#maas 2012-10-01
<roaksoax> bigjools: howdy!! quickly on the squashfs. 1. THere are no precise squashfs images
<roaksoax> bigjools: 2. I don't mind changing to a different path, though the current start of maas-provision wouldn't; allow to install them in the correct path
<roaksoax> bigjools: i wanted to serve these through tftp, but there's a bug in the tftp server that doesn't allow to download more than 32mbs
<roaksoax> bigjools: but allenap said it was decided that it wasn't something important to address for 12.10
 * roaksoax gone
<Daviey> bigjools_: did you see roaksoax's comment?
<Daviey> (before you pinged out?)
<bigjools_> Daviey: I didn't ping out but I did start using a different machine.  And no I didn't see it.
<bigjools_> good morning BTW
<Daviey> morning'
<Daviey> bigjools_: as yes, sorry.. misread
<Daviey> 04:06 < roaksoax> bigjools: howdy!! quickly on the squashfs. 1. THere are no precise squashfs images
<Daviey> 04:07 < roaksoax> bigjools: 2. I don't mind changing to a different path, though the current start of maas-provision wouldn't; allow to install them in the correct path
<Daviey> 04:07 < roaksoax> bigjools: i wanted to serve these through tftp, but there's a bug in the tftp server that doesn't allow to download more than 32mbs
<Daviey> 04:08 < roaksoax> bigjools: but allenap said it was decided that it wasn't something important to address for 12.10
<Daviey> 04:08  * roaksoax gone
<Daviey> 05:44 -!- bigjools-afk [~quassel@60-241-216-21.static.tpgi.com.au] has joined #maas
<Daviey> (i prodded the tftp issue myself, and seemed a pain.)
<bigjools_> Daviey: I feel quite strongly that we should not expose the whole TFTP tree over HTTP
<Daviey> i didn't realise we were..
<bigjools_> we aren't, but roaksoax's branch makes it so
<Daviey> bigjools_: does it actually expose it, or make everything 'work' over http?  Ie, does it include directory listings ?
<Daviey> being able to wget, seems desirable.. directory listings seems not so nice
<bigjools_> the tftp stuff is currently nicely isolated
<bigjools_> it needs to stay that we so we can change it if we want in the future
<bigjools_> mixing in unrelated images into the TFTP tree that are not being served over TFTP is pretty nasty
<bigjools_> Daviey: what are you thinking?
<Daviey> bigjools_: Well, we would be serving it purely under tftp, but the tftp server craps out at >32MB files
<Daviey> so, using http as an alternative.
<bigjools_> Daviey: I know :/
<Daviey> I mean,http could go away, when tftp is fixed.
<Daviey> but it does seem quite desirable to support http.. or certainly ftp.
<Daviey> unrelated, pxe-kexec supports ftp as a alliterative to tftp
<Daviey> alternative*
<Daviey> I agree that directory listings, should probably not be exposed in either case.
<bigjools_> ok
<bigjools_> I'll compromise then, if we remove directory listings
<Daviey> sounds good
<bigjools_> I am nothing if not flexible
<Daviey> I've seen the videos.
<bigjools_> =o=
<rbasak> allenap: do we keep HTTP response code constants in a convenient module somewhere?
<bigjools> rbasak: httplib
<mgz> rbasak: there all in htt... bigjools wins
<bigjools> \o/
<rbasak> bigjools, mgz: thanks!
<allenap> rbasak: They're all in... ug.
<Daviey> rbasak: Yeah, you can find them in.. Oh.
<Daviey> (felt left out)
<rbasak> rbasak: they're in httplib you muppet!
<bigjools> :)
<roaksoax> bigjools: around?
<bigjools> rbasak: I am
<bigjools> argh
<bigjools> roaksoax: I am
<roaksoax> bigjools: howdy! hehe
<bigjools> roaksoax: loads of bugs!
<bigjools> see email to -devel list
<roaksoax> bigjools: so I saw the sqaushfs comments, how is that directory listing should be hidden?
<roaksoax> bigjools: as in, you cannot http://MAAS_server/MAAS/images/amd64
<bigjools> roaksoax: turn off the apache option to show directory listings
<bigjools> requires direct navigation then
<roaksoax> bigjools: as in, you cannot http://MAAS_server/MAAS/images/amd64 ->> you cannot access anything, but it shows page doesn't exist
<bigjools> roaksoax: sorry I am not following you
<roaksoax> bigjools: i mean, if you browse "http://MAAS_server/MAAS/images/amd64" for example, no directory listing is provided
<roaksoax> bigjools: which is why i was asking, is that what you meant about disabling directory listings?
<bigjools> roaksoax: It was, yes
<roaksoax> bigjools: right, so instead, MAAS shows a message that page doesn't exist
<roaksoax> bigjools: but no directory listing is provided
<roaksoax> bigjools: btw.. did you confirm that a rogue celery is running? on package uninstallation maas-celery.upstart will remain *but* it will be stopped before the package gets removed
<bigjools> roaksoax: when I reboot there is a rogue celery
<bigjools> and the config file lives on
<roaksoax> bigjools: the config file lives because the package, on upgrade, is removed, not purged
<bigjools> right :/
<bigjools> so we need some custom stuff?
<roaksoax> bigjools: No, we shouldn't need custom stuff. Upstart should not start the package as the init script should be disabled
<roaksoax> bigjools: oh btw... we need to change the approach how we handle config files
<bigjools> roaksoax: how does it get disabled?
<jam> bigjools: when I do 'make run' and ^C I have a maas-provisioner left running, I wonder if that is related to some of the issues you saw
<roaksoax> bigjools: packaging should do that automatically
<bigjools> roaksoax: it didn't happen though :(
<bigjools> jam: no that's a separate bug
<bigjools> but thanks for pointing it out :)
<jam> bigjools: ah k.
<jam> I ran into it as 'cannot make run' because it hangs if the old process doesn't shutdown.
<bigjools> darn - perhaps one of my guys can fix that
<roaksoax> bigjools: I think adding Conflicts/Replaces <= maas XYZ would fix the issue
<roaksoax> bigjools: for maas package itself
<roaksoax> bigjools: so it forces the removal of 'maas' binary before installing the new one
<bigjools> roaksoax: right
<roaksoax> bigjools: so, i need to discuss this with you
<roaksoax> bigjools: we need to change the approach how config files are being handle
<bigjools> ok
<roaksoax> bigjools: as per policy, we shouldn't not be modifying conffiles
<roaksoax> bigjools: even though we did that in precise, we need to stop doing that for quantal
<roaksoax> bigjools: there's a few alternatives
<roaksoax> bigjools: 1. is to stop treating config files as conffiles for packaging purposes. Say, we for example can ship files under /usr/share/maas/confs and then copy the packaging over /etc/maas/ and make the modifications
<roaksoax> bigjools: that way, on upgrade, the file will be automatically overwritteng and new custom configs will be generated
<roaksoax> bigjools: that causes that, if the user makes a modification, then they won't be preserved
<bigjools> that's not nice :(
<roaksoax> 2. the second option is to ship the config files in a different location, make the package modify those new files, and add new config files in /etc/maas that are *only* meant to be modified by the user
<roaksoax> this would be similar to what we do with maas_local_settings.py and settings.py
<roaksoax> however, instead of modifying automatically maas_local_settings.py we would need to modify settings.py, and leave maas_local_settings.py to be modified by the user only
<roaksoax> in the event they modify the file, on upgrade it will be asked whether they want to isntall the new config file or not, if they decide not to, we hsould still make the customizations in the initial settings.py
<roaksoax> (that's an example of course)
<roaksoax> this however, does not cover *.yaml files for the daemons
<roaksoax> 3. add support for .d/ to the config files, so we ship a base conffile, and the modifications we do are placed under a .d/
<roaksoax> 4. (which could go with any of the options above), would be to put the passwords in a different file, such as etc/maas/passwds, and source those files that need them
<bigjools> what is the reason for this policy?
<roaksoax> bigjools: the only big blocker i see here is the *yaml files for the daemons, since we manually specify what config file to use
<roaksoax> bigjools: basically, it can leave you with a broken system. So only users should modify conffiles, rather than package
<bigjools> so basically no "sed -i" ?
<roaksoax> bigjools: Note that a package should not modify a dpkg-handled conffile in its maintainer scripts. Doing this will lead to dpkg giving the user confusing and possibly dangerous options for conffile update when the package is upgraded.
<roaksoax> that's what policy says
<bigjools> what about removing the files from dpkg's list of installed files?
<roaksoax> bigjools: 10.7 expalins this: http://www.debian.org/doc/debian-policy/ch-files.html
<roaksoax> bigjools: we can do that, which is what option 1 or 2 try to achieve partly
<roaksoax> bigjools: when you install in etc/ they are treated by conffiles, if you install them elsewhere, they are not, and if modified, they will be overwritten in an upgrade
<bigjools> ok
<roaksoax> bigjools: so we remove them form dpkg's list of conffiles
<roaksoax> bigjools: we can simply do that, but then we would have to overwrite the config file each time we upgrade
<roaksoax> bigjools: and user will lose any custom settings
<roaksoax> bigjools: we can display a message an say that the file was overwritten in let the user know that they should merge their changes if any
<roaksoax> bigjools: and preverse a copy
<bigjools> so many options at 23:38 at night
<roaksoax> bigjools: indeed, we need to digest them and start working on them
<roaksoax> so we can do that tomorrow
<roaksoax> bigjools: for for the .py, we can simply install them in /usr/share/maas/ and symlink to etc/maas/, and packaging would modify files directly in /usr/share/maas
<jam> mgz: I just chatted with Francis, essentially rather than trying to do migration via South, just having a simple 'here is a script that will migrate the data' is reasonable for 12.10. And I think that script is essentially 1 line of SQL. So it isn't the first thing we want to do, but when we get to it, it should be pretty straightforward.
<jam> mgz, jelmer: the first priority is to get async building of the node_tags table working (that allows much better scaling stories)
<jam> and then search
<jam> which I'll talk to huw tomorrow about.
<jelmer> jam: okay
<mgz> ...it would be nice is south actually worked though...
<bigjools> roaksoax: ok I'm dead beat here, it might be useful if you summarise in an email and we can look at it later
<mgz> this is at least quite amienable to just being done as a short update script
<roaksoax> bigjools: i will be around tongight so we can discuss it then
<bigjools> ok
<smoser> ok. just a hint for anyone watching rsyslog logs on the maas server
<smoser> (which are sent during install of a node)
<smoser> pid=$(pidof rsyslogd); while date -R && sleep 5; do sudo kill -HUP $pid || break; done
<smoser> basically, rsyslog buffers log files
<smoser> so if you dont tell it to flush the logs, your 'sudo tail -f /var/log/maas/rsyslog/192.168.77.58/2012/10/01/messages' is going to look like its not doing anything.
<jam> mgz: it would be nice if it 'just worked' but if you have to pass special flags to the upgrade, it is going to be a bit hard to just do it in the packaging. So having a separate script should be really straightforward in comparison.
<roaksoax> allenap: howdy
<rbasak> smoser: hey, is there a plan for BOOTIF_DEFAULT at the moment?
<allenap> roaksoax: Hello.
<rbasak> roaksoax: what's the current schedule for the maas-enlist SRU, please?
<smoser> rbasak, it shoudl be there now. in the dialy images (not sure about the released)
<rbasak> smoser: awesome!
<smoser> i just have to start passing it in maas
<rbasak> smoser: so BOOTIF_DEFAULT=eth0 (case sensitive)?
<roaksoax> allenap: so I wanted to talk about the comments on the proposed branch
<smoser> rbasak, yes, case matters in unix
<roaksoax> allenap: what do you mean by [1] symlinking
<smoser> :)
<roaksoax> rbasak: i'll upload today
<rbasak> :)
<rbasak> roaksoax: thanks!
<rbasak> smoser: I'll give it a go - thanks!
<rbasak> smoser: I was going to add for ARM only
<smoser> right
<rbasak> smoser: but thinking about it we could just add it universally without a problem
<roaksoax> allenap: you actually want to symlink the preseed files?
<rbasak> smoser: do you have any preference?
<allenap> roaksoax: You want the same template for both i386 and amd64, so you could call it x86_... and create symlinks from i386_... and amd64_... to it.
<smoser> rbasak, it would seem like we could add it globally, yes.
<smoser> http://paste.ubuntu.com/1254041/ is the script
<roaksoax> allenap: right, but why would you wanna do that? where do you expect that to happen?
<roaksoax> allenap: i personally don't tink it would be a good idea to symlink. Maybe inherit, but not symlink
<allenap> roaksoax: Okay, inherit is good too.
<smoser> in theory the BOOTIF_DEFAULT is only read if BOOTIF is not provided.
<roaksoax> allenap: allenap but TBH that adds complexity.
<roaksoax> allenap: [2]., if squashfs_image is not defined in a template then it shouldn't really affect would it?
<rbasak> smoser: and there are no plans to lose IPAPPEND at all on Intel, so I think universal would be fine. We could always move it later if necessary
<rbasak> smoser: I'm just wondering if there's any conflict with biosdevname but I can't see a problem
<smoser> rbasak, the way its being done now, i'm almost certain biosdevname would not affect it at all
<smoser> unless the biosdevname implementation moved to early in initramfs potentially.
<smoser> as it is now, the devices are left up and network/interfaces says they're "manual", meaning they dont get bounced on boot.
<smoser> meaning biosdevname couldn't change their name if it wanted to
<smoser> as that requires taking them down.
<rbasak> ah yes of course
<rbasak> We went over this. Sorry.
<roaksoax> allenap: allenap hold on, back to [1], I think having a 3 level inheritance (or symlinking for that matter) adds lots of complexity.
<roaksoax> allenap: on [2], the definition of self.squashfs_image is the same as what it is with self.proxy and so on, so there's shouldn't be any issues
<roaksoax> allenap: now tests, you can help me with that :)
<roaksoax> rbasak: btw.. now maas node.architecture is "amd64/generic" "i386/generic" and so on?
<roaksoax> rbasak: or =is there a node.subarchitecture
<rbasak> roaksoax: it's "amd64/generic" or "armhf/highbank" for example
<rbasak> roaksoax: we wanted to do it better but django had a few limitations with derived fields
<roaksoax> rbasak: right, but just wondering.... wasn't it better to simply add a node.subarchitecture attribute?
<rbasak> roaksoax: it's a pain to pass it round everywhere then
<rbasak> roaksoax: ideally it'd be a 2-tuple, and django would translate, but it can't do that even though sqlalchemy can
<allenap> roaksoax: Re. undefined self.squashfs_image not being a problem: that's what we'd want a test for, to assert that that's the case.
<roaksoax> rbasak: right. now, are we looking to have an squashfs image for arm?
<rbasak> roaksoax: I have no idea. I only learnt about the squashfs thing yesterday
<rbasak> roaksoax: if it doesn't go in with already with support, I don't think I'll have time to add it
<allenap> roaksoax: You're right about the complexity, because Tempita's inheritance is weird. OTOH, putting too much logic in a single template gets confusing too, and prevents on-site customisation (well, at least customisation that we can upgrade gracefully with).
<roaksoax> rbasak: right, so i don't mind changing the appracoh of using a subarch, however, it makes no sense doing it if we are not gonna be supporting squaqshfs for arm
<allenap> roaksoax: I may not have time to help with tests until this evening, after 2000 UTC. I assume you'll still be around then?
<rbasak> roaksoax: I've just spent the last few weeks removing all hardcodings of "generic". I'd really prefer not to have to do that again if we have to add support later. Can we not have code go in that understands the arch/subarch correctly even if armhf/highbank doens't work?
<rbasak> roaksoax: that is, read the subarch properly instead of assuming "generic" - that's all.
<roaksoax> rbasak: yeah I don't mind adapting it that way, though it will require many modifications if we ship arm squashfs root filesystems
<rbasak> roaksoax: what modifications would be required?
<rbasak> flacoste: it is ok to skip squashfs support on ARM?
<roaksoax> rbasak: because i'm guessing that eh squashfs image would be provided in different archives, naming convention and stuff
<roaksoax> rbasak: currently for quantal it might even be a temporary naming convention, etc
<roaksoax> rbasak: but i'll adequate the script accordingly
<roaksoax> allenap: no worries i won't work on it till later today
<roaksoax> probably you'll be gone by then
<rbasak> roaksoax: that makes sense and that's OK - I've had to do the same for various other ARM things
<roaksoax> rbasak: so, things like this in the preseeds would need to be changed: {{if node.architecture in {'i386', 'amd64'} }}
<rbasak> roaksoax: the pain is when the full arch and subarch isn't passed around internally so selectively doing the right thing becomes very hard because the predicate isn't available and I have to change how entire sections of code understand arch/subarch
<rbasak> roaksoax: that needs to be in {'i386/generic', 'amd64/generic'} already
<rbasak> roaksoax: or node.architecture.split('/')[0] in {'i386', 'amd64'} - I don't mind which
<rbasak> roaksoax: the important thing is that the full arch/subarch detail is available at that point, so that if statement can change later without having to re-architect the call stack that led to that point
<roaksoax> rbasak: ack. I';ll update my stff accordingly. Though that part is on the proxy stuff for ports.ubuntu.com
<rbasak> roaksoax: ah. That'll be a bug. I need to fix that - thanks
<roaksoax> rbasak: alrigh then :) thanks for the input
<roaksoax> smoser: howdy!! were you able to look at sending the power_type stuff at ipmi
<roaksoax> err
<roaksoax> at the commissioning process/
<roaksoax> in*
<smoser> roaksoax, no. i have not.
<smoser> :-(
<roaksoax> smoser: hehe, alright I think we can get that done by the end ofthe week
<smoser> yes.
<smoser> this epheemeal stuff is a PITA
<smoser> and has been sucking time from me
<smoser> we're *almost* there.
<smoser> i was testing my fix for oauth time sync, but it appears to be broken :-(
<roaksoax> boomer :(
<flacoste> rbasak: the trick is that squadfs makes a big performance difference for provisioning time, so would be really good to have decent provisioning time on ARM
<flacoste> rbasak: i'm fine to doing it last though
<flacoste> rbasak: better to have working but slow arm support, than half-working fast one
<rbasak> roaksoax: do squashfs images exist for ARM at the moment? What would be involved in getting them produced?
<roaksoax> rbasak: no they dont. Daviey was the one who worked on them
<smoser> flacoste, are you making statements above based on experience ? or on conjecture ?
<smoser> "big performance difference" in particular
<flacoste> smoser: conjecture
<flacoste> smoser: we are still awaiting that "one number" from Daviey ;-)
<smoser> when i pressed cjwatson for it, he said "maybe 1 minute"
<smoser> that being part of a 10+ minute install
<flacoste> smoser: only one minute difference between squashfs and d-i?
<smoser> yes
<flacoste> is that experience or conjecture?
<smoser> that was in cjwatson's head when he initially enabled this.
<smoser> err... from his memory of testing it when he enabled it.
<smoser> so its not a hard result either
<flacoste> right, but come from a least some tests
<flacoste> that's disappointing
<smoser> but it seems to me that if your definition of "big" is > 10% you will likely be disappointed.
<flacoste> but still worth a real test
<flacoste> indeed :-)
<smoser> roaksoax, http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/revision/678
<smoser> i've actually tested the oauth time sync issue now. and the url there shows the additional fix needed.
<roaksoax> cool
<smoser> rbasak, did you test BOOTIF_DEFAULT ?
<smoser> if so, i'm gonna do an upload to archive with that.
<rbasak> smoser: not yet, sorry
<rbasak> smoser: got a few things in my queue to test :-/
<smoser> yeah.
<smoser> i'll test it here. i really already have, bug you are the *actual* target.
<jtv> smoser: can I just ask you something about the hostname/domain kernel options?
<smoser> sure
<smoser> domain is gone afaik
<jtv> Gone?
<jtv> *Please* don't tell me that it's meant to be included in hostname now...
<smoser> i dont really recall.
<smoser> well, i guess its not gone.
<smoser> its an option to the debian installer.
<smoser> it probably uses it to determine the fqdn
<jtv> smoser: well my question was: if no domain is given, do I need to provide "local" or somesuch, or can I just leave it out?
<jtv> Hmm... it wouldn't be used to set the search domain, would it?
<jtv> For dns resolution?
<smoser> i dont know.
<smoser> http://hands.com/d-i/
<smoser> see 'domain' there. tells us that it is an alias for 'netcfg/get_domain'
<jtv> Very helpful.
<jtv> This here seems to say that we should pass a default domain if none is given: http://www.hps.com/~tpg/notebook/autoinstall.php
<jtv> (I'm taking a very limited view of the information because my question is exactly that)
<smoser> it would seem that you should pass something, yes.
<jtv> And then the next question is what that would be: "local" or "local.lan" or the enlistment domain...
<smoser> what is "enlistment domain" ?
<jtv> The default domain that nodes go into on enlistment.
<smoser> (i'm looking at netcfg source, and to me it seem slike it would get along without this variable, but then the domain portion of a hostname will not be written to e/tc/hosts)
<jtv> Well the hostname and domain are separate variables.
<jtv> And our Node.hostname includes the domain.
<smoser> which ends up meaning that things that do 'hostname --fqdn' will do dns resolution to figure that out, which is indeerminable.
<smoser> are you thinking of some reason that you'd want it to differe from the enlisted nodes?
<jtv> Yes: the fact that the hostname is initialized to include that domain by default.
<jtv> From which it follows that if the hostname does not include a domain, it is the result of a deliberate action by an admin.
<smoser> jtv, i'm sorry. i cna't come up with a really good suggestion. i think in general you do not want that empty.
<smoser> but i dont know.
<smoser> i know that every time i muck with hostname in cloud-init it breaks something
<jtv> Not very encouraging.
<jtv> I think, given that enlistment domain has a reasonable well-defined goal and this is not it, and given that we only end up with a domain if people have chosen not to accept the one that was set there, the only option we have left is to default to "local."
<smoser> its not entirely un-reasonable to specify none there. really.
<smoser> but your argument seems reasonable to me.
<jtv> The code I had prepared leaves it out if no domain was given, but it sounds as if that's really really undesirable.
<jtv> We can't have any questions on an automated remote install!
<jtv> smoser: is there any meaningful distinction between local and local.lan?
<smoser> http://en.wikipedia.org/wiki/.local
<smoser> it seems maybe inappropriate then for us to use 'local' there
<smoser> jtv, just to be forthcoming, i'm not at all a dns expert. i'm googling same  as you
<smoser> http://superuser.com/questions/117056/how-to-choose-a-sensible-local-domain-name-for-a-home-network
<jtv> smoser: I don't see why it would make "local" inappropriate.  The wikipedia article makes it sound like a sensible, reasonably standardized choice if you can't get an actual proper domain.
<smoser> lots of things specifically say not to use .local
<smoser> because that is basically taken by avahi/mdns and you are explicitly *not* mdns or avahi
<Daviey> smoser: erm, we actually DO want local
<jtv> "Taken" in what way?
<jtv> We're publishing these names on mdns, right?
<smoser> i'm not sure. i dont understand why we'd want .local
<smoser> we're using real dns. and real domain names.
<smoser> err... real dns.
<jtv> We don't want it, but we have to pick something if humans have chosen not to leave the domain name in there.
<jtv> Daviey: you seem to know more about this... please speak!
<Daviey> jtv: No, i'm just a daft manager. :)
<jtv> and allenap, you say a node's domain name should always be taken from its nodegroup... that's not going to be quite the same thing as its zone name, is it?
<Daviey> mDNS and maas-controlled-dns is mutually exclusive, right?
<allenap> jtv: A node/nodegroup doesn't really have a zone name. A nodegroup has a domain name and it has a network. Every node in its network is in its domain, but it's not necessarily true that every node in its domain is in its network, because more than one nodegroup can have the same domain (well, once I land my wretched branch they will).
<jtv> I think I was confused with ddns there for a moment.
<jtv> allenap: "name" was supposed to be its zone name, not its domain name (not that I really know the difference)
<Daviey> jtv: Yeah, so mDNS (.local) and DDNS should also not be used together
<jtv> Daviey: so you're saying that at least it's guaranteed that mdns won't interfere with whatever we do?
<Daviey> no, i'm saying it musn't ;)
<Daviey> I have NFI if it does.
<Daviey> <-- helpful
<jtv> Ahem  :)
<allenap> jtv: Yeah, that's what I thought until the weekend. The "zone name" for a node in a node group might be "example.com." or "168.192.in-addr.arpa.", depending on what context you're thinking of a node in.
<jtv> Hnyargh.
<jtv> That doesn't sound helpful in excessive measures either.
<allenap> Daviey: Where DDNS equals Dynamic DNS (updates)?
<Daviey> jah!
<jtv> allenap: if the zone name can be multiple things depending on how we happen to be thinking at the moment, what is it we're storing in NodeGroup.name?
<allenap> jtv: The domain name, so "maas.canonical.com" for example.
<jtv> Why didn't we know this?
<allenap> jtv: Don't know. Last week was the first time I'd worked on this code, I think.
<jtv> allenap: so... NodeGroup.domain _is_ the domain name then?  And the domain name that is included in the hostname should be ignored?  Any idea why we have the latter?
<allenap> jtv: What? Where did NodeGroup.domain come from?
<jtv> Sorry, NodeGroup.name
<allenap> jtv: Yeah, that sounds right, and I have no idea why the latter has been getting in. Left hand / right hand probably.
<jtv> Urgh.  And we show that to users and invite them to edit it.
<allenap> jtv: We should validate that it's a valid hostname. I wonder if Django has such a form field?
<jtv> Our code inserts the domain name.
<jtv> So I doubt form validation is going to resolve this particular matter.
<allenap> Gah.
<jtv> However at least one thing now becomes clear:
<allenap> It's a prophylactic for next time.
<jtv> an enlisting node (which has no nodegroup) can use the enlistment domain, and an installing one (which does) can use the nodegroup's.
<jtv> I guess a commissioning node (for which there is some special-casing) would naturally fall on the enlistment side.
<allenap> jtv: We ought to be able to figure out which nodegroup any node belongs to by virtue of which network its in... assuming we're managing the network.
<allenap> If not, then what you said makes sense.
<allenap> s/its/it's/ ETIRED
<allenap> jtv: I have to go now, but I'll be back later.
<jtv> It's not quite as easy as it seems to figure out which nodegroup is in.
<jtv> I can't be here when you get back: it's the wee hours.
<allenap> jtv: Okay, get some sleep, see you tomorrow!
<jtv> Can't wait that long -- it'll be after lunch for me when you get back.  :(
<jtv> I think my only viable route right now, given what you said, is to go with what I just described.
<allenap> jtv: If you're still there we can discuss some more. (I had to put Robin in the bath and then bed.)
<jtv> allenap: I'm still sort of here!
<jtv> Updated my branch.
<allenap> jtv: I'll review.
<jtv> Thanks.
<jtv> Changes aren't too big.
<jtv> I think enlistment_domain makes a lot more sense this way.
<jtv> But we need to address the weirdness between those two  domains.
<wkharold> Bug #1028448 (service try to contact zookeeper based on adress no ip) is there a work around?
<ubot5> Launchpad bug 1028448 in juju "service try to contact zookeeper based on adress not ip" [Undecided,New] https://launchpad.net/bugs/1028448
<wkharold> we are trying to bring up a private cloud and this is blocking us
<mgz> wkharold: is that maas related? in general the solution is to use the openstack provider against openstack, rather than the ec2 one.
<mgz> ...also posting the same question in multiple channels simultaniously is annoying >_<
<mgz> #juju is probably the better place
<wkharold> sorry, wasn't sure ... should've asked. this is a private cloud on raw hardware. trying to do a simple test to see if things are working before doing the openstack dance
<mgz> so, it's not that bug then? just the same symptom?
<wkharold> it appears to be that bug ... gettting the same machine-agent stack trace on the blade chosen for the mysql deploy
<mgz> but is juju on raw hardware using the maas provider?
<wkharold> yes
<mgz> does dns work otherwise?
<wkharold> yes ... I'll verify w/my sys admin
<wkharold> tnx
<mgz> if you can't raise anyone else in here, post full details with for conf etc to the maas list, when people are awake they'll respond
<allenap> jtv: +1
<jtv> Thanks allenap
<mercsniper> anyone available to offer assistance?
<roaksoax> allenap: i don't think you are still around are you?
#maas 2012-10-02
<roaksoax> bigjools: waking up early lately :)
<bigjools> roaksoax: it's actually late for me - bad night of twins
 * bigjools otp
<roaksoax> aw :(
<bigjools> roaksoax: where are you guys with the ipmi config?
<roaksoax> bigjools autodetect? needs integration will be done by EOW
<roaksoax> bigjools is trunk broken? i cannot make run
 * bigjools has not tried make run for a while
<roaksoax> lol
<bigjools> packaging baby, packaging
 * roaksoax bbl
<roaksoax> bigjools: around?
<bigjools> in body, yes
<roaksoax> bigjools: you prefer to leave the discussion about the packaging for tomorrow ?
<bigjools> roaksoax: yeah if you don;t mind - I have a million other packaging bugs to fix
<bigjools> I might be more compos mentis tomorrow
<roaksoax> bigjools: cooil, anything ai can help with?
<bigjools> roaksoax: always! :_
<bigjools> :)
<bigjools> roaksoax: bug 1059556, bug 1059459
<ubot5> Launchpad bug 1059556 in MAAS "/etc/init/maas-celery.conf not removed on upgrade" [Critical,Triaged] https://launchpad.net/bugs/1059556
<ubot5> Launchpad bug 1059459 in MAAS "Existing DHCP server not stopped" [Critical,Triaged] https://launchpad.net/bugs/1059459
<bigjools> bug 1059416
<ubot5> Launchpad bug 1059416 in MAAS "When upgrading from the maas package, the cluster controller package doesn't detect MAAS_URL but it could" [Critical,Triaged] https://launchpad.net/bugs/1059416
<bigjools> bug 1059569
<ubot5> Launchpad bug 1059569 in MAAS "Impossible to start cluster controller as maas user" [Critical,Triaged] https://launchpad.net/bugs/1059569
<bigjools> take your pick
<bigjools> I need to fix all these today
<bigjools> oh and bug 1059453
<ubot5> Launchpad bug 1059453 in maas (Ubuntu) "The celery cluster worker is not properly stopped" [Critical,Triaged] https://launchpad.net/bugs/1059453
<roaksoax> bigjools: I think this would solve the first one: http://paste.ubuntu.com/1255220/
<bigjools> yeah we talked about that, thanks, I'll stick it in
<roaksoax> bigjools: i'll test it
<roaksoax> bigjools: i'll grab the latest from PPA and start pushing my fixes there, and if they get fixed I will propose abranch
<bigjools> cool
<roaksoax> i wanted to fix these stuff this morning but got stuck with something else
<bigjools> brb
<roaksoax> bigjools: bug #1059569 is weird, becuase installing the sudoers file should have solved the issue. I think i actually found myself against that error to before
<ubot5> Launchpad bug 1059569 in MAAS "Impossible to start cluster controller as maas user" [Critical,Triaged] https://launchpad.net/bugs/1059569
<roaksoax> gonna check the logs
<bigjools> roaksoax: it's not a sudoers problem
<roaksoax> oh i see, what does the maas-provision command require as root?
<bigjools> roaksoax: we either need to not enforce that maas-provision runs as root, or make it start the celeryd as a different user.  the latter is problematic as we'd have to put more packaging smarts through
<bigjools> I don't know why it is required to be root
<roaksoax> bigjools: i'm pretty sure the old maas-celery didn't have that problem
<roaksoax> and was running as non-root
<roaksoax> so should be the same thing
<bigjools> roaksoax: no - the process is a wrapper around celery now
<roaksoax> bigjools: right, so that shouldn'y have changed anything
<roaksoax> bigjools: so maybe you guys are storing something on a location which is non maas owned?
<roaksoax> bigjools: are logfiles being pre-created with correct permissions?
<roaksoax> maybe is that
<bigjools> roaksoax: you don't understand, let me explain again
<bigjools> maas-provision now starts the cluster celeryd.  maas-provision needs to run as root, but celery needs to run as maas
<bigjools> the upstart conf starts the maas-provision wrapper, not celeryd
<bigjools> if conf is set to run it as maas user, maas-provision bails out as it checks to see if it's root
<bigjools> I don;t know why that check is there
<roaksoax> bigjools: right, I see the point now
<roaksoax> bigjools: so that's design falw then
<roaksoax> bigjools: maas-provisiond should be starting the daemon
<roaksoax> or something
<roaksoax> bigjools: the problme is that we are using a wrapper both a config/admin tool to do certain stuff (such as install pxe images), and you are also using to start a daemon
<roaksoax> and that should not be done that way
<roaksoax> IMHO
<bigjools> I agree
<roaksoax> a dameon is different from a command line tool
<bigjools> it's not a daemon
<roaksoax> bigjools: hence, maas-provision wrapper makes sure we run it as root to do the necessary operations
<roaksoax> bigjools: maybe a good idea would be to have another wrapper that starts the daemon
<roaksoax> without the check
<roaksoax> but either way, celery should be started differently IMHO
<bigjools> I might just remove the check. it's pointless
<roaksoax> bigjools: it is not
<roaksoax> bigjools: it displays nasty errors when run as user
<bigjools> if it needs root, it'll fail anyway
<roaksoax> bigjools: it needs root to be able to write directories
<roaksoax> bigjools: if you do maas-provision -install-pxe-image as normal user it will display a nasty error that should not be displayed
<roaksoax> then, the root check should be done within provisioningserver code
<bigjools> the check should be done elsewhere then
<roaksoax> the wrapper simply checks that the user has appropriate permissions to execute those operations
<bigjools> in fact the code should catch the nasty errors
<bigjools> and display a nice one
<bigjools> wrapping the whole script in a root check is wrong IMO
<roaksoax> k, etiher way i don't agree with having a command line tool starting a daemon
<roaksoax> bigjools: IMO, it is not, because the wrapper runs code that is (or was) meant to be run as root only
<roaksoax> so it should make the check
<roaksoax> but yes, the check should go in the python code now
<roaksoax> that a command line tool starts a daemon
<roaksoax> the daemon should be independent from a command line tool
<bigjools> "meant to be run as root only" - I disagree :)
<roaksoax> bigjools: the wrapper belongs to /usr/sbin
<roaksoax> why? because it messes up with the system
<roaksoax> it does not only mess with the environment of a user
<roaksoax> for that reason, is mean to be run as a privileged user
<bigjools> maas-provision is a command utility. we never intended it to be turned into a root only tool
<roaksoax> bigjools: right, but look, maas-provision is a command line utility that interacts with MAAS operations that require privileged user
<bigjools> yes
<roaksoax> hence, it is a tool that belongs in usr/sbin
<roaksoax> which is meant to be run as root
<bigjools> but not exclusively as rot
<bigjools> root
<roaksoax> bigjools: ok, let e rephrase, privileged user (this means users under sudo).
<roaksoax> bigjools: the check done, is for root user, and sudo users
<roaksoax> bigjools: for example, apachectl can only be run as priviliged user
<roaksoax> maas-provision is exactly the same thing
<roaksoax> the check it has, is to check it is a privileged user
<bigjools> basically we have friction between upstream's design and requirements, and Ubuntu's policies
<roaksoax> bigjools: I dont agree :). These are not Ubuntu policies. These are Operating System policies
<roaksoax> daemons and interaction with daemons are meant to be done by privileged user
<bigjools> the fact that you want it root-only and in sbin is not ubuntu policy?
<roaksoax> bigjools: maybe adding parameter for user/group to run the celery dameon are needed for maas-provision
<roaksoax> bigjools: thta's OS principles, not only ubuntu's
<roaksoax> fedora, redhat, debian, etc etc etc
<bigjools> that was one of my potential solutions yeah
<bigjools> well look, we'll do a separate command
<roaksoax> ok, longtemr thoguh, it should support the arguments for user/group I think
<roaksoax> because it is better to tell the daemon under what user to run
<roaksoax> that way
<bigjools> that's upstart's job isn't it? :)
<roaksoax> bigjools: not completely no
<roaksoax> bigjools: twistd allows passing the user/group that you would like the daemon to run as
<roaksoax> celery doesn't
<roaksoax> but since celery is now run by maas-provision, then maas-provision should probbaly allow passing user/group to run the daemon as
<bigjools> yeah
<bigjools> perhaps that's a better solution here
<roaksoax> indeed
<bigjools> easier, I mean.  jtv ^ ?
<roaksoax> bigjools: and as you mentioned, the other operations should check they are run as root, and display an error accordingly
<roaksoax> bigjools: so that we ditch that check in the wrapper
<jtv> I don't quite understand: if the command to start a cluster will run celery as another user than it itself runs under, that makes it a privileged command.  What's the argument for making it a command separate from maas-provision?
<roaksoax> jtv: right, it makes it a privilleged command, that the upstart job will run as root, and will tell the daemon "run as the user"
<bigjools> jtv: maas-provision is root-only according to roaksoax, so either it runs celery as a different user or another command run by maas has to start it
<roaksoax> jtv: so for exmaple, the upstart job for txlongpoll is invoked as root, but we are telling it that the daemon (twistd) should run as the 'maas' user group
<roaksoax> bigjools: not just according to me. They way I see it, it is a design principle. If a utility messes up with the system, it is a privileged utility
<bigjools> roaksoax: ok you and others :)
<roaksoax> bigjools: hehe isn't it fun to be in between different worlds ? :)
<bigjools> jtv, if it's easy to become a different user in Python, taking a -uid and -gid cmd line option seems ok to me
<jtv> But for maas-provision, or for a separate command?
<bigjools> jtv: for maas-provision's start_cluster_controller
<roaksoax> jtv: the "separate command" was simply a wrapper to start_cluster_controller
<jtv> OK, so keep start_cluster_controller but make it startable as another user.  And obviously that means it'll have to fork() as well.
<bigjools> jtv: indeed
<bigjools> jtv: but fork() is good because it means I can start tracking it properly in upstart
<bigjools> since at the moment it fails to do so
<jtv> Well I say "obviously" -- I don't actually know there's no other way if you're root.  :)
<jtv> Yeah, I just hate to build on two variables.
<jtv> It'd be bloody annoying to find that the solution for changing the user makes the upstart problem impossible to solve!
<bigjools> jtv: fork(), change user, exec().  Sorted.
<bigjools> roaksoax: are you landing that Conflicts: change or shall I?
<roaksoax> bigjools: i'm about to test it
<bigjools> roaksoax: excellent, thanks
<bigjools> roaksoax: what is the best way of stopping/disabling the existing dhcp server when installing maas-dhcp?
<jtv> Added note: have to do it synchronously, I think, or it may keep ours off its port.
<bigjools> yeah
<bigjools> easy
<bigjools> roaksoax: I presume invoke-rc followed by update-rc
<roaksoax> bigjools: yes, if it is using sysvinit script
<bigjools> it is
<bigjools> iirc
<roaksoax> bigjools: this should probbaly done on preinst
<bigjools> roaksoax: oh actually it's an upstart conf
<bigjools> how do you disable those?
<roaksoax> bigjools: probably by an override, let me check
<roaksoax> bigjools: http://upstart.ubuntu.com/cookbook/#disabling-a-job-from-automatically-starting
<bigjools> ah thanks
<bigjools> roaksoax: should that be a package-installed file?
<roaksoax> bigjools: not necessary as long as it gets removed on postrm
<bigjools> ok
<bigjools> so preinst and postrim
 * bigjools hacs
 * bigjools eats first in fact
<jtv> roaksoax: do we still need a pid file, with upstart watching our celery process?
<roaksoax> jtv: i think we do
<roaksoax> jtv: for twistd
<roaksoax> but not for celery
 * roaksoax checks
<jtv> Oh, I meant specifically about celery.
<roaksoax> no we dont
<roaksoax> old celery was exec /usr/bin/celeryd --logfile=/var/log/maas/celery.log --loglevel=INFO --beat --schedule=/var/lib/maas/celerybeat-schedule
<roaksoax>  so no pid
<jtv> Yeah, I'm asking whether that's right though.  Because we had trouble tracking it with upstart.
<roaksoax> jtv: you can specify it if you like
<jtv> I don't like, I'm just not sure whether it's needed!
<roaksoax> jtv: i don't think it is mandatory
<roaksoax> we have been without a pidfile
<roaksoax> so i don't think we need it
<jtv> But it hasn't been working properly, which is why I'm asking.
<roaksoax> jtv: idk TBH. :(
<roaksoax> bigjools: uhm conflicts/replaces didn't seem to work
<jtv> Because it's going to get more complicated to maintain a pidfile with the setuid.
<bigjools> roaksoax: damn :(
<roaksoax> bigjools: maybe just conflicts! i'll give it a try
<bigjools> jtv: it's not working because the wrapper execs in a way that doesn't use fork but somehow changes its pid
<jtv> So we know for sure it's not just lack of pidfile?  Good.
<roaksoax> yes I
<roaksoax> i'd agree with bigjools
<roaksoax> s/i'd/i
<bigjools> jtv: once you use fork we can tell upstart about that
<bigjools> and it'll DTRT
<jtv> Yes, that's the hypothesis.  I just don't know enough about what's going on to be confident, which is why I ask around.
<jtv> bigjools: well, it turns out one of my hypotheses yesterday was right.  We do fork something else before we fork off celery.
<jtv> It's ifconfig.
<bigjools> jtv: gnargh!
<jtv> Yup.
<jtv> So this might be a good time to consider netifaces.
<bigjools> yes
<bigjools> JFDI
<bigjools> and it's in main, unbelievable
<jtv> I was wondering why changing fork-exec to fork-setuid-exec would fix anything :)
<jtv> In good news, it was the tests that brought this to light.
<jtv> What needs doing to add the netifaces dependency?
<bigjools> 2 places: required-dependencies and packaging dependencies
<bigjools> I'm doing loads of packaging changes, I'll do it there for you
<bigjools> just in cluster controller right?
<jam> jam
<jam> morning all
<bigjools> hey jam
<jtv> Hi jam
<jtv> bigjools: yes, just cluster controller.
<jtv> It's lunchtime.  I'm going to try to get some equipment.  Almost bought a nice little lightweight machine, but found that unity probably won't support its AMD graphics chip.  :(
<jtv> Oh crap here comes the rain.  Perfect timing as always.
<bigjools> heh
<jam> bigjools: question for you about using celery
<bigjools> yup
<jam> for the Tag table, we know that when we create a tag, it can take 10s for 10,000 nodes to get checked.
<jam> So we want to push that out into an async job
<jam> (cron or rabbit comes to mind)
<jam> bigjools: how do we integrate that with the system?
<bigjools> jam: pretty easy, I'll take you through the steps:
<bigjools> 1. Edit src/provisioningserver/tasks.py and define the async job
<jam> k
<jam> bigjools: just as a function with @task on it?
<bigjools> it's a function decorated with @task
<jam> k
<bigjools> 2. keep the function small - just call out to something in provisioning server so that the tasks file stays small
<jam> (and jinx, I guess :)
<jam> sure, I expect the logic to stay on the Tag object.
<bigjools> 3. from the appserver code, call it with funcname.apply_async(queue=nodegroup.uuid)
<bigjools> and it'll run on that nodegroup's worker
<jam> bigjools: so we have to loop over N nodegroups?
<bigjools> jam: if your nodes are across multiple groups, yes
<jam> (and the nodegroups talk directly to the DB, or we need another bit to send a message back)
<bigjools> to talk back you need an API method defined that the nodegroup can call
<jam> k
<bigjools> s/nodegroup/celeryd/
<bigjools> the celeryd caches credentials for the API
<bigjools> see report_boot_images() for an example
<jam> bigjools: so what data do the individual nodegroups have access to? (they don't have a local DB do they?)
<bigjools> they don't
<bigjools> they can only see what is passed in the task call
<jam> do they have access to the main? or we need 2 apis, one to scrap the data out of the db, and another to put it back in?
<bigjools> what are you doing, exactly?
<jam> bigjools: we need to take the hardware details, filter it by the xpath statement, and determine a list of Nodes that match particular Tags
<bigjools> ah ok then let's do this in another way
<jam> the long-term idea is that the hardware details would sit on the cluster controler
<bigjools> we have a worker local to the region for this stuff
<jam> however, they don't have anywhere to put it *today8
<jam> bigjools: sounds like the worker we'd like to talk to for the initial implementation, then.
<jam> bigjools: do you set that by the 'queue=...' stuff?
<bigjools> jam: yes
<bigjools> so if you look in etc/celeryconfig_common.py
<bigjools> you'll see WORKER_QUEUE_DNS = 'celery' and WORKER_QUEUE_BOOT_IMAGES = 'celery'
<bigjools> Add another one, WORKER_QUEUE_<something>
<bigjools> set to "celery", which is the name of the region's worker
<bigjools> then when you define the task you can decorate it like this:
<bigjools> @task(queue=celery_config.WORKER_QUEUE_<thing>)
<bigjools> which ensures it always gets sent to the same worker
<jam> so I'm guessing we should have a similar config as WORKER_QUEUE_TAG_??? and then also point it at 'celery'
<jam> right
<bigjools> and you change the apply_async() to a delay()
<bigjools> so function.delay()
<bigjools> and pass any args in the delay params
<jam> bigjools: ah, ok
<bigjools> it sounds like you'll need some way of querying the API for the data you need
<bigjools> and to store it back
<jam> bigjools: so we still need api calls, but it is just run locally.
<jam> NP
<bigjools> right
<bigjools> make sure the api calls run *quick* then the job can take a bit longer
<jam> bigjools: is there plans in say 13.04 to add some sort of local db/storage on the cluster controllers?
<jam> or is the goal for them to be fully stateless?
<bigjools> possibly.  Mark wants to do that.
<bigjools> we might thrash that out a bit more in COP
<jam> bigjools: so how fast is quick? <1s, <100ms, <10ms?
<bigjools> <1s is probably ok
<jam> (shoot for 100ms assuming that sometimes load will cause it to be 1s?)
<bigjools> what I mean is - optimise the DB queries :)
<bigjools> otherwise you're negating the effects of offloading jobs from the appserver
<jam> bigjools: so we have 2 options, we can compute the XPATH content in the DB (postgres has native support for it), or we can compute it in the local process using LXML. It is slightly faster to do it in the DB, because we don't have to read out the XML content, but obviously it adds more load in the DB
<jam> either way we can probably do it in batches
<jam> so we update, say 1000 nodes at a time.
<bigjools> jam: yeah batching is important here for 1000s of nodes
<bigjools> not sure if we have a batching mechanism in Piston
<bigjools> jam: if the xpath runs quick enough on the DB, there's no problem with that
<jam> bigjools: 'quick enough' is 6s for 10,000 nodes.
<jam> so batching at 1,000 nodes would be 600ms, as a ballpark sort of thing.
<bigjools> mmmm might be pushing it
<bigjools> yeah
<jam> but you still spend 6 total seconds doing the work
<bigjools> you'll probably have to manually batch
<jam> or in the 100k node space, you are talking 60s total CPU time.
<bigjools> this is fine to dump on the celery worker
<bigjools> since most of the time it is not doing much ATM
<bigjools> the DB is potentially more precious resources-wise
<jam> bigjools: and the long term goal is to dump down to the individual regions
<bigjools> right
<jam> I'm wondering it should be architectured as: 1) grab the list of nodes that need touching right away, 2) farm out to each nodegroup for their respective nodes
<jam> 3) pass just the node ids
<jam> 4) the provisioning_servers then batch requests for XML content, and parse it.
<jam> 5) and poke the results back into the DB as they go.
<jam> The main trick with all this, is knowing when everyone is "done"
<jam> but I think that gets us a good CPU story
<jam> because the processing is properly farmed out across the 'scaling' portion of the system.
<jam> We have a small bandwidth issue tod
<bigjools> you can do it like that if you like - I'm just saying that the region celeryd is fairly idle
<jam> today, because the data is in the central DB
<jam> but it is where we want it to be done in 13.04 or whatever.
<bigjools> but scaling out is a good plan for the future
<jam> bigjools: from what I can tell, lshw -xml is about 24kB * 10,000 nodes is 240MB being downloaded in these requests.
<jam> Is that too much load?
<jam> on the DB
<bigjools> sounds ok
<jam> (note that they probably compress fantastically well, if that is possible in the API)
<jam> though adding that may be premature optimization at this point.
<jtv> The rain's letting up.  I'm going to make my shopping run.
<jtv> bigjools: any chance of a review?  https://code.launchpad.net/~jtv/maas/netifaces/+merge/127420
<jtv> While I'm out?
<bigjools> one sec
<bigjools> yes, OTP
<jtv> Wow, that's fast!
<jtv> :P
<Fajkowsky> hello guys, I have problem with my maas server I described it here - http://www.tinyurl.pl?QSsXP6oX - its "no instance data found in start local"
<bigjools> Fajkowsky: your link is invalid
<Fajkowsky> http://askubuntu.com/questions/195115/nodes-cant-connect-to-server-after-bootstrap
<Fajkowsky> I try install again maas and add nodes, maybe it works this time.
<bigjools> Fajkowsky: I am just adding an answer
<Fajkowsky> ok
<allenap> rvba: Sorry, I just switched your branch back from Approved.
<rvba> allenap: a) I don't agree with what you say in your comment.  b) we've got many other place where it's done this way.  If we've going to change that behavior, then we better do it everywhere.
<rvba> places*
<allenap> rvba: Well, start here then. That we've done it wrong elsewhere doesn't make it right.
<rvba> allenap: indeed.  But I'm really not sure it's right :)
<rvba> allenap: ok you might be right about that idempotent stuff.  But, if you don't mind me saying that, your method is wrong here.  You should let me land that branch, file a bug about the problem, an *then* someone will pick up that branch and change all the delete methods.
<rvba> allenap: because that bug is marked critical and changing the behavior of all the 'delete' methods is not.
<rvba> It's 'high' at best.
<allenap> rvba: Okay, fair enough; I just wanted to save you an extra branch and proposal for what seems like a simple change.
<rvba> allenap: I just wanted to land a quick fix to unblock Diogo.  And also, who's tell you that I'm going to be the one doing that extra branch ;).  No, seriously, I've got to focus on my UI stuff right now.
<rvba> s/tell/telling/
<allenap> rvba: I wasn't suggesting you fix everything. I was just commenting on this one proposal. I changed it back to Needs Review to stop Tarmac from landing it, so that we could talk about amending it, saving the effort of filing a bug, proposing a merge, making a card, etc. I didn't realise it was a bigger problem. I'm sorry that I caused such distress! :)
<rvba> allenap: no distress, really, but if your goal was to save time for the both of us, then that's a fail :). We definitely will have to file a bug for the other delete() methods so fixing up this one and have it half-done is not the way to go I think.  "Ni fait ni Ã  faire", here is another nice french expression :).
<jtv> Ah, is the "one more small change" worm rearing its ugly head?
<jtv> Meanwhile, I wonder why one of our postinst scripts uses '[a-z]\{0,\}' instead of '[a-z]*'
<jtv> Actually, it's pretty sick to "sed" for an entry in a multi-line dict.  What if some other dict contains the same key?  I think I'd rather define a variable, and have the dict refer to that.  The dict will not need any patching, there'll be no leading whitespace, etc.
<jtv> allenap: maybe you can help me out with this question.  What is the relationship between the patch we have in packaging that sets the db password to 'maas', and the postinst code that sets a proper password?  Why do we have both?
<allenap> jtv: Eugh, I don't know. Intriguing.
<jtv> Or wtf dbc_dbpass comes from... it seems to be coming from thin air.
<jtv> Bit annoying when you want to verify that it really consists only of ASCII letters and digits.
<jtv> (I don't see why the regex needs to check for exact contents of the string: '[^']*' is both easier and afaict, more appropriate)
<jtv> Review needed!  Spot the stupid mistake that I keep repeating... https://code.launchpad.net/~jtv/maas/pkg-bug-1060095/+merge/127451
<allenap> jtv: I'll do it; I haven't a clue about dbc_wibble so I'll keep my head down reviewing.
<jtv> That's good too.  :)  Thanks.
<rvba> jtv: Andres will probably know the answers to these packaging questions :).
<jtv> Yeah.  Wonder if he's here yet... I need to leave soon.
<jtv> roaksoax, are you here?
<allenap> jtv: It's definitely maas_local_settings not maas_local_settings.py, right?
<jtv> allenap: what is?  I think the module is maas_local_settings and the file is maas_local_settings.py.
<jtv> The latter is what we mostly refer to.
<allenap> jtv: It's chmod'ing /etc/maas/maas_local_settings <-- no .py
<jtv> Ooo!
<jtv> Fixed.
<jtv> Good thing you spotted that.
<allenap> jtv, rvba, anyone: I have to stop work early today to collect Robin from school, at 1340 UTC, but I'll be back around this evening, after 1900 UTC.
<jtv> I'll be away for the rest of the night as well.
<jtv> I'll email roaksoax with my questions.
<mgz> allenap: as worded, I don't think bug 1060114 is true, or in need of fixing.
<ubot5> Launchpad bug 1060114 in MAAS "DELETE operations are not idempotent" [Medium,Triaged] https://launchpad.net/bugs/1060114
<allenap> mgz: Fancy rewording it? ;)
<allenap> mgz: Ah, I've just seen your comment on the proposal. Interesting point.
<allenap> mgz: I like your explanation. Can you add it to the bug and mark it Invalid?
<mgz> allenap: sure
<allenap> mgz: You don't happen to be in town today?
<allenap> I'm trying to find an excuse to go out for lunch.
<mgz> allenap: alas :)
<mgz> you're making lunch on thursday though, right?
<mgz> well, as it, getting there, kat will be making it :D
<mgz> *as in
<mgz> bah, I can't type for toffee
<allenap> mgz: Yeah, I'll be there, probably at about 1200, because Chantal and I are collecting a puppy before then.
<mgz> bring the puppy to lunch! :D
<mgz> ...how house trained is it going to be?
<allenap> mgz: Not very, and it has to stay at home until it's fully inoculated so I'm told, so a few weeks.
<mgz> ;_;
<allenap> mgz: We can't even take it out for walks at first. It gets to shit in the back garden, that's it :)
<jtv> Okay folks, I'm off for tonight.
<allenap> Cheerio jtv.
<jtv> nn
<jtv> When Raphers comes up to breathe, tell him I wish him good luck and God speed with his UI branches.  :)
<jam> jelmer: how's it going? Want to skype some more?
<jam> mgz: how are things looking for you?
<jam> allenap: I have some questions about how the celery stuff works. are you knowledgable or should I chat with bigjools tomorrow?
<allenap> jam: rvba is the man on that front, but I might be able to help.
<jelmer> wb jam
<jelmer> jam: making some progress, trying to get some tests going
<jam> the big question is that the workers need someone to call 'record_secrets' before they can do any work
<jam> but the examples we have seem to just drop the request on the floor if they don't haev the secrets yet
<jam> but how do we make sure that the work is always done
<jam> do we need to put the work in the DB as 'todo'
<jam> and then have it marked 'done' by the callback?
<jam> rvba, allenap^^
<mgz> jam: landed some tag sample stuff
<jam> and if the work is still pending who retries it?
<rbasak> smoser: ping, for when you get in
<rvba> jam: (was out having lunch) Celery can handle that for you I think, you just need to get the task retried instead of dropping it on the floor if the secrets are not there.
<jam> rvba: how does one signal that?
<rvba> jam: see rndc_command in src/provisioningserver/tasks.py.
<jam> (and not have it go into a death spiral trying to retry the job)
<rvba> http://docs.celeryproject.org/en/latest/userguide/tasks.html#retrying
<jam> rvba: ah the function itself calls func.retry
<rvba> Yeah.
<mgz> okay, now I feel a lot less clever about not using real xpaths...
<jam> mgz: we have code that asserts they are valid
<jam> like when allocating a new node
<jam> mgz: quick poll for you
<mgz> I'll fake that up here
<jam> we know that after updating a tag
<jam> there will be some time where the node <=> tag mapping is inconsistent.
<jam> Is it better to drop all mapping, and then add them slowly?
<jam> or is it better to slowly update the nodes to make it consistent?
<mgz> ah, interesting
<jam> A user fixes a tag definition, and then goes to start nodes based on that tag.
<jam> is it better to not match anything, and get tried again later
<jam> or better to match something that it used to match, on the premise that a small update is unlikely to actually change the node set dramatically.
<jam> I'm tending towards the former, on the premise that 'juju deploy' will keep trying to fulfill your request
<jam> and then you won't accidentally deploy on machines that no longer match the new tag vaule.
<jam> value.
<mgz> I feel a common "update the same name" case might be to slightly tweak what gets selected
<mgz> so, having an interval where the stuff that used to be selected and will still be selected is not, is probably the most suprising
<jam> mgz: right 'has_nvidia' and you realize that sometimes it is NVIDIA and sometimes nvidia
<jam> so you want to make it case insensitive
<jam> mgz: the flip side is 'big >= 2GB' and you update it to 'big >= 4GB' and you deploy, and it picks a 2GB node.
<mgz> that seems less harmful, having an interval where the old rules apply
<jam> mgz: my argument for 'not selected for a while' is that it will eventually be selected and retrying the query will get you the right value.
<mgz> this is true, if we stick with only using tags as positives
<jam> mgz: so I think jelmer's preference is to do delta updates
<mgz> the other option...
<jelmer> jam: if it's retrying it might be that the tag is still only half up-to-date when the set of nodes is non-empty
<mgz> is to set an update time
<mgz> and if asked to acquire something with an old tag set, defer until the tags are fresh
<jam> jelmer: it may be only have up-to-date, but everything that is tagged *definitely* matches the new value.
<jam> so a 'big' node will always have 4GB after changing the definition.
<mgz> acquire is expected to be slow
<jam> mgz: we do have an updated field already (it is part of the model)
<mgz> adding 6s (currently) or per-cluster update delay, would not be too bad
<jam> and we can do other work to detect if all nodegroups have given their responses.
<mgz> if a cluster doesn't do any acquire till after it's done tag updates, you'd still get some responses fast
<jam> (essentially some sort of db entry that indicates whether nodegroup X has responded for tag Y)
<mgz> (slow/big clusters would tend not to be used straight after tag updates, but that's not terrible)
<jam> mgz: in the end, this sounds like something to bring up at the standup, and we can move forward with what we have until then.
<jam> I have the small feeling that a coin flip may end up involved somewhere.
<mgz> yup, all options are reasonable, and changing strategy after we have running code is fine.
<jam> allenap: also, for the 'nodegroup' changes, are all nodes going to have a nodegroup?
<jam> mgz: it does change the api, 'add_nodes' vs 'update_nodes', etc.
<allenap> jam: They should do already, but let me check.
<rvba> jam: it is already the case.
<jam> rvba: I see that it says 'this should be not null, but we can't do that yet'
<smoser> rbasak, here now.
<allenap> rvba: Node.nodegroup is null=True, blank=False  -- what does that mean on a foreign key?
<allenap> rvba: Ah, I've read the comment now ;)
<rvba> allenap: :)
<smoser> hm... did we get a bug for my issue with daily ppa not installable?
<mgz> bah, pants: TypeError: 'Tag' instance expected, got <Tag: big>
<mgz> south wants to make my life miserable
<mgz> can I refactor that bit out...
<jam> mgz: I think you can do add(Tag.id) but I might be wrong.
<mgz> jam: I'm trying to share code with the migration... when they're really using different model classes
<rbasak> smoser: good morning!
<mgz> passing it in might be th lease stupid option...
<rbasak> smoser: the precise daily ephemeral image seems to work. I didn't add BOOTIF_DEFAULT and it works.
<mgz> *the least
<rbasak> smoser: I had a question though. Looking at making maas-import-ephemerals work for ARM without hackery. It should be a pretty minor change, but I can't remember what we said about adding multiple subarchs in ephemeral images
<rvba> mgz: I don't think you should try to share code with the migration because the migration runs with the models being in a special state (when not all of the migrations have been run yet) so if you change that code later, it might break the migration.
<rbasak> smoser: is the plan to have one image for all of armhf? In which case, how does that work with install_tftp_image?
<smoser> rbasak, it only works for you because you get a dhcp response on both interfaces.
<smoser> s/both/all/
<mgz> rvba: this is all basically a trigger
<mgz> when hardware_details field changes, update these other fields
<smoser> rbasak, for multipel sub-arches' the plan would just to change the format of the tar file so that there were multiple kernels pulled out.
<rbasak> smoser: OK. I want to break it before I fix it in order to test the fix
<smoser> rbasak, fi you want more than "highbank" then ephemeral images and import scripts probably need work.
<mgz> I don't know how to do the migration correctly if setting that field does not also do the (db contents specific) updates to the rest of the stuff
<rbasak> smoser: OK, but does that mean that the maas-provision install-pxe-image interface will need to be changed? Currently it expects an entirely different image directory per subarch, which would be a waste of space I think
<mgz> and it's complex, writing it in two places, one of which is not tested, does not appeal to me.
<rvba> mgz: ok, I don't know what particular problem you're facing but it's just that we've been bitten by that once :)
<mgz> I could just copy the current code though
<smoser> one could just complain that "subarch" should never have been invented :)
<rbasak> :-)
<rbasak> One day we'll have device tree and it'll all go away
<smoser> i'd have to look at it, rbasak, but yeah, our goal would be to have one ephemeral iamge
<smoser> and multiple tftp'd kernels
<rbasak> smoser: ok. So I think it's not practical to get multiple subarch support into maas-import-ephemerals right now, so I think I'll add some kind of ugly hack for highbank and leave it at that for 12.10. Is this OK with you?
<smoser> what is it that you're concerned about?
<rbasak> smoser: right now it doesn't import highbank at all, since "generic" is hardcoded. I need to have this fixed by 12.10, that's all.
<smoser> and it doesn't fail?
<rbasak> smoser: I've been patching it by hand up to now
<rbasak> smoser: and now I'm at the point where I'd like to get it working in trunk and in the package
<rbasak> smoser: and have it import all three arches by default
<sanderj> How come MaaS have a bug not including udev in the initrd script? So it dosn't work to boot up remote vms?
<sanderj> scripts/init-bottom/udev should look like this: http://paste.ubuntu.com/679222/
<rbasak> sanderj: what exactly is the problem?
<sanderj> rbasak, when booting up a vm from spx with maas.. I get the error: cannot find "bnx2-mips-09-6.2.1a.fw"
<rbasak> what's spx?
<sanderj> when I unpacked the initrd and added two lines to the abow script, then it workes.
<sanderj> pxeboot
<sanderj> I mean
<sanderj> Sorry
<rbasak> which lines did you add?
<rbasak> any idea what bnx2-mips is?
<sanderj> network card driver
<sanderj> . /scripts/functions
<sanderj> wait_for_udev
<sanderj> Those two lines I added
<sanderj> into scripts/init-bottom/udev
<rbasak> which initrd did you modify?
<sanderj> The initrd for the kernel maas uses to boot up remote machines.
<rbasak> There are a few
<rbasak> What was the path?
<rvba> allenap: I'm reviewing your branch: https://code.launchpad.net/~allenap/maas/query-strings-and-request-bodies/+merge/127479
<allenap> rvba: Thanks.
<rvba> allenap: I think it will fix many bugs in one go :)
<allenap> rvba: Yeah, I hope so. What *was* I thinking before?
* flacoste changed the topic of #maas to: 1 week until Final Freeze | Discussion of upstream development of Ubuntu's Metal as a Service (MAAS) tool | MAAS jenkins: https://jenkins.qa.ubuntu.com/job/maas-trunk/
<sanderj> rbasak, /var/lib/maas/ephemeral/precise/ephemeral/amd64/20120424
<sanderj> rbasak, /var/lib/maas/ephemeral/precise/ephemeral/amd64/20120424/initrd
<rbasak> sanderj: so I think that's not a maas specific issue, although perhaps maas is the only thing to exhibit it. I think the file you modified is coming from initramfs-tools or some package like that
<rbasak> sanderj: can you check for existing bugs and if you can't find one, then please file a bug report with as much detail as you can?
<sanderj> rbasak, where do I find maas bugs?
<sanderj> rbasak, I think the bug is corrected with ubuntu, but not in maas.
<rbasak> sanderj: https://bugs.launchpad.net/maas/
<sanderj> No bugs reported when searching for udev there.
<jam> mgz, jelmer: https://code.launchpad.net/~jameinel/maas/get-nodes-for-group/+merge/127484
<mgz> jam: looking
<sanderj> rbasak, ok, now it's reported. Let's hope it helps.
<rvba> matsubara: the fix for 1060079 should land any minute now.
<matsubara> rvba, great
<roaksoax> rvba: howdy!! Is make run broken?
<roaksoax> err upstream trunk borken?
<rvba> roaksoax: jenkins seems happy, let me check
<rvba> roaksoax: everything seems fine, what error are you seeing?
<roaksoax> rvba: an import error. give me a sec an i'll show you
<rvba> roaksoax: apt-get install python-netiface maybe?
<roaksoax> rvba: django.db.utils.DatabaseError: relation "maasserver_config" does not exist
<roaksoax> LINE 1: ..._config"."name", "maasserver_config"."value" FROM "maasserve...
<roaksoax> rvba: also this:  it is not being cleaned: setlock: fatal: unable to lock /run/lock/maas.dev.cluster-worker: temporary failure
<rvba> roaksoax: the first error makes me thing that the database is simply not there because "maasserver_config" is an old table that was created months ago.
<rvba> roaksoax: the second one: sometimes celery gets stuck, just killall the celery processes and remove that file (/run/lock/maas.dev.cluster-worker).
<roaksoax> rvba: ImportError: No module named netifaces --> even though it is installed
<roaksoax> there's really something weird going on
<roaksoax> rvba: what do you think might be causing that in my ssytem?
<rvba> roaksoax: difficult to say remotely, can you make sure first that all the processes have been killed?
<roaksoax> rvba: yeah they were, I just had rebooted my machine. IU'm trying again now
<mgz> yeay, working migration
<mgz> now that wasn't at all painful or owt
<mgz> ...which still falls over if run from scratch...
<mgz> hm, the maasserver gets migrated before anything at all is done with metadataserver? that's fun.
<mgz> well, should be easy to fix (ho ho ho)
<roaksoax> rvba: http://pastebin.ubuntu.com/1256069/
<roaksoax> rvba: that's weird postgresql is running
<rvba> roaksoax: you can try to remove db/.s.PGSQL.5432
<roaksoax> rvba: the file doenst exist
<rvba> roaksoax: not even db/.s.PGSQL.5432.lock ?
<roaksoax> nope
<rvba> roaksoax: can you wipe out the db or is there things in there you'd like to keep?
<roaksoax> rvba: so i rm -rf db/ and make run again and same issues
<rvba> roaksoax: does 'make sampledata' work?
<roaksoax> it did
<roaksoax> i'm re-making the whole environment
<rbasak> smoser: I can't get the precise daily armhf image to fail. I tried disabling dhcp on eth1 and it still works
<rbasak> smoser: but whichever way, please could you promote it to a release? Then I can have maas-import-ephemerals import armhf by default without breaking anything
<smoser> rbasak, can you send me a console log ?
<rbasak> smoser: ...of it working?
<smoser> because i dont like that i dont think it should work
<rbasak> OK
<rbasak> smoser: err
<rbasak> smoser: BOOTIF seems to have arrived
<rbasak> Well this is embarrasing
<rbasak> smoser: I'll check after this test but it seems that IPAPPEND support might h ave appeared in the lastest highbank U-BOot update
<smoser> well that'd be neat.
<Daviey> lol
<roaksoax> rvba: make sampledata works
<smoser> roaksoax, ping
<rbasak> smoser: yeah IPAPPEND now works!
<smoser> well that is nice indeed.
<rbasak> smoser: one note though. With DHCP disabled on eth1, everything worked all the way through except after the installation cloud-init hung
<rbasak> smoser: and at that stage it's a local boot so no BOOTIF expected
<rbasak> smoser: thoughts?
<smoser> it looks like little intel-jr is growing up.
<rbasak> :-)_
<smoser> precise
<smoser> ?
<smoser> quantal should work
<rbasak> Yes
<rbasak> I'll check quantal
<smoser> you need a couple fixes SRU'd to precise
<smoser> (they happen to be in that maas-ephemeral ppa , so you could try just adding that ppa and seeing if that makes it magic)
<rbasak> smoser: which ppa please?
<rbasak> smoser: ephermal-fixes?
<smoser> https://launchpad.net/~maas-maintainers/+archive/maas-ephemeral-images
<rbasak> thanks!
<roaksoax> smoser: pong
<rbasak> smoser: is sources.list expected to be wrong in precise still?
<smoser> rbasak, yeah
<smoser> roaksoax, so ... we are to fix ipmi today
<rbasak> smoser: the PPA seems to have fixed it
<smoser> you did an install that quickly?
<rbasak> I just updated
<roaksoax> smoser: ok
<rbasak> smoser: unless first boot is expected to be different for some reason?
<roaksoax> rbasak: quick question... if we upgrade from precise to quantal, how is te change in arch from i386 to i386/generic is handled?
<rbasak> roaksoax: the db migration just slaps /generic on the end of all existing nodes' architectures
<roaksoax> rbasak: cool!
<roaksoax> allenap: aroung?
<roaksoax> around*
<rvba> roaksoax: allenap will be back around 1900 utc.
<roaksoax> rvba: alright. So you might help then :). So for the power related stuff, IPMI specifically, we need to ship a especial config file that will be used every time an IPMI command is executed
<roaksoax> rvba: were do you think the file should live, and how should it be referenced
<roaksoax> i was thinking it should live with the templates
<roaksoax> rvba: oh wai,t you were the one who did the power stuf right?
<rvba> yeah
<roaksoax> rvba: hehe alright so it its you then :)
<rvba> roaksoax: "live with the templates"â¦ which templates? :)
<roaksoax> rvba: http://paste.ubuntu.com/1256181/
<roaksoax> rvba: live with the templates, as in this file should be placed in the power template directory, (were ipmi.template is)
<rbasak> roaksoax: mind that workaround stays in the right place in that patch
<rvba> roaksoax: src/provisioningserver/power/config/ seems like a good place to me
<rbasak> roaksoax: also ipmi-chassis-config might need it too. Probably worth testing with </dev/null
<roaksoax> rbasak: the workaround is not being affected
<rbasak> roaksoax: your patch puts it on the wrong line
<roaksoax> rbasak: ack!
<roaksoax> rbasak: k
<roaksoax> rbasak: i see now
<roaksoax> :)
<rbasak> :)
<roaksoax> rvba: do you know anything about jtv's changes on running maas-cluster-celery under user/pass?
<roaksoax> user/group?
<rvba> roaksoax: yeah, I think he has landed that branch.
<rvba> roaksoax: is there a problem with that change?
<rvba> roaksoax: btw, did you manage to get rid of that weird problem you had?
<roaksoax> rvba: yteah i did manage to get rid of the problme i had
<roaksoax> rvba: and i think there is, i saw maas-cluster-celery bein unable to start
<roaksoax> but now it starts :/
<rvba> Be aware of the fact that it does not start celery instantly, if first need to get the credentials from the region controller.
<roaksoax> rvba: yeah so it seems to do that
<roaksoax> but still
<roaksoax> let me check again
<rvba> roaksoax: arg, it seems the packaging has not been cleaned up: debian/maas-cluster-controller.maas-cluster-celery.upstart still contains setuid maas/setgid maas
<rvba> roaksoax: so apparently, he made the upstream change, but not the related packaging change.
<rvba> roaksoax: and /usr/sbin/maas-provision will refuse to do anything if not run as root.
<rvba> roaksoax: so now I wonder, how can you see it runningâ¦ ?
<roaksoax> rvba: i didn't the problme was that it failed to start on an upgrade
<roaksoax> hence leaving the package unconfoigured
<roaksoax> rvba: i'll upload a fix
<rvba> Ok, thanks.
<rbasak> smoser: just finished testing quantal daily ephemeral for quantal install. It works all the way through without problems.
<rbasak> smoser: can we get the precise armhf daily converted to a release soon now please? Is there anything blocking this? Then I can land a change for maas-import-ephemerals to import armhf by default.
<smoser> i can do that now.
<rbasak> thanks!
<rbasak> Although you changed 'cloud-init boot finished' to 'Cloud-init v. 0.7 finished' so my expect script didn't match for success :-P
<mgz> rbasak: yeah, smoser likes doing small cloud-init changes to break your scripts :)
<mgz> I do now work with lucid->quantal though
<smoser> recent changes can be more blamed on harlowja
<smoser> sorry about that.
<mgz> smoser: the main annoyance is needing to support all versions, the new improved is very nice but having to work with lucid still makes it painful...
<mgz> I'd like to use the file injection stuff now josh implemented it, but having two versions loses the simplifications...
<roaksoax> rvba: do you think it is possible to ship maas_local_settings.py in /usr/share/maas and have that source somthing in /etc/maas/local_settings.py or similar?
<roaksoax> rvba: or have a proper conffile?
<mercsniper_> is the cloud init package still out of date for 12.04?
<melmoth> mercsniper_, i do not know, but last time i used maas (2 weeks ago) i did not experience problem with cloud-init
<mercsniper_> k
<rvba> roaksoax: that's possible but that would simply add one additional level of complexity.
<rvba> roaksoax: what would it give us?
<roaksoax> rvba: ok so the problme is that we can no longer modify /etc/maas/maas_local_settings.py in packaging
<roaksoax> rvba: so it should only be modified by the user
<roaksoax> not the package
<roaksoax> so if the user makes changes, on upgrade it gets prompted
<roaksoax> rvba: it would be like adding .d support
<roaksoax> rvba: cause if it is not done upstream, i'm gonna have to patch maas up and do it myuself
<roaksoax> Daviey: what was the package with the .d support for cobbler?
<Daviey> roaksoax: it was a custom thing i did
<Daviey> never ht the archive, still in my PPA
<rvba> roaksoax: that's definitely doable, we've got a tiny utility method to do that so the change should be simple.  Could you please file a bug with the details.  I'm probably not gonna be able to do it right now but Gavin might be able to do it later today.
<mercsniper_> Is there a way to remove a node if the status is commissioning?
<roaksoax> rvba: cool, that way, we can simply edit /usr/share/maas/maas_local_settings.py or whatever in packaging
<roaksoax> rvba: and if the user wants to override something he can do it
<mercsniper_> melmoth: are you using 1204 or 12.10 for your maas
<melmoth> 12.04
<mercsniper_> still getting commissioning...
<melmoth> mercsniper_, did the machine restarted ? did you try to reboot it ?
<melmoth> i did not understand exactly how the power management thing worked, so some times i just rebooted nodes.
<mercsniper_> I tried rebooting, I get cloud-init-nonet killed(300)
<melmoth> some times they rebooted on their own (i m still puzzled as to why :-) )
<mercsniper_> while it boots i get a landscape-client is not configured
<melmoth> i think i have seen that but did not looked like a real problem.
<melmoth> but i did not see a cloud-ini-nonet killed
<mercsniper_> init: clount-init-nonet main process (269) killed by TERM signal
<melmoth> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1015223
<ubot5> Ubuntu bug 1015223 in cloud-init (Ubuntu) "cloud-init-nonet main process killed by TERM signal" [Low,Triaged]
<melmoth> dont panic i think it says :)
<melmoth> hmm, but there s a link to https://bugs.launchpad.net/ubuntu/+source/maas/+bug/992075 that is worryiing
<ubot5> Ubuntu bug 992075 in maas (Ubuntu) "Commissioning status persists with cloud-init 0.6.3-0ubuntu1" [Undecided,Confirmed]
<melmoth> mercsniper_, is the date and time the same on the maas server and the machine you try to comission ?
<mercsniper_> hm.... i would imagine
<mercsniper_> is there a standard login for nodes?
<melmoth> not untill they got the public juju ssh key injected
<melmoth> if you want a password login you need to make your own image (i dont know how, i saw once a doc telling how to)
<mercsniper_> ah
<melmoth> i m asking about the clock thing because it hit me several time with juju stuff and because of comment 2 in https://bugs.launchpad.net/ubuntu/+source/maas/+bug/992075
<ubot5> Ubuntu bug 992075 in maas (Ubuntu) "Commissioning status persists with cloud-init 0.6.3-0ubuntu1" [Undecided,Confirmed]
<melmoth> mercsniper_, see comment 12 https://answers.launchpad.net/maas/+question/196791
<melmoth> (and what about booting on a live cd and running a ntpade so the local clock is roughly on the correct date and time on next reboot ? )
<mercsniper_> true
<mercsniper_> trying to bootstrap juju, i get an unexpected http 500 code....mean anything to anyone?
<mercsniper_> this directory didnt exist /var/lib/maas/media/storage
<mercsniper_> how long does commissioning take?
<melmoth> couple of minutes
<mercsniper_> then it must not be commissioning properly
<melmoth> the only real long stuff is when you deploy a service, then it s  areal install
<mercsniper_> gonna try virtual box instead of vmworkstation
<roaksoax> smoser: please :) https://code.launchpad.net/~andreserl/maas/packaging_updateS_bzr1134/+merge/127570
<smoser> roaksoax, you copied bug https://bugs.launchpad.net/maas/+bug/1039513 to import-squashfs
<ubot5> Error: ubuntu bug 1039513 not found
<roaksoax> smoser: ack
<roaksoax> yeah we need to do verifications
<rbasak> allenap: thanks for the review!
<allenap> rbasak: Welcome :)
<smoser> https://code.launchpad.net/~smoser/maas/trunk-remove-hostname-kludge/+merge/127571
<smoser> someone can tak that too
<rbasak> allenap: there's https://code.launchpad.net/~racb/maas/arch-detect/+merge/127458 too, and then I'm done :-P
<allenap> Okay, I'll try to look at those both.
<rbasak> thank you!
<rbasak> After that the daily PPA should in theory work for ARM
<allenap> smoser: Is there any way to make set -e work for broken command substitution?
<smoser> what does that mean?
<allenap> wrt. bug 1060411
<ubot5> Launchpad bug 1060411 in MAAS "maas-import-pxe-files does not catch failure of compose_installer_download_files" [Undecided,New] https://launchpad.net/bugs/1060411
<smoser> oh.
<smoser> thats not hte issue.
<smoser> commented in bug.
<smoser> the issue is just 'local' as the declaration succeeds, and that is what is checked.
<smoser> so you just do those on 2 separate lines.
<allenap> smoser: If I do:
<allenap> set -e; echo $(does_not_exist); echo $?
<allenap> I get 0.
<allenap> Ah! But if I do variable=$(something) it will break.
<allenap> Okay, got it.
<roaksoax> allenap: https://code.launchpad.net/~andreserl/maas/use_squashfs_filesystem_2/+merge/127577 --> I adressed 1, the rest is addings tests
<smoser> this is one reason i prefer:
<roaksoax> allenap: if you could take care of that, I'd deeply appreciate it
<smoser>  myfunc && myvar=$_RET || return 1
<smoser> to
<smoser>  myvar=$(myfunc) || return 1
<smoser> in addition to the fact that the second incurs a fork
<smoser> you're welcome to make fun of my hatred of forks. but you'll see why i hate them next time you dist-upgrade.
<melmoth> mercsniper, you install things in virtual machines ?
<roaksoax> smoser: uhmm enlistment doesn't seem to be working :
<roaksoax> :s
<allenap> smoser: Are you sure it incurs a fork, when calling a shell builtin? Try: echo $$ $(echo $$)
<smoser> i'm positive.
<mercsniper> Mel: I am doing this work as a learning experience on my work laptop
<melmoth> how much ram ?
<roaksoax> smoser: did you change the console to which it is displaying the output?
<mercsniper> vmworkstation lets you switch between machines
<smoser> allenap, compare:
<smoser> $ time sh -c 'for i in "$@"; do echo $(echo $i); done' -- $(seq 1 1000)  >/dev/null
<smoser> to
<melmoth> i m using kvm to play with things and learn maas here
<smoser> time sh -c 'for i in "$@"; do echo $i; done' -- $(seq 1 1000)
<melmoth> mercsniper, http://bazaar.launchpad.net/~pierre-amadio/+junk/c6100-jumpstart-maas/view/head:/README.txt
<roaksoax> smoser: nevermind :)
<melmoth> ir you want to give kvm a try instead of virtuabox..Should just work "out of the box"
<allenap> smoser: Yeah, I can see it now; another way to demonstrate it is looking at the process list when running: echo $(read)
<mercsniper_> unfortnately, im on a windows host
<melmoth> hey, good reason to install something new on your laptop :)
<mercsniper_> laptop needs to stay windows per company policy
<mercsniper_> thats why its all virtual
<melmoth> i do it in kvm because it s easier to get one machine with lots and lots of ram, than 10 little ones with switch and wire and stuff
<roaksoax> smoser: updated
 * roaksoax hates chromium crashing
<smoser> reguarding wrap-and-sort, i was only really complaining about the python-netifaces
<roaksoax> allenap: what extra tests did you have in mind for the suqashfs
<smoser> and it turns out i was wrong there anyway
<smoser> :)
<smoser> (i thought that would sort after the ${misc:Depends}
<smoser> i'll ack this because wrap-and-sort is generally a good thing.
<smoser> ah. but you approved already.
<smoser> :)
<roaksoax> smoser: yeah :)
<roaksoax> thanks
<smoser> oh... roak!
<smoser> set -e ?
<smoser> er...
<smoser> set -x ?
<smoser> really?
<roaksoax> smoser: in maas-import-squashfs?
<roaksoax> smoser: yeah I forgot
<roaksoax> smoser: a branch is ready for review that allenap needs to review
<roaksoax> that completes that
<roaksoax> and fixes it
<roaksoax> completes the support and fixes that
<smoser> roaksoax, isntall of maas-dhcp from experimental results in no /etc/maas/dhcpd.conf
<roaksoax> smoser: tbh i hjave not been following up on what they've been doing with maas-dhcp
<roaksoax> smoser: but i'll audit
<smoser> ok. well i'm looking for a way to have a maas functional
<smoser> you suggested experimental
<smoser> that didn't work
<roaksoax> smoser: yeah so that means that's broken somehow
<roaksoax> smoser: i don't think the dhcp server is still functional
<roaksoax> clear
<smoser> matsubara, did you install maas-dhcp recently?
<matsubara> smoser, yes
<matsubara> well
<matsubara> found a bug with the package this morning
<matsubara> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1060237
<ubot5> Ubuntu bug 1060237 in maas (Ubuntu) "apt-get install maas maas-dhcp maas-dns fails" [Undecided,New]
<smoser> matsubara, roak has a fix for that https://code.launchpad.net/~andreserl/maas/packaging_updateS_bzr1134/+merge/127570
<smoser> matsubara, do you have notes available on how you install and configure?
<matsubara> smoser, I follow the checkbox tests and have a local note like this: https://pastebin.canonical.com/75751/
<smoser> where are the checkbox tests?
<smoser> allenap, https://code.launchpad.net/~smoser/maas/trunk-remove-hostname-kludge/+merge/127571
<roaksoax> smoser: you want me to integrate it inside maas-signal right?
<smoser> well, we want maas signal to call it.
<smoser> err..
<smoser> the scripts there to call yours, and then post back the results
<roaksoax> smoser: ok
<roaksoax> smoser: i'm doing this too: http://paste.ubuntu.com/1256796/
<smoser> well config probably shouldnt be executable
<smoser> but other than that i think it looks reasonable
<roaksoax> cool
<smoser> matsubara, how do i enable dhcp?
<matsubara> $ maas-cli api maas node-group-interfaces new master ip=192.168.21.1 interface=eth0 management=2 subnet_mask=255.255.255.0 broadcast_ip=192.168.21.255 router_ip=192.168.21.1 ip_range_low=192.168.21.10 ip_range_high=192.168.21.50
<matsubara> smoser, ^
<smoser> so where do you have those notes?
<smoser> ie, is this part of the "checkbox install" that you mentioned?
<matsubara> smoser, checkbox tests are linked in this doc: https://docs.google.com/a/canonical.com/document/d/1GNrJCL8EyfSw7ypCCYjH0BuIgIEDP2E6Y9Xbb7Gx8rs/edit
<matsubara> and my notes are in the pastebin
<matsubara> and I rely a lot in the shell history as well :-)
<smoser> this *really* needs to not be a private google doc
<smoser> matsubara, and how do you set up a maas user and such ?
<smoser> it seems like you probably have done a lot of things that i want to do
<smoser> and i'm just trying to avoid us both doing them.
<matsubara> sudo maas createadmin --username=admin --password=test --email=example@canonical.com
<roaksoax> smoser: any thoughts "1349210267.806     72 192.168.123.101 TCP_DENIED/403 3728 GET http://192.168.123.2/MAAS/static/images/amd64/generic/quantal/filesystem/filesystem.squashfs - NONE/- text/html"
<roaksoax> ?
<smoser> i would say you are being denied access to that
<smoser> :)
<smoser> check /var/log/apache/*.log
<smoser> (including error)
<roaksoax> smoser: lol yeah I mean, squid-deb-proxy doesn't allow the installer to download the squashfs image
<smoser> i suspect your being expected to oauth
<roaksoax> smoser: any thoughts on how can we fix it?
<roaksoax> smoser: i was thinking on telling squid-deb-proxy to allow access to the maas server in question
<roaksoax> by hacking on the packaging
<roaksoax> but maybe you know of a better way
<smoser> well why is that going through the proxy
<smoser> squid-deb-proxy is explicitly a *deb* proxy
<roaksoax> smoser: because we are telling the installer to use the proxy
<smoser> hm..
<smoser> well that would seem like a bug one way or another
<smoser> either in that we're gelling it there is a generic proxy
<smoser> or in that it is assuming what we said it should use for an archive proxy it can use for other things
<smoser> but yeah, to fix that i guess you will probably have to have it allow proxying of /MAAS/static/images/*
<roaksoax> maybe squid itself  is blocking it
<smoser> roaksoax, i thought htat is what you were saying
<smoser> i'm confused now.
<smoser> were you thinking maas was saying that?
<roaksoax> smoser: i meant squid3 itself (not the instance squid-deb-proxy spawns)
<roaksoax> smoser: but it is squid-deb-proxy
<roaksoax> if I add the IP address of the MAAS server facing that network, it allows it
<smoser> roaksoax, i'll be back in later tonight. (probably 3+ hours from now)
<allenap> roaksoax: I'm looking at use_squashfs_filesystem_2 now.
<roaksoax> smoser: alright, i'll be later here too
<roaksoax> allenap: awesome thank you!
<roaksoax> allenap: are you gonna be at UDS btw?
<allenap> roaksoax: Yeah, you?
<roaksoax> allenap: yeah!! is the rest of the team gonna be there?
<roaksoax> s/team/squad
<allenap> roaksoax: Yeah, I think we're all going. smoser, you at UDS?
<roaksoax> allenap: yeah all of our team is gonna be there
<allenap> Cool :)
<roaksoax> allenap: alright, so I hope you guys don't run away from Peruvian Pisco :P
<allenap> roaksoax: Oh god, I was given Pisco by Nicolas at the Barcelona UDS. I haven't been able to drink spirits since then.
<allenap> I'll give it a go though :)
<roaksoax> lol
<allenap> roaksoax: I've changed is_squashfs_image_present in lp:~allenap/maas/use_squashfs_filesystem_2, and updated the tests. It doesn't test the expansion of the templates, but I need to go and sleep now ;)
<roaksoax> allenap:  is there any example?
<roaksoax> on how to do it?
<allenap> roaksoax: There are general tests for expansion, but not for the specific templates in contrib/...
<allenap> roaksoax: Don't worry about it. It's something we ought to address after 12.10. The confusion about inheritance means that we should revisit this stuff anyway.
<roaksoax> allenap: alright, cool
<roaksoax> thanks for helping out!
<allenap> roaksoax: Pull my branch (it directly follows on from yours), push it up, and I'll +1 that mp.
<allenap> roaksoax: So, sorry I didn't get to this yesterday.
<roaksoax> allenap: no worries :)
<roaksoax> allenap: thank you for helping tho
<roaksoax> allenap: pushed the changes to the MP!! thanks a lot again! and have a good night!
#maas 2012-10-03
<roaksoax> smoser: around already?>
<smoser> here, roaksoax
<roaksoax> smoser: so we are gonna need to pass the "config" to my script, instead of harcoding it
<roaksoax> smoser: so I'm guessing I should pass the config to maas-signal, which will pass it to maas-autodetect-ipmi?
<smoser> what do you mean pass
<roaksoax> smoser: as it tell it where's the template / config
<roaksoax> as in*
<roaksoax> smoser: https://pastebin.canonical.com/75758/
<roaksoax> smoser: that one, we need to tell the location of that template
<roaksoax> template/config
<smoser> i'm sorry. i'm still being dense.
<roaksoax> smoser: bmc-config --commit --filename /location/of/cofig/file.ipmi
<bigjools> hello guys
<roaksoax> bigjools: good morning
<bigjools> I might have done something bad to my debconf database, I am getting this trying to install the maas package after previously purging it: http://paste.ubuntu.com/1255593/
<bigjools> (I need to sh -x the postinst to get that output, otherwise it's fails silently)
<roaksoax> bigjools: what new package are you trying to install? the latest I made available in experimental PPA?
<bigjools> s/it's/it/
<bigjools> roaksoax: one I made myself yesterday
<roaksoax> bigjools: that's fixed then :)
<bigjools> it's not the same problem as the one you fixed
<bigjools> since I'd already found that and tried to fix it and then got held up by this :/
<roaksoax> bigjools: are you referring that this doesn't fix it? http://bazaar.launchpad.net/~maas-maintainers/maas/packaging/revision/113
<roaksoax> see the postinst
<smoser> daily ppa has maas-dhcp installable again
<bigjools> roaksoax: it's hard to see what the effect of the changes are - I was assuming you were referring to the fix for "service isc_dhcp_server stop"
<smoser> bigjools, just curious, what does "fix committed" / "fix released" mean in maas ?
<smoser> bigjools, he was referring to the celery postinst
<smoser> i think
<bigjools> smoser: not a a lot.  Tarmac changes bugs to fix-committed but we don't distinguish that with fix-released on trunk
<smoser> https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1050523
<ubot5> Ubuntu bug 1050523 in maas (Ubuntu) "maas kernel cmdline must include iscsi_initiator" [High,Triaged]
<smoser> so you marked that "fix released"
<smoser> i'm just curios, so i can at least attempt to be consistent.
<bigjools> smoser: yes, they are all fix released if they are in trunk, I am not using fix-committed
<bigjools> then they disappear off bug listings
<roaksoax> bigjools: check the changes made in maas-region-controller.postinst, I removed and if statement
<roaksoax> an*
<bigjools> roaksoax: aha!
<bigjools> nice one
<roaksoax> bigjools: there's a related issue though that I can't find a solution for
<roaksoax> bigjools: if you remove (not purge) and install again, it will show the same error
<roaksoax> bigjools: and I already know how to fix the maas-celery thing, i'll work on it in a bit though
<bigjools> roaksoax: ok - jtv landed a change so we can pass uids I think
<bigjools> bug 1060114
<ubot5> Launchpad bug 1060114 in MAAS "DELETE operations are not idempotent" [Medium,Invalid] https://launchpad.net/bugs/1060114
<bigjools> heh
<smoser> bigjools, https://bugs.launchpad.net/maas/+bug/1058137
<ubot5> Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged]
<smoser> if i just put that in comment 4 into a python script and run it
<smoser> i stack trace
<bigjools> matsubara: that python script works for you?  --^
<smoser> http://paste.ubuntu.com/1257137/
<bigjools> smoser: export DJANGO_SETTINGS_MODULE=path/to/settings.py
<bigjools> /etc/maas/whatever
<matsubara> bigjools, yes, I think I did some path hackery to get it to work
<bigjools> grumble grumble, roaksoax removed my changelog attribution :)
<roaksoax> bigjools: no i didn't :)
<bigjools>  -- Andres Rodriguez <andreserl@ubuntu.com>  Tue, 02 Oct 2012 13:39:47 -0400
<bigjools> you did :)
<roaksoax> bigjools: yeah I made changes to the changelog
<roaksoax> dch -i changes that :)
 * bigjools shakes fist at dch
<bigjools> -u surely?
<bigjools> or -e?
<roaksoax> bigjools: btw... an assumption I am working with if the fact that it doens't matter how many releases we release in PPA
<roaksoax> bigjools: the packaging should remain UNRELEASED
<bigjools> ok
<roaksoax> and only change when we release to archives
<roaksoax> bigjools: but since the changelog is being pilling up I think is ok
<bigjools> roaksoax: well dch -i doesn't change the release it adds a new one, so something changed the attribution I added
<roaksoax> bigjools: because entries of releases that haven't been uploaded to the archives will appear in the changelog even though we didn't
<bigjools> yup
<bigjools> soyuz is great :)
<roaksoax> bigjools: dch -e -DUNRELEASED works?
<roaksoax> errerr dch -e -Dquantal
<bigjools> roaksoax: in r113 the change line was updated, what dch option does that?
<smoser> i tihnk in thie change log there we need to ditch the double bzr branch
<smoser> we get one "for free" from the daily ppa
<smoser> biuld
<smoser> so we should either be pedantic about getting that updated in the code on each commit (or tooling) or just remove the bzr revno from there.
<roaksoax> bigjools: dch -i does it for me
<bigjools> true, but it's needed to build locally from recipe isn't it?
<bigjools> roaksoax: -i adds a new one for me
<smoser> bigjools, well we can just add a script that does the build locally and inserts it.
<bigjools> smoser: just need to fix the debian/rules target I expect
<bigjools> roaksoax: bzr diff -c 113 debian/changelog
<roaksoax> bigjools: dch -e --nomainttrailer
<bigjools> roaksoax: aha
<bigjools> roaksoax: useful, it saves me having to use -k when recipe building :)
<roaksoax> indeed :)
<roaksoax> bigjools: btw.. i already told allenap... no scaping from Peruvian Pisco is allowed at UDS
<roaksoax> smoser: ^^
<bigjools> ?!
<roaksoax> escaping*
<roaksoax> :)
<bigjools> does that involve naked bodies and mud?
<bigjools> otherwise I am totally not up for it
<roaksoax> lol
<bigjools> roaksoax: getting this on the experimental package: http://paste.ubuntu.com/1257167
<roaksoax> bigjools: as an error?
<smoser> ok. this is silly.
<roaksoax> bigjools: can you pastebin a full log?
<smoser> http://paste.ubuntu.com/1257171/
<bigjools> matsubara: can you paste a full hackery for smoser please?
<roaksoax> smoser: do you think i should allow sending several config files rather than just one?
<bigjools> roaksoax: http://paste.ubuntu.com/1257172
<smoser> roaksoax, do you have a reason for doing so
<smoser> ?
<roaksoax> smoser: so if someone sets custom ipmi configs they can be sourced automatically?
<roaksoax> bigjools: yeah that's dbconfig being an asshole i think
<smoser> it doesn't sound unreasonable.
<bigjools> roaksoax: yay
<bigjools> roaksoax: I'll re-purge
<roaksoax> bigjools: yes, just to be sure, remove dbconfig-common too and make sure /etc/dbconfig-common is empty
<roaksoax> bigjools: i discovered that maas.conf under dbconfig-common is preserverd, and we also need to remove that
<roaksoax> bigjools: i'll take care of that along the celery stuff
<roaksoax> along with*
<matsubara> bigjools, smoser: something like this should work: https://pastebin.canonical.com/75761/
<smoser> matsubara, well, thats basically what i did
<bigjools> roaksoax: /etc/dbconfig-common/maas-region-controller.conf exists
<bigjools> weird I'll remove it
<bigjools> roaksoax: on re-install (after removing it) I now get a debconf page telling me it was currently iunstalled and locally mnodified
<bigjools> installed*
 * bigjools installs package version
<matsubara> ok, guys, gotta go. talk to you tomorrow.
<matsubara> have a good night/day
<roaksoax> bigjools: say yes
<bigjools> :)
<bigjools> smoser, roaksoax: how long are you in CPH?
<roaksoax> bigjools: 2 weeks
<bigjools> just UDS?
<bigjools> ah
<smoser> same as roaksoax
<smoser> too long
<roaksoax> smoser: yay!!
<roaksoax> lol
<bigjools> ok plenty of time to meet up for an evening food/drink
<bigjools> yeah too long
<bigjools> my wife is going to self destruct looking after our twins on her own
<jtv> Hi folks
<bigjools> hi jtv
<smoser> i'm really sorry for being a dolt. but some help would be greatly appreciated.
<smoser> http://paste.ubuntu.com/1257179/
<jtv> User isn't a MAAS thing.  It comes with Django.
<jtv> django.contrib.auth.models.User
<jtv> How it got to be in contrib I may learn someday.  How it got to stay there I just won't believe no matter what you tell me.
<bigjools> jtv: should still work
<bigjools> however I get this:
<bigjools>     from django.db import utils
<bigjools> ImportError: cannot import name utils
<bigjools> at the bottom of the trace
<smoser> https://bugs.launchpad.net/maas/+bug/1058137/comments/5
<ubot5> Ubuntu bug 1058137 in MAAS "no way to get first user's first set of api creds" [High,Triaged]
<roaksoax> smoser: http://paste.ubuntu.com/1257181/ with http://paste.ubuntu.com/1257183/ --> what do you think?
<smoser> got it
<bigjools> smoser: phew :)
<smoser> thank you bigjools jtv
<bigjools> np
<roaksoax> bigjools: btw... I think we need to be able to pass kernel arguments
<roaksoax> bigjools: and be able to edit that
<bigjools> pxe template?
<roaksoax> bigjools: console=ttyS0
<roaksoax>  bigjools we need to be able to edit things like that
<roaksoax> bigjools: so yes, it should be added to the pxe template
<roaksoax> bigjools: but should probably be per node
<roaksoax> if one wishes to change that from a node, or set of nodes
<bigjools> roaksoax: how important is this?
<bigjools> should be easy enough to do, but I need to prioritise against the thousand other bugs
<roaksoax> bigjools: ver important i'd say, even sabdlf mention we should be able to edit it. For example, console configuration needs to be done manually directly in the BIOS for ipmi cards
<bigjools> eugh
<roaksoax> bigjools: so they configure serial to be ttyS20
<roaksoax> so maas, as it stands right now, sends console=ttyS1
<bigjools> ok, Node needs a custom "kernel_params" field then, and we can add that to the pxe template
<roaksoax> bigjools: indeed, so we don't really need it to be displayed in the WebUI
<smoser> roaksoax, bigjools well, per-node kernel settings are not all that important really.
<smoser> i opened that bug
<smoser> and i do think they need to be editable
<roaksoax> smoser: right, so we can set global settings, but if the user wants to change them, per node, he needs to
<smoser> but realistically, in a large scale, you're going to have a couple groups of systems
<bigjools> my question is, important for *this release*
<smoser> so per-node would suck anyway.
<smoser> i cant think of them being a hard requirement for this release.
<roaksoax> smoser: right but for example, in sabdlf's cluster I had to manually edit what console to output in maas code, because it was configured differently on the BIOS
<smoser> but the ability to append cmdline arguments without editing python source would be nice.
<bigjools> smoser: I am thinking that the cluster controller would want to set the parameters when it uploads node definitions (in the future)
<smoser> bigjools, ok. heres the whole sucky problem.
<bigjools> all this stuff will/should be automated
<smoser> a.) before a node is enlisted, you know nothing about it. so you can't really customize stuff per-node at that point
<smoser> b.) after that point, you could have some custom settings
<bigjools> roaksoax: https://code.launchpad.net/~julian-edwards/maas/broken-dns-install-bug-1060549/+merge/127621
<smoser> but *really*
<smoser> c.) in a perfect world, give some knowledge of the setup and the node (hw and the like) you could perfectly generate all that anyway.
<bigjools> I disagree with (a) since the cluster controller may know all about its hardware already
<smoser> in which case you're to 'c'
<smoser> so you dont need them
<smoser> :)
<smoser> roaksoax, i think you'd have been fine with the ailbiyt to append cmdline settings.
<smoser> right?
<smoser> and i would have in all my testing been ok to do the same.
<smoser> that would releieve a lot of pain
<roaksoax> bigjools: looks good, make sure +    add_user_group
<roaksoax> 49	+
<roaksoax> is ident'd correctly
<roaksoax> smoser: huh?
<bigjools> roaksoax: GRARGH.  The rest of the file has got TABS in it
<bigjools> I used 4 spaces
<bigjools> TABS ARE EVIL
<roaksoax> smoser: btw... i'm just gonna pass maas-signal with an argument with the json stuff, such as maas-signal --config --power-setings <json format>
 * roaksoax uses tabas for shell scripts and spaces for python  code
<bigjools> TABS ARE EVIL
<bigjools> :)
<roaksoax> xD
<bigjools> because you end up with what you see on the MP
<bigjools> do you care if I s/\t/    /g ?
<roaksoax> bigjools: different schooling I guess.. I was always a TAB user
<roaksoax> bigjools: go for it
<bigjools> we're very particular about our code formatting in the Launchpa^WCloud Engineering team
<roaksoax> smoser: yeah I think we need to be able to append that
<roaksoax> smoser: adding the per-node isn't really that big of a deal, we don't even need to display it in the UI, just be able to modify it per node from maascli
<bigjools> roaksoax: can you approve the MP please
<roaksoax> bigjools: done!
<roaksoax> smoser: ok I think it is done i just need to test it
 * roaksoax heads home
<roaksoax> smoser: i'll go home and test it on the HP mini servers
<bigjools> cheers
<bigjools> jtv: it looks like you didn't fix the upstart conf to expect a fork, or am I missing something?
<jtv> bigjools: I didn't get around to looking into that; I assumed it already expected a fork but got the wrong one.
<bigjools> jtv: ok I 'll sort it out
<jtv> Based on what you said, I thought the packaging branch already expected a fork.
<bigjools> jtv: I tried it but it hung, I'll try it again
<jtv> :(
<bigjools> now there *is* a fork :)
<bigjools> it should work
<jtv> There was a fork before as well.
<jtv> That hasn't changed.
<jtv> But I removed the spurious fork that preceded it.
<bigjools> hmm
<bigjools> jtv: something else is forking
<bigjools> jtv: http://paste.ubuntu.com/1257202
<jtv> Well it can fork right off.
 * bigjools straces it
<bigjools> ARGH
<bigjools> it's the sodding wrapper script
<bigjools> jtv: when starting a python script, straces shows that it forks off this lot: "id, sh, ldconfig, sh, ldconfig, sh, uname"
<bigjools> wtf
<jtv> wtf indeed
<jtv> So celeryd is a wrapper script?
<bigjools> no
<bigjools> maas-provision
<bigjools> the one installed by the package that checks for uid==0
<jtv> Grrr
<bigjools> so basically python is forking a load of stuff off before it starts your actual code
<bigjools> unless it's hiding in maas-provision?
<jtv> I bet it's import side effects.  :(
<jtv> I tried this and found no forks: $ strace python -c 'print' 2>&1 | grep fork
<bigjools> I am doing this:
<bigjools> sudo PYTHONPATH="/usr/share/maas${PYTHONPATH:+:}${PYTHONPATH}" CELERY_CONFIG_MODULE="celeryconfig_cluster" strace -o /tmp/strace.log -fFv /usr/bin/python -m provisioningserver start-cluster-controller http://maas.internal.example.com/
<bigjools> (ignoring the wrapper for now)
<bigjools> oh god
<bigjools> even if we get this right it won't work jtv
<jtv> Yes, my son?
<bigjools> because the fork can happen ages later
<bigjools> so the upstart script will always hang until the admin accepts the cluster
<jtv> Yes
 * bigjools gives up
<jtv> Then maybe we should kill two birds with one stone.
<jtv> Move the polling into the child process, and instead of exec'ing, just have it hang around as basically a wrapper to celeryd.
<bigjools> not sure it'll work ATM anyway
<bigjools> because of these 6 spurious forks
<jtv> That way, start-cluster-controller will hang around for as long as celeryd is running.
<bigjools> I'd rather not do that
<bigjools> but let;s see what roaksoax thinks
 * bigjools lols at jtv's continued wrong merge targets :D
<jtv> Yes, it's become part of the process.  Why change it now?
<jtv> bigjools, roaksoax: I think upstart can manage a process that just never exits.  If so, I think it would simplify our problem a lot if we could just keep the maas-provision process alive, instead of forking & exiting.
<smoser> http://paste.ubuntu.com/1257227/
<smoser> thoughts?
<bigjools> is node-group-interfaces valid?
<smoser> different than http://paste.ubuntu.com/1257229/
<bigjools> ok
 * bigjools looks at code
<bigjools> smoser: nodegroup called "master" does not exist
<jtv> Wait...  did I seriously just catch a snippet of advertising for the miraculous innovation of...  coffee without sugar!?
<bigjools> well, uuid of master I mean
<smoser> so what is 'master' there?
<bigjools> must be the uuid of an existing nodegroup
<bigjools> maas-cli api localmaas node-groups list
<smoser> it would seem like no
<smoser> http://paste.ubuntu.com/1257235/
<smoser> got 'master' as you said
<smoser> hm..
<bigjools> if you list existing NGIs does it show?
<bigjools> hey, this is not a maas bug, where should be be retargeted? https://bugs.launchpad.net/maas/+bug/1060194
<ubot5> Ubuntu bug 1060194 in MAAS "cannot find "bnx2-mips-09-6.2.1a.fw" when booting from maas" [Undecided,New]
<smoser> woohoo.
<smoser> bigjools, initramfs-tools maybe
<bigjools> ok
<smoser> i actually think i saw this.
<bigjools> thanks
<smoser> err... jsut reading code thought "how strange, it brings up networking without looking waiting for udev"
<bigjools> smoser: or cloud-initramfs-tools?
<smoser> no
<smoser> initramfs-tools
<bigjools> ok done
<smoser> incidently, it is probably fixed now
<smoser> because cloud-initramfs-tools code goes looking for NICs and waits for udev
<smoser> :)
<roaksoax> smoser: i'm gonna have to test the cloud-init stuff tomorrow... the mini servers are gonna take forever to install
<roaksoax> smoser: err upgrade
<smoser> upgrade?
<roaksoax> smoser: my maas mini server needs to be upgraded :)
<smoser> bigjools, ok. so my goal in the last escapade is to configure dns
<smoser> http://paste.ubuntu.com/1257248/
<smoser> that gets me a valid response
<smoser> but my goal is to get dhcpd functioning
<bigjools> cool
<smoser> and i still have no /etc/maas/dhcpd.conf
<jtv> Hi roaksoax!  Did you see our problem with the fork-tracking for the cluster controller?
<bigjools> smoser: check the celery log
<roaksoax> jtv: howdy. no I didn't
<roaksoax> jtv: any lp
<roaksoax> jtv: any lp bug?
<bigjools> smoser: 1. has it received secrets from the region, 2. did it get any DNS tasks?
<jtv> roaksoax: bug 1059453.  Turns out maas-provision does a lot of forks before it gets to the one we'd want upstart to track.
<ubot5> Launchpad bug 1059453 in maas (Ubuntu) "The celery cluster worker is not properly stopped" [Critical,Triaged] https://launchpad.net/bugs/1059453
<bigjools> smoser: is https://bugs.launchpad.net/bugs/1060331 fixed now?
<ubot5> Ubuntu bug 1060331 in MAAS "daily ppa install fails fully automated" [Undecided,New]
<smoser> bigjools, sure.
<smoser> i tihnk it was the isc_service_stop thing
<bigjools> yeah
<roaksoax> jtv: i think that's a question for james hunt
<roaksoax> jtv: if not, we are probably have to run it as we used to
<bigjools> roaksoax: oh FWIW there's a trick to conf file removal
<bigjools> I asked colin about it
<roaksoax> bigjools: what did he say?
<bigjools> roaksoax: "man dh_installdeb, man dpkg-maintscript-helper" :)
<bigjools> and search for "conffile removal"
<roaksoax> bigjools: ah yeah, i know that :)
<smoser> can i turn of header output from the cli ?
<bigjools> so basically upstart confs are not treated specially
<bigjools> smoser: not yet :(
<roaksoax> bigjools: nope, so we can use rm_conffile
<jtv> Thanks roaksoax -- I'll see if I can reach JH later.
<bigjools> roaksoax: exactly
<roaksoax> bigjools: we need to do the same for maas.conf under /etc/dbconfig-common
<roaksoax> bigjools: i'll take care of it :)
<bigjools> roaksoax: ok thanks!
<smoser> bigjools, http://paste.ubuntu.com/1257255/
<smoser> that would seem the relevant bit of celery log
<bigjools> yay
<bigjools> smoser: anything for dhcp?
<smoser> http://paste.ubuntu.com/1257259/
<smoser> full log
<bigjools> is maas-dns installed?
<smoser> no
<bigjools> that might help :)_
<bigjools> also maas-dhcp?
<bigjools> well the former will ensure the latter
<smoser> maas-dhcp, yes.
<smoser> you're saying dns ensures dhcp?
<bigjools> yes
<smoser> k.
<bigjools> we can't do dns without dhcp
<bigjools> I suspect a signal is missing to push out the dhcp config when NGI changes
<bigjools> hmm no that's wired
<smoser> bigjools, so am i at least going down the correct path here?
<bigjools> smoser: yes
<smoser> is there some expected way that someone would set this up?
<bigjools> this is fine so far, if it's not working it's a bug
<roaksoax> smoser: ok so the script itself seems to work
<roaksoax> smoser: just need to test in a deployment
<smoser> roaksoax, nice work.
<roaksoax> i'll try to get that tonight
<roaksoax> i just realized that i need to reconfigure the network :(
<roaksoax> my home network :(
<smoser> Setting up maas-dns (0.1+bzr1134+dfsg-0+1136+113~ppa0~quantal1) ...
<smoser> chown: invalid user: `maas:root'
<smoser> dpkg: error processing maas-dns (--configure):
<smoser>  subprocess installed post-installation script returned error exit status 1
<roaksoax> smoser: that's already been fixed by bigjools
<bigjools> smoser: there's a fix just landed
<bigjools> needs rebuilding
 * bigjools hits the daily rebuild button
<bigjools> should be ready (published) in 10-15 mins
<bigjools> I <3 recipes
 * bigjools grabs food
<smoser> bigjools, ok
<smoser> http://paste.ubuntu.com/1257282/
<smoser> ^ results in http://paste.ubuntu.com/1257283/
<smoser> and no /etc/maas/*dhcp*
<smoser> well, i have to go to bed.
<smoser> tell me what stupid thing i've done there.
<smoser> http://paste.ubuntu.com/1257288/ is some node group info.
<smoser> it seems my operation to update my node worked.
<smoser> ok. bed.
<smoser> good night.
<bigjools> smoser: hmmm looks like a bug.  I'll get rvb to look, unless jtv wants to
<bigjools> nn smoser
<jtv> nn smoser
<jtv> I'm looking up the pastes
<smoser> oh. one thing to note. there are things in /etc/bind/maas
<smoser> $ ls /etc/bind/maas/
<smoser> named.conf.maas       rndc.conf.maas                zone.master
<smoser> named.conf.rndc.maas  zone.77.168.192.in-addr.arpa
<bigjools> that looks expected
<smoser> right
<smoser> but no dhcp
<bigjools> yeah dhcp is differnet tasks and they';re not getting fired
 * jtv inserts lame joke about doing the same thing for us
<bigjools> one of us will try to reproduce
<smoser> ok. both of you can get to ubuntu@10.55.60.130
<smoser> to poke around
<smoser> do whatever you want.
<jtv> Thanks.  That'll tell us if it's a missing sudo or an apparmor thing.
<smoser> (hoping through chinstrap
<smoser> later.
<jtv> nn
<jtv> permission denied :(
<jtv> bigjools, can you get into that system?
<bigjools> it's probably on the VPN
<smoser> ubuntu@
<jtv> Yes I did
<jtv> This is a canonistack instance, not the VPN AFAICS
<smoser> right
<smoser> http://paste.ubuntu.com/1257300/
<bigjools> oh that needs spethial setup too IIRC
<smoser> only hoping ghrough chinstrap
<smoser> is all it needs
<jtv> Well I did do it through chinstrap of course
<smoser> Host 10.55.58.* 10.55.60.* 10.55.62.* 10.55.63.* *.canonistack *.canonistack2
<smoser>     ProxyCommand ssh chinstrap.canonical.com nc -q0 %h %p
<smoser> thats all the config you should need.
<jtv> Well, my key is in there.
<jtv> And ssh into canonistack instances normally works for me.
<jtv> And I'm doing the ubuntu@
<jtv> And I'm going through chinstrap.
<jtv> But it denies me public-key auth.
<roaksoax> smoser: so there's various maas regions now?
<roaksoax> err
<roaksoax> canonistafck regions*
<smoser> 2
<roaksoax> ah cool
<smoser> well, password auth is now enabled there
<jtv> Ahhhh this needs my canonistack key, not my regular ssh key
<jtv> So I need to register my canonistack keys as ssh keys.
<smoser> and password is "maas-is-fun"
<smoser> jtv, you dont have to do that.
<jtv> You sick, sick bastard
<jtv> Well it doesn't seem to work without!  My ssh config for canonistack instances is set up to use my canonistack key as IdentityFile.
<smoser> i imported your credentials from launchpad. those should forward through the ssh ProxyCommand above.
<smoser> yeah, i'd recommend ditching that crappy canonistack key
<jtv> But every time you do this, it denies me login.
<smoser> and only ever using your real keys
<jtv> Oh, that's not needed!?
<smoser> euca-import-keypair --public-key-file ~/.ssh/id_rsa.pub jtf
<smoser> then launch instances with '--keypair jtf'
<smoser> (spelled jtv wrong twice)
<jtv> What are the odds
<jtv> Anyway, still not working without that IdentityFile line in my ssh config.  :(
<roaksoax> smoser: http://paste.ubuntu.com/1257310/ alright, we just need to figure out a way how to run it
<roaksoax> smoser: unless you want to run it as a script?
<smoser> well, since it isn't really a script, it doesn't make sense.
<smoser> i'd just add something that runs that script and calls maas signal
<smoser> roaksoax, i can look more tomorrow.
<roaksoax> smoser: yeah, either way I won't test it until tomorrow
 * roaksoax is off
<roaksoax> night all
<jtv> nn roaksoax
<bigjools> smoser: if you are still there, I know what's wrong
<bigjools> we never believe you when you say you're going to bed :)
<jtv> He must be asleep for real.  He wouldn't have been able to resist this one.
<bigjools> smoser: adding a new nodegroup requires a corresponding celeryd to be running, listening on the right queue.  maas-provision start_cluster_controller does that in packaging but you can just run "celeryd -Q <uuid>"
<Fajkowsky> hello I have problem with http://askubuntu.com/questions/195115/nodes-cant-connect-to-server-after-bootstrap
<Fajkowsky> can someone help me with it?
<mgz> what is the lint thing jtv will cry if I don't run?
<bigjools> make lint!
<mgz> ta.
<mgz> sounds counter productive though...
<bigjools> Fajkowsky: answered your Q in #ubuntu-server but better to talk about it here
<mgz> unmake lint!
<bigjools> mgz: hoho!
<Fajkowsky> I have question about maas, I want to check if performence will be grow if I add more nodes to one service. I want to check on minecraft server because I think it's make big diffrence in performance if I add another node, I am right?
<bigjools> Fajkowsky: this is not a maas-specific question, but yes you can scale out easily with juju and maas
<Fajkowsky> whoops , you are right
<jam> bigjools: so we will have the tags getting processed in the async workers eventually, how do we do integration testing of that?
<jam> ATM, we only seem to have unit tests
<jam> that mock out the actual POST/GET requests.
<bigjools> jam: the test harness will run the tasks synchronously
<bigjools> so leave everything unmocked and test for the desired end result
<bigjools> provided you're running with the right TestCase, anyway.  There are diffrerent ones and they are all called TestCase, which is something that needs to be fixed before I strangle a cat in frustration.
<bigjools> and you need the celery test resource loaded, theres plenty of examples in existing tests
<rvba> jam: just be aware of the fact that, in the tests, task.delay() calls the task in a synchronous fashion.
<jam> bigjools: so do all nodegroups have celery workers in the test suite, then?
<jam> (right now the apis are restricted so that only the associated worker is allowed to call them)
<jam> since otherwise you could get around the privacy stuff
<bigjools> jam: not sure how that'd work out, you might need to patch stuff in that case.  You just need to know that apply_async() or delay() just calls the Task func there and then.
<bigjools> obviously there's no queue in this case
<jam> bigjools: well part of this is that I want to have at least one test that things work in a real integration setting (multiple nodegroups, and real celery workers talking via the apis.)
<bigjools> ok
<bigjools> mmmm that might be hard
<jam> I can also update some apis so they can be called by superusers
<bigjools> we've always mocked things out in existing tests
<jam> bigjools: well, it is either that, or all the testing falls onto matsubara
<rvba> The only thing you can do in tests is make sure that each task is routed to the right queue.
<bigjools> you can test either side of the Task easily
<bigjools> and what rvba just said
<jam> rvba: the problem with mocking it out on the celery side, is that you can easily get skew between what you think the API takes and returns, and what reality says.
<bigjools> but a complete integration test is very tricky as we don't start up celeryd
<bigjools> jam: the API should be static anyway
<bigjools> it's versioned
<jam> bigjools: well, it is just being implemented now... :)
<bigjools> well, it's versioned in that we reserved a space for a version :)
<jam> so the testing needs to be done manually for now?
<bigjools> jam: if you patch out the task you can check the params and fail if they change at all
<rvba> jam: you should be able to use the API for real (without mocking it) in a celery task.
<jam> rvba: except for the permissions bits? or how do we work around those?
<jam> (I do know that I had one accidental success if you 'yield' in an api call.
<jam> api stubbing returns the generator, and 'assertItemsEqual' passes just fine)
<bigjools> jam: you should be able to test end-to-end provided you do something about the queue checking, perhaps mock it, I'd need to see it
<jam> but in reality, the HTML level stuff turns the generator into a string
<jam> "<generator ...>"
<bigjools> but the queue given gets passed to the task so it's easily checked
<bigjools> rvb can help you, I gotta EOD now
<jam> bigjools: np, have a good evening
<jam> rvba: I actually need to go pick up my son now, but I'll be back to pick your brain some more later.
<bigjools> cheers
<rvba> jam: by permissions you mean the API credentials?
<rvba> jam: sure, no pb.
<jam> rvba: right, so I added apis that let the nodegroup worker get access to all information about nodes in the nodegroup
<jam> but we can't make that public because that gets around the Node VIEW permission stuff
<jam> so it is only allowed by the oauth key associated with the nodegroup worker.
<rvba> Makes sense.
<jam> which isn't the 'client' that is running the 'create a tag'
<jam> anyway, really gone
<jtv> Any reviewers in the house?  https://code.launchpad.net/~jtv/maas/bug-1059453/+merge/127680
<mgz> what is this netifaces python package the provisionserver wants, why do I not have it, and where can I get it...
<rvba> jam: I think the only thing you need to do is patch the credentials used by the worker to connect to the API (i.e. simulate what refresh_secrets would do)
<mgz> ah, I bet it's because it's been added to required-packages since I last pulled locally...
<jtv> mgz: "make install-dependencies"
<mgz> I should fix my deployment script.
<jtv> And pull regularly.
<jtv> You don't want your branch to fall out of date or you'll be building merge conflicts into branches right from the start.
<mgz> I should change it to use bzr cat lp:maas rather than peeking at my stale branch on this box
<jtv> We've got a pretty high ratio of changes to code right now.
<mgz> the copy I'm working with is up to date, but what I'm telling cloud-init to install is based off a copy of maas I'm not using
<jtv> Ah
<mgz> hence forget to keep up to date.
<mgz> still my fault though :)
<jtv> allenap: want to talk cli?
<allenap> jtv: Sure.
<allenap> jtv: Shall I call you?
<jtv> Call?
<allenap> jtv: It's probably quicker, but here is okay I guess.
<jtv> I mean, do we do a hangout?
<jtv> I'm starting one
<jtv> allenap: https://plus.google.com/hangouts/_/aaedfd9ecc51ae502006d3c55aa21e6680992c86?authuser=0&hl=en-GB
<mgz> jam: the maas lander really hates you...
<mgz> it's not done that "additional revisions which have not been approved" thing to me once...
<mgz> let's see if I've jinxed myself...
<mgz> hm, is there some way I get get django up on localhost without all the rest of the junk for make run? celery is unhappy now...
<mgz> because there's no 'maas' user... what should have created that?
<rvba> jtv: what mgz is experiencing looks like a fallout from what you're doing to fix the start-cluster scriptâ¦
<jtv> otp
<rvba> jtv: I'm just guessing here but if we try to use the maas user to run celeryd on a dev instance, that will be a problem.
<jtv> mgz: try bin/maas-provision start-cluster-controller http://localhost:5240/ -u `pwd` -g `pwd`
<jtv> rvba: that's right, so don't do that.  Use your real identity.
<jtv> mgz: sorry, not pwd
<jtv> the other one
<jtv> whoami
<jtv> and yes I'm conflating your user with your group of the same name and just guessing that there is one
<mgz> ubuntu:ubuntu so yup.
<rvba> services/cluster-worker/run will probably need to be fixed then.
<jtv> Does services/cluster-worker/run use start-cluster-controller already?
<rvba> Yep, since last week.
<jtv> allenap had a cute idea: default not to maas but to the _current_ user, except when that's root.
<mgz> for the record: <http://pastebin.ubuntu.com/1257666/>
<jtv> Unless & until we do that, pass explicit user/group.
<jtv> Yes, that's what happens if you leave it to default to "maas" on a system that doesn't have a maas user.
<jtv> rvba: I'll update the "run" script.  Thanks for updating that btw.
<jtv> I mean, thanks for the update _before_ the one I'm about to make.  :)
<rvba> jtv: great.
<mgz> I take it the early errors in logs/webapp/current are inconsequential as it's repeated until it works?
<mgz> okay, this is all quite nice
<jtv> rvba: aigh.  One problem with updating the cluster services file: start-cluster-controller will no longer exit!  Do we have some utility at hand to wrap it in a daemon?
<jtv> start-stop-daemon -b?
<rvba> jtv: not sure I follow, you've removed start-cluster-controller?
<jtv> No, but it will no longer exit.
<jtv> It'll keep running forever.
<jtv> Well, waiting.  :)
<jtv> And I'm guessing that our services machinery will want it to run in the background.
<rvba> Quite the opposite I think, hence the usage of 'exec' in these files.
<jtv> Ah, that makes sense!
 * jtv gives it a whirl
<jtv> I'm ditching the fghack as well
<jtv> rvba: this is almost funny.  With my change, "make run" seems to add a new celery (with 5 processes) every 10 seconds!
<rvba> jtv: the supervise stuff is probably unable to detect that the process is running ok, so it relaunches one every x seconds.
<jtv> Yeah I guess.  :(
<rvba> jtv: IIRC that's precisely what happens when the process launched by the 'run' script gets daemonized.
<jtv> I wonder if re-introducing fghack helps then...
<jtv> The region worker otoh isn't shutting down properly.
<jtv> Now breaks my wooden shoe.
<jtv> (The literal equivalent of which is a Dutch expression for "WTF????")
<jtv> fghack seems to fix it.
<jtv> Oh well.  Don't question the oracle.
<jtv> I'm out.  allenap, could you have a look at my updated MP?
<mgz> jtv: lgtm, can I land it while you sleep?
<jam> mgz: well, I often do a quick 'cleanup' patch, and then try to submit it.
<jam> ix
<jam> mgz: however, gavin mentioned that you have to wait for the mp to update before you mark it approved. However, I waited until it saw 'Unmerged revisions' but not before the diff was updated.
<jam> it is possible that you need to sing and dance and sacrifice a chicken to get things to go any faster.
<jam> I haven't quite worked out the exact syntax yet.
<mgz> hm, that may be it, I tend to not mark approved instantly after pushing tweaks
<jam> mgz: well 'instantly', I do the tweaks and want to submit it before I forget about it and it sits for a day. I know I need to wait 'a little' but trying to figure out when I can push it out of my mental context is difficult.
<mgz> write a local launchpad script that sleeps for five mins then flips the status :D
<jam> mgz: at least with feed-pqm /pqm-submit we would check that the branch tip was up to date, and not have to wait for the N async processes that go from branch tip changed, to MP noticing, to approve state.
<allenap> mgz: I'll land jtv's branch in his absence.
<jam> jelmer: both of my api branches have landed
<jelmer> jam: \o/
<mgz> allenap: he seemed to have remained awake just long enough to re-mark it approved :)
<mgz> ah, there is one of his pending still though
<allenap> I think we're concerned about different branches :)
<mgz> the scary celery exec change I wouldn't touch :)
<jam> mgz: hows the search stuff looking?
<mgz> I have html edited, need to wire everything up
<jam> mgz: did you know about huw's branch?
<jam> mgz: https://code.launchpad.net/~huwshimi/maas/hardware-search
<jam> (It was on the kanban board, but I realize you don't actually see that stuff)
<mgz> jam: no...
<mgz> I shall get that and examine
<mgz> okay, that seems all reasonably straight forward
<jam> mgz: I also should have some mockups for what it should look like
<jam> let me dig them up
<mgz> in lynx? :)
<mgz> I'll socks proxy so I can see the graphical view :D
<jelmer> mgz: have you tried using w3m or links? They should be able to do graphics if you have a framebuffer..
<jam> mgz: https://docs.google.com/a/canonical.com/drawings/d/1AH_8gCyTYG6LfbjzYYjJQowpT-m8wq_P3oV5kL2XrmY/edit
<jam> and: https://docs.google.com/a/canonical.com/drawings/d/1NnBi_3bzpFjhbC6X8KYbh40qbj-FfxzlPJ40TbmXTGE/edit?pli=1
<mgz> ta
<jam> the idea is that the search bar at the top should be everywhere, but when you get to the nodes/ page itself, it moves down into a larger view inside the page
<jam> in *my* head the main change is for that page to take an optional 'constraints'  (search?) parameter
<jam> which then gets parsed from text form, into a dict, and essentially handed off to the filter code we just landed
<jam> mgz: note that some of that view will not be present in 12.10, some of it is 'future work'. Like if we don't have pagination yet, etc.
<smoser> jtv, did you sort the dhcpd.conf issue i was hitting yesterday?
<jam> and the Node list/search should use the same: 'macaddress (hostname)' that we currently have, rather than the 2 columns that is on that page.
<smoser> ah. i see. reading up.
<jam> rvba: so I can easily test that the helper function returns an indication that the task needs to be retried. Is there a way to test that the retry is done?
<jam> Or do you just patch out retry and assert that it is called?
<rvba> jam: no, you can test the retry for real, see test_rndc_command_is_retried_a_limited_number_of_times in src/provisioningserver/tests/test_tasks.py
<rvba> jam: that's why we've created MultiFakeMethod :)
<jam> rvba: so that test would succeed if the code just raised the exception on the first try
<jam> I don't see it asserting that it is actually retrying
<jam> I see that it is asserting it does eventually stop retrying
<jam> ah, I guess that is 'can_be_retried'
<rvba> jam: indeed, but you're right it's a bit implicit.
<jam> rvba: so how often is the refresh_secrets called?
<jam> rndc only waits 20s before failing
<jam> (10 tries at 2s each)
<rvba> jam: the main reason for retrying the rndc command is that bind sometimes takes some time to start up.
<jam> rvba: so the issue seems to be that it is possible for a queue to exist, but not have credentials to actually talk to the mass server yet
<jam> (mass_url may not be set, you may not have creds, you may not have a nodegroup.uuid yet)
<rvba> jam: I assume s/queue/nodegroup/
<rvba> Queues are simply created on demand.
<jam> rvba: so I was told to do "async_queue(queue=nodegroup.uuid)"
<jam> in order to get them started on the right provisioning_server
<jam> instead of running it on the master
<jam> (each nodegroup's controller is meant to refresh the tag matching for the nodes under its control)
<jam> (so that master doesn't have to update 100,000 linearly, but each cluster can update their <10k nodes in a row.)
<rvba> Makes sense.
<jam> rvba: I would have thought that you need a queue around to make sure it is running on the right machine/worker/something
<rvba> jam: yeah, but queues are autocreated when the region sends tasks to it or when a cluster connects to a queue.
<rvba> Calling task.apply_async('my queue') will create 'my queue' and send task to it.
<rvba> Then the task just sits there until a celery worker feeds from that queue.
<rvba> All I'm saying is that you don't need to create the queues.
<jam> rvba: ok, but that means we need to call refresh_secrets before we call the task we want to do?
<rvba> jam: indeed.  refresh_secrets gives to the cluster worker the credentials it needs to access the API.
<rvba> jam: glancing at the code, that is only called when the cluster controller starts up.
<jam> rvba: so the question is still open whether we can trust that the controller has creds or not. The other api has tests that assert it 'does nothing if it doesn't have creds'
<jam> and we *need* the work to be done
<jam> rvba: I got DC'd for abit.
<roaksoax> morning
<rvba> jam: you're right.  Right now we trust that refresh_secrets was called when the cluster got started.
<jam> so the idea of putting in retry is that we need it to run once we get creds, or we can trust that we always have creds, or we always push creds out before we ask for the work to be done.
<rvba> I understand.  This is a tricky problem.
<jam> rvba: if we could get an error message back we could go with that
<jam> or we can create our own 'queue' of work that is remaining to be done
<jam> in the db tables
<jam> and then create another async process that makes sure the queue is being worked on.
<jam> (since we can assert the transactional nature of the primary db, but not really the rabbit queues)
<rvba> That is exactly what celery-django does for you :)
<rvba> (A package we don't use atm)
<roaksoax> smoser: around already?
<jam> rvba: so should we try to do that, or should we just live with 'things may get out of date and we should add a "manuallly trigger a refresh"' in case stuff ever gets dropped. ?
<smoser> roaksoax, here.
<roaksoax> smoser: where do you tell commissioning to install packages?
<roaksoax> or, extra packages
<roaksoax> smoser: we need freeipmi-tools
<jam> without maas_url, there isn't even a place to put an anonymous 'something failed' handler.
<jam> that the celery worker can inform us it needs to be retried.
<rvba> Indeed, chicken and egg problem.
<jam> rvba: right, so either we write it as 'assume something has failed until the worker has said it succeeded' or ?
<jam> and if we do that, what task sits around making sure it gets retried?
<jam> you can poke at the API to get workers refreshed
<jam> and we can just call that immediately before we try to do work
<smoser> roaksoax, looking
<smoser> roaksoax, also
<smoser> https://bugs.launchpad.net/maas/+bug/1060942
<ubot5> Ubuntu bug 1060942 in MAAS "maas-cluster-celery job dies" [Undecided,New]
<rvba> Yeah, that sounds like a good way to do that quickly.  Long term, it would be good to use celery-django for that kind of stuff.
<rvba> If celery-django gives us that possibility that is.  I haven't looked into it really, but I think that's what it does: track the status of the tasks within the db.
<smoser> roaksoax, sorry for the slow response.
<smoser> you have to do it in that script there (./etc/maas/commissioning-user-data)
<smoser> that script gets sent to cloud-init verbatum as user-data
<smoser> (which it executes because of '#!')
<smoser> so you'll have to 'apt-get update' there
<roaksoax> smoser: ok cool thanks
<smoser> roaksoax, i'm kind of ton on this.
<smoser> on one side i feel ike i should put more into the imgaes
<smoser> (like maas-enlist and freeipmi-tools)
<smoser> but on the other, we have to be able to deliver updated versions of those *anyway*
<roaksoax> yeah
<smoser> so 'apt-get update && apt-get install' is pretty much required anyway
<roaksoax> putting them would indeed speed up the process, but we depend on they being updated
<roaksoax> at least maas-enlist
<smoser> right.
<smoser> so it wouldn't really speed it up
<smoser> in the end
<smoser> well, the deps would help if they're extensive
<smoser> (but they're not i dont think)
<smoser> rvba, roaksoax do you have thoughts on https://bugs.launchpad.net/maas/+bug/1060942
<ubot5> Ubuntu bug 1060942 in MAAS "maas-cluster-celery job dies" [Undecided,New]
<smoser> it looks to me like we've tried twice now to do this and both have failed
<smoser> i'm not exactly sure why celery can't setgid when it starts as root
<roaksoax> smoser: i'm looking at it now
<roaksoax> smoser: the upstart jobs used to set the gid/uid before, when running celerd directly
<roaksoax> celeryd*
<rvba> But now there is communication phase before we can start the celer worker for the cluster controller.
<roaksoax> smoser: which is what maas-region-controller does
<rvba> smoser: could it be caused by a limitation in upstart somewhere?
<roaksoax> rvba: so upstart stats the script as root, the root check in the wrapper passes just fine
<jam> mgz: so I'm heading out for the day, feel free to push up work in progress if you want me to take a look at it tomorrow before you wake up.
<jam> or if there is anything that is blocking you now?
<jam> rvba: nodegroup-ui branch was a merge conflict, not a test failing (from what I can tell)
<jam> rvba: but of course, I read it wrong.
<jam> I agree, the build says 'SUCCESS' at the end.
<mgz> jam: thanks, will do
<mgz> nothing blocking atm
<roaksoax> rvba: what log does cluster-controller creates? celery-cluster.log?
<jam> mgz: we have roughly 1 more day to land it, so I'm trying to make sure everything is as smooth as possible.
<rvba> jam: I don't see the merge conflict but I'll merge trunk anyway, just to be sure :).
<rvba> roaksoax: /var/log/maas/celery.log
<jam> rvba: well, be careful, since otherwise maas lander will reject it because the branch changed after it was approved :)
<smoser> i am confused on whats doing this
<smoser> i added
<smoser> print("user=%s group=%s uid=%s gid=%s curuid=%s curgid=%s" % (user, group, uid, gid, os.getuid(), os.getgid())
<jam> I think I just misread the error message ,which looks to be 100% bogus as you mentioned.
<smoser> right before the fork/seuid/gid
<smoser> and i see
<mgz> jam: I was counting thursday and friday as two, but we want to be all done by tomorrow?
<jam> mgz: we need some time to get it into packaging,etc.
<smoser> user=maas group=maas uid=113 gid=120 curuid=0 curgid=0
<jam> and jelmer is gone on friday regardless.
<smoser> so at that point i'm running as 0:0 and trying to go to 113:120
<jam> so I would say "1-ish"
<mgz> okay, 1-ish it is.
<roaksoax> smoser: yeah that seems to be the issue
<smoser> ?
<smoser> i'm saying it looks right
<roaksoax> smoser: setgid
<smoser> curuid, curgid = (0,0)
<jam> mgz: did you get hardware_details for sampledata ? (not yet, I believe we deprioritized it, vs tag data and having search implemented)
<smoser> i'm just trying to drop gid to 120.
<smoser> why would i not be able to do that?
<roaksoax> smoser: i commented out the setgid and the process starts just fine
<smoser> http://stackoverflow.com/questions/4692720/operation-not-permitted-while-dropping-privileges-using-setuid-function
<mgz> jam: nope, but could trivially do that (something for mem and cpu_count may also be useful)
<smoser> wait. i'm confused by that though.
<roaksoax> smoser: [2012-10-03 09:39:14,571: WARNING/Beat] DBAccessError: (13, 'Permission denied')
<smoser> ah
<smoser> duh
<smoser> stuipd
<roaksoax> smoser: that's besides the gid/uid thing
<jam> mgz: so if you get search up and nobody reviews it, then look at the sampledata stuff :)
<jam> anyway, I'm off, have a good evening
<mgz> later!
<smoser> roaksoax, https://code.launchpad.net/~smoser/maas/lp1060942/+merge/127767
<smoser> roaksoax, where did you see your error ?
<roaksoax> smoser: it is approaved
<rvba> smoser: if that really fixes the problem, you might want to change to comment also :)
<roaksoax> smoser: in /var/log/celery.log
<roaksoax> rvba: where does celery stores its db?
<roaksoax> or beat?
<smoser> yeah, i'm seeing that now
<rvba> roaksoax: don't know what the default isâ¦ hang on.
<roaksoax> http://comments.gmane.org/gmane.comp.python.amqp.celery.user/2375
<roaksoax> smoser: ^^
<rvba> roaksoax: looks like the default is to use /var/run/celerybeat-schedule
<roaksoax> rvba: so how can we do this: http://comments.gmane.org/gmane.comp.python.amqp.celery.user/2375
<rvba> roaksoax: and the region is told explicitly to use /var/lib/maas/celerybeat-region-schedule in the upstart script.
<rvba> roaksoax: are you seing that error?
<smoser> so what is it supposed to be?
<roaksoax> rvba: yeah I'll test/fix
<roaksoax> smoser: it works
<roaksoax> rvba: it works :)
<smoser> what do we need to pass?
<smoser> whats the value we need to pass ?
<roaksoax> smoser: http://paste.ubuntu.com/1258008/
<smoser> are we easily able to pass that in ?
<smoser> as an argument
<roaksoax> smoser: yes, we pass that in the upstart job
<roaksoax> rvba: should be just pass "/var/lib/maas/celerybeat-region-schedule" or should that be detected automatically
<roaksoax> rvba: it guess it would break make run?
<rvba> roaksoax: you're right, we can make it use a param that would be different in the local config.
<rvba> roaksoax: I can put together a branch to do that.
<roaksoax> rvba: awesome! that'd be great
<smoser> rvba, that just gets us to the next error
<rvba> smoser: what's the next error?
<smoser> http://paste.ubuntu.com/1258023/
<rvba> smoser: the error in provisioningserver.tasks.upload_dhcp_leases ?
<smoser> but at least the celery seems up at that point
<smoser> it does suck that it dies permenently on that error
<smoser> hm..
<rvba> Yeah, there is a task failing, we know about this one.
<smoser> yeah, this is really fragile
<rvba> We just need to make the task code deal with the fact that the lease file might not be there.
<smoser> but i'm not sure why upstart isn't re-starting it
<rvba> We're aware of this one.
<rvba> jtv: ^ ;)
<jtv> ?
<rvba> The dhcp task failing because the lease file is not there :).
<jtv> The cluster controller upstart job can't really respawn because the typical reason for failure is that the cluster controller has been rejected.
<rvba> Not urgent but if would be good to tweak the task so that it could cope elegantly with the situation.
<jtv> I agree.
<rvba> Got task from broker: provisioningserver.tasks.refresh_secrets
<rvba> Task provisioningserver.tasks.refresh_secrets[d8c820a8-d7cd-453b-9816-747ee2556927] succeeded in 0.201004981995s
<rvba> Contact with the region controller was ok.
<smoser> jtv, i dont understand the comment above.
<smoser> upstart will handle that most likely
<smoser> if the job dies 5 times in 5 seconds, upstart will give up on it
<jtv> Ah good
<smoser> if it doesn't hit that threshold, then, who cares.
<jtv> Just so long as Right
<jtv> Ahem.
<jtv> Right.
<jtv> Just so long as it doesn't loop wildly.
<smoser> who cares if a daemon keeps spawning that shouldnt
<smoser> it was configured wrong by the admin explicitly
<jtv> In this case we do care, because it makes API requests.
<smoser> in this case.
<smoser> so the way you could do this
<smoser> is a pre-start job
<smoser> that does the check for "am i accepted"
<smoser> and exit 0 if "no"
<smoser> ( i think that sthe right symantics)
<smoser> but then that would only run once
<jtv> We considered something like that, but decided it had to be this way.
<smoser> well then you need to make it re-spawn
<jtv> It needs to repeat in some cases.
<smoser> or its just going to die all the time.
<smoser> upstart respawns daemons for good reason. its generally considered the right thing to do.
<jtv> Well like I said, it's fine here too -- as long as it's not a tight endless loop.
<smoser> see 'man 5 init' look for 'respawn'
<smoser> how did you disable this ?
<jtv> I didn't.
<smoser> is it exiting success on stack trace?
<smoser> i'm confused why it isn't getting respawned
<jtv> I don't have the answer.
<jtv> But Julian told me it wasn't set up to respawn.
<rvba> roaksoax: https://code.launchpad.net/~rvb/maas/explicit-beat-schedule/+merge/127775
<smoser> roaksoax, do you have ideas?
<allenap> smoser: respawn is not in maas-cluster-controller.maas-cluster-celery.upstart?
<allenap> smoser: Also, that setuid/setgid branch slipped in; I guess Tarmac got it too quickly.
<smoser> allenap, bah.
<smoser> i will do a revert branch
<allenap> smoser: If I get time I'll write a test for this ordering.
<allenap> smoser: In fact, I'll do the revert if you want, and add a test at the same time?
<roaksoax> smoser: why isn't what respawned? maas-cluster-celery due to the leases stack trace?
<roaksoax> ok I need to test the IPMI so gonna setup my network properly
<smoser> https://code.launchpad.net/~smoser/maas/revert-1151/+merge/127778
<smoser> roaksoax, thats what i'm confused by
<smoser> allenap, well, there is the revert.
<smoser> it doesn't respawn.
<roaksoax> smoser: why are oyou confused by it?
<smoser> why does it not respawn
<smoser> http://paste.ubuntu.com/1258070/
<smoser> default is to respawn
<smoser> (but i was explicit and added it there)
<roaksoax> smoser: hold on, you want it to respawn due to the failurs of the leases file?
<smoser> it should respawn
<smoser> its an upstart job
<smoser> and if it dies, it should not be fatal
<roaksoax> smoser: it is not fatal
<roaksoax> smoser: i see that error over and over
<roaksoax> smoser: but celery doesn't die
<roaksoax> smoser: it keeps running
<roaksoax> it only error's out
<smoser> i dont think so.
<smoser> it dies
<rvba> Yeah, it's just a failed task.
<smoser> the job dies at least.
<rvba> Nothing critical for celeryd.
<smoser> sudo status maas-cluster-celery stop/waiting
<roaksoax> smoser: did you psas the --schedule?
<smoser> http://paste.ubuntu.com/1258075/
<roaksoax> smoser: and removed the old pyc files?
<smoser> it dies now because of the dhcp
<smoser> roaksoax, wouldn't you expect to see that pid change there?
<roaksoax> smoser: oh i thnk what it is
<roaksoax> smoser: jtv reported and error on which upstart couldn't track all the instances of celeryd
<roaksoax> smoser: that might be related
<jtv> It wasn't quite that.
<jtv> Upstart couldn't track the fork.  As it turned out, there were several forks before the one that actually spawned the celeryd.
<jtv> So upstart couldn't track any instances at all.
<smoser> ok. i'll open a bug here.
<smoser> but this is fairly severe.
<roaksoax> smoser: https://pastebin.canonical.com/75796/
<smoser> or am i incorrect
<roaksoax> smoser: i think i know why
<jtv> I'm getting bits and pieces of the conversation... what is the problem?
<smoser> maas-cluster-celery is fragile
<roaksoax> smoser: becuase maas-provision start-cluster-controller simply tells it to start celeryd and then it returns and exits
<smoser> in that if anything goes wrong it will die and never start
<smoser> requiring a human
<roaksoax> smoser: so maas-provision start-cluster-controller is not really a daemon
<jtv> roaksoax: it no longer does that
<jtv> It keeps running for as long as celeryd does.
<jtv> Well, it execs celeryd once it's got its approval etc. from the server.
<roaksoax> jtv: right, but for some reason the way it is started seems to not be tracked by upstart, and i think it is because it is not a daemon
<jtv> Still not being tracked?  :(
<jtv> And it's so simple now!
<jtv> As it stands, "maas-provision start-cluster-controller" should only exit if (a) the cluster controller has been rejected, or (b) celeryd exits.
<jtv> Wow, that's a pretty lame slogan: "Stiebel Eltron.  Originally German."
<smoser> hm..
<jtv> It's like saying "Pizza Hut.  One of us saw a pizza once and liked it so much we named the company after it."
<smoser> i must be misunderstanding something of upstart
<smoser> roaksoax, well.
<smoser> it doesn't respawn because you need the 'respawn' if you want it to
<smoser> ie, you need both:
<smoser> respawn
<smoser> respawn limit 5 60
<roaksoax> ack
<smoser> suck. why didn't i realize that.
<smoser> anyway
<roaksoax> smoser: why did you revert the setgui/setuid thing?
<smoser> unless allenap was going to do it.
<smoser> allenap, roaksoax just ack https://code.launchpad.net/~smoser/maas/revert-1151/+merge/127778
<smoser> to get my stupid change out
<allenap> smoser: +1 and approved. I'm writing a test for that code; I've noticed another bug at the same time.
<smoser> jtv, so you're working on something to make the dhcp work?
<smoser> rvba, do you know anything about mirrors in maas?
<rvba> smoser: not really, I just know that we have a config option named 'keep_mirror_list_uptodate' that is not used anywhere yet :/
<smoser> right. and i saw a list of mirrors in the ui
<smoser> that contained 'archive.ubuntu.com'
<smoser> and could not be selected or changed
<smoser> :)
<smoser> i'm thinking we're kind of SOL on local mirrors at the moment.
<rvba> smoser: that can be changed and updated. See the 'update from' dropdown.
<rvba> smoser: but the 'Default distro series used for deployment' thing is positioned right in the middle of it.  And that's confusing.
<smoser> hm.. i was just probably remembering incorrectly
<roaksoax> Daviey: do you happen to know how can I force IPMI to obtian a new IP address? (it is set to DHCP but doesn't get address)
<roaksoax> Daviey: nevermind i got it
<Daviey> roaksoax: how did you do it?
<roaksoax> Daviey: I set it to static, and then back to dhcp and it renewed it
<Daviey> hah, that is how i did it :)
<roaksoax> Daviey: :)
<roaksoax> smoser: http://paste.ubuntu.com/1258243/ so look at the first part
<roaksoax> adding the apt stuff that way?
<smoser> http://paste.ubuntu.com/1258248/
<smoser> roaksoax, yes. basically right, but the above is safer
<smoser> (and, fwiw, the '-q' to apt doesn't really mean 'quiet' it means "produces output suitable for logging"
<smoser> which is what we'd want)
<roaksoax> smoser: ok cools
<mgz> yup, basically not using \r to print the same line multiple times with progress updates
<roaksoax> smoser: alright, so it looks something like: http://paste.ubuntu.com/1258286/
<smoser> you have to quote fargs at 42
<smoser> as it probably contains spaces, right?
<smoser> and you didnt patch signal all the way, right?
<roaksoax> smoser: right :)
<smoser> did you want help with that
<roaksoax> smoser: btw... --power-settings is json
<roaksoax> smoser: it reutrns this: ('IPMI', '{"power_address": "192.168.2.111", "power_pass": "8ruHtjzpGdU", "power_user": "maas"}')
<smoser> right.
<roaksoax> smoser: so like this? signal "$fargs" OK "power-settings sent [$power_settings]"
<smoser> rbasak, how easily could you test something
<smoser> specifically https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825
<smoser> roaksoax, let me look.
<smoser> allenap, or rvba, i'd appreciate your input on that merge ^
<smoser> i'm sure the test could be cleaned up
<rbasak> smoser: my test node is being shipped to the MAAS QA lab now!
<rbasak> smoser: I need to switch my testing to another node, which shouldn't take long, but I have a hard stop soon. I can test it for you in the morning though? Normally it'd be about two commands!
<smoser> hm..
<smoser> rbasak, how were you telling cloud-init preserve_source_list off
<smoser> err... on
 * rbasak looks for the patch
<roaksoax> allenap: smoser how do you debug? something seems to have failed :(
<rbasak> smoser: in contrib/preseeds_v2/enlist
<rbasak> smoser: +apt_preserve_sources_list: true
<rbasak> smoser: that's all. And after deployment I fix manually. I presume this still breaks juju though.
<smoser> ah.
<smoser> what breaks juju?
<smoser> this would fix it.
<rbasak> sources.list being wrong after install
<rbasak> since it doesn't use that file on boot
<rbasak> Your fix is a proper fix I presume.
<smoser> right. my patch wuld make juju work.
<smoser> right.
<smoser> but, actually, doe not fix enlistment
<smoser> or commissioning
<smoser> (but those are sort of fixed for you already)
<smoser> by the ephemeral nodes having newer cloud-init
<roaksoax> smoser: https://pastebin.canonical.com/75814/
<smoser> carp
<smoser> you're out of memory
<smoser> how much memory do you have there?
<roaksoax> smoser: memory as in ram?
<roaksoax> ah right
<roaksoax> dah
<smoser> as in ram
<smoser> :)
<roaksoax> 256
<smoser> yeah.
<smoser> do you want to try to work around this?
<roaksoax> smoser: it is a VM
<roaksoax> smoser: so i'm just increasing memory
<smoser> oh. ok.
<smoser> i thought it was one of your servers
<smoser> but your vm wont have ipmi
<roaksoax> smoser: nope, just testing whether the scripts are running properly
<smoser> right
<roaksoax> smoser: btw.. you happen to have the link for the pastebin you got yesterday on how to send power settings back to MAAS?
<smoser> roaksoax, i dont remember pastein
<smoser> from you?
<roaksoax> smoser: from matsubara i thnk
<roaksoax> matsubara: ^^ do you know how to send power settings to maas?
<smoser> oh. i think that was something else.
<matsubara>  maas-cli api maas node update node-7d828c3e-0902-11e2-8461-00e081ddd1cf power_type=ipmi power_parameters_power_address=192.168.22.33 power_parameters_power_user=root power_parameters_power_pass=ubuntu
<matsubara> roaksoax, ^
<roaksoax> matsubara: thanks
<matsubara> np
<smoser> roaksoax, but that doens't help us.
<roaksoax> smoser: i know :(
<roaksoax> smoser: shouldn't we just send it as the other data you are sending?
<smoser> roaksoax, right. your patch looks good. we just have to tell that 'maas-signal' python script how to send the ppower settings.
<roaksoax> yeah
<roaksoax> smoser: we are still going to do the enlistment setting temporary ipmi credentials right?
<smoser> well, we dont have a way to post those at the moment.
<roaksoax> allenap: ^^
<roaksoax> smoser: can i ssh to the commissioning images?
<smoser> i wouldnt think so.
<smoser> roaksoax, what i'd suggest is preparing the ephemeral image to let you in.
<roaksoax> smoser: will do
<roaksoax> smoser: so something is wrong and never returns from commissioning, but I think i know why
<roaksoax> smoser: and don't have console nor a monitor :(
<smoser> what system?
<smoser> physical system?
<smoser> do those hp microservers ipmi have remote serial console ?
<Daviey> seemed not.
<roaksoax> smoser: and the problem is that there's an issue with the kernel not correctly recognizing ipmi port and stuff
<smoser> oh joy.
<Daviey> you have to modprobe with custom stuff
<Daviey> modprobe ipmi_si type=kcs ports=0xca2
<Daviey> But even so, with that - i didn't think you got console?
<roaksoax> Daviey: yeah i added that but doesn't seem to be working,
<roaksoax> smoser: ok it says that module ipmi_si does not exist on /etc/modules
<roaksoax> smoser: which basically tells us that we can't access the bmc :)
<smoser> ?
<smoser> where would that module come from?
<smoser> https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825
<smoser> ^ comments nplease!
<roaksoax> smoser: i'd say it is in the kernel
<smoser> smart @$!$
<smoser> what says that, roaksoax
<roaksoax> smoser: rmmod but i just realizd i probably dont even need it if the modufle hasn;t been loaded
<smoser> so modprobe it?
<smoser> feel free to do that in the commissioning script
<roaksoax> smoser: my default?
<roaksoax> smoser: i alreayd did that but was removing it first, before loading it
<roaksoax> the thing is if the module is loaded already, then you need to remove it to load it again
<roaksoax> so the parameters take effect
<smoser> yes.
<smoser> you do.
<smoser> right.
<smoser> so that sucks.
<roaksoax> smoser: so it commissioned this time but the it didn't seem to have done the ipmi stuff
<roaksoax> smoser: so how can I enable ssh access? besides of course ssh-import-id
<smoser> http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/revision/1089
<smoser> roaksoax, what i'd do is just add another user
<roaksoax> oh wait i guess i can just install openssh-server
<smoser> mount loopback, add user, add ssh keys umount
<smoser> its installed,
<smoser> but by default you wont have access as there is no passowrd and no user and no ssh keys installed
<smoser> :)
<roaksoax> smoser: right, so can't i create a user from the commissioning script?
<smoser> ah. yeah, you could easily enough.
<roaksoax> smoser: does it run as 'ubuntu' user?
<smoser> that runs as root
<roaksoax> ack!
<roaksoax> smoser: and how to prevent it from powering  off?
<roaksoax> trying to avoid having to mount it :)
<smoser> roaksoax, sleep!
<roaksoax> smoser: yeah figured :)
<smoser> i think that script is what powers off, right?
<roaksoax> nope
<smoser> yeah it does
<smoser> just disable the call to 'poweroff'
<smoser> err.. 'shutdown'
<roaksoax> ah lol
<smoser> (it calls that via 'trap' on exit)
<smoser> so just make that function return true
<smoser> or false
<smoser> doesn't actually matter
<smoser> :)
<roaksoax> :)
<roaksoax> smoser: i think what's happening is that it is not detecting IPMI so it continues normally
<roaksoax> smoser: yeah so it seems it is an issue with ipmi detection
<roaksoax> smoser: err modules
<roaksoax> smoser: ok so what i'm gonna do is simply enable the modules from the script, and look for errors i guess
<smoser> well, load the module in the script
<smoser> and generally ignore errors form that.
<roaksoax> smoser: like "modprobe XYZ || true"
<roaksoax> smoser: right so I was thinking, however, that what if the module is loaded (ipmi_si and needs to be reloaded?)
<roaksoax> dah
<roaksoax> duhh
<roaksoax> i guess i can do the same
<smoser> roaksoax, the script isprobably not set -e
<smoser> i dont generally do that.
<roaksoax> smoser: so do you think it is safe to wait 10 seconds for an IP address?
<smoser> safe as in too long or too short?
<roaksoax> smoser: enough time
<roaksoax> yes
<smoser> i think you  might as well give it 60 at least.
<roaksoax> smoser: right, so I will give it 60 only if we had to change from Static to DHCP
<roaksoax> makes better sense that way?
<smoser> well, if you're dhcping in general
<smoser> why would you want to risk giving up early
<roaksoax> smoser: right, but if the machine turns on and starts doing comissioning, then it would be safe to assume that IPMI has an IP address already
<smoser> roaksoax, so then its dhcp request shoudl come back quickly
<smoser> or dhcp was the wrong setting
<roaksoax> smoser: right but for example, what happens if the image doesn't get an IP address quickly?
<roaksoax> smoser: it sits and waits for one right?
<smoser> an image?
<smoser> as in what
<smoser> node booting ephemeral image ?
<roaksoax> smoser: yes
<smoser> pxeboot annoying will time out
<roaksoax> smoser: right, but ok, we pass pxeboot, the image also request the IP address right?
<smoser> well,the kernel does. but only for the interface that got pxe booted
<smoser> (unless there is a bug)
<smoser> and i guess its not really the kernel
<smoser> its the initramfs
<smoser> roaksoax, you know how to access the maas db?
<roaksoax> smoser: yes, what do you need?
<smoser> i ended up getting in with sudo -Hu maas psql maasdb
<roaksoax> smoser: sudo maas shell sshoudl do
<smoser> i needed db
<smoser> not shell
<smoser> how do i put it into debug ?
<smoser> the api server
<roaksoax> smoser: ah lol :) but you can modify the db from the shell
<roaksoax> smoser: that i don't know
<roaksoax> smoser: btw... how can we log all the stuff that the commissioning script does?
<smoser> well, the stuff it calls gets logged back to the server
<smoser> which is what i'm testing)
<smoser> and why i was asking such silly things
<roaksoax> ah!! cool
<roaksoax> smoser: btw.. i think overtime it would be a good idea to make the commissioning/enlistment use the proxy if available
<smoser> proxy?
<smoser> roaksoax, http://paste.ubuntu.com/1258781/
<smoser> ./maas-signal --config my.config --post power_type=ipmi --post "power_parameters={'blob': 'foo'}" WORKING "credsasdasdfasdfasdf"
<smoser> if you call the pastebin there, like that. it will post power params successfully.
<roaksoax> smoser: awesome thanks
<roaksoax> smoser: i'm trying to debug why it is not executing the script automatically but it is doing what it should if I run it manually on the ephemeral image
<roaksoax> smoser: any help would be appreciated
<roaksoax> http://paste.ubuntu.com/1258793/
<smoser> roaksoax, http://paste.ubuntu.com/1258807/
<smoser> replace the one there with this paste ^
<smoser> and then call it like
<smoser> ./maas-signal --config my.config --power-type=ipmi '--power-parameters={}' WORKING "credsasdasdfasdfasdf"
<roaksoax> smoser: ok cool
<smoser> roaksoax, how did you think it was going to run that
<smoser> in your paste i dont see how it would run anything
<roaksoax> smoser: yeah just noticed :)
<smoser> oh.
<smoser> ah
<smoser> add_bin != add_script
<roaksoax> smoser: yeah, so, how do you think it should be called?
<roaksoax> smoser: should we do it from maas-signal directly?
<smoser> hm..
<smoser> roaksoax, i think i'd just have main in that top level script handle this specifically.
<smoser> ie, add_bin to add it
<smoser> and then from main just invoke your script into a temp output
<smoser> then call maas-signal .... --power-par...
<roaksoax> smoser: so maas-signal is only called once right?
<smoser> no.
<smoser> it gets called before and after each script with WORKING
<smoser> and a status [1/X]
<smoser> and then once at the end
<smoser> it posts all the files
<smoser> with OK
<smoser> you can either add the params to the "OK" call
<smoser> or make a WORKING call separate
<smoser> i'm pretty sure the WORKING no longer do anything
<smoser> :)
<smoser> other than return "OK"
<smoser> roaksoax, does that all make sense?
<roaksoax> smoser: http://pastebin.ubuntu.com/1258893/
<smoser> roaksoax,
<smoser> a.) uotes are bad
<smoser> b.) you have to say "WORKING"
<smoser> (and you have to say that before the one runs finished.
<smoser> well, quotes are bad is what it really amounts to.
<smoser> that make sense?
<smoser> i have to run for a few hours
<roaksoax> smoser: ok it works... but the settngs are not set in the server :S
<roaksoax> q
<smoser> roaksoax turn debug on on the server
<smoser> and you might see
<smoser> http://paste.ubuntu.com/1258943/
<smoser> and look in the DB
<smoser> http://paste.ubuntu.com/1258944/
<smoser> that above is after i did:
<smoser>  ./maas-signal --config my.config --power-type=ipmi '--power-parameters={"a":"b"}' WORKING "credsasdasdfasdfasdf"
<smoser> it insists that it be valid json
<smoser> but thats about it really
<roaksoax> smoser: ah yes... i wonder why ...
<smoser> roaksoax, any thing furthre?
<smoser> we're pretty close. definitely the cmdline tool there can now post data.
<smoser> and it seems like you have a pretty good handle on getting it from the ipmi card
<smoser> if you'd like we can add some data manipulation into the 'maas-signal' if that is easier to work with rather than you feeding it a json blob
<roaksoax> smoser: yeah that could be even easier
<roaksoax> smoser: and no i dind't get any further
<smoser> ok.
<smoser> i have to run, and do not intend on being back tonight.
<roaksoax> smoser: ack!
<smoser> you figure out something that makes sense to you as a format
<smoser> have your tool dump that to a file
<roaksoax> smoser: i was just planning on "ip_address,user,pass"
<smoser> and i can write something to read and post it back
<roaksoax> smoser: as an argument
<smoser> sure. thats easy.
<roaksoax> and then split it in python and make it json
<smoser> yep.
<smoser> k. good night.
<roaksoax> night
<bigjools> morning
<roaksoax> bigjools: howdy
<roaksoax> bigjools: the meta-data api stuff for the power parameters doesn't seem to be saving
<bigjools> roaksoax: show me your client code
<roaksoax> bigjools: http://paste.ubuntu.com/1259090/
<roaksoax> bigjools: check the part of power_parms in maas-signal
<roaksoax> bigjools: I see things like : MAASAPIBadRequest: Bad power_type 'ipmis'
<roaksoax> if I sent a bad power type
<bigjools> power_parms
<roaksoax> or bad parameters
<bigjools> there's your prob
<bigjools> oh wait
<roaksoax> bigjools: power_parms is converted into json, then passed to the server, we know it gets there/... but doesn't save
<roaksoax> bigjools: here's the whole thing: http://paste.ubuntu.com/1259093/
<bigjools> roaksoax: `are you putting it in POST data?
<roaksoax> bigjools: yes, it is exaclt the same as sending stuff the other commissioning stuff
<roaksoax> bigjools: so for example, if the power_params are not json, MAAS complains
<bigjools> ok so it gets that far then
<bigjools> how do you know it's not saving ?
<roaksoax> bigjools: http://paste.ubuntu.com/1258943/
<roaksoax> bigjools: becuase they are not displayed on the UI for the commissioned node
<roaksoax> bigjools: so it doesn't auto start the node
<bigjools> does everything else sent in the signal() get saved?
<bigjools> files, basically
<roaksoax> bigjools: yes
<bigjools> well that's odd then
<bigjools> can you check the database itself
<bigjools> rather than UI
<roaksoax> bigjools: >>> node.power_type
<roaksoax> u''
<bigjools> argh
<bigjools> I see the bug
<bigjools> is node.power_parameters set?
<roaksoax> >>> node.power_type
<roaksoax> u''
<roaksoax> >>> node.power_parameters
<roaksoax> {u'power_address': u'192.168.2.121', u'power_pass': u'KP0eOSy9Q9X4RZ', u'power_user': u'maas'
<roaksoax> it seems like it is
<bigjools> the code forgot to set the power_type
<bigjools> you can hack it locally to continue testing
<bigjools> I'll land a  fix
<bigjools> sorry!
<roaksoax> bigjools: hehe no worries :)
<roaksoax>  bigjools now it workds :D
<roaksoax> yuaaaay
<bigjools> \o/
<bigjools> FWIW you should be able to call that any time to update power params
<roaksoax> bigjools: ok cool, but it is authenticated right?
<bigjools> all metadata access is authed
<bigjools> well, 99%
<roaksoax> bigjools: we might need that unauth for enlistment though :(
<bigjools> roaksoax: didn't we have this discussion already? :)
<bigjools> the answer was that you don't, the oauth key should be passed to enlisting nodes
<roaksoax> bigjools: yeah but smoser had another idea that we discussed this morning
<roaksoax> bigjools: it is basically to only set a temporary user/password for IPMI and send it back in enlistent
<bigjools> ok
<bigjools> let me think about this later
<roaksoax> sure, talk to smoser though :)
 * roaksoax has been all day working on this thing
<roaksoax> so happy it works now
<roaksoax> lol
<bigjools> heh
<bigjools> this is why we are developers
<bigjools> better than drugs
<roaksoax> lol
#maas 2012-10-04
<bigjools> damn, need to do something about the slow tests in maasserver
<roaksoax> bigjools: alright, all yours:
<roaksoax> bigjools: https://code.launchpad.net/~andreserl/maas/maas_ipmi_autodetection/+merge/127911
<bigjools> ok
 * bigjools slaps apt
<roaksoax> alright, I'm off
<roaksoax> it is being a long day
<roaksoax> ttyl
<bigjools> cheers
<Daviey> roaksoax: Oh wait!
<smoser> bigjools, i was looking at that signal stuff
<smoser> am i right that WORKING messages just get sent to /dev/null ?
<smoser> i'm not really here, but our the idea was to have enlistment set up a new user in ipmi
<smoser> and post that home
<smoser> but in enlistment we don't have any creds.
<smoser> ie, in enlist we'd create a temporary user 'maas-commissioning' in the ipmi and a password and such.
<smoser> maas would then use that creds for commisioning and would wipe it after that.
<smoser> that way, you always know that 'maas-commisioning' user is not intended for anything
<smoser> and can be thrown away.
<smoser> and after 'accept', its ok to trash the 'maas' user that might have been there.
<smoser> and now, i'm gone
<smoser> good night.
<bigjools> we can add power parameters to the enlistment API call
<bigjools> then you can do what you like
<jam> matsubara-afk: if you are around, for bug #1061286, what value did you enter into juju? (was it an integer or a decimal)? 'cpu_count' should be an integer, so I'm wondering who is converting it (or maybe you just entered 1.0 yourself)
<ubot5> Launchpad bug 1061286 in MAAS "juju bootstrap returned ERROR Invalid 'cpu_count' constraint '1.0'" [Critical,Triaged] https://launchpad.net/bugs/1061286
<roaksoax> bigjools: I addressed smoser's suggedstions, could you please :)? https://code.launchpad.net/~andreserl/maas/maas_ipmi_autodetection/+merge/127911
<bigjools> roaksoax: you might as well land it, I'll approve.  It's hard to make suggestions at this stage because anything I would change would be quite substantial I think.  I don't want to block you.
<roaksoax> bigjools: alright, may I ask what you see wrong with it? :)
<bigjools> no tests for starters :)
<bigjools> nothing fundamental, I'd just structure it very differently based on TDD
<roaksoax> bigjools: right, but there's things that might be more difficult than others to have tested, maybe this is the case, maybe not
 * bigjools â lunch
<bigjools> it's hard time-wise at the moment but we'd be very happy to help with that
<jam> roaksoax: I think the big thing is that if you refactor into small functions, then you can test the functions in isolation, rather than trying to test one-big-main function.
<jam> bigjools: if you're back from lunch, 'make sampledata' is failing because NodeGroup doesn't match. Was there an obvious change recently to fix that?
<jam> it appears that the 'master' nodegroup hasn't been defined?
<jam> though I can see it... :(
<jam> so, if I change all of the references to [master] to be 1 (the number of the master nodegroup) it works fine
<jam> I'm guessing whatever query it is supposed to be doing is querying on the wrong field?
<jam> bigjools: it looks like 'get_by_natural_key' changed from being the nodegroup's name to being the nodegroups UUID, but the sampledata wasn't updated
<jam> or maybe it has always been uuid, but the uuid used to be 'master' as well?
<jam> rvba seemed to do that change, I'll try to bring it up with him
<bigjools> jam: yeah it'll be rvb's change
<jam> bigjools: it looks like he did that Sept 10th, seems odd nobody would have noticed since then.
<jam> Should we be adding 'make sampledata' to the Jenkins run?
<bigjools> jam: we're obviously not depending on sampledata.  Which is good :)
<jam> well, *I* use it for manual testing but yeah I agree with the basic premise
<jam> rvba: can you look at https://code.launchpad.net/~jameinel/maas/sampledata-nodegroup/+merge/127936
<jam> it seems your NodeGroup changes broke the sampledata code.
<rvba> jam: ok
<jam> rvba, allenap: Via the API, how does MAASClient send a list? Do you json.dumps() it and pass it as value=JSON_STR or do you pass it just as value=list and the list get URL encoded into the parameters?
<rvba> jam: your change seems to be reasonable but nothing should be broken right now: when the sample data is loaded, the uuid of the master node group is 'master'.  It's changed to a proper uuid only when the (master) cluster controller connects to it.  What kind of breakage did you see?
<jam> rvba: 'make sampledata' for me fails with a NodeGroup matching query does not exist.
<rvba> jam: on trunk?  I don't get that failure :/
<jam> rvba: and if I look at bin/database shell I see that the nodegroup has a real UUID
<rvba> I see, the master nodegroup uuid has been changed.
<jam> rvba: how do you make sure the db is fully clean? isn't that what 'make sampledata' is doing?
<rvba> And you try to reapply the sampledata.
<jam> I did run 'make distclean' in the middle.
<jam> rvba: well, this is in a tree that I have done work it and done a run in the past, and I'm trying to bring in new sampledata, yes.
<jam> I'm fine nuking it
<jam> but 'make distclean' didn't seem to do that.
<rvba> You should not have to nuke it to load the sampledata.
<jam> rvba: sure, which is why using 'id' which doesn't change makes it easier :)
<rvba> Right, the only problem is that you're now relying on the fact that the master nodegroup has id 1.  That is reasonable but it's still implicit.
<rvba> This nodegroup is created by ensure_master().
<w7z> jam: I've been doing rm -rf db
<jam> rvba: well, if you are changing the UUID we can't trust that, and that is the current 'natural' key.
<jam> w7z: well, with the patch I mentioned, it works whether you rm or not
<jam> w7z: also, don't you need to kill postgres before you delete the dir?
<rvba> jam: true, that's why I've approved your branch.
<jam> rvba: yeah, I would be happy to comment it, but since that isn't possible either :)
<jam> well, maybe it is
<rvba> jam: a better solution would be to have the masternodegroup in the sample data as well with an explicit id=1.
<rvba> Instead of relying on ensure_master to create it with the right id.
<rvba> But I'm probably nitpicking here.
<bigjools> jtv: I'm not sure if https://bugs.launchpad.net/maas/+bug/1061409 is the cause of commissioning getting blocked or not, I hack a fix in and it still blocks but I can't tell if it blocks in the same place
<ubot5> Ubuntu bug 1061409 in MAAS "upload_dhcp_leases runs even when interface is unmanaged" [Critical,Triaged]
<bigjools> oops wrong bug
<bigjools> ubot you suck
<jtv> Ahem.  You're blaming the wrong bot.
<bigjools> that bug is "MAASAPINotFound: Unknown metadata attribute: public-keys during commissioning"
<jtv> I think the fix would be: if there are no SSH keys, return a success response that does not contain any keys.
<jtv> That particular API doesn't use JSON, so AFAICS an empty string would be the appropriate response.
<bigjools> I hacked it to return ""
<jtv> Right.
<jtv> Same error, or different one?
<bigjools> commissioning hangs for me still but like I said I can't tell if it's the same :/
<jam> w7z: there is a bug matsubara just opened about getting a failure trying to commision passing cpu_count
<jam> I tried it locally, and couldn't reproduce (without juju in the loop.)
<jtv> Is that the one where cpu_count is 1.0?
<jam> jtv: yes
<jam> w7z: can you look into it?
<jtv> I would guess that's unrelated.
<w7z> heh... that's interesting
<jam> w7z: bug #1061286
<ubot5> Launchpad bug 1061286 in MAAS "juju bootstrap returned ERROR Invalid 'cpu_count' constraint '1.0'" [Critical,Triaged] https://launchpad.net/bugs/1061286
<jam> w7z: note that I can try 'masscli api m nodes acquire cpu_count=1.0' and it just tells me there are no nodes to acquire, but doesn't give me an InvalidConstraint failure.
<w7z> what's the least stupid way of fixing that? changing int(n) to int(float(n))?
<jam> w7z: read the bug
<jam> do we want to allow float?
<jam> if so, do we want to round?
<jam> what should be happening if you pass something like 'abcda'
<jam> we want it to fail sanely as much as possible
<jam> looking at the report, diogo had a traceback, but it looks like that was just in the log
<w7z> ah, your comments covers some stuff
<jam> it may not have been reported as anything other than BAD_REQUEST
<w7z> right, answer to 1) is everything corretly happened
<w7z> the juju output is good, and I think the maas log is just being thorough
<w7z> can fix in juju maas provider and make maas more liberal
<jam> w7z: what is weird is that when I use 'maascli' I don't get InvalidConstraint
<jam> w7z: where did the 1.0 come from? Is that just the default for 'cpu' ?
<w7z> yup.
<jam> w7z: which means this is critical to fix, since it breaks everyone :)
<w7z> it's trivial at least
<jtv> bigjools: is it possible that you get a traceback on the server if there are no SSH keys *but* the client just shrugs it off and moves on, and then hangs for unrelated reasons, with or without your fix?  The bug as originally reported didn't seem to affect enlistment at all -- it was really just an annoying but otherwise harmless traceback in the server logs.
<bigjools> jtv: that is entirely possible
<w7z> if asked, I'd have said int('1.0') returned 1...
<bigjools> the original bug was commissioning too, not enlistment
<jtv> Strange.  I saw that in the title, but the description said enlistment.
<w7z> jam: if diogo wants to go again, he can set the cpu constraint to something else... I don't think it gets auto floatified
<w7z> right, time for me to leave
<jtv> I thought all numbers got floatified in JS.   :)
<jtv> Just like in good old Sinclair BASIC.
<jtv> And probably a few others that I didn't bother with as much.
<w7z> they do, but fortunately they're porting juju to go, not js :D
<bigjools> jtv: commissioning is grabbing meta data then hanging
<bigjools> user-data is last thing it grabs
<jtv> I guess the server logs would tell you whether that succeeded.
<bigjools> then sits at a login prompt on the console
<jtv> Login prompt?  Hmmm
<jtv> I guess the node's status in the DB stays COMMISSIONING?
<bigjools> yes
<bigjools> something going wrong in cloud-init I guess
<jtv> Or the commissioning script?
<bigjools> yeah
<allenap> jam: Sending a list is the same as sending multiple values in a query string: var=value1, var=value2, etc. Is that what you were after?
<jam> allenap: right, do I send it as a list, or do I json.dumps it first, sounds like I just send a list
<allenap> jam: Yeah, just send a list.
<allenap> jam: Now... I stumbled across something in Piston a while back... it looked like it might be able to accept JSON or YAML in place of multipart/form-data request bodies (pickle too, but they has wisely disabled that).
<jam> mgz: you're at gavins?
<jam> bigjools: hangout URL?
<mgz> jam: at david's, gavin will be here this afternoon as well
<mgz> but I may have trouble getting on hangout
<jam> mgz: want to join up on mumbel?
<mgz> jup
<jam> mgz: https://code.launchpad.net/~jameinel/maas/tag-updating/+merge/127962
<mgz> ta
<Fajkowsky> hey guys I am still have problem with my maas and I am maiking fresh instalation of maas. I want make my server work with existing DHCP server on my router, how can I do this?
<Fajkowsky> In install tutorial is only say how "Set up a fresh DHCP server to work with MAAS."
<bigjools> Fajkowsky: you have to set it up so it works for PXE booting
<bigjools> so add next-server and filename settings
<bigjools> next-server=<your maas server>
<bigjools> filename=pxelinux.0
<Fajkowsky> so I will have two maas servers?
<bigjools> no
<bigjools> you need to tell the existing DHCP server to pxe boot via your maas server
<Fajkowsky> my router will that know?
<Fajkowsky> ok
<bigjools> you need to configure your routers dhcp settings
<bigjools> mgz: I think you broke juju
<bigjools> juju deploy doesn't work against maas
<Fajkowsky> ok I try do this
<mgz> bigjools: see discussion earlier
<bigjools> mgz: on here?
<bigjools> mgz: I don't see any error, it says it was successful but no nodes are allocated at all
<mgz> do --constraints="cpu=1"
<mgz> it's an unfortunate aspect of juju design that errors from deployment don't get reported back to the client in any sane manner
<mgz> you should see one from bootstrap though, so I guess you already had an earlier environment bootstrapped?
<Fajkowski> bigjools: I still don't understand something. How can I add next-server?
<bigjools> mgz: I saw no errors from bootstrap
<bigjools> mgz: deploy just gives me instance-id: pending but I don't see even an "acquire" call in the maas log
<mgz> ...I wonder if that's the same bug then...
<bigjools> Fajkowski: I don;'t know, you have to read your router's manual and see if it supports PXE
<Fajkowski> ok
<bigjools> mgz: indeed.  this is version 0.5.1+bzr566-1 of juju
<mgz> okay, not bug 1061286 and I claim innocence until proved otherwise :D
<ubot5> Launchpad bug 1061286 in MAAS "juju bootstrap returned ERROR Invalid 'cpu_count' constraint '1.0'" [Critical,Triaged] https://launchpad.net/bugs/1061286
<bigjools> mgz: it's not that one no :)
<bigjools> this is weird nonetheless
<Fajkowski> bigjools: But there is option to install on "Existing DHCP server you don't control."
<mgz> bigjools: so, the steps here are #1 run juju with -v and see if that gives more details
<bigjools> Fajkowski: that just means that you have to configure an external DHCP server
<mgz> #2 use nova client to see if a machine got booted at all
<mgz> #3 if so, ssh on and look at cloud-init and juju logs in /var/log for clues
<bigjools> mgz: it's not getting anywhere near that far
<bigjools> mgz: there's simply no "acquire" call being passed to MAAS's API
<bigjools> yet juju decides it got allocated a node.  Bizarre.
<mgz> so, just #1 then.
<bigjools> yup
<mgz> but maybe the logging in the maas provider isn't helpful, and you need to hack a little more in
<bigjools> possible yeah :/
<bigjools> but not sure why this would suddenly break
<mgz> bigjools: any luck? the juju version you're using does not even have any of the maas provider changes, so I don't have any clever guesses
<bigjools> mgz: yeah I just worked that out.  Not really getting anywhere, so trying from square 1
<mgz> poke me if I can help with anything
<bigjools> mgz: thanks
<jtv> Reviewers needed!  https://code.launchpad.net/~jtv/maas/bug-1058282-ditch-api/+merge/127959
<mgz> jtv: the branch name scared me when I got the email, but code looked good :)
<jtv> mgz: I go for shock value
<jtv> BOOM!!
<mgz> jtv: is there a plan on profile name too?
<jtv> Yes.  It'll have to work like juju: there is one, default profile -- until you define a second one and then you must still specify one each time.
<mgz> juju does have an envvar to back that up now as well
<mgz> which is less annoying than -e all the time
<jtv> Funny, that's exactly what I just outlined myself.
<jtv> But time is short.
<bigjools> mgz: so, no nearer to finding anything out.  I added full logging on maas and there's no request to even allocate a node!  Yet it thinks it has got one ...
<bigjools> I suspect a nasty juju bug as there's some stale data hanging around somewhere
<rvba> jtv: could you please have a look at https://code.launchpad.net/~rvb/maas/bug-1061409/+merge/127979
<rvba> jtv: I'll review the branch you just pupt up for review.
<jtv> OK
<rvba> ta
<jtv> My system's not very responsive at the moment, which may complicate things.
<mgz> bigjools: and the juju client logging isn't anything useful? I'd add things there
<bigjools> mgz: I gotta try and remember my way around it :(
<mgz> bigjools: I'd suggest branching lp:juju then adding some log calls in juju/providers/maas/ maas.py and maybe launch.py then `python setup.py install --local`
<bigjools> yep
<bigjools> mgz: .... it's not even calling start_machine .... !
 * bigjools boggles
<jtv> Does it assume that's already done!?
<mgz> bigjools: my guess, you have some stale values in the file storage?
<bigjools> mgz: I destroyed env.... still does it.
<bigjools> but that could be it
<mgz> what does a full list on maas's file-ish backend give?
<bigjools> reluctant to destroy again since it means waiting for a node to install
<mgz> because this seems like a bug that would be worth fixing
<mgz> don't destroy, just list the file stuff
<bigjools> just loads of files from today
<bigjools> mgz: two charm files, bootstrap-verify and provider-state.  Seems normal.
 * jtv reboots in hopes of a better system
<bigjools> mgz, how do I get round the contraints bug on bootstrap?
<bigjools> and wb :)
<mgz> bigjools: so, as I was trying to say before network bounced me, bootstrap-verify and provider-state are what we need to know the contents of
<mgz> bigjools: so, as I was trying to say before network bounced me, bootstrap-verify and provider-state are what we need to know the contents of
<mgz> and my constraints work around doesn't work, but I'm submitting a juju fix now that you can pull in
<bigjools> I went back to system juju for now
 * bigjools relocates
<mgz> bigjools: so, inbetween all my dropping there, did you find an answer to your original mystery?
<mgz> I'm just submitting a juju-side fix now, network-permitting
<bigjools> mgz: no :(
<mgz> (third attempt) so, as I was trying to say before network bounced me, bootstrap-verify and provider-state are what we need to know the contents of
<bigjools> unfortunately they are no more :(
<bigjools> Daviey: are you currently able to run commissioning successfully?  It is hanging for me.
<Daviey> bigjools: I haven't tried.  Want me to?
<jtv> Any news on where it hangs?
<bigjools> Daviey: it would be useful to confirm or deny my setup, thanks
<bigjools> Daviey: use latest daily package
<bigjools> jtv: I see metadata requests, then the console hits the login prompt
<bigjools> and that is it
<jtv> And no "signal" call.  :(
<Daviey> bigjools: i can only really test it on garage lab atm, which i'd rather avoid.
<bigjools> Daviey: figures :)
<Daviey> I'll soon have access to a rig in bluefin
<bigjools> mgz: so even after destroy-environment and wiping all filestorage on maas, it is still failing to acquire a node on deployment !
<rbasak> roaksoax: please could you take a look at bug 1061547?
<ubot5> Launchpad bug 1061547 in MAAS "maas-import-squashfs: line 47: RELEASES: unbound variable" [Undecided,New] https://launchpad.net/bugs/1061547
<jam> bigjools: I'm ready to start doing the maasserver side of farming work out to celery workers. You mentioned there should be lots of tests to crib from
<jam> however, there is only 1 test that 'uses_rabbit_fixture'
<jam> and it is disabled because it broke the test suite in other places.
<bigjools> celery fixture
<bigjools> not rabbit
<jam> bigjools: ah ok, lots of CeleryFixture
<bigjools> :)
<jtv> Any reviewers around?  https://code.launchpad.net/~jtv/maas/bug-1058282-ditch-api/+merge/127959
<mgz> jtv: poking allenap at that now
<jam> jtv: why 'pardir' repeated, instead of 'dirname()' repeated?
<jtv> Because it doesn't expect me to nest.
<jtv> Thanks mgz, but I think jam's already looking
<jam> technically, I think he was looking first :)
<bigjools> smoser: hi - commissioning is currently hanging for me after getting metada, have you seen this?
<jam> jtv: this does mean that the names people login with potentially collide, doesn't it?
<jam> maascli $LOGIN do stuff
<jam> so if we add a module 'foo'
<jam> and somebody called their login 'foo'
<jam> it would be ambiguous
<jam> I'm willing to have people deal with that (or re-add 'api') once we actually support it, though.
<jam> [You could select non-api via an option instead of an argument, for example]
<jam> [maascli --mod=api $LOGIN do stuff]
<jtv> If we do add another module, api will be the default.  In other words, there'll be some way of selecting another module explicitly if you want one.
<jtv> If you look at the MP, it says that "api" is a sensible default even if we add more modules.
<rbasak> roaksoax: also bug 1061577
<ubot5> Launchpad bug 1061577 in MAAS "IPMI configuration on highbank appears to fail" [Undecided,New] https://launchpad.net/bugs/1061577
<jtv> jam: selecting another module would probably happen through an option.  The fact that we can also come up with wrong ways to do it doesn't affect us much.  :)
<jtv> Thanks jam!
<bigjools> mgz: found the source of the fault in the provisioning agent log: http://pastebin.ubuntu.com/1259905/  NFI why that happens though
<mgz> ah, this looks fun
<mgz> guess blame ipv6?
<mgz> error looks like when trying to resolve the maas api endpoint at any rate
<mgz> so, try doing that outside of juju, using twisted, see if you can get the same failure
<jam> rvba, bigjools: So I have a test that properly triggers calling the provisioningserver, however, all my tests are failing while the task code is trying to call back to the API
<jam> (getting a urllib2.open error)
<jam> I'm calling NodeGroup.objects.refresh_workers()
<jam> because if I don't, I get failures that we don't have credentials to contact the maas_url info
<rvba> jam: you probably have to patch the method that calls back to the API so that it talks to the test API.
<jam> rvba: so... how do I actually test that the api calls are valid?
<jam> if I patch both side
<jam> then I'm not actually testing the thing, you know, works
<rvba> jam: you won't patch it entirely, probably just the url used will be enough.
<jam> rvba: so there is a test service running?
<jam> what URL do I need to use?
<bigjools> mgz: what makes you say ipv6?
<jam> (I can certainly set the maas_url to whatever we want)
<jam> I would have thought the tests were already doing that, in fact.
<rvba> jam: well, you need to simulate what self.client is doing.
<rvba> jam: more precisely, patch the thing that calls the API with self.client.
<jam> rvba: well MAASClient has a different api than self.client
<jam> specifically, test.client takes {'op': ...} but MAASClient takes 'op' as a regular argument.
<rvba> What do you mean 'a regular argument'?
<jam> rvba: In a test case you do: self.client.get(PATH, {'op': x, arg1: y, ...})
<jam> in MAASClient you do client.get(PATH, x, arg1=x, arg2=y...)
<rvba> MAASDispatcher.dispatch_query looks like it can be patched to use self.client instead of urllib2.Request
<rvba> def dispatch_query(self, request_url, headers, method="GET", data=None):
<rvba> jam: is that enough to get you going?  AFAIK we haven't done that before.  I'm happy to help you with your branch, especially if you have failing tests I can work on.
<jam> rvba: well lots of things failing because the nodegroup workers can't talk back to the api, but I think I can poke at it for a bit.
<rvba> Ok, cool.
<smoser> bigjools, did you get it resolved?
<bigjools> smoser: no :(  got distracted trying to debug juju problems
<bigjools> smoser: I can debug with you now if you can help?
<smoser> what is "hangs"
<smoser> do you have any console log ?
<bigjools> it doesn't complete - I see metadata requests, then it gets to the login prompt on the console and that's it
<bigjools> the other VTs have nothing on them either
<jtv> Do we still tell the nodes to use the region controller as a log server?  If so, you may find logged information on the server.
<bigjools> we do actually
<bigjools> not sure it's used during commissioning; at least I can't see any commissioning logs
<jtv> :/
<bigjools> I will try again
<smoser> bigjools, sorry for being dense. https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825/comments/275328
<smoser> "untick"
<smoser> but reguarding login prompt. thats expected.
<smoser> ttyS0 (serial console) will have log
<bigjools> smoser: uncheck
<smoser> you can change command line parameters on the kernel to remove ttyS0
<smoser> where is "uncheck"
<smoser> i'm sorry.
<smoser> its probably obvious.
<bigjools> smoser: click the green "extra options" and it opens up more text
<bigjools> and then uncheck "needs review"
<bigjools> it'll create the MP in WIP status
<bigjools> which prevents a diff being emailed out, primarily
<smoser> hm.. wel i actually would have liked ot have the diff mailed out :)
<smoser> bigjools, primarily i'm interested in comments on the test that is there.
<bigjools> smoser: well you created the MP and then changed it to WIP :)
<smoser> bigjools, as i didn't want someone to "approve" just yet
<bigjools> ok so if you do as I mentioned, changing the status to "ready" later will send the email
<bigjools> anyway
<bigjools> I need to remove ttyS0 then ...
<jam> rvba: so there is another impedence mismatch. django.test.client.Client.get() returns python objects, while MAASClient.get() returns a string
<jam> actually, I might be wrong
<jam> I am wrong
<smoser> bigjools, i have another solutoin to get some output on a tty
<smoser> that i thought of yesterday when diego complained about the same thing.
<smoser> fwiw, i still thikn 'console=ttyS0' is the right default.
<bigjools> smoser: it seems to have tty1 as default too though
<bigjools> both, I mean
<rvba> You still might have to change to result of client.get/post so that it matches whaturllib2.urlopen returns.
<rvba> jam: ^
<smoser> for any thing other than 3 systems, ttyS0 has a chance of being logged and remotely viewable, and graphics console has exactly zero percent chance of being logged.
<bigjools> yes
<smoser> bigjools, right. if there is no ttyS0 device, then kernel goes to tty1.
<smoser> (and kernel puts kernel messages on both)
<smoser> but /dev/console ends up as the last valid entry on the cmdline
<smoser> and messages from cloud-init are going to /dev/console
<jam> rvba: yeah, urlopen returns a file-like object that has a .status_code and a .content
<jam> rvba: but that looks a whole lot like what Client.get returns
<rvba> Indeed.
<jam> I imagine they are both based on Httplib's response object.
<bigjools> smoser: will it rsyslog in commissioning mode?
<smoser> it does not rsyslog to maas server.
<smoser> we could make that happen, but cloud-inti woudl have to read the cmdline params to set up syslog. and it does not support that.
<smoser> bigjools, theo ther thing we can do is that cloud-init can do python logging
<smoser> and we can feed it a config file for that
<smoser> via user-data
<smoser> the simpler thing right now is:
<smoser> http://paste.ubuntu.com/1259976/
<smoser> that will get you data on tty2
<smoser> well, more data.
<bigjools> ok I'll do that in 5 mins
<bigjools> mgz: so in another twist, I restarted the PA on the ZK node and now it works.  WTF!
<mgz> o_O
<jam> rvba: ok, so the return signatures sanely overlap,however the arguments do not.
<jam> rvba: MAASDispatcher wants 'data' which is already an encoded_multipart_data
<jam> where some arguments may be in the URL
<jam> and some are in the data itself.
<jam> so we have to parse everything back out of the data
<jam> to get it into a form to pass to a Django.Client.get()
<jam> I think I have to go up a level to MAASClient, and just curry argument parameters.
 * rvba has another at MAASDispatcher.
<rvba> another look*
<jam> rvba: dispatch_query takes a 'data' parameter, which is already encoded as multipart mime
<jtv1> Are you trying to build a dispatcher based on the django client?
<rvba> Yep, that's what he's trying to do.
<jtv1> I tried that once, but the mismatch was great enough to make it a losing proposition.
<jam> jtv1: the provisioning server needs to talk to the api to get work done
<jtv1> It does, yes.
<rvba> Well, that's basically what you're trying to do now, isn't?
<jam> Are there credentials I can pass it so it can just do it
<jam> or do we need to use the testing client and proxy it.
<jam> I'm thinking to not do the dispatcher, because it has already processed the data too much.
<jam> However, doing it at the MAASClient level looks more sane.
<jam> As you have to change some arguments
<jam> but the data is still in python-object form.
<jtv1> Time for me to bug out.  See you all tomorrow.
<rvba> nn jtv1.
<jtv1> nn
<bigjools> nn
<bigjools> smoser: nothing on tty2
<smoser> bigjools, well, i'd ditch the kernel cmdline opt and try again.
<smoser> ie, just console=tty1
<bigjools> ok
<smoser> but if cloud-init got that data, then it really should output something there.
<bigjools> smoser: you won;t believe this
<smoser> i do not
<bigjools> smoser: it just commissioned ok after removing ttyS0
<smoser> bigjools, not possible.
<smoser> its a *watched* pot that doesnt boil
<bigjools> I told you you wouldn't :)
<smoser> not a unwatched pot
<bigjools> is it likely that something is hanging on a non-existent ttyS0?
<smoser> bigjools, the kernel wrote messages to ttyS0
<smoser> and thought that it was good
<smoser> enough to assign /dev/console to ttyS0
<smoser> did you enlist the system?
<bigjools> yes
<smoser> because that does the same thing
<bigjools> so.... wtf is going on
<bigjools> I'll try again with and without
<smoser> bigjools, here.
<smoser> so, delete the system
<smoser> mount the ephemeral image loopback
<smoser> add a user with sudo and some keys there.
<smoser> then ssh into the system and poke around while its haning.
<smoser> i have been working on a script htat would do the above for you
<smoser> as it is useful for debug
<smoser> but not yet finished.
<bigjools> ok
<bigjools> I will do this tomorrow I think.  It's past midnight and I am pretty much about to flake out
<jam> rvba: so there is one more complication, the code wants to be run on N workers (serially or sequentially is fine), but that means the cached 'nodegroup_uuid' needs to be set N times for the N workers.
<jam> I might just fake something by patching the task which gets the 'queue' which happens to match the nodegroup_uuid
<jam> I want to use.
<jam> but it is... icky
<smoser> bigjools, thanks. i'll have "backdoor-image" for you then.
<rvba> jam: or you can call refresh_secrets with the right nodegroup_uuid before each task run.
<bigjools> well I'm just retrying to see if it happens again :)
<rvba> jam: but that's maybe not possible given the structure of the code you're testingâ¦?
<jam> rvba: well it is open to question how it should be working, given our earlier discussion.
<bigjools> smoser: but thanks for that, I also wonder if we shouldn't just throw the admin's ssh key on there for enlist/commission anyway?
<jam> rvba: the only obviously exposed code in Model is NodeGroup.objects.refresh_workers() which refreshes everything.
<bigjools> fwiw with the ttyS0 in, stuff is appearing on tty2 instead of tty1
<jam> though you can refresh_worker.refresh_worker() directly, I guess
<smoser> bigjools, i'm not opposed to that.
<rvba> jam: well, everything={'api_credentials', 'maas_url', 'nodegroup_uuid'}
<smoser> bigjools, but if it doesnt get to the MD, then it still doesnt work
<smoser> (the adding ssh key)
<smoser> bigjools, with the tee it should go to both tty2 and /dev/console
<smoser> (and that log)
<bigjools> smoser: so enlisting with ttyS0 there shows the log on tty2.  when commissioning, it gets as far as starting cloud-init, goes to a login prompt and there's no output anywhere
<bigjools> and on that note, I shall go assume a horizontal position
<bigjools> good night
<smoser> good night.
<jam> rvba: well, I got to the point where I can get 'HttpResponseForbidden' which matches the 'only the actual nodegroup worker can do the work'.
<jam> so that is ~ good :)
<rvba> jam: \o/.  We might want to add the little utility you've created to our testing toolkit.
<jam> rvba: lp:~jameinel/maas/populate-node-rebuilds if you are interested
<jam> the change is in 'test_tag.py'
<jam> class
<jam> which is actually pretty small
 * rvba has a look.
<jam> though the 'use the right credentials' is really snuck in there by a magic patch and structuring the request code to set credentials right before the real request.
 * jam stops for now, dinner & family time.
<roaksoax> query smoser
<roaksoax> err :)
<roaksoax> rbasak: pin
<roaksoax> rbasak: ping
<rbasak> roaksoax: pong, but on a call that's about to start
<roaksoax> rbasak: ipmi_si: Could not set up I/O space
<roaksoax> rbasak: how much RAM do you have?
<rbasak> roaksoax: 4G.
<rbasak> Which is interesting as it's a 32 bit machine
<roaksoax> rbasak: on the arm boards i meant :)
<rbasak> Yeah
<rbasak> On the ARM boards :)
<roaksoax> rbasak: (you've got nice arm boards :P )
<rbasak> :-)
<roaksoax> rbasak: so if you manually do this modprobe ipmi_msghandler modprobe ipmi_devintf modprobe ipmi_si type=kcs ports=0xca2
<roaksoax> it fails
<roaksoax> rbasak: i'd need physical access to one of those to test TBH
<rbasak> roaksoax: the third command fails
<roaksoax> rbasak: if you do this: modprobe ipmi_si
<rbasak> roaksoax: that works
<rbasak> roaksoax: well
<rbasak> roaksoax: the third command didn't fail at the modprobe. THe modprobe succeeded. "ipmi_si: Could not set up I/O space" comes out of dmesg
<rbasak> roaksoax: so it probably works a second time as the module is already "loaded"
<rbasak> rmmod ipmi_si causes a kernel oops
<roaksoax> rbasak: so is that message shown also when modprobe without the arguments?
<rbasak> roaksoax: I'll need to reboot to test again. I'll do that now
<rbasak> roaksoax: modprobe ipmi_si type=kcs ports=0xca2 produces an error and removing the module then oopses
<rbasak> roaksoax: but modprobe ipmi_si works fine and rmmod works fine too
<rbasak> roaksoax: ports=0xca2 probably doesn't make sense on ARM
<roaksoax> rbasak: indeed
<rbasak> roaksoax: can we test behaviour without the parameters?
<roaksoax> rbasak: yeah, just remove the parameters from the commissioning script in /etc/maas
<roaksoax> rbasak: reenlist and then comission
<rbasak> roaksoax: just speaking to dannf about this. He says that ipmi_si should autodetect except where ACPI tables are broken, so shouldn't need parameters on Intel either
<dannf> roaksoax: is there a reason for passing those in general?
<dannf> seems dangerous imo - i could foresee similar problems on intel that rbasak is seeing on arm
<roaksoax> dannf: right, so the testing hardware i'm using doesn't set the port correctly so that's needed otherwise it wouldn't work
<roaksoax> dannf: it would be matter of testing it manually
<dannf> roaksoax: eww - what hw are you using?
<roaksoax> dannf: HP Micro servers
<dannf> really.. and that's a plug-in card, right?
<roaksoax> dannf: yeah, ipmi cards were plugged-in afterwards
<dannf> roaksoax: sounds like something that should get fixed upstream - though i'm not sure the right way atm...
<roaksoax> dannf:i filed a bug about it, and smb said it was more likely to be a buggy BIOS
<roaksoax> dannf: and that there's nothing he could do about it
<dannf> roaksoax: that'd be true it it weren't a plugin card imo - i'm not sure how discovery of plug-in cards should work
<rbasak> roaksoax: just testing now with ipmi_si parameters dropped. What behaviour should I expect to see?
<dannf> roaksoax: imo, its probably better to have a modprobe.d/find-ipmi-card.conf on affected machines than passing the module in general - i think more hw will work that way
<roaksoax> rbasak: no stacktrace :)
<rbasak> roaksoax: of course. I was wondering about MAAS power parameters. SHould these end up getting magically set?
<roaksoax> rbasak: yes
<dannf> for example, i think dell machines use a bt interface instead of kcs.. but they may provide both for backwards compat
 * dannf thinks we need a pile of weird ipmi machines to test
<roaksoax> we do
<roaksoax> Daviey: ^^
<roaksoax> matsubara: do we have intel hardware to test ^^
<matsubara> roaksoax, intel? AFAICT, all those lenovo machines are intel
<Daviey> We know we can't work with everything
<roaksoax> matsubara: can you test the new commissioning stuff? It should be autodetecting IPMI and sending back to MAAS
<Daviey> and IPMI is a poorly standardised service
<Daviey> we can only enable hardware we have access to
<Daviey> .. Canonical Hardware Cert will check this stuff btw
<matsubara> roaksoax, will do.  maas - 0.1+bzr1139+dfsg-0+1168+116~ppa0~quantal1 should have all the changes, right?
<roaksoax> Daviey: ack!
<roaksoax> matsubara: yes should do
<matsubara> cool
<roaksoax> rvba: howdy!! so I have added a dir in src/provisioningserver/power/config/ and I need to source a file on the power scripts
<roaksoax> rvba: http://paste.ubuntu.com/1260164/
<roaksoax> rvba: but i need a way to determine the location of the config dir automatically, how should that be addressed?
<rvba> roaksoax: I think you should add a parameter to the context used when rendering the template.  That parameter will be named 'power_config_path' or something.
<rvba> roaksoax: in the python code, you'll use __file__ to compute that path.
<rbasak> roaksoax: it seems to be better now
<rbasak> roaksoax: enlistment did nothing, so I had to manually power cycle to get into commissioning
<roaksoax> rvba: so look at line 9, that's what I'm adding to the context of the templates
<roaksoax> rvba: but I need to determine the path, somehting like you would do to determine the path of the templates
<rbasak> roaksoax: commissioning seemed to work. I saw Success three times and no stack trace, and it successfully commisioned
<roaksoax> rbasak: cool
<rbasak> roaksoax: power type changed to IPMI, with username of maas and random looking password
<roaksoax> rbasak: \o/
<rbasak> roaksoax: but BMC IP address is 0.0.0.0, so it won't work
<rvba> roaksoax: right, that's where you can use __file__.
<rbasak> roaksoax: is that expected?
<rvba> roaksoax: to compute '<CONFIG-DIR>'.
<rvba> roaksoax: hold on.
<roaksoax> rvba: nope, so that means that your IPMI card didn't receive an IP address
<roaksoax> from DHCP
<rbasak> roaksoax: it definitely does have an IP address
<rvba> rbasak: ^
<roaksoax> :)
<rbasak> roaksoax: what do I do in userspace to extract the IP address in the same way that you're doing?
<roaksoax> rbasak: bmc-config --checkout | grep IP_Address
<roaksoax> rbasak: or bmc-config --checkout --key-pair="Lan_Conf:IP_Address_Source"
<roaksoax> err
<roaksoax> rbasak: or bmc-config --checkout --key-pair="Lan_Conf:IP_Address"
<roaksoax> rbasak: on the UNBOUND varilable bug, I don't see anything.. (and I thought I had fixed that already) : https://pastebin.canonical.com/75908/
<roaksoax> rbasak: are you sure the script is the latest one?
<roaksoax> rbasak: did it have an IP address
<rbasak> roaksoax: for that bug, note that I'm running out of trunk, not packaging
<rbasak> roaksoax: I noted the trunk revision in the bug
<rvba> roaksoax: I think you want to keep all of that on the pserv side: something like (warning, untested code ;)) http://paste.ubuntu.com/1260183/
<rbasak> dannf: http://paste.ubuntu.com/1260191/
<roaksoax> rbasak: ack will further investigate
<roaksoax> rbasak: cool thanks
<rbasak> roaksoax: bmc-config --checkout gives 0.0.0.0, so that'll be where the problem is
<roaksoax> rbasak: so that means that IP doesn't have an IP address :(
<rbasak> roaksoax: it does, because I'm using it
<rbasak> roaksoax: it's how I see the serial output and power control it. We've been doing that for months :)
<rbasak> roaksoax: so I presume it's a driver issue
<roaksoax> rbasak: indeed
<rbasak> roaksoax: can we assume this will get fixed? So I'd like to see ipmi_si without parameters if on ARM please
<roaksoax> rbasak: i might remove this all together
<rbasak> OK
<roaksoax> rvba: alright, so with your branch I simply reference the config_dir in the template itself and do not add a defualt power_params
<rvba> roaksoax: yeah.
<roaksoax> rvba: cool thanks
<rvba> roaksoax: this way, this stays on the cluster controller side.
<rbasak> dannf: http://paste.ubuntu.com/1260207/
<rbasak> roaksoax: in your case, is your BMC set to DHCP?
<rbasak> Do you see Use_DHCP in your version of my paste above?
<roaksoax> rbasak: the scripts sets it to DHCP if it is Static
<rbasak> roaksoax: OK, but what is your bmc-config --checkout output for that section please, once it is set to DHCP?
<roaksoax> rbasak: yes it is exactly that one
<rbasak> roaksoax: with 0.0.0.0, or with an IP address?
<roaksoax> rbasak: IP address
<rbasak> roaksoax: OK, thanks!
<roaksoax> rbasak: it is set to 0.0.0.0 when the card hasn't yet obtained an IP from DHCP
<roaksoax> rvba: so is this correct in terms of the templating? http://paste.ubuntu.com/1260213/
<rvba> roaksoax: I'm not sure why you need to do "ipmi_config={{ipmi_config}}"â¦?
<rvba> Why not simply: config={{ipmi_config}}/{{config_dir}}
<rvba> ?
<roaksoax> rvba: cool, will do that
<dannf> rbasak: hrm.. ok, so maas is using the host to configure access to the BMC - and that is also how it figures out the BMC/HOST mapping?
<rbasak> dannf: that's right
<dannf> s/that is/is that/
<rbasak> dannf: each node has separate and independent IPMI settings
<flacoste> roaksoax, smoser: are we ready to move IPMI auto-configuration to QA, or we are still missing the enlisting piece?
<roaksoax> flacoste: it can be moved to QA for commissioning... enlistment is not yet doing anything IPMI related
<flacoste> roaksoax: so it's not ready for QA then
<flacoste> roaksoax: we need to be able to control power to go to commissioning
<smoser> missing enlistment.
<smoser> we have no way to post creds in enlistment.
<smoser> we'd have to add that to enlist api
<roaksoax> yeah
<smoser> probaly not hard though i would guess
<smoser> hm.
<roaksoax> flacoste: in order for us to do it we need to be able to send un auth settings during enlistment
<roaksoax> flacoste: which maas doesn't yet support
<flacoste> smoser: so we are still missing the API bit there? i thought bigjools was taking care of this
<flacoste> or we missed out the hand off yesterday evening on that?
<roaksoax> flacoste: we do have it for commissioning as part of the metadata API but not for enlistment
<roaksoax> not for MAAS API
<flacoste> right, we are missing the enlistment API change
<smoser> correct
<dannf> roaksoax: fyi - got a ping in w/ an hp contact - he said he'd poke the microserver guys about this
<flacoste> rvba, allenap: any chance one of you could take care of this to unblock smoser and roaksoax?
<roaksoax> dannf: awesome! thank you!
<rvba> flacoste: I won't be able to do that right as I have to step out for a few hours.  Can you take care of that allenap?
<roaksoax> rvba: the config dir thing breaks a lot of tests :(
<rvba> roaksoax: that's surprising.
 * rvba has to step out for a few hours.
<allenap> flacoste, smoser, roaksoax: I'm here for a few minutes now, but I'll be back at ~1900 UTC for longer. What's the problem?
<smoser> we want to have enlistment take power_parameters
<allenap> smoser: Right, okay.
<flacoste> allenap: we are going to create a temporary IPMI user during enlistment, so the enlisment API call needs to accept IPMI creds to associate to the node
<allenap> flacoste, smoser: I've just added a card to the board for that, I'll pick it up later. However, there's an in-review card assigned to bigjools that sounds similar. Do you know anything about that?
<rvba> allenap: it's in the done lane I think.
<rvba> allenap: "Add power parameter setting to metadata API" ?
<rvba> Very similar indeed.
 * rvba really steps out now.
<flacoste> allenap: yep, that was for final IPMI settings after commissioning
<smoser> rvba, allenap flacoste metadata API is done.
<smoser> we need enlistment api
<allenap> Okay.
<roaksoax> allenap: btw... squid-deb-proxy messes up squashfs image installation
<roaksoax> allenap: it would have been great to have that tftp fix to avoid having the proxy messing up with us :(
<allenap> roaksoax: We should not be using tftp for moving such big files. Why/how does squid-deb-proxy affect us?
<roaksoax> allenap: it blocks the download
<roaksoax> allenap: i'm fixing it in packaging
<allenap> matsubara: This one's for you :) https://code.launchpad.net/~allenap/maas/headers-optional/+merge/128075
<matsubara> allenap, 121 +        Otherwise otherwise write it raw to stdout.
<matsubara> I'm not a reviewer, but it looks good to me :-)
<guimaluf> roaksoax, the maas nodes depends on maas server to boot? becausa I shutdown mass server, and my node can't boot anymore, even if I make hard disk first boot choice.
<roaksoax> guimaluf: what's the state of the nodes in MAAS? 'ready'??
<guimaluf> roaksoax, yes
<guimaluf> roaksoax, ready
<roaksoax> guimaluf: so no OS is installed :)
<guimaluf> roaksoax, I though MAAS would install a fully OS inside the nodes. It rather only copy the OS every boot time?
<guimaluf> roaksoax, this makes MAAS server a huge single point of failure, right?
<roaksoax> guimaluf: maas installs the OS when you *deploy* a node
<guimaluf> roaksoax, so, why I can't boot the node with the maas server off?
<roaksoax> guimaluf: becuae ethere's no OS installed
<roaksoax> guimaluf: you enlist, you commissioning, and the node is 'Ready' to be deployed
<guimaluf> roaksoax, it has been deployed yet...
<roaksoax> guimaluf: once you deploy a node (either from the MAAS WebUI/upcoming CLI or juju) then you will have Ubuntu installed on it
<smoser> roaksoax, fyi: http://bazaar.launchpad.net/~smoser/+junk/backdoor-image/view/head:/backdoor-image
<smoser> i've not actually tested that it works. but only that it lookss ane.
<roaksoax> cool
<roaksoax> matsubara: have you seen upgrade errors between versions?
<roaksoax> (as in database errors)
<matsubara> roaksoax, nope, I'm going to try the upgrade soon once I get the test beds ready
<roaksoax> matsubara: ok, cool. Could you please test this though: 1. install quantal maas archive. 2. upgrade to a newer release, 3. update to an even newer release
<roaksoax> matsubara: i'm seeing failues in 3
<roaksoax> smoser: could you please? https://code.launchpad.net/~andreserl/maas/packaging_updates_bzr1170/+merge/128088 thanks :)
<matsubara> roaksoax, ok
<roaksoax> thanks
<allenap> matsubara: Good catch. I'll self-review it I think.
<allenap> jam: Ah, you're around. Want to trade reviews? If you're about to sign off I'll still review yours, and self-review mine.
<jam> allenap: I can give it a view
<jam> no guarantees (midnight here) if it is involved.
<allenap> jam: Ta. https://code.launchpad.net/~allenap/maas/headers-optional/+merge/128075
<jam> Mine is, unfortunately, a bit extensive
<allenap> jam: No worries. Go to bed; leave mine then.
<jam> allenap: get_Response_content_type() why indirect via Message? Because you want a ContentType object?
<jam> vs just the text?
<jam> it might be clearer if the variable name 'content_type' was distinguished from the object being returned.
<jam> allenap: otherwise +1
<allenap> jam: Message has some smarts about separating parameters out from the main part of the content-type header, and I couldn't find them stand-alone anywhere.
<allenap> Message.__init__() is lightweight, so the overhead is minimal. Granted, it does look odd.
<jam> allenap: so maybe a comment about it
<allenap> Cool, good idea.
<allenap> jam: I get a couple of test errors from your branch, http://pastebin.ubuntu.com/1260714/
<jam> allenap: the first one is because nose is overagressive, if you even think the word 'test' it tries to run it.
<jam> so I'll try to rename that.
<jam> The others are as mentioned in the overview, you have to manually specify that you're going to use tags before you use them, so it can patch out the api for the test.
<jam> allenap: any ideas how to make that less terrible are very welcome.
<jam> allenap: I should have a fix for those 3 pushed up as soon as the test suite finishes running here.
<allenap> jam: `from nose.tools import nottest` gets you a decorator to stop it from dry humping anything with "test" in the name.
<jam> allenap: thanks for the heads up. it is hard to write 'test infrastructure' code without using the word test.
<allenap> Yeah!
<smoser> roaksoax, line 83 is strange
<smoser> you check an explicit path for something and then invoke it not by its path
<roaksoax> smoser: i'm checking whether it exists or not
<roaksoax> smoser: if invoke-rc.d exists use it
<roaksoax> smoser: that's how everything else was done
<smoser> that makes sense, ye. but if you check by explicit path then you should use explicit path.
<smoser> hascmd() { command -v "$1" >/dev/null 2>&1 };
<smoser> if hascmd invoke-rc.d; then
<smoser>  invoke-rc.d squid-deb-proxy restart || true
<roaksoax> smoser: right, that would mean refactoring the whole thing :)
<smoser> fi
<smoser> (not really refactoring the whole thing)
<roaksoax> smoser: alright, but will try to take care of it later this week (tomorrow)
<smoser> you might as well not 'grep -qa' for a string if you'r egoing to conditionally replace
<smoser> in the sed
<roaksoax> smoser: it is grep -qs
<smoser> (ie, other than sed not opening RW there is no '#maasurl$" then it does nothing)
<smoser> sed does exactly what you did in the grep
<smoser> is what i'm saying
<smoser> just delete the grep.
<roaksoax> smoser: ah i see
<roaksoax> ok
<smoser> i suspect you're missing some characers that could exist in a hostname
<smoser> at least '-' i think i smissing.
<smoser> whats the Pre-Depends for?
<roaksoax> smoser: debian/maintscripts
<roaksoax> smoser: i copied it from a diff cjwatson had
<roaksoax> smoser: is to remove older files that no longer used
<roaksoax> rm_conffile
<smoser> ok.
<smoser> hm.. i'm not allowed to ack this
<roaksoax> smoser: that's weird, you were always :S
<smoser> (launchapd code reerewers
<smoser> )
<roaksoax> smoser: right bu you are under maas-maintainers right?
<roaksoax> so you should be good
<smoser> do we incur a conf-file prompt now if we change 99-maas ever?
<smoser> i'd think we would.
<roaksoax> yes we will
<roaksoax> we need to handle that differently either way
<smoser> why not just write a file in that directory
<smoser> and remove it
<smoser> rather than packaging it
<smoser> conf-file prompts suck.
<roaksoax> smoser: yeah I'm gonna install in /usr/share/maas/confs/ and copy it over and make the modifications
<roaksoax> smoser: i just wanna do all that handling in 1 branch though
<allenap> jam: Did you look at injecting a custom MAASDispatcher, or patching MAASDispatcher.dispatch_query?
<jam> allenap: dispatch_query takes a 'data' stream that has already been serialized
<smoser> roaksoax, well, its better to land that at once than in pieces
<jam> so I would have had to deserialize it
<smoser> as you're going to incur conf file pompts twice if you do it twice
<jam> to hand the python objects over to Django's Client
<allenap> Right, okay.
<roaksoax> smoser: yeah I'm dealing with a DB upgrade error at the moment
<jam> allenap: the MAASDjangoTestClient is actually quite small, because the only real difference is 'kwargs' vs '**kwargs' and op being a parameter vs a member of kwargs.
<jam> the hard bits are described, 1) do we always inject a MAASDjangoTestClient just in case a test wants to use tags
<allenap> jam: Yeah, I just wondered if it would have been easier to do. I think the division of responsibilities between MAASClient and MAASDispatcher is not great.
<jam> 2) How do we trick the system into giving a new nodegroup worker when we have a single thread.
<jam> allenap: yeah, I did try at that level for a while, but it was post-serialized vs pre-serialized data.
<smoser> roaksoax, so what do you want me to do here.
<smoser> i'd like to see the sed change at least fixed up.
<smoser> but i reallyd ont want 2 separate upgrades to prompt users
<roaksoax> smoser: there's always gonna be upgrades that will prompt users until we have all the config handling fixed
<roaksoax> smoser: conditionally approve it and I'll work on the 99-maas thingy
<roaksoax> and update the branch acordingly
<roaksoax> smoser: i updated the branch with the sed thing and the fix for the db upgrade
<smoser> i did
<roaksoax> smoser: thanks
<smoser> anyone else able to review https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825
<smoser> i think thats reasonable at this point, and QA tested... so, thats good.
#maas 2012-10-05
<bigjools> hi roaksoax and smoser
<smoser> bigjools, hey
<bigjools> this morning would have been better had my wife not woken me up from the deepest sleep ever
<smoser> bigjools, :-(
<smoser> lp:~smoser/+junk/backdoor-image/
<smoser> has "backdoor"
<bigjools> cool
 * bigjools grabs
<smoser> just point it at an image and it will backdoor it
<smoser> user 'backdoor'
<bigjools> I don't normally do backdoor
<smoser> --user bigjools
<bigjools> :)
<bigjools> it's the ephemeral image I want in this case, right?
<smoser> right.
<smoser> /var/lib/ephemeral/.....disk.img
<smoser> its probably most proper to 'restart tgt' afterwards
<smoser> but i've never actually had to do that.
<smoser> but clearly if it had open filehandles, it could be confusing.
<bigjools> yeah
<bigjools> I was going to ask - is it possible to proxy tgt via the clusters?  at the moment ephemerals get pulled from the region
<smoser> i dont know. i think the cluster would just want to run tgt also.
<smoser> you want that to be as close as possible
<smoser> its block level io
<smoser> if i understand you correctly
<smoser> bigjools, oh.
<smoser> i thought about one thign that might be screwing you.
<smoser> i dont think proxy settings or mirrors get sent down to commissioning
<bigjools> it's just that it's a bottleneck right now - we might need to start copying ephemerals to clusters
<smoser> right. thats what i was saying. i think you should plan on copying ephemerals to clusters.
 * bigjools tries commissioning again
<smoser> i have to run
<bigjools> thanks for the script
<smoser> i've tested it on /
<smoser> but not actually on an ephemeral image.
<bigjools> ok :)
<smoser> i've looked at it though (when pointed at an image)
<smoser> but on / it added a user that i could ssh in as
<smoser> so it seemed ok.
<smoser> later.
<bigjools> cheers
<bigjools> oooo
<bigjools> I saw an error flash by
<bigjools> something about tty error
 * bigjools tries to log in
<smoser> tty error.
<smoser> that is strange.
<bigjools> I think it mentioned stderr too
<bigjools> Oct  5 00:27:16 10-0-0-100 kernel: [   25.467047] init: cloud-config main process (899) terminated with status 1
<bigjools> Oct  5 00:27:16 10-0-0-100 kernel: [   25.885676] init: cloud-final main process (1049) terminated with status 1
<smoser> copy off /var/log/cloud-init.log
<smoser> and /var/log/cloud-init-output.log
<smoser> (you cna proably sudo apt-get install pastebinit)
<smoser> and do it that way
<bigjools> ProcessExecutionError: Unexpected error while running command.
<bigjools> Command: ['locale-gen', 'en_US.UTF-8']
<bigjools> Exit code: 1
<bigjools> there
<bigjools> I can scp it then paste, one sec
<bigjools> smoser: no cloud-init-output file, but here's cloud-init.log: http://pastebin.ubuntu.com/1261098/
<bigjools> look around line 246 onwards
<smoser> bigjools, /var/lib/cloud/instance/cloud-config.txt and /var/lib/cloud/instance/user-data.txt.i
<smoser> i really have no idea why 'local-gen' would fail like that. but its not deadly.
<bigjools> cloud-config.txt is empty
<bigjools> user data has plenty
<smoser> ah. yeah, it would be user-data.
<smoser> there is no cloud-config in that.
<smoser> but user-data has that script. and that *shoudl* get run.
<bigjools> http://paste.ubuntu.com/1261109/ is its contents
<smoser> bigjools, something is stopping you from reaching runlevel 3 i think.
<smoser> so the cloud-final job is not running
<smoser> bigjools, what does
<smoser> 'runlevel'
<smoser> and sudo status rc
<smoser> show?
<bigjools> N 2
<bigjools> and
<bigjools> rc stop/waiting
<bigjools> it must be that tty error
<bigjools> smoser: ^
<smoser> oh wait. that is wierd.
<smoser> cloud-final ran
<smoser> try running it again
<smoser> sudo start cloud-final
<bigjools> start: Job failed to start
<bigjools> I can't find any log for it
<smoser> ok. try it more manually. i guess.
<smoser> sudo cloud-init modules --mode=final --debug
<smoser> need --debug before modules
<bigjools> boom
<bigjools> well it has a lot of tracebacks but it did complete
<bigjools> machine powered off
<bigjools> smoser: so something is wrong with the upstart conf perhaps?
<bigjools> log: http://paste.ubuntu.com/1261129
<smoser> how much memory on the system?
<smoser> the FALLBACK stff is not really a big deal. its operating as mostly designed.
<bigjools> 2Gb
<smoser> the issue is that it logs to rsyslog, but then from within it, the userdata its running calls /sbin/poweroff
<bigjools> it's an HP cube
<smoser> and rsyslog gets killed
<smoser> so logging breaks.
<smoser> but id ont understand why it wouldnt have run
<smoser> i have no idea really.
<bigjools> why does it only happen when console=ttyS0 is specified?
<smoser> something failed.
<smoser> it doesn't make any sense to me.
<bigjools> jeez, 29C at 11am, gonna be hot today
<bigjools> I'll retry and see if I can see that error on the console, now I know when to look for it
<bigjools> ah it did it on enlistment too
<bigjools> stty: standard-input: input/output error
<bigjools> I am going to file a critical bug
<smoser> the stty error is maybe not related.
<bigjools> I am thinking that
<smoser> can you get dmesg
<smoser> and basically just tar up
<smoser> /var/log/
<smoser> it just doesn't make sense.
<smoser> bigjools, now that i think about it the tee /dev/tty2 is probably a bad idea.
<smoser> as maybe the tee is in this situation too.
<smoser> since we're /sbin/halting
<smoser> the tee might get killed and cause unnecessary angst.
<smoser> i like the log though.
<smoser> maybe we should have the script do
<smoser> sh -c 'sleep 10 && /bin/poweroff' &
<smoser> i can come up with something better too, but basically just amke sure that cloud-init is done.
<bigjools> smoser: why does "start cloud-final" fail?
<bigjools> isn't that a hint?
<bigjools> https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1061977
<ubot5> Ubuntu bug 1061977 in MAAS "Machine fails to commission when console=ttyS0 is present on kernel opts" [Critical,Triaged]
<smoser> can youg get dmesg
<smoser> and the /var/log attachd there.
<bigjools> yeah
<smoser> it is /dev/console
<smoser> it completely is that
<smoser> the fact that running it trhoug upstart died
<smoser> and then manually did not
<smoser> because its output will go to /dev/console when you do it via upstart
<bigjools> what from /var/log do you want?
<smoser> maybe the kernel has a buffer and just gets fed up at some point.
<smoser> jsut get it all
<bigjools> k
<smoser> if you dont mind.
<smoser> you didn't have any private data there :)
<bigjools> haha :)
<smoser> oh. yeah.
<smoser> while you're on that machine
<smoser> can you just try
<smoser> echo HI MOM | sudo tee /dev/console
<smoser> and see what it does
<smoser> i think we're gonna get an input output error basically.
<bigjools> tee: /dev/console: Input/output error
<smoser> yeah. so it makes sense.
<smoser> but i thought the kernel was supposed to be smarter.
<smoser> you do have 'console=tty1' on the cmdline also, right?
<bigjools> yes
<smoser> so there is no physical serial port, right?
<smoser> just to be sure.
<bigjools> there isn't - unless the BMC is providing one surruptitiosly
<smoser> hm..
<smoser> interesting.
<smoser> bigjools, you see kernel output on the monitor, right?
<bigjools> I do
<smoser> bigjools, the localegen died because it tried to write to its stdout
<smoser> bigjools, so i've got to go afk.
<smoser> but i'd like it if you reviewd my patch for https://code.launchpad.net/~smoser/maas/preserve-sources-list/+merge/127825
<bigjools> smoser: ok
<smoser> particularly, i'm not happy about the test, it seems long winded, but didn't know how to make better.
<smoser> i dont really know what to do for your ttyS0 issue.
<smoser> other than not specifying console= at all
<smoser> which imo seems busted.
<bigjools> smoser: we have to remove it
<smoser> as a default.
<bigjools> busted, but less busted than not booting at all
<smoser> well, to be fair, you are the first person to see this.
<bigjools> how much testing has it had though?
<bigjools> not on VMs
<smoser> more than a few systems.
<bigjools> I bet the others all have serial console
<smoser> well, yes.
<smoser> and so will the real target audience.
<smoser> so changing the default (which is hard coded in source code) to accomodate a little toy system
<smoser> doesn't make a lot of sense to me.
<bigjools> well hang on
<bigjools> you don't know that every system will have a serial console
<smoser> you're right.
<smoser> but i don't that it generally fails so badly if that is the case.
<bigjools> kernel bug?
<smoser> http://www.mjmwired.net/kernel/Documentation/networking/netconsole.txt
<smoser> i've always wanted to play with that.
<bigjools> smoser: I'll improve your test and land your branch
<smoser> thanks. i'll poke the kernel guys tomorrow am.
<smoser> but i dont have any ideas.
<smoser> http://www.mail-archive.com/linux-kernel@vger.kernel.org/msg246433.html
<roaksoax> howdy
<roaksoax> bigjools: howdy
<bigjools> roaksoax: hey
<roaksoax> bigjools: so how do you feel about the packaging?
<bigjools> roaksoax: positive
<roaksoax> bigjools: alright, i'm gonna try to upload to archive tomorrow so we can test upgrades from precise
<bigjools> roaksoax: \o/
<roaksoax> and get community to find more bugs if any
<roaksoax> that's would help us too
<bigjools> I did a fix the other day to prevent questions when ugprading, fingers crossed :)
<roaksoax> bigjools: which one is it?
<bigjools> roaksoax: config for cluster controller
<bigjools> it looks at DEFAULT_MAAS_URL if it can find it
<roaksoax> bigjools: the autodetection of maas-url, yeah I saw
<roaksoax> bigjools: it works
<bigjools> I did test it :)
<roaksoax> bigjools: yeah me too, it works :)
<bigjools> are you still thinking about changing the sed stuff?
<roaksoax> bigjools: i don't know TBH.. doing so will require .d support for the py's
<roaksoax> bigjools: and would require a way to provide .d for the yaml conf's too
<bigjools> I fear that is too much work at this late stage :(
<roaksoax> bigjools: Indeed!! the easier way right now is simply installing to /usr/share/maas and then symlink everything to /etc/maas/
<bigjools> yeah
<roaksoax> bigjools: however, the problme with that is that user settings won't be preserved on upgrade
<roaksoax> bigjools: i also found another bug in upgrading but with dbconfig-common, for some reason it was not preserving the password info and was re-writing the config file.
<roaksoax> the latter is expected, but should have presrved the password, but anyways, i work arounded it by simply sourcing that file and grabbing the password from there instead of letting dbconfig-common to rewrite the config
<roaksoax> bigjools: now to continue on the config, I think it would be the best for now... will have to check with Daviey to see what's his appreciation on this
<roaksoax> bigjools: maybe we can simply send the configs in /usr/share/maas and add some code in the configs itself that source whatever is in /etc/maas
<roaksoax> smoser: still around?
<bigjools> roaksoax: are you going to land your packaging branch?
<roaksoax> bigjools: yeah, i'm just testing one more thing
<bigjools> ok I put the bug back to in-progress :)
<bigjools> tarmac will set it when the branch lands
<roaksoax> cool
<bigjools> roaksoax: I shall force a daily build now
<roaksoax> bigjools: sure
<bigjools> being a build admin has its bonuses
<roaksoax> bigjools: indeed it is
<roaksoax> i envy you
<roaksoax> hahah
<roaksoax> i used to use PPA's extensively
 * roaksoax bed
<bigjools> we bumped the prio permanently on the daily build ppa :)
<bigjools> nn roaksoax
<roaksoax> night
<jam> allenap, rvba: So it turns out that MAASClient doesn't return an object like the Django Client return. It has a 'object.code' vs 'object.status_code' and 'object.read()' rather than 'object.content'
<jam> so... mocking/stubbing/ bad for the real world :)
<jam> allenap: you lied to me... :), list data is not supported by MAASClient. When it gets down into 'make_payload' you support bytes/unicode/IOBase and callable.
<bigjools> hey lifeless, any idea wtf is going on here? http://paste.ubuntu.com/1261414
<bigjools> I think we have a nasty bug with omshell here :/
<bigjools> good grief, I thought I'd seen bad code but this is hideous. http://ftp.fr.netbsd.org/cvsroot/src/usr.sbin/dhcp/common/Attic/parse.c,v
<lifeless> bigjools: looks like a url to me
<lifeless> bigjools: maybe it needs quotes ?
<bigjools> lifeless: I think the base64 parser in omshell blows
<bigjools> quotes don't help
<lifeless> bigjools: that is very odd
<bigjools> looking at the code it treats + and / specially
<lifeless> oh joy.
<bigjools> but the code is sooooo bad that it's taking me a while to work out what
<allenap> jam: Sorry, I didn't realise you were talking about MAASClient. The cli supports multiple values per key, and so does the server side; it's just MAASClient that needs a little love to get it there.
<jam> allenap: yeah, I got it to work with https://code.launchpad.net/~jameinel/maas/maasclient-multipart/+merge/128176
<jam> though it only fixes POST, GET uses urlencode() which just strs the list
<jam> allenap: is there a good way to generate a huge number of nodes for testing? Like I want to populate the DB with 1000, 10000, 100,000 nodes.
<allenap> jam: There's a way. I'm not sure about a good way.
<jam> allenap: well, pressing 'add node' in the web ui is pretty bad
<jam> I can do it in SQL surgery, but I need to generate mac addresses, etc. for each.
<jam> So it makes me think I should do it in python, but via the API or just direct on the db
<jam> but how hard is it to grab maasserver.model objects and play with them
<jam> I imagine 'settings' needs to be set correctly for me to mutate the db
<allenap> jam: How about: make syncdb && make harness; from maasserver.testing.factory import factory; factory.make_node()
<jam> allenap: if make harness will do it, that works for me
<jam> That was the sort of command I was looking for
<jam> w7z: mumble?
<jam> allenap: so what can I do to land the multipart list stuff? I don't actually have a 'mapping' I have an Iterable
<jam> but string is an Iterable
<jam> though apparently getattr(s, '__iter__') fails.
<mgz> there's no sane way to do it without isinstance
<allenap> jam: If it's not a string type, assume it's iterable of string types, and let things blow up if it's not?
<jam> allenap: well there are layering issues. The part that checks if it is a string type only returns 1 payload to attach, so you need to loop at a higher level.
<jam> or change the lower thing to always return a list of payloads
<jam> or..
<allenap> Blast.
<jam> (It also accepts files)
<jam> which are iterable, and you want to upload the whole file as one payload
<allenap> jam: A file isn't a collections.Sequence... but I guess stick with list and put it in the docstring.
<jam> allenap: I could do "if isinstance(x, collections.Sequence) and not isinstance(x, basestring)"
<allenap> jam: Yeah. Or change make_payload to gen_payloads, so allowing it to yield multiple things, then all the isSomething checks can be done there.
<rvba> jam: as I said on the MP, I don't understand ~jameinel/maas/ignore_results, we have a global celery settings for that.
<jam> rvba: where? (my system at least was improved by nuking the mnesia schema)
<jam> rvba: if you can point me to it, I'll happily reject/revert my patch.
<rvba> jam: CELERY_IGNORE_RESULT is set to True in etc/celeryconfig_common.py
<jam> as it could be me killing things repeatedly.
<jam> rvba: so right now, I have to install maas-dns in order to do 'make run' is that true?
<jam> if I don't the service fails to startup properly
<allenap> jam: Something like http://paste.ubuntu.com/1261655/
<jam> and the only log I see is the 'you should install maas-dns'
 * allenap looks forward to "yield from"
<rvba> jam: no, a local named instance is started when you run 'make run.
<rvba> '
<jam> rvba: well, I can't get 'make run' to talk to me, the last error in the terminal is the 'you should install maas-dns on the region controller'
<jam> I had this problem for quite a while this week.
<jam> until it magically fixed itself
<rvba> jam: that message is for the packaged version, I did not bother changing that message in the dev instance.
<jam> (my best guess at this point, is I got enough queues laying around that rabbit would say 'come back later' when asked to update dns, so it wasn't crashing the startup process.)
<jam> allenap: I switched to the 'yield' form, care to approve?
<allenap> jam: Sure.
<rvba> jam: I'm seeing the dns error too, I'll investigate.  But this is only a task failing, it should not cause trouble for the other services.
<jam> rvba: so I installed maas-dns, which installed maas, I then uninstalled it, but I now have a twistd process runinng.
<allenap> jam: +1 :)
<jam> everytime I kill it, it comes back with a new pid
<jam> I'm guessing upstart is keeping it alive?
<jam> ah, maas-txlongpoll
<jam> which didn't get uninstalle
<jam> d
<rvba> jam: as I said, you don't have to install maas-dns on a dev instanceâ¦ also, you'll get into trouble if you install the maas package and try to play with a dev instance at the same time :/
<allenap> Gah, so buildout says it's installing testresources 0.2.5, but then uses the system one, 0.2.4. How completely useless.
<jam> allenap: buildout config as it stands takes system packages over packages it installs.
<jam> I brought it up earlier
<jam> as being useless, too.
<jam> (vs using system packages if there isn't a custom package installed)
<jam> allenap: so you have to uninstall the system package to have buildout get the right one.
<jam> and then re-install it in the future when you want to use the system package for some other project.
<allenap> jam: Ah, right, thanks. At least there's a workaround.
<jam> rvba: uninstalling package 'maas' is triggering maas-txlongpoll to *start*
<rvba> jam: !
<jam> (which naturally prevents uninstalling maas)
<rvba> jam: I've spotted one problem: looks like the wrong celeryconfig is loaded on a dev instance (by the region worker at least).
<rvba> That's why its trying to write to /etc/bind/maas and failing.
<jam> rvba: I'm also getting failures in "_write_temp_file" in provisioningserver/utils.py
<jam> (when I create a node, it triggers writing the dns config again, and I get a failure trying to write the temp file)
<jam> rvba: so it looks like that is the N^2 behavior I was seeing. Every node added is regenerating a full DNS list.
<jam> (it is failing, but it still is pulling together the data which is O(N) and when you allocate N nodes == O(N^2) by the end)
<rvba> jam: the DNS writing task should only be triggered with the node gets an IP address.  Only then do we need to update the DNS config.
<rvba> s/with the node/when the node/
<jam> rvba: well I'm doing factory.make_node() in a loop
<jam> which probably gives addresses?
<rvba> jam: no, there is no lease creation in there as far as I can tell.
<jam> rvba: maybe it knows the dns config didn't get written properly yet?
<jam> it is definitely looping on failing to create it.
<jam> but it may be unrelated to me creating 1k nodes.
<jam> "Sending due task 'upload_dhcp_leases'" is in the log file an awful lot.
<rvba> That's a celerybeat task that is run every minute.
<jam> rvba: I'm seeing it roughly every few seconds
<jam> however, 'make_node' also creates a new nodegroup if you don't pass one in.
<rvba> jam: ah! right, that's what triggering all the DNS stuff.
<rvba> jam: because each nodegroup gets created with a configured interface IIRC.
<rvba> jam: I've just fixed the "wrong celeryconfig" problem.  A stupid mistake in the startup script.
<jam> rvba: \o/
<jam> so creating 1000 nodegroups takes about 15min, but creating 1000 nodes is <1min.
<jam> that seems reasonable.
<rvba> jam: The plan is to remove the DNS config pre-population.  But post 12.10 release.
<jam> rvba: sure, but you won't be creating 100,000 nodegroups in the immediate term, so 15min for 1000 seems fine.
<rvba> jam: right.
<allenap> rvba: Do you know if the [/+] needs to be on both sides for the [/+]no[/+] bug to manifest, or either side?
<rvba> allenap: from Julian's investigation (and the mailing list message we found), I think it's on both sides.
<allenap> rvba: That's one of the weirdest bugs I've ever heard of.
<rvba> allenap: yeah :/.
<allenap> rvba: Do you know anything about the code that submits commissioning results (op=signal)?
<rvba> allenap: not really, maybe I help you with something still?
<rvba> I can*
<allenap> rvba: I'm fixing the code that sets the power parameters. Now, it was broken: it was checking for power_type.upper() in map_enum(POWER_TYPES), i.e. checking that the value of power_type given over the wire matched the Python name of the enum item.
<rvba> allenap: what's wrong exactly?
<allenap> rvba: Well, that works for IPMI, because the Python name == uppercase(enum value).
<allenap> rvba: But not for the others.
<allenap> rvba: I mean, we should be expecting the enum *values* over the wire, right?
<rvba> allenap: right!
<allenap> rvba: Okay, I'm not mad then. The reason I want to ask someone about it is to avoid breaking scripts somewhere else. tl;dr this is what tdd looks like without t.
<rvba> allenap: haha :).  I think Julian mentioned IDD recently.
<rvba> allenap: hm,in his comment in the code, Julian said "if '/' or '+' appear either side of the word 'no'"
<rvba> allenap: rarg, I've tested it and it's either side indeed, not both sides.
<allenap> rvba: Gah. Top marks for you though, for confirming.
<jam> mgz: any progress on search?
<mgz> jam: getting there
<allenap> rvba: Do you have any time this afternoon to review the extra changes I had to make to https://code.launchpad.net/~allenap/maas/anon-power-setting/+merge/128127?
<rvba> allenap: sure, I'll do it right now.
<allenap> rvba: Thanks.
<roaksoax> rvba howdyÂ¡Â¡ i pushed the branch and selected u as reviewer
<roaksoax> did you see itÂ¿
<roaksoax> ?
<rvba> roaksoax: I did :)
<roaksoax> rvba cool then :)
<roaksoax> thanks
<rvba> roaksoax: I'll get to it in ~30 minutes.
<roaksoax> rvba awesome thanks
<roaksoax> allenap thanks for taking care of the power settings for enlistment
<jam> mgz: so I just did a quick 'wrap process_node_tags in lsprof' and the results are: out of 107s under profiling, 7.8s is parsing and processing the XML, 67s is downloading the hardware details, 32s is uploading the system_ids results, and 2.3s is getting the system_ids list.
<jam> I think the MAASClient handling of repeated tags is a bit slow (I think it encodes each value as a separate multipart message section.
<jam> uploading 12,000 values is a lot, but not 32s a lot, given that we can read that many tags in 2.3s.
<allenap> roaksoax: No worries. It hasn't landed yet... when I picked up that rock I found some bugs :)
<mgz> so, it is the passing data around overhead... still seems very high
<jam> And then getting the hardware details is *way* too expensive as well, given if it was a raw DB query, we could get the results in... checking.
<jam> I get the first result here in about 10s.
<jam> 32s to get everything.
<jam> mgz: 32s just to get the content out seems a bit too expensive as well.
<jam> time xpath_exists is still 6.8s, time in lxml on the current code is only 7.8s.
<jam> but "select hardware_details from maasserver_node" > /dev/null is 32s.
<jam> mgz: Is there a lot of quoting that Postgres is having to do?
<mgz> ...no, but maybe it's re-serialising? it shouldn't need to, but the 9.1 support does seem a little unpolished.
<jam> mgz: time to select from an xml column into a new text table is 4.1s.
<mgz> there might be a magic cast that will help?
<mgz> what does adding ::text do if anything?
<jam> mgz: time select content from alt_hardware_details >/dev/null => 32s.
<mgz> gr.
<mgz> might be sane to just ditch some of the db changes and pull from the original location :)
<jam> mgz: interestingly, using 'bytea' instead of text is *slower*
<jam> 1m24s
<mgz> heh, I think that's probably django
<jam> mgz: this is in psql
<jam> no django involved
<jam> psql -c "select ..."
<mgz> ...that's very suprising then
<jam> mgz: so something very strange when 'select DATA from...' is slower than 'select xpath_exists(DATA)' on the same content.
<jam> mgz: if I alter column set storage plain, it fails because it can't fit the 24kB documents in an 8kb page.
<mgz> jam: it would make sense were it storing some custom data structure optimised for querying
<mgz> that's not actually the case though as I understand it, but it's probably doing more work than it needs to for serialisation
<jam> mgz: maybe, but we've confirmed that lxml can take raw text and parse it with an XPath object in ~the same time.
<roaksoax> allenap: so the metadata server now imports the method from the maas API?
<jam> I think the big cost is postgres turning the data into an SQL result.
<allenap> roaksoax: Yeah, and the implementation has changed quite a lot.
<jam> mgz: select substring(content from 24000 for 100); is 3.7s
<jam> so it is all about the number of bytes coming out of psql
<jam> mgz: -A flag
<jam> mgz: time psql -A -c "select content" >/dev/null is 1.5s
<jam> mgz: --no-align :)
<mgz> :)
<jam> so psql is iterating all the data, caching it, figuring out how to align it, before outputting it.
<Daviey> roaksoax: can we remove maas-provision
<rvba> allenap: I'd like to check my sanityâ¦ can you run 'make lint' on trunk?
<jam> mgz: and potentially psql is operating in 'fixed max mem' mode, and so is spooling to disk to do that work.
<rvba> allenap: I'm getting: src/apiclient/multipart.py:74: undefined name 'make_payload'
<jam> but yeah, 1.6s if not aligning, and piping to 'wc -l -c' (if you leave in -w then wc slows things down to 7s looking for word chunks)
<allenap> rvba: Quite a bit of lint.
<allenap> rvba: And I see that too.
<allenap> rvba: I'm free to clean that up.
<rvba> allenap: cool.  Make you can clean the regexp in omshell while you're at it.  I know you like regular expressions :)
<allenap> rvba: As in, the before-or-after thing, or just pretty it up?
<jam> mgz: ts = time(); [node.hardware_details for node in Node.objects.all()]; td = time()
<jam> is 6.8s.
<rvba> allenap: the before-or-after thing :)
<jam> Which is a fair amount slower than 1.6, but nothing like the 60s we see
<rvba> allenap: I think it just needs a conditional expression based on a backrefâ¦ but you know that better than me :)
<allenap> rvba: Can we check for ([/+]no|no[/+)?
<jam> mgz: get_hardware_details is spending 67s total, 18s is reading the response and json.loads it. However, 48.7s is 'post'
<jam> mgz: so I think... MAASClient.post needs some serious poking.
<roaksoax> Daviey: from the archives?
<roaksoax> Daviey: i'd say we can
<roaksoax> and we should
<rbasak> smoser: are you planning to SRU bug 978127?
<ubot5> Launchpad bug 978127 in cloud-init (Ubuntu) "incorrect time on node causes failed oauth" [High,Fix released] https://launchpad.net/bugs/978127
<jam> (it is possible that some of the slowdown is the server processing our request, but 24s in dispatch to upload, 58s in 'encode_multipart'
<jam> allenap: ^^ I'm guessing encoding wasn't tuned for handling 100,000 records, right?
<smoser> rbasak, i suppose you need it after ephemeral also.
<smoser> why can't you just get an architecture that doesn't suck, rbasak ?
<rbasak> smoser: that's the reason I ask, yes. Right now we're still defaulting to a precise ephemeral, and I keep needing to fix the RTC
<smoser> precise ephemeral should be fixed
<rbasak> smoser: can you add a bug task for precise, please? I don't ahve permission
<smoser> if you're still seeing an issue, then the problem is not solved correctly.
<rbasak> smoser: seeing an issue out of today's maas daily
<smoser> rbasak, can you show me?
<smoser> i basically tested this.
<smoser> by having cloud-init's upstart job first break the clock
<rbasak> smoser: half an hour please? I just worked around it for a juju test and d-i just started
<smoser> sure.
<allenap> jam: Ha, wow.
<Daviey> roaksoax: right, i will remove it from the archive... but i want to make sure you have an upgrade path
<Daviey> roaksoax: can you raise a bug requesting removal as it is deprecated etc.. and i'll process it
<mgz> hm, I should make fail an alias for tail -f, that was a funny tyop
<roaksoax> Daviey: right, so maas Conflicts/Replaces on maas-provision, which obviously causes it to be removed. However, it not being in the archives would make the transition smoother, wouldn't it? because the packages would simply be completely removed
<rvba> allenap: I think so yeah.  All I know about this problem is summarized right here ;) : http://paste.ubuntu.com/1262047/
<jam> allenap: apparently 30s of the 80s is in 'set_type'
<allenap> jam: That's... weird. Maybe it's recoding things at that point.
<jam> allenap: well, it is under lsprof, so some things are more expensive than they are in 'real life'. I'm doing a quick test of casting the strings to bytes rather than unicode
<jam> see if that has a big difference
<jam> since bytes payloads don't have set_type called
<rvba> allenap: I've got a tiny branch up for review if you have time: https://code.launchpad.net/~rvb/maas/big-networks/+merge/128269
<jtv> Trunk seems to be failing tests and have lint in it.  :(
<allenap> jtv: I'm fixing those now.
<jtv> Ah
<rvba> roaksoax: did you put your branch up for review?  I don't see it in the "active code reviews" list.
<jam> allenap: interestingly, rabbit can take ~1 minute from the time I put something in the queue, before it actually triggers on the provisioning worker
<roaksoax> rvba: I change the status to work in progress
<rvba> roaksoax: ok, got it.
<allenap> jam: That's not good.
<jam> allenap: (the background is, we can run xpath_exists() in the database taking ~7s to run, or we can dump the raw xml out in about 1.5s, but it takes 42s to read the data via APIS and upload the results back)
<jam> so a 6x overhead
<jam> and we only spend 7.8s in etree.XPATH()
<jam> allenap: as for what causes the rabbit slowdown, I'm not sure. It looks a little like it is recoverying from trying to write the dns config or something.
<jam> (waiting for celery-beat?)
<smoser> rbasak, is there anything you're aware of that you'd like in SRU not on https://bugs.launchpad.net/ubuntu/precise/+source/cloud-init
<mgz> okay, sorted apart from what to do with InvalidConstraint... does the view somehow need to catch it and do something clever... atm you just get a "we broke" error page which is not acceptable for a typo
<flacoste> roaksoax: do you have everything you need for IPMI in enlistment?
<roaksoax> flacoste: yes, just waiting for it to land
<flacoste> roaksoax: right, ok
<roaksoax> thanks :)
<Daviey> roaksoax: state of bug 1052056 ?
<ubot5> Launchpad bug 1052056 in freeipmi (Ubuntu) "[FFe] [MIR] freeipmi" [Undecided,In progress] https://launchpad.net/bugs/1052056
<roaksoax> Daviey: need to address the unused variable compiler warnings, but haven't really have the time to investigate how to do it since I have been pretty much swamped with the other stuff
<Daviey> ok, thanks
<Daviey> roaksoax: do you have your latest debdiff?
<roaksoax> Daviey: yes, hold on
<Daviey> thanks
<roaksoax> Daviey: http://paste.ubuntu.com/1262079/
<Daviey> thanks
<jam> allenap: well, lsprof says that set_type() spends its time calling get_params set_param, __delitem__
<jam> which appears to have to encode/decode the params
<jam> however, it is an lsprof-ism, real-world time is 42s => 41.5s using 'bytes' instead of 'unicode', but lsprof time is 109 vs 85s
<jam> so... ignore that one.
<rvba> roaksoax: http://paste.ubuntu.com/1262102/.  One question: why use the absolute path '/usr/sbin/ipmi-chassis-config' instead of 'ipmi-chassis-config'?  I thought Scott said it was a bad thing to doâ¦
<roaksoax> rvba: following what was done before me
<rvba> Fair enough :)
<roaksoax> rvba: however, i think usr/sbin is not in the path, and hence there was a problem executing the scripts, I don't know if yourecall having us discuss something like that
<rvba> roaksoax: yeah, it rings a bell.
<rbasak> smoser: I'm not so bothered about bug 1028501 any more, since MAAS doesn't need it to work any more
<ubot5> Launchpad bug 1028501 in cloud-init (Ubuntu Precise) "cloud-init selects wrong mirrors for arm" [High,Triaged] https://launchpad.net/bugs/1028501
<rbasak> smoser: I think that means that bug 978127 is the only one I care about in cloud-init for precise. Though I'm worried that I'm missing something else.
<ubot5> Launchpad bug 978127 in cloud-init (Ubuntu Precise) "incorrect time on node causes failed oauth" [High,Triaged] https://launchpad.net/bugs/978127
<roaksoax> rvba: btw.. celeryconfig.py and celeryconfig_cluster.py are not meant to be modified by the user right?
<rvba> roaksoax: no. Only maas_local_celeryconfig_cluster.py and maas_local_celeryconfig.py should be modified by the user.
<roaksoax> rvba: not even celeryconfig_common right?
<rvba> roaksoax: no.
<smoser> rbasak, did your install go? can i see failed oauth now?
<rbasak> smoser: having trouble getting to the point where I can reproduce. I set the clock back to 1970, and now it can't add my PPA for maas-enlist to work around the SRU not being in yet
<rbasak> Changing the hardware clock is a little bit tedious
<rbasak> I have to install some usable OS first, since there's no "recovery disk"
<smoser> boot the ephemeral image.
<smoser> silly.
<rbasak> ...which I can't log in to
<smoser> http://bazaar.launchpad.net/~smoser/+junk/backdoor-image/view/head:/backdoor-image
<rbasak> and yes, there are ways round it
<smoser> use that to add a user 'backdoor'.
<rbasak> The easiest way round it is to install a usable OS
<smoser> i think that script is easier.
<smoser> personally
<smoser> ./backdoor-image /var/lib/ephemeral/......./disk.img
<smoser> then ssh in
<smoser> or login on console.
<rbasak> What about disabling poweroff?
<rbasak> Another step to do
<smoser> this is true.
<smoser> maas needs a "rescue" environment.
<rbasak> I'm going to make an armhf recovery initrd when I get round to it
<smoser> cirros is probably really close
<smoser> to what you ened there.
<smoser> and if you ever do "get round to it" i'd rather you fix cirros to work for you.
<rbasak> I was going to base it on ubuntu core
<rbasak> Which I think might Just Work out of an initrd
<rbasak> Need to test it though
<smoser> well, fo ryou particular use case here, you can probaly just boot the existing initramfs and kernel
<smoser> and pass 'break' on the cmdline
<smoser> if you have console access.
<smoser> if not then you need ssh or the like.
<rbasak> roaksoax: are you planning on landing a fix for the ipmi_si module parameters today? If not I need to file a bug.
<roaksoax> rbasak: that's actually what i'm doing now
<rbasak> roaksoax: OK no problem. I'll leave it then.
<rbasak> smoser: OK, reproduced finally
<rbasak> smoser: now to get you in
<rbasak> smoser: which I presume will involve running your backdoor script ;-)
<roaksoax> rvba: updated the branch, ready for final review. Thanks a lot for working on it
<rvba> roaksoax: np.  Branch approved.
<roaksoax> allenap: are you gonna make changes to the anonymous power settings for enlistment or will you land it?
<roaksoax> rvba: we don't want users to modify the maas-http.conf either right?
<rvba> roaksoax: well, if they want to serve the site over HTTPS they will have to change maas-http.conf.
<roaksoax> rvba: ok, so i'll leave it as is
<roaksoax> rvba: btw.. https://jenkins.qa.ubuntu.com/job/maas-merger-quantal/127/console
<rvba> Yeah, I've seen.  It seems to be related to the changes jam checked in.
<roaksoax> boomer
<rvba> But it's really just a problem in the tests I think.
<roaksoax> yeah but that will be holding off all MP's
<rvba> Likeâ¦ the tests were simply not updated.
<rvba> Good point.
<rvba> hm, I wonder how I could even land a fix then :)
<rvba> I guess I just merge manually.
<roaksoax> rvba: a fix won't make the jenkings job fail, so it would land the branch
<rvba> ah right.
<roaksoax> rbasak:
<roaksoax> https://code.launchpad.net/~andreserl/maas/maas_commissioning_modprobe_params/+merge/128294
<rbasak> roaksoax: approved, thanks!
<roaksoax> allenap: re bug 1059168
<ubot5> Launchpad bug 1059168 in MAAS "MAAS should tell IPMI to PXE once" [High,Triaged] https://launchpad.net/bugs/1059168
<roaksoax> allenap: basically, each team we tell a machine to turn on, it will tell it to PXE boot
<roaksoax> allenap: we don't have to manually configure the BIOS and tell it to PXE boot
<roaksoax> allenap: it will do it on demand
<roaksoax> allenap: sabdfl's request
<roaksoax> s/team/time
<rvba> roaksoax: I'm fixing the build now.  The fix should land shortly.
<roaksoax> rvba: awesome thanks
<Daviey> roaksoax: maas is using squashfs by default now?
<roaksoax> Daviey: for quantal yes
<Daviey> roaksoax: and juju deploy precise uses old method, right?
<roaksoax> Daviey: yes
<roaksoax> Daviey: there are no squashfs images for precise, are they? if there are we could enable it oo
<Daviey> there are not
<Daviey> just wanted to check it worked
<roaksoax> alright :)
<roaksoax> smoser: around?
<smoser> here
<roaksoax> smoser: so, matsubara just pinged me about the commissioning stuff for IPMI since he mentioned that the cards lost its static address
<roaksoax> smoser: since we are working with the assumption
<roaksoax> smoser: that IPMI will alwas DHCP by design
<roaksoax> smoser: so i think we need to address the fact that some people don't wan't to DHCP their IPMI cards, and will pre-configure them
<roaksoax> smoser: what do you think?
<roaksoax> matsubara: ^^
<smoser> roaksoax, i thought you were wokring under that assumption.
<smoser> i thought if the card had an IP you reported it.
<smoser> right?
<roaksoax> smoser: nope not really. The assumption I was told to work bsaed on was if the IUPMI card is set Static addres network source, we should change it to DHCP
<roaksoax> smoser: because it could be pre-configured
<smoser> ah.
<smoser> well i like your new suggestion better.
<smoser> i'd say if it is set to static, and appears to be configured that you should leave it as such.
<smoser> ideally such things are exposed in maas configuration
<smoser> but... its a bit late for that i think
<roaksoax> smoser: right, so I was thinking on simply passing a parameter such as IPMI_DHCP="yes" in commissioning_user_data
<roaksoax> smoser: and if so, send --use-static regardless of what it is
<roaksoax> to the command
<roaksoax> rvba: another build error :( https://jenkins.qa.ubuntu.com/job/maas-merger-quantal/129/console
<rvba> roaksoax: I've seen.  Looks spurious to me.
<matsubara> roaksoax, that'd be helpful. I'm setting up dhcp in the lab ipmi network, but it'd be easier to just use the static ip already configured
<jam> rvba: so I'm pretty sure I did land the change for '?op=' but I'm also surprised that it would have landed if tests would then be failing. I can land a fix if you haven't already
<rvba> jam: doing it right now: https://code.launchpad.net/~rvb/maas/fix-broken-build/+merge/128297
<roaksoax> smoser: something like this: http://paste.ubuntu.com/1262416/
<roaksoax> smoser: or better yet, enable it by default, so if the card is set to static, just use that IP address
<smoser> it looks reasonable.
<roaksoax> alright, i'll get that done then
<smoser> maybe IPMI_CHANGE_STATIC_TO_DHCP=false
<smoser> you did put the ipmi header there, but the name 'use_static' could mean so many things.
<roaksoax> smoser: right :). Will do change that for something more appropriate
<rvba> roaksoax: the fix has finally landed!
<roaksoax> rvba: awesome!
<smoser> allenap, is this known:
<smoser> http://paste.ubuntu.com/1262473/
<smoser> http://paste.ubuntu.com/1262477/
<smoser> it was known. fixed in 1185
<roaksoax> matsubara: could you please configure one of your IPMI cards to static, and apply this patch to the commissioning_user_data conffile http://paste.ubuntu.com/1262508/
<roaksoax> matsubara: and re-enlist, and re-commission and see if it works (leaves it as DHCP and obtains the right address)
<matsubara> roaksoax, sure, using the latest package: maas-0.1+bzr1170+dfsg-0+1192+117~ppa0~quantal1, I take?
<smoser> roaksoax, anyone know anythign about "failed to upload?"
<smoser> well, never mind. looks like we have one.
<roaksoax> smoser: huh? failed to upload where?
<roaksoax> matsubara: yeah
<matsubara> ok, waiting it to be published
<roaksoax> matsubara: ok, it doens't really matter what version of maas as long as it is greater than bzr1170
<roaksoax> matsubara: so you can just patch the file
<matsubara> ah ok
<roaksoax> smoser: for enlistment, do we have a metadata file we can pass same as commissioning?
<smoser> user data for enlistment?
<smoser> contrib/preseeds_v2/enlist_userdata
<roaksoax> smoser: ah right lol, but we are pretty much going to use the same script for enlistment, commissioning
<roaksoax> smoser: so it should probably live in a common place
<roaksoax> smoser: any thoughts? maybe in a package?
<roaksoax> cause i was just thinking on shipping it with maas-enlist
<smoser> roaksoax, thats not a bad idea.
<roaksoax> smoser: i'll make another ipmi package
<roaksoax> err
<roaksoax> another binary package for maas-enlist
<smoser> ?
<roaksoax> smoser: maas-commissioning
<smoser> why separate?
<smoser> due to dependencies?
<roaksoax> smoser: becuase maas-enlist pulls curl, archdetect-deb, libavahi-core7, libavahi-common3
<roaksoax> yeah
<smoser> good enough.
<matsubara> roaksoax, enlisting now with your patch
<roaksoax> matsubara: alright, enlistment wont be affected, only commissioning, but we need to enlist first so that commissioning changes take effect
<matsubara> roaksoax, of the 4 nodes I booted, 3 are static and 1 is dhcp
<roaksoax> smoser: oh btw... if the node is enlisted, and I make changes to commissionining-user-data they don't take effectd
<roaksoax> matsubara: ok cool
<roaksoax> matsubara: that should test it well
<roaksoax> smoser: i have to re-enlist
<smoser> roaksoax, relaly?
<smoser> that is really bad.
<roaksoax> smoser: yep
<smoser> please open a bug for that.
<matsubara> roaksoax, ok. the nodes are declared
<roaksoax> matsubara: ok, now commission them please
<matsubara> roaksoax, now I enter the ipmi configuration normally and commission?
<roaksoax> matsubara: no
<roaksoax> matsubara: don't enter IPMI, let it commission
<roaksoax> matsubara: and see if the IPMI gets set
<roaksoax> and not changed to DHCP
<roaksoax> matsubara: so power them on manually once you've accepted&commission
<matsubara> ok. they're on
<roaksoax> matsubara: did you ifle a bug for the juju issue with the CPU count?
<roaksoax> matsubara: do you know if it's been fixed?
<matsubara> yes, i filed
<matsubara> didn't test yet but i think so
<roaksoax> matsubara: what's the bug number
<matsubara> bug 1061286
<ubot5> Launchpad bug 1061286 in juju "juju bootstrap returned ERROR Invalid 'cpu_count' constraint '1.0'" [Medium,Fix committed] https://launchpad.net/bugs/1061286
<matsubara> roaksoax, ok, nodes are ready
<matsubara> and no IPMI config was set
<roaksoax> matsubara: jum
<roaksoax> matsubara: check in the commissioning-user-data that modprobe ipmi_si has arguments
<matsubara>    modprobe ipmi_si type=kcs ports=0xca2
<roaksoax> smoser: my bad
<roaksoax> smoser: it does work
<roaksoax> smoser: as expected
<roaksoax> i wonder why i thouhght it didnt update
<smoser> hey all
<smoser> http://bazaar.launchpad.net/~smoser/maas/maas-pkg-test/view/head:/maas-ephemeral-test-quantal.txt
<smoser> was mostly functional walk through of ppa test for me
<smoser> i now do not have to touch the UI by hand.
<smoser> whoowhoo
<roaksoax> smoser: what do you think? http://paste.ubuntu.com/1262708/
<smoser> that looks reasonable to me, roaksoax
<smoser> i'm out.
<roaksoax> smoser: hold on
<roaksoax> :/)
<smoser> k
<roaksoax> smoser: give me one min
<smoser> holding
<roaksoax> for you to approave
<roaksoax> smoser: https://code.launchpad.net/~andreserl/maas/commissioning_improvements/+merge/128318 please :)
<smoser> did you test it?
<roaksoax> smoser: yes
<smoser> you asked for review of "launchpad code reviewers" ?
<smoser> what is doing that.
<roaksoax> smoser: it does that automatically
<roaksoax> smoser: julian (i think) changed it that way
<smoser> well that is busted.
<smoser> anyway.
<smoser> i'm out.
<smoser> and i revierwed.
<roaksoax> smoser: awesome, thank you. have a good weekedn
<roaksoax> jtv: if you are making changes to tx-tftp, please make htem in the ubuntu package in archive sa well
#maas 2012-10-06
<jtv> roaksoax: what do I need to do in order to get my tx-tftp fix into the ubuntu package?
<roaksoax> jtv: if you want you can file a bug against the package in ubuntu and attach the patch, and I'll take care of it
<jtv> Thanks.
<roaksoax> jtv: you working todya?
<jtv> Nope.
<roaksoax> :)
<roaksoax> good.. it's been a long weekend for all
<jtv> Not for me.  Major internet outage last night.  :(
<jtv> I added the Ubuntu package to the bug.  And now, I must run!
<roaksoax> jtv: awesome thansk
<allenap> jelmer: Are you around today?
<jelmer> allenap: ish
<jelmer> allenap: how are you?
<allenap> jelmer: Holding up :) You? I was wondering if one of the blues is around to take a second look at https://code.launchpad.net/~andreserl/maas/maas_fix_constraints_arch_mapping/+merge/128338.
<jelmer> allenap: holding op too :)
 * jelmer looks
<jelmer> allenap: Sorry, I'm not sure either
<jelmer> allenap: the API does seem like the wrong place to be doing this sort of mapping
<allenap> jelmer: Okay, I'll leave it as is, and see if John or Martin want to comment. Thanks :)
<jelmer> allenap: but I don't think I am familiar enough with this part of the code to give a useful comment
<jelmer> allenap: okay; sorry I couldn't be of more help
<allenap> jelmer: No worries, thanks for looking.
<allenap> Afternoon
<roaksoax> allenap: still around?
<allenap> roaksoax: Yeah, hi.
<roaksoax> allenap: thanks for the review, addressing your comments :)
<allenap> Cool, ta.
<roaksoax> allenap: comments make sense to you? http://paste.ubuntu.com/1264110/
<allenap> roaksoax: Yeah, they're good.
<roaksoax> allenap: and I also wnat to wait for the blue squad comments on it since i think they didn't forsee the change of the arch including subarch in maas
<roaksoax> and juju passing only arch
<roaksoax> allenap: https://code.launchpad.net/~andreserl/maas/fix_ipmi_command_execution/+merge/128358
<allenap> roaksoax: Fwiw, I normally only resubmit a merge proposal when I've made a mistake with, say, the prerequisite branch, or the proposal is unrecognisable from before. Ordinarily I just push the changes and ping or comment to alert the reviewer to take another look.
<allenap> Although what you did makes sense, it doesn't seem to be the norm.
<roaksoax> ok :)
<roaksoax> thanks for the advice
<allenap> roaksoax: Have a good evening, I'm off now.
<roaksoax> allenap: you do as well
#maas 2013-09-30
<steveubu_> question about maas
<roaksoax> rvba: o/
<roaksoax> where you able to solve yhr image problem? are we ready to upload?
<rvba> roaksoax: we still have bug 1232174. We're in the process of fixing it.
<ubot5> bug 1232174 in MAAS "maas-import-pxe-files fails because Saucy's images information is not available." [Critical,Triaged] https://launchpad.net/bugs/1232174
<roaksoax> rvba: but you dont need to "fix" that just disable saucy from the config filr and that should be it shouldnt it?
<roaksoax> unless we dont support config files
<rvba> roaksoax: we could do that, but I assumed you added Saucy in there for a reason :)
<roaksoax> aince the saucy info comes from.oublished simple streams
<rvba> True, but the simplestreams worked is not yet finished.
<rvba> work*
<roaksoax> rvba: yeah acott needed to deploy so i woild check wih him to have the simplestreams data and images available
<rvba> Okay.
<roaksoax> rvba: he did say thr images were on its way and i think the devel sidenof things are already available
<roaksoax> just not "releases"
<roaksoax> smoser: ^^
<roaksoax> anyway im flying home now ttyl
<jamespage> allenap, roaksoax (or anyone else): how long does a dhcp lease last in maas dhcp server?
<allenap> jamespage: Once the node is allocated, indefinitely.
<allenap> jamespage: A static lease is pushed into dhcpd's config.
<jamespage> allenap, and for other servers that might dhcp from the maas server? i.e. ones that are not registered?
<allenap> jamespage: 30 minutes.
<jamespage> allenap, OK _ that should be good then
<jamespage> allenap, testing the new lxc container stuff with maas that's in juju-core now
<allenap> Cool.
<allenap> jamespage: I may have the units wrong: it's 30 *seconds*, which is surprisingly short.
<jamespage> allenap, hmm
<jamespage> allenap, I saw maas double-allocate an IP to a physical server and an LXC instance
<allenap> jamespage: I suspect it's meant to be 30 minutes.
<jamespage> which surprised me - I wondered whether maas ran out of IP's
<allenap> jamespage: MAAS drive ISC dhcpd, which I'm going to bet tries to not do that sort of thing... unless someone has set a very low lease time.
<allenap> jamespage: However:
<allenap>  1541 Adam Stokes       2013-08-02 [merge]
<allenap>       [r=julian-edwards][bug=][author=adam-stokes] Limit DHCP lease times to 30 seconds for PXE clients.
<jamespage> allenap, blimey - I might not actually have that - this is a raring install
<allenap> jamespage: I can't find what the default is.
<tych0> smoser: (or anyone that knows about kvm/maas): what's the magic i need to do to get maas to understand how to stop a kvm?
<tych0> kvm is on localhost, if that makes a difference
<smoser> tych0, you have to tell it to use virsh as the power control
<smoser> and give it the correct path to virsh.
<tych0> is there an api call for that somewhere?
<smoser> you could probably use
<smoser> register-node in http://bazaar.launchpad.net/~virtual-maasers/+junk/maas-libvirt-utils/files
<smoser> err.. wait.
<smoser> libvirt-domain-to-maas
<smoser> that scrapes libvirt and shoves it into maas.
<smoser> but you can read it. in the end it just callas
<tych0> yep, maas-cli update
<tych0> cool
<smoser> yeah. that one calls 'new'
<tych0> ?
<tych0> oh, i see
<tych0> so this is for a node that's already installed?
<smoser> well, that is fo ra node that exists
<smoser> it reads the libvirt and then adds that node to maas.
<tych0> ok, i'm pxe booting the node and letting maas enlist/provision it
<tych0> i just want maas to be able to start/stop it
<tych0> so maas knows about the node, just not how to control it
<smoser> right. you can use update.
<smoser> or you could just delete the node
<smoser> and shove it in with that script
<tych0> ok, cool
<tych0> i see, $data
<tych0> thanks!
#maas 2013-10-01
<nesusvet> hello everyone. I have a little problem with the proxy. My local server downloads packages trough a remote proxy server, I set up the maas to work with this proxy  and it looks like everything works well. But it seem JuJu tries to get information about servers through the proxy also. Could please help me to disable the requests to the proxy server please.
<nesusvet> hello everyone. I have a little problem with the proxy. My local server downloads packages trough a remote proxy server, I set up the maas to work with this proxy  and it looks like everything works well. But it seem JuJu tries to get information about servers through the proxy also. Could please help me to disable the requests to the proxy server please.
<allenap> nesusvet: Is that the Juju client, or Juju running on the bootstrap node, or another machine in your environment?
<nesusvet> Hi allenap, This is "Juju running on the bootstrap"
<allenap> nesusvet: Do you have a log of the "JuJu tries to get information about servers through the proxy also" bit?
<nesusvet> I got the 401 error with mention about my proxy host
<allenap> nesusvet: I really need a log to understand exactly where that error is being encountered.
<nesusvet> Ok, http://pastebin.com/mgVawVG0 - without proxy
<nesusvet> http://pastebin.com/NrS3u3wA - with proxy
<rvba> nesusvet: why are you using '--upload-tools" (which you should do) only when you define the proxy?  I.e. could you try: juju -v bootstrap  --upload-tools -e maas
<nesusvet> Doing.
<allenap> nesusvet: Right, to summarise: without proxy Juju can talk to MAAS but not to the outside world; with proxy Juju can talk to the outside world, but not MAAS.
<nesusvet> http://pastebin.com/kXykMF53 - here is the result
<allenap> nesusvet: Try that again without http_proxy.
<allenap> As in, unset/blank http_proxy.
<nesusvet> http://pastebin.com/gy24D7RD
<allenap> nesusvet: juju destroy-environment -e maas, then try again.
<nesusvet> http://pastebin.com/nsGfiVsz
<racedo> ping rvba
<rvba> racedo: hi
<racedo> hey rvba
<allenap> nesusvet: I'm not sure if we should worry about the warning. Otherwise that looks like success.
<nesusvet> allenap, http://pastebin.com/fMKNrhtt on this step I alway get the "time out"
<racedo> rvba: we have split the maas-region-controller and the maas-cluster-controller into two servers
<nesusvet> I mean server do nothing
<racedo> rvba: now we want to install multiple cluster controllers, like 7 or 8
<racedo> rvba: so the region controller can communicate to all of them through the same network, right?
<allenap> nesusvet: Give it 5-10 minutes and try again; juju bootstrap returns before the bootstrap node is actually up and running (yeah, weird, I know).
<rvba> racedo: yes
<racedo> rvba: because every cluster controller will be managing different networks
<racedo> rvba: we will have 7 different networks (1 per cluster controller)
<rvba> racedo: right, they all need to have there own network, which they control, plus a way to contact the region
<racedo> rvba: exactly perfect!
<racedo> rvba: thanks :)
<racedo> rvba: so the region controller doesn't provide anything other than API, Web UI and database, right?
<rvba> racedo: that plus DNS
<racedo> rvba: oh ok
<racedo> rvba: but that doesn't affect the communication as DNS doesn't need to be in the same network
<rvba> racedo: right
<nesusvet> allenap, waiting...
<racedo> rvba: thanks again
<rvba> np
<nesusvet> allenap, http://pastebin.com/yc1H9KzY
<nesusvet> the result
<nesusvet> ohh, I forgot to add "-e maas"
<nesusvet> to the same step with a new option
<allenap> nesusvet: Can you ssh into the host - ubuntu@GPT-bart-FE - then look at /var/log/cloud-init-output.log.
<allenap> ?
<nesusvet> yes
<nesusvet> can't reach the host (permissions denied)
<allenap> nesusvet: Can you check in /var/log/maas/rsyslog on the region controller to see if there's anything. Also look at /var/log/maas/pserv.log and the Apache logs for anything suspicious.
<nesusvet> I have restarted the node
<allenap> nesusvet: Have you set the http_proxy in MAAS too?
<nesusvet> yes
<nesusvet> 2013-10-01 03:34:16 DEBUG juju.state open.go:88 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused
<nesusvet> here is the message on the node
<allenap> nesusvet: Okay, I'm wondering if that's the problem. Your proxy can't see internal hosts, but MAAS doesn't know which hosts it should use the proxy for, and which it shouldn't.
<nesusvet> yes, it's a true
<nesusvet> so for this reason I need to restrict the actions between node and proxy
<nesusvet> I mean local network and proxy
<nesusvet> for this reason I want to disable proxy
<nesusvet> on the clients and on the bootstrap machine
<allenap> nesusvet: I agree that's the best thing to try next. MAAS does install a proxy on the region controller. You could configure by hand that to use your existing proxy as an upstream proxy, and remove all proxy settings from MAAS.
<stokachu> what version of maas was http://maas.ubuntu.com/docs/cluster-configuration.html written for? ive got maas 1.2 on ubuntu precise and the gui doesn't have any of these options
<smoser> rvba, around ?
<smoser> where do we stand on maas and saucy right now
<rvba> smoser: Hi.  Well, same as yesterday, the old script errors when trying to download the Saucy images.  The new script is still not fully integrated but jtv is working on it.
<smoser> "old script"
<smoser> there are 2 scripts that import things
<stokachu> actually what version of maas was http://maas.ubuntu.com/docs/ written for
<smoser> import-ephemeral and import-pxe-files
<rvba> In this case I meant the import-ephemeral script
<rvba> Besides, the new script is only a replacement for maas-import-ephemerals, I wonder if tych0 is planning to port maas-import-pxe-images.
<rvba> tych0: ?
<smoser> well, thats an interesting question..
<tych0> not atm, busy with cloud installer stuff
<smoser> because we need server side data if we want that.
<smoser> i *do* want that
<smoser> but ....
<smoser> that probably woudl have to happen "right now".
<smoser> rvba, i really, really want to have something uploaded to ubuntu end of day here that at least doesn't fail to start.
<smoser> so is import-ephemeral the only blocker there ?
<smoser> wait.
<rvba> smoser: the only solution is to remove saucy from the list of series to consider.
<smoser> what is that a solution to ?
<smoser> both import-ephemeral and import-pxe-files ?
<smoser> we can have an ephemeral image later today.
<smoser> can/will
<tych0> FWIW, i just landed https://code.launchpad.net/~tycho-s/maas/port-ephemerals-configs/+merge/187931
<smoser> but if import-pxe-files is busted, we need to fix that.
<rvba> A solution to have a package which does not blow up.
<tych0> which i thought was the only holdup to switching to .py
<rvba> tych0: well, that plus hooking up the new script in MAAS;  plus actual testing :)
<tych0> rvba: right, i meant the only hold up to hooking up the new script
<tych0> and testing, obviously :-)
<smoser> oh we dont actually test anythin ghere.
<smoser> we just type make test
 * smoser was rude. sorry.
<smoser> rvba, is import-pxe-files broken  still ?
<smoser> in the current version?
<smoser> what is that bug ?
<rvba> Bug 1232174
<ubot5> bug 1232174 in MAAS "maas-import-pxe-files fails because Saucy's images information is not available." [Critical,Fix committed] https://launchpad.net/bugs/1232174
<rvba> No idea why the lander marked it as fixed.
<stokachu> the man page for maas-cli doesn't mention anything about " maas-cli maas node-groups import-boot-images" as stated in the online docs so im not sure what version of maas these docs were written for
<rvba> stokachu: these docs are for the version about to be released in saucy.
<smoser> stokachu, i'm really sorry. you're runing 12.04 i suspect.
<stokachu> yea im on precise with maas 1.2
<smoser> stokachu, we're working on getting you a sane solution for 12.04.
<stokachu> ah ok
<smoser> the plan is "cloud-tools" pocket, which is part of the cloud-archive.
<smoser> that pocket will have juju and maas from 13.10 (and dependencies) running on 12.04
<stokachu> ah ok, are those bits available now? or not until saucy goes gold?
<smoser> stokachu, well, they're available... but in a non-yet-working state :)
<smoser> i was hoping by end of today that we'd have a at-least-working state.
<stokachu> smoser: gotcha, mind if i pm you real quick?
<smoser> thats fine.
<roaksoax> o/
<roaksoax> rvba: so how are we looking? everything ready to be released?
<rvba> roaksoax: Hi.
<roaksoax> rvba: o/
<roaksoax> :)
<rvba> roaksoax: I'm relying on smoser's judgment here.  We're not in a position to use tych0's replacement script yet.  And the old import scripts error because of 1232174.  The consequence is not dramatic because the script is written in such a way that all the other images are downloaded first.  But I would be more comfortable removing "Saucy" from the list of images to download.
<smoser> bug 1232174
<ubot5> bug 1232174 in MAAS "maas-import-pxe-files fails because Saucy's images information is not available." [Critical,Triaged] https://launchpad.net/bugs/1232174
<smoser> rvba, ok... i'll look at fixing that script for saucy.
<smoser> i think actually the issue is unfortunately deeper than you think.
<rvba> oh?
<roaksoax> rvba: but we can simply disable the import of saucy for the time being... can tych0's replacement make use of saucy'
<roaksoax> rvba: but we can simply disable the import of saucy for the time being... can tych0's replacement make use of saucy's devel images?
<roaksoax> smoser: devel images are already up right?
<rvba> roaksoax: I'd be in favor of reverting http://bazaar.launchpad.net/~maas-maintainers/maas/trunk/revision/1641.
<smoser> rvba, revert the config file change
<smoser> but not the import_pxe_files
<smoser> and just ship thtat for today
<smoser> thats my suggestion
<smoser> honestly... i *REALLY* would like to see enum.py use python-distro-info
 * rvba nods
<smoser> whifch would take rvba probably 15 minutes to do, and then we'd never have this non-sense about maas refusing to deploy something because it doesn't hknow a string.
<roaksoax> rvba: yeah only revert RELEASES="precise quantal raring saucy"
<rvba> Okay.
<rvba> Next time, please test a change like this before (or right after) landing it.
<rvba> roaksoax: https://code.launchpad.net/~rvb/maas/fix-m-i-p-f/+merge/188610
<roaksoax> rvba: done
<rvba> Ta.
<rvba> roaksoax: the change is merged.
<roaksoax> rvba: thanks! releasing now then
<smoser> rvba, http://paste.ubuntu.com/6179821/
<smoser> that pretty much creates the DISTRO_SERIES and DISTRO_SERIES_CHOICES that are used in enum.py
<smoser> it'd be nice if distro_info gave you that dict that i have to make up. or at least something like it.
<rvba> Indeed.
 * rvba filed bug 1233713.
<ubot5> bug 1233713 in MAAS "The list of supported distro series is hardcoded in src/maasserver/enum.py" [High,Triaged] https://launchpad.net/bugs/1233713
<smoser> http://paste.ubuntu.com/6179846/
<smoser> nothing is simple though.
<mgz> juju-core uses distro-info!
<roaksoax> so we used to have that before I remember but the changes got reverted
<mgz> (we chet a little, have some hardcoded values, then repopulate if something is found, using the ubuntu.csv file)
<roaksoax> rvba: ok so we are good to replace maas-import-ephemerals to the py version?
<rvba> roaksoax: well, no; we haven't tested it yet.  Right tych0?
<roaksoax> rvba: ok
<tych0> well
<tych0> i ran a few tests locally
<tych0> i don't think anyone else has tried it, though
<tych0> which would be good to do
<rvba> Well, I just tried it.
<tych0> did it work? :-)
<rvba> It downloads stuff, but now I can't get my nodes to boot.
<tych0> hmm
<rvba> iscsi error
<rvba> http://people.canonical.com/~rvb/iscsi-error.jpg
<tych0> arg
<tych0> i've seen that.
<tych0> hmm
<smoser> distro_info much harder to work with than i'd like. but this gets a sane array of everything. http://paste.ubuntu.com/6180076/
<byro> hi
<byro> need some help...any volunteer?
<byro> ok, I will try to post my question.....
<byro> I've setup two VMs in virtualbox
<byro> one runs the maas region/cluster controller. let's call this one "master"
<byro> the master vm has three interfaces:
<byro> lost connection
<byro> trying again:
<byro> two VMs: "master", and "node"
<byro> master has following interfaces: eth0: Internal Network - eth1: NAT - eth2: host
<byro> node has only eth0: on Internal network
<byro> I've maas region and cluster controller running on master
<byro> master IP addresses are: eth0:192.168.0.11, eth1=10.0.3.15, eth2=192.168.56.102
<byro> the problem is:
<byro> node does pxeboot, but it receives a preseed with cloud-config-url=http://10.0.3.15/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed log_host=10.0.3.15.....
<byro> I see the DNS address to 10.0.3.15...
<byro> but node doesn't have an interface on that net
<byro> so, what do I do on master to make the DNS show up as 192.168.0.11 instead of 10.0.3.15?
<AskUbuntu> virtualbox, MAAS: wrong seeding? | http://askubuntu.com/q/352363
<smoser> rvba, roaksoax we shoudl have a released ephemeral for saucy now
<Byro> anybody willing to take a stub at http://askubuntu.com/questions/352363/virtualbox-maas-wrong-seeding ?
<Byro> test
<roaksoax> smoser: do you need simplestreams seeded?
<roaksoax> smoser: i was thinking of adding a dependency to simplestreams to maas (before we oficially move to the new import epehemrals script)_
<ppetraki> maas issue[tcpdump]:  http://paste.ubuntu.com/6180595/ , I can bootstrap and destroy an env, but juju status always reports the env is not found
<smoser> roaksoax, yes. please add that dependency!
<smoser> infinity bothers me daily to have it seeded
<roaksoax> smoser: done! I just need to test and will upload to archive
<stokachu> anyone run into issues when installing maas from cloud-tools and postgres not starting
<roaksoax> smoser: great! commissioning seems to be broken!
<roaksoax> oh nevermind
<roaksoax> it just took forever
<roaksoax> smoser: ok so I uploaded it
<Byro> Hi
<smoser> roaksoax, you uploaded maas to saucy ?
<roaksoax> smoser: yes
<smoser> and import-ephemerals and import-pxe-files works out of the box ?
<smoser> did you fix import-pxe-files ? or did it get fixed due to removal of saucy
<Byro> Hi... I see a node booting PXE trying to access 169.254.169.254. Where is that coming from, and what should it actually be?
<Byro> l
<smoser> if its trying to get there, then it didn't find the maas datasource.
<smoser> how did you boot it?
<Byro> where is that supposed to be configured?
<Byro> I have a server VM with maas running...
<Byro> I'm booting the "node" vm in PXE
<Byro> it is correctly getting the IP from the dhcp server in MAAS node...
<Byro> until a few minutes ago I was seeing dhcp0 to be wrong, but fixed that, still that had no effect on this issue
<smoser> Byro, i think you need isc-dhcp-server from https://launchpad.net/~virtual-maasers/+archive/maas-updated-packages
<Byro> smoser: I guess my IRC timed out....
<roaksoax> smoser: yes
<Byro> did you have some wise word for me?
<roaksoax> smoser: removing saucy from the list of releases in /etc/maas/import_pxe_files worked
<smoser> Byro, i suspect you're hitting https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1069570
<ubot5`> Ubuntu bug 1069570 in maas (Ubuntu Raring) "1 MAC Address, two IPs - DNS is "out of sync" with DHCP leases databases" [Undecided,Confirmed]
<smoser> and you can ge ta build with that fix in it at https://launchpad.net/~virtual-maasers/+archive/maas-updated-packages
<smoser> roaksoax, thank you.
<smoser> you'll also need to modify your maas dhcp.conf Byro
<Byro_> the stupid client keeps timing out
<Byro_> but I did see your answer....
<Byro_> I don't have a single mac though
<Byro_> I have three separate eth on the server
<Byro_> each one gets one IP
<Byro_> do you still think that bug would affect me?
<byro> hi
<byro> my client keeps disconnecting
<smoser> byro, but you're surely only booting off of one of them.
<smoser> basically eth0 has to get an ip off of dhcp and has to be ocnnected to amas.
<byro> yes, I'm booting out of 192.168.0.11
<byro> and correctly getting IP
<byro> I also was able to fix the dns0
<byro> it now correctly indicates 192.168.0.11 instead of 10.0.15.3
<byro> but...
<byro> I still get the problem
<smoser> byro, this is commissioning ?
<smoser> or enlistment?
<byro> uhm...
<byro> sorry, not entirely accustomed to jargon yet....
<byro> hold on
<smoser> what have you done so far?
<byro> there are no known nodes to the controller yet
<smoser> yeah.
<smoser> so thats enlistment
<byro> ok
<smoser> the first time the node boots it shoudl pxeboot and then tell maas all of its network interfaes.
<smoser> then you 'accept' the node, and it will "commission"
<byro> right
<byro> ok
<byro> question:
<byro> is this ok?
<byro> #cloud-config datasource:   MAAS:     timeout : 50     max_wait : 120     # there are no default values for metadata_url or oauth credentials     # If no credentials are present, non-authed attempts will be made.     metadata_url: http://10.0.3.15/MAAS/metadata/enlist
<byro> this is the preseed
<byro> it also shows 10.0.3.15
<byro> while the pxe-booted node can't reach that network
<byro> should that be 192.168.0.11 instead (address of maas server on reachable network)?
<smoser> yeah.
<byro> ok...
<smoser> try dpkg-reconfigure maas-provisioning-server
<byro> ok
<byro> dpkg is not installed
<byro> will install
<byro> uhm...
<byro> no info available to install it...
<byro> actually, sorry
<byro> maas-privisioningt server is not installed
<byro> maas-provisioning-server
<byro> ?
<byro> am I still in the channel?
<Byro> hello
<Byro> sorry if somebody already replied, I keep getting kicked out of the channel due to timeouts
<Byro> my preseed have the wrong ip address in it
<Byro> how do I change that?
<Byro> how can I change the metadata_url IP address in the preseed?
<Byro> anbody here?
<stokachu> roaksoax: whats the best way to see whats happening with a node before enlistment? i can see where the maas server offered its dhcp information but not sure where to check status of the node
<tych0> ok, i know i'm forgetting something simple; my pxe nodes can't boot because they can't find pxelinux.0
<tych0> it gets an IP, and tgtd is running on the maas machine
<tych0> but /var/log/maas/pserv.log shows no signs of activity when i try to pxe boot
<tych0> what am i forgetting to start? :-)
#maas 2013-10-02
<roaksoax> stokachu: after pxeboot id say you need to connect to the image and debug it
<roaksoax> tych0: check restart pserv maas-dhcp-server etc
<roaksoax> maas-pser
<tych0> roaksoax: no luck with maas-pserv and maas-dhcp-server
<roaksoax> tych0: try restarting rabbitmq-server maas-cluster-celery in that order
<roaksoax> then maas-dhcp-server and maas-pserv
<tych0> roaksoax: arg, no luck there either. gotta run, though, any other tips are welcome :-)
<roaksoax> tych0: vms?
<roaksoax> killbthe vm and start it again
<roaksoax> dont just restart
<AskUbuntu> Problem with IP Openstack MAAS | http://askubuntu.com/q/352570
<AskUbuntu> MAAS login error | http://askubuntu.com/q/352573
<nesusvet> hello everyone! )
<nesusvet> Node writes the following massage: juju.state open.go:88 connection failed, will retry: dial tcp 127.0.0.1:37017
<nesusvet> I opened the /etc/mongodb.conf file
<nesusvet> and find out that mongodb port is port = 37017
<nesusvet> ohh sorry, I mean 27017
<jtv> nesusvet: you may have to ask someone in #juju what that implies first.
<jtv> Juju runs MongoDB itself.  It may still be a MAAS problem, but the immediate meaning of the error message relates to Juju.
<nesusvet> thanks jtv
<freeflying> can I add mac address to a commissioned node in maas?
<roaksoax> jtv2: around?
<jtv> Hi roaksoax
<roaksoax> jtv: howdy! so i figured things out so no worries :)
<jtv> Ah
<stokachu> is maas-region-controller forcing a preconfigure even if setting non interactive flag in apt?
<stokachu> im attempting to force non interactive with maas install " apt-get install -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' -f -q -y maas maas-dns maas-dhcp" however the maas/installation-note template is being shown regardless
<dpb1> stokachu:  DEBIAN_FRONTEND=noninteractive in front of all that?
<stokachu> dpb1: yea
<dpb1> and what about < /dev/null at the end?
<stokachu> ah didnt add that
<stokachu> lemme try with that
<roaksoax> allenap: around?
<roaksoax> allenap: could you please add maas-maintainers as bug subscriber for https://launchpad.net/ubuntu/+source/djorm-ext-pgarray
<allenap> roaksoax: Sure.
<roaksoax> allenap: thanks!
<stokachu> dpb1: sweet that worked!
<stokachu> thanks :D
<dpb1> nice.  when in doubt, old shell tricks to the rescue
<stokachu> dpb1: most definately
<allenap> roaksoax: Where's the best place to see commissioning errors?
<roaksoax> allenap: in the image itself
<roaksoax> allenap: you'd need to connect to the image
<allenap> roaksoax: Doesn't it shut down when commissioning is complete?
<roaksoax> allenap: you can prevent it from doing so: https://lists.launchpad.net/maas-devel/msg00808.html
<roaksoax> allenap: that's why we were requesting a way to store that info in maas and make it available
<allenap> roaksoax: That shouldn't be too hard to do now, with built-in commissioning scripts.
<roaksoax> allenap: indeed, but I think they do report back to maas
<roaksoax> smoser: ^^
<allenap> roaksoax: Btw, thanks for your help.
<roaksoax> np
<kentb> which cloud-tools ppa do I need to be connected to to get the latest stable juju / maas packages for precise? Is it "deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main" ?
<stokachu> kentb: yea thats what ive been using
<kentb> stokachu: ok. cool. thank you
<stokachu> np
<smoser> kentb, stokachu, roaksoax uploaded a new maas to saucy yesterday
<smoser> but its not into that cloud-tools yet.
<kentb> ok. thanks for the heads up
<smoser> roaksoax, allenap you should be able to post back results. errors or othereise.
<smoser> right ?
<stokachu> smoser: cool thanks
<roaksoax> stokachu: we are waiting for approval from the release team
<allenap> smoser: Yeah, you can.
<allenap> smoser: It's limited to 1MB right now, but we could extend that.
<smoser> i'd think you shoudl extend it, but that shoudl'nt be a problem.
<smoser> realistically, thats a hell of a lot of text
<stokachu> im running maas in a vagrant-lxc container
<stokachu> pretty sweet
<roaksoax> smoser: how do we enable the use of curtin now?
<smoser> tags i think
<smoser> http://bazaar.launchpad.net/~smoser/+junk/xinstall/view/head:/maas-usage.txt
<roaksoax> smoser: thanks
<kentb> so if I wanted to run maas - 1.4+bzr1551 on raring, is there a way to get that?
<stokachu> kentb: if you needed it right now you could use backportpackage
<stokachu> and just backport from saucy
<kentb> stokachu: ok. yeah, good idea
<stokachu> kentb: im not sure what maas devs release schedules are like to know when it would land in raring
<kentb> stokachu: ok. makes sense.
<stokachu> kentb: i would say use this but it hasn't been built yet
<stokachu> https://launchpad.net/~maas-maintainers/+archive/dailybuilds?field.series_filter=raring
<stokachu> kentb: looks like i have access to request builds
<stokachu> kentb: just launched it for raring
<kentb> stokachu: sweet! thanks!
<stokachu> np :)
<stokachu> kentb: crap it just does 1.3
<stokachu> kentb: i would just use backportpackage then
<kentb> stokachu: ok
<smoser> roaksoax, tell me you fixed bug 1231693
<ubot5> bug 1231693 in MAAS "maas-dhcp backport does not start on precise" [High,Confirmed] https://launchpad.net/bugs/1231693
<roaksoax> smoser: nope, I thought you were looking at it
<smoser> well, i have a fix attached to it. but i was hoping you would hyave picked it up.
<smoser> can you take a look and see if you have a better idea for how to patch it?
<roaksoax> smoser: looks good to me
<smoser> k. i'll do a MP. to lp:ubuntu/maas
<roaksoax> smoser: could you please do it to lp:maas-maintainers/maas/packaging?
<smoser> yeah
<roaksoax> smoser: that way It can be solved in the next upload
<smoser> what is the relationship to that and trunk ?
<smoser> ah
<smoser> i see
<roaksoax> smoser: packaging trunk is lp:maas-maintainers/maas/packaging
<smoser> trunk has no debian/
<roaksoax> correct!
<roaksoax> smoser: let me know when you propose for merging the upstart thing so I can upload a new maas today
<smoser> roaksoax, i'm working on it.
<roaksoax> smoser: did narinder talk to you about enabling curtin in saucy?
<roaksoax> with the new preseed and stuff?
<smoser> asked about enabling, yes
<smoser> roaksoax, i dont think that this bug should exist
<roaksoax> smoser: how so?
<smoser> dpb1, bug 1231693
<ubot5> bug 1231693 in MAAS "maas-dhcp backport does not start on precise" [High,Confirmed] https://launchpad.net/bugs/1231693
<smoser> because current packaging only assumes precise
<smoser> err...
<smoser> assumes the behavior found in quantal
<smoser> (quantal or later)
<smoser> of isc-dhcp-server
<smoser> and cloud-archive cloud-tools will only have isc-dhcp-server at that level.
<roaksoax> smoser: right, but also assumes that we would be using the isc-dhcp-server on precise was the same as is on the precise archives (and the CA is backporting a version of isc-dhcp
<smoser> look at the current packaging branch
<smoser> debian/maas-dhcp.maas-dhcp-server.upstart
<smoser> there is no "if precise" logic htere.
 * dpb1 reads scrollback
<roaksoax> if [ "$RELEASE" = "precise" ]; then
<roaksoax>         owner_group="dhcpd:dhcpd"
<roaksoax>         dhcpd_owner_opts=""
<roaksoax>     else
<roaksoax> smoser: ^^
<smoser> roaksoax, where did you see that ?
<roaksoax> smoser: in debian/maas-dhcp.maas-dhcp-server.upstart for saucy (which is what you are backporting to the CA)
<roaksoax> s/you/we/
<roaksoax> smoser: the problem is that the cloud-tools-next pocket is *also* backporting a isc-dhcp from *saucy*
<smoser> what the hel
<roaksoax> smoser: hence, the upstart jobs goes "if release is precise, I assume we are using isc-dhcp *from* precise"
<roaksoax> hence, set owner_group to dhcpd:dhcpd
<dpb1> backporting is fun
<roaksoax> smoser: but since the cloud-tools-pocket next is providing a isc-dhcp-server that requires ownership to be *root:root* then it fails
<smoser> shoot.
<smoser> i had out of date bzr here.
<smoser> the fix that was put in
<smoser> adding that logic around "if release"
<smoser> is what actually broke this for us if we had cloud archive
<roaksoax> smoser: not really the logic, because the logic assumes that we are using isc-dhcp *from* precise
<smoser> since cloud-archive is quantal-style behavior for isc-dhcpd
<smoser> right.
<roaksoax> and the cloud-archive is providing isc-dhcp *from* saucy
<smoser> but if they didn't add that logic
<smoser> they would have acted like saucy
<smoser> and worked
<smoser> because cloud-archive isc-dhcp-server is saucy
<roaksoax> smoser: yes, but that logic was added under the assumption that we *will* be using isc-dhcp *from* precise
<roaksoax> not from saucy
<smoser> right.
<smoser> and that assumption is wrong.
<smoser> so
<smoser> we have 2 options
<dpb1> smoser: didn't your patch change it to a behavior based evaluation, not release-based?
<roaksoax> smoser: not really, it is not wrong because it was the assumption that the fix for client-uids would have landed in precise too
<smoser> a.) code a more complex fix (like shown int hat bug) that accounts for different behaviors of isc-dhcp
<smoser> b.) do not support trunk maas on precise without isc-dhcp-server > 4.2
<smoser> roaksoax, it would still be wrong there.
<smoser> as if we backported the client-uuids to precise it would not change the ownershp behavior to saucy/quantal behavior
<roaksoax> saucy/quantal use root:root
<smoser> right.
<smoser> and that is waht cloud-archive uses
<smoser> as it uses saucy isc-dhcp-server
<roaksoax> smoser: correct, but the idea was to backport the minimum possible in the CA
<roaksoax> correct?
<dpb1> change the package dependencies, and take out the if clause, is that what you are proposing, smoser in b.)?
<roaksoax> so that client=-uids fix would have landed in precise with the appropriate technical board exception
<roaksoax> hence, not requireing us to backport an isc-dhcp to the CA
<roaksoax> which means, that the fix would have worked
<roaksoax> that is the reason why that distiction exists
<smoser> roaksoax, wrong. the idea is to use ubuntu-current-release versions backported.
<roaksoax> because there was the assumption that the client-uids fix will land in *precise* (which never did), *and* at the time, we never had the idea that we will have a cloud-tools pocket
<smoser> ah. yeah, so that would be a possbility.
<smoser> if we want ot support that.
<smoser> i dont really want to
<roaksoax> smoser: correct, so at the end of the day, I agree that we can drop that from saucy'
<smoser> the easiest thing to do is just depend on > 4.2 and say sorry, trunk maas only works on precise with cloud-archive version of isc-dhcp-server
<roaksoax> smoser: correct, so at the end of the day, I agree that we can drop that from saucy's packaging
<roaksoax> i was just making sure why this was done that way
<smoser> right
<roaksoax> so you could understand the logic placed there
<roaksoax> smoser: ok, so let's just drop that
<roaksoax> and we should be good to go
<smoser> roaksoax, funny
<smoser> it doesn't work anyway
<smoser> i'im confused on how it workd
<roaksoax> it should work
<smoser> because if you try to run this on 12.04 isc-dhcp-server
<smoser> then it app-armor fails
<smoser> [ 6638.305979] type=1400 audit(1380748171.914:45): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/dhcpd" name="/var/lib/maas/dhcp/dhcpd.leases" pid=16582 comm="dhcpd" requested_mask="r" denied_mask="r" fsuid=106 ouid=106
<smoser> ah. no i see m
<smoser> maas delivers a fix for that and  I didn't have it.
<smoser> roaksoax, http://paste.ubuntu.com/6185681/
<smoser> thats my complex patch
<smoser> i'm way past "have to go now"
<smoser> could you just disable that "run on precise" logic and fix it?
<smoser> roaksoax, https://code.launchpad.net/~smoser/maas/packaging.1231693/+merge/188932
#maas 2013-10-03
<smoser> roaksoax, are you around ?
<smoser> roaksoax, there are two MP there. i'm really ok with you chosing either one.
<roaksoax> smoser: sorry wasn
<roaksoax> feeling so good
<roaksoax> smoser: i'll take a look tomorrow
<androoo> hey, is anyone here willing to help out a noobie?
<racedo> ping rvba
<rvba> racedo: Hi (otp right now).
<racedo> rvba: oh sorry
<racedo> rvba: quick one when you can
<racedo> rvba: when splitting maas region controller and maas cluster controller into different servers, the juju client will never talk directly to the maas cluster controller, right?
<rvba> racedo: correct
<racedo> rvba: many thanks :)
<rvba> Welcome :)
<racedo> rvba: and we can have just one juju bootstrap node for all the maas cluster controllers?
<racedo> rvba: or should we commission one per cluster controller?
<rvba> racedo: yes, as long as all the nodes managed by all the cluster controllers can talk to each other.
<racedo> rvba: ok, so multiple juju bootstrap nodes is kind of pointless then
<rvba> Yes.
<racedo> cool, thanks again rvba
<AskUbuntu> Migration from generic OpenStack to Ubuntu Openstack | http://askubuntu.com/q/353127
<roaksoax> rvba howdy
<rvba> Hi roaksoax
<roaksoax> rvba: hold on those ladt two mps
<roaksoax> last*
<rvba> roaksoax: don't worry, I don't intend to land them now.
<roaksoax> rvba: ok ;) ill review them in a bit
<rvba> roaksoax: they are mostly to get us started on testing the new script.
<roaksoax> rvba: ok but im releasing maas again today
<rvba> Okay.
<roaksoax> rvba: ok, commented on them
<roaksoax> rvba: other than that if they can be merged today
<roaksoax> rvba: it would be great
<rvba> roaksoax: Thanks for the review.  We will need more testing before we land it; in particular, we want to make sure the old config gets migrated ok.
<roaksoax> rvba: ok
<roaksoax> rvba: though it would have been simple to keep the old config style as it is with import_pxe_files
<roaksoax> y
<rvba> roaksoax: we will rewrite maas-import-pxe-files in Python too.  We thought it was a better idea to unify the configs.
<roaksoax> rvba: ok
<freeflying> rvba, if import-pxe-files support specify mirror, might be more convenient
<rvba> freeflying: true, we're still working on maas-import-ephemerals and import-pxe-files is our next target but you're right, we need to be able to configure where simplestreams will fetch its information.
<stokachu> is the web interface the only way to manually enlist nodes?
<kurt_> roaksoax: ping
<roaksoax> kurt_: pong
<roaksoax> stokachu: if you have a live image on the node, you could install maas-enlist
<stokachu> roaksoax: ok ill try that
<kurt_> roaksoax: something I was thinking through with MAAS/juju - there is no way to guarantee a particular node with particular resources is used for a enlistment/commission/juju deployment - is that right?
<kurt_> if you don't have control over WOL, you are in bad shapee
<kurt_> shape
<kurt_> actually that doesn't even matter from juju's perspective it seems
<roaksoax> kurt_: ideally you need control over ipmi, but enlistment/commuissioning are steps that have to be done in order for the machines to be available to juju
<roaksoax> kurt_: now, you would need to add tags to the noeds to identify them based on constraints
<roaksoax> and that way, when you juju deploy something you can gfarantee that a particular node with mparticular hw detauls is assigned to juju by specifying the constraints
<kurt_> tags?
<kurt_> first I'v heard of this
<roaksoax> kurt_: http://maas.ubuntu.com/docs/tags.html
<kurt_> thanks I'll have a look - that integrates with juju?
<kurt_> ahhhâ¦I seee
 * kurt_ lightbulb lights up
<smoser> stokachu, you can enlist nodes with the cli too
<stokachu> smoser: i was able to get it to work with maas-enlist, is there another preferred way?
<roaksoax> stokachu: maas-nelist when manually
<roaksoax> stokachu: it is much easier, and that's what it is actually used when running the enlistment process
<stokachu> cool that worked like a champ
<smoser> stokachu, are you using real nodes ?
<stokachu> smoser: virtualbox atm
<smoser> virtualbox.
<smoser> for real?
<smoser> are you running this on windows!
<stokachu> haha
<kurt_> smoser: I have been very successful with VMWare workstation too
<stokachu> im trying to automate a maas+10 node setup
<kurt_> and Fusion on Mac OSX
<kurt_> I'm in the middle of blogging the experience
<smoser> stokachu, http://bazaar.launchpad.net/~virtual-maasers/charms/precise/virtual-maas/trunk/files
<smoser> see README-nojuju.txt
<smoser> that is basically what you're after
<smoser> and you dont have to use virtualbox
<stokachu> ok ill try that way next
<kurt_> what does virtual-maas do differently?
<stokachu> uses kvm
<kurt_> in what way?  I've got a hypervisor
<stokachu> you manage all your nodes and maas server with libvirt
<kurt_> regulÃ¤r maas has that built in too, does it not?
<smoser> maas has support for "power" control via virsh
<smoser> virtual-maas just uses that.
<smoser> and http://bazaar.launchpad.net/~virtual-maasers/charms/precise/virtual-maas/trunk/files
<smoser> has tools to create nodes and insert them into maas.
<smoser> and remove them.
<kurt_> ok because virsh is a part of libvirt
<stokachu> kurt_: this helps you automagically deploy maas+nodes
<smoser> virsh is a part of libvirt, yes. it wouldn't be hard to extend maas's power control to cover virtualbox.
<kurt_> ok, I just figured out how to do this all manually on my own, so I'm curious about the benefit
<smoser> and if you have some reason that you want virtualbox (like you want to use this on a mac) then that might make sense to re-invent things.
<smoser> but what is there in 'virtual-maas' is very close to "just works".
<smoser> as a "all in one maas" with virtual nodes.
<kurt_> for some reason I remember reading virtual-maas not going to be updated too
<stokachu> kurt_: i posted a blog article on setting up maas within lxc
<kurt_> stokachu: I'm working on blogging my results too
<stokachu> kurt_: http://astokes.org/running-maas-vagrant/
<kurt_> I'll have a look
<smoser> kurt_, sothere are some parts of virtual-maas that are hacky.
<smoser> its basically all in one.
<smoser> it doesn't survive reboot
<smoser> those htings can be fixed for sure
<kurt_> I just found out about the tagging thing, which will make life exponentially better
<kurt_> I've managed to get MAAS/Juju and Openstack fully up and running
<kurt_> on VMWare Fusion and Workstation that is
<kurt_> anyways - you have now piqued my curiosity
<kurt_> to see how much work I did for NOTHING :D
<kurt_> lol
<kurt_> smoser: so is virtual-maas officially supported on precise?
<smoser> "officially", no
<smoser> it will probably work.
<smoser> its really a development thing.
<kurt_> right
<smoser> but there is a lot of generally usable things there.
<smoser> ie, scripts to add a node to libvirt with 2 disks and X memory and then shove that node into maas
<smoser> and then (generically) power that machien on and install it with release X
<smoser> the only hacky things really are in getting maas up and going and listening on the maasbr0 network and the networking involved.
<kurt_> ok.
<stokachu> so has anyone run maas in virtualbox on windows? :)
<kurt_> workstation :)
<smoser> kurt_, really. give it a try. if you have a system with 8G memory, its fairly easy.
<stokachu> kurt_: did you use vagrant or just manually through vmware?
<smoser> slow, but easy.
<smoser> and you can play. it is a very easy way to get a juju going
<kurt_> manually
<kurt_> as I said, it works quite well
<stokachu> kurt_: ok, ive got a pretty automated thing ive setup if you dont mind testing on windows
<stokachu> im not done yet but hopefully tomorrow
<stokachu> it would use virtualbox though
<kurt_> stokachu: thing = a package, scripts?
<stokachu> kurt_: yea using vagrant+virtualbox
<stokachu> but automated
<stokachu> just curious if it is portable across mac/windows
<kurt_> if it means I have to take down my workstation install, I'm afraid its going to have to wait - I'm knee deep in blogging
<stokachu> it wont touch your vmware stuff
<stokachu> in any case just let me know when/if youre interested
<kurt_> ok.  I'm playing with ceph next too
<stokachu> kurt_: nice, dont forget to write your findings i'd be interested in looking at that too
<kurt_> will do ;)
<stokachu> is there a way to alter the preseed file or do i need to edit it manually
<stokachu> http://maas.ubuntu.com/docs/development/preseeds.html i see this page
<stokachu> im trying to force the enlist server to be on a specific network device
<stokachu> seems to default to the first device no matter what
<stokachu> whats the best way to override http://paste.ubuntu.com/6189451/
<stokachu> thats from maas_local_settings.py
<kurt_> stokachu: can't you go in to the interface and specify which device its doing dhcp on?
<kurt_> interface = web gui
<stokachu> lemme check
<stokachu> eth1192.168.50.255/24Manage DHCP and DNS
<kurt_> right
<stokachu> so its using eth1
<kurt_> correct
<stokachu> but the default maas url is set to 10.0.2.15
<stokachu> which is eth0
<kurt_> you need to dpkg-reconfigure
<kurt_> set it to the IP of the maas url you want
<stokachu> in my dhcpd.conf?
<kurt_> no
<kurt_> hang on
<stokachu> dpkg-reconfigure maas-region-controller lets me change the maas url
<kurt_> right
<kurt_> that's what I meant
<stokachu> ah ok, how do i do this programmatically though
<kurt_> you mean from the maas-cli?
<stokachu> sure or any other avenue
 * kurt_ crickets
<stokachu> lol
<stokachu> hmm if only i could pass an ip to dpkg-reconfigure
<stokachu> during setup
<kurt_> there has to be a way
<kurt_> did you fully research the maas-cli options?
<stokachu> ill look through it again and see if im missing something
<kurt_> hmmâ¦well that wouldn't work too well
<kurt_> because the authentication is tied in to the url for the maas-cli
<stokachu> i found this http://paste.ubuntu.com/6189525/
<kurt_> that's a roaksoax or bigjools question
<stokachu> not sure if enlistment_domain would work or not
<kurt_> why do you need to change the maas url once or twice?
<stokachu> so when i install maas onto a server with eth0 and eth1 i need mass to only listen and respond on eth1
<stokachu> for whatever reason eth0's ip is being picked for the maas default url
<kurt_> you could hack IP forwarding to make it not matter :)
<stokachu> true
<kurt_> which is what I've done as I recall
<stokachu> ive edited the maas_local_settings.py for now which seems to provide a proper preeseed
<kurt_> but I've always just gone back in and reconfigured as you put in above
<stokachu> ok
<kurt_> I need IP forwarding on my virtual network anyways
<kurt_> that's the only way my maas nodes are  going to get access to the internet
<kurt_> so its always just worked for me
<kurt_> stokachu: btw this is the best guide on easy IP forwarding in case you didn't have an easy source
<kurt_> http://askubuntu.com/questions/95199/two-network-cards-and-ip-forwarding
<stokachu> thanks
<stokachu> i think cloud-tools archive died
<stokachu> The following packages have unmet dependencies: maas : Depends: maas-region-controller but it is not going to be installed
<stokachu> smoser: ^ was there a recent push to archive
<stokachu> guess that should be roaksoax ^
<dpb1> should this work?  curl http://169.254.169.254/2009-04-04/meta-data/  (from a maas node) ?
<stokachu> bah ithink the archive is broken :(
<dpb1> Anyone seen this error with juju/maas from the UCA:     agent-state-info: '(error: could not access file ''tools/juju-1.14.1-precise-i386.tgz'':
<dpb1>       gomaasapi: got error back from server: 401 UNAUTHORIZED (Nonce already used:
<dpb1>       33410643))'
<dpb1> machines 1-18 went fine, on deploying this one, I got this error.
<stokachu> is anyone else using the cloud-tools archive?
<stokachu> it seems to be busted
<stokachu> roaksoax: ^
<roaksoax> stokachu: why is that?
<roaksoax> stokachu: what are you experiencing?
<roaksoax> 5
<roaksoax> 4
<roaksoax> 3
<roaksoax> 2
<roaksoax> 1
 * roaksoax dead
<roaksoax> :)
<stokachu> roaksoax: sec
<stokachu> roaksoax: http://paste.ubuntu.com/6190310/
<roaksoax> stokachu: are you using saucy?
<stokachu> this is on precise using the cloud-tools repo
<roaksoax> this seems because tierh package has a higher version and it is trying to install a lower version
<roaksoax> stokachu: weird, I'll test later
<roaksoax> gotta go
<stokachu> ok
<stokachu> deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main
<stokachu> deb-src http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main
<stokachu> thats my sources.list
#maas 2013-10-04
<stokachu> roaksoax: i think i know whats wrong, maas relies on python-curtin
<stokachu> but thats no where in the archive
<stokachu> smoser, roaksoax: python-curtin only exists in saucy atm
<stokachu> i believe thats why the archive is failing
<AskUbuntu> region controller does not know whether any boot images have been imported | http://askubuntu.com/q/353426
<Byro> hi
<Byro> I'm stuck with maas configuration...
<Byro> I posted a question on askubuntu
<Byro> http://askubuntu.com/questions/353426/region-controller-does-not-know-whether-any-boot-images-have-been-imported
<Byro> I'd really appreciate it if somebody could take a look
<Byro> anybody around?
<dpb1> hi Byro, I ran into something like this recently
<dpb1> Byro: the script stalls there forever?
<dpb1> how long did you wait?
<dpb1> Byro: I hit this issue: https://bugs.launchpad.net/maas/+bug/1212434
<ubot5> Ubuntu bug 1212434 in MAAS "maas-import-pxe-files breaks on 404" [Critical,Fix committed]
<jtv> dpb1: we landed a fix for that in the development codebase...  Which packages are you using?  Daily?
<jtv> The script really can't stall forever, but the downloads can take a long time.  You can tweak the RELEASES and ARCHES settings to eliminate downloads you don't need.
<dpb1> jtv: ya, I'm past the issue, I was just replying to Byro up there.
<jtv> Ah OK
<dpb1> Byro, see jtv's question ^
<jtv> We're just in the process of re-doing those download scripts, especially the one for the commissioning images.
<dpb1> jtv: ya, I was reading about that a couple days ago.  makes sense, given maas is useless if they don't work. :)
 * jtv ponders people carrying USB sticks around
<jtv> Yes, pretty much.  :)
<jtv> Now, as far as the maas-import-pxe-files script is concerned, if you're running a reasonably recent version of the package, you could just run the latest upstream version.  No big changes.
<jtv> But it calls the maas-import-ephemerals script, which is a wholly different beast and AFAIK will still break if downloads fail.
<dpb1> jtv: ya, I had a bug about that oo
<dpb1> too
<jtv> I need to step out for some time...  But if it's something I can help with, please explain here and I'll get back to it!
<jtv> (Or if appropriate of course, just file a Launchpad bug :)
<dpb1> jtv: oh, it's the same bug I think.  in the comments. :)
<dpb1> jtv: cheers. :)
<gnuoy> hi, I have a newly created maas and I'm finding that commissioning is failing most of the time. when a node fails commissioning it never seems to come up on the network and the remote serial connection shows it loading initrd.img for ~1 minute and then reports it failed and prompts to press a key to continue to nothing else. I'd like pass console=ttyS1,38400 to the kernel which I'm hoping will give me more info
<gnuoy> how would I go about doing that ?
<rvba> Hi gnuoy, right now, you need some manually hackery to do that.  If you find maas' kernel_opts.py file, you could change compose_arch_opts() to return the parameter you want. (Don't forget to 'sudo service apache2 restart' for the modification to take effect.)
<gnuoy> rvba, perfect, thank you
<rvba> welcome
<AskUbuntu> Build And Host my web application on my own private cloud | http://askubuntu.com/q/353508
<gnuoy> fwiw turns out my issues was Bug#1155556
<tych0> bug 1155556
<ubot5> bug 1155556 in python-tx-tftp (Ubuntu) "HP ProLiant DL380 G7 tftps kernel, but initrd tracebacks in tftp server. DL380 G6 succeeds." [Undecided,Fix released] https://launchpad.net/bugs/1155556
<rvba> roaksoax: Hi Andres.  I'm seeing something weird with the package that is in Saucy right now: when I install the package with other stuff, the installation breaks: see http://paste.ubuntu.com/6192266/
<rvba> roaksoax: any idea what might be wrong?
<roaksoax> rvba: ill check in a bit
<rvba> Thanks.
<rvba> roaksoax: also, but it's really not urgent, I'd be happy if you could have a look at the power-related bugs recently filed: https://bugs.launchpad.net/maas/+bugs?field.tag=power
<roaksoax> rvba: when did you experience the install issue?
<rvba> roaksoax: just now
<roaksoax> rvba: 3 of those appeared yesterday
<roaksoax> rvba: two of which we cant do anything
<roaksoax> rvba: and anotherone is on your side of things
<roaksoax> rvba: bug #1234910
<ubot5> bug 1234910 in MAAS "maas unsets ipmi protocol when commissioning" [High,Triaged] https://launchpad.net/bugs/1234910
<roaksoax> rvba: is something I had asked before
<roaksoax> and i was told "you cannot update power parameters individually"
<rvba> roaksoax: right, I need to look into thatâ¦ would you mind adding comments on the others and maybe marking them invalid (if we can't do anything).
<rvba> roaksoax: that's a bit bizarre, because we can update the power parameters individually; maybe I'm missing something here.
<roaksoax> rvba: no, if i have pwoer_user and power_pass set, and then I update power_user to something else, power_pass resets to default, which is ''
<roaksoax> rvso you pretty much have to update all the power parameters if you dont want them reset
<rvba> roaksoax: I see; well, yes, they basically behave as one piece of information.
<roaksoax> exatly
<roaksoax> rvba: so if a user changes A on the webui
<rvba> roaksoax: but the script manipulating the power parameters can workaround this.
<roaksoax> rvba: and then goes to the CLI to change B
<roaksoax> then A gets reset
<roaksoax> and shouldn't be that way
<roaksoax> the particular case here was
<roaksoax> machine enlisted, authentication was changed by the user to LAN_@_0
<roaksoax> and then on commissioning, since commissioning doesn't touch authentication method nor changes it
<roaksoax> it gets reset
<roaksoax> because commissioning only updates power_user, power_pass, power_address
<roaksoax> hence the power_type (i think is the variable) gets reset
<roaksoax> so the correct is to not reset other parameters when updating some
<roaksoax> rvba: ok just did a clean maas install without any issues...
<rvba> roaksoax: we have two fields: power_type and power_parameters.  They should be independent from one another.  I'll look into this.
<rvba> roaksoax: did you install other packages with the same "apt-get install" command?
<rvba> That's when the problem happens.
<roaksoax> i'll re-try
<rvba> roaksoax: try sudo apt-get install -y maas python-nose python-testtools juju python-zmq
<rvba> roaksoax: also, I don't understand why installing maas installs maas-dns, it's supposed to be optional IIRC.
<roaksoax> rvba: not anymore
<roaksoax> rvba: maas-dns and maas-dhcp are now default
<rvba> roaksoax: ah okâ¦ I don't have a problem with that but I suspect this change caused the problem I'm seeing.
<roaksoax> rvba: uhmmm maybe not cause I had my system and had been upgrading it without any issues
<roaksoax> rvba: where were you installing maas from though?
<roaksoax> in precise? saucy?
<rvba> roaksoax: saucy
<rvba> roaksoax: just the version in the archive
<roaksoax> rvba: ok testing your command now
<roaksoax> stokachu: ping
<stokachu> roaksoax: pong
<roaksoax> stokachu: so what was your issue installing from the cloud archive again?
<roaksoax> rvba: ok, know what the issue is
<roaksoax> rvba: maas-dns uses the 'maas' command
<stokachu> roaksoax: python-curtin is required and it doesn't exist in the cloud-tools archive
<smoser> stokachu, i'm fixing.
<roaksoax> stokachu: k thanks
<stokachu> smoser: cool thanks
<smoser> python-curtin *is* availalbe
<smoser> its (source=djorm-ext-pgarray ) python-djorm-ext-pgarray that was not
<rvba> roaksoax: I'm wondering why the installation doesn't fail when one installs only maas.
<smoser> i had failed to copy from -next to -staging.
<smoser> just did that now,
<stokachu> ah ok
<smoser> and then i'll push through -propsed and -updates
<roaksoax> rvba: yeah that's weird
<roaksoax> rvba: but anyway, maas-dns cannot be a dependency of maas-region-controller
<rvba> roaksoax: Right.
<roaksoax> rvba: which causes an issue because there would be no way to install it automatically
<roaksoax> and gets unseeded
<roaksoax> rvba: btw.. i just noticed that maas-dns depends on maas-dhcp, that's not needed... is it?
<rvba> roaksoax: I don't see why we would have such a dependencyâ¦
<rvba> roaksoax: hum, hang onâ¦
<smoser> stokachu, if you want to just get past this...
<smoser> grab the binary deb from https://launchpad.net/~ubuntu-cloud-archive-private/+archive/cloud-tools-proposed/+packages
<smoser> wait. wrong link
<stokachu> smoser: yea thats what i did
<smoser> ah. ok.
<smoser> form -next is probably bettre.
<rvba> roaksoax: if you want the dns stuff to work you have to let maas control the dhcp.  So the dependency actually makes sense.
<smoser> stokachu, and thank you for being willing to be a guinea pig. tryin gthis really helps.
<stokachu> smoser: not a problem at all :D
<roaksoax> rvba: right, but that's why the cluster controller is the one that provides dhcp isn't it?
<roaksoax> rvba: so in a multi environment maas, the dhcp server on the region (which is being installed by maas-dns)
<roaksoax> does really work
<roaksoax> and it is just installed for nothing
<rvba> roaksoax: yeah, you're right.
<roaksoax> rvba: i think i'd rather leave it as is
<roaksoax> rvba: being so late in the cycle
<roaksoax> we don't want to break anything
<rvba> Sounds sensible.
<roaksoax> rvba: i just uploaded maas_1.4+bzr1656+dfsg-0ubuntu2~ppa1 to ppa:maas-maintainers/experimental
<roaksoax> rvba: so we can test the multi package install
<roaksoax> and see if we experience the same issue
<rvba> roaksoax: okâ¦ well, the test will be easy.
<roaksoax> indeed
<rvba> roaksoax: arg, the package in the archive takes precedenceâ¦ is there a trick to force your package to be installed?  I know how to specify that a particular version must be used for one package, but if it's more than one package, it becomes tedious.
<roaksoax> rvba: that cant be
<roaksoax> rvba: the one in ppa has higher version
<roaksoax> it might have not been released yet
<rvba> roaksoax: you're right, it's not published yet.
<roaksoax> rvba: it is built but not released yet
<roaksoax> yep
<roaksoax> rvba: published now
<roaksoax> rvba: so what script will replce import_ephemerals?
<roaksoax> i mean what config file?
<rvba> gnuoy: As I said on the bug, I found out the feature to add custom kernel params is already there.  Sorry for the confusion, I'm so used to hack things manually that I sometimes forget about the user-facing features ;)
<rvba> roaksoax: the pserv config file.
<roaksoax> rvba: hold on, the pserv config file will include the config for importing ephemeral images?
<rvba> roaksoax: yes
<roaksoax> why?
<roaksoax> the ephemeral and import-pxe files
<roaksoax> should have its own separate config
<roaksoax> from the pserv service
<rvba> why?
<roaksoax> these are scripts
<roaksoax> that should be possible to run
<roaksoax> byt the user
<roaksoax> itself
<roaksoax> smoser: ^^
<rvba> Having a common config is not a problem for that.
<roaksoax> rvba: well, I see things differently but as long as maas-import-ephemerals is not a script limited to a maas daemon and the user *can* run it
<roaksoax> then pserv.yaml could be used as a config source
<rvba> That's the plan.
<rvba> (Note that we're still working on it)
<roaksoax> but given that it is a "external" script, it should have its own config
<rvba> roaksoax: hum, it makes senseâ¦ I'll talk to jtv (who is doing this work) about it, see what he thinks.
<rvba> roaksoax: your package installed correctly
<rvba> roaksoax: I'm calling it a day.  See you next week.
<roaksoax> rvba: so AFAIK, is it is place din pserv.yaml, in order for the script to actually catch the config, wouldn't we have to restart maas-pserv all the time too?
<roaksoax> because AFAIK the config gets loaded on pserv, and maas-import-ephemerals would obtain the config from there
<rvba> roaksoax: well, no.
<roaksoax> so if we change the config, then we would need to restart pserv
<roaksoax> rvba: alrigh! thanks for testing!
<rvba> I don't think so, the script is still run "as a script".
<rvba> But I'll check.
<rvba> Like I said, this is still WIP.
<roaksoax> rvba: ok! thanks!
<roaksoax> let's catch up on monday then
<rvba> Okay.
<smoser> roaksoax, i agree that the user should be able to run theset things
<smoser> stokachu, 'apt-get install maas' should work on proposed or updates now
<smoser> (i just tested)
<roaksoax> smoser: i did a new upload to archive that is sitting waiting for approval
<roaksoax> it is just a packaging change though
<smoser> rvba, roaksoax so does the import-ephemerals run regularly ?
<smoser> it really needs to. same with import-pxe-files
<roaksoax> smoser: it should
<smoser> and can we boot saucy now ?
<smoser> cause thats also moderately important :)
<smoser> roaksoax, did you fix http://paste.ubuntu.com/6193171/
<smoser> ?
<smoser> is that what your upload fixed?
<stokachu> smoser: ok ill test it now
<stokachu> thanks for the quick response
<roaksoax> let me see
<roaksoax> smoser: yeah that's the one i fixed
<smoser> stokachu, see above. it wont work :)
<smoser> but a couple more poushes. and we might get ther.e
<stokachu> ah lol ok cool
<stokachu> ive got a few ansible playbooks setup so i can switch between saucy and precise
 * roaksoax lunch
<stokachu> smoser, roaksoax: also lxc in the cloud-tools archive wont work unless running the lts-raring hardware enablement kernel i believe
<stokachu> not by default anyway unless there is a way to disable namespaces in lxc config
 * stokachu hasn't checked
<smoser> stokachu, what do you mean "work" ?
<smoser> current lxc works fine on precise.
<stokachu> lxc version 1.0?
<stokachu> from a default install of lxc from cloud-tools archive it won't start a container
<stokachu> spawns errors like 'get_cgroup" failed
<ppetraki> hey, how do I grab the power_parameters for a given node from the CLI?
<ppetraki> I'm trying to back up  a maas
<ppetraki> also, https://code.launchpad.net/~peter-petrakis/maas/maas/+merge/189407
<stokachu> ppetraki: maas-cli maas nodes -h
<stokachu> should give you an idea of what you can pull from it
<stokachu> ppetraki: http://maas.ubuntu.com/docs/maascli.html#node
<ppetraki> stokachu, yeah, tried that: maas-cli maas node  read node-eccf979e-2d23-11e3-9b02-525400052fb9 power_parameters_power_id
<ppetraki> u'power_parameters_power_id' is not a name=value pair
<stokachu> read only takes system_id
<ppetraki> right, it doesn't return enough information: http://pastebin.ubuntu.com/6193532/
<stokachu> what does power_parameters_power_id refer too
<stokachu> the virsh stuff?
<ppetraki> in this case it's for virsh
<ppetraki> but I'm really targeting this towards ipmi
<ppetraki> should still work though
<stokachu> ppetraki: they dont show user/pass there either
<stokachu> i wonder if the id is under the same umbrella
<ppetraki> that would suck
<stokachu> let me look at the source and see if there are any comments
<ppetraki> we show it in plain text in the gui, the cli is authenticated, but I can't get the same level of data back
<ppetraki> thanks
<stokachu> ppetraki: yea could just be an oversight
<stokachu> ppetraki: probably should file a bug to have the api expose that
<ppetraki> stokachu, ok, will do
<ppetraki> I'll try and hack my existing instances to expose those fields so I can back it all up
<ppetraki> stokachu, https://bugs.launchpad.net/maas/+bug/1235421
<ubot5> Ubuntu bug 1235421 in MAAS "power_parameters_* should be returned by nodes resource" [Undecided,New]
<stokachu> ppetraki: thanks
<Nik_> Hi. I'm having an issue with maas tags trying to identify machines that have more than 4 disks.. not sure if anytone is available who can help?
<Nik_> "definition": "count(//node[starts-with(@id,\"disk\") and @class=\"disk\"]) >= 4"
<Nik_> xmlstarlet sel -T -t -v 'count(//node[starts-with(@id,"disk") and @class="disk"]) >= 4' /tmp/lshw.xml
<Nik_> utput is "true"
<Nik_> but node doesn't get tagged
<Nik_> some do, but not that particular one
<AskUbuntu> Maas out of sync with lshw | http://askubuntu.com/q/353814
#maas 2013-10-06
<guntbert> the page http://maas.ubuntu.com/docs/install.html is (at least in part) out of date
<guntbert> it still tell me to initiate the import of the boot-images from the command line
<guntbert> *tells
<guntbert> whereas Julian Edwards says "Also, you should use the UI to import the pxe files, the script is there just as a backup. " in a comment on bug #1161927
<ubot5> bug 1161927 in MAAS "bogus missing boot images warning" [Undecided,Expired] https://launchpad.net/bugs/1161927
#maas 2014-09-29
<jtv> rvba, if you want to see my workaround for the BooleanField problem: https://code.launchpad.net/~jtv/maas/bug-1374388/+merge/236291
<rvba> jtv: okay, I'll review this right now.
<jtv> Thanks.
<jtv> rvba: I'm just pushing some improvements to the comment that explains it all.
<rvba> jtv: just approved the branch but I suggested an improvement that could help us generalize this fix, see my inline commentâ¦
<jtv> Great.
<jtv> rvba: how do I add a field to the UI form without also getting it as part of an API form submission?
<rvba> jtv: add a mixin on top of the form that is used in the UI.
<jtv> But what form is it that's used in the UI and not in the API?
<rvba> jtv: I'm thinking about adding a hierarchy of forms where the UI uses the base form (the one used by the API) + a mixin that add the 'ui-specific' hidden field.
<rvba> adds*
<rvba> jtv: does that make sense?
 * jtv cringes at the thought of yet more mixins
<jtv> I do see the argument against doing it from the templates though.
<rvba> We could probably avoid doing this with a mixinâ¦
<rvba> We could have this logic (add the 'ui-specific' field) in a base view class.
<rvba> i.e. our own version of Django's django.views.generic.edit.ProcessFormView
<jtv> How would we make use of that new view?
<jtv> I don't see any use of ProcessFormView in our node views at the moment... is a base class for some other class we use?
<rvba> Our views NodeListView, etc would derive from it instead of django.views.generic.edit.ProcessFormView.
<jtv> What I'm asking is: AFAICT the views for which we need this don't derive from ProcessFormView, so how does this help them?
<jtv> UpdateView.  That's the one we want, I think?
<rvba> The views for which we need this *do* derive from ProcessFormView, but indirectlyâ¦
<rvba> NodeView â UpdateView â ProcessFormView.
<jtv> How do you change the last step in that chain without a mixin?
<rvba> I know, this isn't possible without a mixin.
<jtv> Right.
<rvba> Either we change UpdateView (like you said) or we use a mixin.
<jtv> I think UpdateView would be fine â because the problem is specific to model forms.
<rvba> True.
<jtv> Can we just add a field there and have it show up in UI-originated form submissions?
<jtv> Or were you thinking to change get_form_class?
<rvba> Well, the view itself isn't the form.  It's in charge of building the form.
<rvba> Yeah, that's what I had in mind.
<Ditma> Hello, is someone here who can help me out with creating a custom boot image?
<jtv> Ditma: blake_r is probably your best bet right now, but he won't be in yet.
<Ditma> jtv: allright thanks. I think i will try to contact him later then.
<jtv> OK
<jtv> allenap: I did have a question about OS support that you might know answers to, actually... any chance of a quick chat?
<allenap> jtv: Sure.
<jtv> Thanks.
<jtv> allenap: conflict in your branch.  :(
<allenap> jtv: Ta, Iâll fix that.
<allenap> Done.
<Ditma> Is someone online who made some experiences with MAAS and link aggregation?
<rick_h_> hello all, I'm running into https://bugs.launchpad.net/maas/+bug/1350925 it appears and there's reference for trying the 'release candidate' is there a ppa or something I can get that from?
<ubot5> Ubuntu bug 1350925 in MAAS "Unable to get RPC connection for cluster 'maas'" [Critical,Incomplete]
#maas 2014-09-30
<rick_h_> so are preseeds a way I could suggestion someone use ansible with maas w/o juju?
<rick_h_> basically a way to run some custom cloud-init when the node is brought up
<thetrav> trying to learn MAAS. Is there a command to re-image a node?
<thetrav> everything seems to work properly to get one up and running, but then I do my configuration, break it horrible, and want to start again
<thetrav> s/horrible/horribly
<thetrav> maybe I can commission it with a different release, then switch it back
<thetrav> seems a bit long winded though
<thetrav> trying to get MAAS to re-image a machine... so far it seems to just be re-booting it with the existing OS in place.  Anyone know how to get it to start from scratch again?
<jtv> thetrav: if you just release it, it will be reinstalled on next boot.
<thetrav> how do I release it?
<thetrav> there's no button for that in the web ui :P
<thetrav> maas -h also doesn't lead me to any answers
<jtv> There should be, if it's currently allocated to you.
<jtv> Oh, you want to re-commission?
<thetrav> there is a "stop" button
<jtv> I fell into the middle of what you were saying earlier, so I'm missing some context.
<thetrav> I think what I was saying earlier is the same
<jtv> Ah yes, older version than what we're working with.  Sorry.  That "stop" button released the node.
<thetrav> or it was meant to be
<thetrav> I'm working with whatever is in the ubuntu package repo
<thetrav> I figured that would be "stable"
<jtv> We're developing the next version, and so we're more actively familiar with it.
<thetrav> right
<thetrav> makes sense
<thetrav> is there a way to do it in the old version?
<jtv> If you hit Stop, the node should then go into the Ready state.
<thetrav> yeah, it does do that
<jtv> Good.  Now, if you re-allocate and restart it again (the Start button in your version), it should reinstall.
<thetrav> re-allocate?
<thetrav> if I click the start button it just powers up the machine.  Still has the existing file system and ubuntu install
<jtv> Yeah.  There's two steps to deploying a machine, which are more clearly distinct in the next version: first you allocate the machine, then you fire it up.
<jtv> If it went through Ready and is now in Allocated state, it should _not_ have the same install...
<jtv> If this doesn't boot you back into the installer, I suspect it's just not netbooting.
<thetrav> right
<jtv> Question is, if it's not netbooting, how did it install before?
<thetrav> so... one of the things I noticed
<thetrav> in the preseed there's a line that says "turn off PXE netboot"
<jtv> Yeah, that happens at the end when the node's installed and deploying.
<thetrav> and when I go look at the bios in CIMC, PXE is not in the Actual Boot Order
<thetrav> is MAAS supposed to be modifying the boot order?
<thetrav> when it shuts down?
<jtv> When the node gets released, it should set it to netboot again.
<thetrav> right
<thetrav> so I think that's not happening right
<jtv> Sounds like.
<thetrav> what mechanism is it using to do that?
<jtv> IIRC it's a parameter to the power command: "come up, and when you do, netboot."
<thetrav> right, so in this case I've configured it to IPMI v2
<thetrav> I do recall seeing something about a bug with cisco integrated management controller and MAAS power management
 * thetrav searches
<thetrav> so it may be that I just have to manually set it to netboot whenever i restart
<jtv> That'd be annoying.
<thetrav> yep
<thetrav> something like this: https://bugs.launchpad.net/maas/+bug/1300476
<ubot5> Ubuntu bug 1300476 in maas (Ubuntu) "Unable to setup BMC/UCS user on Cisco B200 M3" [Critical,Fix released]
<jtv> (You _can_ restart a node that you currently own without MAAS' involvement, of course, and that won't need the change.  But still.)
<thetrav> although this is not a B200, it's a C240-M3S
<thetrav> yeah, so I told it to netboot using KVM and it's re-imaging now
<thetrav> would be nice if I didn't need KVM though
<jtv> I don't suppose this hardware supports the UCS power method?
<thetrav> I believe UCS requires expensive hardware that we haven't purchased
<thetrav> I don't really know though
<thetrav> I"m a software guy more than a hardware guy
<thetrav> this space is all pretty new to me
<thetrav> when you say UCS do you mean the cisco unified computing system dealy?
<thetrav> or is there another meaning for that achronym?
<jtv> I thinkthat's the one... Unified Computing and Servers?
<thetrav> yeah, so everything I've read and been told (admittedly by cisco sales guys) about that is that I can't use it without a 25k fabric interconnect
<thetrav> or maybe it's the fabric extender
<thetrav> point is, it's some seriously expensive fabric
<thetrav> for my budget of $0
<thetrav> that bug is not the issue btw
<thetrav> I checked and it has created a maas user for the IPMI stuff
<thetrav> it's just not adjusting the boot settings
<thetrav> at least on server shut down
<jtv> Yeah your problem is a different one from that bug.
<jtv> I'm not sure it should be adjusting on shutdown â I think it does that on power up.
<ram_> how to validate if maas is installed correctly on the server? - I do not get the URL <ServerIP>/maas working in my setup
<rick_h_> any maas folks around to help me get through this ssh problem. We're trying to qa updates to quickstart to enable maas support and having some fun
<roaksoax> rick_h_: how can we help you?
<rick_h_> roaksoax: I've got maas running on 3 nucs, we thought we had everything good but keep having issues with ssh and juju from the maas controller to the two nucs it's controlling
<rick_h_> roaksoax: I'm confused about how the amt/maas control stuff is meant to work so maybe things look setup right but aren't
<roaksoax> rick_h_: what are the issues?
<rick_h_> roaksoax: juju is unable to ssh https://bugs.launchpad.net/juju-core/+bug/1314682 it looks a lot like that bug
<ubot5> Ubuntu bug 1314682 in juju-core "Bootstrap fails because of virt-manager config" [High,Triaged]
<rick_h_> roaksoax: so I think my amt control isn't 100% correct
<rick_h_> roaksoax: so if I can fire a couple of questions maybe it'll lead to something
<rick_h_> roaksoax: on the amt node, I started out with it setup dhcp, but chnged it to static ip in an effort to make sure the node is always in the same place
<rick_h_> roaksoax: I entered that into the maas power settings
<roaksoax> rick_h_: did you add AMT credentials for each of the nodes and confirmed MAAS power's them on on a juju bootstrap?
<rick_h_> roaksoax: and when the machine is commissioned, it gets a different ip, is that ok? amt has 10.0.0.101 and comissioned one gets 10.0.0.250?
<rick_h_> roaksoax: well that's the thing. in maas it's 'start/stop' but that doesn't seem to really control power on or power off?
<roaksoax> rick_h_: what version of MAAS are you using?
<rick_h_> roaksoax: I moved to the daily ppa last night trying to work around a different issue
<rick_h_> 1.6.1+bzr2550+2551+295~ppa0~ubuntu14.04.1
<rick_h_> roaksoax: it's setup at maas.jujugui.org and happy to help give access if it helps in debugging
<roaksoax> rick_h_: can you test ppa:maas-maintainers/experimental ?
<rick_h_> roaksoax: sure thing
<roaksoax> rick_h_: who is giving IP Address ot AMT? I'd suggest you configure the IP manually for each AMT host and not on a range that MAAS manages
<rick_h_> roaksoax: right, that's what I've done. I've hard coded the amt ip now on both nucs
<rick_h_> roaksoax: and then maas gives the machine a dynamic space ip when it comissions
<roaksoax> rick_h_:yes that's fine. The static IP allocation means that MAAS pics an IP and assigns it to the node on *start*
<rick_h_> roaksoax: ok
<rick_h_> roaksoax: 1.7 installing now
<rick_h_> roaksoax: hmm, change to maas_local_settings.py in upgrade there removing all rabbitmq?
<rick_h_> roaksoax: sent diff in pm if that's expected?
<roaksoax> rick_h_: Y
<roaksoax> rick_h_: that's expeceted
<rick_h_> roaksoax: ok cool thanks for sanity check
<rick_h_> smaller one on pserv.yaml accepted as well
<roaksoax> rick_h_: i'll try to handle that automatically
<rick_h_> ok, new ui loaded :)
<rick_h_> "Boot image import process not started. Nodes will not be able to provision without boot images. Start the boot images import process to resolve this issue." warning
<roaksoax> rick_h_: ah yeah, og to the Images tab
<roaksoax> and import images again :)
<roaksoax> rick_h_: but the images should be there
<roaksoax> rick_h_: we just haven't migrated
<roaksoax> yet
<rick_h_> ah cool yea got it
<rick_h_> ok, see some more options on the nodes as well
<rick_h_> roaksoax: so stop == shutdown?
<roaksoax> rick_h_: yes! please files bugs if you thinkg that should be changed :)
<rick_h_> roaksoax: so I could not start/stop the node with the error that the config didn't allow it
<rick_h_> roaksoax: so I went in to edit the node and in the 'power type' I have a select list with no options
<rick_h_> roaksoax: so it seems I lost my power info on the node in the upgrade and have no valid types to choose from now
<roaksoax> rick_h_: so what if you do: sudo service apache2 restart && sudo service maas-cluster-register restart && sudo service maas-cluster restart
<roaksoax> rick_h_: wait a bit
<roaksoax> rick_h_: and check under the 'Clusters' tab to see if the cluster is connected
<rick_h_> roaksoax: running now
<roaksoax> rick_h_: once connected, you should get that back
<rick_h_> roaksoax: rgr
<roaksoax> rick_h_: did it work this time?
<rick_h_> roaksoax: so it grabbed some ips on the wrong network so editing the pserv.yaml and maas_cluster.conf to update those ips and restarting
<roaksoax> rick_h_: sudo dpkg-reconfigure maas-cluster-controller :)
<rick_h_> roaksoax: good to know
<rick_h_> roaksoax: ok, restarted and connected cluster now
<rick_h_> roaksoax: ok, so my power config is back and set
<roaksoax> rick_h_: ok great, let's try to test this time
<rick_h_> roaksoax: is the mac addr diff or the same from amt to the 'machine'? (I have them as the same but curious if I should be looking for a diff one)
<roaksoax> rick_h_: i think it might be the same
<rick_h_> machine is off, trying to start it brings up same error "The action "Start selected nodes" could not be performed on 1 node because its state does not allow that action.
<roaksoax> rick_h_: i cna't remember.. don't have a NUC in hand now unfortunately
<rick_h_> roaksoax: all good
<roaksoax> rick_h_: Allocate machine first
<roaksoax> rick_h_: and then you can start it
<rick_h_> allocate == commission?
<rick_h_> it's showing as status of ready atm
<rick_h_> ah, more details in the new edit details page
<rick_h_> Failed to query node's BMC â Node could not be queried node-ee9f70b4-48aa-11e4-8a8c-eca86bffcfed (nuc1) amt failed with return code 2: Missing amttool (amtterm package)
<roaksoax> rick_h_: commissioning is the stage where MAAS lears about the machine
<roaksoax> rick_h_: there you go, sudo apt-get install amtterm :)
<roaksoax> rick_h_: good thing that MAAS shows what's going on nowadays :)
<rick_h_> no kidding
<rick_h_> never would have realized it didn't already come with that stuff ootb
<roaksoax> rick_h_: that's why mark was happy in nuremberg :)
<rick_h_> oh yay color icon shows off now
<rick_h_> roaksoax: ok, I've got a power status and the button on the edit to 'check power status' shows it's off correctly
<rick_h_> roaksoax: but when I go to 'start' I get 'The action "Start selected nodes" could not be performed on 1 node because its state does not allow that action.
<rick_h_> '
<rick_h_> roaksoax: from a 'ready' state currently
<rick_h_> roaksoax: no new error in the edit view, just the same 6min old 'amtterm' missing
<roaksoax> rick_h_: yeah you need to *own* the machine first
<roaksoax> rick_h_: so *allocate* the machine first and then *start*
<rick_h_> roaksoax: allocate == commission or acquire or ?
 * rick_h_ isn't seeing a allocate button and is feeling a bit like a dumb user
<rick_h_> roaksoax: ok yea so the 'acquire' gave me the status of 'allocated' so I'd give user feedback of making the terms consistent
<roaksoax> rick_h_: agreed, please do files bugs
<rick_h_> ok, and now I can start and the machine is coming up woot!
<rick_h_> roaksoax: will do ty
<rick_h_> roaksoax: ok, more fun. So I managed to acquire and then start. Now I can't shut down or do anything else. It shows green started, and I'ge for tftp request node events
<roaksoax> rick_h_: what's the status of the node? Deploying? Deployed?
<rick_h_> roaksoax: I've tried to "Abort operatoin" and "stop node" and I get The action "Stop selected nodes" could not be performed on 1 node because its state does not allow that action.
<rick_h_> deploying
<roaksoax> rick_h_: ok, I think that's xpected since it is in the process of being deployed
<rick_h_> but that failed, the machine came up, got a tftp timeout, and now is sitting at the 'reboot or select proper boot device"
<roaksoax> rick_h_: humm so it never really started?
<rick_h_> roaksoax: correct
<rick_h_> it 'turned on' but never got rolling due to the tftp timeout
<roaksoax> rick_h_: did you add an SSH key? can you show output of /var/log/maas/maas.log and /var/log/maas/pserv.log
<rick_h_> roaksoax: https://pastebin.canonical.com/117948/ and https://pastebin.canonical.com/117948/ respectively
<rick_h_> roaksoax: yes, ssh key is added to maas
<rick_h_> roaksoax: tftp error was "PXE-E32: TFTP open timeout
<roaksoax> rick_h_: yeah I don't see a PXE boot here: https://pastebin.canonical.com/117949/
<roaksoax> rick_h_: is the cluster controller correctly configured to do DHCP/DNS?
<rick_h_> roaksoax: as far as I'm aware. it's connected, managed interfaces: 1, nodes 1, images synced, with "Manage" set to DHCP & DNS
<rick_h_> roaksoax: ok, I can release the node and now there's an error where amt disagreed that the machine was going to power off, yet it did
<rick_h_> https://pastebin.canonical.com/117950/ but it did occur so at least it's not tied up on the broken deploying now
<roaksoax> rick_h_: weird... maybe AMT issues?
<rick_h_> roaksoax: maybe, my first experience with it.
<roaksoax> rick_h_: the PXE boot issue i think it has been seen before but that's something we are investigating
<rick_h_> ok, had to release it twice but now it's actually released, back in a powered off 'black' state
<rick_h_> roaksoax: ok, cool. Well, this is farther than I was with better debug info. I'll add the second node and see if I can get it to work at all with juju now.
<rick_h_> roaksoax: thanks for the help and pointer to the experimental stuff.
<roaksoax> rick_h_: ok, great
<rick_h_> I'll file a few bugs on things based on the experience as feedback
<roaksoax> rick_h_: awesome! thanks
<rick_h_> roaksoax: can't add a my second node, does this look like something you've seen? https://pastebin.canonical.com/117958/
<roaksoax> rick_h_: interesting... it seems like it would be trying to pxe boot a node but it doesn't fine it
<roaksoax> rick_h_: can youplease file a bug and point it to me?
<rick_h_> roaksoax: will do
<rick_h_> roaksoax: that's in the add node UI after entering the info to add a new one
<roaksoax> rick_h_: ah yes, we spotted a bug when adding via webui
<roaksoax> rick_h_: it is in the process of being fixed
<rick_h_> roaksoax: ok cool
#maas 2014-10-01
<rick_h_> anyone know what action you can take on a node that 'releasing failed'? having a lack of success trying to figure out what to do
<jtv> rick_h_: that's a very recent addition... are there no actions you can take on the node, such as mark it as Broken?
<jtv> Thanks for the review allenap â and sorry for being difficult about yours.
<jtv> rvba: I'm landing a branch that changes the way we check for lint, and hopefully speeds it up.  Could you see what it does on your system?
<rvba> jtv: sure
<jtv> Thanks.
<jtv> It was getting out of hand.
<allenap> jtv: No worries, Iâll fix it :)
<rvba> jtv: yeah, big improvements with your change: it takes ~9s as opposed to ~[20s-25s]
<dimitern> jtv, hey
<frankban> hi maas devs, I am trying to set up maas for quickstart tests. I have two kind of problems currently: 1) nodes don't have access to the network (i.e. ping 8.8.8.8 fails) and 2) while I am able to "ssh <node address>", trying "ssh <node address> /bin/bash" hangs (and that's what juju would do when setting up the bootstrap node). any ideas?
<dimitern> jtv, so maas will bring up only the primary network interface on the machine by default?
<rick_h_> jtv: yea, that's what I ended up doing was broken/fixed and then got it going as ready
<jtv> Hi dimitern.  Yes, that's what it does at the moment isn't it?
<jtv> Actually I'm thinking to bring up "any network interface that's on a managed network."  (Or if there are none, "any network interface we have a MACAddress model object for.)
<jtv> Any reviewers up for an easy branch?  https://code.launchpad.net/~jtv/maas/clean-up-networking_preseed/+merge/236657
<dimitern> jtv, +1 I was thinking the same thing, if it's not too much more work perhaps
<jtv> Perfect.
<jtv> allenap: thanks for the review â I do agree with what you say and that's also why I used defaultdict initially.  But given the RPC boundary, I wanted to be fully in charge of the data structures and avoid any unnecessary differences across the RPC boundary.
<jtv> More easy reviewing: https://code.launchpad.net/~jtv/maas/identify-auto-interfaces/+merge/236688
<kindjal> Is there MaaS documentation describing how I can manually import boot images, for example, Lucid?
<kindjal> Can I just provide a kernel and an initrd?
<MilesDenver> I had heard there may be some CentOS images
<kindjal> I just canât figure out how to add images.
<kindjal> docs talk about pointing to âsources"...
<kindjal> but those look like something called a simplestream repo.
<MilesDenver> yea. I don't actually knowâ¦. but I remember Alessandro mentioning CentOSâ¦. searching on Google. isn't working for me.
<sarnold> kindjal: I think maas-import-pxe-files is the place to start looking
<kindjal> yes Iâve been reading about that...
<kindjal> you give it a source file with a url and  some other items...
<kindjal> but there is no specification of what the URL is supposed to host...
<kindjal> can I just point it at ubuntu cloud images?
<kindjal> http://cloud-images.ubuntu.com/?
<kindjal> or netboot images?
#maas 2014-10-02
<mscheel> Hi, I'm following http://maas.ubuntu.com/docs/install.html#pkg-install to get the 1.7 maas
<mscheel> I have already installed maas 1.5 (the server is Ubuntu 14.04), I'd like to upgrade
<mscheel> I do the add-apt-repository cloud-archive:tools but I get:
<mscheel> cloud-archive only supported on precise
<mscheel> (so for fun I installed a server on precise, but then it only gets maas 1.2)
<mscheel> Is there a way to get the 1.6 or 1.7 maas via packages?
<mscheel> (thanks in advance)
<mwhudson> mscheel: there must be a ppa somewhere
<mwhudson> although, is 1.7 out yet?
<jtv> Not as such.
<mwhudson> mscheel: https://launchpad.net/~maas-maintainers/+archive/ubuntu/dailybuilds ?
<jtv> Yeah, that should give you pre-release packages.
<rick_h_> mwhudson: I had to get the experimental to get 1.7
<rick_h_> vs dailybuilds, but then again I guess 1.7.0 is listed there
<rick_h_> https://launchpad.net/~maas-maintainers/+archive/ubuntu/experimental is a few revs ahead it looks like
<mscheel> mwhudson: thanks!
<mscheel> I didn't realize 1.7 wasn't out
<mscheel> I need the static IP support, which doesn't appear to be present in the 1.5 UI at least
<jtv> Yeah that's 1.7.
<jtv> If you upgrade, you'll also want to edit your cluster interface to have separate static & dynamic IP address ranges.
<mscheel> yeah we followed http://maas.ubuntu.com/docs/cluster-configuration.html
<mscheel> didn't realize that 1.5 was different
<mscheel> I don't think we can use daily/experimental/beta maas for the project I'm working on, unfortunately
<mscheel> btw, one of the things that tripped me up early in my experiments, was when I configured the cluster controller, I used 127.0.0.1 as the region controller address (or at least that's what I thought I was doing :)  - then when my first node started to come up, it just sits there, on a ci-info screen, never times out or prints that it is failing because it is trying to connect to 127.0.0.1
<mscheel> eventually I managed to get a screen shot of the boot process that showed that IP address as the controller it was connecting to, and just had to dpkg-reconfigure the cluster controller
<jtv> mscheel: thanks â I'll file a bug on the package, to make the question clearer.
<mscheel> thanks - I wasn't really sure where the "bug" was because the error was mine :)
<mscheel> I assume there's some reason the client node just sits waiting for a response
<jtv> Well if it can't reach the server, there isn't all that much we can make it do.
<mscheel> yeah not really
<mscheel> an error message would have been helpful, or a statement of what step it was on, maybe
<mscheel> "registering with maas cluster controller @ <ip: x.y.z.a>"
 * mscheel shrugs
<mscheel> if I'm the only one it happened to, not a big deal :)
<jtv> Then again, there could be any number of people running into it and not getting back to us.
<jtv> I see a small change that could be made: it's not just how the cluster controller reaches the API, it's also the basis for how the nodes reach the API.
<mscheel> ah, yeah, that makes sense
<jtv> allenap: if you look at lp:~jtv/maas/use-compose-curtin-network-preseed now and run src/maasserver/tests/test_preseed.py, you'll see that RPC failure I mentioned.
 * allenap looks
<jtv> allenap: found it!  The fake handler for that RPC method was hidden away in a mix-in.
<mgz> allenap: where are you?
<allenap> jtv: http://paste.ubuntu.com/8478304/ is how Iâd start.
<allenap> mgz: Gah. Iâm at home.
<jtv> allenap: thanks â I think I need this to pass by default though, because it affects semi-unrelated tests.
<mgz> allenap: :D
<allenap> mgz: Iâm not going to make it :-(
<mgz> allenap: ;_;
<mgz> you brussels though?
<allenap> mgz: Nope, last minute reprieve. Austin, yes.
<jtv> rvba: can you think of an easy way to add the region controller to the DNS mappings?
<jtv> So that I can test with a DNS server that resolves a hostname for the region controller in both IPv4 and IPv6?
<rvba> jtv: let me have a look at the codeâ¦
<jtv> Thanks.
<rvba> jtv: I don't really see a way other than pass the region ip â hostname info to DNSForwardZoneConfig in src/maasserver/dns/zonegenerator.py, the same way we pass dns_ip.
<jtv> rvba: that's probably not so bad, actually.  Thanks!
<rvba> np
<jtv> (In fact I've been wondering if the region shouldn't just provide a fixed hostname for itself in its own DNS...)
<rvba> blake_r: Hi Blake.  It seems src/maasserver/models/tests/test_bootsource.py:TestBootSource.test_calls_cache_boot_sources_on_create is flakyâ¦ any idea why (see http://paste.ubuntu.com/8479003/)?
<blake_r> rvba: no I will have to look into it
#maas 2014-10-03
<rvba> gmb: question for you: when I'm performing a bulk operation on two nodes that are both being powered up, instead of the expected summary message (about the fact that the 2 operations can't be performed), I get only one message about one node.
<rvba> gmb: is that expected?
<rvba> gmb: from the logs it seems there is only failure happening, the 'abort commissioning' operation fails with the first node and the second operation (aborting the second node) isn't even tried.
<gmb> rvba: I don't know about expected, but remember that under-the-hood, although Node.objects.start_nodes() accepts a collection of Nodes to start, it will get called thus:
<gmb> Node.objects.start_nodes([node_1])
<gmb> Node.objects.start_nodes([node_2])
<gmb> That may be why you're only getting one error.
<gmb> Though I don't know the entire context, so I don't know why you'd expect to see both.
<gmb> rvba: Ah, althoughâ¦ because it's called one at a time, and each one raises an error, and the error handling middleware adds a message and then redirectsâ¦ that's probably why you'd see only one message.
<rvba> gmb: I've done another experiment:
<rvba> I've got two nodes.  I commission one of them and waits until the power command is executed.
<rvba> wait*
<rvba> Then I commission another node and quickly after that (i.e. while the lock on the second node is still there) I try to abort the commissioning operation for the *2* nodes.
<rvba> gmb: guess what happened :)
<rvba> I got a message about the fact that it's impossible to cancel the commission for node 2â¦ and node 1 didn't get touched!
<gmb> rvba: Both aborts aborted?
<gmb> Oh!
<gmb> Haha
<gmb> Yeah, that doesn't actually surprise me. Again, see aboveâ¦ Your request hit an error, so *everything* in that transaction got aborted
<gmb> (i.e. both aborts aborted)
<rvba> Right.  I understandâ¦ but that's a bit surprising from a user's perspective.
<gmb> agreed
<gmb> rvba: This is a problem that allenap and I were aware of; that we're going to end up with these kind of something-should-be-unwound-something-shouldn't-be states. But we punted on it at the time because things were getting complex enough as it was.
<rvba> Ok.  I'll still file a bug about it.
<gmb> rvba: Please do!
<rvba> We should probably use subtransactions here.
<allenap> rvba: Yep.
<allenap> Agreed.
<rvba> gmb: allenap: actually, it's much worse than that: in my example, the first node (which was half way through commissioning) got shutdown (but was still left in the commissioning state because the DB transaction got rolledback)!
<rvba> gmb: allenap: this is because the power operation is not part of the transaction and didn't get rolled back :).
<gmb> Yep.
<allenap> rvba: For that I think we should commit the transaction before issuing the power command. It may be that commit-before-rpc â or commit-before-doing-it â should be the approach everywhere.
<gmb> allenap, rvba, jtv: Review for one of you, if you please: https://code.launchpad.net/~gmb/maas/add-alerts-for-disconnected-clusters-bug-1341121/+merge/237039
<jtv> I'll take it.
<gmb> Ta
<rvba> allenap: gmb:https://bugs.launchpad.net/maas/+bug/1377099
<ubot5> Ubuntu bug 1377099 in MAAS "Bulk operation leaves nodes in inconsistent state" [Critical,Triaged]
<rvba> gmb: testing your branch now
<gmb> rvba: Ta
<rvba> gmb: looks nice :)
<jtv> Thanks Gav.
<rvba> gmb: not a proper review but I added a couple of minor comments to your branch.
<gmb> k
<jtv> One reason to dislike IPv6: the digit is right in the dead zone on the keyboard.
<gmb> rvba, jtv: Thankee
<allenap> rvba: Iâve commented.
<gmb> rvba:Thanks for the painful bugs list :)
<rvba> gmb: welcome :)
<rvba> gmb: some of these bugs are not only painful but also painful to reproduce unfortunately.
<gmb> Yeah.
<gmb> rvba, allenap: Are either of you working on those bugs? If so, which? Need help?
<rvba> I assume allenap is still working on the RPC stuff.
<rvba> gmb: I've worked on 1376782 but it isn't fixed completely yet.
<rvba> gmb: it's one of the bugs that is really hard to reproduce.
<gmb> Right
<gmb> rvba: Can you be more specific on the bug about why the problem might still occur? (I ask because if I pick it up I'll need to refer to it later, and my memory is a sieve today due to jet lag)
 * gmb -> jetlag break; bbiab
<rvba> blake_r: btw, great job on the import boot images JS, I've tested it and it's great!
<blake_r> rvba: thanks
<blake_r> rvba: glad you like it
<blake_r> rvba: only works for Ubuntu images, if you want to extend a FFe I can get the others as well, ;-)
#maas 2015-09-28
<mup> Bug #1500454 opened: There is not API endpoint for MAAS keys <MAAS:New> <https://launchpad.net/bugs/1500454>
<mup> Bug #1500454 changed: There is not API endpoint for MAAS keys <MAAS:New> <https://launchpad.net/bugs/1500454>
<mup> Bug #1500454 opened: There is not API endpoint for MAAS keys <MAAS:New> <https://launchpad.net/bugs/1500454>
<binoy> http://askubuntu.com/questions/660580/post-request-using-python-maas-client/679311#679311
#maas 2015-09-29
<mup> Bug #1500678 opened: maas allows use of the maas controllers IP in dhcp ranges <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1500678>
<mup> Bug #1500683 opened: Changing MaaS default behaviour for DNSSEC <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1500683>
<mup> Bug #1500683 changed: Changing MaaS default behaviour for DNSSEC <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1500683>
<mup> Bug #1500683 opened: Changing MaaS default behaviour for DNSSEC <canonical-bootstack> <MAAS:New> <https://launchpad.net/bugs/1500683>
<mup> Bug #1500683 changed: Changing MaaS default behaviour for DNSSEC <canonical-bootstack> <MAAS:Opinion> <https://launchpad.net/bugs/1500683>
<mup> Bug #1500683 opened: Changing MaaS default behaviour for DNSSEC <canonical-bootstack> <MAAS:Opinion> <https://launchpad.net/bugs/1500683>
<mup> Bug #1500683 changed: DNSSEC should be disabled in MAAS by default <canonical-bootstack> <MAAS:Opinion> <https://launchpad.net/bugs/1500683>
<binoy> how to install latest version of maas
<binoy> i want to install maas 1.8 atleat, how can i do that
<binoy> https://anzweb.wordpress.com/2015/07/07/installing-maas-1-8-on-ubuntu-14-04-lts/
<mup> Bug #1499869 changed: maas wily deployment to HP Proliant m400 arm64 server cartridge fails <cloud-init:Confirmed for smoser> <curtin:New> <cloud-init (Ubuntu):New> <linux (Ubuntu):Fix Committed by timg-tpi> <linux (Ubuntu Vivid):In Progress by timg-tpi> <cloud-init (Ubuntu Wily):New> <linux
<mup> (Ubuntu Wily):Fix Committed by timg-tpi> <https://launchpad.net/bugs/1499869>
#maas 2015-09-30
<mup> Bug #1501112 opened: Node Networking - Users can edit network info while commissioning <MAAS:New for blake-rouse> <https://launchpad.net/bugs/1501112>
<mup> Bug #1501135 opened: maas-import-pxe-files fails when MAAS_URL is quoted <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Trusty):New> <https://launchpad.net/bugs/1501135>
<mup> Bug #1501135 opened: maas-import-pxe-files fails when MAAS_URL is quoted <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Trusty):New> <https://launchpad.net/bugs/1501135>
<mup> Bug #1501135 changed: maas-import-pxe-files fails when MAAS_URL is quoted <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Trusty):New> <https://launchpad.net/bugs/1501135>
<mup> Bug #1501135 opened: maas-import-pxe-files fails when MAAS_URL is quoted <MAAS:New> <maas (Ubuntu):New> <maas (Ubuntu Trusty):New> <https://launchpad.net/bugs/1501135>
<mup> Bug #1393616 changed: Juju destroy-environment --force left unit in deployed state <landscape> <MAAS:Expired> <https://launchpad.net/bugs/1393616>
<mup> Bug #1462136 changed: 1.8rc2: One out of 4 regions is not connected to the cluster <oil> <MAAS:Expired> <https://launchpad.net/bugs/1462136>
<binoy> https://answers.launchpad.net/ubuntu/+source/maas/+question/271919
<mup> Bug #1501180 opened: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501180 changed: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501180 opened: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501180 changed: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501180 opened: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501400 opened: set-boot-disk yields in a machine not being able to deploy <MAAS:New> <https://launchpad.net/bugs/1501400>
<mup> Bug #1501404 opened: Node networks detail - default values <MAAS:New> <https://launchpad.net/bugs/1501404>
<mup> Bug #1501180 changed: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501180 opened: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<mup> Bug #1501180 changed: how we do post in maas-api (1.8.2) <MAAS:New> <https://launchpad.net/bugs/1501180>
<Slugs_> my region contoller states that is not connected, im not sure how to troubleshoot this
<Slugs_> if i change the IP address of the region controller im assumiing i would need to edit a maas configuration file to match the new ip
<Slugs_> not sure where to find that
<Slugs_> got it
<Slugs_> i had to reconfigure the maas-region-controller package
<Slugs_> put the correct ip
<mup> Bug #1501527 opened: Adding a node with a hostname in the CLI, automatically creates a different hostname <MAAS:New> <https://launchpad.net/bugs/1501527>
#maas 2015-10-01
<mup> Bug #1501605 opened: Upgrade to 1.9 turned a device into a node <MAAS:New> <https://launchpad.net/bugs/1501605>
<mup> Bug #1501613 opened: Tag explosion on upgrade to 1.9a2 <MAAS:New> <https://launchpad.net/bugs/1501613>
<mup> Bug #1501753 opened: test_dehydrate_partitions_returns_list_of_partitions fails spuriously <tests> <MAAS:Triaged> <https://launchpad.net/bugs/1501753>
<mup> Bug #1501753 changed: test_dehydrate_partitions_returns_list_of_partitions fails spuriously <tests> <MAAS:Triaged> <https://launchpad.net/bugs/1501753>
<mup> Bug #1501753 opened: test_dehydrate_partitions_returns_list_of_partitions fails spuriously <tests> <MAAS:Triaged> <https://launchpad.net/bugs/1501753>
<mup> Bug #1501607 opened: Bootstrap failed on MAAS 1.9 <cloud-install-failure> <MAAS:New> <https://launchpad.net/bugs/1501607>
<mup> Bug #1501785 opened:  Cannot resolve keyword 'block_device' into field <MAAS:New> <https://launchpad.net/bugs/1501785>
<mup> Bug #1501607 changed: Bootstrap failed on MAAS 1.9 <cloud-install-failure> <MAAS:New> <https://launchpad.net/bugs/1501607>
<mup> Bug #1481091 changed: MaaS Install guide incorrectly refers to 12.04 cloud PPA <MAAS:Fix Released by andreserl> <https://launchpad.net/bugs/1481091>
<mup> Bug #1498601 changed: Errors that occur during startup inside "yield deferToThread(inner_start_up)" are hidden <support> <MAAS:Triaged> <https://launchpad.net/bugs/1498601>
<mup> Bug #1501527 changed: Adding a node with a hostname in the CLI, automatically creates a different hostname <MAAS:Invalid> <https://launchpad.net/bugs/1501527>
#maas 2015-10-02
<mup> Bug #1501978 opened: Too many logs when posting to the status API <MAAS:New> <https://launchpad.net/bugs/1501978>
<mup> Bug #1501978 changed: Too many logs when posting to the status API <MAAS:New> <https://launchpad.net/bugs/1501978>
<mup> Bug #1501978 opened: Too many logs when posting to the status API <MAAS:New> <https://launchpad.net/bugs/1501978>
<mup> Bug #1501978 changed: Too many logs when posting to the status API <MAAS:New> <https://launchpad.net/bugs/1501978>
<mup> Bug #1501978 opened: Too many logs when posting to the status API <MAAS:New> <https://launchpad.net/bugs/1501978>
<mup> Bug #1501982 opened: Network's filter doesn't show networks. <MAAS:Triaged> <https://launchpad.net/bugs/1501982>
<mup> Bug #1502253 opened: Partition table should be deleted after all partitions have been removed <storage> <MAAS:New> <https://launchpad.net/bugs/1502253>
<mup> Bug #1502253 changed: Partition table should be deleted after all partitions have been removed <storage> <MAAS:New> <https://launchpad.net/bugs/1502253>
<mup> Bug #1502253 opened: Partition table should be deleted after all partitions have been removed <storage> <MAAS:New> <https://launchpad.net/bugs/1502253>
<mup> Bug #1502259 opened: default install failed in lvcreate <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1502259>
<mup> Bug #1496469 changed: MAAS still uses 3.13 kernel which does not work on power <blocks-hwcert-server> <MAAS:Confirmed> <https://launchpad.net/bugs/1496469>
<mup> Bug #1502333 opened: MAAS allows multiple block devices and partitions to be mounted at the same mount point <storage> <MAAS:New> <https://launchpad.net/bugs/1502333>
<mup> Bug #1502333 changed: MAAS allows multiple block devices and partitions to be mounted at the same mount point <storage> <MAAS:New> <https://launchpad.net/bugs/1502333>
<mup> Bug #1496469 opened: MAAS still uses 3.13 kernel which does not work on power <blocks-hwcert-server> <MAAS:Confirmed> <https://launchpad.net/bugs/1496469>
<mup> Bug #1496469 changed: MAAS still uses 3.13 kernel which does not work on power <blocks-hwcert-server> <MAAS:Confirmed> <https://launchpad.net/bugs/1496469>
<mup> Bug #1502333 opened: MAAS allows multiple block devices and partitions to be mounted at the same mount point <storage> <MAAS:New> <https://launchpad.net/bugs/1502333>
#maas 2015-10-03
<mup> Bug #1502360 opened: Editing the filesystem on partition does not trigger a node update <storage> <MAAS:Triaged> <https://launchpad.net/bugs/1502360>
<mup> Bug #1502435 opened: Clicking "Edit" node details and clicking "check now" for power status in the ui 1.9.0 (alpha3+bzr4345) results in redirection to http://xx.xx.xx.xx/MAAS/#/nodes <1.9.0> <alpha3> <ui> <web> <MAAS:New> <https://launchpad.net/bugs/1502435>
<JZTech101> I tried to reinstall maas today
<JZTech101> I got this:
<JZTech101> https://gist.github.com/anonymous/f42af9dba6bd012026db
<JZTech101> help?
#maas 2015-10-04
<mup> Bug #1502566 opened: Rack ID for nodes <suggestions> <MAAS:New> <https://launchpad.net/bugs/1502566>
<mup> Bug #1502566 changed: Rack ID for nodes <suggestions> <MAAS:New> <https://launchpad.net/bugs/1502566>
<mup> Bug #1502566 opened: Rack ID for nodes <suggestions> <MAAS:New> <https://launchpad.net/bugs/1502566>
<mup> Bug #1502566 changed: Rack ID for nodes <suggestions> <MAAS:New> <https://launchpad.net/bugs/1502566>
<mup> Bug #1502566 opened: Rack ID for nodes <suggestions> <MAAS:New> <https://launchpad.net/bugs/1502566>
#maas 2016-10-03
<batkins61> But 2.0 doesn't provide for static IP address assignment, other than snippits, correct?  I require that, because I have to slot these servers into specific addresses.
<roaksoax> batkins61: 1.9+ allos you to xonfigure the interface in 3 ways "auto-assign" which is mAAS picks an IP for you automatically and writes e/n/i for it
<roaksoax> batkins61: "static" you can change the IP to what you want (and MAAS writes e/n/i)
<roaksoax> batkins61: or DHCP, which means what ever ip address you can get
<batkins61> When I tried to select "static", the menu refused to stick, and always reset to Unconfigured
<roaksoax> batkins61: check the logs in /var/log/maas/*.log and see if there is an error
<roaksoax> batkins61: when you are trying to change it as static
<batkins61> After commissioning, the interface is set to auto assign.  I tried to change it to static, but it ignores that.  Setting to DHCP with snippits works, and the nodes commission and deploy
<batkins61> I can ICMP ping them, but they refused connect on 22.
<roaksoax> batkins61: i have not tested what you've done, but 'static' should work, and if it doesn't there must be an error going on
<roaksoax> batkins61: it you could please look at /var/log/maas/*.log while setting the interface as static
<roaksoax> batkins61: and if you could file  abug
<batkins61> Ok, let me try again.
<roaksoax> even better
<roaksoax> so we can fix it
<roaksoax> batkins61: did it work ?
<pmatulis> roaksoax, hi, can we get the topic in this channel to point to the new doc site?
<mup> Bug # changed: 1052870, 1237723, 1253940, 1272018, 1301833, 1304613, 1306372, 1308802, 1309593, 1311119, 1313580, 1313723, 1315071, 1317677, 1317978, 1326827,
<mup> 1330762, 1340254, 1344125, 1350459, 1351247, 1352938, 1361673, 1361773, 1376395, 1386482, 1389033, 1389809, 1389852, 1392378, 1459441, 1462635, 1623598
* roaksoax changed the topic of #maas to: World's best bare-metal provisioning tool | Docs: http://maas.io/docs | Mailing list: https://launchpad.net/~maas-devel
<mup> Bug # changed: 1379155, 1379636, 1390145, 1430129, 1430324, 1437221
<mup> Bug # opened: 1379155, 1379636, 1390145, 1430129, 1430324, 1437221
<mup> Bug # changed: 1379155, 1379636, 1390145, 1430129, 1430324, 1437221
<mup> Bug #1629868 opened: times out because of no dbus <maas-ipv6> <cloud-init (Ubuntu):New> <maas (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<mup> Bug #1629896 opened: [2.1] Deployment defaulting to hwe-16.04 instead of ga-16.04 <MAAS:New> <https://launchpad.net/bugs/1629896>
<mup> Bug #1629896 changed: [2.1] Deployment defaulting to hwe-16.04 instead of ga-16.04 <MAAS:New> <https://launchpad.net/bugs/1629896>
<mup> Bug #1629896 opened: [2.1] Deployment defaulting to hwe-16.04 instead of ga-16.04 <MAAS:New> <https://launchpad.net/bugs/1629896>
<mup> Bug #1629909 opened: [FFE] New upstream release MAAS 2.1 <maas (Ubuntu):New> <https://launchpad.net/bugs/1629909>
<mup> Bug #1629915 opened: Specifying the default gateway in an interface different than the provisioning one does not work <MAAS:New> <https://launchpad.net/bugs/1629915>
<mup> Bug #1629909 opened: [FFE] New upstream release MAAS 2.1 <maas (Ubuntu):New> <https://launchpad.net/bugs/1629909>
<ybaumy> hi. im using 2.1.0~alpha4+bzr5397-0ubuntu1~16.04.1 and now im having connection issues. the hosts are getting dhcp ipv4 addresses but then... http://imgur.com/a/7yWBT
<ybaumy> and the commissioning fails
<ybaumy> seems like the nodes are trying to use ipv6 addresses
<ybaumy> 2016-10-03 18:07:42 [RegionServer,1,::ffff:192.168.10.1] Successfully configured DHCPv4 on rack controller '4y3h7n'.
<ybaumy> 2016-10-03 18:07:42 [RegionServer,1,::ffff:192.168.10.1] Successfully configured DHCPv6 on rack controller '4y3h7n'
<pmatulis> ybaumy, i'm getting something similar. commissioning fails due to DHCP
<baldpope> ybaumy, does the network your handing out dhcpd leases from allow traffic through firewall to internet?
<ybaumy> yes it does
<ybaumy> the nat is configured on the ASA
<ybaumy> at least on the maas server its working
<pmatulis> ybaumy, whoah, weird. after rebooting my maas yet again my node has enlisted. that wasn't happening before
<ybaumy> hmm i dont know whats going on. how do i disable ipv6 completly. im not using it anyways
<pmatulis> ybaumy, no idea. but commissioning seems to be proceeding as normal now (for me)
<ybaumy> hmm maybe i know what the problem is. my dns servers changed and the proxy log shows still the old ones. which file is used for the dns servers in maas
<pmatulis> ybaumy, i noticed when i was in a broken state the logs where showing that the rackd and regiond had lost connectitivy
<pmatulis> this was also reflected in the web UI
<pmatulis> yup, my node is in the 'Ready' status now
 * baldpope sigh
<baldpope> is there a current doc for deploying openstack/autopilot ?
<baldpope> the one listed on ubuntu's site appears out of date with the current UI
<HeOS> Hello! Could you say me, how can I debug a situation, when maas can't connect to SuperMicro's BMC?
<HeOS> I see, that mass got an IP-address of the BMC, but it can't operate by power actions on the server.
<HeOS> I've tired to run ipmitool on the controller, it works, I can connect to the server and operate it, but MAAS can't.
<mup> Bug #1629868 opened: times out because of no dbus <maas-ipv6> <MAAS:New> <cloud-init (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<mup> Bug #1629868 changed: times out because of no dbus <maas-ipv6> <MAAS:New> <cloud-init (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<kiko> HeOS, that's odd -- what happens when MAAS tries to connect?
<HeOS> kiko, when I add my credentials there, then I see the following error: "Connection timed out while performing power action. Check BMC configuration and connectivity and try again. "
<HeOS> I don't see nothing in the maas' logs.
<kiko> HeOS, I guess the first question I have is why hasn't auto-enlistment work for you?
<HeOS> auto-enlistment is working, I see credentials there, but maas can't power on, for example, my server. So, my question is the following: how can I figure out why it's happened?
<HeOS> kiko, I don't know, what I should to do. :(
<kiko> HeOS, is there an IP address problem on the rack/cluster controller?
<kiko> i.e. is it using the wrong IP address to connect?
<kiko> HeOS, there will be something in /var/logs/*
<HeOS> kiko, I can connect to the IPMI's ip-address from the controller by using ipmitool, it works.
<kiko> HeOS, the question is how does it fail?
<kiko> and the logs can tell us
<HeOS> kiko, I'm using the following version: 2.0.0+bzr5189-0ubuntu1.
<HeOS> kiko, what a kind of logs can help?
<mup> Bug #1629868 changed: times out because of no dbus <maas-ipv6> <MAAS:New> <cloud-init (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<kiko> HeOS, /var/log/maas/* on the controller should help
<HeOS> There are two files: maas.log rackd.log.
<kiko> yep, look through those when attempting to boot the machine
<kiko> i.e. sudo tail -f /var/log/maas/*
<mup> Bug #1629868 opened: times out because of no dbus <maas-ipv6> <MAAS:New> <cloud-init (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<HeOS> I'm going to repeat my actions right now to get all related information from the logs. :)
<kiko> thanks!
<HeOS> kiko, I see only this files changed: http://paste.openstack.org/show/584061/.
<kiko> HeOS, what did tail -f show when you clicked the button?
<kiko> that file isn't very useful
<kiko> roaksoax, blake_r, mpontillo: any hint as to what could be wrong above?
<mup> Bug #1629868 changed: times out because of no dbus <maas-ipv6> <MAAS:New> <cloud-init (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<mup> Bug #1629868 opened: times out because of no dbus <maas-ipv6> <MAAS:New> <cloud-init (Ubuntu):New> <https://launchpad.net/bugs/1629868>
<HeOS> kiko, what a button should I click?
<HeOS> Do you mean the "Save" button in the IPMI-related config?
<HeOS> No new information in the logs.
<HeOS> I only see "Power error check now" message in the MAAS UI.
<kiko> that's what we want
<HeOS> kiko, can I enable more informative log level for the tool?
<kiko> HeOS, you see nothing in the logs?
<kiko> that's so strange
<blake_r> HeOS: MAAS uses ipmipower to power control the machine, try that versus using ipmitool
<HeOS> kiko, yes, I'm.
<HeOS> BMC-related information: http://paste.openstack.org/show/584065/.
<HeOS> blake_r, kiko, thanks, looks like ipmipower or BMC related problem. I'll try to inve
<HeOS> *investigte it.
<kiko> HeOS, thanks -- it's odd
<baldpope> HeOS, i'm using supermicro mobo with maas - power control works fine - what's going on
<baldpope> kiko, is there a newer doc for deploying openstack via maas/autopilot?
<kiko> baldpope, it's called conjure-up :-) let me find a doc
<kiko> http://conjure-up.io/
<baldpope> kiko, looks cool
<kiko> baldpope, however, note the "preview release"
<baldpope> does this replace the work with juju?
<kiko> well, it drives juju
<kiko> but through a very nice front-end
<baldpope> my mistake
<baldpope> do you still need to go through the prep with maas - re: discover/commission of the new blades?
<kiko> baldpope, yep
<kiko> the entire stack depends on MAAS doing what it needs to
<baldpope> k - stupid question - i wiped and re-installed my maas box - i noticed on the 16.04 install, an option for maas / rack controller install option
<baldpope> versus the standard ubuntu install
<baldpope> if the end results is intended to be a maas box, should I pre-select the appropriate install, or am I fine with installing via apt - post system install
<kiko> baldpope, it doesn't really make that much difference.
<kiko> baldpope, the maas install just leaves the machine with maas preinstalled
<mup> Bug #1629972 opened: networking stop incorrectly disconnects from (network) root filesystem <maas-ipv6> <MAAS:New> <ifupdown (Ubuntu):New for lamont> <ifupdown (Ubuntu Xenial):New> <https://launchpad.net/bugs/1629972>
<HeOS> baldpope, looks like, I have old version of the IPMI's firmware.
<baldpope> HeOS, i had firmware issues with ikvm - supermicro support was reasonable in getting me an updated one
<HeOS> Yay, it works, when I use the following command: ipmipower --workaround-flags=nochecksumcheck,noauthcodecheck -h <host> -u <user> -p <pass>
<mup> Bug #1629982 opened: [2.1] After upgrade from 2.0 to 2.1, iscsi cannot be updated. <MAAS:New> <https://launchpad.net/bugs/1629982>
<baldpope> yo kiko , for maas-oath - should use the api key?
<kiko> baldpope, I think the answer is yes?
<baldpope> you guys are killing me with being unable to paste the api key
<kiko> baldpope, what do you mean?
<kiko> baldpope, the API key in the MAAS web UI can't be pasted?
<baldpope>  copy the api key from web ui, attempt to paste in conjure-up (also through openstack-install) , paste no work
<baldpope> quit conjure, and I can paste in the putty terminal
<baldpope> actually, maas-oauth must be looking for something different, because conjure doesn't prompt for a username, just the maas-server and maas-oauth
<stokachu> baldpope: did you try shift+insert
<stokachu> to paste in putty
<baldpope> i did not stokachu , i'll try that
<kiko> stokachu, baldpope has a point though, does the token include the user information in it?
<stokachu> its just the api key you get from your accounts page
<stokachu> you dont enter a username or anything
<baldpope> http://imgur.com/W4TNZF1
<baldpope> ok, i'll try again
<kiko> so the key is globally unique
<kiko> ok :)
<baldpope> shift insert worked, ty stokachu
<baldpope> that shit was driving me nutz
<stokachu> yea shift+insert is not well known for some reason
<stokachu> but it's what i always use
<baldpope> i've always just used right click
<stokachu> baldpope: yea that won't work with the ncurses stuff we're doing
<baldpope> appreciate the tip - you know how many times I've had to type that api key out
<stokachu> yea :( i know your pain
<baldpope> almost enough to make me ditch and go to virtual pc ;)
<baldpope> or hyper-v
<baldpope> hah
<baldpope> ok, after putting in maas server ip, and api key and clicking add credential, i get an error - unable to find home/user/.local/share/juju/accounts.yaml
<baldpope> http://imgur.com/93nzldG
<baldpope> odd, i removed the log file and ran again, worked this time - odd...
<baldpope> i'm at deploying openstack-base
<baldpope> also, not sure if it's supposed to have or not, but the public key tied to my user in maas did not got applied to the ubuntu account during the initial deployment when running conjure-up
<baldpope> the assumption is that it should have?
<cmart> Does anyone know how to get MAAS to release an IP address which was previously statically auto-assigned to a node, but that node has been deleted?
<roaksoax> cmart: yes
<roaksoax> cmart: hwen you delete a node it should delete all ip addresses owned by that node
<cmart> I don't think that happened. My Subnet view in the web interface shows that the IP address is owned by "MAAS". Usage "Unknown", Interface "Unknown", Type "Static". I'm looking for a way to remove this entry in both the GUI and the CLI, and coming up short.
<cmart> I'm only trying to re-use the IP address. MAAS won't let me statically assign it to a node, says that it conflicts with an existing IP.
<cmart> roaksoax: "IP address is already in use."
<roaksoax> cmart: that seems like a bug
<roaksoax> cmart: can you please file a bug and point me to it ?
<PCdude> hey all :)
<PCdude> I have some questions about openstack, to prevent myself from spamming this IRC channel, I have put it in an askubuntu question
<PCdude> http://askubuntu.com/questions/832736/openstack-with-autopilot-some-networking-clear-up
<PCdude> Some are MAAS specific therefore I post it here
<mup> Bug #1630034 opened: Cannot re-use static IP address <maas-cli> <ui> <MAAS:New> <https://launchpad.net/bugs/1630034>
<cmart> ^ roaksoax ^ as requested :) Thanks so much for taking a look.
<cmart> roaksoax I pasted the output of a couple SQL queries as a comment. I'm thinking of just deleting the row in maasserver_staticipaddress
<roaksoax> cmart: thank you for filing, we will inviestigate that
<cmart> roaksoax: more detail in the comments, but I worked around/resolved the issue for myself
<roaksoax> cmart: great! thank you!
<cmart> by deleting the static IP from the database. it's not a great solution (doesn't identify the root cause of how the address got in that state), but got me past the problem.
<roaksoax> cmart: you could have tried doing it via the webui :)
<cmart> I did, there was no way that I could find to delete it. I tried all I could in the web UI and API before touching the DB directly.
#maas 2016-10-04
<mup> Bug #1630123 opened: OpenStack base 45 not being deployed with Juju GUI <MAAS:New> <https://launchpad.net/bugs/1630123>
<PCdude> Hi all
<PCdude> I have some questions about openstack on ubuntu
<PCdude> I have put it in an askubuntu question
<PCdude> http://askubuntu.com/questions/832736/openstack-with-autopilot-some-networking-clear-up
<PCdude> Some about them are MAAS related
<Sujeet_> Hi Kiko, Hi Roaksoax
<mup> Bug #1630123 changed: OpenStack base 45 not being deployed with Juju GUI <juju-core:New> <juju-gui:New> <https://launchpad.net/bugs/1630123>
<baldpope> good morning kiko roaksoax
<baldpope> fyi - ran through the conjure at about 4pm yesterday, 13hrs or so and it's just sitting idle, no progress
<baldpope> i'll dig into it more this morning
<stokachu> baldpope: what do you mean sitting idle?
<stokachu> at what point?
<baldpope> finished inputting in maas server ip, and the api key
<stokachu> baldpope: are you seeing machines being deployed in maas?
<baldpope> then the next screen shows which modules will be selected, i presumed the default would be sufficient, so left untouched
<baldpope> of the 5 nodes, 1 is deployed (previously in the ready state) the other 4 are sitting in the idle state
<baldpope> the deployed host is online, and ssh is on, but I cannot ssh into it - the ssh key I added is not accepted when I attempt to connect as ubuntu
<baldpope> ssh key I added to the webui
<stokachu> it's probably using the ssh key generated by juju
<stokachu> you can do juju switch controller;juju ssh 0
<baldpope> from the maas host?
<stokachu> yea
<stokachu> well wherver you ran conjure-up
<baldpope> right
<baldpope> sysadmin@ubuntu-ap-brk:~/.local/share/juju/ssh$ juju switch controller
<baldpope> finch:admin@local/conjure-up -> finch:admin@local/controller
<baldpope> sysadmin@ubuntu-ap-brk:~/.local/share/juju/ssh$ juju ssh 0
<baldpope> ERROR no API addresses
<stokachu> your bootstrap failed then
<stokachu> what does juju models show
<baldpope> cannot list models - no api addresses
<stokachu> yea your bootstrap failed
<stokachu> are you able to deploy a node via maas ui?
<baldpope> i believe so,
<baldpope> 1sec
<baldpope> yea, was able to deploy a new bloade with ubuntu 16.04 lts
<baldpope> and i can login with the ssh key assigned in webui
<stokachu> can you access the internet from that node?
<stokachu> also what version of juju are you running
<baldpope> 2.0-rc2-xenial-amd64
<baldpope> apt install worked (presumably through maas controller) but running lynx www.google.com fails
<stokachu> and can you access the internet from that node? or run a `sudo apt update`
<stokachu> yea
<stokachu> your network isn't configured properly
<baldpope> apt update worked
<stokachu> thats from the maas proxy
<stokachu> do you have IP forwarding enabled on the maas server?
<baldpope> am I mistaken, but I thought maas also acted as squid proxy ?
<baldpope> stokachu, shit .. i'll bet not
<stokachu> yea assuming you setup the network config on the maas server properly :)
<stokachu> you also want to NAT that traffic
<stokachu> so you'll need to add that rule in iptables
<baldpope> i have a forward rule, but I don't have any nat rules
<baldpope> sigh .. did I miss a step somewhere, thought I followed closely
<stokachu> /docs/en/users/#customize-headless-mode
<stokachu> err
<stokachu> http://paste.ubuntu.com/23275006/
<stokachu> baldpope: thats what i use for my private network
<stokachu> just change that to whatever network you're using
<baldpope> that's in /etc/iptables/rules.save or something?
<stokachu> http://paste.ubuntu.com/23275007/
<stokachu> see the pre-up line
<stokachu> you can add it there
<stokachu> or save it
<stokachu> do all of your nodes have 2 nics?
<baldpope> yea
<stokachu> all eth0/eth1?
<baldpope> well, 6
<baldpope> enp9s0f0 and enp9s0f1
<stokachu> ok so when you get to the application list make sure to configure your neutron br-ext to be enp9s0f1
<stokachu> it defaults to eth1
<baldpope> the other 4 not currently plugged in
<stokachu> i would plug those in because you can't select the machine you want to use for neutron
<baldpope> the nics, you mean?
<stokachu> yea
<stokachu> well the machine that is housing neutron
<stokachu> you can't specifically select that machine, juju will grab one at random
<baldpope> stokachu, (or anyone else) if I've got the masq rule in place, do I also need to be running squid, or will it just forward the traffic?
<baldpope> wait..
<baldpope> ok, i may not have setup networking correctly, the 'internal' network I created is routable through firewall - is not required to go through maas controller
<baldpope> and in this case, the individual nodes do not have access through firewall out -
<baldpope> so the question I have now is - should the nodes be required to go through the maas controller, or is it ok for them to have direct access out?
<stokachu> the easiest solution is to route everything through your maas server
<baldpope> hm
<baldpope> that's interesting
<baldpope> i would think I would want to use maas for dpeloyment, but not necessarily to act as the end-all routing uplink..
<stokachu> it doesn't have to be
<stokachu> thats just the easiest solution
<baldpope> if that's the case, more care should be taken on the maas controller to use a bridge interface with sub interfaces for redundancy
<baldpope> so eth0 and eth1 in bridge with br0.1 as the wan and br0.2 as lan?
<baldpope> along with any other sub interfaces as required
<stokachu> yea that will work too
<baldpope> hm
<baldpope> ok, not going to mess with that just yet
<baldpope> I can update the firewall to allow my private segment out
<baldpope> but I'm not sure how that resolves the juju deploy?
<stokachu> because juju needs to resolve things like streams.canonical.com:443
<baldpope> ah
<baldpope> i thought that was being done from the maas controller
<baldpope> my mistake
<baldpope> in my 5 node environment, after the juju controller has already been deployed, am I limited to using the remaining 4 for compute, so the 1 head is lost?
<stokachu> yea unfortunately
<baldpope> hm
<stokachu> juju 2 requires a node for the controller
<baldpope> well, that by itself isn't terrible
<baldpope> if you plan ahead, you can pick a host that would suffice for juju, but might not be the ideal compute/storage node
<baldpope> that's what I thought I was deploying on the maas box (an older dell poweredge)
<stokachu> so you can select which machine to perform the bootstrap on
<baldpope> well - not exactly
<stokachu> JUJU_BOOTSTRAP_TO=host.maas conjure-up -d openstack
<stokachu> ?
<baldpope> ah
<baldpope> sorry - thought you were asking me a question
<stokachu> this only works for maas and i haven't documented it yet
<baldpope> stokachu, that would work perfectly, if I had a spare host to deploy to
<stokachu> most people usually create a VM to house the controller
<baldpope> not complaining - just trying to understand the environmen
<stokachu> and just register that in the maas
<baldpope> yea, that makes sense
<baldpope> stokachu, not sure if it's progressing or not
<baldpope> http://imgur.com/LalU5zS
<stokachu> did it get passed fetching juju agent yet?
<baldpope> no, been sitting here for the last couple of minutes
<stokachu> yea it still can't get out to streams.canonical.com
<baldpope> and both the node and maas controller have unfiltered access out
<baldpope> hm
<baldpope> looking at firewall, no blocked traffic
<stokachu> deploy another node via maas and just see if you can wget from streams.canonical.com
<stokachu> like wget http://streams.canonical.com/juju/tools/agent/2.0-rc2/juju-2.0-rc2-xenial-amd64.tgz
<baldpope> testing - need a minute to deploy again
<baldpope> thanks for taking a few minutes stokachu
<stokachu> np
<baldpope> i've got pages of notes on my side where I've made mistakes, thinkgs I've forgotten after cleaning/reinstalling/deploying
<baldpope> will be happy to share any relevant bits once I get it cleaned up and repeatable
<stokachu> yea im sure roaksoax and the docs team would want to look at that
<shubjero> How does one configure a large amount of servers in maas with a specific disk layout?
<shubjero> I'd like to have 36 machines configured with the same partition layout
<shubjero> I'd rather not have to go through 36 times and do it all
<baldpope> stokachu, ok, a bit lost now... i've deployed a new node - i can perform nslookup on www.google.com, with the reply coming from maas controller, but attempting wget fails, though traffic is not blocked
<baldpope> i can ssh directly into the node using the key provided through webui
<baldpope> default route is out through firewall
<baldpope> stokachu, i appear to have a routing issue - have to work on this and report back - but will have to be later...
<stokachu> baldpope: ok
<mup> Bug # changed: 1392763, 1394792, 1459888, 1481285, 1508975, 1589640, 1593388, 1623110, 1623634, 1623878, 1625711, 1625714, 1627019, 1627038, 1627039, 1627363, 1628052, 1628213, 1628298, 1629004, 1629008, 1629011, 1629019, 1629022, 1629045, 1629142, 1629402, 1629491, 1629868, 1629896
<mup> Bug #1630343 opened: [2.1] upgrade from 2.0 to 2.1 broken <MAAS:New> <https://launchpad.net/bugs/1630343>
<mup> Bug #1629026 changed: [2.1] Images have been imported, but can't add a chassis <MAAS:Invalid> <https://launchpad.net/bugs/1629026>
<mup> Bug #1630361 opened: [2.1 ipv6] MAAS should refuse to deploy a host with bad address-family config <maas-ipv6> <MAAS:New> <https://launchpad.net/bugs/1630361>
<mup> Bug #1616232 changed: [2.1, 2.0] Installs should use GPT by default if volume is larger than 2TB <MAAS:Fix Released by blake-rouse> <MAAS 2.0:Triaged> <MAAS trunk:Fix Released by blake-rouse> <https://launchpad.net/bugs/1616232>
<mup> Bug #1630343 changed: [2.1] upgrade from 2.0 to 2.1 broken <MAAS:Fix Released by blake-rouse> <https://launchpad.net/bugs/1630343>
<mup> Bug #1616232 opened: [2.1, 2.0] Installs should use GPT by default if volume is larger than 2TB <MAAS:Fix Released by blake-rouse> <MAAS 2.0:Triaged> <MAAS trunk:Fix Released by blake-rouse> <https://launchpad.net/bugs/1616232>
<mup> Bug #1630343 opened: [2.1] upgrade from 2.0 to 2.1 broken <MAAS:Fix Released by blake-rouse> <https://launchpad.net/bugs/1630343>
<mup> Bug #1616232 changed: [2.1, 2.0] Installs should use GPT by default if volume is larger than 2TB <MAAS:Fix Released by blake-rouse> <MAAS 2.0:Triaged> <MAAS trunk:Fix Released by blake-rouse> <https://launchpad.net/bugs/1616232>
<mup> Bug #1630343 changed: [2.1] upgrade from 2.0 to 2.1 broken <MAAS:Fix Released by blake-rouse> <https://launchpad.net/bugs/1630343>
<mup> Bug #1630394 opened: [2.1] Bootloaders not downloaded on initial import <MAAS:Confirmed for ltrager> <https://launchpad.net/bugs/1630394>
<mup> Bug #1630398 opened: [2.0] EFI system fails to PXE boot: PXE-E23, Maas server returns TFTP error for bootx64.efi <oil> <MAAS:New> <https://launchpad.net/bugs/1630398>
<wililupy> MAAS 2.0 is not seeing all my interfaces on my server. How can I manually add them?
<wililupy> clarification, my rack controller can't see all of its interfaces.
<mup> Bug #1630398 changed: [2.0] EFI system fails to PXE boot: PXE-E23, Maas server returns TFTP error for bootx64.efi <oil> <MAAS:New> <https://launchpad.net/bugs/1630398>
#maas 2016-10-05
<solefald> hello gents. i am trying to figure out difference between commands in curtin_userdata file. why do some of them require square brackets and others don't?
<solefald> ie  ["curtin", "in-target", "--", "sh", "-c", "mkdir /blah"] vs curtin in-target -- sh -c "/bin/echo -e 'test' > /tmp/file"
<mup> Bug #1630552 opened: [FUJ] The user should be able to progress to Dashboard even if they have not imported an SSH key (MAAS next) <ui> <MAAS:New> <https://launchpad.net/bugs/1630552>
<mup> Bug #1630591 opened: Rename "Networks" tab to "Subnets" <MAAS:Triaged> <https://launchpad.net/bugs/1630591>
<mup> Bug #1630591 changed: Rename "Networks" tab to "Subnets" <MAAS:Triaged> <https://launchpad.net/bugs/1630591>
<mup> Bug #1630591 opened: Rename "Networks" tab to "Subnets" <MAAS:Triaged> <https://launchpad.net/bugs/1630591>
<mup> Bug #1630633 opened: Unable to select nodes <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1630633>
<mup> Bug #1630636 opened: [2.1 ipv6] YAML error when maas_url has an IPv6 IP <maas-ipv6> <MAAS:New> <https://launchpad.net/bugs/1630636>
<solefald> hello gents. i am trying to figure out difference between commands in curtin_userdata file. why do some of them require square brackets and others don't?
<solefald> ie  ["curtin", "in-target", "--", "sh", "-c", "mkdir /blah"] vs curtin in-target -- sh -c "/bin/echo -e 'test' > /tmp/file"
<mup> Bug #1630667 opened: MAAS fails to deploy systems with 3+ TB disks <regression> <MAAS:New> <https://launchpad.net/bugs/1630667>
<agordeev> hello there. could someone shed more light on how automatic discover works for maas? where i can get any information about the PXE image for automatic discovery? any docs/code/scripts?
<mup> Bug #1630679 opened: [2.1] Can't use custom image repository <MAAS:New> <https://launchpad.net/bugs/1630679>
<mup> Bug #1630681 opened: MAAS 2.0 cannot auto discover interfaces on whitebox switches <MAAS:New> <https://launchpad.net/bugs/1630681>
<mup> Bug #1630745 opened: [2.1] Nodes faild to deploy due to 503 error in archive <cdo-qa> <cdo-qa-blocker> <MAAS:New> <https://launchpad.net/bugs/1630745>
<mup> Bug #1630755 opened: Updated to MAAS 2.1 and all my boot resources are gone <hwcert-server> <MAAS:New> <https://launchpad.net/bugs/1630755>
<mup> Bug #1630756 opened: maas.io stream in 2.1 does not provide Trusty <hwcert-server> <MAAS:New> <https://launchpad.net/bugs/1630756>
<mup> Bug #1630757 opened: MAAS should be able to disable discovery (and services) on a per-network basis. <hwcert-server> <MAAS:New> <https://launchpad.net/bugs/1630757>
<mup> Bug #1630756 changed: maas.io stream in 2.1 does not provide Trusty <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630756>
<mup> Bug #1630756 opened: maas.io stream in 2.1 does not provide Trusty <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630756>
<mup> Bug #1630756 changed: maas.io stream in 2.1 does not provide Trusty <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630756>
<mup> Bug #1630786 opened: MAAS 2.1 fails to set power type on BMC system <hwcert-server> <MAAS:Incomplete> <https://launchpad.net/bugs/1630786>
<mup> Bug #1630794 opened: Unable to commission nodes in MAAS 2.1, no external route provided <hwcert-server> <MAAS:Incomplete> <https://launchpad.net/bugs/1630794>
#maas 2016-10-06
<mup> Bug #1630817 changed: [ERROR] Unable to find a copy of bootppc64.bin in the SimpleStream or a previously downloaded copy. <MAAS:New> <https://launchpad.net/bugs/1630817>
<mup> Bug #1630755 opened: [2.1b1] Updated to MAAS 2.1 and all my boot resources are gone <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630755>
<mup> Bug #1630755 changed: [2.1b1] Updated to MAAS 2.1 and all my boot resources are gone <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630755>
<mup> Bug #1630817 opened: [ERROR] Unable to find a copy of bootppc64.bin in the SimpleStream or a previously downloaded copy. <MAAS:New> <https://launchpad.net/bugs/1630817>
<mup> Bug # changed: 1445941, 1611949, 1612203, 1614659, 1619262, 1625676, 1627362, 1628645, 1628761, 1630394, 1630591
<mup> Bug #1630552 changed: [2.1, FUJ] The user should be able to progress to Dashboard even if they have not imported an SSH key <ui> <MAAS:Opinion> <https://launchpad.net/bugs/1630552>
<mup> Bug #1631022 opened: [2.1b1] 'Registering existing rack controller' <MAAS:New> <https://launchpad.net/bugs/1631022>
<Sujeet_> Hi Kiko, Hi Roaksoax
<Sujeet_> how to handle the errors in the user defined commissioning script?
<mup> Bug #1631024 opened: [2.1b1] Dashboard column widths for discovered items are wonky <MAAS:New> <https://launchpad.net/bugs/1631024>
<hscomp> Hi, where do I find the secret used for the sudo maas-rack register command?
<kiko> hscomp, is that not documented at maas.io/docs?
<brendand> hscomp, http://maas.io/docs/en/manage-cli-advanced#install-a-rack-controller
<brendand> "you will be prompted for the secret key that is stored in file /var/lib/maas/secret on the region controller."
<kiko> thanks brendand
<hscomp> brendand, yes but it has to be encoded and the secret stored there wasn't. So I encoded it an after using it in this command the encoded version was stored under /var/lib/maas/secret
<brendand> hscomp, what do you mean it wasn't encoded?
<hscomp> brendand, message was: Secret could not be decoded: Non-hexadecimal digit found
<brendand> hscomp, odd
<brendand> hscomp, which version of maas was this?
<hscomp> 2.0
<brendand> hscomp, where was that message?
<hscomp> brendand, as root: maas-rack register
<brendand> hscomp, then you entered the secret in that file?
<brendand> hscomp, do you happen to have a copy of what the contents of the file were? i assume you changed it so maybe you don't anymore
<mup> Bug # changed: 1229458, 1621507, 1621615, 1625809
<kiko> hscomp, this deserves a bug if you did everything based on the instructions we provide
<kiko> hscomp, can you outline the steps you took to get to this situation?
<kiko> <allenap> kiko: The secret is decoded to binary in memory, but it's saved as base16/hex in /var/lib/maas/secret.
<kiko> hscomp, ^^
<hscomp> kiko, I really do not remember when I put it into /var/lib/maas/secret
<kiko> hscomp, the question is how you got the text to put in there
<hscomp> kiko, yes I'll try to repeat all the steps tomorrow
<mup> Bug #1631061 opened: Deferred being returned from thread <MAAS:Triaged> <https://launchpad.net/bugs/1631061>
<mup> Bug #1631062 opened: [2.1] MAAS Subnets 'tab' should have an icon telling the user Active Discovery is enabled <MAAS:New> <https://launchpad.net/bugs/1631062>
<mup> Bug #1631064 opened: [2.1] MAAS Subnets 'tab' should have an icon telling the user that DHCP is enabled <MAAS:New> <https://launchpad.net/bugs/1631064>
<mup> Bug #1631079 opened: [2.1, UI] Other reserved IP ranges disappear when one of them is deleted on Subnet details page. <MAAS:Triaged> <https://launchpad.net/bugs/1631079>
<mup> Bug #1630786 changed: [2.1b1] MAAS 2.1 fails to set power type on BMC system <hwcert-server> <MAAS:Incomplete> <https://launchpad.net/bugs/1630786>
<mup> Bug #1631083 opened: grub fails to install on maas 1.9.4 with ubuntu 16.04 <MAAS:New> <https://launchpad.net/bugs/1631083>
<mup> Bug #1631113 opened: Doc for "Backdoor (add a login) to ephemeral image" needs updating <MAAS:New> <https://launchpad.net/bugs/1631113>
<braven_> what is the command to reset a node IP
<mup> Bug #1631129 opened: Support for Microsoft Hyper-V would be nice <MAAS:Triaged> <https://launchpad.net/bugs/1631129>
<braven_> ??
<brendand> braven_, what do you mean by 'reset'?
<mup> Bug #1631129 changed: Support for Microsoft Hyper-V would be nice <MAAS:Triaged> <https://launchpad.net/bugs/1631129>
<mup> Bug #1631129 opened: Support for Microsoft Hyper-V as a Chassis <MAAS:Triaged> <https://launchpad.net/bugs/1631129>
<mup> Bug #1631132 opened: jquery script failure on MAAS UI Nodes page <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1631132>
<mup> Bug #1631132 changed: jquery script failure on MAAS UI Nodes page <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1631132>
<mup> Bug #1631132 opened: jquery script failure on MAAS UI Nodes page <cdo-qa> <MAAS:New> <https://launchpad.net/bugs/1631132>
<mup> Bug #1467635 changed: Image import cannot be aborted <ui> <ux> <MAAS:Fix Released by blake-rouse> <https://launchpad.net/bugs/1467635>
<mup> Bug #1631152 opened: [2.1] Failed to mount a partition and it doesn't surface error. <MAAS:New> <https://launchpad.net/bugs/1631152>
<mup> Bug #1631152 changed: [2.1 UI] Failed to mount a partition and it doesn't surface error. <MAAS:New> <https://launchpad.net/bugs/1631152>
<mup> Bug #1631152 opened: [2.1 UI] Failed to mount a partition and it doesn't surface error. <MAAS:New> <https://launchpad.net/bugs/1631152>
<mup> Bug #1630794 changed: [2.1b1] Unable to commission nodes in MAAS 2.1, no external route provided <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630794>
<mup> Bug #1630794 opened: [2.1b1] Unable to commission nodes in MAAS 2.1, no external route provided <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630794>
<mup> Bug #1630794 changed: [2.1b1] Unable to commission nodes in MAAS 2.1, no external route provided <hwcert-server> <MAAS:Invalid> <https://launchpad.net/bugs/1630794>
#maas 2016-10-07
<mup> Bug #1630817 changed: [ERROR] Unable to find a copy of bootppc64.bin in the SimpleStream or a previously downloaded copy. <MAAS:New> <https://launchpad.net/bugs/1630817>
<Sujeet_> Hi Kiko
<Sujeet_> How to handle errors in the commissioning script?
<mup> Bug #1631358 opened: [2.1]  Incorrect logging message - showing SERVICE_STATE.ON <MAAS:New> <https://launchpad.net/bugs/1631358>
<mup> Bug #1631358 changed: [2.1]  Incorrect logging message - showing SERVICE_STATE.ON <MAAS:New> <https://launchpad.net/bugs/1631358>
<mup> Bug #1631358 opened: [2.1]  Incorrect logging message - showing SERVICE_STATE.ON <MAAS:New> <https://launchpad.net/bugs/1631358>
<mup> Bug #1630745 changed: [2.1b1] Nodes faild to deploy due to 503 error in archive <cdo-qa> <cdo-qa-blocker> <MAAS:Invalid> <https://launchpad.net/bugs/1630745>
<mup> Bug #1630745 opened: [2.1b1] Nodes faild to deploy due to 503 error in archive <cdo-qa> <cdo-qa-blocker> <MAAS:Invalid> <https://launchpad.net/bugs/1630745>
<mup> Bug #1630745 changed: [2.1b1] Nodes faild to deploy due to 503 error in archive <cdo-qa> <cdo-qa-blocker> <MAAS:Invalid> <https://launchpad.net/bugs/1630745>
 * baldpope sigh
<baldpope> figured out my networking issue from inside lan - forgot to turn on nat at perimeter firewall..
<fbarilla> hi, can't restart maas-dhcpd
<fbarilla> root@hnode1-8:~# service maas-dhcpd restart Failed to restart maas-dhcpd.service: Unit maas-rackd.service is masked.
<roaksoax> fbarilla: restart maas-rackd
<roaksoax> fbarilla: that should restart maas-dhcpd
<mup> Bug #1631403 opened: [2.1b2] Service 'ntp' is SERVICE_STATE.ON but not in the expected state of 'running', its current state is 'exited'. <oil> <MAAS:New> <https://launchpad.net/bugs/1631403>
<fbarilla> root@hnode1-8:~# service maas-rackd restart Failed to restart maas-rackd.service: Unit maas-rackd.service is masked.
<mup> Bug #1631403 changed: [2.1b2] Service 'ntp' is SERVICE_STATE.ON but not in the expected state of 'running', its current state is 'exited'. <oil> <MAAS:New> <https://launchpad.net/bugs/1631403>
<mup> Bug #1631403 opened: [2.1b2] Service 'ntp' is SERVICE_STATE.ON but not in the expected state of 'running', its current state is 'exited'. <oil> <MAAS:New> <https://launchpad.net/bugs/1631403>
<mup> Bug #1631420 opened: [2.1 UI] Images page "Queued for download" is confusing when selections are not saved <MAAS:New> <https://launchpad.net/bugs/1631420>
<mup> Bug #1631421 opened: [2.1b2] failure to deploy all systems that lasted until reboot of maas container <oil> <MAAS:New> <https://launchpad.net/bugs/1631421>
<braven_> hello
<braven_> Is anyone on line?
<brendand> yes
<brendand> braven_, ^
<braven_> I am having an issue. I keep getting a internal error
<braven_> when try to save a interface to dhcp
<braven_> The cluster controller will not accept the changes
<braven_> Code<UNKNOWN>: Unknown Error
<brendand> braven_, was it you was asking about the secret yesterday?
<braven_> no
<braven_> We have the cluster Controller connected
<braven_> it was working. We tried adjust the dhcp range. It would not save
<braven_> when we try to save the changes to the interface we get "Internal server error."
<sean5384> hi there
<braven_> hello
<sean5384> i hope someone can shed some light for me, i am trying to install MAAS and it says a few sentences down about the clusters tab, where the heck is it at?
<braven_> what version of MAAS
<sean5384> i think it is 2.0, let me check
<sean5384> if i'm running ubuntu 16.04 LTS, it should be 2.0 right?
<braven_> yeap.
<sean5384> ok then, thats what i have running then.
<sean5384> is the clusters tab, no more then or something?
<braven_> I am using the old version because my hardware. But do remember read they change it
<sean5384> i also can't seem to get my private network functioning correcty within ubuntu itself either for some reason, would that have something to do with it maybe?
<braven_> They something new called Fabric
<sean5384> ok, i have seen that.
<brendand> sean5384, do you have a link to the docs you are reading?
<braven_> Have used the older version of MAAS
<sean5384> i was trying to install via the documentation, but you can't follow that now i can see.
<brendand> braven_, which version are you using?
<braven_> 1.8
<brendand> ahhh
<brendand> on which version of ubuntu
<braven_> http://maas.io/docs/en/installconfig-rack  for Sean
<braven_> 14.04
<braven_> the hardware I have is not supported by 16
<sean5384> i just saw the documentation on top of this site.....
<braven_> lol
<sean5384> this makes more sense now.
<brendand> braven_, 1.8 is not really supported. 1.9 is the version in Trusty
<sean5384> i was getting distraught because there was no clusters tab at the top of the page anywhere.
<sean5384> lol
<braven_> MAAS is awesome when it works. We were able to deploy 60 nodes in about 3 hour
<sean5384> wow
<braven_> how do I know what version I am running?
<sean5384> i'm trying to get just 5 going right now.
<brendand> braven_, it should say at the bottom of the web ui
<brendand> or you can do dpkg -l | grep maas on the maas server
<brendand> braven_, that largely depends on your hardware/network too, we regularly deploy 40+ nodes in about 15-20 mins
<brendand> haven't timed it though so it could be a bit longer
<braven_> wow
<braven_> I wonder when we configured wrong
<brendand> it will use a lot of bandwidth, is one thing
<braven_> 1.7.6
<braven_> so can we upgrade without loosing all the nodes?
<brendand> braven_, you *should* be able to
<braven_> do know why it might be give that error when I am trying to save a cluster interface
<brendand> could be anything really. haven't used 1.7 much
<brendand> braven_, but some logs would be helpful
<brendand> braven_, if you're willing to upgrade that would be a good first step, then if it still happens, file a bug
<smgoller> Hi all, if I want to control VMs on a vCenter server, what has to be on the same subnet as the rack controller, the vCenter appliance, or the individual esxi servers?
<batkins61_> Nodes won't enlist, even though booted under maas.  Metadata service appears unreachable.  Version 1.9 on 14.04.
<batkins61_> Where should I look for that?
<batkins61_> Is there any help on this channel?  Is there somewhere else to ask questions?
<baldpope> with recent update to conjure, i get error after selecting which type of openstack to deploy
<stokachu> baldpope: which is?
<mup> Bug # changed: 1470202, 1473254, 1553647, 1563779, 1563799, 1563807, 1592666, 1604901
<baldpope> conjure-up -> openstack for maas -> hawk (only option) -> error
<baldpope> unable to list models: error cannot list models: no api addresses
<stokachu> baldpope: sounds like a failed bootstrap
<stokachu> what does juju status return?
<baldpope> error no api address
<stokachu> yea your bootstrap failed
<baldpope> i'm seeing this on the ubuntu maas host
<baldpope> all nodes in ready - none currently deployed - i thought this prepped the first node?
<baldpope> or can I make this maas head the juju controller?
<stokachu> yea so you have a state where juju thinks a controller exist but maas has nothing deployed
<stokachu> i would juju kill-controller hawk
<baldpope>  Unable to open API: no API addresses Unable to connect to the API server, destroying through provider
<stokachu> thats fine it'll just kill it manually and delete it from it's database
<stokachu> then you can try conjure-up again
<stokachu> baldpope: nah we have a seperate queue that handles all that logic so you can proceed until you see the screen showing the bootstrap output
<baldpope> looks like it's making progress
<baldpope> i see progress in the conjure window - hadn't see that before
<stokachu> baldpope: :D
<baldpope> stokachu, hm, seems like it's stalled again
<stokachu> baldpope: the machine is still in deploying state?
<baldpope> nope, deployed
<stokachu> what the wait screen showing?
<baldpope> oh wait
<baldpope> just finihed
<baldpope> wow - long time?
<baldpope> it's now deploying the rest
<stokachu> yea maas is depending on the hardware
<stokachu> so if the systems take a minute to boot up then it'll prolong that wait
<stokachu> etc
<baldpope> gotcha
 * baldpope w000t-- progress
<stokachu> but honestly aws takes about the same amount of time from bootstrap to deploy
<baldpope> i don't mind failure as long as I learn something in the progress
<stokachu> how many machines do you have available in maas?
<stokachu> after the first one deployed?
<baldpope> 5 total
<baldpope> blade(1-5)
<stokachu> ok cool
<stokachu> all with 2nics?
<baldpope> the first, is where juju (I believe) completed the it's deployment
<baldpope> 6 technically, but 2 are plumbed
<stokachu> yea the first deployed machine is your controller node
<stokachu> ok
<stokachu> baldpope: and your network interfaces are coming up eth0/eth1 as well?
<baldpope> side note, i'm all in on openstack for the foreseeable future at work - so anything you guys need to test - i'm available
<stokachu> i know we fixed that yesterday
<baldpope> on blade1, it's shows as br-enp9s0f0
<baldpope> as the active interface
<baldpope> and then en01, enp9s0f0, enp9s0f1, ens3f1, 2, 3,
<stokachu> ok neutron may fail with a 'config-changed' hook that we can fix with juju config neutron-gateway br-ext=en01 etc
<stokachu> and then juju resolved neutron-gateway/0
<stokachu> you could probably set that now if you want
<stokachu> to avoid the hook error
<baldpope> so from the maas head, run what?
<stokachu> juju config neutron-gateway br-ext=<second interface name>
<baldpope> ok, quick question before I do that - the remaining 4 are now in deployed state
<baldpope> but everything is in waiting or maintenance state
<stokachu> yea they are all going through processing hooks etc
<stokachu> all the things needed to get the software up
<baldpope> should I still attempt to change the neutron-gateway?
<stokachu> yea it should queue it up and apply it
<baldpope> k
<stokachu> if the charm breaks because of performing that action it is a bug in the charm
<stokachu> unless the interface was entered incorrectly
<baldpope> and the second int name is the one on the node, correct?
<stokachu> though it should really let you know via status-set blocked
<stokachu> baldpope: yea the one on that node that is running neutron-gateway
<baldpope> ERROR unknown option "br-ext"
<baldpope> juju config neutron-gateway br-ext=enp9s0f1
<baldpope> and conjure-up just errored out - neutron-gateway/0: hook failed: config-changed
<stokachu> sec
<stokachu> baldpope: ah you need to setup bridge-mappings
<stokachu> https://jujucharms.com/neutron-gateway/xenial/3#charm-config-bridge-mappings
<stokachu> so like if you have a bridge setup on that node you can do
<stokachu> juju config neutron-gateway bridge-mappings=mynet:br0
<baldpope> stokachu, while setting up a bridge makes sense (and I'd want to get there eventually) I did not setup bridge mode (either in maas or on the switch)
<baldpope> i assumed the br-enp9s0f0 was stock from the existing charm?
<stokachu> does that bridge exist on the system?
<stokachu> if you juju ssh neutron-gateway/0
<stokachu> more info as well: https://ask.openstack.org/en/question/94206/how-to-understand-the-bridge_mappings/
<baldpope> http://pastebin.ubuntu.com/23290786/
<stokachu> yea so you can assign the bridge mapping to br-enp9s0f0
<stokachu> baldpope: additional info https://github.com/openstack/charm-neutron-gateway#port-configuration
<stokachu> you can access all those config options from conjure-up if you click on configure->advanced configuration
<baldpope> silly question stokachu - i feel like I've completely skipped a ton of reading - attempting to just follow the ubuntu docs on deploying open stack - is there a primer I skipped/missed?
<stokachu> baldpope: networking is complex and you'll want to be familiar with that
<batkins61> Do target hosts have to have internet access during elistment?
<baldpope> stokachu, not gonna lie - a lot to take in
<stokachu> baldpope: yea openstack is a complex beast
<baldpope> stokachu, for the sake of conversation i've got the 6 ports total, 2 on board, and 4 on the add-on card - the long-term idea being that I'd use the some combination to dedicate shared storage and some remain to dedicate to service networks
<baldpope> all that is done through juju ahead of deployment, or post deploy?
<baldpope> i can see where that made little sense, i'll try again... dedicate 4 ports to the shared storage, and the 2 onboard to network services
<baldpope> with the appropriate lag/bridge ports in place for redundancy and throughput
<baldpope> ok, so I did the following: juju config neutron-gateway bridge-mappings=br-enp9s0f0
<baldpope> and am re-running conjure against the existing deployment
<baldpope> we'll see
<baldpope> bridge-mappings appears to be the current non-deprecated command
#maas 2016-10-08
<mup> Bug #1610475 changed: [2.0 rc3] DNS not responding - rndc -c /etc/bind/maas/rndc.conf.maas reload` returned non-zero exit status 1 <oil> <oil-2.0> <MAAS:Expired> <https://launchpad.net/bugs/1610475>
<mup> Bug #1631595 opened: [IPMI] MAAS can't operate a Supermicro server with IPMI firmare version 1.76. <ipmipower> <MAAS:New> <https://launchpad.net/bugs/1631595>
#maas 2017-10-02
<mup> Bug #1711320 opened: [2.3, UI] Can't 'Save changes' and 'Cancel' on machine/device details page <tags> <MAAS:Fix Committed by blake-rouse> <https://launchpad.net/bugs/1711320>
<kukacz> hi, is there anything special on updating maas 2.1(.5) to 2.2(.2)? or is `apt upgrade` enough to make it done? it's a small all-in-one installation with 15 managed nodes
<roaksoax> kukacz: sudo apt-get update && sudo apt-get dist-upgrade should do
<kukacz> roaksoax: thanks!
#maas 2017-10-03
<Baqar>   
<mup> Bug # opened: 1721105, 1721106, 1721108, 1721111
<mup> Bug #1721113 opened: [2.3, UI, HWTv2] Machine details cards - StoraIf multiple disks, condense the card instead of showing all disks <MAAS:New> <https://launchpad.net/bugs/1721113>
<mup> Bug #1721113 changed: [2.3, UI, HWTv2] Machine details cards - StoraIf multiple disks, condense the card instead of showing all disks <MAAS:New> <https://launchpad.net/bugs/1721113>
<mup> Bug #1721113 opened: [2.3, UI, HWTv2] Machine details cards - StoraIf multiple disks, condense the card instead of showing all disks <MAAS:New> <https://launchpad.net/bugs/1721113>
#maas 2017-10-04
<mup> Bug # changed: 1507712, 1684085, 1696270, 1711320, 1717287, 1718209, 1718270, 1718294, 1718686
<mup> Bug # opened: 1721266, 1721268, 1721270, 1721273
<mup> Bug #1721276 opened: [2.3, UI, HWTv2] Hardware Test tab - Table alignment for the results doesn't align with titles <MAAS:New for ltrager> <https://launchpad.net/bugs/1721276>
<gull> hello everyone !
<shubjero> I have a bunch of servers that are unknown to MaaS but have been configured to PXE boot and MaaS is serving up a PXE environment but the servers never make it in to MaaS as 'New'. If I add the server in to Maas manually I can get it to work/comission/etc. Anyone know whats up with that?
<shubjero> I used to be able to just pxe boot new servers and they would automatically appear in maas after going through a pxe boot.. from there i was able to comission and deploy them
<shubjero> I guess whats not working is the 'enlistment'
<roaksoax> shubjero: check that /etc/maas/rackd.conf points to the right IP address
<shubjero> yeah, it does
<roaksoax> shubjero: do you have a console log to look at ?
<shubjero> i have a virtual console to the server
<shubjero> it reaches an ubuntu 16.04 login but there are some errors before it gets to the login prompt
<shubjero> and im not sure how to log in to the system at that state
<roaksoax> shubjero: would be good to see those errors
<shubjero> yeah but i cant get at them
<shubjero> flys by so quick
<mup> Bug #1721360 opened: "ubuntu xenial is configured as the commissioning release but it is not selected for download" but xenial is synced <cdo-qa> <cdo-qa-blocker> <MAAS:New> <https://launchpad.net/bugs/1721360>
<shubjero> roaksoax: https://imgur.com/a/bywNj - here's the error
<roaksoax> shubjero: seems like the machine cannot find the datasource
<roaksoax> shubjero: try making sure that /etc/maas/regiond.conf has the correct maas_url to an ip the machine can access
<shubjero> maas_url: http://localhost/MAAS
<shubjero> is that right? :/
<roaksoax> shubjero: change that to an  IP
<roaksoax> shubjero: e.g. http://<ip>:5240/MAAS
<shubjero> the maas server ip, right?
<roaksoax> shubjero: yes
<shubjero> yeah, ill give that a try
<shubjero> give me a bit, got distracted with something else
<shubjero> thanks for your help
<shubjero> roaksoax: works, you da man
#maas 2017-10-05
<mup> Bug #1717543 opened: [FFe] Standing Feature Freeze Exception for MAAS 2.3 <verification-needed> <verification-needed-artful> <maas (Ubuntu):New> <https://launchpad.net/bugs/1717543>
<mup> Bug #1717543 changed: [FFe] Standing Feature Freeze Exception for MAAS 2.3 <verification-needed> <verification-needed-artful> <maas (Ubuntu):New> <https://launchpad.net/bugs/1717543>
<mup> Bug #1721454 opened: Finding how to rename a node is difficult <MAAS:New> <https://launchpad.net/bugs/1721454>
<boolman> I'm having trouble with my Maas, I can pxe boot. but the servers wont register. where can I get a complete list of network port requirements?
<boolman> for my firewall
<mup> Bug #1717543 changed: [FFe] Standing Feature Freeze Exception for MAAS 2.3 <verification-needed> <verification-needed-artful> <maas (Ubuntu):Fix Released> <https://launchpad.net/bugs/1717543>
<roaksoax> boolman: what's the error you see ?
<roaksoax> d you have a console log ?
<mup> Bug #1721523 opened: [2.3] tgt continues to get restarted and managed even when disabled <MAAS:Triaged> <https://launchpad.net/bugs/1721523>
<mup> Bug #1721524 opened: [2.3, UI, HWTv2] When upgrading from older MAAS, Storage HW tests are not mapped to a disk <MAAS:Triaged> <https://launchpad.net/bugs/1721524>
<mup> Bug #1721525 opened: [2.3, UI, HWTv2] Storage card on machine details page missing red bar on top if there are failed tests <MAAS:Triaged by newell-jensen> <https://launchpad.net/bugs/1721525>
<mup> Bug # changed: 1389811, 1706462, 1707674, 1707675, 1710867, 1715337, 1715687
<mup> Bug #1394306 changed: maas 1.7 upgrade: document potential need for maas-proxy.conf manual setup migration <canonical-bootstack> <canonical-is> <MAAS:Won't Fix> <maas (Ubuntu):Won't Fix> <https://launchpad.net/bugs/1394306>
<mup> Bug # changed: 1394306, 1443644, 1537095, 1540539, 1540548, 1587201, 1612755, 1715345, 1715353
<mup> Bug #1394306 opened: maas 1.7 upgrade: document potential need for maas-proxy.conf manual setup migration <canonical-bootstack> <canonical-is> <MAAS:Won't Fix> <maas (Ubuntu):Won't Fix> <https://launchpad.net/bugs/1394306>
<mup> Bug # opened: 1394306, 1443644, 1537095, 1540539, 1540548, 1587201, 1612755, 1715345, 1715353
<mup> Bug #1394306 changed: maas 1.7 upgrade: document potential need for maas-proxy.conf manual setup migration <canonical-bootstack> <canonical-is> <MAAS:Won't Fix> <maas (Ubuntu):Won't Fix> <https://launchpad.net/bugs/1394306>
<mup> Bug # changed: 1352582, 1379566, 1394306, 1443644, 1537095, 1540539, 1540548, 1575625, 1587201, 1612755, 1715345, 1715353
<mup> Bug #1352582 opened: MAAS doesn't detect invalid YAML in curtin_userdata <canonical-is> <internal> <tech-debt> <MAAS:Fix Released> <https://launchpad.net/bugs/1352582>
<mup> Bug #1379566 opened: maas-cluster depends on secure_path settings in sudoers <canonical-is> <MAAS:Fix Released> <https://launchpad.net/bugs/1379566>
<mup> Bug #1575625 opened: Can't find way for MAAS to autodiscover IPMI settings <canonical-is> <MAAS:Invalid> <https://launchpad.net/bugs/1575625>
<mup> Bug #1352582 changed: MAAS doesn't detect invalid YAML in curtin_userdata <canonical-is> <internal> <tech-debt> <MAAS:Fix Released> <https://launchpad.net/bugs/1352582>
<mup> Bug #1379566 changed: maas-cluster depends on secure_path settings in sudoers <canonical-is> <MAAS:Fix Released> <https://launchpad.net/bugs/1379566>
<mup> Bug #1575625 changed: Can't find way for MAAS to autodiscover IPMI settings <canonical-is> <MAAS:Invalid> <https://launchpad.net/bugs/1575625>
<mup> Bug #1593487 changed: [2.0b7]maas-region-controller failed to preconfigure exit 127 <cdo-qa> <internal> <MAAS:Invalid> <https://launchpad.net/bugs/1593487>
<mup> Bug #1628338 changed: 2.1a4:syslog flooded with whoami requests <cdo-qa> <internal> <MAAS:Invalid> <https://launchpad.net/bugs/1628338>
<mup> Bug #1721548 opened: [2.3] Failure on controller refresh seem to be causing version to not get update <MAAS:Triaged> <https://launchpad.net/bugs/1721548>
<mup> Bug #1721548 changed: [2.3] Failure on controller refresh seem to be causing version to not get update <MAAS:Triaged> <https://launchpad.net/bugs/1721548>
<mup> Bug #1721548 opened: [2.3] Failure on controller refresh seem to be causing version to not get update <MAAS:Triaged> <https://launchpad.net/bugs/1721548>
<saudk> Hello Guys
<saudk> I am trying to import the ubuntu 16.04 image in MAAS
<saudk> and seeing this error
<saudk>  [ERROR] Failed to download images: [Errno 2] No such file or directory: u'/tmp/maas-Ud4i5y/usr/lib/shim/shim.efi.signed'
<saudk> any ideas?
<saudk> thanks
<xgrid> exit
<mup> Bug #1721587 opened: [2.3, UI, HWTv2] Commissioning logs (and those of v2 HW Tests) are not being show <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721587>
<roaksoax> saudk_: what version of MAAS are you using ?
<junaidali> roaksoax: I had a private chat with saudk, he is using maas 1.9 on trusty
<junaidali> with maas 2.2, he was facing an issue where importing image gets stuck. Moving on to 1.9, the image gets imported but in maas.log and clusterd.log, it errors out with the msg he shared
<roaksoax> junaidali: he needs to upgrade to latest 1.9 in -proposed
<shubjero> Anyone here work with Intel PCH RAID (available on supermicro servers and others).. I've got two SSD's configured as RAID-1 in the intel raid bios but MaaS still see's two separate SSD's.. I was expecting MaaS to see 1 virtual disk
<junaidali> roaksoax: noted, thanks
<roaksoax> shubjero: that could actually be an issue with the OS itself, since maas uses lsblk
<shubjero> i would think the os would just show 1 device even with lsblk though.. since the whole point of a raid controller is to add a layer of abstraction
<shubjero> im going to ignore the fact that maas see's the two disks, and just configure one of them to boot & install the os on and see what happens inside the os once it's deployed :)
<roaksoax> shubjero: that's my point, it could be an issue with the os
<shubjero> ok, fingers crossed.. here goes :)
<shubjero> roaksoax: weird, the deploy crashed on ntp stuff? https://pastebin.com/ULXEc8Rg
<roaksoax> shubjero: that's a known bug, are you sure you using your latest images ?
<roaksoax> shubjero: it was an issue with the kernel
<shubjero> mine are set to sync automatically
<roaksoax> shubjero: what version of MAAS ?
<shubjero> 2.1, but i see there are updates to my maas packages so im upgrading to 2.2
<mup> Bug #1721611 opened: [2.3] Rack refresh failures may be causing controllers to become degraded <MAAS:Triaged> <https://launchpad.net/bugs/1721611>
<JoeJulian> I cannot find a release schedule. When is 2.3.0 expected to go GA?
#maas 2017-10-06
<saudk_> Hey Guys
<saudk_> I am trying to provision a server using MAAS
<saudk_> and see this error while importing images
<saudk_> No such file or directory: u'/tmp/maas-MEt55A/usr/lib/shim/shim.efi.signed'
<saudk_> any ideas
<saudk_> ?
<saudk_> thanks
<parlos> Good Morning Maas!
<boolman> roaksoax: thanks for the reply, it was my own fault. I have a custom resolver in which I have 3 views. so the client got the wrong zone and can't do a lookup for the maas server
<boolman> its all sorted out now
<mup> Bug #1721743 opened: [2.3b2] Rack and region controller versions still not updated <MAAS:New> <https://launchpad.net/bugs/1721743>
<mup> Bug #1721759 opened: no error message when virsh pod not accessible <MAAS:New> <https://launchpad.net/bugs/1721759>
<mup> Bug #1721743 changed: [2.3b2] Rack and region controller versions still not updated <MAAS:New> <https://launchpad.net/bugs/1721743>
<mup> Bug #1721823 opened: [2.3, UI, HWTv2] No way to surface a failed test that's non CPU, Mem, Storage in machine listing page <MAAS:Triaged> <https://launchpad.net/bugs/1721823>
<mup> Bug #1721824 opened: [2.3, HWTv2] Overall health status is missing <MAAS:Triaged> <https://launchpad.net/bugs/1721824>
<mup> Bug #1721825 opened: [2.3, HWTv2] Tests are not run in sensible order <MAAS:Triaged> <https://launchpad.net/bugs/1721825>
<LikeLuc> hey, i habe some problems with my "20 mins old" mass installation after i configured the DHCP i started a node and get a cloud-init fail ;( any idea?
<LikeLuc> cloud-init[1302] maybe this can help u ?
<mup> Bug #1721825 changed: [2.3, HWTv2] Tests are not run in sensible order <MAAS:Triaged> <https://launchpad.net/bugs/1721825>
<LikeLuc> okay, maybe u can tell me how i update to MAAS 2.3.0alpha3
<mup> Bug # opened: 1721825, 1721827, 1721829, 1721831
<roaksoax> LikeLuc: check that /etc/maas/rackd.conf has maas_url pointing at a IP and not at localhost
<roaksoax> saudk_: upgrade to the latest 1.9.5, a fix should have been made available in -updates already
<roaksoax> like yetserday i think
<LikeLuc> @roaksaoax in /etc/maas/rackd.conf i add my ip, now there stand "http://10.1.0.100:...." do i need this "http"?
<LikeLuc> @roaksoax *
<LikeLuc> *IT WORKS TY <3
<LikeLuc> so now i have one more question which power type i have to use with Hyper V
<roaksoax> LikeLuc: manual, we can't talk directly to hyperv to manage vm's
<LikeLuc> ;( so i have to use manual
<LikeLuc> right
<jamesbenson> hey roaksoax, have a minute?
<roaksoax> jamesbenson: hey, shoot!
<jamesbenson> hey
<jamesbenson> sorry about the delay roaksoax
<jamesbenson> It's an odd question (I think).  My servers have 2 nics in use.  One completely internal (no external access at all) and used for PXE, the other used for internet access.  https://i.imgur.com/NF5laT7.png
<jamesbenson> Something like that... (sorry no text)
<jamesbenson> I am able to deploy nodes, but once deployed, I need to fix the DNS and gateway to the public side.  By default the gw is set to the internal maas IP and DNS set to maas.  MaaS does have internet access and can resolve names.  How do I get it so the internet works "out of the box"
<jamesbenson> So I don't need to fix DNS and interfaces file...
<jamesbenson> It's got to be something simple that I'm overlooking.
<roaksoax> jamesbenson: so you want to set the default gateway to the external IP & the DNS ?
<roaksoax> jamesbenson: something like this would work: maas <user> interface set-default-gateway <system_id> <interface>
<jamesbenson> roaksoax: and there is no way doing that in the gui?
<roaksoax> nope
<jamesbenson> what about the DNS?
<roaksoax> jamesbenson: you can override the DNS byt setting it on the subnet
<roaksoax> jamesbenson: go to the subnet in MAAS< and set the DNS IP there
<jamesbenson> and delete the others?
<jamesbenson> roaksoax: Also any good way to get maas to reboot hyper-v nodes?
<jamesbenson> roaksoax: feature improvement with having an option to do that in the gui, similarly to the storage is done...
<roaksoax> jamesbenson: yeah
<roaksoax> we cannot power manage hyper-v
<jamesbenson> roaksoax: yeah, sort of torture for me right now...
<jamesbenson> I'm setting up juju with the controller on a VM in hyper-v attached to my maas cloud.
<jamesbenson> hyper-v VM1 has juju and maas... my juju controller is hyper-v VM2 (with IP's under one rack, R4), and the rest of the maas cloud are actual servers in the same rack, R4....
<roaksoax> nice
<roaksoax> yeah we dont suport hyperv
<roaksoax> well we can deploy it
<roaksoax> ew just can't manage vm's inside it
<jamesbenson> roaksoax: how do you unset the default gateway as well?  I looks like I have to in my route -n...
<mup> Bug #1721867 opened: bcache defined filesystems configured as /dev/bcacheX in fstab which causes inconsistent mounting in future boots <canonical-bootstack> <canonical-is> <MAAS:New> <https://launchpad.net/bugs/1721867>
<mup> Bug #1721869 opened: [2.3] DHCP address doesn't show <MAAS:Triaged> <https://launchpad.net/bugs/1721869>
<jamesbenson> roaksoax: I think I got it.  :-)
<jamesbenson> roaksoax, no dice...
<jamesbenson> I'll continue on monday :-)
<mup> Bug #1721886 opened: [2.3, UI, HWTv2] Hardware Test tab doesn't auto-update <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721886>
<mup> Bug #1721887 opened: [2.3, HWTv2] No way to override a machine that Failed Testing <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721887>
<mup> Bug #1721886 changed: [2.3, UI, HWTv2] Hardware Test tab doesn't auto-update <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721886>
<mup> Bug #1721887 changed: [2.3, HWTv2] No way to override a machine that Failed Testing <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721887>
<mup> Bug #1721886 opened: [2.3, UI, HWTv2] Hardware Test tab doesn't auto-update <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721886>
<mup> Bug #1721887 opened: [2.3, HWTv2] No way to override a machine that Failed Testing <MAAS:Triaged by ltrager> <https://launchpad.net/bugs/1721887>
#maas 2017-10-07
<faizz> hi
<faizz> how are you
#maas 2017-10-08
<amir_> hi, On ESXi 6.5, I am trying to PXE install maas VM node via existing dhcp server, the node finds the maas controller VM pxe and boots the install process. But at some point the node VM shuts down.
<amir_> Anyone can help?
<chamar> Love MAAS.  Thanks for that!
<mup> Bug #1722090 opened: [2.3, HWTv2] Logs do not provide an idea of why a test failed <MAAS:Triaged> <https://launchpad.net/bugs/1722090>
#maas 2019-09-30
<mup> Bug #1842287 changed: Node deploy options without writing to secondary disks or storage <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1842287>
<mup> Bug #1842287 opened: Node deploy options without writing to secondary disks or storage <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1842287>
<mup> Bug #1842287 changed: Node deploy options without writing to secondary disks or storage <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1842287>
<mup> Bug #1845928 opened: [Feature request] Configurable maas-proxy ports <MAAS:New> <https://launchpad.net/bugs/1845928>
<mup> Bug #1845928 changed: [Feature request] Configurable maas-proxy ports <MAAS:New> <https://launchpad.net/bugs/1845928>
<mup> Bug #1845928 opened: [Feature request] Configurable maas-proxy ports <MAAS:New> <https://launchpad.net/bugs/1845928>
<mup> Bug #1770141 changed: [2.4b3] When SMART is not supported on a drive, some tests pass and some are skipped <MAAS:Invalid> <https://launchpad.net/bugs/1770141>
<mup> Bug #1845928 changed: [Feature request] Configurable maas-proxy ports <MAAS:Invalid> <https://launchpad.net/bugs/1845928>
<mup> Bug #1845788 changed: Failed to deploy Disco i386: Unable to locate package grub-efi-i386 <MAAS:Invalid> <https://launchpad.net/bugs/1845788>
#maas 2019-10-01
<mup> Bug #1846141 opened: Send SSL updates over websocket <MAAS:New> <https://launchpad.net/bugs/1846141>
<mup> Bug #1846259 opened: [API] Get /machines filtering by status/status_name <api> <MAAS:New> <https://launchpad.net/bugs/1846259>
<ebsalberto> Hi all. I use static IPs on public interfaces for ~900 nodes. Whenever we recommission a machine, that interface will simply revert back to an Unconfigured state. I've searched discourse and launchpad to see if anyone is facing this issue, but without luck. Could this be a bug or am I missing
<ebsalberto> something?*
<ebsalberto> I have `retain network configuration` checked btw
<ebsalberto> $100 if you help us with this, email is eduardosoubihe@gmail.com
#maas 2019-10-02
<mup> Bug #1846286 opened: List authorisation tokens over websocket <MAAS:New> <https://launchpad.net/bugs/1846286>
<mup> Bug #1846348 opened: [ui, websocket-api] New data shape for node testing statuses <ui> <websocket-api> <MAAS:New> <https://launchpad.net/bugs/1846348>
#maas 2019-10-03
<mup> Bug #1846441 opened: [2.4.2] Custom layout is not applied with custom Image <sts> <MAAS:New> <https://launchpad.net/bugs/1846441>
<mup> Bug #1846457 opened: [2.6] Composing a VM with an IP address on its interface is in Auto Assign state after commissioning <MAAS:New> <https://launchpad.net/bugs/1846457>
<mup> Bug #1846441 changed: Custom layout is not applied with custom Image <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1846441>
<mup> Bug #1846441 opened: Custom layout is not applied with custom Image <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1846441>
<mup> Bug #1846441 changed: Custom layout is not applied with custom Image <sts> <MAAS:Invalid> <https://launchpad.net/bugs/1846441>
<TimRiceSE> Hi! Has some change been released to maas/images recently? Suddenly today we're unable to deploy machines with a bond interface configured... cloud-init seems to fail with a "Not all expected physical devices present" runtime error... Exact same server was deployed two days ago with the exact same configuration :/
#maas 2019-10-04
<mup> Bug #1846764 opened: Deploying many machines (like 30-50) at once via the UI leaves some in Ready state <MAAS:Triaged> <https://launchpad.net/bugs/1846764>
<mup> Bug #1846764 changed: Deploying many machines (like 30-50) at once via the UI leaves some in Ready state <MAAS:Triaged> <https://launchpad.net/bugs/1846764>
<mup> Bug #1846764 opened: Deploying many machines (like 30-50) at once via the UI leaves some in Ready state <MAAS:Triaged> <https://launchpad.net/bugs/1846764>
<fallenour> is this the maas channel? am I at the right place?
<fallenour> can maas use LXD to deploy instances?
<ltrager> fallenour: not yet, we're working on it :)
<fallenour> I dunno
<fallenour> I found this setting in lxd
<fallenour>  maas.api.key:
<fallenour> It makes me wonder
<fallenour> BTW, I love maas, you guys do incredible work with it
<fallenour> Ive used it for years since I found it.
<ltrager> fallenour: thanks, I'm not sure how LXD is using that yet
<ltrager> fallenour: we're started integration so it may be part of that
<fallenour> honestly, me either, but you guys respond a lot faster than they do, and I can beat on it a lot if you guys can walk me through some of the features of maas
<fallenour> I have a lotta hardware, a lot of spare time, and nothing to really do with either.
<fallenour> BTW, I initially deployed my maas unit as bare metal, but Id like to move it into an lxd container so I can shrink down my amount of active hardware. Is that supported?
<fallenour> and if so, whats the safest way for me to do that without causing an outage?
<ltrager> fallenour: I actually run MAAS in an LXD container normally for testing. I haven't moved one from metal to a container though
<ltrager> fallenour: what you'd have to do is move the data in Postgres
<ltrager> fallenour: /var/lib/maas and /etc/maas also has some configuration that would be good to move as well
<fallenour> Is it a postgres database?
<ltrager> fallenour: yes
<fallenour> If so you should be able to just put the postgres db in maas into ha mode
<fallenour> and simply replicate the data over to the maas system in the container
<fallenour> thats what Id recommend
<fallenour> if you want I can try that once I get this up and running
<fallenour> is the postgres db on maas a snap or a regular postgrs db install?
<ltrager> fallenour: that should work, I've just never done it. My test environment is simple enough I can just rebuild it quickly
<ltrager> fallenour: depends how you installed MAAS
<fallenour> snaps make me cry
<ltrager> fallenour: deb with deb
<ltrager> fallenour: If you use the MAAS snap we bundle postgres with the snap
<fallenour> mmk
<fallenour> Ill see if I cant put something together
<fallenour> OO!
<fallenour> While its on my mind, is it possible to make my maas box a local ubuntu apt repo? That way I can have a local package repo instead of eating bandwidth to do so
<ltrager> fallenour: by default MAAS is configured to use a squid proxy so it shouldn't be using to much bandwidth
<ltrager> fallenour: you can setup a mirror and configure MAAS to use it as well - https://maas.io/docs/package-repositories
<fallenour> hey sorry if I missed your response, my nvidia drivers crashed again. absolutely killing me
<mup> Bug #1833618 opened: MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken (USB pendrive in this case). <curtin:Invalid by rafaeldtinoco> <MAAS:In Progress by rafaeldtinoco> <sg3-utils (Ubuntu):In Progress by rafaeldtinoco> <sg3-utils (Ubuntu Bionic):In Progress by rafaeldtinoco>
<mup> <sg3-utils (Ubuntu Disco):In Progress by rafaeldtinoco> <sg3-utils (Ubuntu Eoan):In Progress by rafaeldtinoco> <https://launchpad.net/bugs/1833618>
<mup> Bug #1833618 changed: MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken (USB pendrive in this case). <curtin:Invalid by rafaeldtinoco> <MAAS:In Progress by rafaeldtinoco> <sg3-utils (Ubuntu):In Progress by rafaeldtinoco> <sg3-utils (Ubuntu Bionic):In Progress by rafaeldtinoco>
<mup> <sg3-utils (Ubuntu Disco):In Progress by rafaeldtinoco> <sg3-utils (Ubuntu Eoan):In Progress by rafaeldtinoco> <https://launchpad.net/bugs/1833618>
<mup> Bug #1833618 opened: MAAS can't deploy Ubuntu if ID_SERIAL of any block device is broken (USB pendrive in this case). <curtin:Invalid by rafaeldtinoco> <MAAS:In Progress by rafaeldtinoco> <sg3-utils (Ubuntu):In Progress by rafaeldtinoco> <sg3-utils (Ubuntu Bionic):In Progress by rafaeldtinoco>
<mup> <sg3-utils (Ubuntu Disco):In Progress by rafaeldtinoco> <sg3-utils (Ubuntu Eoan):In Progress by rafaeldtinoco> <https://launchpad.net/bugs/1833618>
<Fallenour> hey guys!
<Fallenour> Im back
<Fallenour> Does anyone use maas with juju? im having issues deploying lxd images atm. Im on 2.6.9, and Im having isues getting lxd containers to deploy
<Fallenour> it just constantly gives this error: failed to start machine 1/lxd/3 (acquiring LXD image: no matching image found), retrying in 10s (10 more attempts)
#maas 2019-10-05
<mup> Bug #1846897 opened: Feature request: Support signed SSH public keys/SSH certificates <MAAS:New> <https://launchpad.net/bugs/1846897>
<mup> Bug #1846897 changed: Feature request: Support signed SSH public keys/SSH certificates <MAAS:New> <https://launchpad.net/bugs/1846897>
<mup> Bug #1846897 opened: Feature request: Support signed SSH public keys/SSH certificates <MAAS:New> <https://launchpad.net/bugs/1846897>
#maas 2019-10-06
<mup> Bug #1846959 opened: MAAS GUI shows incorrectly displays false failure message from prior operation <gui> <MAAS:New> <https://launchpad.net/bugs/1846959>
