=== CyberJacob|Away is now known as CyberJacob === vladk|offline is now known as vladk === CyberJacob is now known as CyberJacob|Away [07:38] jtv, rvba: did either of you have a chance to look at my schema changes? [07:38] Not yet, not yet... [07:40] bigjools: not yet :/ [07:40] no worries, I need allenap to look too anyway :) [07:40] I made a load of changes to the design doc [07:56] There goes the lander again! Something is wrong. [07:56] Notice how many rabbit brokers get reported when this happens? I wonder if we're leaking any. [08:16] jtv: looks like you got a different error this time! FAIL: maasserver.tests.test_js.YUIUnitTestsLocal.test_YUI3_unit_tests [08:17] Gah! Three in one branch! [08:17] I suspect there is something wrong with the lander's machine. [08:17] Yes, that last one sounds like it might be related to memory leaks also. [08:17] After all, browser tests are normally the most taxing for the system. [08:18] Yep. [09:27] jtv: Branch landed \o/ [09:29] rvba: thanks — I hope the G figures it out because this is intowewwabwe [09:30] I shouldn't have named the branch after the Punic wars — those took several attempts too. [09:44] Unable to access MAAS nodes | http://askubuntu.com/q/473350 === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk [13:15] Hi all. I was wondering if someone could help me troubleshoot. I have a fresh maas install and I am getting a metric *ton* of oops logs when pxe tries to run on the nodes. I generated an sos report and put it here: https://drive.google.com/file/d/0BwO0l99GoKIxeUFxTHgxaGR0Sk0/edit?usp=sharing [13:15] but mostly the oops logs just repeat Failure: twisted.internet.error.TimeoutError: User timeout caused connection failure. [13:28] actually guys, I just got it. Thanks anyway [13:43] is it possible to have the MAAS management interface be different than the PXE/Provision interface? Say I want to control the box through the web interface through 172.16.103.92 and I want the cluster to work through a second ethernet interface with an ip of 10.0.0.1. Is this possible? [13:45] ophuk: if you're talking about power management through e.g. IPMI, then yes, it's a matter of choosing the node's "power address." [13:45] That's for turning the node on or off. Is that the type of control you have in mind? [13:46] AMT should be able to do the same, from what I hear, although by default it simply sits on the same interface. [13:46] jtv: not really. I have one box with two network interfaces, one public and one private. I want to be able to access the web interface through the public interface and I want the private interface there for the PXE, DHCP, and DNS for the cluster [13:47] Oh, we'er talking about the network interfaces _on the server_? [13:47] jtv: yes, sorry [13:47] Sorry, I thought you meant the network interfaces on the node. [13:47] Yes, those are meant to be different things (which can of course be the same NIC if you want). [13:48] jtv: no worries. I feel like it should be possible but my googling is coming up with anything. It could be because I'm looking for the wrong thing [13:48] I'll have to refresh my memory, but at a guess I would say that Apache (which runs the region controller) listens on all interfaces. [13:49] Yes, Apache listens on all interfaces. [13:49] In addition to that, there's a setting (again, rusty memory) for the URL which nodes should use when talking to the region controller. [13:49] ophuk: this should work out of the box. [13:50] The installation does have a question for "which interface on the region controller faces the nodes," but I don't think you typically get to see that. [13:50] I'll shut up for a bit and refresh that memory. [13:50] jtv: ok, so if I do sudo dpkg-reconfigure maas-cluster-controller and set that to the public IP, this allows me to view the web interface from the public IP. But how do I tell it to use the other interface for PXE, [13:51] It's sort of the other way around. [13:51] ophuk: you want to use the private IP for that. [13:51] What rvba said. [13:52] The UI is on any of the server's addresses; what needs to be known is which address is reachable for the nodes. [13:52] That's going to be the "private" address in your case. [13:52] rvba: ok - so that should be set to my internal IP. That's what I had originally but going to the public interface through a web browser wouldn't bring up the MAAS web ui. I was getting a server 500 error [13:52] That sounds like you _were_ reaching it, but something else went wrong. [13:52] What jtv said. [13:53] Gotta love symmetry. [13:53] jtv: yeah apache was working - my public IP is 172.16.103.92 but going to 172.16.103.92/MAAS/ wasn't. I just attributed that to the wrong interface [13:53] great minds think alike:) [13:53] Thank you... :) [13:54] Shall we have a look at the original problem then, and see if that solves the rest? [13:54] First place to look is the Apache error log. [13:54] jtv: yeah. I did the the reconfigure maas back to the internal IP [13:57] jtv: one sec - uploading a pastebin [13:57] jtv: rvba here you guys go - this is the error log, http://paste.ubuntu.com/7536756/ [13:57] I moved the original and cleared it out and tried to reconnect. That is what I got [13:58] Looks like the connection to RabbitMQ timed out. [13:59] Innnteresting. [14:00] also as I side not I'm not terribly attached to this install, if you feel like something might of gotten messed up during the install I can do it again - it just takes forever to reimport the isos [14:01] Well that can be solved by moving /var/lib/maas/boot-resources/cache out of the way somewhere, and moving it back into the new install. :) [14:02] jtv: well that is good to hear:) [14:05] ophuk: rabbit does tend to get upset when its IP address changes. I don't know if that's related. [14:06] (In this case that'd be a little silly since from the sound of it, its communication will be entirely local to the machine.) [14:06] jtv: it is quite possible because it originally pulled an IP from DHCP in the 172 range and then I had to change it to the public and private IP's. I don't remember an option during the install for setting up both nics [14:07] Ah! [14:07] Then I think that explains it. I've had some luck in the past just editing the rabbitmq config myself, but that wasa before doing much actual work with the setup. [14:08] jtv: hmm...ok. Where is this config file? [14:10] Looking for it... Curses, I've done this before. === vladk is now known as vladk|offline [14:11] jtv: I'm not sure where the piece of documentation that I'm writing belongs… The best place is probably the online documentation but you've documented the bootsource/bootsourceselection stuff in the import script's manpage so I'll have to redo/copy that part to add the relevant context… that's a bit suboptimal. [14:11] jtv: /etc/rabbitmq/rabbitmq.config ? I don't have that file [14:11] o.O [14:12] ophuk: same here, and I could have sworn I did just a few months back. [14:13] This may just be a rabbit change. Change is great, but one wishes the rest of the world would ask one's permission before making any. :) [14:13] jtv: lol - is it possible to do this during install, i.e. configure a static IP on both nics? [14:13] rvba: the man page should ideally cover only the command-line script, not the API/UI/CLI-triggered "proper" import. [14:14] jtv: agreed. What's suboptimal is that I'll have to copy part of the documentation from the script to the online doc. [14:15] ophuk: I seem to remember some confusion when reconfiguring afterwards, along the lines of "is this a URL or just an IP address?" What I did in the past was just grep /etc for my IP address, and see where the fixes need to be. [14:15] rvba: I don't suppose you can move it, and just make the script refer to the online version? [14:16] jtv: I'd like to keep the description of the config file's format in the manpage. [14:17] Ah, that part.. I agree, though it's been a bit of a pain tracking changes. === vladk|offline is now known as vladk === vladk is now known as vladk|offline === vladk|offline is now known as vladk [14:47] I am new to maas development. I am trying to make it so the maascli (i.e. I will need to make API changes) that allow us to do: maas node read [14:48] so we would be adding it so that we can read a node via the system_id (current) or hostname (new) [14:48] Curious where in the API this change would need to occur [14:49] jtv: ^^ [14:49] rvba: ^^ [14:50] * newell is happy to be doing it in the right channel finally [14:50] roaksoax: newell & I have been conversing about this, we just moved to a new channel. :) [14:50] heh [14:50] jtv: ah ok :) [14:51] newell: I think what you'd do at the moment is: maas nodes list hostname= [14:51] There must be some convenient command-line tool for manipulating/querying JSON... [14:51] jtv: so, what if we would like to change anything that needs system_id to also be possible to specify hostname [14:52] jtv: for example, we have been experiencing difficulty when adding/removing tags [14:52] roaksoax: then I'd do some broader planning first, just because it's a technical departure. [14:52] jtv: because every time we want to do it, we have to first obtain the system_id, so we would like to move to a world where I can just do API operations specifying a node's hostname, as it is unique [14:53] jtv: so, I'd definitely say that we should support both, system_id|hostname [14:53] First though, doesn't the Tag API let you do this? [14:53] Because this is a relationship between Tag and Node, and Node is the side where the scale is. [14:54] So maybe it'd make the most sense to say something like "add tag X to the nodes I specify as parameters." [14:54] At that point, there's no technical departure and you no longer need to loop over all nodes for mass updates. [14:55] jtv: so what do you think is more appropriate? Being able to do *any* node related operation (that currently needs system_id) with the hostname of the node instead (or alternatively) of using system_id? or for 1 or 2 methods being able to specify hostnames? [14:55] jtv: it is my perspective that management via the CLI would be highly improved by being able to specify a node's hostname instead of a system_id [14:56] jtv: because in the latter case, we are just doing a "work around" [14:56] I don't think so. It's just the more scalable way to phrase the request. [14:56] jtv: obviously, there will be places where we could experience issues, such as when we are to update a node's details [14:57] jtv: right, but right now, for any operation we want to make, we have to do the following: 1. obtain list of nodes. 2. grep for X node. 3. get its system_id. 4. perform operation [14:57] jtv: usability wise, that is *very* painful [14:57] Since hostnames really only make sense while a node is allocated, I guess another factor that comes into this is under which circumstances the operation makes sense on a given node. [14:58] I don't think you understand what I said. [14:58] jtv: they don't really only make sense when node is allocated. Hostnames make sense anytime because it is a nice way to identify a node, That's why DNS was created [14:59] Given that you can tag a bunch of nodes which you pass as parameters, there's nothing against passing them _as hostnames_. [15:00] jtv: right I got your point, however, what I'm trying to explain is that does solve the problem IMHO [15:00] jtv: right now, if I want to update a node, I have to do 1. 2. 3. and 4. above [15:00] if I want to update a tag, I have to do the same thing [15:00] No you don't, is my point. [15:00] there's many operations that require system_id to be passed [15:01] maas maas tag update-nodes my_tag add="" [15:01] What I'm saying is this needs some more careful thought since it's a technical departure. [15:01] how come when you're installing maas from a live CD do you don't have an option to set a static IP when creating a new MAAS server? [15:01] for me to do that, i need to 1. obtain list of nodes. 2. grep for X node. 3. get its system_id. 4.perform oeperaiton [15:01] And you're just pushing for the solution you have in mind, which may or may not be fine, but we can end up going down a blind alley if we run with the first suggestion. [15:01] jtv: so yes, in this particular case being able to specify hostname instead of system_id does make sense [15:02] roaksoax: not quite what you need to do. You can ask explicitly for just the node with the given hostname. [15:03] jtv: right, but you are still asking for a node with a given hostname [15:03] jtv: 1. ask for a node with a given hostname. 2. obtain its system_id. 3. perform any other operation I need [15:03] jtv: why can't we just go to 3 directly by using its hostname instead of obtaining a system_id [15:03] Think of it as OO if you prefer: you ask for the node, and you get its URL. [15:04] jtv: right, that might be the case, but the CLI is a usability thing [15:04] I see that you want to cut out that step, and I'm trying to help, but we do need to discuss the broader problem without pushing for the original suggestion. [15:05] jtv: CLI as to make admin's life easier, not more complicated [15:05] I'm trying to help you, but pushing and pushing instead of going through the design process makes that harder, not easier. [15:06] jtv: I'm not pushing, I'm trying to figure out what is needed to make this happen (which is going through the design process) [15:06] And I'm sorry for seemingly distracting you from your path to that solution, really I am. But experience taught us that it's something we have to do. [15:07] So the problem we have on the table is "I want it to be simpler to write changes to a node." Right? [15:13] roaksoax, ^^ [15:14] Not just, "I want it to be easier to manipulate tags on a node." [15:17] jtv: for every single node related operation (other than updating hostname), we need to obtain the system_id. That is affecting usability, so basically, I'd say that every single operation that currently requires a node's system_id, should accept the hostname (other than when updating a node's hostname of course) [15:19] That's much better, thanks. [15:36] roaksoax: the original architectural guideline for the API, which we've been lax about, was to focus on batch-oriented operations for scalability. Unfortunately of course scale and friendliness are not good friends, and we do very much want more friendliness in the CLI. [15:38] Some ideas for things we might want to look into in exploring your suggestion: [15:39] 1. How would hostnames show up in URLs? System IDs are "safe ASCII," hostnames have their own restrictions (including hopefully unicode at some point!) [15:39] I think it's probably OK of course, just noting to check. [15:40] 2. Is there any chance of ambiguity? [15:40] Need to check whether a hostname can also be a well-formed system ID or vice versa. [15:47] jtv: agreed. [15:47] newell: did you join the maas-devel ML? [15:47] newell: can you raise some discuss about it? [15:48] jtv: then we would be able to estimate work effort and we can decide from there if this is somethign we can fit this cycle [15:48] hmm not sure let me check [15:48] roaksoax, you want me to send out an email to the list summarizing the conversation here that you and jtv have mostly had? === vladk is now known as vladk|offline [15:52] newell, roaksoax: it's currently possible for the same identifier to be one node's system ID and another node's hostname... that's probably something we could just forbid if we have to. It's a UUID, so not a lot of risk. [15:53] newell: yeah an email talking about the possibility of allowing the API use for all operations that currently require the system_id, and some background, and the ideas raised by jtv [15:53] (Unless somebody tries to rename nodes as a way of getting one to take over another's role or something) [15:54] roaksoax, okay will do [15:54] Thanks. Mailing list is good for this. [15:54] newell: thank you [15:54] npo [15:54] jtv: and agreed, yes we need t consider various variables [15:54] np* [15:54] Yeah. Sorry for being difficult about it — we've had some really bad experiences going in to quickly with a solution, and had to force ourselves to do this. [15:55] It always feels a bit like sabotage... [15:55] But I'm glad that you're looking into user-friendliness improvements to the CLI! [15:56] We're OK on the hostname-in-URL issue, I think. Once we start supporting full unicode I guess we'll have to percent-encode — no drama. [15:59] jtv: oh not at all, please rest assured that me bringing this up was to simply raise some discussion about the issue to reach a solution that would benefit all of us and our users [17:11] I've got a node in a ready state but I can't click on start node - there is a blue message up top stating an adequately configured DHCP server can boot this node. I did the DHCP/DNS server stuff through the web UI, is there something I'm missing? [17:16] ophuk: did you add the SSH key for the user? [17:17] roaksoax: this part, maas-cli login maas http://10.98.0.13/MAAS/api/1.0 and then the key from the web UI? Yes [17:18] ophuk: nope [17:18] ophuk: go to the webui [17:18] and go to the User Settings [17:18] and add a SSH Key (public key) [17:18] so you can ssh into the nodes after deployment [17:23] roaksoax: ok, recommissioning and seeing what happens. [17:36] ophuk: there should be a hover message on the greyed-out button saying why you can't use it. [17:37] jtv: ok, I'll check it [17:40] jtv: it is working now [17:40] roaksoax: that fixed it i believe...thanks:) === vladk|offline is now known as vladk [18:36] Tips for troubleshooting IPMI in MAAS 1.6 | http://askubuntu.com/q/473587 [19:32] Hello... I'm trying to work with preseed_master by making a copy "amd64_generic_trusty" but when I do this I get an error on line 70 [19:33] name 'self' is not defined at line 70 column 3 in file /etc/maas/preseeds/amd64_generic_trusty [19:33] This is something easy :) [19:36] --- was reading about user-provided preseeds here http://maas.ubuntu.com/docs/development/preseeds.html [20:27] Figured it out. Apparently inheritance is odd. [21:43] okay something so basic, anyone can help [21:44] After I have a node allocated and built in MaaS how to I force a rebuild? [21:44] want the OS loaded again to test my preseed scripting. [21:45] jfarschman: stop and start the node [21:45] blake_r_: but how do I put it in a state where it knows to rebuild? [21:46] it should not rebuild on every reboot. [21:46] jfarschman: what do you mean rebuild? [21:46] reinstall the OS, invoking the preseed modifications I just made [21:46] jfarschman: when you stop a node, it is considered no longer in use [21:46] jfarschman: when you start it again it will be re-installed [21:46] I did an install yesterday that was generic... now I need to do an install that adds puppet [21:46] oh... hahaha. [21:47] jfarschman: start means turn on, and give me a clean installed system [21:47] if it still exists in maas, the os will not be reloaded. [21:47] jfarschman: so be careful with that stop button, :) [21:48] otherwise a power failure would cause every server to reload OS [21:48] designated: if the node is allocated and you click stop it will power off, and become unallocated [21:48] designated: next time you click start it will allocate to you and install [21:48] yea... I just assumed it was going to use IPMI and shut it down for me. but I like this method. [21:48] designated: if you restart the node, from the machine it will reboot, and will not reinstall [21:49] blake_r_, correct, it must be issued from within maas. simply power cycling the node will not kick off a new OS load. [21:49] jfarschman: yes it will use IPMI and power off the node [21:50] designated: power cycling? if you use ipmi directly or un-plug it and back in, it will nto re-install [21:50] designated: if you use maas to control the node, it will re-install [21:51] designated: so in a data center and the power goes off, they will all restart to where they were before power cycle [21:51] blake_r_: that language, "stop selected nodes" is a little deceptive. Perhaps deprovision [21:51] blake_r_, we're saying to same thing. I was trying to make sure jfarschman understood that maas had to be used, simply power cycling the node would not initiate OS load. [21:52] jfarschman, exactly [21:53] jfarschman: https://bugs.launchpad.net/maas/+bug/1311224 [21:53] Ubuntu bug 1311224 in MAAS ""Start node" and "Stop node" terminology confusing to newbies" [Medium,Triaged] [21:54] the maas documentation isn't entirely clear on a lot of things, like advanced networking for example. How do you bond interfaces for LACP and use VLAN tagging? [21:54] designated: good question? [21:54] designated: idk, :) [21:54] designated: never done that! [21:55] some things in the documentation are outright wrong. [21:55] designated: are you using the correct version of the documentation, for your version [21:55] blake_r_, yes [21:55] designated: please create a bug, if its wrong [21:56] I've been hesitant because this is the most active I have ever seen this IRC channel. doesn't seem like a very active community, figured it was a waste of time. [21:56] designated: we are very active, on launchpad [21:57] blake_r_, ahh i see [21:57] designated: will try to be more on here [21:59] having access to more advanced networking features, like LAGG and VLAN tagging, as well as the ability to adjust disk partitions without having to manually write a preseed are essential to success for a project like this, especially if you plan on using juju to deploy software. juju for example works great when deploying to a cloud but when deploying to bare metal, you need access to hardware, like multiple NICS or bonded and trunked inte [21:59] rfaces. [22:00] designated: I know that VLAN tagging is supported, just don't know to what extent and the ability of MAAS [22:00] i tried deploying openstack on juju and maas, it was a nightmare because of limited network support and disk partitioning options. who deploys openstack on hardware with a single interface? [22:01] designated: juju neutron uses a second nic interface [22:01] designated: adjust for partition would be hard, who would you handle different size hard drives? [22:01] blake_r_, if the community is truly active, I would definitely be interested in getting involved. I just don't want to adopt something that isn't actively developed and wind up with problem after problem... [22:02] designated: it is actively developed [22:02] designated: and a key project for canonical [22:03] i did write a bash script to solve the problem of mac to name nic mapping inconsistencies across a pool of servers that gets pulled in during preseed, that was a huge issue i was having when trying to use juju with maas [22:03] blake_r_, i believe it is, it's just knowing how to get ahold of the developers to interact with them :) [22:03] One more.... My provisioning of Trusty stops and [!!] Partition Disks and asks me to make a choice. This is normal? [22:03] jfarschman: no it should not ask you any questions [22:04] jfarschman: you must have not answered all of the questions required in your preseed [22:04] jfarschman, i would think if the disk partitioning portion of the preseed was complete, it wouldn't do that. I've never had that issue. [22:04] designated: talking to one now! [22:04] It did on the first install with the default preseed and now with my modified one. I'll work it out. Thanks [22:07] blake_r_, i thought maas 1.6 was supposed to be the default on 14.04 but it installs 1.5 [22:07] designated: 1.6 is trunk current development [22:07] designated: trusty is 1.5 [22:07] designated: we do backport hardware enablement and critical fixes to 1.5 [22:08] designated: you can install from a ppa if you want 1.6 [22:08] designated: it is not recommended as its in active development [22:08] designated: it can be broken [22:09] designated: depending on when you use it, best to stick with 1.5 unless you really need something from 1.6 [22:09] blake_r_: I appreciate that. When learning a new tool, predictability is important [22:09] jfarschman: np === blake_r_ is now known as blake_r === CyberJacob|Away is now known as CyberJacob [22:13] blake_r, i thought we would be able to bond interfaces in 1.5 according to https://blueprints.launchpad.net/maas/+spec/t-cloud-maas-advanced-networking [22:14] blake_r, I haven't seen anything in the documentation explaining how to do this. [22:15] blake_r, the ability to do a per node preseed would be extremely beneficial, or the ability to group nodes for a specific preseed, like using a certain preseed based on assigned tags. [22:15] designated: https://bugs.launchpad.net/maas/+bug/1254755 [22:16] Ubuntu bug 1254755 in MAAS "Feature Request - bonding on MAAS provisioned nodes" [Wishlist,Triaged] [22:16] designated: you can do a per node preseed [22:17] designated: so in /etc/maas/preseeds [22:17] designated: if you make a file names liek this [22:17] designated: curtin_amd64_generic_trusty_hostname [22:17] designated: that is for a custom preseed that uses the curtin install known as "fastpath" [22:18] designated: so its {prefix}_{node_arch}_{node_subarch}_{release}_{node_name} [22:18] designated: by tag is not possible, but please make a bug for that, I agree that would be helpful [22:19] blake_r, it's to address issues like my storage nodes for example would need different network configuration as well as disk partitioning than say a server that simply runs an application. === vladk is now known as vladk|offline [22:22] designated: I'm using puppet for that sort of thing. When the config gets complicated and needs to be granular, puppet. [22:23] jfarschman, that's great but I might as well manually configure those items, if juju can't request a resource from maas that is configured ready to go. [22:23] designated: you can also use juju, to do that as well [22:23] /query/win 3 [22:25] blake_r, didn't think of juju handling that configuration, i suppose it would have to happen before installing/configuring any software. [22:25] designated: fair enough, but not realistic yet. One tool to rule them all is still a dream. [22:25] designated: yeah, you can modify your charm to perform the configuration [22:25] designated: of create a subordinate charm to do it [22:26] crowbar supposedly does all of that, but their dev community doesn't seem very active. lol I can't even get crowbar up and running on 14.04 and no one in their irc channel ever speaks... [22:27] plus from what I heard, dell stopped developing it. [22:29] blake_r, i would prefer to use maas/juju, it seems a lot more versatile, the hesitation was quite simply, I couldn't ever get ahold of anyone to troubleshoot/network with and the documentation hasn't been helpful in a lot of situations. === CyberJacob is now known as CyberJacob|Away [22:34] designated: feel free to tag me in a question so I see it, and I can respond [22:34] blake_r, thanks for the chat. you have inspired me to give maas/juju another chance. [22:35] designated: cool, let me know, if you need any help