=== Jeff_ is now known as Guest70936 [03:34] Hey everyone. I'm having a very weird occurance with MaaS. For some reason the TFTP server gets a SIGTERM signal. In the /var/log/maas/pserv.log I find lines like this: "2014-04-01 23:16:51-0400 [-] Received SIGTERM, shutting down." just randomly. Can anyone help me out with this or at least tell me how to turn the tftp server back on? [03:35] Guest70936: can you paste the whole log somewhere please [03:36] And, are you sure that SIGTERM isn't just from an upgrade, reboot, or installation of other related packages? [03:37] and which version of maas are you using [03:39] bigjools: This is the last 1000 lines of pserv.log: http://pastebin.com/p6prSEHd [03:39] jtv: I mean I am simply doing a juju bootstrap and then juju status -e maas [03:40] Guest70936: it looks like a regular shutdown to me [03:40] So when all the nodes in MaaS go back to Ready it shuts down the TFTP server? [03:40] bigjools: ^^ [03:41] Oh you mean it looks like a reboot or shutdown of the computeR? [03:41] well that is not supposed to happen, no, but SIGTERM is sent by upstart when t wants to close it down [03:41] and it happens 35 minutes after the TFTP session finished [03:42] so yes it looks like a reboot [03:42] bigjools: Ok thanks. Yeah I noticed that I just have had some weird problems with servers being able to pxe boot and the problem is always a pxe boot timeout. Could it have to do with the switches on the network and "portfast"? [03:43] Also after bootstrapping juju on the maas controller and setting nameserver to be 127.0.0.1 a juju status -e maas won't work either. [03:44] Guest70936: ok first thing, it could be portfast, or it could be something else on the network. I sometimes see timeouts when mixing gigabit and 100M network switches [03:44] Guest70936: what does "won't work" mean, what happens? [03:47] bigjools: Alright we are mixing a 1Gig with 10Gig? And won't work means it hangs for a while and then says something along the lines of "Can't connect to environment." I also looked in the .juju/environment/maas.yaml and noticed there were a couple of ports like 17010 and 31017 or something similar and after checking the open ports using netstat I couldn't find that either of the ports were open. Could that have something to do with [03:49] Guest70936: yes you need to open ports for mongo and the jujud [03:50] bigjools: I am not running a firewall and hitting the MaaS server with Juju locally. [03:50] ok then I expect juju is not installed on the bootstrap node [03:51] bigjools: juju is installed locally on the MaaS controller and I have setup the environments.yaml pointing to 127.0.0.1 and with the MaaS api key, etc... [03:52] Oh on the bootstrap node. [03:52] How do I push it there? [03:52] Does the juju bootstrap command do that? [03:59] bigjools: Any ideas? [03:59] Guest70936: juju bootstrap will start up a node in MAAS and install juju on it [03:59] did you do that already? [04:00] bigjools: Yeah I just did it and I guess the previous times I just didn't wait long enough. I'm waiting to see the server hit the tftp server. [04:00] right, it can take a long time [04:00] which version of maas? [04:01] bigjools: Is there a command I can run to find that? [04:01] dpkg -l maas [04:02] bigjools: Thanks, 1.4+bzr1693+dfsg-0ubuntu [04:02] Guest70936: ok you're using saucy? [04:02] 13.10, that is [04:02] Yes 13.10? [04:02] Yes. [04:03] if you are not using the fast installer, it takes 20-30 minutes for the bootstrap to complete [04:03] can you see the console of the node it is booting? [04:04] bigjools: Ok sounds good. No I am currently not in the office. I will be able to see it tomorrow. One last question for you: Should the boot order of the server always start pxe first? [04:05] Of the nodes? [04:05] They should prefer PXE over hard disk. [04:05] Guest70936: the node will make a DHCP request then a PXE request [04:05] Of course you may still want CD, USB etc. first, but that's up to you. [04:05] all done in the BIOS [04:05] oh yes, what jtv said, PXE first, always [04:06] bigjools, jtv: Thanks! Is the fast installer recommended? [04:06] Guest70936: it's very rough and ready in 13.10 but you can try it [04:06] there should be a button on the node page to enable it [04:07] It's much, much faster. [04:07] yeah, takes ~3 minutes to install the OS instead of 30, then juju takes around 5-10 to install itself [04:07] The non-fast installer depends on the speed of your connection to the archive; the fast one basically doesn't. [04:07] jtv: it still does actually [04:07] "Basically" [04:07] :) [04:08] The "classic" installer takes hours for me; the fast-path installer takes minutes. [04:09] bigjools, jtv: This is by far the most I've understood this system; even after reading all the documentation. Thanks again! Ok and one thing that I noticed is that the classic installer won't install over a previously installed system. But I would think that it should be able to wipe away the previous one and reinstall. Am I correct in this thinking? [04:10] Guest70936: no it should wipe it [04:10] Guest70936: it means you're doing something wrong [04:10] Yes, it does wipe a previously installed system. I've got one doing that now. [04:10] But it won't install while the node is still allocated. [04:10] Make sure the node is in the Ready state. [04:10] you are either booting a node manually and it boots the old installation, or you're not letting maas control the node [04:10] MAAS only installs a node when deploying it, i.e. when somebody's allocated it. [04:11] bigjools: Interesting. It would give random errors when installing over something saying it couldn't write the kernel and sometimes just saying that it couldn't write a file. We are writing the system on SD cards on blades. [04:11] Guest70936: oh weird. [04:11] Problem with the SD drive being mounted too late in the process, or something along those lines..? [04:11] Guest70936: this is not maas-specific, it is running debian installer at that point [04:11] bigjools: Ok thanks I'll look into other things. [04:11] Guest70936: I have seen it fail when there are RAID partitions [04:12] bigjools: There are raid partitions. [04:12] Oh, another wild guess: maybe the SD drive is getting a different device name in different OS releases. [04:12] aha! [04:12] bigjools: I haven't setup the preseed file yet to mount it properly. [04:12] existing RAID partitions confuse d-i [04:12] I had to delete all of mine [04:12] there is work coming in the near future to handle custom partitioning [04:12] bigjools: The raid is simply there as a virtual drive, it isn't partitions. [04:12] partitioned*. [04:13] yeah any raid present confuses d-i [04:13] because the kernel tries to mount it [04:13] bigjools: Oh ok. So how do I deal with that? [04:13] delete the raid [04:13] bigjools: Ok and then I guess let software run it? I can't use hardware RAID currently? [04:13] better still, go to #ubuntu-server and ask them :) [04:14] oh is it hardware raid? [04:14] I am talking about software raid [04:14] bigjools: Yeah it's hardware RAID that has a virtual drive setup. [04:14] ok [04:14] check with the server guys, this is beyond my knowledge here [04:15] bigjools: Ok. Thanks again for all your help! [04:15] jtv: Thanks for your help! [04:15] no worries [04:15] Have a good night! [04:15] afternoon here :) [04:15] Have a good day then! [04:35] 1.5 release branch just cut [04:38] bigjools: you remember a month or so ago i was working on a test using maas to deploy desktops and you wanted me to let you know how it went? [04:38] ging: yes! [04:38] well it seems although the maas part worked out really well, it's not going ahead [04:39] the company has gone for windows 7 because they could not get a lot of thier spreadsheets and things to work with open office [04:39] ah darn it [04:40] well, thanks for the feedback, it's good to hear at least maas went well [04:40] they liked maas for deploying nodes, it worked well on the 5 test machines we used [04:41] What a shame. [04:46] we set them up some rdesktop app stuff running on a windows server for the windows things they need, but it wasn't good enough on window 2003 server that they had, and paying out for server 2007 because of microsofts complicated licensing where you seem to need to have licenses per client etc, it doesn't cost much more to just have all windows 7 licenses [04:53] prob would be cheaper to redesign those spreadsheets to work with Open Office [05:22] probably be even cheaper to completely replace them with something more efficent and sack 3/4 of thier work force === CyberJacob|Away is now known as CyberJacob === CyberJacob is now known as CyberJacob|Away [07:56] jtv, are you around? [07:56] I am. [07:57] jtv, just a quick question re bug 1299114 [07:57] bug 1299114 in MAAS "'ValidationError' object has no attribute 'error_dict' when creating a network" [Critical,Fix committed] https://launchpad.net/bugs/1299114 [07:57] Shoot. [07:57] jtv, maybe i'm getting it wrong but am I supposed to be able to add both a network (i.e. 192.168.50.0/24) and a vlan on that same range (192.168.50.0/30) ? [07:58] No. [07:59] jtv, looking at the docs, somewhere was mentioned that maas should know about your networks/vlans so that the cluster controller can talk to the nodes on the configured netwokrs [07:59] docs/networks.rst in the source tree. [08:00] MAAS really has two notions of "network" that are completely separate for now: [08:00] jtv, so with a setup like this (on the controller): eth0=192.168.50.0/24,managed dhcp+dns, eth1=192.168.111.0/24, also managed dhcp+dns [08:01] That's one of the two notions of network. But currently you're defining the other type of network. [08:01] (Yes, the two should learn more from each other) [08:01] jtv, I have 2 networks defined: vlan0:192.168.50.0/30 tag:42 and net1=192.168.111.0/24, i though I also have to define net0=192.168.50.0/24 so maas will know all of them that can be bound to NICs inside a node [08:02] MAAS doesn't support defining different networks with identical or overlapping IP ranges. [08:03] When you allocate a node using a network constraint, you can specify an IP address on that network. The region controller has to be able to figure out exactly which network that means. [08:03] It's something that could be changed in the future, but we figured having IP addresses that could be in several networks would be a recipe for confusion anyway. [08:04] jtv, I see, so you're supposed to add only those networks which are not known by maas (i.e. other than the first NIC on the node, which gets its ip from maas dhcp) [08:04] (Similarly, it's perfectly possible technically to have two separate networks with the same VLAN tag, or one VLAN with different tags as seen on different parts of the network, but we didn't allow either) [08:04] No, it's fine to add networks that MAAS knows about. [08:05] But this goes back to there being two completely separate notions of "network." [08:05] and now you *have* to link a network to all the nodes that need to be on it by mac address, right? [08:05] One is: a network as attached to a cluster controller interface. We're really only interested in these when the cluster controller is supposed to manage them. [08:06] right, that's easy [08:06] The other is: a network as attached to nodes. We're really only interested in these for the purpose of allocation constraints. [08:06] For those, yes, if you want the constraints to work, you have to tell MAAS what nodes attach to what network. [08:06] (And obviously we'll want to automate that at some point.) [08:07] right [08:07] i guess what i'm really asking is: can i rely on the following assumption [08:07] Now, it's perfectly fine to define e.g. your 192.168.111.0/24 network as a network, and attach nodes to it. [08:07] That way you have the network defined in both senses. No worries. [08:08] What you can't have is: [08:08] any node with networks attached will also have mac addresses specified in maas [08:08] Any node has MAC addresses specified in MAAS. (Unless you delete them, but...) [08:09] However, MAAS does not yet auto-discover what MAC addresses are attached to which networks. [08:09] What you can't have is: [08:09] so if you haven't configured the correct macs that's too bad - we won't be able to find them by mac on boot and bring the NICs up appropriately [08:09] sorry, i go on :) [08:10] s/i// [08:10] The node's MACs get auto-discovered. [08:10] You can boot [08:10] and the NICs will come up, and DHCP. [08:10] But you need to configure "MAC x is attached to network y" in order for network-based placement constraints to work. [08:11] are any extra nics supposed to be using dhcp? because i can't see how i can decide what static ips to assign.. [08:12] Well technically, what happens after that is up to the preseed/userdata I think. [08:13] ok, that's a relief - i can rely on mac addresses being known by maas (and get-able from the networks list) [08:14] the only issue that remains is how to know which mac belongs to which physical NIC at cloudinit time [08:14] You can get MACs based on the node. But you can only get them from the networks list if you first tell MAAS which MAC is on which network. [08:15] yeah, i think we can safely assume the user configured maas correctly (at least for the MVP phase of nets/vlans support now) [08:16] As far as we're concerned though, when it comes to the core MAAS codebase, configuring networks is entirely optional. [08:17] right [08:18] I was looking at the xml info that's gathered at commissioning time with all the hardware - is this the same format regardless of the type of node? (i.e. physical or vm) [08:18] Should be, though of course the actual data might differ. It's just lshw output. [08:19] my idea is to get the mapping NIC<->MAC address from there before starting the node so I can generate the appropriate cloudinit scripts [08:20] It'd be nice to be able to do it at runtime. We have seen some cases where NICs still get different names on different boots. [08:20] We've been thinking along the lines of "MAC is a UUID for a network interface." [08:21] So forget about eth0 etc; the MAC identifies the NIC both within the system and throughout the MAAS. [08:21] we'll do it at runtime (or rather at machine agent startup) eventually, but for now they'll be configured at boot (and set to start on reboot) once [08:22] Should do for most cases. [08:22] yeah, that's nice but I need to identify the nic for ifup/down (unless you can refer to nics by their macs - have to check) [08:22] Would be nice if more networking tools supported MAC-based addressing of interfaces, no? It's not like walking through a list of 6 items and comparing data structures is particularly expensive any more. [08:23] :) agreed [08:53] jtv: Approved your branch, finally. Sorry for the day. [08:53] jtv: about bug 1300294, the idea that, for some reason, these nodes might be "owned" and in the 'ready' state is a good idea, I vaguely remember we saw a bug about that once. [08:53] bug 1300294 in MAAS "Standard users cannot see all unallocated nodes." [High,Incomplete] https://launchpad.net/bugs/1300294 [08:55] Thanks gmb. [08:55] rvba: yes, but I ruled that out with reasonable confidence — we have a long-standing assertion against it. [08:55] I do wonder, however, whether it might be something along the lines of "I'm listing nodes within the Juju environment" [08:59] jtv: yeah, might be juju-related somehow. That's why I asked them to try to see if they can see the same problem when using the API to list the nodes. [09:00] allenap: btw, I see Julian asking about that yesterday: Blake confirmed that the UEFI stuff has been ported to the new import script correct? [09:01] rvba: Yes, I’m pretty sure it has. [09:01] blake_r: ^ Please can you confirm when you come online? [09:03] blake_r: also, can you please confirm that you've QAed that change with the daily PPA? [09:03] s/daily PPA/daily package/ [09:12] Hi guys. I'm back from the sprint. I plan to work on some functional tests for our new script to pull boot resources. Like 'here is a tricky metadata which boot resources you'd pull'. Does it look reasonable to work on this? [09:16] Hi strikov [09:17] Yes, that would be useful. The important thing with this one is to make sure the tests run at the right level of integration. [09:18] We're already writing end-to-end tests; it would be good to have tests that exercise and verify specific behaviours of the various components. [09:19] jtv: as a first goal I plan to verify that script pulls correct product if multiple available for a specific arch/subarch/release/label and that it pulls correct version of the product if we specify some tricky label like 'beta2' [09:21] That sounds good — we do not have as much understanding of the simplestreams part as we'd like. [09:21] Normally I'd say "don't bother testing what should already be tested in libraries we're using," but in this case there are enough unknowns in the interaction. [09:21] For most tests though, I would hope to be able to stub out most of the simplestreams code. [09:22] I'm currently writing a test that simulates a successful import based on controlled simplestreams import data — but I'd hate to have too much of that kind of detail in there! [09:32] jtv: Right, I just want to test the code which makes decision which image to pull, this logic is inside the script (RepoDumper + boot_merge) [09:33] We'd be most grateful for tests and documentation there! [10:52] Yay! Big deep end-to-end test of the main boot-resources import function is up for review: https://code.launchpad.net/~jtv/maas/import-script-top-level-unit-test/+merge/213809 [10:56] jtv: wow, looks good [10:58] Thanks. :) [10:59] There are probably helpers in simplestreams that could make this easier, but I never quite found my way in the documentation. [10:59] jtv: I wonder if I need to extend this code for my own check or I want to create a separate one [11:00] jtv: but anyway, let me come up with my tests and we decide how to integrate them [11:02] Better to have a separate one, I think — assuming you can avoid most of the complexity. [11:02] Maybe the best thing to do is to specify, in the form of tests, what exactly should be in the meta file for given inputs — without any simplestreams interaction. [11:03] "For inputs X, I expect output Y." [11:03] Best form of test. :) [11:07] And with that, I'm out. Bye! [11:39] conversion from commencing to ready state in maas | http://askubuntu.com/q/442350 [13:51] I'm trying to run test.pserv. It complains about sst unavailable (but I ran make install-dependencies before). Do I need to install this test dependencies manually or we have some other way to do it automatically? [14:01] strikov: Hi Oleg. make install-dependencies install the prod and run-time dependencies, 'make' should install the dev dependencies (sst is one of these). [14:03] strikov: After `make` has been run, you should have a directory named 'eggs' which should contain sst-0.2.2-py2.7.egg. Isn't that the case? [14:11] rvba: Thanks. Yeah I have this egg installed. Is it okay to run bin/test.pserv manually with PYTHONPATH in the env? [14:12] strikov: yes, running bin/test.pserv is the way to do it. [14:13] rvba: I'm getting the following error: http://pastebin.ubuntu.com/7194420/ [14:13] rvba: while this module is definitely exists and in the PYTHONPATH [14:14] strikov: that's because you have python-maas-client installed. [14:14] strikov: you need to remove all the maas packages to develop maas (otherwise you get that kind of conflicts): sudo apt-get remove python-maas-client [14:14] rvba: Oh, how did you manage to figure this out from the error log? :) THanks [14:15] strikov: I know that error :). This should probably be in the FAQ ;) [14:17] rvba: now all the tests pass, thanks! [14:18] np [15:50] gmb: thanks for adding a lander for the new branch! [15:54] I am running MAAS on 13.10 and believe I am running into https://bugs.launchpad.net/maas/+bug/1237364 [15:54] Ubuntu bug 1237364 in maas (Ubuntu) "Commissioning with a Saucy image sets node status to "Failed tests"" [Critical,Fix released] [15:54] anyone else had to deal with this? [16:07] Wondering if someone could clue me in on why maas reinstalls on stop/start?! [16:08] Is this by design or am I doing something wrong? [16:14] Ruetobas maybe you can help? [16:16] onezerohosting: Can you clarify what you mean by “maas reinstalls on stop/start”? [16:17] "Stop node" and "Start node" on the Nodes listing of maas [16:17] It seems to me that it should simply shut down a machine on stop and when restarted power it back up. However I'm getting a complete re-install. [16:18] Only way to avoid this behavior so far is to set to not boot from PXE however I'd like to leave that setting alone. [16:23] onezerohosting: That’s by design; when you stop a node it goes back into the set of nodes available for other users. When they start it, they’ll start with a clean slate. Alternatively, you’re using Juju to orchestrate deployments, in which case Juju expects the machien to be pristine when it gets access to it. [16:45] onezerohosting: Does that make sense? [16:46] Any one know how to get a node to commission? No matter what ver of Ubuntu I use to commission, I get: failed [2/5] ( 00-maas-01-lshw 00-maas-02-virtuality) [16:51] gmb. Thanks. Yes it does make sense. However the one caveat I'm running into (and the reason I was restarting to begin with) is DNS. [16:52] Maas is managing DNS and in order to change a hosts name, it seems that a restart is required per the web console. [16:52] Of course I can change it from the command line but I wanted the hostname to be reflected at the web interface. Is there any way to do this? [16:52] gmb, rvba: Got time for a shortish review? https://code.launchpad.net/~allenap/maas/dont-crash-when-selecting-fpi--bug-1293676/+merge/213887 [16:53] When you try to edit a node name while in use you get the following error, "Can't change hostname to compute2d: node is in use." [16:53] allenap: Sure. [16:53] gmb: Thanks :) [16:54] blahRus... I've only been able to get Raring to commission properly. Otherwise I get the same error. [16:54] onezerohosting: So, MAAS *has* to manage the hostname… changing it on the command line on the node doesn’t really make sense in a MAAS context. [16:55] Didn't mean command line on the node... meant within Maas' DNS server config... zone files. [16:56] onezerohosting: If MAAS is managing DNS you don’t need to edit the zone files manually. In fact that could theoreticaly cause problems down the line. [16:57] So how do I change a nodes name per MAAS after it's already running? [16:58] onezerohosting: You don’t. The workflow would typically be: [16:58] 1. Enlist (and accept the node) [16:59] 2. Update its details (cluster, zone, power type etc.) [16:59] (etc. include FQDN) [16:59] 3. Start the node [16:59] onezerohosting: that's what I am trying to load now :/ [17:00] Got ya. Thanks gmb. [17:00] onezerohosting: No worries. [17:05] this seems to be the error: http://paste.ubuntu.com/7195086/ [17:07] blahRus: your node(s) didn't manage to install the packages needed for commissioning. [17:09] rvba: hurm… does it sound like I missed a step? [17:10] rvba: do the HD on the systems need to be empty? [17:10] blahRus: not at all. [17:10] blahRus: maybe a network config step [17:10] rvba: the boxes are pxe'ing just fine [17:10] blahRus: the nodes use the region's proxy to download packages. [17:11] blahRus: the packages are downloaded from the ubuntu archive (the pxe request doesn't step out of your network). [17:13] rvba: perfect, they are getting publicly accessible IPs, maybe the DNS piece isn't running right on maas [17:14] maas-dns is installed, does it run as a service? [17:14] blahRus: try to get the logs for the other commissioning scripts. The earlier scripts install the required package. Let's double check that these didn't fail. [17:14] packages* [17:15] alright, bind9 is running [17:16] rvba: http://paste.ubuntu.com/7195139/ [17:19] blahRus: looking… [17:19] ty [17:22] blahRus: http://paste.ubuntu.com/7195161/ [17:22] Confirmed, your node couldn't reach the archive. [17:22] got it, so the node PXE'ed couldn't get to ubuntu archive [17:22] Right. [17:23] blahRus: Unless you changed the proxy setting in MAAS, your nodes are using the region's proxy (configured by MAAS) to access the archives. [17:24] blahRus: Please have a look at the region's proxy's log to see if you can spot the problem. [17:25] i'm getting 'The DHCP leases file does not exist' and it tells me to install maas-dhcp, but maas-dhcp is installed [17:25] what process should i look for to check if dhcp is operating? my pxe requests are not getting responses [17:26] xmltok: sudo service maas-dhcp-server status [17:26] xmltok: did you configure the DHCP service on the cluster's page? [17:26] i guess not [17:27] i have managed dhcp set up for the cluster [17:27] i may have just needed to start the process [17:28] xmltok: it's restarted by MAAS each time the settings are changed. [17:29] hmm, lost my connection. so the maas-dhcp-server service wasn't running but it is now [17:31] ok its pxe'ing but i guess some of the pxe files didnt import, ill try that again [17:39] allenap: nice use of the view infrastructure to fix bug 1293676 ;). Don't forget to backport your fix to 1.5. [17:39] bug 1293676 in MAAS "internal server error when marking nodes as using fast-path installer" [Critical,In progress] https://launchpad.net/bugs/1293676 [17:59] rvba: how do I set BIND to allow the lookups to archive, we don't' need to use a proxy in our case. [18:00] cool, the node registered with maas [18:00] can i only deploy nodes using juju? === roadmr is now known as roadmr_afk [18:07] blahRus1: MAAS only adds its own section to BIND's config. What I mean is that you should be able to configure BIND anyway you like as long as you keep MAAS' section in the config. [18:08] rvba: think we got past that, but now I am getting: http://paste.ubuntu.com/7195367/ [18:08] xmltok_: no, using the API or the UI you can deploy nodes (as in: get machines up and ready). Juju simply automates the next step: deploying _services_ onto the nodes. [18:11] rvba: is it trying to default to ipv6 (which we are not serving up at all) [18:11] blahRus1: I think you want to configure an upstream DNS server for queries outside the real of MAAS. This is done via the 'upstream_dns' config option (available on the 'settings' page, at the bottom — or using the CLI/API). [18:12] rvba: not seeing that anywhere on the settings page [18:13] blahRus1: arg, you must be using an earlier version then. Hang on. [18:13] we tired the beta in 14.04 but didn't have much luck so we we back 13.10 [18:14] blahRus1: you need to add http://paste.ubuntu.com/7195395/ to your DNS config. [18:14] lmk if we should be running via a different mirror [18:14] im getting failures trying to power up the nodes now, i think its probably a firewall problem [18:15] rvba: does that go in options? /etc/bind/named.conf.options [18:15] blahRus1: yes [18:22] rvba: bind was ok with the config, server still boots up, then ubuntu haults it [18:25] blahRus1: can you resolve archive.ubuntu.com using MAAS' DNS server? [18:25] ya [18:25] rvba: http://goo.gl/3REiXv [18:26] blahRus1: is this at the end of the commissioning cycle? [18:27] current status is Declared [18:27] Not yet commissioned then. [18:27] so click commission now, and power back up? [18:28] If you're using IPMI (or any other power mechanism supported by MAAS), you don't need to power up your nodes manually. [18:28] Just click "commission." [18:29] alright, so it's normal to halt before commission right? [18:30] blahRus1: yes [18:30] Hello everyone I am using the fast installer to load a ubuntu image on to some blades using MaaS. I am having a problem after the fast installer installs the system on reboot when it tries to PXE boot it does what it's supposed to and drops to the local disk. But when it does it comes up with an error that says "boot sector signature not found". [18:30] smoser: care to give us a hand here? ^ [18:32] Jeffrey_, what version of maas / curtin ? [18:32] rvba: latest http://paste.ubuntu.com/7195472/ [18:32] What's more interesting is that after I see that error if I boot the computer selecting the internal SD card it will boot properly. === CyberJacob|Away is now known as CyberJacob [18:33] smoser: 1.4+bzr1693+dfsg-0ubuntu [18:35] blahRus1: There is still a problem apparently: http://paste.ubuntu.com/7195483/. [18:35] E: There are problems and -y was used without --force-yes [18:35] Traceback (most recent call last): [18:35] File "/tmp/user_data.sh.DMFkVb/combase64: invalid input [18:35] (While installing the lldpd package) [18:35] Jeffrey_, you're missing part of that string. [18:35] what version of curtin ? [18:36] you could be seeing https://bugs.launchpad.net/curtin/+bug/1244026 [18:36] Ubuntu bug 1244026 in curtin "curtin block-meta selects first entry in unordered list" [Undecided,Fix committed] [18:36] but that should be fixed if you have all updates applied to saucy [18:36] smoser: I'm using dpkg -l maas/curtin and I got that for the version of maas and it says that curtin is not installed. [18:37] Jeffrey_, dpkg-query --show python-curtin [18:37] smoser: python-curtin 0.1.0~bzr94-0ubuntu1.13.10.1 [18:38] maas 1.4+bzr1693+dfsg-0ubuntu2.3 [18:38] rvba: glob not defined [18:38] rvba: https://www.dropbox.com/s/j770hq7thqqxmnc/Screenshot%202014-04-02%2011.37.56.png [18:39] rvba: followed by: https://www.dropbox.com/s/fgrqpnxtaqmevas/Screenshot%202014-04-02%2011.38.02.png [18:39] blahRus1: that last bit is normal. The node is powered down at the end of the commissioning process. [18:39] blahRus1: the first error is… scary… let me look into it. [18:40] Jeffrey_, h.. i don't really know then. [18:40] this is intel ? === roadmr_afk is now known as roadmr [18:40] smoser: Yes intel cpus, duel cpu blades. [18:41] and the sd card is in the bios boot order ? [18:41] basically curtin installs to the thing it thinks is the first disk [18:41] anad if that disk is "internal sd card", then, well thats what you got. [18:43] smoser: Yes it installs it on the SD card successfully, it's just after the install when the computer is rebooting it tries to pxe boot again and then drops local, but doesn't boot the SD card properly. If I then boot the computer manually and select the SD card it boots. It's like MaaS isn't booting the proper disk. [18:44] Jeffrey_, well, maas (via pxe) just says "boot from local disk" [18:44] its the bios that interprets that [18:44] smoser: Ok sounds good. I guess I maybe have a BIOS configuration problem or something. [18:45] maybe. [18:45] the bios might also not consider the sd card to be a "disk" or something. [18:45] 2 other paths are: [18:45] * tell curtin to install on some other target that *would* be found [18:45] * tell curtin to grub-install to more things [18:46] both of these are probably acheivable, and the first possibly through config. [18:46] but from my lack of documentation and lack of time to help you find a solution, i don't have a "this will work" for you answer. [18:46] sorry. [18:46] blahRus1: there is something weird with the problem you found. It seems to be a stupid import error but I wonder how we missed that thus far. You didn't customize maas_ipmi_autodetect_tool.py by any chance? [18:47] smoser: Ok thanks for your help! I'll try and figure it out. [18:48] not at all [18:48] rvba: ^ [18:48] I am running 1.5.4 [18:49] rvba: it works fine for powering on the server [18:53] blahRus1: can you try editing /etc/maas/templates/commissioning-user-data/snippets/maas_ipmi_autodetect_tool.py to add 'import glob' underneath 'import re'. Then sudo service apache2 restart. Then re-commissiong your node. [18:54] blahRus1: hang on [18:54] blahRus1: do you have all the updates installed? [18:55] rvba: double checking [18:58] rvba: yes [18:59] blahRus1: can give me the output of `apt-cache policy maas`? [19:00] rvba: https://www.dropbox.com/s/8g6njl6k0dhj7si/Screenshot%202014-04-02%2012.00.24.png [19:00] ya [19:01] rvba: http://paste.ubuntu.com/7195575/ [19:02] blahRus1: all right, looks like an ugly bug. While I'm looking into it, can you try what I described above? [19:03] rvba: yup [19:07] rvba: latest http://paste.ubuntu.com/7195596/ [19:08] blahRus1: looks like you got the same error. [19:09] rvba: not resolving? [19:10] blahRus1: the error about the missing import. [19:11] rvba: have we moved past the resolving part then? [19:11] update to more recent code? [19:11] blahRus1: definitely [19:11] blahRus1: did you make the change to maas_ipmi_autodetect_tool.py and restart apache? [19:11] yes [19:11] #!/usr/bin/python [19:11] import commands [19:11] import subprocess [19:11] import re [19:11] import glob [19:11] Looks good. [19:12] blahRus1: can you capture the node's console while it's commissioning again? [19:12] rvba: let me try [19:15] rvba: https://www.dropbox.com/s/1tgqrpm4tabw5ju/test.avi [19:20] blahRus1: okay, so the change fixed the problem. But your node is still unable to reach the archive. [19:21] 403 forbidden? [19:21] blahRus1: yeah [19:22] rvba: our IP getting blocked? [19:22] blahRus1: I don't see how this could happen. [19:23]