| === freeflying_away is now known as freeflying | ||
| hobbyBobby | got a problem trying to get dns working with vms if anyone is willing to help | 02:07 |
|---|---|---|
| hobbyBobby | wow im daft | 02:18 |
| === mwhudson is now known as zz_mwhudson | ||
| BjornT | bigjools_: hi :) i don't know the terminlogy, but i'm talking about cloud-init for sure. i'm not sure what the part-001 script is called, though, that cloud-init tries to run, but that one seems to be coming from maas at least. | 07:20 |
| bigjools_ | hey BjornT! | 07:21 |
| bigjools_ | BjornT: juju gives it to MAAS to feed to cloud-init | 07:21 |
| bigjools_ | AFAIK | 07:21 |
| bigjools_ | assuming you're deploying a node? | 07:22 |
| BjornT | bigjools_: yes, i am. | 07:22 |
| bigjools_ | BjornT: all of the maas commissioning scripts start with a number, like 01-lshw | 07:24 |
| BjornT | bigjools_: so this is most likely a bug with juju, and not maas? | 07:27 |
| bigjools_ | BjornT: almost certainly | 07:27 |
| bigjools_ | BjornT: where are you seeing the failure? | 07:27 |
| bigjools_ | and can you show me a log? | 07:28 |
| BjornT | bigjools_: i would show you the log, except that when i ran the script manually it succeeded, and the log got rewritten. i'm seeing this when deploying on garage maas. i haven't run into this issue when deploying to vms. | 07:29 |
| bigjools_ | BjornT: how do you know it fails normally? | 07:36 |
| BjornT | bigjools_: looking at juju status and noticing that one machine is stuck in pending for a long time. then i ssh in and check the logs. i'm going to redeploy now to see if i can reproduce it. | 07:38 |
| bigjools_ | ok thanks | 07:38 |
| BjornT | bigjools_: btw, you don't know a way of speeding up the bootstrap process? it takes something like 7 minutes to download and upload all the tools. | 07:40 |
| bigjools_ | BjornT: fraid not, that's all in juju's hands | 07:40 |
| bigjools_ | assuming you're using the fast installer on maas? | 07:40 |
| BjornT | bigjools_: yes | 07:41 |
| bigjools_ | it was quicker int he Python juju days :) | 07:42 |
| bigjools_ | *cough* | 07:42 |
| rvba | jtv: I wonder if each subnet shouldn't include a reference to the interface where it should be "offered" (in the DHCP config). | 08:16 |
| rvba | jtv: I found traces of such a config and it seems to work (i.e. the DHCP server starts) but I can't find a proper mention of this in the documentation. | 08:16 |
| jtv | rvba: so basically the server might or might not be ignoring that interface spec? | 08:28 |
| rvba | jtv: yeah. | 08:29 |
| === zz_mwhudson is now known as mwhudson | ||
| jtv | rvba: I have no idea how dhcpd decides which interface to serve what on... | 08:39 |
| rvba | jtv: It expects the NICs to have fixed IP addresses and then matches subnets to interfaces based on network membership. | 08:40 |
| jtv | Doesn't sound as if the interface config really helps then... | 08:47 |
| rvba | jtv: well, adding the "interface <itf>" statement inside the subnet declaration is a way to override this behavior. | 08:53 |
| rvba | That's my guess. | 08:53 |
| rvba | But I can't find proper documentation for this :/. | 08:53 |
| jtv | Sounds sensible — any particular reason to worry about it? | 08:54 |
| jtv | I have a few physical machines here, so I could experiment in the evening if it helps. | 08:54 |
| jtv | (Much better if you can _see_ that nobody's doing anything clever inbetween) | 08:55 |
| rvba | If you have machines with multiple NICs, I'd be happy if you could test this. | 08:56 |
| jtv | I have one, yes. | 08:57 |
| jtv | Two NICs. | 08:57 |
| jtv | I'm thinking: install dhcpd there, configure with whatever you dictate, hook up to two client machines, see that they get DCHP addresses each on their own network. | 08:58 |
| rvba | Sounds good. | 08:58 |
| BjornT | bigjools_: i've attached the cloud-init log and the part-001 of a failing node to the bug | 08:58 |
| bigjools_ | cheers | 08:59 |
| rvba | Hi BjornT, sorry I misguided you yesterday, I thought you were having a problem commissioning a node. Looks like the problem happens further down the deployment process. | 09:00 |
| BjornT | rvba: no worries. do you need the maas logs as well? | 09:06 |
| rvba | BjornT: let me have a look at the documents you just posted. | 09:09 |
| rvba | jtv: that's an example config: http://paste.ubuntu.com/6831026/ | 09:23 |
| jtv | rvba: OK, I can test that sometime after the call. | 09:24 |
| rvba | jtv: cool, thank you. | 09:24 |
| rvba | jtv: err, this version contains the "interface <>" statements: http://paste.ubuntu.com/6831032/ | 09:25 |
| jtv | OK | 09:26 |
| rvba | BjornT: the script in question downloads stuff from MAAS (curtin's install image) so attaching /var/log/apache2/* and /var/log/maas/maas.log to the bug might help us see if there is a problem on MAAS' side. | 09:28 |
| BjornT | rvba: done | 09:32 |
| rvba | Ta | 09:32 |
| === mwhudson is now known as zz_mwhudson | ||
| BjornT | rvba: do you know if there's some way of making the part-001 script show more debug information, to see where it fails? | 09:46 |
| rvba | BjornT: (sorry, was otp) I think this script is generated by curtin so it's not obvious how to do this, let me look into it… | 10:01 |
| rvba | jtv: here you go: https://code.launchpad.net/~rvb/maas/dhcp-multiple-intf/+merge/203325 | 10:04 |
| jtv | OK | 10:05 |
| jtv | By the way, I do see another way of checking for clashing networks, using IPSet, but if anything it looks _more_ complicated than what I had in mind. | 10:05 |
| === CyberJacob is now known as CyberJacob|Away | ||
| rvba | smoser: Hi, could you please have a look at bug https://bugs.launchpad.net/maas/+bug/1273296 ? Maybe you'll be able to help us debug the problem. | 14:04 |
| ubot5 | Ubuntu bug 1273296 in MAAS "cloud-init sometimes fails to run the part-001 script" [Undecided,Incomplete] | 14:04 |
| smoser | rvba, cloud-init ran the program. | 14:09 |
| smoser | thats clear in the log, and it prints WARNING that it failed to run it. | 14:09 |
| smoser | Jan 28 08:34:13 maas-1-16 [CLOUDINIT] util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [3] | 14:10 |
| rvba | smoser: right, but we can't figure out *why*. Also, BjornT ran the script manually a second time and this time it worked… | 14:10 |
| smoser | i think you probably have output of the command when it ran in /var/log/cloud-init-output.log | 14:11 |
| smoser | (which wasn't collected) | 14:11 |
| rvba | BjornT: ^ (Not sure you still have the node available to get this log file…) | 14:12 |
| smoser | bah. | 14:17 |
| smoser | this is from garage maas | 14:17 |
| smoser | i wouldn't trust the maas installation on that system. | 14:17 |
| smoser | or curtin installation | 14:17 |
| smoser | # smoser disable growpart as it is causing mount issues | 14:18 |
| smoser | # the issue is really teh partition table writing (due to > 2TB disks) | 14:18 |
| smoser | (thats a comment in the part-001) | 14:18 |
| rvba | ugh | 14:19 |
| smoser | there are local changes there. | 14:19 |
| smoser | its quite possible there *is* a bug though. | 14:19 |
| smoser | the good thing about being garage maas is that i can see console logs. | 14:19 |
| BjornT | rvba, smoser: there was no cloud-init-output.log, iirc. the node isn't running anymore, but i can try to reproduce it. | 14:26 |
| tomixxx | hi, matsubara or jtv online? | 14:27 |
| tomixxx | if i connect to the maas server via 10.0.0.9/MAAS/ i get "internal server error" in the browser | 14:28 |
| tomixxx | however, directly after booting the server, i could connect to the web-interface | 14:28 |
| tomixxx | log says : "scheduling error: couldnt apply scheduled task report-boot-images: [Errno 113] no route to host | 14:33 |
| tomixxx | and after this, the system shut down | 14:33 |
| tomixxx | my network config is like the following: http://pastebin.ubuntu.com/6762617 | 14:34 |
| smoser | ok. so if there wasn't a /var/log/cloud-inti-output.log, then that is a bug in maas that it should send that cloud-config. | 14:34 |
| smoser | we sould fix that. | 14:34 |
| tomixxx | so i have two interfaces | 14:34 |
| tomixxx | one connects me to the university-network, the other one connects to me to my switch which connects to other nodes | 14:35 |
| tomixxx | can someone help me please? | 14:35 |
| tomixxx | i guess this is because the server tries to download the images but cannot connect to the internet | 14:36 |
| tomixxx | ? | 14:36 |
| jtv | Hi tomixxx. I don't _think_ it's that... It looks more like a problem when the region controller tries to order the cluster controller(s) to report what images they have downloaded. | 14:37 |
| tomixxx | hi jtv :-) | 14:38 |
| jtv | But that shouldn't interfere with the web app. | 14:38 |
| jtv | The server will try to download the images, yes, and that will fail if there's no internet... but it shouldn't cause this. :/ | 14:38 |
| jtv | It's not the proxy again? | 14:38 |
| jtv | Because the cluster controller(s) will try to download the images through the proxy running on the region controller. | 14:39 |
| tomixxx | i dont know, but immadietaly after rebooting, i could enter the maas-web-interface | 14:40 |
| tomixxx | a minute later, "internal server error" occured | 14:40 |
| jtv | Any tracebacks in the apache logs? | 14:41 |
| smoser | BjornT, could you open a bug stating that there is no /var/log/cloud-init-output.log during install phase. | 14:41 |
| tomixxx | jtv: raise.sockeet.error, msg \r\n error: [Errno 113] no route to host | 14:43 |
| tomixxx | jtv: is the last entry | 14:43 |
| jtv | But no context about where that error happened? | 14:43 |
| tomixxx | jtv: client 127.0.0.1 | 14:43 |
| tomixxx | jtv: the whole message is written as follow: "[error] [client 127.0.0.1] raise socket.error, msg ..." | 14:44 |
| tomixxx | ah ok, i see what u mean | 14:45 |
| tomixxx | jtv: last recent call | 14:45 |
| tomixxx | File "/usr/share/maas/sgi.py", line 30, in <module> | 14:45 |
| BjornT | smoser: bug 1273705 | 14:46 |
| ubot5 | bug 1273705 in MAAS "No cloud-init-output.log during install phase" [Undecided,New] https://launchpad.net/bugs/1273705 | 14:46 |
| smoser | rvba, it'd seem to get that fixed, we just ened to modify contrib/preseeds_v2/curtin_userdata | 14:47 |
| smoser | i *think* | 14:47 |
| jtv | tomixxx: I guess wsgi.py, not sgi.py? | 14:47 |
| smoser | to add | 14:47 |
| smoser | output: {all: '| tee -a /var/log/cloud-init-output.log'} | 14:47 |
| tomixxx | jtv: yes, sorry | 14:47 |
| jtv | tomixxx: that's the very top level... It's not a full traceback? | 14:47 |
| tomixxx | jtv: yes, should i post it? | 14:48 |
| rvba | smoser: sounds good to me. | 14:48 |
| jtv | tomixxx: that'd be great, yes | 14:49 |
| tomixxx | jtv: http://pastebin.ubuntu.com/6832375 | 14:49 |
| jtv | Thanks. | 14:49 |
| jtv | That does look like the attempt to send commands to the cluster controller is failing... | 14:50 |
| tomixxx | ok | 14:50 |
| jtv | Specifically, it looks like a problem with RabbitMQ. | 14:51 |
| jtv | IIRC RabbitMQ is a bit sensitive about IP addresses changing after it was set up. | 14:51 |
| jtv | Is rabbit running? It may have logged a hint of what's wrong at its end. | 14:52 |
| tomixxx | ok, how can i check this? | 14:52 |
| jtv | Look for errors logged in /var/log/rabbitmq | 14:53 |
| tomixxx | jtv: no errors in any log file | 14:55 |
| jtv | :| | 14:55 |
| jtv | Is Rabbit running? | 14:56 |
| jtv | Try: ps -ef | grep rabbit | 14:56 |
| tomixxx | just to repeat what i have done so far: set the dhcp and dns settings in the cluster-controller, and changed the main-url in maas_local_settings.py | 14:57 |
| tomixxx | jtv: ok | 14:57 |
| tomixxx | jtv: it prints me some text with red words | 14:58 |
| tomixxx | jtv: red words = rabbit | 14:58 |
| === freeflying is now known as freeflying_away | ||
| tomixxx | jtv: network settings: http://pastebin.ubuntu.com/6832421 | 14:59 |
| tomixxx | jtv: eth1 connects me (successfully) to the i-net | 15:00 |
| tomixxx | jtv: eth0 is the interface of the cluster-controller of the maas-server | 15:00 |
| jtv | tomixxx: if the "ps -ef" output mentioned erlang, then rabbit is running. | 15:00 |
| jtv | The red words are normal: "grep" highlights matching words. | 15:01 |
| tomixxx | jtv: there is an entry /usr/lib/erlang/erts-5.85... | 15:01 |
| tomixxx | jtv: do i have to bridge the two interfaces in some way? what did you exactly mean by "is it the proxy again" ? | 15:02 |
| jtv | tomixxx: I may be misremembering... I thought on a previous occasion you had some problem with the http proxy that maas starts on the region controller. But it may have been someone else. | 15:03 |
| jtv | Anyway, it doesn't look to be the proxy. This problem involves rabbit. | 15:03 |
| tomixxx | jtv: kk | 15:03 |
| tomixxx | jtv: as far as i remember, i had no problems with the region controller (so far :D) | 15:03 |
| jtv | No, I was probably just misremembering who ran into that. IIRC it was simply running out of memory in that case. | 15:04 |
| tomixxx | the funny thing is, it worked for around one minute after system-reboot... i could navigate till the preferences-page if i remember correct | 15:05 |
| jtv | rvba: success! With your DHCP config, clients on the two networks get IP addresses in those respective networks. | 15:05 |
| rvba | jtv: \o/ | 15:05 |
| rvba | Thanks for the test! | 15:06 |
| jtv | tomixxx: it is infuriating... It looks as if rabbit accepts messages for a while, and then either breaks down or discovers that it couldn't connect in the first place... | 15:06 |
| jtv | I wonder if smoser or roaksoax might know more about what could be going wrong there. | 15:06 |
| tomixxx | jtv: hmm, maybe i should reboot again | 15:08 |
| jtv | Always worth a try. :) | 15:08 |
| tomixxx | k, internet works, now i open 10.0.0.9/MAAS | 15:11 |
| tomixxx | damn, this time i got the error message immadietly | 15:11 |
| tomixxx | could a firewall be the problem? | 15:12 |
| jtv | It'd have to be blocking local communication... doesn't seem likely. | 15:14 |
| tomixxx | there sth more in the apache2 log i will post | 15:14 |
| tomixxx | ...cannot be loaded as pytthon module | 15:15 |
| jtv | That sounds suspicious! | 15:16 |
| jtv | Could mean that one of the configs is incorrect. | 15:17 |
| tomixxx | http://pastebin.ubuntu.com/6832521 | 15:17 |
| tomixxx | i guess this is the whole log since reboot | 15:17 |
| jtv | Alas. It looks like the "cannot be loaded" error is just a result of the "no route to host" one. :( | 15:18 |
| tomixxx | ok | 15:18 |
| jtv | Is there anything in /etc/rabbitmq? | 15:20 |
| tomixxx | y, a folder and a config file | 15:20 |
| tomixxx | the folder is empty | 15:20 |
| jtv | You might want to look there to see if the config mentions any IP address that can't be reached in the current setup. | 15:20 |
| tomixxx | jtv: no ip mentioned | 15:21 |
| tomixxx | jtv: is there a way to reset the whole maas-server? | 15:22 |
| jtv | Oh, and: the machine's host name did not change since you installed, right? | 15:22 |
| tomixxx | jtv: the machine host name changed, because when i installed maas first time, i had only interface with another ip (assigned by an extern dhcp from the university) | 15:23 |
| jtv | One thing you can always do is uninstall the packages, with the --purge option. But in this case I get the impression the problem is with rabbit. | 15:23 |
| tomixxx | jtv: later on, i added a 2nd interface, and now the 2nd interface with 10.0.0.9 connects the server to the other nodes | 15:24 |
| tomixxx | former, the ip was sth like 143.xxx.xxx.xx | 15:24 |
| jtv | Then the problem could simply be that the changes confused rabbit. It's a creature of habit. | 15:24 |
| tomixxx | oh, ok | 15:24 |
| jtv | I'm afraid I need to go now, but you could try uninstalling rabbitmq and purging its config. This will uninstall maas, but just make sure that it doesn't purge the maas config. After that, re-install maas and it should pick up your old config. | 15:26 |
| jtv | Or maybe dpkg-reconfigure will work on rabbit... sounds safer. | 15:26 |
| tomixxx | sudo dpkg-reconfigure rabbitmq? | 15:27 |
| jtv | I think the package name is rabbitmq-server. Let me look. | 15:28 |
| jtv | Yup, that's the one. | 15:28 |
| * jtv → zzz | 15:28 | |
| tomixxx | ok, ty for the hint but does not work :( | 15:29 |
| jtv | :( | 15:29 |
| tomixxx | it seems i have to reinstall-maas | 15:29 |
| jtv | Well if you don't purge the configuration, it should be much easier to re-install. | 15:30 |
| jtv | If it really is a rabbit issue, then probably the rabbit config is the only thing that needs purging. | 15:30 |
| tomixxx | so i should delete the config file in the rabbit folder? | 15:31 |
| jtv | allenap: didn't you run into problems with rabbit getting confused by networking changes after setup? | 15:31 |
| jtv | No, no need to delete — if you uninstall rabbit with apt-get's --purge option, config will be removed. | 15:31 |
| tomixxx | ok, could u spell the command please, and the command to install maas again? | 15:32 |
| jtv | Well you may want to search the internet for more about rabbitmq networking problems before you do anything drastic, but the commands would be: | 15:33 |
| jtv | sudo apt-get --purge remove rabbitmq-server | 15:33 |
| jtv | (To uninstall rabbitmq-server and everything that depends on it, and purge rabbitmq's config) | 15:33 |
| jtv | Followed by: | 15:33 |
| jtv | sudo apt-get install maas | 15:34 |
| jtv | (To re-install maas, plus everything it needs including rabbitmq) | 15:34 |
| jtv | As always, be careful to back up important things etc. | 15:34 |
| jtv | Inserting lots of disclaimers here. :) | 15:34 |
| tomixxx | ok, good thing is i can do whatever i want with my 3 phyical nodes here ^^ | 15:34 |
| tomixxx | as long as i do not interfere the university network xD | 15:35 |
| jtv | Always nice! | 15:35 |
| jtv | Must sleep now... best of luck, alles gute! | 15:35 |
| tomixxx | ok, ty so far, i will try | 15:35 |
| tomixxx | huhu, could it be that the command $ maas-cli maas node-groups import-boot-images is not supported? | 15:47 |
| tomixxx | from this site: http://maas.ubuntu.com/docs/install.html#post-install | 15:47 |
| tomixxx | i only get a usage-hint as response | 15:48 |
| tomixxx | i have reinstalled maas now and downlaoded the images via sudo maas-import-pxe-files but the yellow error message on 10.0.0.9/MAAS does not disappear | 16:13 |
| tomixxx | :( | 16:13 |
| tomixxx | i had to change the ip with sudo dpkg-reconfigure maas-region-controller | 16:13 |
| tomixxx | OKOK it works, yellow message disappeared ... | 16:26 |
| tomixxx | jep, one node READY :D | 16:38 |
| tomixxx | BUT, i have seen that the node was not able to connect to the i-net and download additional packages | 16:39 |
| === zz_mwhudson is now known as mwhudson | ||
| === mwhudson is now known as zz_mwhudson | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === zz_mwhudson is now known as mwhudson | ||
| === freeflying_away is now known as freeflying | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!