| freeflying | http://paste.ubuntu.com/6379554/ | 01:29 |
|---|---|---|
| freeflying | maas stop to update dns, a lot of error like this can be found from maas.log | 01:30 |
| bigjools | what api call is causing that? | 01:31 |
| freeflying | very likely update_dns | 01:34 |
| freeflying | when celery-region try to update dns maybe | 01:35 |
| bigjools | can you show me the celery-region log please | 01:35 |
| bigjools | also the celery-cluster log | 01:35 |
| freeflying | http://paste.ubuntu.com/6379577/ | 01:35 |
| bigjools | rndc is failing - it was previously working. What changed inbetween> | 01:37 |
| bigjools | ? | 01:37 |
| bigjools | I suspect the cluster is not authorised as well, it does periodic lease updates on the api (but I can't tell for sure that's what's failing as you chopped the timestamp on the maasserver log) | 01:38 |
| freeflying | i found it fails to reolves some node's name, then checked, found ip has changed, but dns not updated, so restart named, it can't be stopped | 01:39 |
| freeflying | then did kill it, and restart it again | 01:39 |
| bigjools | try this: | 01:39 |
| bigjools | maas set_up_dns | 01:39 |
| bigjools | it'll re-write everything DNS-related | 01:40 |
| bigjools | also show me your cluster celery log please | 01:41 |
| freeflying | No handlers could be found for log "metadataserver" | 01:41 |
| freeflying | http://paste.ubuntu.com/6379619/ | 01:43 |
| bigjools | ok so there's failures earlier on for import_boot_images and restart_dhcp_server, are those ok now? | 01:45 |
| bigjools | also are you still seeing those api failures? If so can you look at the apache log file and for the corresponding timestamp of a failure look up what the request was | 01:46 |
| freeflying | bigjools, they're fine now | 01:47 |
| freeflying | bigjools, but no idea maas.log grows that bigger | 01:47 |
| bigjools | sorry that doesn't make any sense, are you saying maas.log is growing? | 01:48 |
| bigjools | I need to know if you are still seeing those authorization errors for an api call | 01:48 |
| freeflying | bigjools, I mean the import_boot image seems fine now, we did it 3 day ago | 01:48 |
| freeflying | bigjools, there is no error from maas.log since 10:36 | 01:50 |
| freeflying | now is 10:50 on the server | 01:50 |
| bigjools | and is everything working? | 01:50 |
| freeflying | don't have new node deployed, so not sure, is it normal that if no new ip in lease file, then celery-region won'r have anything to do | 01:52 |
| freeflying | bigjools, from celery-region.log, there is no action for a while | 01:52 |
| bigjools | correct, the celery-region just writes out changes to DNS as required | 01:53 |
| freeflying | bigjools, ic, thanks | 01:54 |
| bigjools | as I said above you can force a re-write with the "maas set_up_dns" command. | 01:54 |
| freeflying | yea | 01:54 |
| === freeflying is now known as freeflying_away | ||
| === freeflying_away is now known as freeflying | ||
| Azendale | bigjools: here's my /var/log/maas/pserv.log file (with just the stuff from today, otherwise it was 1.5 million lines!) http://paste.ubuntu.com/6379827/ | 03:04 |
| === freeflying is now known as freeflying_away | ||
| === freeflying_away is now known as freeflying | ||
| === freeflying is now known as freeflying_away | ||
| bigjools | Azendale: your pserv log looks fine. Can you snip the appropriate bits for dhcpd out of syslog? | 06:05 |
| Azendale | bigjools: sure | 06:10 |
| Azendale | bigjools: This time, the VM's booted, but the login prompt has something like "192-168-178-68" as the the host name | 06:13 |
| Azendale | bigjools: I can ssh to the individual node, but of course have no idea of the credentials to login | 06:14 |
| bigjools | Azendale: how did you allocate and start it? | 06:14 |
| Azendale | bigjools: juju deploy | 06:15 |
| bigjools | then juju puts your ssh key on it, assuming it installed ok | 06:15 |
| Azendale | bigjools: log as requested http://paste.ubuntu.com/6380436/ (syslog filtered through grep "dhcpd") | 06:19 |
| bigjools | Azendale: there's nothing that leaps out to explain why you had problems. | 06:20 |
| bigjools | are things very reliable when booting one at a time? | 06:21 |
| Azendale | bigjools: it seems to work just fine when I bring them up staggered a bit | 06:21 |
| Azendale | bigjools: I do see in the log "Dynamic and static leases present for 192.168.178.86" | 06:22 |
| bigjools | that's normal | 06:22 |
| bigjools | and it's definitely the dhcp stage that's timing out, not tftp? | 06:22 |
| Azendale | bigjools: I've tried to get this to work so many times, it's hard to be sure for 100% of the time | 06:23 |
| Azendale | bigjools: But I know for sure I have seen cases where the PXE boot never gets an address. That didn't seem to happen in this case because I can ssh to the node | 06:24 |
| bigjools | I am going to guess that you saturated your network with udp packets | 06:24 |
| Azendale | bigjools: I'm not sure what would cause the machine to come up with no host name (but still with an IP)? | 06:25 |
| bigjools | it's pulled its host name from DNS I expect, which means cloud-init failed | 06:26 |
| Azendale | bigjools: what all does cloud init depend on to succeed? | 06:26 |
| bigjools | there's a ton of stuff that can go wrong | 06:27 |
| bigjools | hard to see without looking at its log | 06:27 |
| bigjools | and if you can't ssh in then you can't get it | 06:27 |
| bigjools | we're working soon on making that easier, but for now you have to use a special ephemeral that hard-codes a backdoor user + password | 06:28 |
| Azendale | bigjools: is the log stored on the harddisk? I might be able to power off the machine and then retrieve it, it's a .qcow2 disk image | 06:28 |
| bigjools | yes /var/log/cloud-init/ | 06:28 |
| Azendale | bigjools: ok, I'll see if I can get a copy | 06:29 |
| bigjools | ok | 06:29 |
| bigjools | I'm leaving soon so I'll catch you Monday, it's Friday here :) | 06:30 |
| Azendale | bigjools: ok. If you think it would help, I could try running through a few cycles of boot various number of machines, and collect data if that would help. If so, just give me an idea of what information it would be helpful for me to gather | 06:32 |
| bigjools | not sure tbh, if it's a networking problem there's not much I can do | 06:33 |
| bigjools | are you on 100M, GigE? | 06:33 |
| Azendale | bigjools: it's virtio tap interfaces to a linux bridge on the host | 06:36 |
| Azendale | bigjools: http://paste.ubuntu.com/6380463/ cloud init log | 06:37 |
| bigjools | ah ok | 06:41 |
| bigjools | well that log looks like it booted without maas asking it to boot | 06:42 |
| Azendale | bigjools: hm, weird. I have MaaS set up with the virsh option for power management. The machine would have been triggered to turn on (via virsh) by maas when I ran juju deploy. Wouldn't MaaS expect the machine to reboot automatically after installing? | 06:45 |
| bigjools | hmm ok | 06:46 |
| bigjools | yes it reboots and then installs juju | 06:46 |
| Azendale | bigjools: so, if I'm keeping you, just let me know | 06:49 |
| bigjools | I am going now actually | 06:49 |
| Azendale | bigjools: ok, I guess I'll talk to you Monday | 06:50 |
| bigjools | yep - let's continue this | 06:51 |
| bigjools | bye | 06:51 |
| jtv | rbasak, do let me know if you want help adding the ssh-key-file option to uvtool! | 06:57 |
| === CyberJacob|Away is now known as CyberJacob | ||
| rbasak | jtv: it's in and build in the PPA | 07:58 |
| rbasak | built | 07:58 |
| rbasak | jtv, rvba: though --ssh-public-key might more more consistent than --ssh-public-key-file. | 07:59 |
| * rbasak doesn't know | 07:59 | |
| === freeflying_away is now known as freeflying | ||
| jtv | rbasak: great news, thanks! The "-file" does make it nicely clear that this is not a key ID. | 08:43 |
| gnuoy | Hi, I want to configure bonding on a server under maas control the port I'll be pxe booting via is part of the bond. Any advice on how to go about this ? I can add a script to setup the bonding with the pressed config but it'll be to late then I think. | 09:00 |
| gnuoy | ...as the switch will be configured to bond the interfaces right from the start | 09:00 |
| gnuoy | Do I need to doctor the initrd image ? or is there a whole world of pain waiting around the corner ? | 09:01 |
| === freeflying is now known as freeflying_away | ||
| jtv | Grrr why can't I do self.useFixture(TempDir()) in KVMFixture!? | 10:40 |
| jtv | AttributeError: 'KVMFixture' object has no attribute '_cleanups' | 10:40 |
| jtv | allenap: it sounds like the sort of thing you'd know all about... ^ | 10:40 |
| jtv | (And __init__ and setUp do their upcalls, so it's not that) | 10:41 |
| jtv | Arrrrgh I have it. The fixture is never set up. | 10:48 |
| allenap | jtv: Was the call in __init__? | 10:51 |
| allenap | That dosnae work; needs to be in setUp(). | 10:51 |
| jtv | Yeah, it was just a test that skipped setUp(). Sorry! | 10:58 |
| rbasak | allenap: got time at some point to talk about the node subarch field thing? | 11:02 |
| allenap | rbasak: Sure. I'm also reading your release combos email right now, so we could talk about that. I need to refresh my mind about the subarch thing. 1145 okay? | 11:03 |
| rbasak | allenap: Sure, thanks! | 11:04 |
| allenap | rbasak: Cool. | 11:04 |
| gnuoy | I've have a Global Kernel Parameter set but one of my nodes its not showing it on the display screen and it doesn't seem to be being passed when the node boots. Any ideas how I can fix that ? | 12:11 |
| gnuoy | all nodes are in the "ready" state | 12:18 |
| gnuoy | I've tried removing the kernel option and adding it back in again | 12:18 |
| gnuoy | maas-cli seems to returns the correct value | 12:21 |
| gnuoy | $ maas-cli maas maas get-config name=kernel_opts | 12:21 |
| gnuoy | "console=ttyS1" | 12:21 |
| rbasak | gnuoy: I think (but am not sure) that the kernel parameters there are for enlistment, commissioning and firing off the installer. For the deployment itself, I think you need to preseed. I'm not sure about what you need to do for fast-path, though. | 12:23 |
| gnuoy | rbasak, preseed is fine, but its very odd that the gui isn't showing the kernel params for just one node | 12:24 |
| rbasak | gnuoy: ah. I didn't realise you meant that it was missing from the gui. I'm not sure about that, sorry. | 12:25 |
| gnuoy | rbasak, ok, thanks for the preseed suggestion. | 12:26 |
| rbasak | gnuoy: no problem. Note that I might be inaccurate. My knowledge is a little dated. | 12:28 |
| gnuoy | rbasak, ok, noted. :-) | 12:29 |
| rvba | gmb: could you please have a look at https://code.launchpad.net/~rvb/maas-test/vm-network4/+merge/194522 if you have time today? I'd like to get that landed before my long weekend if possible so that you guys will have it on Monday. | 14:33 |
| gmb | rvba: Sure, I'll take a look now. | 14:33 |
| rvba | gmb: well, okay, I also want Jeroen to deal with all the conflicts this will create with his branch ;) | 14:33 |
| gmb | lol | 14:34 |
| rvba | Ta. | 14:34 |
| gmb | rvba: Approved with a couple of comments. | 15:11 |
| rvba | gmb: Thanks! | 15:34 |
| gmb | rvba, allenap: I'm almost done for the day; I thought I'd get to jtv's branch but didn't; it's here should either of you want to round out your friday nicely... https://code.launchpad.net/~jtv/maas-test/ssh-key/+merge/194488 | 15:42 |
| allenap | gmb: Okeydoke. Have a good weekend dude. | 16:03 |
| gnuoy | If anyone was following along at home with regards to my earlier problem with installing with bonded nics, in the end initial registration and commissioning worked fine, with the nics on the switch in an active channel-group but the install went dark almost immediately after the initrd was loaded. I got around this by shutting down the second nics interface on the switch. | 17:22 |
| mthaddon | gnuoy: I'd like to make sure that's documented somewhere - allenap, how would we get something like that into the maas docs? | 17:27 |
| gnuoy | mthaddon, ok, I'll follow that up | 17:31 |
| mthaddon | thx | 17:32 |
| mthaddon | save someone else your pain :) | 17:32 |
| allenap | mthaddon, gnuoy: The docs at http://maas.ubuntu.com/docs are generated from rst files in lp:maas/docs. Add something there - any of us will be more than happy to help - then propose a merge. | 17:36 |
| allenap | (That's the docs/ directory in lp:maas.) | 17:36 |
| gnuoy | allenap, lovely, thanks | 17:36 |
| mthaddon | thx | 17:37 |
| allenap | gnuoy: Adding docs to MAAS entitles you to a bit wet kiss. Unfortunately, as the Red Squad representative in Norfolk, I would be the one authorised to deliver your reward. | 17:38 |
| gnuoy | allenap, I'll settle for a big wet pint | 17:39 |
| allenap | gnuoy: Deal. | 17:39 |
| allenap | gnuoy: Or you can wait for gmb to visit the county? Take your pick. | 17:40 |
| gnuoy | I'll wait ! | 17:40 |
| allenap | Think of all those bristles... | 17:41 |
| === Spideyman is now known as Spideyman_afk | ||
| === Spideyman_afk is now known as Spideyman | ||
| allenap | gnuoy: I've just replied to your weird kernel options bug. | 18:08 |
| allenap | gnuoy: Have a good weekend! | 18:09 |
| === Spideyman is now known as Spideyman_afk | ||
| === Spideyman_afk is now known as Spideyman | ||
| === freeflying_away is now known as freeflying | ||
| === freeflying is now known as freeflying_away | ||
| === CyberJacob is now known as CyberJacob|Away | ||
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!