| === matsubara is now known as matsubara-lunch | ||
| === matsubara-lunch is now known as matsubara | ||
| allenap | matsubara: Hi there, have you just installed python-tempita on the build machine by any chance? | 13:07 |
|---|---|---|
| allenap | matsubara: Ah ha, no, it was rvba. Thanks rvba :) | 13:08 |
| matsubara | allenap, I haven't but I can if you need. | 13:08 |
| matsubara | :-) | 13:08 |
| matsubara | the joys of shared maintenance | 13:08 |
| matsubara | btw, I should be able to start the migration from ec2 to the QA lab jenkins instance today | 13:08 |
| matsubara | finish with the maza migration as their instance crashed last week and the team is blocked | 13:09 |
| allenap | matsubara: Oh cool, nice. | 13:09 |
| matsubara | s/finish/finishing/ | 13:09 |
| allenap | rvba: Interesting failure now; it wants confirmation for deleting the test database. I'll sort it out. | 13:09 |
| rvba | allenap: yep, I'm seing this. | 13:10 |
| rvba | matsubara: that's great news! | 13:10 |
| roaksoax | rvba: howdy! so I was wondering if there are significant changes in maas trunk yet that require new packaging and stuff? | 13:21 |
| rvba | allenap: build fixed. I think there is an option in Django to make the command "make check" non interactive. This would avoid the problem we just saw. | 13:22 |
| rvba | roaksoax: hi. Well, we've changed the architecture quite a bit since we are working on removing cobbler. | 13:23 |
| roaksoax | rvba: ok... is it still functional in trunk? What packaging changes you think it needs? What new dependencies hae been introduced? | 13:24 |
| rvba | roaksoax: it is functional, we're removing things bits by bits. | 13:25 |
| roaksoax | rvba: ok. On the other hand, what version of libraphael and libyui you guys need as the plan is to remove them from shipping them on trunk | 13:26 |
| roaksoax | so we need to ensure the packages are updated accordingly | 13:26 |
| rvba | roaksoax: indeed. I think I'd like to use yui 3.5 (we currently use 3.4.1) but I need to make sure everything is fine with YUI 3.5 first. I'll do that this week and get back to you. | 13:27 |
| roaksoax | rvba: ok, other than that, If I get to that this week, I'll update to yui 3.4.1 and then we can upgrade | 13:27 |
| roaksoax | rvba: what about libraphael? | 13:27 |
| rvba | roaksoax: we use 2.1.0. | 13:28 |
| roaksoax | ok | 13:28 |
| roaksoax | i'll take ccare of that | 13:29 |
| rvba | roaksoax: again, I'll check if there is not a major release that has been done in the meantime. I guess it would be more profitable to other people if we ship major milestone version as opposed to just the version we need. | 13:29 |
| rvba | roaksoax: cool, I'll test MAAS with YUI 3.5. Then, once the packaging is done, I'll get able to get rid of the version of YUI that we ship… that will be a very cleanup. | 13:30 |
| roaksoax | indeed | 13:31 |
| rvba | roaksoax: I guess it won't be a problem for you to upgrade once the initial packaging will be done right? It's just a bunch of static files as far as you're concerned I guess. | 13:31 |
| rvba | allenap: I've got another branch up for review if you feel like it: https://code.launchpad.net/~rvb/maas/ui-power-parameters/+merge/109621 | 13:33 |
| rvba | allenap: and then I'll be done with the power parameters \o/ | 13:33 |
| cheez0r | hrm, i can't figure out if this is a maas issue or a juju issue; I've got 11 nodes in my maas, they're all allocated to me, I've done a juju bootstrap and a bunch of deploys, but some of my machines in juju stay in agent-state: not started; and on the console these nodes get stuck partway through their pxe boot. | 13:34 |
| cheez0r | they seem to lock up, with nothing but the pink ncurses background screen showing. A reboot brings them back through the pxe process and they hang at the same spot. | 13:34 |
| cheez0r | since it's not making it through pxe I figured I'd talk to you guys first ;) | 13:35 |
| cheez0r | The last thing it does is I see it pull the kickstart, then load 'additional components', then NTP, then partitioning, and then it just sits there. | 13:41 |
| roaksoax | rvba: yeah raphael and yui shouldn't really be an issue | 13:55 |
| roaksoax | cheez0r: those those machines *don't* pxe boot? or they pxe boot but doesn't show up the installer? | 13:57 |
| cheez0r | roaksoax: they pxe boot, they load the pxelinux image, they do the initial boot sequence, they do the initial system configuration (network addressing, NTP time, etc) | 13:58 |
| rvba | roaksoax: I'm testing with YUI 3.5.1 and everything seems fine. Also, libraphaeljs 2.1.0 is the most recent release. | 13:58 |
| cheez0r | then they hang after partition manager. | 13:58 |
| cheez0r | I'm watching the console of one right now: loading additional components: partman, pkgsel, etc. Setting up the clock/NTP; detecting disks; flash; starting up the partitioner, flash, flash, nothing. | 13:59 |
| cheez0r | flash = too quick to be read | 13:59 |
| roaksoax | rvba: ok cool | 14:01 |
| allenap | rvba: Sure, I'll review that. | 14:02 |
| roaksoax | cheez0r: is there a way you could get the syslog of the machine being installed? | 14:03 |
| rvba | allenap: ta. | 14:03 |
| allenap | rvba: Sorry for the delay; someone at the door. | 14:03 |
| cheez0r | roaksoax: it never boots to a point where it can be accessed. | 14:03 |
| roaksoax | cheez0r: right, but the installer is running, which means it has a ssyslong | 14:03 |
| cheez0r | I have been watching cobbler.log; it stalls directly after generating and downloading its kickstart. | 14:03 |
| roaksoax | syslong | 14:03 |
| rvba | roaksoax: apart from that we've added two new dependencies: python-celery python-tempita | 14:03 |
| cheez0r | roaksoax: right, but I can't get to it without being able to log into the system | 14:04 |
| roaksoax | rvba: awesome. so it is still using cobbler | 14:04 |
| rvba | roaksoax: we will need to update the packaging scripts when we will want to start the celery workers but we need to think about how to do this properly first. | 14:04 |
| rvba | roaksoax: yes, right now it does. | 14:04 |
| rvba | allenap: hehe, no worries. | 14:05 |
| roaksoax | rvba: ok so the first step, I'll provide a new release in quantal with what is in trunk now introducing the new dependencies and filing MIRs | 14:05 |
| roaksoax | and then step by step we can be improving the packaging | 14:05 |
| roaksoax | rvba: as I'm sure they will be some cleanup to be done | 14:05 |
| rvba | roaksoax: if we don't start the celery workers this won't be functional. | 14:06 |
| rvba | roaksoax: because the only part that replaces cobbler with our stuff uses celery. | 14:06 |
| roaksoax | cheez0r: https://help.ubuntu.com/community/Installation/NetworkConsole | 14:06 |
| roaksoax | rvba: right, but right now you said it is using cobler still, so we don't need it yet? | 14:06 |
| rvba | roaksoax: like I said, we're replacing it bits by bits. Most of the application uses it still but the tiny part which does not use it uses celery. | 14:07 |
| roaksoax | rvba: alright, I guess i could release a partially working maas in order for me to introduce the new dependencies and get things working | 14:07 |
| rvba | roaksoax: if by "partially working" you mean "not really working" then yes… but I'm not sure it's worth it really. | 14:09 |
| roaksoax | rvba: right, but I much rather have all the new dependecies in main now than later | 14:09 |
| roaksoax | rvba: how do you usually start celery workers? | 14:09 |
| cheez0r | roaksoax: I'm not following what you're telling me to do; are you basically saying create this netboot iso, boot the system with it, log into it, and somehow recover the log from the prior install attempt where it never wrote a partition to a disk, so there can't be any log retained? | 14:09 |
| roaksoax | cheez0r: not the netboot ISO. use the network-console part | 14:09 |
| roaksoax | cheez0r: it will allow you to login while the installer is running | 14:10 |
| cheez0r | okay, so how do I implement this in the pxe boot image file? | 14:10 |
| rvba | roaksoax: Very simple: add the maas package to the PYTHONPATH and then run 'celeryd'. | 14:10 |
| roaksoax | cheez0r: you only need to modify the preseed file | 14:10 |
| cheez0r | ah ok | 14:11 |
| roaksoax | cheez0r: /var/lib/cobbler/kickstarts/maas.preseed | 14:11 |
| rvba | (note that it also need the rest of the python packages provided by maas but IIRC they will be on the general python path) | 14:11 |
| roaksoax | rvba: so is it similar to how we run twistd? | 14:11 |
| rvba | roaksoax: in the most simple case yes, but then we want to be able to run the workers on another machine and also run many workers. | 14:12 |
| roaksoax | cheez0r: sthis 3 are the ones you need: | 14:12 |
| roaksoax | d-i network-console/password password SECRET123 | 14:12 |
| roaksoax | d-i network-console/password-again password SECRET123 | 14:12 |
| roaksoax | d-i preseed/early_command string anna-install network-console | 14:12 |
| rvba | roaksoax: and we haven't started the work for these complex setup yet. | 14:12 |
| roaksoax | rvba: right, so as a first step, wouldn't it make more sense to ship a wrapper that sources PYTHONPATH and runs celeryd and/or upstartify it? | 14:13 |
| cheez0r | I've already got a partman/early_command string; that can coexist with preseed/early_command string? | 14:13 |
| cheez0r | (just confirming) | 14:13 |
| roaksoax | rvba: and eventually ship it in i.e. maas-worker | 14:13 |
| roaksoax | rvba: and provide a config to say how many workers are needed to be run? | 14:14 |
| roaksoax | cheez0r: yes, just add network-console to the end | 14:14 |
| rvba | roaksoax: That could be first step indeed. Yeah, the plan is to ship the workers within another package. | 14:14 |
| rvba | roaksoax: but this is very much WIP. Unless you really really want to do a release, I suggest waiting a little bit so that we can think of a plan. | 14:15 |
| roaksoax | rvba: awesome then, so we *do* need a wrapper for celeryd or it is just the same way as how twistd was used? | 14:15 |
| === vibhav_ is now known as vibhav | ||
| roaksoax | rvba: well I was talking to Daviey last week and we agreed on having a new release in archives ASAP cause there | 14:16 |
| roaksoax | there's much to take care. i.e. New versions of raphael/yui/ File MIR's for celery ., etc etc | 14:16 |
| rvba | roaksoax: ok then. But we definitely will need to iterate on that first release. You've been warned ;) | 14:16 |
| roaksoax | rvba: :) no worries... after what I've dealt with last cycle this is piece of cake now :) | 14:17 |
| rvba | roaksoax: haha. What would the wrapper do then? | 14:17 |
| roaksoax | rvba: TBH, I rather identify issues and start working on getting things package now than be in trouble later | 14:17 |
| roaksoax | rvba: that's my question, do we really need a wrapper that starts as many workers wanted? or for me we can just run celeryd from an upstart job | 14:18 |
| rvba | roaksoax: sure. We're in the middle of reachitecturing the whole thing that's all :) | 14:18 |
| rvba | roaksoax: we need only one worker for now. | 14:18 |
| roaksoax | rvba: ok, so that means using an upstart job is enough | 14:19 |
| rvba | Starting celeryf with the right packages on the path should be enough. | 14:19 |
| rvba | celeryd* | 14:19 |
| rvba | Yes I think so. | 14:19 |
| roaksoax | rvba: cause, once we need various workers, we can just have a wrapper that starts as many as cfonfigured and replace it with the upstart job | 14:20 |
| roaksoax | s/with the upstart job/within the upstart job | 14:20 |
| rvba | Well, it will be more complicated than that because how many workers to start is an information that will be in the DB so starting workers and stuff will need to be done at the application level. Or at least controlled at the application level. | 14:21 |
| rvba | But let's start with the simple case. | 14:21 |
| cheez0r | roaksoax: similar behavior- I connect to the network console, I start the install, it finishes with setting up the partition manager, it kicks me out of the network console. | 14:22 |
| rvba | roaksoax: so, you need to start celeryd with all maas' packages available on the path, plus the celery configuration file (etc/celeryconfig.py). | 14:22 |
| cheez0r | now ssh back in no longer works, but the console is still stuck on "Continue installation remotely using SSH" screen | 14:22 |
| roaksoax | rvba: ok, awesome. As always I'll seek your advice when I get to that part | 14:23 |
| rvba | allenap: I'm thinking that if we want to do the massive code reorg we talked about the other day, we should probably do that ASAP. | 14:23 |
| rvba | roaksoax: sure, no problem. | 14:23 |
| roaksoax | cheez0r: uhmmm | 14:23 |
| roaksoax | cheez0r: TBH I hjave no idea what it could be | 14:23 |
| cheez0r | k | 14:23 |
| roaksoax | cheez0r: smoser` Daviey ^^ any ideas for cheez0r's issue? | 14:24 |
| allenap | rvba: I'm losing interest in that because of the Django app issue :-/ Do you want to try moving metadataserver to maas.metadata (or something like that)? If you can get that to work then I think we're back on. | 14:24 |
| rvba | allenap: I can try that. What was the issue? | 14:25 |
| rvba | allenap: if we don't manage to pull it off then I think we should at the very least rename 'apiclient'. | 14:28 |
| allenap | rvba: Django complained that it couldn't load the app, but it didn't make a lot of sense. I suspect a bug. | 14:30 |
| rvba | allenap: ok, I'll see what I can do. | 14:30 |
| allenap | Cool· | 14:32 |
| cheez0r | roaksoax: so I ran the network connection as a shell and continued the installer on the console, and tailed syslog | 14:37 |
| cheez0r | I've got the output of that if you'd like it- it appears to be querying memory modules and CPUs and then just dies | 14:37 |
| cheez0r | the last line I get is kernel: [ ###.######] lowmem_reserve{}: 0 869 64806 64806 and then the beginning of another line before "Connection closed by remote host." | 14:38 |
| roaksoax | cheez0r: so that might be a kernel issue with your hardware? or faulty hardware even? | 14:40 |
| cheez0r | could be; all of the blades in this chassis are identical though, so hardware maybe, and it's passing all of the on boot/POST tests | 14:40 |
| cheez0r | I'm also getting this issue on five of eleven blades | 14:40 |
| cheez0r | I'm checking hardware configuration right now to try and isolate differences | 14:41 |
| roaksoax | alright!/win 3 | 14:46 |
| allenap | Diff against target: 4466764 lines (+3848606/-587797) 5320 files modified <-- rvba does not muck around. | 15:51 |
| rvba | allenap: haha :). I look forward to removing YUI our tree. | 15:52 |
| rvba | from* our tree | 15:53 |
| roaksoax | rvba: still around? | 17:07 |
| === smoser` is now known as smoser | ||
| marcoceppi_ | So, as I continue to venture into a more stable BTRFS setup, I've been trying to research if USB3 is just as bad as USB2 for connecting disks over long periods of time. Is USB3 any better than USB2 in regards to consistency of disk connections? Or should I still be hunting for a better method of connection not using USB at all | 17:36 |
| marcoceppi_ | FUUUU, wrong room | 17:36 |
| === matsubara is now known as matsubara-afk | ||
| kurt_ | General help question: I'm having difficulty getting a node to accept an ssh key and allow auto-logon. Other nodes don't have this problem. I've tried deleting and re-adding the node via maas, and the node re-bootstraps, but still no go with the key. Any suggestions? | 23:12 |
| kurt_ | Anyone watch this list? :) | 23:22 |
| kurt_ | General help question: I'm having difficulty getting a node to accept an ssh key and allow auto-logon. Other nodes don't have this problem. I've tried deleting and re-adding the node via maas, and the node re-bootstraps, but still no go with the key. Any suggestions? | 23:59 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!