| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === jfarschman is now known as MilesDenver | ||
| === CyberJacob is now known as CyberJacob|Away | ||
| jtv | Thanks rvba. | 08:14 |
|---|---|---|
| rvba | np | 08:14 |
| === jfarschman is now known as MilesDenver | ||
| bigjools | allenap: I was planning on doing a crappy TimerService derived service in pserv and making it do exactly as discussed in that email, do we need to pre-imp this further? | 08:56 |
| allenap | bigjools: Try to avoid the crappy part, but otherwise that sounds fine. | 08:57 |
| bigjools | haha | 08:58 |
| bigjools | allenap: it's gonna call the func I landed yesterday every hour and then kick off a deferToThread() for the import | 08:58 |
| bigjools | allenap: I still think we need a queueing mechanism for jobs | 08:58 |
| allenap | bigjools: This is something I’ve discussed a lot with rvba. There should be only one job running per node at any time. For example, it doesn’t make sense to have both an install and a shutdown job running concurrently... | 09:00 |
| allenap | However, there are subtleties. | 09:00 |
| bigjools | allenap: what about related jobs like this one? | 09:00 |
| bigjools | unrelated, sorry | 09:00 |
| allenap | If an install job is running and a “check the power state” question comes along, we can probably answer that question without waiting for the install to finish. | 09:01 |
| bigjools | and I agree - but enforcement should be done in a different level | 09:01 |
| allenap | bigjools: There should only be one import job at a time. | 09:01 |
| bigjools | allenap: not talking about that per se | 09:01 |
| bigjools | I mean total jobs across the pserv | 09:01 |
| bigjools | if it's managing thousands of nodes, we could potentially have thousands of threads | 09:02 |
| allenap | If we start hitting concurrency problems then we can add limits, but let’s get it working first. | 09:02 |
| bigjools | all strangling CPU | 09:02 |
| bigjools | I think we should consider this from the star | 09:03 |
| bigjools | t | 09:03 |
| allenap | Don’t use threads too freely then :) Use the Twisted, Luke. In truth, many of those threads will be blocking at any one time. | 09:03 |
| bigjools | well, thanks to the GIL :) | 09:03 |
| allenap | Yes, but mostly because of IO. | 09:03 |
| bigjools | simplestreams is not twisted | 09:03 |
| allenap | I know, but it’s largely IO bound. | 09:04 |
| bigjools | yes - it's a candidate | 09:04 |
| bigjools | but it's no I/O bound when it builds the tarballs | 09:04 |
| allenap | When you use deferToThread, the thread is obtained from a thread pool, to which we can apply limits. | 09:04 |
| allenap | We can also create separate thread pools and defer tasks into them specifically. | 09:04 |
| bigjools | that would be great | 09:05 |
| allenap | The default thread pool is limited to 10 threads. | 09:06 |
| bigjools | celery limited to total cores | 09:07 |
| bigjools | we have to be absolutely sure that long running things like this don't interfere with power_on and suchlike | 09:08 |
| bigjools | and for that we need a QA plan | 09:08 |
| allenap | bigjools: If simplestreams is using tarfile, the Python module, then high CPU there could pose a problem to pserv. If it’s using /bin/tar then I’m not going to worry. | 09:08 |
| bigjools | I can't remember, let me look | 09:09 |
| allenap | bigjools: twisted.internet.reactor.getThreadPool().adjustPoolSize(min, max) | 09:09 |
| bigjools | it shells out | 09:09 |
| bigjools | it loop mounts the extracted image | 09:10 |
| bigjools | but will obviously hammer the disks | 09:10 |
| allenap | bigjools: That’s not new though, and we could ionice it if it’s swamping things. | 09:11 |
| bigjools | allenap: so that's adjusting the default pool size? | 09:11 |
| bigjools | aye | 09:11 |
| allenap | Yep. | 09:11 |
| bigjools | allenap: too easy | 09:11 |
| bigjools | what happens if you exceed that? does it queue, or reject new thread requests? | 09:12 |
| allenap | bigjools: It queues. | 09:13 |
| bigjools | I wonder if we shouldn't set a limit on that too | 09:13 |
| allenap | bigjools: If we subclass t.python.threadpool.Threadpool and override callInThreadWithContext we can add that behaviour. | 09:16 |
| allenap | s/Context/Callback/ | 09:16 |
| allenap | biab | 09:17 |
| bigjools | allenap: excelente! | 09:23 |
| === jfarschman is now known as MilesDenver | ||
| allenap | bigjools: Actually, more Twisted would be to use a DeferredSemaphore, say, to manage concurrency. | 09:33 |
| rvba | allenap: I've got a branch up for review that I'd like you to review if you have time (it's tiny): https://code.launchpad.net/~rvb/maas/ipmi-power-state/+merge/226974 | 09:50 |
| rvba | I contains shell script… and my thinking is "shell script" → allenap ;) | 09:50 |
| rvba | s/I contains/It contains/ | 09:50 |
| bigjools | allenap: can you have limits on those? | 09:53 |
| allenap | rvba: Sure. | 09:56 |
| rvba | Ta | 09:56 |
| allenap | bigjools: Yep, you set the limit on the semaphore size when creating it. We’d want to check the sem.waiting queue before calling sem.acquire() or sem.run(…). | 09:59 |
| bigjools | allenap: excelente! | 09:59 |
| === jfarschman is now known as MilesDenver | ||
| jtv | Who wants to review my branch for the QA lab tests? https://code.launchpad.net/~jtv/maas/qa-alt-ngi-name-fix/+merge/226991 | 10:29 |
| rvba | jtv: I'll take it | 10:29 |
| jtv | Thanks. | 10:29 |
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| === roadmr is now known as roadmr_afk | ||
| === CyberJacob|Away is now known as CyberJacob | ||
| === jfarschman is now known as MilesDenver | ||
| === roadmr_afk is now known as roadmr | ||
| === jfarschman is now known as MilesDenver | ||
| === Solution-X|AFK is now known as Solution-X | ||
| === jfarschman is now known as MilesDenver | ||
| === jfarschman is now known as MilesDenver | ||
| allenap | blake_r: Have you noticed any memory problems using tgt for iSCSI? I’m using a laptop with 4GB memory to run a region+cluster for my NUCs but the boot import crashes with out-of-memory errors when registering images with tgt at the end. | 22:17 |
| blake_r | allenap: no I have not in don't have anything with that low of memory | 22:21 |
| allenap | blake_r: Hehe, yes, it’s somewhat underspecified. It’s pink too. | 22:22 |
| blake_r | allenap: unless the targets are mounted I wouldn't think it should take that much | 22:22 |
| blake_r | allenap: the laptop is pink? | 22:22 |
| allenap | blake_r: Yes, the laptop is pink. It’s a cast-off. I have set a lot of swap but it doesn’t care. Okay, thanks anyway, I may sniff around to see what I can discover. | 22:23 |
| blake_r | allenap: Haha. Okay. | 22:25 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!