=== jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === CyberJacob|Away is now known as CyberJacob === jfarschman is now known as MilesDenver === CyberJacob is now known as CyberJacob|Away [08:14] Thanks rvba. [08:14] np === jfarschman is now known as MilesDenver [08:56] allenap: I was planning on doing a crappy TimerService derived service in pserv and making it do exactly as discussed in that email, do we need to pre-imp this further? [08:57] bigjools: Try to avoid the crappy part, but otherwise that sounds fine. [08:58] haha [08:58] allenap: it's gonna call the func I landed yesterday every hour and then kick off a deferToThread() for the import [08:58] allenap: I still think we need a queueing mechanism for jobs [09:00] bigjools: This is something I’ve discussed a lot with rvba. There should be only one job running per node at any time. For example, it doesn’t make sense to have both an install and a shutdown job running concurrently... [09:00] However, there are subtleties. [09:00] allenap: what about related jobs like this one? [09:00] unrelated, sorry [09:01] If an install job is running and a “check the power state” question comes along, we can probably answer that question without waiting for the install to finish. [09:01] and I agree - but enforcement should be done in a different level [09:01] bigjools: There should only be one import job at a time. [09:01] allenap: not talking about that per se [09:01] I mean total jobs across the pserv [09:02] if it's managing thousands of nodes, we could potentially have thousands of threads [09:02] If we start hitting concurrency problems then we can add limits, but let’s get it working first. [09:02] all strangling CPU [09:03] I think we should consider this from the star [09:03] t [09:03] Don’t use threads too freely then :) Use the Twisted, Luke. In truth, many of those threads will be blocking at any one time. [09:03] well, thanks to the GIL :) [09:03] Yes, but mostly because of IO. [09:03] simplestreams is not twisted [09:04] I know, but it’s largely IO bound. [09:04] yes - it's a candidate [09:04] but it's no I/O bound when it builds the tarballs [09:04] When you use deferToThread, the thread is obtained from a thread pool, to which we can apply limits. [09:04] We can also create separate thread pools and defer tasks into them specifically. [09:05] that would be great [09:06] The default thread pool is limited to 10 threads. [09:07] celery limited to total cores [09:08] we have to be absolutely sure that long running things like this don't interfere with power_on and suchlike [09:08] and for that we need a QA plan [09:08] bigjools: If simplestreams is using tarfile, the Python module, then high CPU there could pose a problem to pserv. If it’s using /bin/tar then I’m not going to worry. [09:09] I can't remember, let me look [09:09] bigjools: twisted.internet.reactor.getThreadPool().adjustPoolSize(min, max) [09:09] it shells out [09:10] it loop mounts the extracted image [09:10] but will obviously hammer the disks [09:11] bigjools: That’s not new though, and we could ionice it if it’s swamping things. [09:11] allenap: so that's adjusting the default pool size? [09:11] aye [09:11] Yep. [09:11] allenap: too easy [09:12] what happens if you exceed that? does it queue, or reject new thread requests? [09:13] bigjools: It queues. [09:13] I wonder if we shouldn't set a limit on that too [09:16] bigjools: If we subclass t.python.threadpool.Threadpool and override callInThreadWithContext we can add that behaviour. [09:16] s/Context/Callback/ [09:17] biab [09:23] allenap: excelente! === jfarschman is now known as MilesDenver [09:33] bigjools: Actually, more Twisted would be to use a DeferredSemaphore, say, to manage concurrency. [09:50] allenap: I've got a branch up for review that I'd like you to review if you have time (it's tiny): https://code.launchpad.net/~rvb/maas/ipmi-power-state/+merge/226974 [09:50] I contains shell script… and my thinking is "shell script" → allenap ;) [09:50] s/I contains/It contains/ [09:53] allenap: can you have limits on those? [09:56] rvba: Sure. [09:56] Ta [09:59] bigjools: Yep, you set the limit on the semaphore size when creating it. We’d want to check the sem.waiting queue before calling sem.acquire() or sem.run(…). [09:59] allenap: excelente! === jfarschman is now known as MilesDenver [10:29] Who wants to review my branch for the QA lab tests? https://code.launchpad.net/~jtv/maas/qa-alt-ngi-name-fix/+merge/226991 [10:29] jtv: I'll take it [10:29] Thanks. === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver === roadmr is now known as roadmr_afk === CyberJacob|Away is now known as CyberJacob === jfarschman is now known as MilesDenver === roadmr_afk is now known as roadmr === jfarschman is now known as MilesDenver === Solution-X|AFK is now known as Solution-X === jfarschman is now known as MilesDenver === jfarschman is now known as MilesDenver [22:17] blake_r: Have you noticed any memory problems using tgt for iSCSI? I’m using a laptop with 4GB memory to run a region+cluster for my NUCs but the boot import crashes with out-of-memory errors when registering images with tgt at the end. [22:21] allenap: no I have not in don't have anything with that low of memory [22:22] blake_r: Hehe, yes, it’s somewhat underspecified. It’s pink too. [22:22] allenap: unless the targets are mounted I wouldn't think it should take that much [22:22] allenap: the laptop is pink? [22:23] blake_r: Yes, the laptop is pink. It’s a cast-off. I have set a lot of swap but it doesn’t care. Okay, thanks anyway, I may sniff around to see what I can discover. [22:25] allenap: Haha. Okay.