[08:14] <jtv> Thanks rvba.
[08:14] <rvba> np
[08:56] <bigjools> allenap: I was planning on doing a crappy TimerService derived service in pserv and making it do exactly as discussed in that email, do we need to pre-imp this further?
[08:57] <allenap> bigjools: Try to avoid the crappy part, but otherwise that sounds fine.
[08:58] <bigjools> haha
[08:58] <bigjools> allenap: it's gonna call the func I landed yesterday every hour and then kick off a deferToThread() for the import
[08:58] <bigjools> allenap: I still think we need a queueing mechanism for jobs
[09:00] <allenap> bigjools: This is something I’ve discussed a lot with rvba. There should be only one job running per node at any time. For example, it doesn’t make sense to have both an install and a shutdown job running concurrently...
[09:00] <allenap> However, there are subtleties.
[09:00] <bigjools> allenap: what about related jobs like this one?
[09:00] <bigjools> unrelated, sorry
[09:01] <allenap> If an install job is running and a “check the power state” question comes along, we can probably answer that question without waiting for the install to finish.
[09:01] <bigjools> and I agree - but enforcement should be done in a different level
[09:01] <allenap> bigjools: There should only be one import job at a time.
[09:01] <bigjools> allenap: not talking about that per se
[09:01] <bigjools> I mean total jobs across the pserv
[09:02] <bigjools> if it's managing thousands of nodes, we could potentially have thousands of threads
[09:02] <allenap> If we start hitting concurrency problems then we can add limits, but let’s get it working first.
[09:02] <bigjools> all strangling CPU
[09:03] <bigjools> I think we should consider this from the star
[09:03] <bigjools> t
[09:03] <allenap> Don’t use threads too freely then :) Use the Twisted, Luke. In truth, many of those threads will be blocking at any one time.
[09:03] <bigjools> well, thanks to the GIL :)
[09:03] <allenap> Yes, but mostly because of IO.
[09:03] <bigjools> simplestreams is not twisted
[09:04] <allenap> I know, but it’s largely IO bound.
[09:04] <bigjools> yes - it's a candidate
[09:04] <bigjools> but it's no I/O bound when it builds the tarballs
[09:04] <allenap> When you use deferToThread, the thread is obtained from a thread pool, to which we can apply limits.
[09:04] <allenap> We can also create separate thread pools and defer tasks into them specifically.
[09:05] <bigjools> that would be great
[09:06] <allenap> The default thread pool is limited to 10 threads.
[09:07] <bigjools> celery limited to total cores
[09:08] <bigjools> we have to be absolutely sure that long running things like this don't interfere with power_on and suchlike
[09:08] <bigjools> and for that we need a QA plan
[09:08] <allenap> bigjools: If simplestreams is using tarfile, the Python module, then high CPU there could pose a problem to pserv. If it’s using /bin/tar then I’m not going to worry.
[09:09] <bigjools> I can't remember, let me look
[09:09] <allenap> bigjools: twisted.internet.reactor.getThreadPool().adjustPoolSize(min, max)
[09:09] <bigjools> it shells out
[09:10] <bigjools> it loop mounts the extracted image
[09:10] <bigjools> but will obviously hammer the disks
[09:11] <allenap> bigjools: That’s not new though, and we could ionice it if it’s swamping things.
[09:11] <bigjools> allenap: so that's adjusting the default pool size?
[09:11] <bigjools> aye
[09:11] <allenap> Yep.
[09:11] <bigjools> allenap: too easy
[09:12] <bigjools> what happens if you exceed that?  does it queue, or reject new thread requests?
[09:13] <allenap> bigjools: It queues.
[09:13] <bigjools> I wonder if we shouldn't set a limit on that too
[09:16] <allenap> bigjools: If we subclass t.python.threadpool.Threadpool and override callInThreadWithContext we can add that behaviour.
[09:16] <allenap> s/Context/Callback/
[09:17] <allenap> biab
[09:23] <bigjools> allenap: excelente!
[09:33] <allenap> bigjools: Actually, more Twisted would be to use a DeferredSemaphore, say, to manage concurrency.
[09:50] <rvba> allenap: I've got a branch up for review that I'd like you to review if you have time (it's tiny): https://code.launchpad.net/~rvb/maas/ipmi-power-state/+merge/226974
[09:50] <rvba> I contains shell script… and my thinking is "shell script" → allenap ;)
[09:50] <rvba> s/I contains/It contains/
[09:53] <bigjools> allenap: can you have limits on those?
[09:56] <allenap> rvba: Sure.
[09:56] <rvba> Ta
[09:59] <allenap> bigjools: Yep, you set the limit on the semaphore size when creating it. We’d want to check the sem.waiting queue before calling sem.acquire() or sem.run(…).
[09:59] <bigjools> allenap: excelente!
[10:29] <jtv> Who wants to review my branch for the QA lab tests?  https://code.launchpad.net/~jtv/maas/qa-alt-ngi-name-fix/+merge/226991
[10:29] <rvba> jtv: I'll take it
[10:29] <jtv> Thanks.
[22:17] <allenap> blake_r: Have you noticed any memory problems using tgt for iSCSI? I’m using a laptop with 4GB memory to run a region+cluster for my NUCs but the boot import crashes with out-of-memory errors when registering images with tgt at the end.
[22:21] <blake_r> allenap: no I have not in don't have anything with that low of memory
[22:22] <allenap> blake_r: Hehe, yes, it’s somewhat underspecified. It’s pink too.
[22:22] <blake_r> allenap: unless the targets are mounted I wouldn't think it should take that much
[22:22] <blake_r> allenap: the laptop is pink?
[22:23] <allenap> blake_r: Yes, the laptop is pink. It’s a cast-off. I have set a lot of swap but it doesn’t care. Okay, thanks anyway, I may sniff around to see what I can discover.
[22:25] <blake_r> allenap: Haha. Okay.