[01:26] <haasn> Hmm.. some update on the IPMI issue I had: Turns out when I said it was working.. well, not fully. It works for --stat, but it doesn't work for turning the power on or off
[01:27] <haasn> But the odd thing is: For power on/off it doesn't even seem to be launching ipmipower at all
[01:27] <haasn> I don't see any processes created the way I see processes created every time I click “check now”
[01:31] <haasn> Ah, it's going through ipmi-chassis-config - of course
[04:22] <mup> Bug #1524853 changed: DHCP lease allocation does not match deployed node's IP address <MAAS:Expired> <https://launchpad.net/bugs/1524853>
[09:57] <haasn> What's the easiest way to customize the images usable in MAAS, preferably in a uniform way? For example, I want to make a change to the default /etc/cloud/cloud.cfg to change the way new systems are deployed
[09:58] <haasn> But I don't really want to maintain my own custom image - I'd rather this change be “universal”, if that makes sense
[10:04] <haasn> Hmm. I guess I want to be modifying preseeds/curtin_userdata?
[10:08] <haasn> No, that doesn't seem right. Curtin only deals with the installation process - but I want to customize things *after* the installation is successful (i.e. from within the new system, before cloud-init runs)
[11:05] <mup> Bug #1519810 changed: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>
[11:14] <mup> Bug #1519810 opened: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>
[11:17] <mup> Bug #1519810 changed: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>
[11:40] <haasn> I think part of what I want to accomplish is done by adding late_commands to curtin_userdata. I tried adding two commands (to run `apt-get update` and `apt-get upgrade` automatically on newly deployed nodes) but now I get some weird behavior: when deploying, the log eventually changes from “deployed” to “failed deploying” but the node itself still gets displayed as ‘deploying’
[11:41] <haasn> Seems like there's some sort of weird inconsistency in the MAAS database itself here, where the log clearly thinks the status got changed but the UI disagrees
[11:41] <haasn> Without the lines I added it works fine
[11:41] <haasn> The change I made is adding the two apt-get lines here: https://0x0.st/Xt8.txt
[11:42] <haasn> Anybody know what's wrong with them? It's not like I can see a log output because the UI doesn't even think installation failed (yet)
[11:47] <haasn> Including just the update line seems to work fine, it's the upgrade line that's causing problems
[11:47] <haasn> Maybe because it takes too long?
[11:50] <haasn> The command does seem to be running “just fine”
[12:04] <haasn> Ah, there's some errors in the install log still visible after the failed deployment
[12:05] <haasn> I think I need to run the upgrade from inside the newly booted system
[12:05] <haasn> rather than curtin in-target
[12:24] <haasn> seems like I would want it to go in user_data, but I can't set this via the web UI nor do I want to manually set it for every single node, wouldn't it be possible to have some kind of global “default user_data”?
[12:50] <roaksoax> haasn: why not use the daily images instead? and if you see failed deployment after update/ugrade then something is failing
[16:42] <roaksoax> road/win 4
[17:57] <mup> Bug #1543707 opened: MAAS 1.9+ should not allow whitespace characters in space names <MAAS:New> <https://launchpad.net/bugs/1543707>
=== CyberJacob is now known as Guest83242
[19:15] <haasn> roaksoax: daily images? I am using them, at least as far as I'm aware
[19:15] <haasn> “last update 09 feb 2016”
[19:15] <haasn> but deploying a machine and logging into them still requires like 300 package upgrades
[19:19] <haasn> there is some mention on the bug tracker of this https://bugs.launchpad.net/maas/+bug/1492531
[19:19] <haasn> looks like it's not using daily images by default and afaik there's no way to change that without changing the source?
[19:20] <haasn> The milestone is set as MAAS v1.9.0, perplexingly enough
[19:21] <haasn> 1.9.0 was released and I'm using itb ut the bug still seems to be open?
[19:21] <haasn> Ah, the base URL is a setting in the settings menu
[19:24] <haasn> Unrelated: It seems maas and all of its deployed nodes all automatically live in Etc/Utc timezone, which does make debugging logs etc. slightly more difficult than it needs to be. Is this intentional or could I go ahead and change them all to a german timezone if all my servers are going to be in germany?
[19:35] <roaksoax> haasn: the bug refers to using the images from the 'releases' stream, not from the 'daily' stream. Did you change the stream of images in the settings page?
[19:36] <haasn> roaksoax: I did as soon as I was made aware that you could change the stream of images on the settings page. (I was looking for the setting in the conf files to no avail)
[19:38] <roaksoax> haasn: oh! i think that may  be becuase there's an issue with the image publishing right now, which may not be publishing newer images
[19:38] <roaksoax> haasn: but will have a look in a sec
[19:39] <haasn> roaksoax: I'm confused. I already solved my problem, didn't I?
[19:40] <roaksoax> haasn: right, if you are using daily images then you should be ok, becuase the releases images are out of date
[19:40] <roaksoax> haasn: releases are more in line with current release than latest packages
[19:40] <roaksoax> haasn: but if you using the dailies, and it still has tons of packages to update
[19:40] <roaksoax> then there's two options
[19:40] <roaksoax> 1. new images not being published
[19:41] <roaksoax> 2. or something else is going on
[19:41] <haasn> I dunno, I haven't tested the dailies yet. (It's still synchronizing them)
[19:41] <haasn> I'm going to strongly assume the problem will be fixed after switching to the dailies
[19:41] <roaksoax> haasn: ah ok! then you should be good to go with daily
[19:51] <haasn> Hmm. Is it normal to have ridiculously high IO usage on postgresql after syncing images?
[19:52] <haasn> Like, the machine is nearly unresponsive
[19:55] <haasn> `service restart postgresql` seems to have fixed it?
[19:56] <roaksoax> haasn: for HA, the images are stored in the DB
[19:56] <roaksoax> haasn: and tehn synced to the clusters
[19:58] <haasn> Ah, after waiting a bit the io usage is going up again
[19:58] <haasn> roaksoax: Should I get a dedicated disk for postgresql just to avoid this clogging up the system?
[19:59] <haasn> Or what's the intended design here?
[20:02] <haasn> It's running on a RAID10 of 15000rpm server drives
[20:02] <haasn> And has 10+ GB of RAM left unused
[20:03] <haasn> Maybe if I could get postgresql to keep more in memory and write it slower to the disk that would be fine
[20:07] <roaksoax> yeah that should be fine
[20:07] <roaksoax> haasn: we don't have intended design for postgres
[20:07] <roaksoax> haasn: but with MAAS 2.0, the intention is to have the region controller separated from the cluster controllers  *if* we want to support HA
[20:08] <roaksoax> haasn: so we can have Postgresql in HA, Multiple Region Controllers, Multiple cluster controllers
[20:08] <haasn> Will it still be possible to have the region controller and cluster controller on the same machine for small setups?
[20:08] <haasn> I'm using it to organize a max of 10-20 machines
[20:10] <roaksoax> haasn: yes you will
[20:15] <haasn> roaksoax: I had a chat with #postgresql and it seems ubuntu's defaults for checkpoint_segments is too low
[20:15] <haasn> postgresql is getting overloaded with the large writes and writing checkpoints constantly, hence the high load
[20:15] <haasn> Not sure if it would somehow be possible to tune this parameter when installing maas but I have no idea if there's a favorable interaction here
[20:19] <haasn> (same for checkpoint_completion_target and checkpoint_timeout)
[20:20] <roaksoax> haasn: maybe it would be good to change the defaults altogether
[20:25] <haasn> roaksoax: Deployed from the daily image now, only like 5 packages needed upgrading so it seems to be working. Thanks for the suggestion!
[20:33] <roaksoax> haasn: no prob! we will have better UX for that in 2.1 hopefully
[20:37] <haasn> roaksoax: I would still love an easier way to set up a list of scripts to be run on fresh installs
[20:37] <haasn> for example use cases I may want all new nodes to install a puppet client or set up some ssh certificates
[20:38] <haasn> My ideal UX for this would be a list of shell files I can dump into a postinstall/ directory inside /etc/maas/preseeds
[20:38] <roaksoax> haasn: right, there's definitely a lot of things we have to improve overall
[20:38] <roaksoax> haasn: but we are getting there
[20:38] <haasn> just having shell files like XX_setup_puppet.sh that gets run after a new install would be great
[20:38] <haasn> and not that hard to implement
[20:39] <haasn> Btw, how much “security” is in the design of MAAS? Should I ensure that all communications between the cluster controller and nodes it's setting up happen on a private network? What about API keys, is it safe to access the maas API over the internet using an API key, or does that mean anybody else could steal my API key and do stuff?
[20:40] <haasn> i.e. does the API key function like a private/public system to ensure integrity and identity or is it just a replicatable token
=== menn0_ is now known as menn0
[20:42] <roaksoax> haasn: region/cluster communication is not *yet* happening securely, but we are in the process of getting that in the roadmap for 2.1 or 2.2
[23:10] <mup> Bug #1543828 opened: Install of 1.10 fails due to simplejson dependency <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1543828>