haasnHmm.. some update on the IPMI issue I had: Turns out when I said it was working.. well, not fully. It works for --stat, but it doesn't work for turning the power on or off01:26
haasnBut the odd thing is: For power on/off it doesn't even seem to be launching ipmipower at all01:27
haasnI don't see any processes created the way I see processes created every time I click “check now”01:27
haasnAh, it's going through ipmi-chassis-config - of course01:31
mupBug #1524853 changed: DHCP lease allocation does not match deployed node's IP address <MAAS:Expired> <https://launchpad.net/bugs/1524853>04:22
haasnWhat's the easiest way to customize the images usable in MAAS, preferably in a uniform way? For example, I want to make a change to the default /etc/cloud/cloud.cfg to change the way new systems are deployed09:57
haasnBut I don't really want to maintain my own custom image - I'd rather this change be “universal”, if that makes sense09:58
haasnHmm. I guess I want to be modifying preseeds/curtin_userdata?10:04
haasnNo, that doesn't seem right. Curtin only deals with the installation process - but I want to customize things *after* the installation is successful (i.e. from within the new system, before cloud-init runs)10:08
mupBug #1519810 changed: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>11:05
mupBug #1519810 opened: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>11:14
mupBug #1519810 changed: cluster controller failing to detect interfaces on bootstrap <doc> <sts> <MAAS:Invalid> <MAAS 1.8:Invalid> <https://launchpad.net/bugs/1519810>11:17
haasnI think part of what I want to accomplish is done by adding late_commands to curtin_userdata. I tried adding two commands (to run `apt-get update` and `apt-get upgrade` automatically on newly deployed nodes) but now I get some weird behavior: when deploying, the log eventually changes from “deployed” to “failed deploying” but the node itself still gets displayed as ‘deploying’11:40
haasnSeems like there's some sort of weird inconsistency in the MAAS database itself here, where the log clearly thinks the status got changed but the UI disagrees11:41
haasnWithout the lines I added it works fine11:41
haasnThe change I made is adding the two apt-get lines here: https://0x0.st/Xt8.txt11:41
haasnAnybody know what's wrong with them? It's not like I can see a log output because the UI doesn't even think installation failed (yet)11:42
haasnIncluding just the update line seems to work fine, it's the upgrade line that's causing problems11:47
haasnMaybe because it takes too long?11:47
haasnThe command does seem to be running “just fine”11:50
haasnAh, there's some errors in the install log still visible after the failed deployment12:04
haasnI think I need to run the upgrade from inside the newly booted system12:05
haasnrather than curtin in-target12:05
haasnseems like I would want it to go in user_data, but I can't set this via the web UI nor do I want to manually set it for every single node, wouldn't it be possible to have some kind of global “default user_data”?12:24
roaksoaxhaasn: why not use the daily images instead? and if you see failed deployment after update/ugrade then something is failing12:50
roaksoaxroad/win 416:42
mupBug #1543707 opened: MAAS 1.9+ should not allow whitespace characters in space names <MAAS:New> <https://launchpad.net/bugs/1543707>17:57
=== CyberJacob is now known as Guest83242
haasnroaksoax: daily images? I am using them, at least as far as I'm aware19:15
haasn“last update 09 feb 2016”19:15
haasnbut deploying a machine and logging into them still requires like 300 package upgrades19:15
haasnthere is some mention on the bug tracker of this https://bugs.launchpad.net/maas/+bug/149253119:19
haasnlooks like it's not using daily images by default and afaik there's no way to change that without changing the source?19:19
haasnThe milestone is set as MAAS v1.9.0, perplexingly enough19:20
haasn1.9.0 was released and I'm using itb ut the bug still seems to be open?19:21
haasnAh, the base URL is a setting in the settings menu19:21
haasnUnrelated: It seems maas and all of its deployed nodes all automatically live in Etc/Utc timezone, which does make debugging logs etc. slightly more difficult than it needs to be. Is this intentional or could I go ahead and change them all to a german timezone if all my servers are going to be in germany?19:24
roaksoaxhaasn: the bug refers to using the images from the 'releases' stream, not from the 'daily' stream. Did you change the stream of images in the settings page?19:35
haasnroaksoax: I did as soon as I was made aware that you could change the stream of images on the settings page. (I was looking for the setting in the conf files to no avail)19:36
roaksoaxhaasn: oh! i think that may  be becuase there's an issue with the image publishing right now, which may not be publishing newer images19:38
roaksoaxhaasn: but will have a look in a sec19:38
haasnroaksoax: I'm confused. I already solved my problem, didn't I?19:39
roaksoaxhaasn: right, if you are using daily images then you should be ok, becuase the releases images are out of date19:40
roaksoaxhaasn: releases are more in line with current release than latest packages19:40
roaksoaxhaasn: but if you using the dailies, and it still has tons of packages to update19:40
roaksoaxthen there's two options19:40
roaksoax1. new images not being published19:40
roaksoax2. or something else is going on19:41
haasnI dunno, I haven't tested the dailies yet. (It's still synchronizing them)19:41
haasnI'm going to strongly assume the problem will be fixed after switching to the dailies19:41
roaksoaxhaasn: ah ok! then you should be good to go with daily19:41
haasnHmm. Is it normal to have ridiculously high IO usage on postgresql after syncing images?19:51
haasnLike, the machine is nearly unresponsive19:52
haasn`service restart postgresql` seems to have fixed it?19:55
roaksoaxhaasn: for HA, the images are stored in the DB19:56
roaksoaxhaasn: and tehn synced to the clusters19:56
haasnAh, after waiting a bit the io usage is going up again19:58
haasnroaksoax: Should I get a dedicated disk for postgresql just to avoid this clogging up the system?19:58
haasnOr what's the intended design here?19:59
haasnIt's running on a RAID10 of 15000rpm server drives20:02
haasnAnd has 10+ GB of RAM left unused20:02
haasnMaybe if I could get postgresql to keep more in memory and write it slower to the disk that would be fine20:03
roaksoaxyeah that should be fine20:07
roaksoaxhaasn: we don't have intended design for postgres20:07
roaksoaxhaasn: but with MAAS 2.0, the intention is to have the region controller separated from the cluster controllers  *if* we want to support HA20:07
roaksoaxhaasn: so we can have Postgresql in HA, Multiple Region Controllers, Multiple cluster controllers20:08
haasnWill it still be possible to have the region controller and cluster controller on the same machine for small setups?20:08
haasnI'm using it to organize a max of 10-20 machines20:08
roaksoaxhaasn: yes you will20:10
haasnroaksoax: I had a chat with #postgresql and it seems ubuntu's defaults for checkpoint_segments is too low20:15
haasnpostgresql is getting overloaded with the large writes and writing checkpoints constantly, hence the high load20:15
haasnNot sure if it would somehow be possible to tune this parameter when installing maas but I have no idea if there's a favorable interaction here20:15
haasn(same for checkpoint_completion_target and checkpoint_timeout)20:19
roaksoaxhaasn: maybe it would be good to change the defaults altogether20:20
haasnroaksoax: Deployed from the daily image now, only like 5 packages needed upgrading so it seems to be working. Thanks for the suggestion!20:25
roaksoaxhaasn: no prob! we will have better UX for that in 2.1 hopefully20:33
haasnroaksoax: I would still love an easier way to set up a list of scripts to be run on fresh installs20:37
haasnfor example use cases I may want all new nodes to install a puppet client or set up some ssh certificates20:37
haasnMy ideal UX for this would be a list of shell files I can dump into a postinstall/ directory inside /etc/maas/preseeds20:38
roaksoaxhaasn: right, there's definitely a lot of things we have to improve overall20:38
roaksoaxhaasn: but we are getting there20:38
haasnjust having shell files like XX_setup_puppet.sh that gets run after a new install would be great20:38
haasnand not that hard to implement20:38
haasnBtw, how much “security” is in the design of MAAS? Should I ensure that all communications between the cluster controller and nodes it's setting up happen on a private network? What about API keys, is it safe to access the maas API over the internet using an API key, or does that mean anybody else could steal my API key and do stuff?20:39
haasni.e. does the API key function like a private/public system to ensure integrity and identity or is it just a replicatable token20:40
=== menn0_ is now known as menn0
roaksoaxhaasn: region/cluster communication is not *yet* happening securely, but we are in the process of getting that in the roadmap for 2.1 or 2.220:42
mupBug #1543828 opened: Install of 1.10 fails due to simplejson dependency <MAAS:In Progress by allenap> <https://launchpad.net/bugs/1543828>23:10

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!