[21:43] mornin [21:43] g [22:20] morning thumper [22:32] o/ [22:48] booked your flights yet thumper? [22:48] ha, no [22:48] need to do the initial paperwork and signoff right? [22:48] yeah [22:48] I did that bit [22:48] doesnt help having a travel provider 9 time zones away [22:49] :) [22:50] i booked my own tickets [22:57] i'm going to start cutting 1.13.3 now [23:00] davecheney: excellent, i'll commit the bootstrap chsnge when the release is done [23:02] wallyworld_: yup, i wanna unblock y'all as soon as possible [23:12] oh fuck [23:12] ubuntu@ip-10-248-63-202:~$ df -h [23:12] Filesystem Size Used Avail Use% Mounted on [23:12] /dev/xvda1 7.9G 7.8G 0 100% / [23:12] showstopped [23:12] showstopper [23:12] :( [23:12] thumper: this is an upgrade problem [23:12] davecheney: where is that? [23:12] this is a 1.12 env upgraded to 1.13 [23:13] is it logging? [23:13] yup [23:13] what is it logging? [23:13] 1.12 did not contain logrotate [23:13] ubuntu@ip-10-248-63-202:~$ ll /var/log/juju/ [23:13] total 3356424 [23:13] drwxr-xr-x 2 root root 4096 Aug 29 00:46 ./ [23:13] drwxr-xr-x 12 root root 4096 Sep 1 06:45 ../ [23:13] -rw-r----- 1 syslog adm 3226935296 Sep 1 23:13 all-machines.log [23:13] -rw-r--r-- 1 root root 5508 Aug 29 00:46 juju-gui.log [23:13] -rw-r--r-- 1 root root 108191300 Sep 1 23:13 machine-0.log [23:13] -rw-r--r-- 1 root root 101818940 Sep 1 23:13 unit-juju-gui-0.log [23:15] thumper: wallyworld_ here is the problem [23:15] if you have a 1.13 environment, log rotate works [23:15] if you have a 1.12 and you upgrade [23:15] log rotate does not work [23:16] i didn't realise we had added log rotate [23:16] wallyworld_: uh, you were the one that did it ... [23:16] haha [23:16] nope, i didn't add log rotate [23:16] that is a metric fuck-ton of logging [23:16] WTH is being logged? [23:16] wallyworld_: there are two bugs about log rotate [23:17] sure, one of the fixes by andrew was to tweak the rsyslof conf [23:17] i'm sure you closed at least one of them [23:17] but that is not log rotate [23:18] thumper: one of the uniters has gone mad [23:18] the fix i'm aware of was to stop rsyslog's processing of rules so it didn't keep adding stuff [23:18] davecheney: gone mad in which way? [23:18] https://bugs.launchpad.net/juju-core/+bug/1218616 [23:19] thumper: appears to have been doing config-changed over and over again [23:19] https://bugs.launchpad.net/juju-core/+bug/1191651 [23:19] ^ wallyworld_ i was wrong [23:19] there are two bugs, both open [23:19] ok, phew, i thought i had missed something [23:20] hmmm, ok, this isn't a blocker [23:20] it's broken in 1.12 and 1.13.x [23:20] i hope you don't want to run your environment for more than 4 days [23:20] ... [23:21] so this is more about a uniter going made [23:21] mad [23:21] and logging constantly? [23:22] thumper: counting [23:23] thumper, the api rpc stuff also logs constantly, though not runaway style [23:23] hazmat: it's not that [23:23] but it is annoying [23:23] yeah [23:23] that was turned off [23:23] I'm working on a branch that should fix this [23:23] how did it get turned back on again ? [23:24] I propose setting the log level of logging every message and response as "trace" [23:24] for juju.api [23:24] +1 [23:25] thumper: ubuntu@ip-10-248-63-202:~$ grep -c config-changed /var/log/juju/all-machines.log [23:25] 3181195 [23:25] config-changed hook has gone ape shit [23:25] hmm [23:25] while amusing, not helpful [23:26] thumper: large log is large [23:26] i'm grabbing it now [23:26] not sure where to put 100 odd gig of logs [23:26] yeah, we probably don't need all of it [23:27] you could probably safely just grab the last few megs [23:27] thumper: wallyworld_ can one of you please merge those two bugs [23:27] and that would be more than enough [23:27] and mark the remaining one as blocker for 1.13.4 [23:27] davecheney: ok [23:28] wallyworld_: while technically it sohuld be a blocker for 1.13.3 [23:28] we need to release taht today [23:28] and make a 1.14 branch [23:28] i'll start a backport document for bugs we have to backport [23:28] they are separate issues though [23:28] no no point marking as dupe [23:29] wallyworld_: ok, i don't really care, as long as it gets fixed [23:29] i know hte one from kurt is really 'do debug-log another way' [23:29] sure [23:29] that won't get done this week [23:29] so maybe the other one from prodstack is the better choice [23:32] hmm... [23:32] it seems that we do now have rsyslog things added [23:32] but I bet on upgrade, we don't add them [23:33] thumper: yup, that was what I thought the problem was initially [23:35] I do wonder where it would specify size... [23:37] rsyslog can do log rotation: http://www.rsyslog.com/doc/log_rotation_fix_size.html [23:38] so we can add that for next release [23:53] wallyworld_: release is tagged, go nuts [23:53] davecheney: awesome, thanks