[21:43] <thumper> mornin
[21:43] <thumper> g
[22:20] <bigjools> morning thumper
[22:32] <thumper> o/
[22:48] <bigjools> booked your flights yet thumper?
[22:48] <thumper> ha, no
[22:48] <thumper> need to do the initial paperwork and signoff right?
[22:48] <bigjools> yeah
[22:48] <bigjools> I did that bit
[22:48] <bigjools> doesnt help having a travel provider 9 time zones away
[22:49] <thumper> :)
[22:50] <wallyworld_> i booked my own tickets
[22:57] <davecheney> i'm going to start cutting 1.13.3 now
[23:00] <wallyworld_> davecheney: excellent, i'll commit the bootstrap chsnge when the release is done
[23:02] <davecheney> wallyworld_: yup, i wanna unblock y'all as soon as possible
[23:12] <davecheney> oh fuck
[23:12] <davecheney> ubuntu@ip-10-248-63-202:~$ df -h
[23:12] <davecheney> Filesystem      Size  Used Avail Use% Mounted on
[23:12] <davecheney> /dev/xvda1      7.9G  7.8G     0 100% /
[23:12] <davecheney> showstopped
[23:12] <davecheney> showstopper
[23:12] <thumper> :(
[23:12] <davecheney> thumper: this is an upgrade problem
[23:12] <thumper> davecheney: where is that?
[23:12] <davecheney> this is a 1.12 env upgraded to 1.13
[23:13] <thumper> is it logging?
[23:13] <davecheney> yup
[23:13] <thumper> what is it logging?
[23:13] <davecheney> 1.12 did not contain logrotate
[23:13] <davecheney> ubuntu@ip-10-248-63-202:~$ ll /var/log/juju/
[23:13] <davecheney> total 3356424
[23:13] <davecheney> drwxr-xr-x  2 root   root       4096 Aug 29 00:46 ./
[23:13] <davecheney> drwxr-xr-x 12 root   root       4096 Sep  1 06:45 ../
[23:13] <davecheney> -rw-r-----  1 syslog adm  3226935296 Sep  1 23:13 all-machines.log
[23:13] <davecheney> -rw-r--r--  1 root   root       5508 Aug 29 00:46 juju-gui.log
[23:13] <davecheney> -rw-r--r--  1 root   root  108191300 Sep  1 23:13 machine-0.log
[23:13] <davecheney> -rw-r--r--  1 root   root  101818940 Sep  1 23:13 unit-juju-gui-0.log
[23:15] <davecheney> thumper: wallyworld_ here is the problem
[23:15] <davecheney> if you have a 1.13 environment, log rotate works
[23:15] <davecheney> if you have a 1.12 and you upgrade
[23:15] <davecheney> log rotate does not work
[23:16] <wallyworld_> i didn't realise we had added log rotate
[23:16] <davecheney> wallyworld_: uh, you were the one that did it ...
[23:16] <thumper> haha
[23:16] <wallyworld_> nope, i didn't add log rotate
[23:16] <thumper> that is a metric fuck-ton of logging
[23:16] <thumper> WTH is being logged?
[23:16] <davecheney> wallyworld_: there are two bugs about log rotate
[23:17] <wallyworld_> sure, one of the fixes by andrew was to tweak the rsyslof conf
[23:17] <davecheney> i'm sure you closed at least one of them
[23:17] <wallyworld_> but that is not log rotate
[23:18] <davecheney> thumper: one of the uniters has gone mad
[23:18] <wallyworld_> the fix i'm aware of was to stop rsyslog's processing of rules so it didn't keep adding stuff
[23:18] <thumper> davecheney: gone mad in which way?
[23:18] <davecheney> https://bugs.launchpad.net/juju-core/+bug/1218616
[23:19] <davecheney> thumper: appears to have been doing config-changed over and over again
[23:19] <davecheney> https://bugs.launchpad.net/juju-core/+bug/1191651
[23:19] <davecheney> ^ wallyworld_ i was wrong
[23:19] <davecheney> there are two bugs, both open
[23:19] <wallyworld_> ok, phew, i thought i had missed something
[23:20] <davecheney> hmmm, ok, this isn't a blocker
[23:20] <davecheney> it's broken in 1.12 and 1.13.x
[23:20] <davecheney> i hope you don't want to run your environment for more than 4 days
[23:20] <davecheney> ...
[23:21] <thumper> so this is more about a uniter going made
[23:21] <thumper> mad
[23:21] <thumper> and logging constantly?
[23:22] <davecheney> thumper: counting
[23:23] <hazmat> thumper, the api rpc stuff also logs constantly, though not runaway style
[23:23] <davecheney> hazmat: it's not that
[23:23] <davecheney> but it is annoying
[23:23] <hazmat> yeah
[23:23] <davecheney> that was turned off
[23:23] <thumper> I'm working on a branch that should fix this
[23:23] <davecheney> how did it get turned back on again ?
[23:24] <thumper> I propose setting the log level of logging every message and response as "trace"
[23:24] <thumper> for juju.api
[23:24] <davecheney> +1
[23:25] <davecheney> thumper: ubuntu@ip-10-248-63-202:~$ grep -c config-changed /var/log/juju/all-machines.log
[23:25] <davecheney> 3181195
[23:25] <davecheney> config-changed hook has gone ape shit
[23:25] <thumper> hmm
[23:25] <thumper> while amusing, not helpful
[23:26] <davecheney> thumper: large log is large
[23:26] <davecheney> i'm grabbing it now
[23:26] <davecheney> not sure where to put 100 odd gig of logs
[23:26] <thumper> yeah, we probably don't need all of it
[23:27] <thumper> you could probably safely just grab the last few megs
[23:27] <davecheney> thumper: wallyworld_ can one of you please merge those two bugs
[23:27] <thumper> and that would be more than enough
[23:27] <davecheney> and mark the remaining one as blocker for 1.13.4
[23:27] <wallyworld_> davecheney: ok
[23:28] <davecheney> wallyworld_: while technically it sohuld be a blocker for 1.13.3
[23:28] <davecheney> we need to release taht today
[23:28] <davecheney> and make a 1.14 branch
[23:28] <davecheney> i'll start a backport document for bugs we have to backport
[23:28] <wallyworld_> they are separate issues though
[23:28] <wallyworld_> no no point marking as dupe
[23:29] <davecheney> wallyworld_: ok, i don't really care, as long as it gets fixed
[23:29] <davecheney> i know hte one from kurt is really 'do debug-log another way'
[23:29] <wallyworld_> sure
[23:29] <davecheney> that won't get done this week
[23:29] <davecheney> so maybe the other one from prodstack is the better choice
[23:32] <thumper> hmm...
[23:32] <thumper> it seems that we do now have rsyslog things added
[23:32] <thumper> but I bet on upgrade, we don't add them
[23:33] <davecheney> thumper: yup, that was what I thought the problem was initially
[23:35] <thumper> I do wonder where it would specify size...
[23:37] <wallyworld_> rsyslog can do log rotation: http://www.rsyslog.com/doc/log_rotation_fix_size.html
[23:38] <wallyworld_> so we can add that for next release
[23:53] <davecheney> wallyworld_: release is tagged, go nuts
[23:53] <wallyworld_> davecheney: awesome, thanks