bigjoolsmorning thumper22:20
bigjoolsbooked your flights yet thumper?22:48
thumperha, no22:48
thumperneed to do the initial paperwork and signoff right?22:48
bigjoolsI did that bit22:48
bigjoolsdoesnt help having a travel provider 9 time zones away22:48
wallyworld_i booked my own tickets22:50
davecheneyi'm going to start cutting 1.13.3 now22:57
wallyworld_davecheney: excellent, i'll commit the bootstrap chsnge when the release is done23:00
davecheneywallyworld_: yup, i wanna unblock y'all as soon as possible23:02
davecheneyoh fuck23:12
davecheneyubuntu@ip-10-248-63-202:~$ df -h23:12
davecheneyFilesystem      Size  Used Avail Use% Mounted on23:12
davecheney/dev/xvda1      7.9G  7.8G     0 100% /23:12
davecheneythumper: this is an upgrade problem23:12
thumperdavecheney: where is that?23:12
davecheneythis is a 1.12 env upgraded to 1.1323:12
thumperis it logging?23:13
thumperwhat is it logging?23:13
davecheney1.12 did not contain logrotate23:13
davecheneyubuntu@ip-10-248-63-202:~$ ll /var/log/juju/23:13
davecheneytotal 335642423:13
davecheneydrwxr-xr-x  2 root   root       4096 Aug 29 00:46 ./23:13
davecheneydrwxr-xr-x 12 root   root       4096 Sep  1 06:45 ../23:13
davecheney-rw-r-----  1 syslog adm  3226935296 Sep  1 23:13 all-machines.log23:13
davecheney-rw-r--r--  1 root   root       5508 Aug 29 00:46 juju-gui.log23:13
davecheney-rw-r--r--  1 root   root  108191300 Sep  1 23:13 machine-0.log23:13
davecheney-rw-r--r--  1 root   root  101818940 Sep  1 23:13 unit-juju-gui-0.log23:13
davecheneythumper: wallyworld_ here is the problem23:15
davecheneyif you have a 1.13 environment, log rotate works23:15
davecheneyif you have a 1.12 and you upgrade23:15
davecheneylog rotate does not work23:15
wallyworld_i didn't realise we had added log rotate23:16
davecheneywallyworld_: uh, you were the one that did it ...23:16
wallyworld_nope, i didn't add log rotate23:16
thumperthat is a metric fuck-ton of logging23:16
thumperWTH is being logged?23:16
davecheneywallyworld_: there are two bugs about log rotate23:16
wallyworld_sure, one of the fixes by andrew was to tweak the rsyslof conf23:17
davecheneyi'm sure you closed at least one of them23:17
wallyworld_but that is not log rotate23:17
davecheneythumper: one of the uniters has gone mad23:18
wallyworld_the fix i'm aware of was to stop rsyslog's processing of rules so it didn't keep adding stuff23:18
thumperdavecheney: gone mad in which way?23:18
davecheneythumper: appears to have been doing config-changed over and over again23:19
davecheney^ wallyworld_ i was wrong23:19
davecheneythere are two bugs, both open23:19
wallyworld_ok, phew, i thought i had missed something23:19
davecheneyhmmm, ok, this isn't a blocker23:20
davecheneyit's broken in 1.12 and 1.13.x23:20
davecheneyi hope you don't want to run your environment for more than 4 days23:20
thumperso this is more about a uniter going made23:21
thumperand logging constantly?23:21
davecheneythumper: counting23:22
hazmatthumper, the api rpc stuff also logs constantly, though not runaway style23:23
davecheneyhazmat: it's not that23:23
davecheneybut it is annoying23:23
davecheneythat was turned off23:23
thumperI'm working on a branch that should fix this23:23
davecheneyhow did it get turned back on again ?23:23
thumperI propose setting the log level of logging every message and response as "trace"23:24
thumperfor juju.api23:24
davecheneythumper: ubuntu@ip-10-248-63-202:~$ grep -c config-changed /var/log/juju/all-machines.log23:25
davecheneyconfig-changed hook has gone ape shit23:25
thumperwhile amusing, not helpful23:25
davecheneythumper: large log is large23:26
davecheneyi'm grabbing it now23:26
davecheneynot sure where to put 100 odd gig of logs23:26
thumperyeah, we probably don't need all of it23:26
thumperyou could probably safely just grab the last few megs23:27
davecheneythumper: wallyworld_ can one of you please merge those two bugs23:27
thumperand that would be more than enough23:27
davecheneyand mark the remaining one as blocker for 1.13.423:27
wallyworld_davecheney: ok23:27
davecheneywallyworld_: while technically it sohuld be a blocker for 1.13.323:28
davecheneywe need to release taht today23:28
davecheneyand make a 1.14 branch23:28
davecheneyi'll start a backport document for bugs we have to backport23:28
wallyworld_they are separate issues though23:28
wallyworld_no no point marking as dupe23:28
davecheneywallyworld_: ok, i don't really care, as long as it gets fixed23:29
davecheneyi know hte one from kurt is really 'do debug-log another way'23:29
davecheneythat won't get done this week23:29
davecheneyso maybe the other one from prodstack is the better choice23:29
thumperit seems that we do now have rsyslog things added23:32
thumperbut I bet on upgrade, we don't add them23:32
davecheneythumper: yup, that was what I thought the problem was initially23:33
thumperI do wonder where it would specify size...23:35
wallyworld_rsyslog can do log rotation: http://www.rsyslog.com/doc/log_rotation_fix_size.html23:37
wallyworld_so we can add that for next release23:38
davecheneywallyworld_: release is tagged, go nuts23:53
wallyworld_davecheney: awesome, thanks23:53

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!