| thumper | mornin | 21:43 |
|---|---|---|
| thumper | g | 21:43 |
| bigjools | morning thumper | 22:20 |
| thumper | o/ | 22:32 |
| bigjools | booked your flights yet thumper? | 22:48 |
| thumper | ha, no | 22:48 |
| thumper | need to do the initial paperwork and signoff right? | 22:48 |
| bigjools | yeah | 22:48 |
| bigjools | I did that bit | 22:48 |
| bigjools | doesnt help having a travel provider 9 time zones away | 22:48 |
| thumper | :) | 22:49 |
| wallyworld_ | i booked my own tickets | 22:50 |
| davecheney | i'm going to start cutting 1.13.3 now | 22:57 |
| wallyworld_ | davecheney: excellent, i'll commit the bootstrap chsnge when the release is done | 23:00 |
| davecheney | wallyworld_: yup, i wanna unblock y'all as soon as possible | 23:02 |
| davecheney | oh fuck | 23:12 |
| davecheney | ubuntu@ip-10-248-63-202:~$ df -h | 23:12 |
| davecheney | Filesystem Size Used Avail Use% Mounted on | 23:12 |
| davecheney | /dev/xvda1 7.9G 7.8G 0 100% / | 23:12 |
| davecheney | showstopped | 23:12 |
| davecheney | showstopper | 23:12 |
| thumper | :( | 23:12 |
| davecheney | thumper: this is an upgrade problem | 23:12 |
| thumper | davecheney: where is that? | 23:12 |
| davecheney | this is a 1.12 env upgraded to 1.13 | 23:12 |
| thumper | is it logging? | 23:13 |
| davecheney | yup | 23:13 |
| thumper | what is it logging? | 23:13 |
| davecheney | 1.12 did not contain logrotate | 23:13 |
| davecheney | ubuntu@ip-10-248-63-202:~$ ll /var/log/juju/ | 23:13 |
| davecheney | total 3356424 | 23:13 |
| davecheney | drwxr-xr-x 2 root root 4096 Aug 29 00:46 ./ | 23:13 |
| davecheney | drwxr-xr-x 12 root root 4096 Sep 1 06:45 ../ | 23:13 |
| davecheney | -rw-r----- 1 syslog adm 3226935296 Sep 1 23:13 all-machines.log | 23:13 |
| davecheney | -rw-r--r-- 1 root root 5508 Aug 29 00:46 juju-gui.log | 23:13 |
| davecheney | -rw-r--r-- 1 root root 108191300 Sep 1 23:13 machine-0.log | 23:13 |
| davecheney | -rw-r--r-- 1 root root 101818940 Sep 1 23:13 unit-juju-gui-0.log | 23:13 |
| davecheney | thumper: wallyworld_ here is the problem | 23:15 |
| davecheney | if you have a 1.13 environment, log rotate works | 23:15 |
| davecheney | if you have a 1.12 and you upgrade | 23:15 |
| davecheney | log rotate does not work | 23:15 |
| wallyworld_ | i didn't realise we had added log rotate | 23:16 |
| davecheney | wallyworld_: uh, you were the one that did it ... | 23:16 |
| thumper | haha | 23:16 |
| wallyworld_ | nope, i didn't add log rotate | 23:16 |
| thumper | that is a metric fuck-ton of logging | 23:16 |
| thumper | WTH is being logged? | 23:16 |
| davecheney | wallyworld_: there are two bugs about log rotate | 23:16 |
| wallyworld_ | sure, one of the fixes by andrew was to tweak the rsyslof conf | 23:17 |
| davecheney | i'm sure you closed at least one of them | 23:17 |
| wallyworld_ | but that is not log rotate | 23:17 |
| davecheney | thumper: one of the uniters has gone mad | 23:18 |
| wallyworld_ | the fix i'm aware of was to stop rsyslog's processing of rules so it didn't keep adding stuff | 23:18 |
| thumper | davecheney: gone mad in which way? | 23:18 |
| davecheney | https://bugs.launchpad.net/juju-core/+bug/1218616 | 23:18 |
| davecheney | thumper: appears to have been doing config-changed over and over again | 23:19 |
| davecheney | https://bugs.launchpad.net/juju-core/+bug/1191651 | 23:19 |
| davecheney | ^ wallyworld_ i was wrong | 23:19 |
| davecheney | there are two bugs, both open | 23:19 |
| wallyworld_ | ok, phew, i thought i had missed something | 23:19 |
| davecheney | hmmm, ok, this isn't a blocker | 23:20 |
| davecheney | it's broken in 1.12 and 1.13.x | 23:20 |
| davecheney | i hope you don't want to run your environment for more than 4 days | 23:20 |
| davecheney | ... | 23:20 |
| thumper | so this is more about a uniter going made | 23:21 |
| thumper | mad | 23:21 |
| thumper | and logging constantly? | 23:21 |
| davecheney | thumper: counting | 23:22 |
| hazmat | thumper, the api rpc stuff also logs constantly, though not runaway style | 23:23 |
| davecheney | hazmat: it's not that | 23:23 |
| davecheney | but it is annoying | 23:23 |
| hazmat | yeah | 23:23 |
| davecheney | that was turned off | 23:23 |
| thumper | I'm working on a branch that should fix this | 23:23 |
| davecheney | how did it get turned back on again ? | 23:23 |
| thumper | I propose setting the log level of logging every message and response as "trace" | 23:24 |
| thumper | for juju.api | 23:24 |
| davecheney | +1 | 23:24 |
| davecheney | thumper: ubuntu@ip-10-248-63-202:~$ grep -c config-changed /var/log/juju/all-machines.log | 23:25 |
| davecheney | 3181195 | 23:25 |
| davecheney | config-changed hook has gone ape shit | 23:25 |
| thumper | hmm | 23:25 |
| thumper | while amusing, not helpful | 23:25 |
| davecheney | thumper: large log is large | 23:26 |
| davecheney | i'm grabbing it now | 23:26 |
| davecheney | not sure where to put 100 odd gig of logs | 23:26 |
| thumper | yeah, we probably don't need all of it | 23:26 |
| thumper | you could probably safely just grab the last few megs | 23:27 |
| davecheney | thumper: wallyworld_ can one of you please merge those two bugs | 23:27 |
| thumper | and that would be more than enough | 23:27 |
| davecheney | and mark the remaining one as blocker for 1.13.4 | 23:27 |
| wallyworld_ | davecheney: ok | 23:27 |
| davecheney | wallyworld_: while technically it sohuld be a blocker for 1.13.3 | 23:28 |
| davecheney | we need to release taht today | 23:28 |
| davecheney | and make a 1.14 branch | 23:28 |
| davecheney | i'll start a backport document for bugs we have to backport | 23:28 |
| wallyworld_ | they are separate issues though | 23:28 |
| wallyworld_ | no no point marking as dupe | 23:28 |
| davecheney | wallyworld_: ok, i don't really care, as long as it gets fixed | 23:29 |
| davecheney | i know hte one from kurt is really 'do debug-log another way' | 23:29 |
| wallyworld_ | sure | 23:29 |
| davecheney | that won't get done this week | 23:29 |
| davecheney | so maybe the other one from prodstack is the better choice | 23:29 |
| thumper | hmm... | 23:32 |
| thumper | it seems that we do now have rsyslog things added | 23:32 |
| thumper | but I bet on upgrade, we don't add them | 23:32 |
| davecheney | thumper: yup, that was what I thought the problem was initially | 23:33 |
| thumper | I do wonder where it would specify size... | 23:35 |
| wallyworld_ | rsyslog can do log rotation: http://www.rsyslog.com/doc/log_rotation_fix_size.html | 23:37 |
| wallyworld_ | so we can add that for next release | 23:38 |
| davecheney | wallyworld_: release is tagged, go nuts | 23:53 |
| wallyworld_ | davecheney: awesome, thanks | 23:53 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!