[00:15] <menn0> wallyworld: verbose test output https://github.com/juju/juju/pull/6514
[00:29] <wallyworld> looking
[00:30] <wallyworld> menn0: i would even love a make arg to add gocheck.vv
[00:30] <wallyworld> so that we can diagnose better hung individual tests
[00:30] <wallyworld> but the change as submitted looks good
[00:45] <menn0> wallyworld: agreed but -check.vv makes things a bit hard to follow though
[00:46] <wallyworld> menn0: yeah, i ws thinking of it as a maek arg so that it was off by default
[00:59] <menn0> wallyworld: interestingly, we already have per package test timings
[00:59] <menn0> wallyworld: it varies a lot. for run-unit-tests-xenial-amd64 I've seen the state package tests take anywhere from 11mins to 19.2 mins.
[01:00] <menn0> wallyworld: maybe we just need to bump the timeout
[01:00] <wallyworld> maybe yeah
[01:04] <anastasiamac> bumping timeout is kind of like sweeping under the rag... is there really no way we can shorten the time the tests run?
[01:04] <anastasiamac> rug*
[01:11] <menn0> anastasiamac: thumper fixed a bunch of unnecessarily long running tests in state recently
[01:11] <menn0> anastasiamac: I don't think there's much low hanging fruit left
[01:12] <menn0> anastasiamac: we keep adding functionality and tests so the test time keeps increasing
[01:12] <axw> menn0: FWIW, in the prometheus metrics I'm gathering from my azure env I can see the effects of periodic txnpruner in terms of deletion ops going up every 2 hours, but there's no corresponding spike in CPU in jujud
[01:12] <menn0> anastasiamac: it also seems like the test run time is highly variable in the virtualized environments in which we run the tests
[01:12] <anastasiamac> menn0: i c... so based on the times u've put ^^, what timeout r u thinking? 25min?
[01:13] <axw> but then I'm not seeing CPU spikes in my env at all
[01:13] <menn0> axw: ok, so it's probably not that
[01:14] <axw> menn0: I'm going to try hammering the disk to see if it triggers
[01:14] <menn0> axw: did you manage to have a look at the logs for the system that is seeing the problem? i.e. is the pruner running for long periods there?
[01:14] <axw> menn0: I think the log level was set to WARNING, I'll have another look
[01:15] <axw> menn0: yeah, set to the default of <root>=WARNING
[01:16] <anastasiamac> menn0: i have a tiny question but it may need context, could u HO or prefer IRC?
[01:17] <menn0> anastasiamac: i'm easy
[01:18] <anastasiamac> in standup?
[01:18] <menn0> anastasiamac: ok
[01:39] <wallyworld> damn, roofing guys caused my power to drop out. hopefully back now for  bit
[01:40] <wallyworld> anastasiamac: it's not necessarily sweeping under rug. if there are X tests and each test takes Y time, then the tests will take X*Y to run. there's not much we can do about mgo startup time etc. of couse we can tune tests, but there's only so much that can be done
[01:43] <natefinch> IIRC, Gustavo said we were doing something stupid with mongo - restarting it for every test or something.  Probably unavoidable per package, but maybe worth looking into to make sure that we're not restarting it for every test?
[01:45] <wallyworld> i *think* someone did look at that. not retarting has other issues, eg test isolation
[01:45] <wallyworld> not sure of the final outcome there
[02:20] <natefinch> heh, I'm running the tests serially to get nicely ordered output and whoo boy, they take a lot longer that way
[02:21] <natefinch> 35 minutes so far.... 35 minutes so far, and usually it's like... I dunno, 15ish?
[02:34] <natefinch> heh: 44m31.434s
[02:42] <anastasiamac> menn0: wallyworld how relevant is this to us now, with new logging/rsyslog in 2.0? https://bugs.launchpad.net/juju/+bug/1562088
[02:42] <mup> Bug #1562088: flexible /etc/rsyslogd.d/25-juju.conf configuration <canonical-bootstack> <juju:Triaged> <https://launchpad.net/bugs/1562088>
[02:47] <menn0> anastasiamac: it's not. 1.25 only
[02:54] <anastasiamac> natefinch: \(ˆ˚ˆ)/
[02:54] <anastasiamac> natefinch: or even better ヽ(´o｀；
[02:56] <anastasiamac> menn0: tyvm
[03:10] <natefinch> PASS: bakerystorage_test.go:72: BakeryStorageSuite.TestExpiryTime	58.037s
[03:10] <natefinch> that's one test
[03:10] <natefinch> PASS: syslog_test.go:220: syslogSuite.TestConfigChange	31.715s
[03:10] <natefinch> PASS: syslog_test.go:193: syslogSuite.TestLogRecordForwarded	31.731s
[03:10] <natefinch> PASS: local_test.go:547: localServerSuite.TestStopInstanceSecurityGroupNotDeleted	29.046s
[03:17] <menn0> natefinch: ouch
[03:17] <menn0> natefinch: we should get that one fixed :)
[03:17]  * menn0 is EOW
[13:01] <dimitern> rick_h_: ping
[13:01] <rick_h_> dimitern: pong
[13:01] <dimitern> rick_h_: do you have a few minutes for a quick HO?
[13:01] <rick_h_> dimitern: otp atm, will have time possiblly before standup or right after
[13:02] <dimitern> rick_h_: sure
[13:10] <abentley> balloons: So our candidate for 2.02 is  8fcc982d ?
[13:11] <abentley> balloons: I mean 2.0.1
[13:33] <rick_h_> dimitern: free if you want to chat
[13:34] <dimitern> rick_h_: ok, joining standup HO
[13:34] <rick_h_> dimitern: rgr omw
[13:47] <voidspace> mgz: ping
[13:47] <mgz> voidspace: yo
[14:00] <rick_h_> dimitern: voidspace natefinch mgz dooferlad ping for standup
[14:00] <mgz> omw
[14:01] <voidspace> rick_h_: omw
[14:06] <katco> natefinch: funny you should send that email out re. tests. i had a test fail bc "`exec /usr/bin/mongod` file not found". i was sad for a bit
[14:06] <katco> and begrudgingly installed mongo =|
[14:07] <natefinch> haha
[14:07] <natefinch> so sad that our "unit" tests depend on installing mongo
[14:08] <katco> natefinch: well beyond that... why are we trying to *start* mongo by just running the binary if it's not running? just fail and let the user maintain a semblance of control over their machine
[14:08] <natefinch> katco: good point
[14:22] <mgz> natefinch: basically, sent message to list just now about ssh from windows client
[14:23] <mgz> atm my fix uses github.com/andybalholm/crlf to wrap stdin
[14:24] <mgz> which is probably not enough code to be bothered importing (not even sure I want the golang.org/x/text dep)
[14:25] <mgz> wondered if you'd come across a modern go idiomatic method of translating newlines
[14:28] <natefinch> mgz: you can do it pretty easily with just a io.reader wrapper... the naive implementation might be slightly worse performance-wise than that implementation, but this isn't something we're using to read hundreds of megabytes either
[14:29] <natefinch> I'll make an example, one sec
[14:29] <mgz> natefinch: ta
[14:38] <natefinch> mgz: https://play.golang.org/p/aS3IFXBgdl
[14:40] <mgz> natefinch: thanks!
[15:27]  * rick_h_ goes to family lunch
[15:45] <redir> alexisb: ping
[16:14] <alexisb> heya redir
[16:14] <alexisb> I am going to go drop off kiddo
[16:15] <alexisb> but will ping when I am back
[17:33] <alexisb> redir, ping
[17:33] <redir> pong
[19:57]  * redir heads out for a bit.