mupBug #1634390 changed: jujud services not starting after reboot when /var is on separate partition  <uosci> <juju:Triaged> <juju-core:Won't Fix> <https://launchpad.net/bugs/1634390>00:03
* thumper relocates while care gets serviced00:12
blahdeblahAny ETA on 2.2 release?  Asking for a friend's HA controller. :-)00:28
wallyworldblahdeblah: 2.2 rc1 this week or more likely early next week; 2.2 fa soon after01:17
blahdeblahanastasiamac: or s/fa/ga/ even :-)01:18
blahdeblahwallyworld: thanks01:19
anastasiamacblahdeblah: as long as u've figured :)01:19
anastasiamacdetails... you know?... :D01:19
wallyworldbabbageclunk: free now if you want to chat01:21
babbageclunkwallyworld: hey, yes please - back in standup?01:24
wallyworldsure  be there in a sec01:25
axw_wallyworld: we've got a lot of landing failures in the api package, going to look at that before windows things01:46
wallyworldaxw_: ok. i'm free now too whenever you wanted to talk01:47
axw_wallyworld: ok, see you in standup then01:47
babbageclunkwallyworld: yay, it turns out that MachineAgent.apiserverWorkerStarter would just leak the state if any error occurred when creating the apiserver in MachineAgent.newAPIserverWorker. The syslog tests pass now.02:11
axw_anastasiamac: would you kindly review https://github.com/juju/juju/pull/7428?02:47
anastasiamacme looking02:47
anastasiamacaxw_: lgtm, tyvm!!02:48
thumperaxw_: is this one that either you or wallyworld have fixed? https://bugs.launchpad.net/juju/+bug/166504003:45
mupBug #1665040: Race in github.com/juju/juju/worker/peergrouper <ci> <race-condition> <regression> <unit-tests> <juju:Triaged> <https://launchpad.net/bugs/1665040>03:45
wallyworldthumper: yep, axw i think, i'd need to check the pr03:46
axw_thumper wallyworld: pretty sure I fixed a different one, checking now03:46
wallyworldbut there's no more peer grouper races in the lastest runs03:46
axw_thumper: actually my PR would have fixed a bunch of tests, it was related to some common code. so yes03:47
thumperaxw_: can you put the pr in that bug?03:47
axw_wallyworld: https://github.com/juju/juju/pull/7429 should fix the windows test failure. going for a ride, bbs04:18
axw_feel free to $$merge$$ if you're happy with it04:18
wallyworldaxw_: alright. after that i need to talk to you about storage04:18
thumperveebers: ping04:54
thumperveebers: are you able to jump on a quick hangout?04:54
veebersthumper: yep, real quick have a standup coming up :-)04:56
thumperoh, you go to jam's?04:56
veebersaye, most of the time04:56
babbageclunkha ha, state has 270 public methods.05:01
veebersthumper: any luck with that test now?05:17
thumperveebers: got some time?05:37
thumperI only have a few minutes before heading out05:37
thumpertaking Maia to guides05:37
thumperveebers: I'm still in quick05:40
thumperveebers: nm, have to head out now05:44
thumperI'm running an attempt at a test fix05:44
wallyworldbabbageclunk: burton-aus: i think this failure may just be a slight difference in file content, but i haven't looked closely http://reports.vapour.ws/releases/5321/job/log-forward/attempt/123805:44
jamaxw_: babbageclunk: I've been doing some tweaking on the internals of mgo/txn, and while I'd usually reach out to menn0, he's not around to discuss them. Are either of you interested?05:45
babbageclunkjam: I would be, but I need to drop soon for child feeding and hosing down, sorry.05:47
jambabbageclunk: well, these are not high priority, so if you're interested in the area, we can schedule it for the future05:47
babbageclunkjam: yeah, definitely!05:47
babbageclunkwhoa, that was probably more enthusiasm than I intended.05:48
babbageclunkBut I definitely am interested.05:48
veebersugh, sorry thumper was peeling potatoes :-\05:49
babbageclunkwallyworld, burton-aus: I haven't looked very hard at that, but there are definitely forwarded messages in the logs there, so I think it might be a test issue?05:53
wallyworldbabbageclunk: yeah. my initial thought was that the changes done should have been transparent, so if the test was passing before, it should pass now also05:54
babbageclunkwallyworld: now that I think about it there might be ordering differences (since the logs for each model would be forwarded independently), or the forwarding might only be set up for the controller model in the test (expecting that would also forward the model logs, which isn't true any more).05:56
wallyworldbabbageclunk: the latter sounds more plausible05:57
wallyworldwe'll have to get the test updated05:57
babbageclunkwallyworld: want me to take a look at that? I think I'm finished the log collection splitting. Just fixing state tests that look directly in the logx collection, and then migration steps.05:58
babbageclunkI mean, upgrade steps05:58
wallyworldbabbageclunk: i think it maybe better to continue your wip05:59
wallyworldbabbageclunk: burton-aus might get to it first, otherwise you could look after putting up the log split PR06:02
wallyworldeven if we just identify that the test needs fixing, we can sort out something to unblock the release06:02
axw_wallyworld: sorry, you wanted to chat storage? 1:1?06:15
wallyworldaxw_: yeah, standup ho?06:25
wallyworldaxw_: https://github.com/wallyworld/juju/compare/cleanup-removes-app-artefacts...wallyworld:cleanup-removes-app-artefacts2?expand=106:26
axw_jam: if you have something written down about your changes, I'd be interested to read - I don't know enough about the insides of mgo/txn to provide immediate useful feedback06:41
=== axw_ is now known as axw
jamaxw: sure. the specific changes in this case are doing some caching and preloading of db requests06:42
babbageclunkwallyworld: yeah, I thought that too, just checking.06:54
veeberswallyworld, burton-aus: with the log-forwarding test, the test uses a regex to look for log entries that come from the other machine, that might need to be updated (maybe simplified)07:10
burton-ausveebers this is the one I guess:07:11
burton-aus"^[A-Z][a-z]{,2}\ +[0-9]+\ +[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}\ machine-0.3ec9b846\-9520\-4d40\-87f2\-5c9114c8a28f\ jujud-machine-agent-3ec9b846\-9520\-4d40\-87f2\-5c91\ .*$"07:12
burton-ausveebers though that machine related string is just fetched from the run.07:12
veebersburton-aus: aye, that's the one.07:12
veebersbabbageclunk, wallyworld, burton-aus: It's kind of hidden but the failure is in ensure_multiple_models_forward_messages, which adds a new model and deploys something, then checks that logs from that model appears in the rsyslog machine logs07:16
veebersso Looking at what babbageclunk mentioned, perhaps there is some extra config needed to make sure those logs get forwarded as well?07:17
wallyworldveebers: that bit should be transparent IIANM07:21
veeberswallyworld: I'm sorry I don't understand, which part, that there needs to be extra config for the models? Or that there shouldn't be any need for extra config?07:23
wallyworldno need for extra config07:23
wallyworldif the test is checking logs from a model, that bit should work the same as before07:24
wallyworldaxw: i think we have an issue still - cleanupDyingUnit calls cleanupUnitStorageAttachments() with remove=false, so the storage removal doesn't happen, and EnsureDead() fails. i can't see that the processing of a dying unit adds a cleanup job to remove dying storage07:24
veeberswallyworld: ah cool, thanks for clarifying. It's possible the regex check needs tweaked (and or relaxed) if the format has changed a bit07:25
wallyworldi am likely missing something07:25
axwwallyworld: just a minute, looking07:25
wallyworldveebers: the format should be the same also07:25
wallyworldveebers: xtian will need to look into it a bit07:25
axwwallyworld: right, so cleanupDyingUnit causes the storage attachments to go to Dying (detach but don't remove)07:26
wallyworldveebers: it could be a test tweak as well, we just don't know yet07:26
axwwallyworld: then the uniter will run detach-storage hooks07:26
axwand will then remove them07:26
wallyworldaxw: ah right, i need to run that bit manually as well07:26
veeberswallyworld: ack ok, keep us posted :-)07:32
thumperjam: just here to beg a review07:33
jamthumper: of?07:33
veebersthumper: I missed your ping before, available now if you like07:34
thumperveebers: see PR07:34
thumperveebers: can you just check the CI test aspects?07:35
veebersthumper: link to PR?07:36
thumperveebers: two lines above your mention07:36
veebersthumper: ah ha :-) looking now07:37
thumperit really is pretty simple07:39
thumper7 files, +10 −607:39
veebersthumper: sweet, commented. LGTM07:40
thumperjam: ?07:40
jamthumper: was otp, do you want it right away?07:40
thumperI'll poke axw07:41
thumperI was wanting to kick off the merge07:41
thumperit's very very simple07:41
* thumper looks at axw07:41
* thumper will pop back in 10min07:42
* thumper needs to clean house a bit07:42
thumperveebers: I'm assuming for develop we use the in tree tests and charms07:42
jamthumper: are we guaranteed never to see .log before it becomes .log.gz07:42
veebersthumper: not yet. That's something we're working toward (won't be far away)07:44
thumperveebers: oh... well the CI test will fail then07:45
thumperyou do see a .log before it becomes .log.gz, but just very briefly07:46
veebersthumper: ack, once that branch lands we can do the separate "update all" which will propagate the changes (then re-run the test again if needed)07:47
thumperveebers: but... but.. then the tests will fail from older versions07:47
thumperor have you fixed that?07:48
axwthumper: sorry I was afk, you wanted a review from me?07:48
thumperaxw: discussing ^^07:48
thumperjam: I guess if we hit a weird timing issue, we can add a sleep 5 to the action07:48
thumperbut it passed here07:49
thumperon lxd with ssd07:49
thumperwell... I guess we'll find out07:49
* thumper needs to head off now...07:49
axwveebers: I can merge tim's branch, but CI will start failing - how much longer are you around? isn't it already past your EOD?07:56
veebersaxw: aye it is, the CI test will fail on the revision-build, we can make it pass though by updating the nodes once it lands07:59
veebersIt's a bit messy as we're in the process of making it so testing is done from in tree07:59
axwveebers: ok. looks like this is meant to be going into rc1, so I'll merge and hopefully balloons can sort it out when he wakes up08:00
veebersaxw: ack, I'll email and let him kknow08:01
axwveebers: thanks :)08:02
=== salmankhan1 is now known as salmankhan
wallyworldrogpeppe: hey, i'm told  a recent change to add/use a dns cache may have added a flakey test, TestDNSCacheUsed... any chance you could look? we are trying to get an rc out this week. here's an example of a failure http://reports.vapour.ws/releases/5325/job/run-unit-tests-xenial-amd64/attempt/25512:09
rogpeppewallyworld: was that before https://github.com/juju/juju/pull/7429 landed?12:12
wallyworldrogpeppe: it's off the latest CI run, let me check to see that rev it is12:13
rogpeppewallyworld: thanks12:14
wallyworldrogpeppe: yeah, the CI run is from testing PR 7430 which landed 5 hours after12:15
rogpeppewallyworld: OK, i'll look into it12:16
wallyworldrogpeppe: tyvm, i'm off to bed real soon12:16
wallyworldwe are looking to get a good CI run for the morning in australia12:16
wpkaround midnight UTC?12:18
rogpeppewallyworld: there's one problem that really should be fixed before release12:18
rogpeppewallyworld: https://bugs.launchpad.net/juju/+bug/169290512:20
mupBug #1692905: cert error on public controller: cannot validate certificate  <juju:New> <https://launchpad.net/bugs/1692905>12:20
rogpeppewallyworld: i'm working on the fix12:20
balloonshey wallyworld ;)12:22
balloonsLooks like if you land the unit test fix the only issue will be with the windows deploy test. We think the slave is sick12:24
balloonsrogpeppe, will you have a fix for that today?12:25
rogpeppeballoons: i am hoping to, yes12:25
rogpeppeballoons: i've fixed the code - just writing tests for it12:26
balloonsAwesome. So we can get a bless on that landing. Changing any dependencies?12:27
rogpeppeballoons: here's a fix for another flaky test of mine... https://github.com/juju/juju/pull/743412:27
balloonsrogpeppe, ack. Good stuff.12:29
=== akhavr1 is now known as akhavr
mupBug #1694988 opened: AWS instances created by juju don't have an associated IPv6, even if "auto-assign IPv6 addresses" is enabled for the subnet <juju-core:New> <https://launchpad.net/bugs/1694988>13:28
=== pathcl is now known as path
=== path is now known as pathcl
=== akhavr1 is now known as akhavr
rogpeppethis PR fixes juju bug 1692905: https://github.com/juju/juju/pull/7438; reviews appreciated17:15
mupBug #1692905: cert error on public controller: cannot validate certificate  <juju:New> <https://launchpad.net/bugs/1692905>17:15
wpkrogpeppe: unit tests are failing18:00
cmarswpk, could i please get a review of https://github.com/juju/juju/pull/7439 ?18:17
wpkcmars: done.19:04
cmarswpk, thanks19:04
rogpeppewpk: looking19:12
rogpeppecmars: you could look at the server version and warn if it's 2.2 or greater, I guess19:14
rogpeppewpk: ok, a bunch of fairly trivial things; i was too lazy to run the tests my own machine, can you tell? :)19:15
rogpeppewpk: i'd very much appreciate a review if you're up for it, BTW19:26
rogpeppewpk: tests should be fixed now19:29
wpkrogpeppe: full suite kills my laptop, so I always do the bare minimum and then test it in Jenkins :)19:40
rogpeppewpk: at least the tests are now run before you hit $$merge$$19:40
babbageclunkveebers: hey, sorry to miss the discussion last night - I think if you set up the model defaults with log forwarding settings before creating the model it should start forwarding logs straight away.21:14
veebersbabbageclunk: that's contrary to what wallyworld said re: the settings being transparent isn't it?21:18
wallyworldveebers: you always did need to set up log forwarding in the initial config21:19
babbageclunkveebers: It's transparent if you were already setting the model defaults. ;)21:19
veebersah right21:19
* veebers checks what the test is doing now21:19
babbageclunkveebers: There is a change in behaviour - before if you set up forwarding for the controller it would automatically forward for all models. Now if you want that you need to put the settings in model defaults.21:21
veebersbabbageclunk: ah right ok, I think that's the missing part in my thinking, cheers21:22
babbageclunkveebers: Sorry, I probably should have mentioned that earlier!21:23
veebersbabbageclunk: you have docs re: what the settings values are for that?21:24
externalrealityDoes anyone happen to know if there exists CI jobs that depend on the fact that GO is being installed by Juju's top level Makefile?21:26
veebersballoons: is it only unit tests that we expect juju to install go as part of it's own setup?21:28
balloonsexternalreality, yes we do depend on it21:28
balloonsveebers, we we run the merge jobs we use the makefile on the new instance to test21:29
babbageclunkveebers: no, I don't think it's documented anywhere. The settings haven't changed - still logforward-enabled, syslog-host, syslog-ca-cert, syslog-client-cert, syslog-client key.21:32
babbageclunkveebers: Just how they're used has changed.21:33
veebersbabbageclunk: do you need to set the syslog-* stuff on the model config too?21:37
babbageclunkveebers: yup - they could be independent, in theory.21:39
babbageclunkveebers: Want to have a quick hangout about it?21:40
veebersbabbageclunk: would love to, just in release call, will be a little bit before i'm free, can I ping you?21:44
rogpeppethumper, axw, wallyworld: you might wanna take a look at this PR - I'm hoping it can land before the release: https://github.com/juju/juju/pull/743821:45
wallyworldrogpeppe: might be able to :-) btw there's still a dns cache related race21:46
babbageclunkveebers: yup yup21:46
rogpeppewallyworld: got a link to a failure?21:47
wallyworldgive me a sec to find it21:47
wallyworldrogpeppe: TestDNSCacheUsed. I think you fixed a different one. http://reports.vapour.ws/releases/5331/job/run-unit-tests-race/attempt/283121:48
rogpeppewallyworld: ooh, an actual race!21:48
rogpeppewallyworld: thanks21:48
wallyworldrogpeppe: we are delaying rc1 till next monday/tuesday, so that will get time for your pr to land21:49
wallyworldrogpeppe: yeah, we have fixed several actual races this week elsewhere as well. so close to getting a proper blessed CI run21:50
rogpeppewallyworld: ok, that's a trivial fix21:52
babbageclunkIs anyone else having trouble with pushes to github taking a long time?21:58
veebersbabbageclunk: I did last night and thought it was just my internet (or that I did something wrong)21:58
rogpeppewallyworld: https://github.com/juju/juju/pull/744021:59
wallyworldrogpeppe: you rock, ty, will look real soon22:00
wallyworldi'll merge22:00
wallyworldrick_h: just finishing meeting, be there in a sec22:00
rick_hwallyworld: all good22:00
babbageclunkveebers: Mine's just sitting here with a git pack-objects process doing nothing. I guess the other end of the connection is loaded?22:03
babbageclunkugh, finally!22:05
thumperbabbageclunk: can you join the release call plz?22:08
babbageclunkthumper: sure22:08
wallyworldrick_h: still in release call, can we delay for 15 mins, or defer?22:08
thumperrick_h: I'm feeling left out, you haven't asked me for a call22:08
rick_hwallyworld: rgr, just setup something that works for you next week if that's ok22:08
rick_hthumper: well, I like wallyworld :P22:08
wallyworldrick_h: sure, and sorry, 2.2 release is so close22:09
wallyworldwe need to get stuff sorted22:09
rick_hwallyworld: <322:09
rick_hwallyworld: thumper the one thing for 2.2 I wanted to bring up is if this article effects instance type availability and needs to be mentioned. https://goo.gl/VNe9oC22:11
rick_hwallyworld: other than that I'll catch you later22:11
wallyworldrick_h: we'll look at it22:11
wallyworldbabbageclunk: could you also tweak the controller setting max-txn-log-size when you do the mustString() thing for the other ones?23:14
babbageclunkwallyworld: ok23:15
babbageclunkwallyworld: we don't ww23:15
babbageclunkwallyworld: covfefe23:15
babbageclunkwallyworld: We don't want to run the log pruner per-model do we?23:16
wallyworldit should be like the status history pruner23:16
wallyworldi think that's per model23:16
babbageclunkwallyworld: oh, ok - I'll take a look at that23:17

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!