[00:06] sinzui: i don't know mongo very well but those errors could be mongo just refusing to start for whatever reason :-( [00:14] wallyworld_, that's what I see too. I think something has changed and we might need to adapt juju to it [00:16] perrito666: no luck yet, the problem is nothing in that commit would seem to have a direct impact on backup and restore in this fashion. It is really odd. [00:16] wallyworld_, the current tip of juju is failing the same test [00:16] sinzui: the other possibility - we set a mongo socket timeout of 21 sec, which is twice the mongo heartbeat interval. those socket connection errors occur about 30 after the sockets are opened. perhaps mongo is closing its client connections because it thinks they areunused [00:17] the socket timeout was recently increaed from 10s [00:17] so if anything that should have made it more tolerant [00:18] but i'm just guessing that the error is due to mongo socket closure, may not be [00:21] wallyworld_, I don't think recent juju changes are the problem. We know exactly which change the failure happened with https://github.com/juju/juju/commit/635a06380b2df20f829f205e45e2d3a7e9476d0d [00:21] I don't think that is relevant. [00:22] sinzui: the socket timeout increase happened last week before that change [00:22] i was trying to make sense of what the most relevant culprit may have been [00:23] sinzui: but deploy to other clouds works [00:24] wallyworld_, only a few intermittent failures caused by timeouts or resources over the last week. This set rarely failed before a few hours ago. It is the only test on aws though that is bootstrapping with juju-monogodb like this [00:24] wallyworld_, yep all clouds and all the locals love bootstrapping [00:24] so just aws [00:25] If I could boot utopic there, I bet it would fail the same way [00:25] precise doesn't fail...it has mongodb-server [00:28] hmmm [00:28] The ami is two weeks old === Guest70819 is now known as wallyworld === wallyworld is now known as Guest14539 === Guest14539 is now known as wallyworld === wallyworld is now known as Guest41478 === Guest41478 is now known as wallyworld [01:31] sinzui: i just bootstrapped an amazon env on trusty from my local machine [01:33] wallyworld, CI is still trying. Look at this extraordinary effort http://juju-ci.vapour.ws:8080/job/aws-deploy-trusty-amd64/643/console [01:33] wow [01:33] wallyworld, I could kill the job, see if it restart and complete...if so then I think aws was denfintely doing something bad [01:34] sinzui: i wonder if it is region related [01:34] hmm [01:34] 2014-06-20 00:57:20 INFO juju.provider.ec2 ec2.go:643 started instance "i-193ae232" in "us-east-1c" [01:34] that is what just worked for me [01:34] it picked az c [01:34] after it was rejected by a and b as being congested [01:34] wallyworld, us-east-1a [01:35] well, that is the on I left alive anyway [01:35] 2014-06-20 00:57:17 INFO juju.provider.ec2 ec2.go:627 "us-east-1a" is constrained, trying another availability zone [01:35] 2014-06-20 00:57:19 INFO juju.provider.ec2 ec2.go:627 "us-east-1b" is constrained, trying another availability zone [01:35] 2014-06-20 00:57:20 INFO juju.provider.ec2 ec2.go:643 started instance "i-193ae232" in "us-east-1c" [01:35] so i wasn't allowed to use a or b [01:35] wallyworld, that one that is taking forever is us-east-1c [01:36] hmmm ok, that's the one that worked for me [01:36] and it just failed. now the other aws jobs will start [01:36] not sure what instance type it picked, hw is "hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M" [01:37] m1.small maybe [01:37] so i wonder why mine worked [01:37] wallyworld, We get m3.medium by bootstrapping with mem=2G [01:38] i just used the default constraints [01:38] i can try with mem=2G [01:38] did I count wrong? I see the test restarting [01:39] oh this is wrong, CI is only allow to test this 5 times. It is now doing 7 [01:39] wwitzel3: well tomorrow is a holiday here so I wont be here, this is your problem for a day :|, I think we need to find out what is re-writing agents.conf and in which documents is that info stored to see which one we are missing, my guess is that wallyworld's commit fixed something and that the fact that restore was working before was actually a bug [01:40] we are missing an update somewhee, eiter peergrouper or instancepoller [01:40] anyway, its late and I am very very tired [01:40] perrito666: ok. is there info on the bug, i have not much idea what has been done etc [01:41] eg how agents.conf is involved [01:42] wallyworld: there is not much info, basically, we are doing some stuff in a bash script embedded in restore.go [01:42] which is being undone [01:42] ok, np [01:42] that is the issue (in few words [01:42] and running the exact same steps by hand works? [01:42] yup [01:42] wtf [01:43] which line does it fail at? [01:43] wallyworld: none, that is what makes this wtf of an issue impossible to find [01:43] the restore ends succesfully [01:44] hmmm, ok. i've not even looked at the restore script before so not sure how far i'll get [01:44] and then when jujud-* starts, it all goes back as it was, courtesy of some worker [01:44] i've got some other issues to deal with as well [01:44] wallyworld: in any case wwitzel3 will be working on that too [01:44] ok [01:44] wallyworld, something interesting is happening here. AWS doesn't see any sign of the two services being deployed [01:45] wallyworld: this is the script https://github.com/juju/juju/blob/master/cmd/plugins/juju-restore/restore.go#L106 just in case you get inspired [01:45] which services, the test ones? [01:45] now gentlemen, I really need sleep [01:45] cheers [01:45] perrito666: ok,good night [01:46] did I mention that joyent manta went down for 5 hours yesterday? I had a power cut so it was difficult to report the outage [01:46] perrito666: got it, have a good night [01:46] ah... [01:47] wallyworld, let's say the ec2 issue was region. this test suddenlty came to life and may pass http://juju-ci.vapour.ws:8080/job/aws-deploy-trusty-amd64/644/console [01:48] sinzui: i just bootstrapped with mem=2G, got same region/zone as before [01:48] 2014-06-20 01:43:22 INFO juju.provider.ec2 ec2.go:627 "us-east-1a" is constrained, trying another availability zone [01:48] 2014-06-20 01:43:24 INFO juju.provider.ec2 ec2.go:627 "us-east-1b" is constrained, trying another availability zone [01:48] 2014-06-20 01:43:25 INFO juju.provider.ec2 ec2.go:643 started instance "i-4a528a61" in "us-east-1c" [01:48] - i-4a528a61 [01:48] and it worked [01:48] the test officially passed [01:49] And I was watching the moment it got unstalled [01:49] I didn't intervene [01:49] so did the failing and passing tests use different regions / zones? [01:50] wallyworld, no, the test that passed and the one I kept alive were the same region [01:50] but I think your supposition of region or maybe bad machine were at fault [01:50] maybe it is just mongo flakiness [01:51] wallyworld, the failed bootstraps, the two timeouts, an the pass were all us-east-1c [01:52] \o/ [01:52] wallyworld, I don't believe mongo flakiness because there is no prehistory of it in the test [01:52] just seems strangethat mongo starts but then has connection errors [01:53] wallyworld, oh...I lie [01:53] wallyworld, there are two previous occurrences of this error in the last week. They happened twice in a row, It resolved itself in less than 30 mintutes http://juju-ci.vapour.ws:8080/job/aws-deploy-trusty-amd64/614/console [01:54] hmmmm [01:54] this case though is 4 hours long [01:54] anyway. Lets close this bug. maybe I need to walk away from CI and let it mind itself [01:55] sinzui: would have been interesting to use placement to run the tests on a different region/az when the test fails in us-east-1 [01:56] hmm, well I think I can add such as test after the release, or before if we loose faith that we make juju and CI happy in the next 16 hours [01:56] ok [02:08] wallyworld, new info about joyent: https://bugs.launchpad.net/juju-core/+bug/1329123 [02:08] <_mup_> Bug #1329123: status outage during upgrades in joyent, Juju is broken

[02:09] sinzui: looking into that one now [02:10] I am looking for something like termination protection to keep this instance alive [02:10] ok [02:17] wallyworld, I have an instance. I clobber something on it to keep it alive when the test ended [02:18] sinzui: ok, i'm bootstrapping a 1.18.4 env now and will see what the upgrade does [02:32] far out joyent is slooooow [02:39] wallyworld, it is faster than azure [02:39] surely you jest [02:39] as for Hp, it now varies between 9 minutes and 21 minutes [02:40] yay cloud [02:40] azure is consistently 18 to 21 minutes [02:42] wallyworld, canonistack tests have been passing since I switch the test to pull tools from s3. I suspect swift is the principle cause of speed and out right failures in canonistack [02:42] sigh [02:50] sinzui: so it seems on joyent the api server does not start again after an upgrade [02:50] :( [02:52] i think mongo is ok, looking into it [03:59] sinzui: so it's caused by joyent offering up machine addresses that juju doesn't know how to pick out a mongo replicaset peer from. juju's address processing changed in 1.19 and is not compatible with joyent [04:00] :( [04:00] i'll see if i can figure out a fix [04:00] i'm somewhat new to this part of juju [04:01] wallyworld, I pleased that we know why. I feel relieved that we have really triaged the issue [04:01] yeah, me too [04:02] sorry i had not enought ime earlier to look [04:03] sinzui: sort of argues against that upgrades are generic and can just be tested on one cloud, although i'm confused why a deployment worked as it should have had the same problem [04:04] wallyworld, yeah. abentley argued a different perspective that we certify upgrade works. We need to do it for each cloud to gather evidence that things work and that upgrade isn't different. [04:05] wallyworld, well maybe the deploy case works because juju is exploring the possible addresses. upgrade doesn' gather addresses. [04:05] sinzui: there's not enough debugging to see the root cause, i'll add some and retest, but first i have to go and pick up my son as he finished school early today [04:06] wallyworld, All fine with me. I am going to sleep. [04:06] ok, i'll send an update [05:09] wallyworld: I don't really understand this rsyslog stuff. I can manage to get it to work by changing the default ruleset, but I don't understand why that works... probably going to have to check with wwitzel3 and/or voidspace [05:10] axw: fair enough, i don't fully grok it either [05:10] maybe add a bug comment and we can ping them later [05:21] where should I be reporting bugs? [05:21] marcoceppi: launchpad [05:22] even for things like gomaasapi ? [05:22] that's not even on github yet is it? [05:22] I think launchpad too [05:22] * marcoceppi has no idea [05:22] cool [05:23] maybe not when it's on github. matters for juju for integration points, probably doesn't matter for libraries [05:30] wallyworld: is there anything else pressing I can look at? or just addresses/azure? [05:31] axw: i've made progress with the joyent thing so yeah, just addresses/azure. let's try and wrap up the wip addresses stuff [05:31] ok. gonna write some tests now and propose part of it [05:41] great [06:13] clear [06:13] derp [06:41] wallyworld: axw https://github.com/juju/juju/pull/135 [06:41] looking [06:41] bah, beat me to it [06:41] wallyworld: the fix ? [06:42] no, looking :-) [06:46] axw: davecheney: might explain some of our mongo test issues [06:47] * wallyworld handwaves [06:48] maybe, not sure. it could cause sockets to not be released [06:48] davecheney: reviewed [06:50] bueno [06:50] axw: nice review [06:50] i don't know what the cost of leaking cursors is [06:50] or the rate we were leaking them [06:50] but given the usual nonsense we see with mongo [06:50] i'm confident to bet it wasn't helping [06:51] morning all [06:51] morning voidspace [06:51] o/ [06:53] voidspace: when you have a moment, can you please cast your eye over https://bugs.launchpad.net/juju-core/+bug/1332358 and my comment on it? [06:53] <_mup_> Bug #1332358: creating a local environment stops the syslog (1.19.3)

[06:54] axw: small one https://github.com/juju/juju/pull/136 [06:54] looking [06:55] fixes last critical for 1.19.4 release [06:55] axw: ok [06:56] axw: ouch [06:56] voidspace: does that ruleset stuff mean something to you? [06:56] axw: yep [06:56] axw: I wrote the rulesets [06:57] axw: I can switch to looking at this [06:57] axw: but today is my last day for a week [06:57] axw: so maybe the best thing I can do is write up a quick explanation of the rulesets to help wayne or someone else debug it [06:57] voidspace: sounds good, thank you [06:57] axw: wayne did the rest of the rsyslog stuff (but not the ruleset) [06:57] ok [06:58] voidspace: i've moved that bug to 1.19.4 so if there's any way to get it sorted before eod that would be great [06:58] either you or wayne [06:58] as it would be bad to ship 1.20 with that issue still there [06:58] axw: wallyworld i THINK i've got all the occurances [06:58] wallyworld: what was setting blank addresses? [06:58] i've searched for them a bunch of different ways [06:58] axw: is the problem that with local provider, we kill the rsyslog configuration for anything else on the system? [06:58] i wanted to use the new pointer analysis feature in godoc 1.3 [06:58] davecheney: oracle? :) [06:58] heh [06:58] axw: an old version of the instance poller [06:59] but there isn't enough memory in the world to process juju [06:59] wallyworld: ok [06:59] i can't recall the exact bug [06:59] axw: Having a bug description with full sentences would help :-/ [06:59] the stdlib consumes 1.7gb on my machine [06:59] but it rings a bell [06:59] axw: i could try the oracle [06:59] voidspace: yes, everything else stops it seems [07:00] davecheney: I have 16GB, I'll see if I can run it. [07:00] axw: and you can reproduce it? [07:00] voidspace: yes, trivially [07:00] axw: what are the repro steps [07:00] voidspace: bootstrap local provider, wait for rsyslog worker to kick in, /var/log/syslog stops showing other bits [07:00] ah [07:00] voidspace: I just did what's in the bug [07:01] hah, there are repro steps [07:01] axw: it's 8am here and I haven't had coffee :-) [07:01] so coffee [07:01] :) [07:01] axw: so I guess the problem is that I change the default ruleset to forward all logs to the state server [07:01] and the state server only logs juju messages and drops everything else [07:02] and on local provider the state server is "the machine" [07:02] so all other logs are dropped [07:02] it looked like they both do that though? [07:02] both rulesets I mean [07:02] axw: i also have a bunch of junk in my $GOPATH [07:02] which upset the analyser, it freaks on first error [07:02] axw: one should just forward messages (local) the other should actually log them (remote) [07:03] but coffee first [07:03] ok [07:04] (so "remote" ruleset is for logs received by TCP - and they should be logged. The "default ruleset" is "what to do with messages generated locally" and we want to forward those to all state servers - so that's the "local ruleset" which forwards messages.) [07:04] For local provider we need to be smarter and log messages that aren't juju ones [07:04] in fact we can do that for all providers [07:04] so it should be a simple change [07:05] thanks davecheney [07:05] but I/we need to know how to specify "do what you would have done anyway" for the non juju messages [07:05] the remote ruleset just logs messages - which is why switching to remote by default works [07:06] but it means that log messages don't get broadcast to other state servers [07:06] so it isn't correct [07:06] voidspace: isn't that just for non-juju log messages? [07:06] we only want our log file to handle juju messages [07:06] wallyworld: why the "" check in bestAddressIndex too? [07:07] axw: the problem is that it is for non-juju log messages too [07:07] axw: but yes - we have a tag format for juju messages, so for all *non juju* ones we need to say "just process them as you would have done" [07:07] I don't know how to specify that [07:08] axw: because somehow "" addresses still end up in there and i'm not sure how [07:08] nor do I [07:08] needs digging into [07:08] probably keeping a reference to the normal "default ruleset" before replacing it [07:08] voidspace: axw: there's a ~ syntax i think [07:08] or something [07:08] yeah ~ means "stop processing" [07:08] or something like that [07:08] we only want to do that for juju ones [07:09] but we're also replacing the default ruleset, so the old processing rules might not even be used [07:09] there must be a way to do it, just needs looking up and trying... [07:09] i'm sure between you and wayne you'll figure it out :-) [07:09] heh, thanks [07:10] wallyworld: I'll get wwitzel3 up to speed when he comes online if I haven't already fixed it [07:10] voidspace: awesome, thanks. [07:11] voidspace: i'll mark it as in progress to you so if curtis looks he'll know it's being worked on, and you can assign to wayne to finish off if need be [07:12] voidspace: what's you lp id? [07:14] never mind [07:14] found it [07:19] ? github.com/juju/juju/rpc/rpcreflect [no test files] [07:19] ok github.com/juju/juju/state 370.446s [07:19] ok github.com/juju/juju/state/api 24.052s [07:19] what the f has happened to the state package [07:19] it was 156 seconds a few days ago [07:19] hm, I just ran it and it took 115.711s [07:19] ok, what the f is wrong with the bot ? [07:22] wallyworld: the real problem is in the joyent provider [07:22] instance.go, Addresses method [07:22] it's initialising a slice with len, and then appending to it [07:22] let me look [07:23] so it is [07:23] i'll fix [07:23] thanks [07:23] np [07:23] we did also have that instance poller issue so i jumped to conclusions [07:32] * wallyworld taps fingers impatiently waiting for joyent env to start up [07:32] I thought it was meant to be fast [07:33] not when the instance has to do the apt dance [07:36] how come you guys rewrite the apt mirrors ? [07:36] are the ec2 mirrors busted ? [07:37] davecheney: dunno, that is inherited from CI I think. [07:38] it's probably not that important [07:38] 2mb downloaded [07:39] the real time is taken writing the data [07:39] is that using ephemeral storage ? [07:39] that instance [07:40] axw: bollocks, just the provider fix without the other work still fails [07:40] so i'll have to add back the other check [07:41] wallyworld: :( [07:41] davecheney: non capisco. is what using ephemeral storage? [07:42] the ec2 instance [07:42] it has some, it just shows up as /mnt [07:43] aww crap [07:43] ok, ignore my suggestion, won't work [07:43] davecheney: we use /mnt/tmp as $TMPDIR when running the tests [07:56] mgz: ping [07:57] axw: i added your fix and retained one of mine (and removed the other) and it all works [07:58] wallyworld: cool. which one of yours did you have to keep? [07:58] the merge addresses one [07:58] okey dokey. I guess that needs to be there cos it was buggered before [07:59] that's my guess but i don't know 100% [07:59] because it looks like we're not moving to kr's godep tool that soon, i've just pushed a couple of updates to godeps which should make it a little more pleasant to use [07:59] hard to find out for little gain [07:59] rogpeppe1: what changes? [07:59] wallyworld: no worries, LGTM [07:59] ta [08:00] specifically, there's now a -f flag which, if set and an update fails, tries to fetch new changes from the repo before trying again [08:00] i did git pull upstream master and the conflicts were horrid [08:00] how do I undo that ? [08:00] i have no committed [08:00] rogpeppe1: nice, thank you [08:00] i have not committed [08:00] axw: also (actually this changes has been around for a while) it tries to do as much work as possible in the face of failure [08:00] axw: so if one repo is dirty, it doesn't stop others being updated [08:00] davecheney: "git reset HEAD" I think? [08:01] rogpeppe1: ah excellent [08:01] axw: nope [08:01] sorry [08:01] davecheney: "git checkout ." in the root. I dunno if there's a better way to do it. [08:01] axw: i'd appreciate it if you could try it out and tell me if it seems to work ok. i didn't have the energy to write tests :-) [08:01] axw: thanks [08:01] rogpeppe1: will do [08:01] that worked [08:02] git totally fucked the change to state/apiserver/root.go [08:02] davecheney: you might find it useful too FWIW [08:04] rogpeppe1: will do [08:04] much love for godeps [08:05] it's funny, i've been thinking for ages i couldn't make the time to do the auto-fetch stuff, and then it only took 30 mins to do :-) [08:06] #winning [08:09] wtf [08:09] Updating juju-core dependencies to the required versions. [08:09] launchpad.net/godeps (download) [08:09] # cd .; bzr branch https://launchpad.net/godeps /mnt/jenkinshome/jobs/github-merge-juju/workspace/tmp.PFiIQenwfH/RELEASE/src/launchpad.net/godeps [08:09] bzr: ERROR: exceptions.RuntimeError: if we move self._source_infos, then we need to change all of the index pointers as well. [08:09] Traceback (most recent call last): [08:09] File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 920, in exception_to_return_code [08:09] jenkns is busted [08:09] i'll hit the retry button [08:10] rogpeppe1: ^^ did you break it? :) [08:10] we grab the latest godeps I think... [08:11] axw: it's quite possible, although i tried to make it so the current behaviour wouldn't change unless explicitly specified [08:11] axw: specifically, it doesn't try to fetch stuff unless the -f flag is given [08:11] okey dokey [08:12] hmm actually it looks like it's trying to branch godeps, not run it [08:12] axw: i'm off to soccer, if my 2nd retry fails, and there's a fix to anything, can you type $$merge$$ for me again? [08:12] wallyworld: sure [08:12] ta [08:13] i'll be back later anyway, just want to get this sucker landed [08:13] so lp just had a lie down ? [08:14] maybe. I just branched it and it worked fine [08:14] on my machine [08:14] well, jenkins is still thinking about godeps... [08:17] hmm. *still* thinking. that's not a great sign. [08:18] same problem again [08:18] looks like it might be a bzr issue [08:18] :( [08:19] I'll jump on the machine and take a peek [08:19] axw: try making sure that the godeps repo dir is deleted [08:19] axw: that might make a difference, i suppose [08:19] it goes to a tempdir [08:19] axw: hmm [08:20] axw: it's perhaps indicative of something that it took so long to fail [08:20] i'll try fetching it from scratch myself [08:21] worked fine for me [08:22] rogpeppe1: works when I do it on the machine too [08:22] le sigh [08:22] axw: hrmph [08:24] rogpeppe1: did you do "go get" or "bzr branch"? [08:24] axw: go get [08:25] axw: which, i hope, is what the bot is doing too [08:25] yeah that works too [08:25] axw: in fact, i know it is [08:25] yep [08:25] it is [08:25] axw: oh yes, the other new feature in godeps is that it now understands package wildcards just like the go tool [08:26] rogpeppe1: ah nice, now I don't have to remember how to list the packages :) [08:26] axw: cudos to github.com/kisielk/gotool - which made it a 2-line change [08:28] axw: i could uncommit and push -f if we need to [08:33] rogpeppe1: just going to see if there are any env vars that are influencing this first [08:33] axw: good thought [08:44] fwereade: if you haven't seen it, i think you'll like this proposal: https://docs.google.com/document/d/1e8kOo3r51b2BWtTs_1uADIA5djfXhPT36s6eHVRIvaU/edit [08:45] rogpeppe1, is that the bundles one? haven't looked at it properly yet I'm afraid [08:45] fwereade: no, it's a go proposal [08:45] fwereade: but i'd appreciate your thoughts on the bundles thing too, if poss [08:46] fwereade: 'cos we're actually implementing it right now :-) [08:46] rogpeppe1, I read the first sentence and thought "woohoo!" [08:46] fwereade: :-) [08:47] rogpeppe1, that's awesome [08:47] fwereade: i think so too [08:48] fwereade: it means we can have a clear division in juju-core between internal and external APIs [08:48] fwereade: i think we could restructure it like that even before the go tool supports it actually [08:48] rogpeppe1, yeah, and we can make small packages without annoying you *too* much ;) [08:49] fwereade: :- [08:49] ) [08:52] axw: any joy? [08:52] rogpeppe1: nope :( sticking a "bzr branch" in there to see if it makes a difference [09:01]