davecheneyrick_h__: ping00:18
davecheneydid you get my email about winblows ?00:18
rick_h__davecheney: yes, sorry. Meant to ping alexis on it and see if she moved forward before I sent it over00:19
rick_h__davecheney: let me check my email for getting at the msdn stuff.00:19
davecheneyrick_h__: thanks00:19
davecheneyi've got as far as I can without being able to test locally00:19
sinzuidavecheney, I ca start a win instance from snapshot used to start the win-build-server00:23
rick_h__davecheney: see pm00:25
sinzuiwallyworld, rerunning an older revision is easy, but it take 4 hours now that we test so much and don't have the slaves doing their own builds00:44
wallyworldsinzui: ok, np00:45
sinzuiwallyworld, But I don't see any reason to test master until 1.20 is good So I will start the test00:45
sinzuiwallyworld, The build is started. We are about 2.5 hours from when the function*restore tests will run00:48
wallyworldsinzui: i'm manually testing restore, not done it before, ran into problem: i took a backup, then destoryed my env, then restored. i fails saying it can't re-bootstrap without a control bucket :-(00:48
wallyworldso it seems it wants stuff from the jenv file which no longer exists00:48
sinzuiwallyworld, I have not seen that before00:49
wallyworld[master]ian@wallyworld:~/juju/go/src/github.com/juju/juju$ juju-restore juju-backup-20140702-1042.tgz00:49
wallyworldextracted credentials from backup file00:49
wallyworldre-bootstrapping environment00:49
wallyworlderror: cannot re-bootstrap environment: control-bucket: expected string, got nothing00:49
sinzuiwallyworld, what kind of env are you testing with?00:49
wallyworldmaybe i needed to keep the jenv file00:50
sinzuiwallyworld, CI doesn't set control-bucket for any env :(00:50
wallyworldsinzui: i think doing destroy-environment after the backup was a mistake00:51
sinzui...and unittests and win-build-installer are now playing00:51
wallyworldbut it seems unfortunate we can't fully restore from that situation00:51
sinzuiwallyworld, well, yes...00:51
sinzuiwallyworld, use the console to kill the instance00:52
wallyworldsinzui: cause you know, restore is meant to recover from stuff going away00:52
sinzuiwallyworld, that will simulate a state-sever failure00:52
wallyworldi should be able to hand over the backup tar gz to someone and they should be able to restore00:52
sinzuiwallyworld, and you need on service up to verify that the service learns about the new state-server00:52
sinzuione service i mean no on server00:53
wallyworldhmmm. true for running a test. but in general, restore should be able to habdle a full restore from nothing00:53
sinzuiwallyworld, I don't think so. the control-bucket is only removed by destroy-env...that is human fault00:54
wallyworldsure, but here it failed because there was no jenv to read the control buvket from00:55
wallyworldrequiring some artifacts to still be up for a restore to be successful implies restore is broken00:56
sinzuiwallyworld, your test isn't valid. the case is for when hardware fails, or the like when canonistack loses a machine00:56
sinzuithat is the scenario fwereade outlined to me00:56
wallyworldthere's no reason why restore can recognise there's no control bucket and recreate it and redeploy the charms listed in state00:56
wallyworldimagine if hard drive or data backup worked that same way00:57
sinzuiwallyworld, possibly, but to test what was written and given to customers, we just simulate a bootstrap node failure by terminating the instance using the provider's tools00:57
wallyworld"sorry, restore can't copy your stuff back from tape because you no longer have your original login details to your pc that was destroyed"00:58
wallyworldunderstood, i was just using restore how i thought it should work00:58
wallyworldand got disappointed00:58
wallyworldit's not what i would accept as a customer00:58
wallyworldaxw: hiya, i landed your 1.20 keep alive branch. landings still quite problematic01:08
axwwallyworld: thanks. yeah :(01:08
wallyworldbut a bunch did go through last night01:08
axwit seemed marginally better after some of hte fixes landed, but could just be my bias01:08
wallyworldwe still have the could of root cause issues i think01:08
axwwallyworld: is it okay if I modify the unit test job to create a tmpfs and use that for $TMPDIR?01:09
wallyworldi also want to fix the root cause issues as well01:10
wallyworldaxw: could i ask you to take a quick look to ensure we think the bootstrap-timeout option still works. it seems with CI joyent tests, joyent wasn't bootstrapping due to that 1.18 bug but the sessions weren't being closed after 10 minutes01:10
axwwallyworld: ok01:11
axwthere was a problem with the timeout option originally, it never took effect I think01:11
axwgot fixed somewhere along the way. I'll verify in a bit01:12
wallyworldsinzui: i get the same restore "cannot connect to bootstrap instance: EOF" error in my manual test on 1.19.5 :-(01:26
wallyworldthe instance is there and i can ssh to it01:28
wallyworldi'll see what else might be happening01:28
axwwallyworld: 2014-07-02 01:28:44 ERROR juju.provider.common bootstrap.go:119 bootstrap failed: waited for 30s without being able to connect01:28
axwStopping instance...01:28
axw(it works)01:29
wallyworldaxw: ok, thanks. sinzui ^^^^^01:29
wallyworldnot sure why timeout was failing for joyent01:29
axwthat was on ec201:29
axwwallyworld: was it 1.18?01:30
wallyworldmay have been01:30
wallyworldwas there a fix post 1.18?01:30
axwbootstrap-timeout used to be busted01:30
axwnot sure when it was fixed01:30
wallyworldok, we'll ascribe the issue to that then :-)01:30
axwwallyworld: not sure if you're aware of this yet: #127114401:31
_mup_Bug #1271144: br0 not brought up by cloud-init script with MAAS provider <cloud-installer> <landscape> <local-provider> <lxc> <maas> <regression> <juju-core:Fix Released by axwalk> <juju-core (Ubuntu):Fix Released> <juju-core (Ubuntu Trusty):Confirmed> <https://launchpad.net/bugs/1271144>01:31
axwwe fixed that a while ago, but there's a problem with maas assuming that the interface is eth001:32
wallyworldhadn't seen that01:32
axwwell, sorry, the maas provider01:32
wallyworldso maas is broken for 1.20?01:32
axwit's the same now as it is in trusty01:32
axwI don't *think* that particular bug is new01:33
axwwallyworld: I don't think we need to fix it for 1.20 necessarily, just letting you know it exists because it's had some activity01:33
wallyworldthanks, we'll target for 1.20.101:34
wallyworldsinzui: hacking the restore code to just pick up from the bit where it attempts to connect to the newly started replacement bootstrap instance worked the second time through. so it looks like a mongo startup issue related to timing. i can put the connect bit of code in a retry loop01:39
sinzui:) you are a hero01:40
wallyworldi'll target the bug to 1.20 also01:40
sinzuiwallyworld, Do I dare take master our of testing to make 1.20 the only priority01:41
wallyworldsinzui: i would :-)01:41
wallyworldjust for today01:41
* sinzui does01:41
axwdavecheney: are you updating juju to the new loggo? if not, I can01:53
* thumper needs to think things through, so going to walk the dog01:58
davecheneyaxw: https://github.com/juju/juju/pull/21201:59
davecheneysorry, took a whlie to run all the tests01:59
davecheneyaxw: say the words, SAY THEM!02:00
axwluks gud02:00
axwdavecheney: reviewed02:01
davecheneyermehgerd, wreeevu02:01
wallyworldsinzui: there are a few possibilities where the restore is falling over, i have to track down the place and then i can fix. not as obvious where to put the retry as i first hoped.02:05
wwitzel3when writing tests for apis .. if the state/api tests execute the code paths in state/apiserver .. do I need to copy and paste those tests minus the caller? what is the stance on that?02:06
wwitzel3I have unit tests for the methods internal to state/apiserver , but the method we expose to state/api (GetCapabilities) is being fully executed by the tests in state/api.02:07
wallyworldsinzui: well, i know the top level api call, but one part of the implementation does retry, and there's a section that doesn't. i'll have a fix soon i hope02:09
wallyworldwwitzel3: our api tests are not really complete - i believe the client tests involve calling all the way through to a functioning server02:09
wallyworldthe apiserver tests are a subset02:10
wallyworldi just follow what others have done when i add tests there02:10
wallyworldwwitzel3: i think from memory the apiserver tests have full coverage, and the client tests just check that it's all wired up02:11
wallyworldso they just run a few representative calls02:11
wwitzel3wallyworld: ok, well then I will move my state/api assertions in to state/apiserver and then have the state/api test just ensure it is all wired up correctly.02:15
wallyworldwwitzel3: yeah, i think so. see what say keymanager does for example02:15
wallyworldit's sorta crap, but it was how it was all done originally02:16
wallyworldpeople just didn't stub out properly02:16
wallyworldthe the pattern has been cargo culted over and over :-(02:16
wwitzel3I've been cargo culting for the last 3 months .. sorry02:21
wwitzel3I'm new, I don't know what to do!02:21
wwitzel3just got done looking at the keymanager example02:22
wwitzel3I will adjust accordingly02:22
wwitzel3thanks :)02:22
waiganihttps://labix.org - security certificate has expired02:25
wallyworldwwitzel3: cargo culting reference wasn't meant for you - it's a reflection of the poor state of the original tests02:39
wallyworldi've done the same thing in that section of code02:39
axwwallyworld: wow, that tmpfs change I made had quite an impact on the time taken to run the state package tests02:40
axw325s down to 120s02:40
axwthose disks are slow as02:40
wallyworldmust be :-(02:40
axwit's meant to have SSDs02:40
wallyworldi use a ramfs for my tmpfs cause i got 16G02:40
wallyworldi don't want to wear out my ssd02:41
* axw nods02:41
axwwoot, ran all the tests in 19 mins and passed first go02:41
wallyworldaxw: if they pass 10 times in a row..... that will be a nice surprise02:42
wallyworldregardless, there's still race conditions to fix :-(02:42
axwwallyworld: that's shaved 8 minutes off the last successful first-time run02:42
axwI am looking at all the races now02:42
wallyworldand when we get nailed up instance, goodbye to another 4-8 minutes02:43
lifelesseven SSDs are slow if they have to actually do IO02:43
axwsure, but the difference on my laptop's SSD and tmpfs is about 30s in this test, not 2 minutes02:44
davecheney^ another state related fix03:02
davecheneyactually related to the footgun that is suite.TearDownTest03:02
davecheneyoh fuck03:02
davecheneythe branch is mixed in with my loggo change03:02
davecheneythe change is in  state/initialize_test.go03:03
davecheneyif you want to review03:03
davecheneyi'll repropose it after the bot has landed my previous branch03:03
axwdavecheney: this is where rebase is handy. "git rebase -i master" and delete the loggo commit line03:06
davecheneyaxw: hmm03:14
davecheneyi tried03:14
davecheneybut failed03:14
davecheneythe commits came back :*03:14
axwmmkay. I never had any problems with it so I don't know what to suggest03:16
davecheneyaxw: i'm batting 0.00 with git rebase03:17
thumperoh man...03:20
thumperhead is spinning with auth shit03:20
* axw calls victory over the lander03:21
axw3 in a row passed first time, 19m 17m and 17m03:22
davecheneyaxw: i remain unconvinced03:23
davecheneyi suspect there is more pain in there03:23
wallyworldgreat :-) those were the times we saw when this was all first set up - it has degraded over time since :-(03:23
thumperspeaking of shit,  antipathy.nonserviam.net/utilities/shit  <== beware, perl03:23
davecheneybasically any time mongo shits itself coming up03:23
davecheneythe test explodes in a blame misdirecting way03:23
thumpershow image in terminal03:23
thumpersaw it last nigh03:23
axwdavecheney: there are still problems, but I reckon it's due to slowness03:23
thumperpretty interesting03:24
thumpersomeone should write it in go03:24
thumperit could be called "go shit"03:24
davecheneygo is03:24
* thumper chuckles03:24
thumpersfw - examples imgur.com/a/kmOhv#003:25
davecheneythumper: ha, fool me trice03:25
wallyworldsinzui: so i have a working solution. i tried to push the retry logic down to the root cause failure (login), but the underlying connection got closed underneath so i had to move it up higher. will propose and land in 1.20. took a while to get it right due to the iteration time03:25
wallyworldonly took one retry loop before it connected03:26
davecheney^ this is what a sick commit bot looks like03:30
wallyworldsinzui: ah bollocks, it failed in a different place right near the end of the restore. ffs03:31
wallyworldmight be spurious, will retry03:32
* thumper out to collect daughter, back shortly03:32
axwwallyworld: woohoo, now the github merge failures are back03:33
axwcan't win03:34
axwbbs picking up my daughter03:34
davecheneybloody helll04:38
davecheneyhow many panics are there in there !?!?04:38
jcw4davecheney: local tests? The last build failure seemed to be a merge issue...04:45
davecheneyjcw4: this problem goes deeper than the last test faliure04:46
davecheneyif mongo shits itself coming up04:46
davecheneythe test helper will panic in one of a dozen different ways04:46
davecheneythe panic is captured by the testing suite04:46
davecheneywhich then goes to tear down the test04:46
davecheneythat usually generates a far more fatal panic04:47
jcw4davecheney: and mongo failing on startup is random?04:47
davecheneyjcw4: random as clockwork04:47
jcw4but... but... mongodb is network scale04:48
davecheneyit's significantly less stable since we combined replica sets and tls into an unhoyly offspring04:48
jcw4or something like that04:48
davecheneywallyworld: axw http://paste.ubuntu.com/7734835/04:49
davecheneyopenstack tests fail reguarly for me04:49
davecheneycomplain they can't find tools and crap themselves04:49
jcw4davecheney: what are the address type warnings about ipv6?04:50
jcw4davecheney: seems possibly related to not finding resources on
davecheneyour tests leak shitloads of pingers04:51
davecheneyjcw4: no idea04:51
thumperdavecheney: got a minute to chat?05:01
thumperdavecheney: particularly about tests and pingers05:01
davecheneythumper: sure, send me a hangout link05:02
davecheneyi'll go upstairs05:02
thumperdavecheney: https://plus.google.com/hangouts/_/canonical.com/testing05:03
wallyworlddavecheney: what changes were made before those tests were run?05:11
davecheneywallyworld: not sure I follow05:26
davecheneyi think those leaky pingers have been ther for a while05:26
davecheneyi think thye are leaking out of the mongodb driver05:26
wallyworlddavecheney: i mean the tools errors05:26
jcw4PTAL: https://github.com/juju/juju/pull/21605:26
davecheneysorry, i'll try to capture more errors05:26
davecheneyit doens't happen consistently05:26
davecheneyand when I run the tests in a loop i usually find another failure05:26
davecheneywhich distracts me05:26
davecheneyfor example, http://paste.ubuntu.com/7734852/05:27
jcw4jam: since you'll be on before me tomorrow: https://github.com/juju/juju/pull/216 :)05:32
jamjcw4:  ?05:32
jcw4on call reviewer :)05:32
jcw4jam: pretty straightforward05:33
jcw4jam: what timezone are you?  I thought you were closer to GMT?05:33
jamjcw4: UTC + 405:34
jamI'm in Dubai05:34
jcw4jam: Ah! that's the second time I've forgotten that05:34
jamjcw4: well it is 20+ people to keep track of :)05:35
jcw4jam: I told fwereade a couple weeks ago I chatted with you late at night 'cause I thought you were down under05:35
jcw4jam: and I was embarrased to learn you were in dubai; now I know it's just that you're up bright and early05:35
jamjcw4: in your defense, I don't keep to a very strict 8-hours-a-day work schedule. so I *might* show up at random times05:36
jammy dog usually wakes me up for a walk at about 5:30-6am05:36
jcw4jam: :) that's too early.  well I'm UTC-7 so I'm off to bed myself05:36
jamand my family was away for a week recently, which led to a bit more work times.05:36
jamjcw4: rest well, it is quite late for you05:36
jcw4yep, my self imposed bedtime is 30 minutes ago05:37
davecheneywallyworld: http://paste.ubuntu.com/7735005/05:44
wallyworlddavecheney: ok, that appears to be a test failure due to c.Check(info2, gc.DeepEquals, info) - what happens on failure is all the debug log stuff gets shown. simplestreams debug logs what happens as it looks on the search path for metadata, and sometimes there's nothing there so it logs it for diagnostic purposes05:46
wallyworldso it appears to not be an issue so much05:46
davecheneyand it's upset because the test bound to
wallyworldi'm not sure why that happened without looking into it05:49
wallyworldaxw: bot is a lot happier now that /tmp is not on those slow disks06:02
axwyeah :)06:02
axwso much blue06:03
=== vladk|offline is now known as vladk
wallyworldbuys us time to nail the remaining intermittent errors06:07
wallyworldcause running on a slow disk should not break the tests :-)06:07
davecheneycloud is always gonna be slow06:08
davecheneythat's what it does06:08
axwindeed, really do need to get to the bottom of it06:13
=== vladk is now known as vladk|offline
=== vladk|offline is now known as vladk
jamfor the well motivated: https://github.com/juju/juju/pull/218 and https://github.com/jameinel/juju/pull/407:03
jamdimitern: fwereade: ^^07:03
wallyworldaxw: did the race detector find those races you fixed?07:27
axwwallyworld: yes07:27
wallyworldjeez, there's a lot of them07:27
axwindeed :)07:28
wallyworldwe need to set up a CI job07:28
wallyworldcause we won't pick them all up at review time07:28
axwwallyworld: all I did was "go test -race ./..."07:28
wallyworldtakes along time though right?07:29
axwnot particularly long07:29
axwI didn't time it07:29
wallyworldi still think a job will be helpful07:29
jammorning dimitern07:35
dimiternmorning jam07:36
dimiternjam, i'm looking at your PRs07:36
jamdimitern: and thanks for fixing the edit of the networking spec07:36
dimiternjam, np, i should've done that initially07:37
jamdimitern: hard to realize when you're own editing of it works just fine :)07:40
jamvladk: thanks for reviewing https://github.com/juju/juju/pull/211, one point of note. When you finish a review, it is good to give a summary comment on the whole thing. That way we know that you are done reviewing, and we know whether you are giving it an overall LGTM or whether it is something that needs to come back for another review.07:48
TheMuejam: vladk: thx for review, good comments07:53
voidspacemorning all08:06
jammorning TheMue and voidspace08:11
* jam needs to go get food08:11
dimiternjam, both reviewed08:45
axwdimitern: does the coming network stuff need bridge-utils? I'm just tidying up our cloud-init, wondering if it can be removed for non-maas08:49
dimiternaxw, we should leave it to the networker to install that and the vlan package as needed, but let's leave it for now until it does?08:51
axwdimitern: ok, will leave it there for now08:52
dimiternaxw, cheers08:52
dimiternaxw, if you're changing cloudinit anyway, perhaps add a TODO about dropping bridge-utils and letting the networker install it as neede\08:53
axwthere's a bug about it, I'll add a comment and ref to that08:54
dimiternaxw, awesome08:56
dimiternaxw, btw is the relation addresses work on hold pending approval?08:59
axwdimitern: yes. I did a more modest change to trigger config-changed on address change, but relation addresses are on hold for now09:00
dimiternaxw, right, ok09:00
axwneeds some more discussion and/or arm twisting09:00
dimitern:) tell me about it09:00
mgzaxw: want to do mini standup?09:01
axwmgz: sure, brt09:01
dimiternwe've been going over the model back and forth for the past 3 weeks, but at least we're now into minor implementation details mostly09:01
voidspacejam: so I've finally managed to find the right place to hardcode the ipv6 mongo address09:32
voidspacejam: and I can get a connection to mongo on [::1]:37017 working fine09:32
voidspacejam: which I think validates that mongo works ok locally with ipv609:32
voidspacejam: and from his reading of the mongo codebase vladk thinks that we should be fine with both ipv4 and ipv6 connections to the same mongo(s) if we need it09:34
dimiternvoidspace, great news!09:50
jamvoidspace: sounds good, indeed.10:01
menn0voidspace: ping?10:05
TheMuejam: you’ve got another part I can support network coding?10:05
voidspacemenn0: pong10:17
jamdimitern: I think I've addressed all of your requests on https://github.com/juju/juju/pull/218 if you want to give it another quick pass.10:30
dimiternjam, cheers, will look in a bit10:30
jamTheMue: I don't think I quite understand? Just wanting more work to do ?10:31
jamTheMue: we'll talk about it in 15 min anyway, so I'm going to just go make coffee and see you guys there.10:32
TheMuejam: ok10:32
vladkvoidspace, dimitern, jam, TheMue: here is my investigations on replicaset with IPv610:41
dimiternvladk, thanks for sharing!10:42
TheMuevladk: interesting notation when setting addr and port. go would dislike it10:43
dimiternjam, standup? also - your PR is ready to land I think10:46
fwereadeaxw, ping10:51
rogpeppemgz: i'm going to update godeps. worth watching the 'bot to check that it doesn't fall over10:53
jam1wallyworld: mgz: http://juju-ci.vapour.ws:8080/job/github-merge-juju/324/console failed because the script couldn't be found: /tmp/hudson6827171313236674227.sh: line 121: /home/ubuntu/jenkins-github-lander/bin/lander-merge-result: No such file or directory10:53
mgzrogpeppe, jam1: that was too fast, right? :)10:56
jam1mgz: too fast?10:56
rogpeppemgz: i think so10:56
mgzto blame rog :)10:56
jam1mgz: it failed a long while ago10:56
jam1mgz: so yes, not rog's fault10:56
rogpeppemgz: i don't think it had even merged by the time that message came here10:56
rogpeppephew :-)10:56
mgzyeah, it's yesterday. I wasn't fiddling with it then, but someone may have been10:58
mgzwould just try rerunning it, stuff has landed since then.10:58
axwfwereade: pong. getting kids to bed and dinner on. I will bbl11:12
fwereadeaxw, no worries, but if you have some time I would appreciate your thoughts on https://github.com/juju/juju/pull/189/files -- you've been in there a lot more recently than I have11:12
fwereadeaxw, I think I'm saying things that are roughly sane but your oversight would be good11:13
axwfwereade: nps, will take a look later11:27
=== vladk is now known as vladk|offline
=== vladk|offline is now known as vladk
wwitzel3so I noticed there was state/policy.go and inside there is an interface EnvironCapabilities. So I should probably extend that to have the InstanceTypes and AvailabilityZones (it already has SupportedArchitectures) instead of defining my own interface (which was also called EnvironCapabilities). If I do that, I get an import cycle with state since I need to in include it in the apiserver code to use this interface, but it is also included 11:55
wwitzel3do I move the interface to its own package under state? do I move the interface out of state and use the one I've defined in provider?11:57
wwitzel3do I just have two different, but very similar interfaces?11:58
wwitzel3I feel like state is the wrong place for this interface anyway. It made sense to me existing in provider/common.12:01
wwitzel3so I'm leaning towards just moving it there, since the interface itself is only used by providers anyway (afaict)12:02
wwitzel3looks like it is used by state in a few places, hrmph12:08
jamvladk: yay, your patch landed this time12:09
vladkjam: hooray12:09
TheMuejam: back from lunch, shall I grab one coding task from our planning lane?12:12
jamTheMue: so, we can have you (a) try to pair with voidspace on what he's doing with ipv6 and mongo12:14
jam(b) have you look at fixing the firewall rules for providers to disallow external DB access12:14
voidspacejam: TheMue: the first part of that is trivial and will be done soon12:14
voidspacejam: TheMue: when I come back from lunch we could pair on the peergrouper stuff12:14
jamvoidspace: yeah, I expect '—ipv6' to be easy enough. IT is more about pairing for replica set and peergrouper12:14
voidspacesounds good12:15
jamvoidspace: though if you put that up for review for TheMue to look at before you go to lunch, he can approve it by the time you get back :)12:15
TheMuejam: voidspace: ok12:15
TheMuesounds good12:15
jamTheMue: if you have some time free now, getting developer docs on using API versioning is probably quite worthwhile while it is fresh in our memories12:16
TheMuejam: yes, I’m currently looking into your changes and the doc to see where we differ from the old API12:17
TheMuejam: just digging in PR 21812:18
TheMuejam: would btw like two APICall() versions, one like now with explicit version and using BestFacadeVersion(), and another one doing that implicitely12:22
TheMuejam: APICallBestFacade()12:22
TheMuejam: but that’s only convenience ;)12:22
jamTheMue: so we have FacadeCaller that provides the second form12:23
jamalong with BestAPIVersion for things that need to check compatibility12:23
jamwhich I would expect most poeple would use12:23
jamsince they *also* just want to use calls on a single facade12:23
TheMuejam: oh, not yet found it, nice12:23
=== vladk is now known as vladk|offline
dimiternjam, how about instead of 1 suite, with methods to start ipv4/ipv6/both servers, instead to have 3 separate suites, that do each one separately at SetUpTest ?12:23
voidspacejam: so in testing/mgo.go we hardcode the address to localhost:port (MgoInstance.addr)12:24
voidspacejam: ah, no problem forget that12:24
jamdimitern: its possible, however I would think that you would have both kinds of tests in one file, and it might be more obvious to have them next to eachother.12:24
dimiternjam, because it's really not that easy to reason with MgoSuite and make it *not* start mongod in SetUpTest and assert it was closed in TearDownTest12:24
voidspacejam: I just want to test I can *connect* to it with ipv612:24
jamjam: "localhost" isn't a problem, right?12:24
jam127.0.0.1 would be12:24
voidspacejam: no, it's fine - I was confusing starting the mongo instance and connecting to it12:25
jamdimitern: I would be happy enough to have us split out Mongo testing from API Server testing.12:25
voidspacehaving started it I just want to test connecting to it12:25
jamwhat we probably care the most about at this layer is API Server testing12:25
voidspaceI thought I wanted to change the way it was started12:25
dimiternjam, the problem is we can't start an API server without a connected backing state.State12:26
jamdimitern: I'm just thinking that if I was writing tests, I would probably write: "TestIPv6APIServer()" and then "TestDualStackServer", etc.12:26
jamWhich *could* be separate Suites, but it wouldn't be as obvious (I think)12:26
jamalternatively, we have a test mixin12:26
jam(so a common set of tests that get run with each base suite)12:26
dimiternjam, I'll think about it some more12:27
jamdimitern: so I'm less stuck on Mongo must be connected to the same way as the API server.12:28
jamif it is easier to tease them apart for testing.12:28
dimiternjam, at least i found out a smaller prereq i'll propose shortly - the api server needs to take Addrs []string (or []network.HostPort, but I'd rather not bind these two) and listen on all given EPs, not just one12:28
jamdimitern: I think that's better, though I wonder if we want to just pass ":Port" and have it bind to all ?12:28
dimiternjam, that's better actually and simpler12:30
jamdimitern: well, I'm looking into the Python 'socket' code, and you can't just do that, at least.12:34
jamdimitern: at least in python, when you declare a socket, you have to specify AF_INET or AF_INET612:34
jamand then sock.bind(('': 12345)) vs sock.bind('::', 12345, 0, 0))12:35
jam(bind takes 4 args vs 2)12:35
jamdimitern: http://golang.org/pkg/net/#Listen12:36
jamdimitern: says that it must be "tcp" "tcp4" or "tcp6"12:36
jammaybe "tcp" will give us both?12:36
dimiternjam, right, we need "tcp" and "tcp6" and 2 Listen() calls12:36
dimiternjam, I'll do some experiments12:37
jamdimitern: well if we need 2 Listen calls, then we probably need 2 Listener objects, and if we want to support dual stack, then the inner select needs to select on both of them.12:38
voidspaceI get LOG messages "connection accepted", followed by a "no reachable servers" error12:41
voidspacedoes the "connection accepted" log message mean that we *did connect*, or is it possible that the "no reachable servers" message is genuine12:42
voidspacedimitern: ping ^^^12:45
dimiternvoidspace, I think it means you connected to one of the mongos, but it itself couldn't reach the other replica set members perhaps?12:47
dimiternrogpeppe will know better I guess ^^12:48
jamdimitern: I did some web searching, everything looks like you need 2 sockets to do it right.12:48
voidspacedimitern: this is testing/mgo.go12:48
voidspaceso there's no replicaset12:48
jamvoidspace: the log message "connection accepted" just means TLS connected. I think we still need to Login, etc. after that.12:49
voidspacejam: so mongo accepted the connection but login failed - something like that12:49
voidspaceThis is how I'm constructing the connection: http://pastebin.ubuntu.com/7736622/12:50
voidspaceessentially: info.Addrs = []string{fmt.Sprintf("[::1]:%s", testing.MgoServer.Port())}12:50
voidspacelet me try with to make sure it can possibly work12:50
jamvoidspace: I forget the exact circumstances, but I've seen us go into a tight loop with 1000s of lines of "connection accepted"…. yeah, if it was accepted, why aren't you doing anything… :)12:52
voidspaceheh, no that fails12:52
voidspaceso the test construction is bad, nothing to do with ipv612:52
voidspacehah, it was my python knowledge that was causing it to fail13:00
voidspacedimitern: jam: %s is not the way to interpolate an int into a string with Sprintf ...13:00
voidspacewhen I use %v the test passes13:00
voidspaceand without the ipv6 flag the test fails13:01
voidspaceship it13:01
voidspaceTheMue: https://github.com/juju/testing/pull/1713:03
dimiternjam, it actually works with Listen("tcp", ":54321") - the same listener accepts both 4 and 6 connections13:10
voidspaceTheMue: https://github.com/juju/juju/pull/22113:11
TheMuevoidspace: looking13:12
TheMuevoidspace: just to be sure, when adding —ipv6 mongo only additionally enables ipv6, but still ipv4 in parallel, doesn’t it?13:15
voidspaceTheMue: that's correct - and the rest of the test suite works13:18
TheMuevoidspace: fine13:20
TheMuevoidspace: so the fist LGTM has been easy  ;)13:22
TheMuevoidspace: and the second one too13:23
jamdimitern "tcp" sounds good. I was trying to trace through the code to see if it would work, but didn't get to the root of it.13:30
jamdimitern: I'm off until our call13:30
dimiternjam, ok13:30
voidspaceTheMue: thanks13:30
voidspaceI'm going on lunch13:30
jamvoidspace, TheMue: so we need to do the same thing for github.com/juju/testing13:31
jamsince that sets up our MgoTestSuite13:31
voidspacejam: I had pull requests for both13:32
voidspacejam: and both reviewed13:32
voidspacejam: probably need to bump revision in dependencies.tsv - but we're not yet *depending* on that13:32
* voidspace really goes on lunch13:33
sinzuiabentley, wallyworld jam fwereade I am manually re-running the two failed restore function tests 1 just passed. I think the feature is brittle, but better than 1.18. I hope to start a release in an hour13:56
jamsinzui: would this be 1.19.5 or would we be jumping to 1.20?14:03
jamI personaly would like to go the "dev release becomes stable with only version.go changes" but I realize we're close on time.14:04
sinzuijam I am releasing 1.20.0 from the 1.20 branch14:04
sinzuijam Once I do that, I will change master to be 1.20-alpha114:05
jamsinzui: I realize that, just that we have a fair number of code changes since 1.19.414:05
jamsinzui: I think master becomes 1.21-alpha114:05
sinzuijam, master has not passes in a week, so I am not inclined to attempt a 1.19.5. The plan last week was to only merge the safe and stable changes into 1.2014:07
jamsinzui: you would do 1.19.5 as a pre-release for 1.20 from the 1.20 branch14:08
jamI would certainly not do a 1.19.5 from master14:08
jamsinzui: I think *master* is already not a 1.20 branch, but a 1.21-pre branch14:08
jamsinzui: my point is that at one point in time, we wanted to have a stable release be *exactly the same code* as a dev release, except for the version.go line.14:09
jamand while we've been doing 'stable' patches to 1.19.4, that is still patches from the last dev release.14:09
sinzuiWe created the 1.20 branch our of desperation. I didn't do it until it was clear that was the only way to get a release out14:10
sinzuijam. I think we need a policy change to get to "only the version is different". When we have a regression, we stop the line and everyone  fixes it14:10
jamsinzui: we can just do the release you want to do, but call it 1.19.5, and when we're happy it really is stable we do 1.20 with no other changes.14:11
sinzuijam, master always has multiple regressions, and I think devs are building on top of the broken code14:11
sinzuijam, release 1.19.5 today, the 1.20 tomorrow?14:11
jamsinzui: given the timeframe, I'm not sure if it is actually worthwhile. If we were giving it 1 week then I think there would be genuine benefit for a stable release.14:13
jamsinzui: as for getting into a bi-weekly release cadence for dev releases, we've talked about whether we have to split of each one for stabilization. We have a track record that says we need to.14:14
jamanyway, I really need to go, but hopefully the release happens, it sounds good.14:14
dimiternvoidspace, vladk|offline, TheMue, a minor PR to enable and test the apiserver listens on ipv4+ipv6 - https://github.com/juju/juju/pull/224/14:23
dimiternrogpeppe, you might be interested as well ^^14:24
rogpeppedimitern: thanks14:24
katcoare there any sort of architectural diagrams for juju?14:24
katcoto help identify subsystems, etc.?14:24
ericsnowkatco: try https://github.com/juju/juju/tree/master/doc14:25
katcoty, sir!14:25
ericsnowkatco: more text than diagrams though14:25
katcoit's a start :)14:26
TheMuedimitern: looking14:27
dimiternTheMue, cheers14:28
voidspacedimitern: that looks great14:53
TheMuedimiterm: so far here too, but now 1:1 with alexisb14:54
dimiternvoidspace, yeah, it turned out simpler than I expected14:54
dimiternTheMue, np14:54
voidspacemgz: no merge bot on juju/testing ?15:00
TheMuedimitern: LGTM by my side too15:01
dimiternTheMue, thanks!15:02
ericsnowvoidspace, wwitzel3, perrito666: standup?15:03
perrito666ericsnow: I am in a rather unhappy place right now, I would say that you go on without me, there are no changes since our last standup on my side15:04
ericsnowperrito666: :(  Do you need a break to come visit us in the land of the living?15:05
perrito666ericsnow: I currently am 200km from my usual workplace, I am working at a friends house for the day, I had to travel to get a certificate that entitles my wife to drive my car (yes, you read that correctly)15:05
ericsnowperrito666: good luck15:06
perrito666ericsnow: oh I got the required paper, it took me 5 min :p15:06
perrito666its the 200km part that bugs me15:06
perrito666I need to do this for every person I want to authorize to drive my car15:07
perrito666and now there is also a kid crying next to me :p15:08
wwitzel3ericsnow: not sure why my highlight doesn't work when it is in a list with other names, but just saw this.15:17
wwitzel3ericsnow: I just got out of the tosca meeting 5 minutes ago, you still want to do standup?15:18
ericsnowwwitzel3: I wouldn't mind15:18
wwitzel3ericsnow: meet you there15:19
voidspaceericsnow: not me15:19
wwitzel3also fwereade I sent you that branch to look at15:19
voidspaceericsnow: I don't like you lot these days15:19
wwitzel3ericsnow: in moonstone15:19
katcointeresting, i see a lot of mutexes in the logging framework. isn't the idiomatic way to do synchronization in go message passing?15:20
perrito666voidspace: you do not like us now that you have better internet, you have changed15:20
voidspacehah, I wish my internet was better15:20
voidspaceperrito666: it's still horrible :-/15:20
voidspaceI've complained to the ISP now anyway15:21
perrito666voidspace: its a good thing you dont have to spend a week locked with us :p15:21
perrito666voidspace: they said new, not better15:21
voidspaceTheMue: you up for / available for pairing?15:32
voidspacedimitern: ping15:32
=== urulama is now known as urulama-away
TheMuevoidspace: now again, had been afk for giving my wife a helping hand in the garden ;)15:41
voidspaceTheMue: cool :-)15:44
voidspaceTheMue: there are three tickets we could plausibly work on15:44
voidspaceTheMue: peergrouper IPv6 testing - but I think that one is better waiting for the test suite that dimitern is working on15:45
voidspaceTheMue: Have tests for Mongo in IPv6 replicaSet tests15:45
voidspaceTheMue: Client: juju/api.go Don't drop IPv6 addresses15:46
voidspaceTheMue: I get the feeling that jam would prioritise "Have tests for Mongo in IPv6 replicaSet tests" slightly higher15:46
voidspaceTheMue: but there is no additional description in the card15:46
TheMuevoidspace: ack15:47
TheMuevoidspace: yeah, cards often have only the title ;)15:47
voidspaceTheMue: I guess the most important thing is that members of a replicaset can talk to each other over ipv615:48
mbruzekI am using the local provider.  I deleted the /var/cache/lxc/cloud-trusty/ubuntu*cloudimg*.tar.gz, destroyed and re-bootstrapped.  I don't see a new image in that directory, but juju "deploys".  I thought juju would download a new image.15:48
voidspaceand possibly in "dual stack" scenarios, but I think we're going to (initially at least) require that all HA state servers be on either ipv6 or ipv4 (so no mixed environments)15:48
voidspaceTheMue: on the other hand, the other ticket "Client: juju/api.go Don't drop IPv6 addresses" seems nice and straightforward15:49
voidspaceTheMue: although I think we can't actually use that until we can connect to the api server with ipv615:49
TheMuevoidspace: sounds logical. even if it may be possible it alway can lead to hassle15:50
TheMuevoidspace: exactly, that’s a dependency15:50
TheMuevoidspace: so the order would be replicaSet, no dropping in api and finally peergrouper15:51
voidspaceTheMue: cool15:52
voidspaceTheMue: hangout?15:52
TheMuevoidspace: can do, mom, have to fetch headset15:52
mgzdid you just refer to voidspace as mom? I've clearly missed something TheMue15:53
TheMuemgz: hehe, no15:53
voidspacethat's what I thought15:53
TheMuemgz: mom or mom pls is simply for „one moment please"15:53
voidspacemgz: hey, for juju/testing do I need to manually approve the merge15:54
TheMuemgz: how would you abbreviate it?15:55
mgzvoidspace: sorry, missed that question earlier - yeah, for anything where github gives you the option of manually merging, you need to run the tests yourself and hit that button on the page15:55
mgzI'm planning to start switching stuff over shortly, but atm it's only a few projects that are landing bot controlled15:56
voidspaceok, cool15:56
voidspacemgz: thanks15:56
TheMuevoidspace: so, calling you in hangout ;)16:00
TheMuevoidspace: hmm, doesn’t work. taking the saphire hangout? https://plus.google.com/hangouts/_/canonical.com/juju-sapphire16:02
voidspaceTheMue: I'm there16:02
TheMuevoidspace: lost you16:13
* TheMue lost connection16:54
TheMuevoidspace: I’ve got to step out, will ping you tomorrow morning16:55
voidspaceTheMue: ok, sorry for the delay17:04
=== revagomes__ is now known as revagomes
katcowould anyone have some time to answer some questions about the logging systems?17:22
sinzuialexisb, I just sent an email about the 1.20.0 release. I need to eat. Perhaps you want to talk in about 30 minutes about it17:27
alexisbsinzui, I will be out for a bit for lunch, but can ping  you whne i am back17:27
* alexisb goes an dlooks at email17:27
hazmatdimitern, what's the status on container addressability?17:40
dimiternhazmat, it's on hold since ipv6 gained priority17:42
hazmatdimitern, ic, thanks17:42
mfoordgrrr, new phone arrived today17:43
dimiternhazmat, btw why does the deployer still connect to state directly? (somebody mentioned it today)17:44
hazmatdimitern, it doesn't17:45
hazmatdimitern, you mean juju deployer or unit deployer in juju?17:45
dimiternhazmat, juju deployer17:45
hazmatdimitern, it doesn't connect to mongo... it primarily uses the api or cli.17:46
dimiternhazmat, we'll be closing the mongo port 37017 in 1.20.x soon17:46
hazmatdimitern, there's a separate tool .. juju db inspector that does17:46
dimiternhazmat, ah, ok then, that's fine17:46
hazmatdimitern, primarily the value there (state inspection with db inspector) is being able to look at unit relation data17:47
dimiternhazmat, right17:47
dimiternhazmat, if it's just that, why not use juju run relation-get ...17:47
hazmatdimitern, good question ;-) .. amulet goes through all kinds of tricks to try and do just that (inserting proxy charms and subordinates) for charm testing. prolly cause it predates juju run17:48
hazmatmarcoceppi, ^17:48
dimitern:) i see17:49
marcoceppiOh, it predates the idea of there even being a juju run17:49
marcoceppidimitern: also, juju run relation-get requires you know do several other runs to get relation, relation-id, list of units, etc17:50
marcoceppibut juju-run is a "good enough" alternative for sure17:50
dimiternmarcoceppi, you can run a script that does all that in one juju run call17:51
=== vladk|offline is now known as vladk
marcoceppidimitern: you could, and I guess the unit subordinates could install that script17:51
* marcoceppi wanders off to contemplate this17:51
dimiternmarcoceppi, yeah, or you can just juju scp it :)17:52
=== vladk is now known as vladk|offline
=== vladk|offline is now known as vladk
mbruzekI am getting an error in Juju  panic: runtime error: invalid memory address or nil pointer dereference18:01
hazmatmbruzek, file a bug with the traceback from the machine log18:02
mbruzekThanks hazmat I will do that.  Is there anything else they would need?18:04
hazmatmbruzek, juju status output for some context might help..18:06
katcoi'm having trouble figuring out where juju instantiates log files. can anyone give me a pointer?18:57
sinzuiperrito666, anyone help18:58
perrito666katco: look for DefaultLogDir18:58
katcoperrito666: ty18:59
perrito666sinzui: going18:59
sinzui^ The job fails for 1.20, but we haven't landed related changes to it18:59
sinzuiI have amassed a lot of test results, but not consistent failure to say what is wrong18:59
* perrito666 uses a lot ofadjectives 18:59
perrito666sinzui: ... odd as hell19:02
perrito666says jenkins taht this broke at https://github.com/juju/juju/commit/e7f77fc119:05
sinzuiperrito666, that is master, not 1.2019:08
sinzuiperrito666, commit 9ece6097 could be what broke the test19:09
perrito666sinzui: I just clicked on the link you gave me and then the first failing one19:09
sinzuijenkins doesn't understand branches. It is a moron19:10
sinzuiperrito666, This commit is first failure in 1.20 https://github.com/juju/juju/commit/9ece6097f4b21db8e934885ff9cb1373afa579f419:11
perrito666oh Isee19:11
sinzuiI suspect cloud issues rather than code changed, but it does indicate brittleness.19:11
sinzuiperrito666, I am starting a conversation about releasing 1.20.0 without support for this test19:12
perrito666sinzui: I dont have the setup for that test at hand19:14
perrito666It does look like restore is taking a lot19:14
perrito666I am really hoping it is only that19:15
sinzuiperrito666, setup takes longer. I am updating the test to work with modern HP. I am 30 minutes into it without achieving HA19:16
perrito666sinzui: I really need to EOD today but Ill be back later (I need to take 3h bus ride)19:26
sinzuiperrito666, Thank you for your time19:27
perrito666so if you throw in more data I will be glad to take a look19:27
perrito666C U ALL19:27
JoshStroblHey guys, I'm having issues with deploying my charm locally (it sits in /home/vagrant/charms/trusty/). I exported JUJU_REPOSITORY as /home/vagrant/charms and deploy with juju deploy local:trusty/metis however the log (.juju/local/log/unit-metis-0.log) claims the install file does not exist (it does in the metis/hooks). The charm proceeds to stay in a dying state after that (I have to destroy the machine and then service in19:43
JoshStroblorder to try again). Any tips? Full unit log at http://paste.ubuntu.com/7738369/19:43
sinzuilazypower, mbruzek ^ Do you have any experience with JoshStrobl's issue?20:00
lazypowerlooking now20:01
lazypowerhey JoshStrobl o/20:01
JoshStroblhey lazypower o/20:01
mbruzekHi JoshStrobl is this a local charm ?20:01
sinzuiJoshStrobl, the file is exec able? 777?20:01
lazypowerthats interesting... the log leads me to believe the hooks weren't copied over.20:01
sinzuifor 75520:01
JoshStroblYes, all the files in hooks are executable20:01
mbruzekJoshStrobl, have you checked in a branch to bzr yet?20:01
mbruzekJoshStrobl, Do you have some time to share you screen on G+?20:02
JoshStroblmbruzek: I have an existing branch that is out of date. My work takes place via Git and GitHub and the charm will be residing on Launchpad.20:02
JoshStroblmbruzek: Sure thing, let me get my mic set up.20:02
JoshStroblmbruzek: It'll be a few minutes :)20:03
JoshStroblmbruzek: gotta install the plugin :D20:05
mbruzekno worries.20:05
JoshStroblmbruzek: the link says the party is over20:09
mbruzekJoshStrobl, I am in the room right now20:09
JoshStroblstill says it after refreshing :\20:09
lazypowerJoshStrobl: you may need to clse out of chrome/firefox and restart it, the plugin is notoriously terribad at first time installation.20:10
=== tvansteenburgh1 is now known as tvansteenburgh
=== vladk is now known as vladk|offline
alexisbsinzui, I responded to your mail, but I am also happy to chat if you like20:42
sinzuialexisb, Thank you. I am happy for the leads to speak up.20:43
* JoshStrobl looks at Microsoft and curses out loud. Then opens up gEdit.20:52
mbruzekJoshStrobl, looks like dos2unix would help you out there Josh.20:53
lazypowersinzui: found the issue - it was windows line endings in character encodings.20:53
lazypowerFYI - one more thing we need to be aware of as our story with windows users grows.20:53
lazypower(josh, you're a pioneer using windows line endings in linux.)20:53
JoshStroblI would like to say I'm proud of that, but I'm not.20:53
JoshStroblI don't even know how that happened...20:54
lazypowerid dint know gedit had that as an option myself20:54
sinzuilazypower, maybe charm proof could assist us20:54
JoshStroblI blame gEdit...because you know, it is totally never the developers fault /s20:54
lazypowerbut its nice to know we support the desktop interop20:54
lazypowersinzui: really good idea. i'll file a bug20:54
JoshStrobllazypower: ping me the launchpad bug when it's up, I'll mark myself as affected20:55
sinzuilazypower, There was a file in juju core's win script that had a single win ling ending in it. IT was maddening to find.20:55
_mup_Bug #1336936: Charm proof to detect character encoding <Juju Charm Tools:New> <https://launchpad.net/bugs/1336936>20:56
JoshStrobllazypower: thanks20:56
lazypowersinzui: wow - epic. a single line?20:56
lazypowerhow does that even happen?20:56
lazypowersmallest broken merge, ever.20:56
sinzuiyeah, how does that happen?20:56
JoshStroblYou guys saved me several hours and probably a few euro at the pub. You guys rock.20:58
lazypowerJoshStrobl: we aim to please. tell your friends about us.21:01
JoshStrobllazypower: Will do.21:01
JoshStroblOh and by the way, the dos2unix tool works flawlessly. I'd recommending making note of that tool in the docs for Winderps users. I was able to just do dos2unix * in the hooks directory and it converted all the scripts to unix "format".21:04
=== thumper is now known as thumper-afk
wwitzel3fwereade: ping22:14
davechen1ywallyworld__: http://paste.ubuntu.com/7739086/22:22
davechen1ymore problems with the openstack provider22:22
davechen1ywallyworld__: gah, sorry, here is the real eror22:23
davechen1ybug report coming22:23
katcowallyworld__: hey another question, looking at this comment. it says that the default for rsyslog is 0640; is that sufficient, or do we want to exclude the group as well?22:24
_mup_Bug #1336980: provider/openstack: map ordering issue in tests <juju-core:New> <https://launchpad.net/bugs/1336980>22:24
davechen1yspot the bug22:24
katcowallyworld__: ah, and i remembered my other question. open to the room as well: is there a faster way to recreate the log-files in /var/log than to destroy-environment and bootstrap a new one?22:36
wallyworld__katco: on a call, will respond soon22:37
katcowallyworld__: no worries and no rush. thank you, sir!22:37
=== revagomes_ is now known as revagomes
wallyworld__katco: sorry, off call now. today is my meeting day, lots of them. using juju proper, there's no faster way that i know of. you could though change the rsyslog conf file and stop and restart rsyslog (deleing the all machine log in between) to see any permissions changes take effect23:21

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!