[00:10] wallyworld_, looking at the file dan uploaded .. there's a wierd 5 minute gap http://paste.ubuntu.com/7502559/ [00:10] where the agent restarts itself [00:11] oh.. right after it reconfigures logging to WARN.. hence no log [00:25] Is anyone about to help me diagnose a local kvm issue on a ppc64 machine in the lab? [00:25] http://pastebin.ubuntu.com/7503638/ [00:28] sinzui: my gut says you are missing a package [00:29] juju-local does not depend on the entire set of packages required to boot ubuntu images [00:29] thank you davecheney I will investigate [00:29] (based on my experiences 6 weeks ago) [00:29] something like ubuntu-cloud-template [00:29] hang on [00:29] i'll stop guessing [00:29] davecheney, I did install the list of packages it suggested when I tried it first, but I understand the list might no be accurate [00:29] possibly you need lxc-templates [00:33] davecheney, lcv works fine. CI now tests lxc on ppc64el for every rev we change [00:34] ok, i don't know jack about kvm [00:34] it didn't even work on ppc when I was using it [00:34] as all the machines i had access to were themselves inside kvm [00:34] which might be the problem [00:38] davecheney, yeah. I know kvm in kvm works. This is my last hope to get this working. I can see no cloud images were downloaded. Maybe there aren't ppc64el cloud-images suitable for this case [00:50] wallyworld_: axw http://paste.ubuntu.com/7503679/ [00:50] getting closer every day [00:50] farking awesome [00:51] davecheney: nice [00:51] hmm, what's that replicaset one [00:51] replica set stuff is always busted [00:51] sorry to whoever wrote it [00:51] its really unreliable [00:51] yeah I know, I've been working on making it more reliable [00:51] also on golang go :-( [00:51] apparently still not [00:51] davecheney: I have a fix for the utils one [00:52] davecheney: TestSecureClientNoAccess failure - that's because we're trying to connect to a non local address [00:52] just about to approve [00:52] wallyworld_: it's cos of $http_proxy [00:52] ah right [00:52] https://bugs.launchpad.net/juju-core/+bug/1322392 [00:52] I've got a CL to clear it [00:52] <_mup_> Bug #1322392: utils: test failure due to proxy

[00:52] great [00:52] ^ sorry, just raised this, it's probably a due [00:52] dup [00:53] yeah I opened one yesterday [00:53] do we have a bug for TestConstraintsValidator? [00:54] wallyworld_, no [00:54] ok, that's an easy fix anyway [00:55] wallyworld_: can you please take a look at https://codereview.appspot.com/100730043/ to make sure I haven't buggered up the intent of the test [00:55] sure [00:57] davecheney: seen this yet? https://code.google.com/p/gosmith/ [00:57] axw: looks fine [00:57] wallyworld_: thanks [00:58] axw: yeah, dmitry has been ripping the compiler a new one with that [00:58] it's great [00:58] now it's llgo's turn [01:00] axw, Do you have any insights into this? http://pastebin.ubuntu.com/7503710/ [01:01] sinzui: 'fraid not [01:01] thank you axw [01:02] about to get on a meeting, I will take a look at the kvm code later to see if there's somethign else you can run [01:14] axw: I found the issue. [01:14] uvt-kvm: error: no images found that match filters ['release=trusty', 'arc [01:14] h=ppc64']. [01:16] wap, wap, wah [01:18] hang on [01:18] is that 'cos the arch is wrong [01:18] it should be ppc64el, or ppc64le [01:19] davecheney, exactly what I suspect. [01:20] I see ppc64el release [01:20] http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.json [01:20] davecheney, but lxc works so WTF [01:20] sinzui: we have several functions inside juju that translate btween the various aliases [01:20] it is possible lxc knows more than kvm [01:21] or even that the lxc images have a different naming scheme [01:21] I could test this by creating an alternate stream, but my weekend has started. I am not keen to start a project at this hour [01:23] davecheney, before the ports were opened of for stilson-06, I hacked up a https server and the hosts file on the machine to help juju find the tools [01:28] sinzui: any chance I could get direct access to the trusty CI machine that runs the unit tests? there's one in particular that I cannot reproduce anywhere else [01:29] axw, absolutely. one moment while I install your key into one of them [01:31] axw, who else is in tanzanite? [01:31] sinzui: wallyworld_ and mgz [01:32] yes please [01:38] davecheney: should be a couple less test failures on trunk now [01:40] axw: i'm working on a mega patch that will remove 90% of the uses of DeepEquals in our tests [01:41] IMO they are almost all wrong [01:41] mmkay [01:41] for example [01:41] c.Assert(manifest, jc.DeepEquals, set.NewStrings(dummyManifest...)) [01:41] this is a set [01:41] order is unimportant [01:42] set is a map underneath I think [01:42] so DeepEquals will work anyway? [01:42] logically your testing for set equality [01:42] not ordering [01:48] wallyworld_, axw, and mgz: I sent an email about accessing all the ppc64 env I manage [01:48] thanks [01:57] wallyworld_, axw. I reported bug 1322401 about the issue. I will work on other tests next week [01:57] <_mup_> Bug #1322401: kvm on ppc64el cannot fine images

[01:58] sinzui: sorry if you've been talking to me, my client didn't realise it was disconnected === axw_ is now known as axw [01:58] axw, I wasn't all is fine. and I didn't say the bug had to be fixed this week [01:59] okey dokey [02:01] oh, axw, or wallyworld_ I was supposed to rebuild 1.19.2 to make ppc64el and arm64 tools. Since ppc64el has so many names, should I always make the tool with a specific name? when the script sees "ppc64el" debs, make a "ppc64" tools? [02:01] ppc64 is the arch name we use in the juju code [02:10] axw, to restate your brief remark. Juju knows about ppc64. It will look for tools by that name. when the assemble script sees ppc64el, ppc64le, or power8, it will make a ppc64 tools from the first one, and ignore the rest. [02:11] ah right. sounds reasonable to me [02:13] axw, I think that is right since that is the rule I hacked together to deploy the jenkins slave to stilson-06. I didn't want to put it in production without a dev having a chance to say It was dangerously wrong [02:15] sinzui: davecheney would probably know better, but AFAIK ppc64el is the right dpkg arch to match against [02:15] axw, that is the common one. [02:16] axw, http://ports.ubuntu.com/pool/universe/j/juju-core/ [02:16] ^ powerpc is still there, there was a ppc64le for a few weeks [02:18] powerpc eh... fairly sure we don't care about that [02:20] The assembly script makes a tool for each one. There are not tools for ppc64 [02:23] maybe I should make an extra tool each time a ppc64el is found [02:26] davecheney, I /think/ juju is looking for ppc64 tools, not ppc64el tools. Should I be making I make tools named arm64 each time I find a new ppc64el deb? [02:27] davecheney, I think I need to do that because the streams I made to deploy jenkins-slave to stilson-06 didn't work with ppc64el [02:29] axw: I think you've fixed it https://bugs.launchpad.net/juju-core/+bug/1322392 [02:29] <_mup_> Bug #1322392: utils: test failure due to proxy

[02:29] sinzui: I don't know [02:29] this keeps changing [02:29] davecheney: excellente [02:30] afaik when you use --upload-tools the name of those tools were -ppc64el.tar.gz [02:31] davecheney, users aren't supposes to use it [02:31] sure [02:31] davecheney, since I have an env with tools that were never published, and my computer is amd64, upload-tools is usless [02:31] but that leads me to think that the the published name of the tools should also have the same exension [02:31] so [02:31] https://juju-ci.s3.amazonaws.com/tools/streams/v1/com.ubuntu.juju:released:tools.json [02:32] ^ I created a stream of just the tools we run ci on, including a tools for ppc64. I chose that label because ppc64el didn't work [02:34] sinzui: https://bugs.launchpad.net/juju-core/+bug/1290654 [02:34] <_mup_> Bug #1290654: juju must not rely on the output of uname -m

[02:34] possibly related [02:35] davecheney, yep, suffered that bug with i386 too [02:35] I have a hack in the run-unit-tests jobs to ensure i386 is matched correctly [02:36] I think the safest thing to do is to make two tools for each ppc64el [02:41] sinzui: do you have a fixed ec2 instance for running the amd64/trusty unit tests? or do you start/stop instances each time? [02:41] axw, neither, we start an instance with an ami, then run apt-get update (and possibly add ctools) [02:42] sinzui: each run? [02:42] yes [02:42] k [02:42] which AMI? I want to try to reproduce this EnsureAdminUser test failure [02:43] cannot repro anywhere [02:43] axw, http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/trunk/files < run-unit-test is for amd64 and i386 [02:43] thanks [02:44] axw, run-unit-tests m1.xlarge ami-018c9568 [02:45] axw, some of the tests now have ami's embedded in them. It was a problem when many bit rotten while we were in vega [02:45] alright, will give that a shot... [02:46] * sinzui updates each test in http://juju-ci.vapour.ws:8080/ to show what AMIs are used [02:52] axw, This is the whole test script http://pastebin.ubuntu.com/7503961/ [02:52] sinzui: thanks [03:45] davecheney: i'm about to fix the TestConstraintsValidator test failure for gccgo in the joyent tests [03:45] are you alreayd doing that one? [03:47] wallyworld_: can you please take a look at https://codereview.appspot.com/96540047/ [03:47] sure [03:48] axw: take my +1 with a grain of salt since i got it wrong the first time [03:48] sure :) [03:49] axw: so it seems like you've changed it to how it should have been done first time round when it ws originally written - take use tomb [03:49] yep [03:49] nfi why it wasn't to start with [03:49] nate wrote it originlly and roger +1ed :-) [03:49] i think [03:49] it was fine, but only on gc [03:50] yeah :-( === Ursinha is now known as Ursinha-afk [04:23] axw: remind me - we reduced the mgo dial timeout to 30 seconds because it used to be 10 minutes and sometimes tests wouldn't notice a connection had disappeared and so waited 10 minutes for the timeout to expire, right? [04:30] wallyworld_: hey sorry, otp [04:30] just a mo === Ursinha-afk is now known as Ursinha [04:32] wallyworld_: it was reduced to 30s because it shouldn't need to be any longer than that; other connections should time out and retry. Yes, some tests were failing to connect for 10m and then the test got killed [04:32] so we wouldn't get useful output [04:55] axw: so i might have to add a retry look around it because of the bug where a user's slow machine can't start mongo fast enough and bootstrap dies [04:56] wallyworld_: around what? [04:56] MaybeInitiateMongoServer [04:56] or an upstream caller [04:56] wallyworld_: ahh.. right, in bootstrap it doesn't retstart [04:56] restart* [04:56] bugger, sorry I missed that detail [04:56] in bootstrap if it can't connect after 30s yeah it dies [04:57] np, simple enough fix [04:57] wallyworld_: you could just set the timeout explicitly in bootstrap [04:57] rather than use the default [04:57] could do [04:58] i think a loop and smaller timeouts are perhaps better [04:58] maybe not, doesn't really matter for this situation i don't think [04:59] I don't think it matters. whatever works and is elegant [04:59] with preference to the former :) [04:59] sure [05:00] I have nfi about this EnsureAdminUser thing. I can't reproduce on the same instance type as used in CI [05:00] also trying a faster instance type on ec2 [05:00] no luck [05:01] I'm going to try crafting up a separate program soon to cut down the time initialising mongo, speed up the test cycle === vladk|offline is now known as vladk [05:29] davecheney: have you done a recent gccgo test run? on my machine, i get compiler errors and seg faults [05:29] lots of tests pass though [05:33] davecheney: when you get a chance could you do another test run and pastebin the results? [05:49] review please (fairly easy one): https://codereview.appspot.com/94710046 === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [06:23] wallyworld_: I may go back on what I said before about longer timeout vs. retry [06:24] wallyworld_: if we use a retry loop, it may help with these i/o timeout errors [06:24] that's sorta what i was thinking [06:24] i'll add a loop [06:25] bbiab, gotta go get kid from train station [06:25] k [06:35] wallyworld_: interesting thing- on CI, the agent and agent/mongo package tests take a long time to run (>15s). on my machine, they take ~1s each [06:36] on another run, they took closer to 40s each [06:36] on CI [06:37] just a guess, but maybe the machine is still busy initialising when the tests start up [07:00] could be [07:01] maybe we delay starting the tests [07:09] axw: do you know how dave has been running his tests? on amd64 using -compiler=gccgo or on ppc? [07:11] on ppc I think [07:17] hi all, [07:18] hey rogpeppe [07:18] axw: yo! [07:18] axw: how's tricks? [07:18] not too shabby [07:18] how are you? have a good break? [07:18] morning [07:18] morning TheMue [07:18] TheMue: hiya [07:24] wallyworld_: I don't think it's worth expending any more effort on these i/o timeout bugs until we try adding a delay at the beginning of CI [07:24] agreed [07:24] wallyworld_: I can extend timeouts and so on, but I think it'll be neverending [07:25] I'll pick something else up for now [07:25] 1305014? [07:26] yep [07:26] or 1322281 [07:26] we are sort of expecting more info on those top 2 cards to come in [07:26] the cloud init logs [07:27] davecheney: you around per chance? [07:29] axw: yeah, had a great break thanks [07:30] axw: two weeks on cyprus [07:31] nice [07:50] fwereade: hiya, i've done another pass over that doc if you have a chance to look, no rush [07:51] wallyworld_, cheers [07:52] fwereade: i have standup soon and then will be back from soccer in about 4 hours if you needed to chat or whatever [07:53] wallyworld_, nothing hugely springs to mind, but I'm here if you want me :) [07:53] morning all [07:53] fwereade: i meant about the doc [07:53] once you read it [07:54] wallyworld_, I have lots of other stuff I have to figure out today so I might not get there in time, but yu never know your luck [07:54] np, let's tak monday [09:01] mgz: you around for standup? [09:01] joining [09:20] morning [09:20] mornin' perrito666 [09:21] mgz, perrito666: morning [09:39] hello === vladk is now known as vladk|offline [09:52] my connection is ropey today [09:57] anyone know if the bot automatically pulls new deps? i *thought* it did now, but this seems to indicate otherwise: https://code.launchpad.net/~frankban/juju-core/update-testing-package/+merge/220772 [10:05] rogpeppe: it doesn't [10:07] mgz: what's the process for updating a git repo in tarmac? should we ask Curtis? [10:08] frankban: I'll do it [10:08] mgz: thanks [10:13] mgz: since we want to migrate some packages to github, we will need to update the dependencies file several times. I am also taking notes about the migration process. It would be nice to have the bot to pull changes automatically, but for the time being, is it ok if I include something like "ask mgz or sinzui to pull the last revision of the updated package in the tarmac machine before landing the dependencies.tsv c [10:13] hange"? [10:14] frankban: this is all changing, this week [10:15] mgz: as part of the git migration? [10:16] yeah [10:17] frankban: for now, bugging me is fine [10:24] mgz: cool thanks, please ping me when I can re-submit the branch [10:27] anyone have strong feelings about breaking the utils.Apt* functions into a separate package? [10:27] I'm a big fan of small, focused package, so seems like a good idea to me [10:28] also potentially moving the proxy-specific functions in osenv into their own package too [10:28] fwereade: ^ [10:29] +1 to small, focused packages [10:29] i'm focusing more on dependency hygiene than small packages per se [10:30] i don't mind larger packages if they're coherent [10:30] coherent packages, small files [10:31] e.g. focussing on individual types [10:31] i don't mind large files either, if the contents of each file makes sense as whole [10:31] s/as whole/as a whole/ [10:31] I find it tought to build large packages with large files if I am making things coherent [10:33] rogpeppe, +1 to dependency hygiene, small files, and small packages [10:33] i think the most important thing is that for every exported thing, it's easy to say "that thing definitely *belongs* in this package" [10:33] rogpeppe, I think you have a greater tolerance for the latter two than many of us [10:33] fwereade: possibly. i like to follow the examples demonstrated by the go stdlib [10:35] pwd [10:35] rogpeppe, I get antsy past a few hundred lines in a file, and past a few files in a package, tbh [10:36] rogpeppe, mybrain has to start swapping things out at around that point [10:36] fwereade: i think there's a lot to be said for intra-package hygiene too. [10:37] fwereade: (for avoiding that situation) [10:37] rogpeppe, definitely, but it's *really* hard to get that right in a large team -- package boundaries don't get violated accidentally, but it's easy to miss those sorts of screwups within a package [10:38] rogpeppe: I'm hesitant using the stdlib (of any language) as a guide for how you should craft packages that aren't part of the stdlib. [10:38] rogpeppe, sometimes they still get violated deliberately because someone doesn't understand the situation, but it's usually more obvious that the exposed surface has just changed significantly [10:39] fwereade: with too many packages, it's very easy to have packages with no proper raison d'etre - they tend to have fuzzy boundaries and end up having random collections of not-really-related functionality in them [10:39] axw: ping, you still around [10:41] rogpeppe, IME that's more a problem with large packages -- if the bias is to create small packages at the drop of a hat, existing packages keep their focus [10:41] rogpeppe, and if two packages really are so closely related as to be better as one, it's an awful lot easier to merge than to split [10:42] fwereade: it's all trade offs as usual :-) [10:42] rogpeppe, yeah :) [10:42] lc [10:44] fwereade: tbh i've found it pretty easy to split packages in the past === vladk|offline is now known as vladk [10:45] TheMue, standup? [10:49] wwitzel3: the standard library for Go is pretty good standard to go by for coding standards. Occasionally there are functions which are hard for mere mortals to follow, but in general, it's a pretty good template. [10:50] natefinch: morning [10:51] voidspace: morning [10:52] natefinch: I guess I've just found that things done for the sake of the "stdlib" are not always the best conventions to be used in your own packages. [10:53] natefinch: that said, my exposure to the Go stdlib at this point is still limited, so I'm still open to the idea [10:55] natefinch: and I've always pissed and moaned that the stdlib for a language wasn't the defacto go to place to learn good practices for building packages, so hopefully as I get more exposure to the Go stdlib, it will be able to promote it as the language that does just that. [10:57] wwitzel3: I hope so :) [11:05] fwereade: you have a moment for a hangout? [11:05] wwitzel3, sure [11:05] wwitzel3, moonstone? [11:06] fwereade: yep === vladk is now known as vladk|offline [11:29] trivial automatically-made change to review anyone? https://codereview.appspot.com/97670044 [11:30] natefinch, fwereade, wwitzel3, voidspace: ^ [11:30] (it just sorts all imports standardly) [11:31] voidspace: sorta here === vladk|offline is now known as vladk [11:33] rogpeppe: LGTM [11:33] natefinch: ta [11:33] voidspace: (re rsyslog review) that's why I suggested to remove the upgrade step; the immutability of syslog-port will cause the upgrade-step to fail now [11:33] voidspace: although... there is a way to open a state connection that doesn't validate environ config changes [11:35] axw: so, when an upgrade is done is the code upgraded first - or the upgrade steps run *then* the code update [11:35] axw: hmmm... the upgrade must be run with the current codebase [11:35] axw: so it can't work [11:35] voidspace: it runs with the new code [11:35] axw: do we want to *aim* to allow a 1.16 -> 1.20 upgrade [11:35] if so we need to bypass validation [11:35] if not we need to remove those upgrade steps [11:35] voidspace: I think that was the aim, yeah. so I guess we should leave it in but reopen state without the policy [11:36] axw: how do I do that? :-) [11:36] voidspace, axw: we certainly aim for that to work, yes [11:36] so yes, with my current mp the upgrade tests fail [11:36] fwereade voidspace: can we just leave a TODO for that for now? [11:36] for the obvious reason [11:36] ah [11:36] I should have run that before proposing of course [11:37] axw, voidspace: well, we want 1.20 relatively soon, so I'm not sure how good an idea that it [11:37] axw: how hard is it to bypass validation for an upgrade step? [11:37] voidspace: not sure, just taking a look now [11:37] axw: thanks, I appreciate it [11:39] voidspace: the upgrade steps get an agent.Config, so can do state.Open [11:39] voidspace: state.Open has an optional policy argument, which you'd want to pass in as nil [11:40] voidspace: hrm, but I'm not sure if it'll work, because syslog-port is provider-independent [11:41] wow [11:41] ok launchpad.net/juju-core/provider/joyent 388.185s [11:41] voidspace: I don't think it's going to work [11:41] rogpeppe: yeah, wallyworld_ has a pull request to the joyent packages to fix that [11:42] axw: hmm [11:42] apparently it's due to repeated crypto calls [11:42] axw: it wouldn't be the first time... [11:43] voidspace: so I don't see an easy fix here. it's possible to modify the settings docs in mongo directly, which is a little bit horrible [11:44] axw: instead of adding it to the immutableAttributes, can we have custom validation? [11:44] axw: which will allow setting it to the default but disallow all other values [11:44] axw: so the upgrade will work [11:45] voidspace: actually I may be wrong, I think it *does* skip all config validation. I'll investigate a bit more... [11:46] axw: let me look at the failing test, it may be failing when it tries to set the value *before* running the upgrade [11:46] ehh, could do that, but I'd rather not have super special case things [11:46] axw: sure [11:48] nope, it's running UpdateRsyslogPorts that fails [11:49] fwereade, natefinch, wwitzel3, voidspace: another fairly trivial review. just a code move. please ignore the dubious import formatting - i wanted to avoid extra diffs from merging the above CL before it landed. https://codereview.appspot.com/100730044 [11:49] this is just a code move (plus some associated renaming) [11:50] natefinch: where is your tool for stripping log output from test runs? [11:50] voidspace: opening without the policy allows you to bypass the validation altogether [11:50] provider-specific or not [11:51] just tested it in the state package [11:51] voidspace: go get github.com/natefinch/nolog [11:51] natefinch: thanks [11:52] voidspace: so, in the upgrade step- state.Open(agentConfig.StateInfo(), state.DefaultDialOpts(), nil) [11:52] awesome [11:52] I was just wondering how to get the Info [11:56] rogpeppe: lgtm [11:56] natefinch: thanks! [11:56] I love the way Go namespacing means you don't have to put "Apt" in the name of all the types and functions [11:57] fwereade, dimitern, natefinch: ISTM that we should probably allow a single `import "foo"` statement in a file so that we can work nicely with goimports, which will format things that way if it adds a single import. [11:58] fwereade, dimitern, natefinch: it would make life easier for ourselves, and it seems a fairly trivial thing [11:58] rogpeppe: yeah, that seems fine. not wrestling with standards that conflict with goimports seems like a good idea in case anyone wants to use it. [11:59] natefinch: lots of people run goimports on save [12:00] rogpeppe: yep. Certainly for things like this where it doesn't really matter either way, if one makes goimports happy, that seems a reasonable standard. [12:02] my lunch date is here [12:02] * voidspace lunches [12:08] rogpeppe, goimports leaves it alone if you reformat it [12:09] rogpeppe, but i'm fine either way for a single import, as long as it's consistent across the code base [12:09] dimitern: yeah, but if it adds a new import when there was none already, it doesn't add the () [12:09] dimitern: i'm happy allowing either form, tbh [12:09] dimitern: i don't think total consistency matters that much here [13:22] fwereade, mgz - any chance of getting https://codereview.appspot.com/98260043/ reviewed and landed? === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha [13:51] jcw4, approved, keep half an eye on it to make sure it lands [13:52] fwereade: go bot will publicly shame him if it doesnt :p [13:57] lol... tx fwereade [14:14] natefinch: this is my fix https://codereview.appspot.com/91660043/patch/20001/30010 [14:16] * perrito666 sees the noise in the standup was generated by a trunk pumping an overflowed sewer... thanks cthulu my nose is a bit clogged today [14:16] heh [14:22] jcw4: sorry, I thought we'd said yesterday to just land it, [14:23] jcw4: branch seems to have a text conflict with trunk which will need fixing [14:23] jcw4: then it needs a commit message, which the bot should then pick up and merge [14:30] bodie_: you around? [14:30] hullo [14:30] I was looking at the json schema package [14:30] aye [14:31] I have some concerns [14:31] all ears! [14:31] it looks like it could be ok if it were fixed up [14:32] I was wondering what Dave Cheney meant about 66% coverage -- the author claims 246/248 tests passing on json-schema [14:32] that's the only reason I thought it would be acceptable [14:32] I think he means code coverage, so like 33% of the code doesn't get touched by the tests [14:33] ah, is there a way to check that that I'm not aware of? [14:34] bodie_: go test -cover :) [14:34] mgz: thanks... I saw the text conflict message, I'll try to figure out how to resolve it... should I just merge trunk into my branch? [14:34]