[00:00] menn0: what is the error? === axw_ is now known as axw [00:00] wallyworld: did you see my last message? sorry, dodgy wifi [00:00] axw_> "bad HTTP response" means the API server failed to find the tools locally, and failed to find them remotely [00:00] perrito666: some machine agents are unable to download tools for upgrade. they get a HTTP 400 error. [00:01] axw: yeah, i found where that's being generated - the sendError(400). but we need extra info [00:01] hence my earlier request for more logging [00:01] in that area [00:01] actually hrm, it also means it found an entry in simplestreams but failed to download it [00:01] weird [00:01] yeah [00:04] menn0: mm I doubt it but you could very well try a rever tand see what happens [00:04] perrito666: I'm pretty certain it's not your change [00:10] man hot oil realy burns [00:18] wallyworld: https://github.com/juju/juju/pull/718 [00:19] looking [00:20] let's land this and see wtf is happening [00:29] wallyworld: hmm, wtf [00:29] there's weirdness in the simplestreams [00:30] wallyworld: http://paste.ubuntu.com/8304581/ [00:30] is that version at the top meant to be there? [00:30] I guess it's not actually a problem though, since it's finding the tools... [00:32] that version at the top will be used for the containing elements unless overridden. ideally we wouldn't be generating mixed streams like that [00:49] sinzui: bug 1366887 is starting to look like it's caused by that known mongo issue https://jira.mongodb.org/browse/SERVER-11807 [00:54] interesting, IBM just sent a patch to gccgo to support Linux s/390 [00:56] uff, sounds large review task [01:05] wtf mate [01:05] thumper: is this related to identity changes? [01:05] Error details: [01:05] invalid entity name or password [01:05] I just upgraded from 1.20.6 to master, got that [01:07] axw: I believe wtf is the stronges language I have see coming from you [01:09] I meant, oh bother [01:10] axw: you are going to go into a craziness rampage during a sprint and kill us all right? [01:10] * axw twitches [01:23] davecheney: thanks for the review. did you see my replies to https://github.com/juju/juju/pull/715 ? [01:25] axw: probably [01:25] axw: oh fuck [01:25] oh fuck oh fuck ... [01:25] heh [01:25] oops [01:26] shit shit shit shit shit... [01:26] * thumper thinks [01:27] waigani: this relates to the change we landed yesterday where the env user is now needed for api login [01:27] um... [01:27] axw: actually, no [01:27] * perrito666 notices this is coprolalia day [01:27] axw: maybe? [01:27] axw: we should chat [01:30] axw: you'd get that error response if you tried to access the API as a client before the upgrade steps ran to add in the env user [01:30] thumper: sorry went afk for a bit. I tried again and it worked... [01:31] axw: yeah, we changed auth a little yesterday [01:31] axw: there now needs to be an envuser in the environment [01:31] thumper: I tried it a few times in that run, over a few minutes [01:31] axw: which gets added in an upgrade step [01:31] so maybe the upgrade went pear shaped [01:31] will collect logs if I see it again [01:32] cheers [01:35] wallyworld: are you OK if I JFDI the landing for https://github.com/juju/juju/pull/717? [01:35] wallyworld: at the very worst, we get more logging context for failures [01:35] wallyworld: at the best (?) we don't fail as much [01:35] i think that sounds ok [01:36] just wanted someone else to agree with me :) [01:36] scaredy cat [01:36] is it JFDI or __JFDI__ [01:36] can't recall [01:36] due diligence [01:36] * wallyworld was yanking thumper's chain [01:37] * thumper consideres his chain yanked [01:37] wallyworld: did you remove the "retry the tests if you fail first time" in the bot? [01:37] yup [01:39] cool [01:39] * thumper goes to walk the dog and get antibiotics for sick child [01:39] * thumper back soon [01:40] thumper: beware, dont mi those two activities [02:11] wallyworld: getting a different error now, unexpected deletion of resource catalog entry with id "941af6aa084ab96c30440ee74d4a321761c47bc5f069b8297d387578b919e12544227eed4fc9db1eb03d978c998e33b1": resource with id "941af6aa084ab96c30440ee74d4a321761c47bc5f069b8297d387578b919e12544227eed4fc9db1eb03d978c998e33b1" not found [02:12] one sec, otp [02:18] axw: so something is deleting the same record twice, not sure without digging why [02:18] wallyworld: in the branch, I don't remove blobs at all. blobstore must be doing that [02:18] wallyworld: it does handle concurrent uploads to the same path right? :) [02:18] supposed to [02:19] if uses the assert txn stuff [02:19] and ref counts [02:20] axw: ah, that message is a bit misleading; it's an error doing a Get() [02:21] it's saying it tried to fetch something that wasn't there [02:21] wallyworld: I don't think so, this is in the code that's trying to cache the tools to storage [02:21] wallyworld: full error message: [02:21] 2014-09-10 02:11:01 ERROR juju.apiserver tools.go:55 GET(/environment/a7aca243-c2ef-4912-883b-049182059330/tools/1.21-alpha1.1-trusty-amd64?%3Aenvuuid=a7aca243-c2ef-4912-883b-049182059330&%3Aversion=1.21-alpha1.1-trusty-amd64&) failed: error fetching tools: error caching tools: cannot store tools tarball: unexpected deletion of resource catalog entry with id "09d4e310662142df8fc5304484ca33fc5b1ab9d1e5e1fecc09cef1369227fa71aa38c63be34dca95d1c7 [02:21] 6ff6f488df64": resource with id "09d4e310662142df8fc5304484ca33fc5b1ab9d1e5e1fecc09cef1369227fa71aa38c63be34dca95d1c76ff6f488df64" not found [02:22] look in putResourceReference [02:22] it's a sanity check at the end [02:23] yep, I see it [02:23] it could be there's a race there [02:23] axw: I'm getting a vet warning when I push. looks like something you did recently [02:24] maybe existingResourceId comes back as != "" [02:24] axw: logger.Infof("%v tools not found locally, fetching") [02:24] in apiserver/tools.go [02:24] missing arg to Infof [02:24] menn0: sorry, was adding logging in a rush, will fix [02:24] axw: np [02:25] axw: how often does that error occur? [02:25] wallyworld: that's what's tripping up CI from the looks of things [02:26] it's occurring over and over again [02:26] hmm, ok. but it works locally for you [02:26] 2014-09-10 02:06:33 WARNING juju.storage managedstorage.go:183 cleaning up resource catalog after failed put [02:26] yep [02:44] axw: i might have an idea [02:44] wallyworld: I've reproduced it locally now... not sure if it's timing related or not [02:44] wallyworld: agh... write error: No space left on device [02:44] wallyworld: for the bot [02:44] wallyworld: that is a try again error isn't it? [02:44] thumper: retry [02:44] yup [02:44] why does it happen? [02:44] it seems it could happen if a 2nd upload happens and the ref count is attempted to be updated before the first upload finishes [02:45] thumper: nfi [02:45] wallyworld: got a minute to talk through a failure? [02:45] looking at the port in use one [02:45] thumper: sure, give me a sec [02:45] wallyworld: can you point at code in blobstore? [02:46] axw: putResourceReference [02:46] // Sanity check - ensure resource catalog entry for resourceId still exists. [02:46] _, err = ms.resourceCatalog.Get(resourceId) [02:46] if the blob is still being uploaded, that Get() will fail [02:46] wallyworld: ah, because pending [02:47] right? [02:47] so if it's the 2nd caller, it's suposed to +1 to ref count [02:47] ye [02:47] p [02:47] but if 2nd caller occurs before upload is done, then boom [02:47] i think [02:47] wallyworld: so we should only error if err != nil && err != ErrUploadPending [02:48] i think so, let me take one more look [02:48] though I suppose PutForEnvironment shouldn't really return nil if it's pending still... [02:49] yeah, i think for now, if err != nil && err != ErrUploadPending should do it [02:50] axw: if you can reproduce, can you try that fix? [02:50] wallyworld: I'll have a play with it [02:50] ty [02:50] thumper: did you want to talk now? [02:51] wallyworld: just on with menn0, shortly? [02:51] sure [03:08] waigani: I'm confused, didn't you tell me to delete that comment? [03:08] wwitzel3: which one? [03:08] waigani: the RunCommandsArgs [03:09] waigani: also thanks for the review :) [03:10] wwitzel3: If I did it was a mistake, exported types should always have comments [03:10] wwitzel3: I'm just trying to see if/where I said that ?! [03:12] wwitzel3: btw a lot of the error cases are not covered by tests, but often the error states are from funcs/packages that have already been covered. But just as a suggestion maybe think if any new functions you've introduced need to be tested for expected behaviour when they crap out. [03:19] waigani: ok [03:19] waigani: thanks :) [03:20] np [03:25] wallyworld_: onyx hangout? [03:25] ok [03:36] davecheney: do we have a rule on underscores in var names? [03:36] davecheney: I haven't seen them in juju, but a PR uses them [03:40] wallyworld_: https://github.com/juju/juju/pull/720 [03:40] waigani: you know i'm not a big fan of rules [03:40] i'm more into shouting and throwing my weight around [03:40] lol [03:41] this_isnt_considered_kosher() [03:41] butThisIs() [03:41] is all i know [03:41] okay, I'll s/rule/tend to [03:42] davecheney: I stand corrected: format_1_18Suite [03:44] waigani: plx fix - http://paste.ubuntu.com/8305704/ [03:44] waigani: pls fix - http://paste.ubuntu.com/8305704/ [03:44] waigani: should be "equal(now) || after(now)" [03:44] wallyworld_: https://github.com/juju/blobstore/pull/15 [03:45] looking [03:45] thumper: is that trunk? [03:45] waigani: yes [03:45] waigani: slipped through [03:45] yikes, on it [03:45] waigani: just hit it landing a branch, ta [03:50] axw: thanks for fixing that, if tests pass i'll merge [03:50] wallyworld_: they do, I'm just going to test in juju now... [03:51] ok [03:51] let me know and i'll merge then [03:51] wallyworld_: do you think it would be prudent to use a different path in storage when uploading tools? e.g. generate a UUID to add to the path [03:52] I'd rather not, seems like blobstore should be doing this [03:52] axw: you mean and then make them available at the right path when done? [03:53] wallyworld_: actually... I don't think the path matters does it. the resource catalog entry is based on the hash of the content, riught? [03:53] yep [03:53] thumper: https://github.com/juju/juju/pull/721 [03:57] wallyworld_: juju likes that better [03:57] it still surfaces ErrUploadPending, but it recovers [03:57] i.e. the upgrading agent bounces and tries again [03:58] axw: that's good enough for now :-) [03:58] that pending thing should be fixed in the blob store i agree [03:58] it just wasn't implemented [04:00] thumper: searched, found one other place, updated. [04:01] thumper: merge? [04:02] wallyworld_: https://github.com/juju/juju/pull/722 please [04:02] anyone actually [04:05] thanks waigani [04:05] axw: that was a big one! [04:06] I know, it's a hard life :( [04:08] wwitzel3: shouldn't you be asleep around now ? :) [04:09] axw: sorry, lost my network again right when i saw your PR [04:14] boo... hiss... FAIL: firewaller_test.go:460: FirewallerSuite.TestRemoveMultipleServices [04:15] new intermittently failing test [04:15] * thumper hasn't had much luck landing this branch [04:18] wallyworld_: no worries [04:32] thumper: regarding our earlier conversion regarding clearing the upgradeinfo doc... it will mean a pre 1.21 client won't be able to clear it. only newer clients will have the client side support to sort out a problem [04:32] thumper: does that matter? [04:33] menn0: I don't think so [04:33] thumper: good. that's what I was thinking too but thought I'd better check [04:43] davecheney, waigani: here is the user tag branch - https://github.com/juju/juju/pull/723 [04:43] davecheney: I'd like you to look, but perhaps as a mentor for waigani [04:44] wallyworld_: how do I check the landing bot status? [04:44] http://reports.vapour.ws/ i think [04:45] h, maybe not [04:45] it should accept new jobs as soon as the blocker is marked fit comitted [04:46] bad-record-mac ... again [04:46] * thumper head desks [04:46] I think I'm approaching a record again [04:46] thumper: oh, you want to look at your job? [04:46] http://juju-ci.vapour.ws:8080/job/github-merge-juju/ [04:47] click on the dot next toy yours to see console output [04:47] ah [04:47] ta [04:47] the sooner we move to mongo 2.6, the better [04:48] true that [04:52] thumper: AAAAAAAAAAAAARGH [04:53] params.Status is SO INCIDIOUS [04:53] there is an interface defined in state which reuqires a params.Status [04:53] they joined like simaese twins [05:02] ick [05:02] wallyworld_: success finally \o/ [05:02] \o/ [05:02] davecheney: so is the spelling of insidious [05:02] * menn0 ducks [05:07] har har [05:07] i'm useless without the little red line [05:07] sorry :) [05:07] menn0: anyway, back at the pooit [05:07] i don't know how to undo this [05:07] yeah the actual point sucks [05:07] we have types in the apiserver that have to implment the state.StatusSetter interface [05:08] and that StatusSetter interface depends on types in the apiserver/params package [05:08] i've just stashed a whole buttload of changes [05:08] and i'm going to see what happens if I move state.StatusSetter to the apiserver [05:08] as it's really the api that presents that interface now [05:08] nobody gets to talk to the state server directly anymore [05:09] so that fact that state implements that interface is tangental [05:11] menn0: does that sound sane ? [05:11] state doesn't need to implement StatusSetter as that is an api method now, not a state method [05:11] * davecheney waves hands furiously === uru_ is now known as urulama [05:21] uh, why is there an apisever/client package [05:21] how is that not [05:21] api/ ??! [05:23] davecheney: I think that is the server side part of the client (read CLI) facade [05:23] as opposed to worker facades [05:24] o_O [05:25] no, wait [05:25] _ /| [05:25] \'o.O' [05:25] =(___)= [05:25] U [05:25] ACK! [05:29] go test here/goes/nothin [05:32] jam: yeah, probably, sometimes it is just more productive late at night :) [05:37] wallyworld_: ping [05:37] wallyworld_: did you get my e-mail? [05:37] wwitzel3: I just remember you also waking up at 5am... [05:37] urulama: hi, um, not sure, when did yuo send? [05:37] wallyworld_: 10min ago [05:38] urulama: ah, you are uros [05:38] yes, and i just replied :-) [05:38] jam: yeah, it is more like 7 now, since no one on my team is around at 5 anymore. [05:38] so even if I don't get to bed until 2, I still get a solid 5 hours [05:38] wwitzel3: I'm not sure if 5 can be considered "solid", but if it is enough for you [05:39] wallyworld_: great, saw that PR for the fix in the e-mails, yes. thanks [05:39] urulama: i copied william so he's in the loop about the use of env namespace [05:41] jam: generally more is better, I really just mean it is better than 3 [05:43] wallyworld_: ok, i'll do that as well in the future [05:49] wallyworld_: are you still doing something with https://github.com/juju/juju/pull/714 ? [05:51] wallyworld_: are you still doing something with https://github.com/juju/juju/pull/714 ? [05:52] i made gocheck panic, do I get a prize ? [05:52] you probably should [05:53] axw: i wasn't going to because it just reduces the spam, rather than eliminating it. i did test my solution and it worked, but didn't seem worth it [05:53] i should close the pr i guess [05:53] ok. it will be less spammy in 1.21 anyway [05:53] yes please [05:53] yeah [05:54] axw: i was procrastinating as to whether tp proceed or not [05:54] nps, just wondering what to review next [06:02] aaaaaaaaaaaaand, kaboom [06:02] func (s *BoolSuite) TestNonBooleanValues(c *gc.C) { c.Assert(nil, jc.IsFalse) [06:02] } [06:07] oh no, and other tests depend on this behavior [06:09] axw: wallyworld_ can you do [06:09] go test github.com/juju/testing/checkers [06:09] for me [06:09] i think the test suite is broken [06:09] sure [06:10] davecheney: what commit are you on? [06:10] commit 503e61bd033592d7b6003174389f68454deb7b7a [06:10] Merge: 9f90119 ed4eedd [06:10] Author: Andrew Wilkins [06:10] yup, still broken at tip [06:11] davecheney: works for me [06:11] ooops [06:11] me too [06:11] sorry lads [06:11] PEBKAC [06:11] nps [06:11] i had local changes [06:11] thanks for confirming [06:11] happens to the best of us, and tim :-) [06:12] * davecheney rimshot [06:12] while i have you pity [06:12] does anyone know if there is a c.Check check that does not mark the build as failued [06:12] failure [06:12] basically, I need to write a check that will fail [06:12] hmmm, not sure [06:13] but it should fail, not panic [06:13] maybe I can use c.Panics or something [06:13] c.Assert(nil, jc.IsFalse) [06:13] ^ this panics [06:13] isn't that what Check does: [06:13] ? [06:13] i have a fix, but can't write a test [06:14] davecheney: c.Check(func(), gc.PanicMatches("text")) [06:14] maybe I can just call the checker direclty [06:14] jam: the problem is c.Assert(nil, jc.IsFalse) [06:14] panics gocheck [06:14] but thereis no value of nil that will ever be jc.Istrue [06:14] davecheney: c.Check(func() {c.Assert(nil, jc.IsFalse}, PanicMatches()) ? [06:14] jam: thanks, will try [06:14] or you don't want it to panic, but to just assert false [06:15] well, nil is neither false nor true [06:15] so we can't evne write nil, gc.Not(jc.False) [06:15] maybe I should just call the checker directly [06:15] so even if the checker doesn't panic [06:15] it will still mark the test as a fail [06:16] to be pedantic, gc.Not(jc.False) != jc.True, exactly because of stuff like nil (or a digit, or string, etc). I don't really know what the expectation is for jc.IsFalse, but I personally would expect it to fail the assert if it got nil [06:18] hmm, bool_test.go:42: c.Assert(nil, jc.IsFalse) [06:18] ... obtained = nil [06:18] ... expected type bool, received [06:18] ths is the problem [06:18] i have to test this [06:20] sod it, i'll just call the checker directly [06:26] https://github.com/juju/testing/pull/34 [06:26] jam: https://github.com/juju/testing/pull/34 [06:26] i was making it way to hard for myself [06:30] davecheney: lgtm [06:30] jam: ta [06:31] now back to what is _was_ doing === urulama-afk is now known as urulama [07:44] morning all [07:57] mattyw, morning [07:58] anyone know why merges are still blocked? The 2 blockers that are currently in place are marked as "Fix Committed" [08:12] menn0: ping [08:13] rogpeppe: pong [08:13] menn0: just looking at https://codereview.appspot.com/92560043/ [08:14] rogpeppe: really? that's from over 3 months ago [08:14] menn0: wondering if there was a particular reason for disallowing a leading digit in user names there [08:14] right [08:14] * menn0 reads [08:14] menn0: the issue was raised as to why we don't allow leading digits in user names when launchpad does allow them [08:15] menn0: and it looks like that CL was when the change was introduced [08:15] menn0: (before then, we did allow leading digits) [08:15] I'm pretty sure the regex has changed again since that PR [08:15] but I see it still doesn't allow leading digits [08:16] menn0: right [08:16] I know there was talk at one point about making juju accept whatever LP does [08:16] I think someone looked at LP's code [08:17] but clearly we don't have exact parity [08:17] menn0: clearly :-) [08:17] I don't think there's any particularly good reason but I haven't been too involved with this work [08:17] thumper and waigani are the best ppl to talk to [08:18] menn0: ok, thanks [08:18] I don't think you'll find much argument if this needs to be changed [08:18] who raised the issue? [08:55] dimitern: voidspace: TheMue: I'm probably going to miss the standup today, my son's b-day party at school got moved to the end of the school day. [08:56] jam: ok [08:56] jam: ok, won't wonder then [08:56] jam: have fun and grats to your son [08:57] jam, sure, have fun! :) [09:40] jam: TheMue: dimitern: I have a doctors appointment tomorrow morning - vaccinations for India - so I'll be a bit late starting [09:41] jam: TheMue: dimitern: back in plenty of time for standup though [09:41] voidspace: ah, ok, some vaccinations? had it once I gave talks in India too. [09:42] voidspace: but it has been a fantastic trip, I liked it. [09:42] voidspace, no worries [09:44] * TheMue remembers the fantastic food there === gsamfira1 is now known as gsamfira [10:14] voidspace: im going to india next feb [10:14] what vaccinations do I need ? [10:18] dimitern, ping? [10:19] davecheney: ah [10:19] tasdomas, hey [10:19] davecheney: heh, I'll tell you when I come back from the doctors [10:19] voidspace: right, brimming over with confidence [10:19] davecheney: they did tell me a few weeks ago what they would do to me [10:19] davecheney: but I've forgotten [10:19] davecheney: it does depend where in India you're going - I'm going to Bangalore [10:19] davecheney: where are you going? [10:19] dimitern, the bug you assigned to me - what version of juju did you experience this with? [10:20] voidspace: same, bangaloe [10:20] davecheney: I'll let you know tomorrow then :-) [10:20] It was four injections IIRC [10:21] tasdomas, sorry, which one? [10:21] go tool 7l a.7 m.7 [10:21] tasdomas, I'm going over your port ranges handover document and reassigning the bugs to myself [10:21] splendid [10:22] dimitern, ah, ok [10:22] tasdomas, the one I reassigned back to you is https://bugs.launchpad.net/juju-core/+bug/1359837 I believe, a fix for which landed with your PR #667 [10:22] Bug #1359837: expose open_port fails with no-op change

[10:22] dimitern - I was talking about the postgresql on local provider one [10:22]