| davecheney | wallyworld: axw bot is still being a prick | 01:04 |
|---|---|---|
| davecheney | did jams' fix to the data race in the api server get fixed ? | 01:04 |
| wallyworld | sadly yes | 01:04 |
| wallyworld | i think so | 01:04 |
| wallyworld | there's 2 mongo races that i think are the cause of the remianing issues | 01:04 |
| axw | I'll take a look at it this morning | 01:04 |
| axw | oh | 01:04 |
| axw | wallyworld: new ones though? | 01:05 |
| axw | because it wasn't this bad last week | 01:05 |
| wallyworld | axw: not sure, haven't looked in detail but there's been various changes this week | 01:05 |
| wallyworld | axw: could you take a look at this first up, it attempts to fix another critical blocking 1.19.4 https://github.com/juju/juju/pull/155 | 01:06 |
| axw | wallyworld: sure, looking | 01:06 |
| wallyworld | ta | 01:06 |
| davecheney | wallyworld: haven't been able to land a branch all morning | 01:08 |
| wallyworld | davecheney: so last week, we had the 2 intermittent mongo failures which only happened every so often. lots has changed this week. stuff has broken :-( | 01:10 |
| wallyworld | i do think we've got systemic race conditions in our code which no one has really got a handle on | 01:10 |
| wallyworld | and/or mongo is shit | 01:11 |
| axw | wallyworld: do you know why the peergrouper is getting an empty set of machines in the first place? | 01:14 |
| wallyworld | axw: not sure, it's all quite involved, i guessed that it could have been due to stuff initialising and replicaset stuff not being ready (likely during upgrade from non ha to ha env) and timer inside worker triggers publish of machine addresses but they aren't known yet | 01:16 |
| wallyworld | there are 2 triggers to publish - machine watcher and timer | 01:16 |
| axw | wallyworld: then I would expect it's just going from empty to empty? hmm. anyway, I'll LGTM because it doesn't make anything worse AFAICT | 01:17 |
| wallyworld | axw: yes but the watcher won't have fired | 01:18 |
| wallyworld | my analysis is in bug comment 5, i can't see an other reason for the issue :-( | 01:20 |
| davecheney | wallyworld: ok, i'll focus on finding those race conditoins | 01:27 |
| wallyworld | davecheney: we'll look as well | 01:27 |
| wallyworld | davecheney: the 2 i was aware of were one in bug 1305014 and another inside the apiserver code (not sure of bug off hand) | 01:28 |
| _mup_ | Bug #1305014: panic: Session already closed in TestManageEnviron <intermittent-failure> <test-failure> <juju-core:Triaged by rogpeppe> <https://launchpad.net/bugs/1305014> | 01:28 |
| * thumper goes to make a coffee | 01:33 | |
| axw | davecheney wallyworld: jam's fix has not landed | 02:05 |
| axw | may explain why I'm getting panics and the race detector isn't happy with cmd/jujud | 02:05 |
| wallyworld | oh, bollocks, i thought it had | 02:05 |
| davecheney | axw: what can I do to help land that today | 02:06 |
| wallyworld | axw: davecheney the fucking bug was marked as fix committed | 02:07 |
| wallyworld | hence i thought it had landed | 02:07 |
| davecheney | wallyworld: yay issue tracking | 02:07 |
| wallyworld | but github says otherwise | 02:07 |
| axw | davecheney: merge was attempted and tests failed, could just try again. rogpeppe has made added a remark about the solution being non-ideal | 02:08 |
| wallyworld | i've hit the "merge" button again | 02:08 |
| wallyworld | we can fix the non-ideal bit after it lands, we need to get this release out imo | 02:08 |
| davecheney | seconded | 02:08 |
| axw | sgtm | 02:08 |
| axw | trivial review please: https://code.launchpad.net/~axwalk/gwacl/vnet-location-docalignment/+merge/224254 | 02:10 |
| wallyworld | looking | 02:10 |
| wallyworld | axw: dumb question, how is 2012 more recent than 2013? | 02:11 |
| axw | wallyworld: "which is actually older than what was in the code." | 02:12 |
| wallyworld | yeah read that but still don't understand | 02:12 |
| axw | wallyworld: 2012-whatever is the most recent published version for that API | 02:12 |
| wallyworld | so we were using an incorrect date? | 02:12 |
| axw | wallyworld: yeah | 02:12 |
| wallyworld | ok | 02:12 |
| axw | I don't think it actually matters, but I'd rather go by the docs | 02:13 |
| wallyworld | yup | 02:13 |
| thumper | davecheney: re: https://github.com/juju/juju/pull/156 | 02:27 |
| thumper | davecheney: axw has approved but I have a few questions | 02:28 |
| thumper | davecheney: just letting you know | 02:28 |
| davecheney | thumper: sure | 02:29 |
| davecheney | thumper: i'm not trying to land anything til the race fix lands | 02:29 |
| thumper | davecheney: done with comments | 02:31 |
| davecheney | thumper: yup | 02:31 |
| davecheney | i see your comments | 02:31 |
| davecheney | i think oyu found one of my TODo's there | 02:31 |
| davecheney | fwiw, almost all this stuff was just a string slung straight to the api server | 02:31 |
| davecheney | but I will adjust the signatures to mandate the correct type | 02:32 |
| davecheney | test 0 | 02:32 |
| davecheney | ... Panic: Couldn't create temporary directory: mkdir /tmp/gocheck-3784560248718450071: file exists (PC=0x41078D) | 02:32 |
| davecheney | wowo | 02:32 |
| thumper | haha | 02:34 |
| thumper | pseudo-random | 02:34 |
| thumper | davecheney: should it be the actual type or an interface? | 02:35 |
| thumper | just wondering | 02:35 |
| davecheney | thumper: i'm thnking the concrete type | 02:35 |
| thumper | ok | 02:35 |
| davecheney | we want to say "only machine tags here" | 02:35 |
| davecheney | or units | 02:35 |
| davecheney | or whatever | 02:35 |
| * thumper nods | 02:35 | |
| thumper | This gives the compiler control rather than runtime assertion | 02:36 |
| thumper | better I think | 02:36 |
| davecheney | thumper: yup | 02:37 |
| davecheney | to fix this i'll need to change most of hte places where we have a struct like | 02:38 |
| davecheney | type Machine struct { tag names.Tag } | 02:38 |
| davecheney | to | 02:38 |
| davecheney | type Machine struct { tag names.MachineTag } | 02:38 |
| davecheney | and this is a good thing | 02:38 |
| davecheney | thumper: consider yourself nagged for that sprint | 02:42 |
| davecheney | its < 30 days | 02:42 |
| thumper | ack | 02:42 |
| === _mup__ is now known as _mup_ | ||
| menn0 | bodie_: I just finished reviewing 141 | 02:56 |
| menn0 | all good from my perspective | 02:57 |
| davecheney | wallyworld: http://juju-ci.vapour.ws:8080/job/github-merge-juju/252/console | 03:10 |
| davecheney | what ? | 03:10 |
| davecheney | tests didn't fail | 03:10 |
| davecheney | did someone nuke this build ? | 03:11 |
| wallyworld | davecheney: looks like connection to ec2 instance disappeared half way through | 03:11 |
| wallyworld | faaaark | 03:12 |
| wallyworld | retry i guess | 03:12 |
| bodie_ | menn0, good to hear :) | 03:12 |
| sinzui | wallyworld, the utopic ami expired a few hours ago. I just configured tests to use that to fix on test...maybe i need to check the ami used by git-merge-juju | 03:32 |
| wallyworld | sinzui: ah, could be it | 03:33 |
| wallyworld | although i thought we ran trusty | 03:33 |
| wallyworld | or even precise actually for the tests | 03:33 |
| wallyworld | yes, i'm sure it's precise | 03:33 |
| sinzui | wallyworld, yeah, and trusty tests are happy. I will double check | 03:33 |
| wallyworld | ok, but it's waaaay past your eod | 03:34 |
| sinzui | wallyworld, oh... | 03:35 |
| sinzui | wallyworld, I think there is a correlation between git-merge-juju and a number of hung instance from a few hours hours ago. I had to terminate them. | 03:35 |
| wallyworld | really? oh | 03:36 |
| wallyworld | maybe you terminated a git-merge-juju one as well? | 03:36 |
| sinzui | wallyworld, I have request an additional 20 instances. | 03:36 |
| wallyworld | great | 03:36 |
| sinzui | Maybe I can also update the tests to switch to hp which is very good now that we addresses the ram instance ratio | 03:37 |
| * sinzui forces a rebuild | 03:38 | |
| wallyworld | i think that couldn't do any harm | 03:38 |
| wallyworld | mgz is moving to get a nailed up hp cloud instance working to run the tests | 03:38 |
| sinzui | I can do that in 30 minutes. | 03:39 |
| sinzui | I can get slaves and slave helpers up quickly for trusty. | 03:40 |
| sinzui | I have failed to create a fat 386 to test local deploy. I am pondering the consequences of never testing that 386 local provider works. | 03:41 |
| wallyworld | i don't think too many people would be using it | 03:41 |
| sinzui | a surprising number download it from the ppa | 03:42 |
| wallyworld | once the nailed up instance is there, mgz needs to do a little script tweaking but it should be easy | 03:42 |
| wallyworld | wow, ok | 03:42 |
| wallyworld | i386 is so last century | 03:42 |
| wallyworld | decade maybe | 03:43 |
| sinzui | wallyworld, https://pastebin.canonical.com/112317/ | 03:48 |
| wallyworld | jeeez, ok | 03:48 |
| * wallyworld is surprised | 03:48 | |
| sinzui | Indeed. abentley intends to break the numbers down by week to find proper trends is adoption and decline | 03:49 |
| sinzui | your beta tester numbers are still generating | 03:50 |
| wallyworld | ok | 03:51 |
| wallyworld | i'd be interested in seeing those when done | 03:52 |
| menn0 | potentially silly question: are juju controlled machines (specifically state servers) able to SSH between each other? I'm thinking about something that would involve distributing a file to all state servers. | 03:53 |
| sinzui | menn0, They don't have the private keys | 03:56 |
| menn0 | sinzui: ok thanks. (I think you mean the public keys right?) | 03:57 |
| sinzui | menn0, they have public keys, that is why juju ssh workd | 03:57 |
| sinzui | menn0, there isn't a private key on any of the juju machines that would allow an agent to ssh to another machine | 03:58 |
| menn0 | sinzui: got it | 03:58 |
| menn0 | thanks | 03:58 |
| * menn0 should have just looked in ~/.ssh ... | 03:58 | |
| sinzui | menn0, also, manual provision can happen across networks, so the design is that the agents can all find the state server, but not necessarily the each other | 03:59 |
| sinzui | juju ci's state server cannot reach the lab, but the 4 machines I provisioned there can reach the state server. | 03:59 |
| menn0 | sinzui: ok that's good to know. | 04:00 |
| menn0 | I'm wanted to get a file generated on one state server to all other state servers | 04:00 |
| menn0 | well that's one design I'm looking at | 04:00 |
| sinzui | menn0, surely the master state-server is known to all machines. so all machines can ask for a task to pickup the file | 04:01 |
| sinzui | wallyworld, beta testers https://pastebin.canonical.com/112318/ | 04:03 |
| menn0 | sinzui: right, so you're thinking: a state server generates the file, tells the master to come and get it which it does, and then pushes it out to all other state servers. | 04:03 |
| sinzui | menn0, I think that is possible | 04:04 |
| menn0 | that could work | 04:04 |
| sinzui | menn0, My experience is from observation. I really don't know how juju works. | 04:04 |
| wallyworld | 1.11.2 was big | 04:05 |
| menn0 | sinzui: alternatively I think I can make this work by adding extending the API a little... | 04:05 |
| sinzui | on the other hand, I have done the neigh impossible because I didn't read the code http://curtis.hovey.name/2014/06/10/building-trans-cloud-environments-with-juju/ | 04:05 |
| menn0 | sinzui: :) thanks for your thoughts. you've nudged me in the right direction. | 04:07 |
| sinzui | wallyworld, It you stare at the 1.19.x numbers you can see the transition from precise users to trusty users. | 04:11 |
| * wallyworld needs another coffee before staring at numbers | 04:11 | |
| bodie_ | if there's any chance I could get a LGTM on https://github.com/juju/juju/pull/140 it would be quite helpful in moving Actions forward :) | 04:11 |
| bodie_ | (LBTM equally appreciated, though) | 04:12 |
| wallyworld | bodie_: doesn't look like william's issues with tags and canAccess function have been fixed? | 04:17 |
| waigani | do we have any helper functions for merging two slices of structs, or should I just do it by hand? | 04:30 |
| sinzui | wallyworld, Are you getting more information about bad tests from walk-unit-tests-amd64-trusty than for the other unittests? | 04:38 |
| wallyworld | sinzui: the idea was that we'd try and solve the intermittent failures. that job was useful but with the switch to running the tests on ec2 instead of canonistack/tarmac, the main landing tests have become unreliable instead :-( | 04:40 |
| wallyworld | we can pass that job until things settle down a bit | 04:40 |
| wallyworld | pause | 04:40 |
| sinzui | done | 04:40 |
| wallyworld | ok, ta | 04:40 |
| waigani | thumper: I've got a question, hangout? should be quick | 04:46 |
| thumper | sure | 04:46 |
| waigani | thumper: https://plus.google.com/hangouts/_/g4rgbccmbg5lchy6qcx7hsny6qa?hl=en | 04:46 |
| bodie_ | wallyworld, I'll have to have a look tomorrow morning, well past eod here, but I think I've addressed tags. I'm not totally certain what he was looking for with canAccess | 04:54 |
| bodie_ | for the queries from api to apiserver, everything is encapsulated by tags | 04:54 |
| wallyworld | ok | 04:54 |
| bodie_ | thanks for having a look! good night all | 04:55 |
| sinzui | wallyworld, I need to sleep. I am a little concerned that local-upgrade on precise is not passing. maybe there is something in one of these logs if CI continues to fail http://juju-ci.vapour.ws:8080/job/local-upgrade-precise-amd64/1414/ | 04:59 |
| wallyworld | ok, thanks, we'll look | 04:59 |
| davecheney | thumper: wallyworld | 05:39 |
| davecheney | [LOG] 0:02.374 DEBUG juju.environs.simplestreams fetchData failed for "http://127.0.0.1:47994/v1/1/juju-test-e76bfc6736366a8a/images/streams/v1/index.json": failed to GET object images/streams/v1/index.json from container juju-test-e76bfc6736366a8a | 05:39 |
| davecheney | caused by: Resource at http://127.0.0.1:47994/v1/1/juju-test-e76bfc6736366a8a/images/streams/v1/index.json not found | 05:39 |
| davecheney | caused by: request (http://127.0.0.1:47994/v1/1/juju-test-e76bfc6736366a8a/images/streams/v1/index.json) returned unexpected status: 404; error info: 404 Not Found | 05:40 |
| davecheney | The resource could not be found. | 05:40 |
| davecheney | an exmple of the random test failures I get on my machine | 05:40 |
| wallyworld | i'd need to see the whole log of the filed test and which test it is | 05:41 |
| === vladk|offline is now known as vladk | ||
| wallyworld | axw: if you get a chance at any point, could you look at https://github.com/juju/juju/pull/124 ? william is happy with the implementation but hasn't had time to do a more detailed review | 06:01 |
| axw | wallyworld: yep, no worries | 06:02 |
| wallyworld | ta, i have to go out for an hour or so, back later | 06:02 |
| axw | adios | 06:03 |
| vladk | dimitern: morning, I started to work on NetworkInterfacesWatcher. I am going to change NetworkInterfaceDoc according to the final draft. Is it ok? | 06:07 |
| dimitern | vladk, better not i think, because we'll need to add subnets and other things as well | 06:12 |
| dimitern | vladk, we can still watch ifaces as they are though | 06:13 |
| vladk | dimitern: so I filter interfaces by machineId, rather than _id? | 06:19 |
| dimitern | vladk, what do you mean? | 06:21 |
| dimitern | vladk, i suppose there will be a machine.WatchInterfaces() method in state, returning a StringsWatcher reporting the ids of all interfaces for that machine, as they are created, enabled, disabled, or removed | 06:22 |
| dimitern | vladk, you do need to add the IsDisabled flag to the iface doc though, but just that - for the other things that need changing, i think we should start with networks and subnets first, then the rest | 06:23 |
| vladk | dimitern: to read initial set of interfaces I need filter interfaces from the sate by machineId | 06:28 |
| dimitern | vladk, yes, the interface doc has a machineId - you can use that, right? | 06:35 |
| vladk | dimitern: yes | 06:35 |
| TheMue | jam: ping | 07:59 |
| jam | TheMue: pong | 08:07 |
| TheMue | jam: seen the latest comments on https://github.com/juju/juju/pull/150 ? | 08:08 |
| jam | I haven't, I'll go look | 08:10 |
| TheMue | jam: we have different levels to fix this issue. from low-level inside the jsonrpc package up to the command. I prefer ParseSettingsStrings in charm.Config | 08:11 |
| jam | TheMue: well, I posted that you might even go for the 'schema' package, which is the stuff that validates already | 08:12 |
| jam | however, I really don't want us to get too derailed here, this is mostly a passing-by stuff, and we thought things were working differently than they are. | 08:13 |
| jam | so for now, document that we get \ufffd when you pass in invalid binary data (part of the actual api Client level tests), so that we at least assert the current behavior and know if we change it. | 08:13 |
| jam | and I can live with whatever other results we get, as it won't be *worse* than what we've lived with forever. | 08:13 |
| jam | TheMue: does that make sense to you? | 08:14 |
| TheMue | jam: so you don’t expect cmd.Main to return something else than success? because we simply accept and document this behavior? | 08:17 |
| TheMue | jam: as the json encoding always „corrects“ the invalid sequence to \ufffd | 08:18 |
| jam | TheMue: I can live with it, given we've been living with it forever. | 08:18 |
| jam | TheMue: it isn't what I prefer, but we don't have to fix everything, we have more important things today | 08:18 |
| TheMue | jam: yes, that have been my thoughts this morning too. we only discovered this issue by thinking about it, but it never happened in reality so far | 08:19 |
| TheMue | jam: ok, then it makes sense and I’ll change the latest added tests to be a bit more explicit and documenting, so that in case of somebody having to fix this error is having a good base to start from | 08:20 |
| jam | TheMue: sounds good to me | 08:20 |
| jam | vladk: just a reminder that you're On-Call-Reviewer today | 08:21 |
| jam | and TheMue you're OCR tomorrow | 08:21 |
| TheMue | jam: tested the json marshalling yesterday evening a bit in both directions as well as taking a look into the source. interesting that they removed the error for invalid utf-8 with go 1.2 | 08:21 |
| TheMue | jam: thx for the hint | 08:22 |
| jam | TheMue: seems like a pretty significant change, but I guess that's what we have to live with. | 08:22 |
| TheMue | jam: yeah, especially as we use the websocket json codec directly, so we don’t even see the marshalled bytes. they are directly written to the socket | 08:23 |
| * jam heads to lunch | 08:27 | |
| fwereade | davecheney, ping | 08:48 |
| axw | fwereade: https://github.com/juju/juju/pull/158 - FYI | 08:55 |
| fwereade | axw, thanks | 08:55 |
| davecheney | fwereade: ack | 09:01 |
| wallyworld | mgz: standup? | 09:03 |
| fwereade | davecheney, can we have a quick chat about where you're heading with the tags stuff? there's clearly plenty to like but I don't have a clear idea of the plan | 09:09 |
| davecheney | fwereade: sadly no | 09:09 |
| davecheney | i am not at home | 09:09 |
| davecheney | fwereade: you speak to tim frequently | 09:10 |
| davecheney | he can represent me on this issue | 09:10 |
| fwereade | davecheney, cool, thanks, I'll chat to him this evening | 09:10 |
| wwitzel3 | morning all | 09:41 |
| TheMue | wwitzel3: morning | 09:50 |
| perrito666 | morning | 09:59 |
| TheMue | vladk, jam: (hopefully) final changes https://github.com/juju/juju/pull/150. so please review (again) | 10:19 |
| wallyworld__ | jam: not sure if my "fix" for that agent.conf issue will do any good or not but it's all i could come up with as something to try from reading the code | 10:36 |
| dimitern | jam, standup? | 10:46 |
| === wallyworld__ is now known as wallyworld | ||
| === jcsackett is now known as idioteque | ||
| === idioteque is now known as foobarbazqux | ||
| * fwereade bbiab | 12:12 | |
| perrito666 | stumble upon's logo reminds me of something | 12:33 |
| sinzui | Looks like wallyworld's fix got upgrade broken local precise upgrades | 12:59 |
| * sinzui reports bug | 12:59 | |
| === alexisb_bbl is now known as alexisb | ||
| sinzui | natefinch, jam, fwereade, alexisb : We have a regression cause by the fix for upgrade bug reported yesterday https://bugs.launchpad.net/juju-core/+bug/1334273 | 13:08 |
| _mup_ | Bug #1334273: Upgrades of precise localhost fail <local-provider> <precise> <regression> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1334273> | 13:08 |
| alexisb | sinzui, lovely | 13:08 |
| alexisb | perrito666, wwitzel3, mgz ^^^ you guys available to take a look at this bug | 13:10 |
| bodie_ | morning all | 13:15 |
| natefinch | wwitzel3: can we push our 1 on 1 to this afternoon? | 13:15 |
| mgz | alexisb: nothing is jumping out at me from the change or the log in the bug... | 13:15 |
| === anthonyf` is now known as anthonyf | ||
| alexisb | mgz, well of course it couldn't be obvious that would be waaaay to easy ;) | 13:19 |
| wwitzel3 | natefinch: sure | 13:19 |
| sinzui | mgz, The stall in the test log. where a single status call takes 10 minutes, then fails the test looks like the issue joyent upgrades had. Wallyworld said the fix was to ensure the state server got a sane list of addresses | 13:20 |
| mgz | sinzui: so, all the change does is return an error if the publish function is given an empty list | 13:22 |
| mgz | I don't particularly see how that regresses us... it must be something fun and side-effecty | 13:22 |
| sinzui | mgz #136 from wallyworld/ignore-empty-machine-addresses | 13:23 |
| sinzui | mgz, I am speculating...I really don't know | 13:23 |
| alexisb | sinzui, mgz : does that mean we think that this current bug is a result of joyent shotcomings? | 13:23 |
| sinzui | alexisb, no. I am just remarking that the fix for yesterdays critical broke another test in a similar way to a bug from last week | 13:24 |
| sinzui | alexisb, CI loves juju yesterday. I could release something from yesterday as 1.19.4 | 13:25 |
| mgz | sinzui: I don't see that error propogated back up in the logs anywhere either | 13:25 |
| perrito666 | sinzui: github says https://github.com/juju/juju/commit/e01ac93e is a different commit | 13:25 |
| mgz | which it *should* be if that change actually affected anything | 13:25 |
| alexisb | sinzui, ok | 13:26 |
| sinzui | perrito666, I give you the latest data, there have been two broken revisions, I just to give you a log from what fails 15 minutes ago | 13:28 |
| perrito666 | sinzui: I was talking about "With the introduction of e01ac93e "Merge pull request #155 from wallyworld/peergrouper-publish-ignore-empty"" | 13:28 |
| sinzui | perrito666, http://juju-ci.vapour.ws:8080/job/local-upgrade-precise-amd64/ shows all the failed tests over the last 9 hours | 13:29 |
| sinzui | oops | 13:29 |
| perrito666 | sinzui: as I say, the first failure comes from https://github.com/juju/juju/commit/e01ac93e0cbee312bf19ead78035a78ed2428871 | 13:30 |
| sinzui | #153 from davecheney/112-state-life-takes-a-tag is the one | 13:30 |
| perrito666 | which is not wallyworld's tag | 13:30 |
| perrito666 | s/tag/rev | 13:30 |
| sinzui | I updated the bug | 13:31 |
| perrito666 | mgz: you might find more answers there ^ | 13:31 |
| mgz | that does seem a little more promising | 13:31 |
| alexisb | mgz, is there an obvious fix or is a commit we should consider backing out for this current dev release? | 13:32 |
| mgz | I think we should consider backing out the change and seeing if that resolves the issue | 13:33 |
| mgz | this might mean pain for upgrades later, this is an api change | 13:35 |
| mgz | of sorts | 13:36 |
| alexisb | mgz, that seems very reasonable to me, sinzui your thoughts? | 13:36 |
| sinzui | mgz...well, I have the tarball and installer from the previous pass. I can start the release now with fa4f6106 | 13:38 |
| sinzui | mgz, are you preparing to backout the rev to test? | 13:49 |
| bodie_ | anyone happen to know whether fwereade is around? | 13:51 |
| natefinch | alexisb: I presume I stick around with Gabriel and skip tosca this week? | 14:01 |
| alexisb | natefinch, yes | 14:02 |
| === foobarbazqux is now known as jcsackett | ||
| === vds` is now known as vds | ||
| sinzui | mgz, perrito666 Is anyone working on bug 1334273 or planning to backout the revision where the error started? | 14:45 |
| _mup_ | Bug #1334273: Upgrades of precise localhost fail <local-provider> <precise> <regression> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1334273> | 14:45 |
| perrito666 | I am not, I was under the impression mgz was on it | 14:46 |
| mgz | sinzui: I can do the backout if we want that | 14:47 |
| mgz | I'm not completely sure it'll resolve the issue, but seems worth trying | 14:47 |
| sinzui | mgz we can do that or I release the version before it. | 14:47 |
| sinzui | mgz, if the test passes after the backing out, then we know the rev is bad, that is all we learn | 14:48 |
| mgz | sinzui: I guess the question is, do we want the wallyworld change that comes after | 14:48 |
| mgz | because *that* bug was the blocking cause previously right? (the thing wallyworld's change tries to fix, panic on not having addresses) | 14:49 |
| sinzui | mgz, yes, if it is independent. wallyworld's change was a hope to fix the other critical | 14:49 |
| mgz | okay, I'll do the revert, we send tip with that through ci, and release that if it's clean, otherwise release the rev before | 14:50 |
| mgz | sounds good? | 14:50 |
| alexisb | mgz, yes sounds good :) | 14:51 |
| sinzui | mgz, alexisb. I have a lot of anxiety regarding the delay of the devel release. We have more than 50 issues people aren't testing beause we always find one more critical bug. I am inclined to revert to my old behaviour of selecting a blessed revision from the past. In general, a bug that is in older versions of juju does not block a devel release, only recent regressions can block | 14:52 |
| perrito666 | +1 | 14:53 |
| alexisb | sinzui, +1 | 14:54 |
| mgz | can I get a check/stamp on pr160 please | 14:59 |
| mgz | perrito666: ^ | 14:59 |
| perrito666 | natefinch: checking | 15:00 |
| mgz | >_< | 15:00 |
| perrito666 | aghh sorry, off by one in the kb | 15:02 |
| perrito666 | mgz: check | 15:02 |
| mgz | okay, sent to bot | 15:02 |
| wwitzel3 | perrito666, natefinch: standup or are you guys busy with the regression? | 15:02 |
| perrito666 | wwitzel3: coming | 15:03 |
| perrito666 | natefinch: ? | 15:10 |
| natefinch | wwitzel3, perrito666: let's meet this afternoon, sorry, in a virtual sprint with the cloudbase guys until 1-ish | 15:10 |
| perrito666 | cool, I can then use my bw to continue downloading broken sword :p | 15:11 |
| mgz | sinzui: revert has landed | 15:23 |
| mgz | sinzui: the upgrade job is blue, but from before the revert landed anyway. | 15:33 |
| sinzui | ? | 15:35 |
| mgz | if I'm looking at the jenkins job correctly, the latest run is blue, but it's from an earlier landing anyway | 15:35 |
| alexisb | what does blue mean? | 15:36 |
| mgz | er, it passed. blue dot in the jenkins ui. | 15:36 |
| sinzui | damn it | 15:36 |
| sinzui | mgz, a race condition where upgrade won? There were a lot of failures and the rate is much worse than it was before the rev | 15:38 |
| sinzui | mgz, if the job passes quickly in the next run, then I am convinced that a race condition was avoided | 15:38 |
| mgz | yeah, makes seeing particular blame on the change much harder | 15:39 |
| mgz | we have a few more trunk revs to see | 15:39 |
| bodie_ | does anyone know how to retrieve the Unit making the query in an APIserver endpoint? | 15:49 |
| bodie_ | I was passing the calling Unit's tag along with the entity request, but I was told that the UniterAPI has the calling Unit as a member variable | 15:50 |
| === tvansteenburgh1 is now known as tvansteenburgh | ||
| alexisb | jam1, did you want to meet? | 16:43 |
| jam1 | alexisb: yeah, I think so | 16:43 |
| alexisb | ok, our normal hangout | 16:43 |
| wwitzel3 | natefinch, perrito666, ericsnow: is now good? | 17:00 |
| ericsnow | wwitzel3: good for me | 17:01 |
| perrito666 | wwitzel3: coming | 17:01 |
| natefinch | wwitzel3: can we do it in an hour? | 17:02 |
| perrito666 | lol | 17:02 |
| perrito666 | not coming | 17:02 |
| perrito666 | ping me when you are ready | 17:05 |
| * perrito666 cooks | 17:05 | |
| mgz | sinzui: I've not seen any more runs on the local upgrade job, | 17:14 |
| mgz | but I've proposed a restore of dave's changes as it doesn't seem to have had an effect | 17:15 |
| sinzui | mgz, It runs in a later phase | 17:15 |
| sinzui | mgz, I was just looking if I could force the test to run earlier | 17:15 |
| sinzui | mgz, were you planning to create a dedicate HP test machine? | 17:16 |
| mgz | sinzui: yup, though not completely sure about the hp part | 17:17 |
| sinzui | mgz, HP is good now that we fixed the RAM to instances inbalance | 17:17 |
| sinzui | mgz, I can provide a slave for you before you wake tomorrow | 17:17 |
| mgz | do I need a la | 17:18 |
| mgz | *slave, the unit test script just takes any machine right? | 17:18 |
| sinzui | mgz, I have been pondering an update to the run-unit* to provision nova instances | 17:19 |
| mgz | that seems reasonable | 17:19 |
| sinzui | A slave in our cloudy case means some jobs jump the queue. That helps you | 17:19 |
| mgz | okay | 17:21 |
| sinzui | We could choose to run tests on the slave machine. I am going to try that for some tests. I really want to minimise our aws resources...then I thought changing the run* tests to use nova would help me | 17:21 |
| sinzui | oh | 17:21 |
| sinzui | mgz, your tests try once fast, then try with -p 2. Should I be doing the same? | 17:22 |
| sinzui | I try test 4 times, maybe I could your your technique to minimose setups | 17:22 |
| mgz | sinzui: , ideally not, but I was considering proposing a change to the script to take a flag that does that | 17:22 |
| bac | i'm getting this error from jenkin-slave even though the file listed exists and is world-readable: 2014-06-25 17:39:24 ERROR juju runner.go:220 worker: exited "upgrader": cannot read tools metadata in tools directory: open /var/lib/juju/tools/1.17.4.1-trusty-amd64/downloaded-tools.txt: no such file or directory | 17:47 |
| bac | any ideas? | 17:47 |
| natefinch | perrito666, ericsnow: standup? | 18:03 |
| perrito666 | natefinch: sure | 18:03 |
| ericsnow | natefinch: coming | 18:04 |
| sinzui | I wont be helping a test job to run sooner. My effort lead to a collision in lxc/mongod/jujud on the jenkins master. That 60 minute delay was not faster than letting tests run as they naturally want to | 19:12 |
| wwitzel3 | all the contents of the turkey wrap I was eating for lunch blew out the bottom and the cat got scared, ran through it, and I just finsihed wiping up all the mustard cat prints in my house. | 19:39 |
| ericsnow | wwitzel3: just imagining that gave me a chuckle :) (hopefully it wasn't too much work) | 19:42 |
| ericsnow | wwitzel3: I've had the same thing happen with small children | 19:42 |
| wwitzel3 | ericsnow: thankfully, wood floors | 19:46 |
| thumper | fwereade: around? | 20:08 |
| thumper | natefinch: o/ | 20:08 |
| natefinch | thumper: | 20:08 |
| natefinch | howdy | 20:08 |
| perrito666 | ericsnow: do you eat small children? :p | 20:10 |
| ericsnow | perrito666: only when they misbehave ;) | 20:10 |
| thumper | natefinch: got time for a quick hangout? | 20:12 |
| natefinch | yeah | 20:13 |
| thumper | natefinch: https://plus.google.com/hangouts/_/g44xyud6z5kz3g6p775ebv6cfya?hl=en | 20:13 |
| jcw4 | I'd appreciate a review on https://github.com/juju/names/pull/11 | 20:36 |
| jcw4 | Its an ActionTag refactor to hide the implementation details of extracting a UnitTag from an ActionTag | 20:36 |
| jcw4 | thumper: I think it's Thursday in your timezone >:-} ^^ | 20:38 |
| jcw4 | Of course it's still Wednesday for mgz ;) | 20:40 |
| thumper | hi jcw4 | 21:02 |
| thumper | yes, thursday here | 21:02 |
| jcw4 | hi thumper :) | 21:02 |
| thumper | just after 9am | 21:02 |
| perrito666 | jcw4: there you go, you might want to look for better reviewers than me nevertheles | 21:02 |
| jcw4 | perrito666: your review comments are great | 21:02 |
| thumper | I'll take a quick look too if you like | 21:02 |
| jcw4 | thumper: excellent, thank you yes | 21:03 |
| jcw4 | thumper: re: exported attributes... I wasn't clear if I could access the internal attributes outside of a method on the internalStruct | 21:07 |
| jcw4 | I'll change it and make sure it still works | 21:08 |
| thumper | jcw4: all internal attributes are accessable inside the same package | 21:08 |
| thumper | as long as it is only used inside the names package, you should be fine | 21:08 |
| jcw4 | thumper: perfect | 21:08 |
| thumper | hmm... | 21:10 |
| * thumper writes a longer general comment. | 21:11 | |
| * bodie_ grabs his binoculars and tries to catch a peek | 21:13 | |
| * jcw4 counts the minutes, wondering what could cause such long deliberation... | 21:15 | |
| * jcw4 is scared now | 21:17 | |
| fwereade | thumper, hey, I thought there was some meeting in 40 mins? don't seem to be invited any more | 21:18 |
| thumper | hmm... this is getting longer | 21:18 |
| thumper | fwereade: yes it is still on | 21:18 |
| fwereade | thumper, since I seem to be up, I might swing by, it looked important iirc | 21:18 |
| fwereade | thumper, what was it? | 21:19 |
| thumper | docker stuff | 21:19 |
| thumper | the meeting that is | 21:19 |
| thumper | I'd like to chat with you about identity and multi-env stuff | 21:19 |
| thumper | but reviewing jcw4's branch just now, | 21:19 |
| thumper | with you shortly | 21:19 |
| fwereade | thumper, and I with you, I might be back in a bit or I might do it after that meeting if you'll be free? | 21:20 |
| thumper | fwereade: standup after the other emeting | 21:20 |
| thumper | fwereade: how about pre other meeting, in 20 minutes say? | 21:20 |
| fwereade | thumper, ok, sgtm, see you soon | 21:20 |
| thumper | ack | 21:20 |
| thumper | jcw4, bodie_: there you go | 21:25 |
| thumper | damn the markdown squishing spaces | 21:25 |
| jcw4 | thumper: thanks! | 21:25 |
| thumper | I should work out how to write code inside comments | 21:25 |
| jcw4 | thumper: excellent suggestion, thanks | 21:26 |
| thumper | np | 21:26 |
| jcw4 | thumper: I considered it, but thought it might be too radical | 21:27 |
| jcw4 | :) | 21:27 |
| thumper | nah | 21:27 |
| jcw4 | with your blessing though... | 21:27 |
| jcw4 | :) | 21:27 |
| thumper | with the upcoming identity work, I'm going to do something similar | 21:27 |
| thumper | the identity tag will have two parts: username@provider | 21:27 |
| thumper | so I had thought through all this already :) | 21:27 |
| jcw4 | :) | 21:28 |
| sinzui | hi wallyworld thumper: I am creating a slave dedicated to running git-merge-juju. If you are contemplating running the unit tests on the same machine, perhaps in lxc, I can give it extra cpus to do that | 21:28 |
| sinzui | otherwise I will use a medium instance and let the job create other instances to run the tests | 21:29 |
| perrito666 | thumper: ``` | 21:29 |
| thumper | sinzui: um... not sure I have enough context here, do we need to run tests on it? | 21:29 |
| thumper | perrito666: yes? | 21:30 |
| perrito666 | thumper: ```code``` | 21:30 |
| thumper | perrito666: oh.. cool | 21:30 |
| thumper | ta | 21:30 |
| sinzui | thumper, the job currently creates a large ec2 instance to run tests. I was pondering support for hp instances, but maybe the core devs are thinking about running the unit tests in an lxc | 21:30 |
| thumper | I've not thought about that, but others may have :) | 21:31 |
| bodie_ | I usually use ```go \n $code \n ``` | 21:35 |
| bodie_ | you get highlighting by language | 21:35 |
| bodie_ | which is lovely | 21:35 |
| perrito666 | bodie_: good hint | 21:36 |
| bodie_ | perrito666, the other way is still good for inline monospaced code, though | 21:37 |
| jcw4 | bodie_, perrito666 (thumper); for inline monospace you just need `blah blah` <-- only one tick | 21:41 |
| thumper | ta | 21:42 |
| perrito666 | ok ppl EOD, ill still hang around bc I have no life | 21:43 |
| perrito666 | but I might not answer a lot since I just purchased the broken sword collection and I intend to play it :p | 21:43 |
| bodie_ | ooo | 21:47 |
| bodie_ | ... to the single backtick ;) though I have been known to have no life as well | 21:48 |
| === vladk is now known as vladk|offline | ||
| jcw4 | bodie_: haha, I saw your ooo and thought you were talking about perrito666's broken sword collection | 21:54 |
| menn0 | morning all | 22:03 |
| waigani | thanks menn0 | 22:06 |
| menn0 | waigani: np | 22:06 |
| menn0 | it was top of my inbox and was straightforward :) | 22:06 |
| waigani | menn0: nice, all my PRs should aim for that! | 22:07 |
| wallyworld | sinzui: we are contemplating running unit tests in a container so extra cpus would be good | 22:12 |
| sinzui | :( I just completed my setup of a 4 cpu machine | 22:12 |
| wallyworld | that's ok | 22:13 |
| wallyworld | we'll see how it goes | 22:13 |
| wallyworld | it probably won't be something we can get done immediately | 22:13 |
| sinzui | wallyworld, I can remake it. if my you are unhappy with the hp/nova provisioning I am adding to tests | 22:14 |
| wallyworld | sinzui: ok, np. my view is that the immediate goal is to get the landing tests running in a nailed up instance of some sort | 22:15 |
| sinzui | oh? | 22:16 |
| wallyworld | containers would simplify isolation, but initially we can just do what we did in tarmac | 22:16 |
| sinzui | wallyworld, I will restart this then. you and mgz will still have a dedicated slave in an hour | 22:16 |
| wallyworld | sinzui: did we need a quick chat? seems like we are not quite aligned? | 22:17 |
| wallyworld | although i have a meeting in 10 minutes | 22:17 |
| sinzui | wallyworld, for the slave or f*ing bug 1334273 | 22:18 |
| _mup_ | Bug #1334273: Upgrades of precise localhost fail <local-provider> <precise> <regression> <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1334273> | 22:18 |
| bodie_ | mornin' | 22:20 |
| wallyworld | sinzui: either or both. i'll ping you in an hour if you are still around. i'll also look into that bug. i was hoping it was ok when it started passing again yesterday :-( | 22:21 |
| sinzui | wallyworld, mgz reverted Dave's commit and CI is playing it now. It has been a sad day. I will *not* try to make the tests run faster because I last effort cause to lxc-based tests to collide | 22:25 |
| jcw4 | thumper, perrito666 I pushed an updated branch for https://github.com/juju/names/pull/11 | 22:26 |
| jcw4 | thumper: if you have the time and inclination we can discuss my decision to make func ParseActionId(string) (ActionTag, bool) {} exported | 22:26 |
| wallyworld | sinzui: ok, i'd still like to still ping you for a chat if you are free and around in an hour | 22:27 |
| thumper | ok | 22:27 |
| sinzui | wallyworld, I have no social life | 22:27 |
| wallyworld | i hear you | 22:28 |
| * jcw4 takes a mid afternoon break | 22:28 | |
| wallyworld | sinzui: if you are free https://plus.google.com/hangouts/_/gyul7vioiw2m7xnhi2tkfvpsjya | 22:41 |
| sinzui | wallyworld, Google hates me | 22:43 |
| * sinzui tries to hack the url | 22:43 | |
| wallyworld | i can send an invite | 22:44 |
| sinzui | wallyworld, send me an invite, G is the suck | 22:45 |
| wallyworld | already sent | 22:45 |
| sinzui | hmm, why is it ringing | 22:46 |
| wallyworld | sinzui: should i try another hangout? | 22:48 |
| sinzui | yes, It keeps ringing and you don't hear it | 22:48 |
| thumper | jcw4: looking now | 22:48 |
| wallyworld | sinzui: https://plus.google.com/hangouts/_/gzdeubxezekdktt42jeva7ui5ma | 22:49 |
| sinzui | party's over | 22:49 |
| thumper | awww... | 22:49 |
| thumper | no party for sinzui | 22:49 |
| thumper | jcw4: why have ParseActionId public? what is the rationale? | 22:51 |
| jcw4 | thumper: in juju/state/action.go the Action type returns a names.Tag | 22:58 |
| jcw4 | thumper: the Action uses ParseActionId to get that tag | 22:58 |
| jcw4 | thumper: the Action doesn't have a reference to a Unit or a UnitTag | 22:59 |
| thumper | um... no | 22:59 |
| thumper | jcw4: where in state/action.go? | 22:59 |
| jcw4 | line 84 now... | 22:59 |
| jcw4 | but that code isn't pushed yet | 22:59 |
| thumper | if it is returning a names.Tag | 23:00 |
| thumper | you should use a type assertion, not more parsing | 23:00 |
| thumper | action, ok := (names.ActionTag)(tag) | 23:00 |
| * thumper thinks | 23:00 | |
| thumper | that's the right syntax isn't it? | 23:01 |
| jcw4 | thumper: it doesn't have a tag, it's just part of the Entity interface to return one | 23:01 |
| jcw4 | action, ok := tag.(names.ActionTag) | 23:01 |
| thumper | ok.. that | 23:01 |
| thumper | the action object though should have a unit and a sequence, yes? | 23:01 |
| jcw4 | thumper: yes...ish | 23:02 |
| jcw4 | the unit is encoded in the action Id | 23:02 |
| thumper | so you return names.NewActionTag(unit, sequence) | 23:02 |
| jcw4 | thumper: so we | 23:02 |
| thumper | ah... | 23:02 |
| thumper | really? | 23:02 |
| jcw4 | thumper: it's partially due to watchers | 23:03 |
| * thumper takes a deep breath | 23:03 | |
| jcw4 | we want to watch the actions collection for actions specific to a unit | 23:03 |
| thumper | right, so the document should have a unit attribute | 23:03 |
| thumper | and not encoded in some id field | 23:03 |
| thumper | it may well be encoded in id *as well* | 23:03 |
| jcw4 | thumper: +1 | 23:03 |
| thumper | but it shouldn't be the only source | 23:03 |
| jcw4 | thumper: that's how it is now but I agree | 23:04 |
| * thumper nods | 23:04 | |
| * thumper realised he was late for the standup | 23:04 | |
| jcw4 | thumper: thanks~ | 23:04 |
| jcw4 | thumper: at your convenience; you'd prefer an actual unitName on the actionsDoc, over a method that parses it out of the _id? | 23:08 |
| bodie_ | jcw4, thumper -- it appears the UnitTag uses the parsing method as follows -- https://github.com/juju/names/blob/master/unit.go#L53-L63 | 23:19 |
| bodie_ | fwiw | 23:19 |
| thumper | jcw4: ack | 23:19 |
| bodie_ | we have a working ActionEvents filter channel :) | 23:24 |
| bodie_ | going to attempt to push down to the Hook level tomorrow | 23:24 |
| bodie_ | https://github.com/juju/juju/pull/163 | 23:26 |
| bodie_ | any review would be very welcome :) | 23:26 |
| jcw4 | thumper: and I've updated https://github.com/juju/names/pull/11 | 23:28 |
| thumper | bodie_: I'm a reviewer today, and will try to get to it :) | 23:33 |
| jcw4 | thumper: given that perrito666 also reviewed, am I clear to merge? | 23:39 |
| thumper | jcw4: yup | 23:39 |
| jcw4 | thumper: ta | 23:39 |
| jcw4 | thumper: et. al. do you use checklists when doing code reviews? That was one of the suggestions in the article jam1 linked to on the email list | 23:45 |
| thumper | bodie_: it seems I'm commenting on other parts of the work in the wrong PR | 23:46 |
| thumper | jcw4: I have a mental checklist, but nothing formal at this stage | 23:46 |
| jcw4 | maybe we could compile the checklist items into a doc/code-review.md or something | 23:46 |
| thumper | gym time... back later | 23:47 |
| jcw4 | have fun thumper | 23:47 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!