| menn0 | thumper, axw or wallyworld: review pls http://reviews.vapour.ws/r/3073/ | 00:10 |
|---|---|---|
| menn0 | this makes a feature test i'm writing a lot cleaner | 00:11 |
| thumper | menn0: done | 00:13 |
| * thumper goes to walk the dog in the sun | 00:13 | |
| menn0 | thumper: thanks | 00:13 |
| davecheney | OH WOW | 00:40 |
| davecheney | the maas provider demands yaml.v2 | 00:40 |
| davecheney | but also calls a utilty function with requires yaml.v1 | 00:41 |
| davecheney | mgz_: thumper http://reviews.vapour.ws/r/3074/ | 01:35 |
| davecheney | please view | 01:35 |
| davecheney | should break the arguments about which version of yaml.v2 juju/utils can use | 01:35 |
| davecheney | also, spot the WTF in that pach | 01:36 |
| davecheney | also, spot the WTF in that patch | 01:36 |
| === natefinch-afk is now known as natefinch | ||
| natefinch | davecheney: heh, looks familiar: https://github.com/natefinch/atomic/blob/master/atomic.go#L17 | 01:56 |
| natefinch | davecheney: looks like I should steal some safeguards from yours, though. | 01:57 |
| perrito666 | mm, the chmod wont work in windows, but since it is not tested it wont break anything either | 01:58 |
| natefinch | perrito666: chmod is just a noop on winows | 01:58 |
| perrito666 | natefinch: not entirely | 01:58 |
| perrito666 | there is an implementation in go that does something stupid | 01:58 |
| perrito666 | it changes fs properties for the file but these are ignored by windows | 01:59 |
| natefinch | effectively a noop :) | 01:59 |
| perrito666 | natefinch: well the fact that those things can be changed means that something is using them | 02:00 |
| perrito666 | I have no clue what that is | 02:00 |
| perrito666 | most likely windows 3 :p | 02:00 |
| thumper | menn0, davecheney, axw: team meeting | 02:02 |
| natefinch | davecheney: seems like this must be a bug: http://reviews.vapour.ws/r/3074/#comment19133 | 02:07 |
| davecheney | coming | 02:33 |
| davecheney | oh | 02:33 |
| davecheney | i'm 33 mins late | 02:33 |
| davecheney | natefinch: thanks for your comments | 02:36 |
| davecheney | PTAL | 02:37 |
| natefinch | davecheney: we're still talking on the hangout btw | 02:37 |
| davecheney | ok, coming | 02:37 |
| natefinch | davecheney: tests? | 02:49 |
| natefinch | thumper: Looking at that retry package, seems like it would be useful to be able to encapsulate call args into a value, and then be able to call value.Call(somefunc) .... so you could use the same retry semantics with any number of different functions. Also it then doesn't hide the function you're retrying inside a huge list of args. | 02:52 |
| natefinch | thumper: also you should call the package sisyphus | 02:53 |
| davecheney | natefinch: this is only temporary | 02:56 |
| davecheney | once everyone is at yaml.v2 i'll be deleting those forks | 02:56 |
| davecheney | my goal is not to move code from other packages into juju | 02:56 |
| davecheney | in fact the opposite | 02:56 |
| davecheney | but this yaml.v2 dep keeps fucking that plan up | 02:56 |
| cherylj | what's the difference between stateaddresses and apiaddresses in agent.conf? | 03:08 |
| cherylj | thumper, menn0 ^^ ? | 03:10 |
| menn0 | cherylj: iirc stateaddresses has the mongodb server addresses and apiaddresses is the API server addresses | 03:11 |
| * menn0 checks something | 03:12 | |
| thumper | natefinch: interesting idea | 03:13 |
| menn0 | cherylj: yep, that's right | 03:13 |
| thumper | natefinch: and pretty easy to implement | 03:13 |
| menn0 | cherylj: really stateaddresses shouldn't be there except on the state/controller servers, but it is | 03:14 |
| cherylj | ok, thanks, menn0 | 03:14 |
| menn0 | cherylj: it's probably a historical vestiage - non state agents used to connect directly to mongo for some things | 03:14 |
| thumper | menn0: could I get you to cast your eyes over http://reviews.vapour.ws/r/3072/ ? | 03:15 |
| thumper | cherylj: menn0 is right in his musings :) | 03:15 |
| menn0 | cherylj: related tidbit: a lot of (probably older) code uses "state" to mean "the mongodb server". not at all confusing. | 03:18 |
| menn0 | thumper: looking | 03:18 |
| cherylj | ha, ok :) | 03:18 |
| natefinch | thumper: also, if you're going to make retry a standalone package, you gotta move clock into a standalone package | 03:25 |
| thumper | natefinch: axw is moving the clock out in his one | 03:26 |
| natefinch | thumper: awesome | 03:26 |
| thumper | natefinch, axw: although perhaps we should have a common top level one... | 03:26 |
| thumper | axw: perhaps juju/clock ... | 03:26 |
| axw | natefinch: FYI, this is the branch talked about on the call: https://github.com/axw/juju-time | 03:26 |
| axw | thumper: I was thinking juju/time/clock, but *shrug* | 03:26 |
| natefinch | thumper: yeah. thats what I meant, just juju/clock | 03:27 |
| thumper | axw: I was thinking more to have a common parent for the scheduler one too, rather that the clock with the scheduler | 03:27 |
| axw | thumper: yes, definitely not in the one package | 03:28 |
| axw | we have enough utils already :) | 03:28 |
| thumper | I was thinking not in the same repo | 03:28 |
| thumper | agreed | 03:28 |
| natefinch | thumper: btw, I think the error handling in your retry code could be improved by following davecheney's advice to assert behavior: http://dave.cheney.net/2014/12/24/inspecting-errors .. so like have a 'Last() error' method on the error types, etc. | 03:34 |
| * thumper looks | 03:35 | |
| menn0 | thumper: well that sucked. review done. lots of stuff missed. | 03:35 |
| thumper | menn0: :( | 03:35 |
| menn0 | thumper: the review process sucked... not the PR | 03:35 |
| thumper | menn0: it isn't complete... | 03:35 |
| thumper | geez | 03:35 |
| thumper | there is another one following... | 03:35 |
| thumper | I think | 03:35 |
| thumper | at least | 03:35 |
| * thumper looks | 03:35 | |
| menn0 | thumper: there's lots of test names that still say System and bits of help text missed and a few other things | 03:36 |
| thumper | ok | 03:36 |
| menn0 | thumper: I basically just did a search in my browser for "system" looked to see if it came up on the right side of the diff :) | 03:37 |
| menn0 | thumper: also "server" comes up a bit | 03:37 |
| * thumper sighs | 03:37 | |
| menn0 | not sure if you're fixing those | 03:37 |
| thumper | so much stuff to change | 03:37 |
| natefinch | I can't believe we're spending so much time just to change the names of things, instead of, like, implementing features. | 03:42 |
| natefinch | oh yeah, hey, I had a 1.26-alpha1.1 server running, and tried to interact with it via a client built from master, and was getting this error message printing out with a lot of my commands: 2015/11/04 21:27:00 warning: discarding cookies in invalid format (error: json: cannot unmarshal object into Go value of type []cookiejar.entry) | 03:44 |
| natefinch | davecheney: what's wrong with this picture? https://github.com/juju/persistent-cookiejar/blob/master/jar.go#L10 | 03:47 |
| davecheney | natefinch: the type isn't public ? | 03:50 |
| natefinch | davecheney: repo name is "persistent-cookiejar" :/ | 03:50 |
| natefinch | package name is "cookiejar | 03:50 |
| davecheney | oh for fucks sake | 03:50 |
| * davecheney throws something | 03:51 | |
| natefinch | also, using that package seems to break client-server compatibility | 03:51 |
| natefinch | I saw this: | 03:51 |
| natefinch | $ juju destroy-environment local -y | 03:51 |
| natefinch | ERROR cannot connect to API: cannot load cookies: json: cannot unmarshal object into Go value of type []cookiejar.entry | 03:51 |
| natefinch | whewn my client was a different version than my server... presumably one of them was using the old cookiejar and one the new | 03:52 |
| * davecheney bursts into tears | 03:52 | |
| davecheney | natefinch: i'm convencined we've passed more than 100% technical debt to gdp | 03:53 |
| natefinch | davecheney: time to rewrite juju in rust | 03:53 |
| davecheney | that wasn't the exact point I was trying to make ... | 03:55 |
| natefinch | davecheney: well, we're working on tech debt at oakland, right? One week should just about do it. | 03:57 |
| davecheney | i'll be late | 04:05 |
| * natefinch calls it a night to go read his new Go programming book | 04:07 | |
| === akhavr1 is now known as akhavr | ||
| thumper | davecheney: https://github.com/howbazaar/clock-proposed | 04:28 |
| thumper | davecheney: I know you like non-util named packages :) | 04:28 |
| thumper | davecheney: to become github.com/juju/clock | 04:28 |
| thumper | axw: ^^ | 04:28 |
| thumper | added in the testing clock from juju/juju/testing/clock.go | 04:28 |
| thumper | and added a few tests | 04:29 |
| thumper | and type assertions in the test file | 04:29 |
| axw | thumper: LGTM, but I'd like it if you renamed clock/testing to clock/clocktest | 04:32 |
| axw | thumper: saves aliasing everywhere | 04:33 |
| thumper | clock/testclock? | 04:33 |
| axw | thumper: I was thinking clocktest as in testing things related to the clock package, not as in a package that contains a TestClock | 04:33 |
| thumper | I agree with renaming it | 04:33 |
| axw | thumper: much like net/http/httptest | 04:34 |
| thumper | ah.. ok | 04:34 |
| thumper | happy to follow precident | 04:34 |
| axw | thumper: still not really sure about having a whole repo to its own. we're goingto want other time-related things which would be nice to group together | 04:36 |
| axw | thumper: e.g. delay functions for retry/scheduler/backoff thing that bogdan is working | 04:36 |
| thumper | axw: let's ask fwereade, given tech-lead hat :) | 04:37 |
| axw | thumper: I'm thinking they'd live in juju/time/delays or something like that... and then having clock over by itself feels a bit odd | 04:37 |
| axw | thumper: SGTM | 04:37 |
| davecheney | \o/ all praise the clock | 04:43 |
| davecheney | 15:32 < axw> thumper: LGTM, but I'd like it if you renamed clock/testing to clock/clocktest | 04:43 |
| davecheney | ^ yes, a million times, yes | 04:43 |
| thumper | I'm just asking our architect and TLs about guidance around separate repo for clock, or one for time that includes the scheduler that axw has | 04:44 |
| thumper | davecheney: the clock-proposed repo now has that package renamed | 04:44 |
| davecheney | thumper: also if it's going to be a juju project, it needs a cute name | 04:44 |
| davecheney | what about clocky ? https://upload.wikimedia.org/wikipedia/commons/4/4b/Clocky_almond_panorama1680.jpg | 04:44 |
| thumper | :) | 04:45 |
| thumper | axw: got the link to your time proposed repo? | 04:46 |
| thumper | axw: I'll include it in the email | 04:46 |
| axw | thumper: https://github.com/axw/juju-time | 04:46 |
| thumper | ta | 04:46 |
| menn0 | thumper: here's juju-dumplogs. I'm just doing the /usr/local/bin symlink now. | 04:47 |
| menn0 | http://reviews.vapour.ws/r/3075/ | 04:47 |
| thumper | menn0: I'll continue reviewing shortly, got to go and drop of kids to guides/show | 04:50 |
| thumper | bbs | 04:50 |
| menn0 | thumper: np | 04:51 |
| jam | morning all | 07:54 |
| dimitern | dooferlad, hey, can you check if you open http://imgur.com/a/ky3cl please? | 09:13 |
| fwereade | dimitern, nice | 09:13 |
| dooferlad | dimitern: yep | 09:14 |
| dimitern | fwereade, dooferlad, thanks :) | 09:14 |
| dimitern | wasn't sure I need to actually sign up for imgur to "public" the album - never used it, and their UI is confusing | 09:14 |
| dooferlad | dimitern: I have an environment in EC2 that has had its agent-state-info stuck in "Request limit exceeded" since last night. | 09:20 |
| dooferlad | dimitern: any thoughts? | 09:20 |
| dimitern | dooferlad, hmm - is it the shared account? | 09:21 |
| dooferlad | dimitern: https://console.aws.amazon.com/trustedadvisor/home?#/dashboard says I am below the service limits | 09:21 |
| dooferlad | so I expect this is a cached response :-| | 09:22 |
| dimitern | dooferlad, if it's the shared account, I can have a look | 09:23 |
| dooferlad | dimitern: it is shared, in eu-central | 09:23 |
| dimitern | dooferlad, ok, looking | 09:23 |
| dooferlad | dimitern: and, the fun part is, I only have 3 machines running after I did an ensure-availability -n 3 and had a charm on machine 1. | 09:24 |
| dooferlad | dimitern: so it looks like Juju tried to add another machine then gave up | 09:25 |
| dooferlad | dimitern: http://pastebin.ubuntu.com/13111330/ | 09:25 |
| dimitern | dooferlad, that smells like the instancepoller getting overexcited | 09:27 |
| dimitern | and polling more often than needed | 09:27 |
| dooferlad | dimitern: even though we are, at the moment, under limit? | 09:27 |
| dooferlad | dimitern: ignore that comment, I was looking at service limits, not request limits | 09:28 |
| dooferlad | dimitern: we definitely need exponential backoff in our EC2 API with automatic retries. Is it supposed to have it already? | 09:33 |
| dimitern | dooferlad, IIRC we already have that | 09:33 |
| dimitern | dooferlad, or maybe it was just for the instancepoller | 09:33 |
| dooferlad | dimitern: it shouldn't be related to the bug I am looking at anyway since the customer is using MAAS. | 09:35 |
| dimitern | dooferlad, yeah | 09:35 |
| dimitern | dooferlad, can you try destroy-machine --force on those with the error and add new ones? | 09:37 |
| dooferlad | dimitern: possibly, but it isn't important right now. Was just wondering about a quick fix. | 09:38 |
| dooferlad | dimitern: just seemed odd | 09:38 |
| dimitern | dooferlad, indeed - before destroying the env, it might be useful to get the machine-0.log for some insight | 09:39 |
| jam | frobware: sorry I missed standup. Was still meeting with Mark. are you guys still chatting? | 10:37 |
| frobware | jam: yep | 10:38 |
| frobware | jam: into topics "catacomb and rate limiting" | 10:38 |
| jam | k. using the restroom and I'll be right there | 10:39 |
| fwereade | jam, it's a `.Kill(nil)`, so that's even easier :) | 11:06 |
| * perrito666 applies for a visa for the first time in around 20 years | 11:10 | |
| perrito666 | axw: you have a pretty strict migration policy :p | 11:10 |
| lazypower | dimitern - Help me out for a sec. Whats the name of your juju-core team? | 12:31 |
| rick_h__ | lazypower: sapphire | 12:31 |
| lazypower | ah! thank you | 12:31 |
| lazypower | is there a chart/roster somewhere? | 12:31 |
| lazypower | It'll help when blogging and pointing credit arrows :D | 12:32 |
| rick_h__ | lazypower: honestly I use the canonical directory | 12:32 |
| lazypower | Ah, allright | 12:32 |
| rick_h__ | lazypower: if you look up dimiter it shows "Juju Core - Sapphire" as his team | 12:32 |
| rick_h__ | lazypower: I'm sure there's others but just what I tend to use when I forget. | 12:32 |
| dimitern | lazypower, hey :) | 12:33 |
| perrito666 | it would be really nice to have an api for the directory so I can write a plugin for my irc client | 12:33 |
| rick_h__ | perrito666: come on, web scrapers ftw! :P | 12:34 |
| lazypower | perrito666 - automate away the pain. Embedded profiles for IRC | 12:34 |
| jam | perrito666: do you know about mup? | 12:40 |
| perrito666 | jam: I do I would like my client to show me faces on hover though :) | 13:31 |
| jam | perrito666: would be good | 13:32 |
| cherylj | frankban: I see that you're reverting your update for crypto. Was it updated originally for a particular reason? | 13:48 |
| mup | Bug #1513466 opened: Different behavior on ServiceDeploy with Config/ConfigYAML <juju-core:New> <https://launchpad.net/bugs/1513466> | 13:50 |
| mup | Bug #1513468 opened: imports github.com/juju/juju/workload/api/internal/client: use of internal package not allowed <blocker> <ci> <regression> <testing> <wily> <juju-core:New> <juju-core 1.25:Fix Released> <https://launchpad.net/bugs/1513468> | 13:51 |
| frankban | cherylj: I don;t remember specific reasons, probably just ended up there because it was updated in my GOPATH by some other project | 14:05 |
| cherylj | frankban: ok, thanks! | 14:05 |
| frankban | cherylj: should I merge it? | 14:06 |
| cherylj | frankban: did your local tests pass? | 14:06 |
| frankban | cherylj: I alway have some intermittent failures locally, but they seems not related to the downgrade, CI will tell us I guess | 14:07 |
| * cherylj curses intermittent failures | 14:07 | |
| cherylj | frankban: yes, please merge | 14:07 |
| frankban | cherylj: done, how do we know if this fixes armhf? | 14:08 |
| cherylj | frankban: I don't think we have any way to test that without requesting images be built for 1.26-alpha 1 | 14:09 |
| frankban | cherylj: ok, does merging this automatically unblock master? | 14:10 |
| cherylj | frankban: no, we'll need to verify that the fix worked first | 14:10 |
| perrito666 | someone gave more machine to CI? curses are coming faster | 14:11 |
| frankban | cherylj: Does not match ['fixes-1513236'], I wrote fixes-1513236 in the pr comment, what else should I do? | 14:11 |
| perrito666 | frankban: $$fixes-1513236$$ should do the trick | 14:11 |
| cherylj | frankban: use "$$fixes-1513236$$" | 14:11 |
| cherylj | rather than $$merge$$ | 14:11 |
| frankban | ok done | 14:12 |
| cherylj | thanks, frankban! | 14:14 |
| abentley | frankban: If master is blessed, your bug will automatically be marked fix-released. If there are no other blockers, that will unblock master. | 14:40 |
| frankban | abentley: sounds good | 14:40 |
| frankban | cherylj: downgrade branch landed | 14:45 |
| abentley | frankban: master bec300366 now testing. | 14:47 |
| mup | Bug #1513492 opened: add-machine with vsphere triggers machine-0: panic: juju home hasn't been initialized <add-machine> <panic> <vsphere> <juju-core:Triaged> <https://launchpad.net/bugs/1513492> | 15:00 |
| cory_fu | With the release of Juju 1.25, we seem to be seeing shorter idle times on the agent-state, possibly due to update-status hook being called more frequently. Was there a specific change related to that in 1.25? | 15:00 |
| perrito666 | bbl lunch | 15:01 |
| === jog_ is now known as jog | ||
| perrito666 | cory_fu: update status hook will call before entering idle status | 15:01 |
| perrito666 | but the added time there might make idle time shorter as a consequence | 15:01 |
| perrito666 | fwereade: correct me if that changed | 15:01 |
| perrito666 | now yes, bbl | 15:02 |
| fwereade | perrito666, cory_fu: yes, I wouldn't expect a non-errored agent to be idle for longer than 5 mins (I think that's the update-status period) | 15:03 |
| fwereade | cory_fu, but if it never gets close to that there might be something up | 15:03 |
| cory_fu | I'm not sure if I understand. The issue I'm running up against is that I'm waiting for a 30s idle period and am not seeing it within a 30min window. This seems to mostly happen on Azure | 15:04 |
| cory_fu | Which is notoriously slow for these deploys, so that may be a factor. | 15:05 |
| tvansteenburgh | cory_fu: i have examples on other clouds too | 15:05 |
| cory_fu | tvansteenburgh: But it's not 100% consistent on other clouds? | 15:05 |
| tvansteenburgh | cory_fu: azure is notable b/c it hasn't passed on 1.25 at all. hp on the other hand, has passed at least twice :P | 15:06 |
| katco | wwitzel3: standup? | 15:07 |
| wwitzel3 | katco: trying .. :/ | 15:08 |
| tvansteenburgh | fwereade, cory_fu: seems to me that agent-status.since should not be updated unless the value of agent-status.current actually changes | 15:25 |
| fwereade | tvansteenburgh, right, but it executes a hook every 5 minutes | 15:25 |
| fwereade | tvansteenburgh, is workload-status also shortened? | 15:25 |
| fwereade | tvansteenburgh, I would expect ~5mins of idle, separated by (likely) sub-second blips of executing | 15:26 |
| tvansteenburgh | fwereade: no, in our examples, wordload-status.since is much older - 8 to 15 minutes older in the one i'm looking at | 15:27 |
| fwereade | tvansteenburgh, cool, I think that's what I'd expect | 15:27 |
| fwereade | tvansteenburgh, is it unhelpful to you? | 15:27 |
| tvansteenburgh | fwereade: i'm not sure. we've been using agent-status.since to determine when an environment had "settled" - all agents idle for 30 seconds. that worked well prior to 1.25, but now we see most deployments never reaching that settled state. trying to figure out what changed | 15:30 |
| frankban | cherylj, abentley: bec30036 failed, errors don't seem related though | 15:30 |
| fwereade | tvansteenburgh, well, the update-status thing is the clear proximate cause, but ultimately I think it's incorrect to depend on agent status as a proxy for environment stability | 15:31 |
| fwereade | tvansteenburgh, that should be what workload-status is for -- assuming the charm implements it | 15:32 |
| abentley | frankban: no, but it does seem like a legit failure. The same test failed for the last test of master. | 15:32 |
| tvansteenburgh | fwereade: yeah, fair enough. we were trying to accommodate the charms that don't, but i see your point | 15:34 |
| fwereade | tvansteenburgh, cool -- forwarded you a mail where I go into a bit more detail, in case it's relevant :) | 15:35 |
| tvansteenburgh | fwereade: thanks! | 15:35 |
| frankban | abentley: how do we check that https://bugs.launchpad.net/juju-core/+bug/1513236 is fixed? | 15:36 |
| mup | Bug #1513236: Cannot build trusty armhf with go1.2 on from master <armhf> <blocker> <go1.2> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1513236> | 15:36 |
| katco | davecheney: did you have a golang issue or something to justify unblocking master for bug 1513236 ? | 15:36 |
| katco | davecheney: (just now seeing your message) i'd like to update that bug with justification before untagging it as a blocker | 15:37 |
| abentley | frankban: Regardless of whether that bug is fixed, we don't want to unblock until we get a bless. | 15:37 |
| frankban | abentley: I don't question that | 15:38 |
| abentley | frankban: That's tricky, because we don't build armhf as part of testing, because we don't have suitable hardware to test with. I'd talk to sinzui. | 15:40 |
| sinzui | abentley: mgz: I beleive we could crossbuild it, which would catch the error. | 15:41 |
| natefinch | abentley, frankban, sinzui : note that davecheney said that this was very likely an upstream go 1.2 bug that is not likely to be fixed anytime soon, possibly ever | 15:41 |
| sinzui | abentley: mgz: I was also thinking of asking for armhf hardware, but mgz might be able to prove the cae for us with this chromebook | 15:42 |
| sinzui | natefinch: yeah, Go has moved on to newer versions | 15:42 |
| cherylj | hey dimitern, is there any update on bug 1483879? I know the fix isn't trivial, but the bug was brought up in the cross team call and I wanted to make sure it was still in progress... | 15:50 |
| mup | Bug #1483879: MAAS provider: terminate-machine --force or destroy-environment don't DHCP release container IPs <bug-squad> <destroy-machine> <landscape> <maas-provider> <sts> <juju-core:Triaged> <juju-core 1.24:Triaged> <juju-core 1.25:In Progress by dimitern> <https://launchpad.net/bugs/1483879> | 15:50 |
| dimitern | cherylj, it still is - I'm close to proposing 1.24 fix (needed to fix my maas setup to test it properly, which I finished a couple of hours ago) | 15:52 |
| cherylj | dimitern: great, thanks! I'll put an update in the bug that it's close for 1.24. | 15:52 |
| cherylj | dimitern: will it be difficult to forward port? | 15:52 |
| dimitern | cherylj, the 1.24 fix is the most difficult, as it needs some cherry-picking from 1.25 | 15:53 |
| cherylj | ah, ok | 15:53 |
| dimitern | cherylj, the 1.25 and master forward ports should be much simpler | 15:53 |
| natefinch | wwitzel3, ericsnow: fyi, a few tests failed on the lxd build tags merge, I can fix easily, but just letting you know. | 15:58 |
| ericsnow | natefinch: k | 15:58 |
| wwitzel3 | natefinch: ty | 15:59 |
| natefinch | alexisb, katco: I notice the "roomie" thing on the oakland spreadsheet says N/A for everyone. Does that mean we all get our own rooms? | 16:01 |
| katco | natefinch: that is my understanding | 16:02 |
| natefinch | katco: awesome :) | 16:02 |
| katco | natefinch: yes, as an introvert that makes me extremely happy | 16:02 |
| katco | natefinch: i.e. i'll actually have a place i can recharge | 16:03 |
| * dooferlad is a happy introvert as well at this news | 16:03 | |
| natefinch | katco: yeah, totally understand that | 16:03 |
| natefinch | katco: I'm generally fine with sharing a room... right up until the actual sleeping part... then I really just want my own room, thankyouverymuch. | 16:04 |
| katco | natefinch: it's pretty disastrous for me as i feel like i have to be "on" for 24/7 for an entire week | 16:05 |
| wwitzel3 | I usually wake up at 7am and don't go to bed until 3am during sprints .. they should really do a timeshare thing, probably save money | 16:07 |
| katco | wwitzel3: 1:1? | 16:07 |
| wwitzel3 | I also feed off the engery of introverts, so that helps | 16:08 |
| natefinch | wwitzel3: rofl | 16:08 |
| katco | haha | 16:08 |
| lazypower | > I feed off he energy of introverts | 16:16 |
| lazypower | strange place to join the conversation, but knowing wwitzel3 i'm not surprised... | 16:16 |
| wwitzel3 | lazypower: :) | 16:21 |
| natefinch | no wonder wwitzel3 likes programming so much... neverending supply of food. | 16:22 |
| frobware | cherylj, would you have time to try a fix for https://bugs.launchpad.net/juju-core/+bug/1412621 | 16:31 |
| mup | Bug #1412621: replica set EMPTYCONFIG MAAS bootstrap <adoption> <bootstrap> <bug-squad> <charmers> <cpec> <cpp> <maas-provider> <mongodb> <oil> <juju-core:Triaged by frobware> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1412621> | 16:31 |
| natefinch | man, prices for flights went up like 25% from yesterday afternoon :/ | 16:36 |
| perrito666 | natefinch: nearing christmas | 16:36 |
| natefinch | perrito666: I think we just crossed the "30 days from flight day" mark | 16:37 |
| perrito666 | ah, ok, that makes me panic | 16:37 |
| perrito666 | my before trip todo is especially long | 16:38 |
| ericsnow | natefinch: I found a couple typos in your build constraints patch (left a review) | 17:03 |
| natefinch | ericsnow: thanks! | 17:06 |
| ericsnow | natefinch: np | 17:06 |
| ericsnow | natefinch: found it while rebasing my patches on yours :) | 17:06 |
| natefinch | ericsnow: "how did this compile before?" exactly my question | 17:08 |
| ericsnow | natefinch: I don't think it did :/ | 17:08 |
| natefinch | ericsnow: there were two compiler errors before I even changed anything. My guess is that they were because of a merge/rebase | 17:10 |
| ericsnow | natefinch: yep | 17:10 |
| natefinch | ericsnow: fixed those spots btw | 17:10 |
| ericsnow | natefinch: thanks | 17:11 |
| natefinch | ericsnow: the only thing left is some provisioner tests that are failing | 17:11 |
| ericsnow | natefinch: weird | 17:12 |
| ericsnow | natefinch: look for instance.LXD in those tests | 17:12 |
| mup | Bug #1513552 opened: master cannot deploy charms <blocker> <charm> <ci> <deploy> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1513552> | 17:12 |
| natefinch | the letters lxd don't even exist in files in this directory :/ | 17:12 |
| natefinch | ericsnow: it's suspicious because it's in container_initialisation_test.go .. which implies it is our fault (maybe we're being punished for the UK spelling in the filename) | 17:15 |
| perrito666 | bbll | 17:39 |
| * perrito666 goes to curse ha screaming and will be back later | 17:40 | |
| cherylj | natefinch: can you take a look at bug 1513552? It looks like some of your recent commits are causing widespread CI failures | 17:41 |
| mup | Bug #1513552: master cannot deploy charms <blocker> <charm> <ci> <deploy> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1513552> | 17:41 |
| ericsnow | alexisb: sorry, zillow killed my browser | 17:48 |
| alexisb | lol | 17:48 |
| alexisb | :) | 17:48 |
| cherylj | could anyone tell me when we would see SECURE_STATESERVER_CONNECTION: "false" in agent.conf rather than "true"? | 18:11 |
| natefinch | cherylj: ok looking | 18:38 |
| natefinch | hmm... many of those failures mention deployer... seems suspicious | 19:12 |
| natefinch | sinzui: I wonder if deployer is doing something different than juju-core... because I can deploy charms just fine with juju deploy | 19:13 |
| natefinch | sinzui: re: that "cannot assign unit" problem | 19:13 |
| natefinch | katco: FYI, spending some non-trivial time looking into bug 1513552 | 19:14 |
| mup | Bug #1513552: master cannot deploy charms <blocker> <charm> <ci> <deploy> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1513552> | 19:14 |
| sinzui | natefinch: the deployer jobs that stand up openstack report the same problem | 19:15 |
| katco | natefinch: yep thx for the heads-up | 19:15 |
| sinzui | natefinch: eg: http://reports.vapour.ws/releases/3270/job/OS-deployer/attempt/502 | 19:15 |
| sinzui | natefinch: the quickstart error is also the same (though reported differently) http://reports.vapour.ws/releases/3270/job/aws-quickstart-bundle/attempt/1282 | 19:16 |
| sinzui | ^ landscape bundle | 19:17 |
| natefinch | sinzui: do we have tests that just use juju deploy? | 19:17 |
| natefinch | (not being snarky, real question... want to make sure I'm right in thinking it's deployer etc) | 19:18 |
| sinzui | natefinch: to deploy a bundle? not yet | 19:18 |
| natefinch | sinzui: no, to deploy a charm | 19:19 |
| sinzui | natefinch: many tens of tests deploy a charm | 19:20 |
| natefinch | sinzui: and those all pass? So it's just deployer and quickstart? | 19:20 |
| sinzui | natefinch: no. http://reports.vapour.ws/releases/3270 clearly shows "deploy", "deployer", and "quickstart" are broken on all substrates, all series, all archs | 19:21 |
| sinzui | natefinch: this shows every test that failed in the two revs that were tested http://reports.vapour.ws/releases/issue/563b902c749a563ed218b6cf | 19:21 |
| sinzui | natefinch: I think I see the issue :) | 19:26 |
| sinzui | natefinch: nm, I am looking at stale data | 19:26 |
| cherylj | aww, I had gotten my hopes up | 19:27 |
| natefinch | man I wish we didn't redact the API parameters | 19:48 |
| ericsnow | fwereade: isn't it gloomy in the catacombs? | 19:55 |
| ericsnow | fwereade: I suppose they have a certain charm :) | 19:55 |
| fwereade | ericsnow, terribly glum, yeah :) | 19:55 |
| thumper | menn0: test failure: [LOG] 0:00.206 ERROR juju.apiserver debug-log handler error: tailer stopped: tailable cursor requested on non capped collection | 20:00 |
| wwitzel3 | katco ericsnow natefinch ping | 20:00 |
| katco | wwitzel3: brt | 20:02 |
| natefinch | sinzui: I bet this is a race condition where the script is calling expose or something before the unit has a machine assigned, and that's causing us to go down a codepath we didn't previously go down, because unit creation and assignment happened in lock step. | 20:10 |
| natefinch | sinzui: because a simple deploy definitely still works | 20:10 |
| sinzui | natefinch: maybe. the deploy_stack.py copies a lot of deployment where expose is called immediately after deploy and add-relation. I thnk deployer though deferred that operation until relations were added, and relations were deffered until all units were up | 20:13 |
| sinzui | natefinch: http://reports.vapour.ws/releases/3270/job/aws-deploy-trusty-amd64/attempt/2476 does show that deploy, add-relation, and export were all called withing seconds of each other | 20:15 |
| katco | wwitzel3: ericsnow natefinch sorry, almost there | 20:17 |
| natefinch | sinzui: definitely looks like expose and/or add-relation is the problem, I can repro if I do juju deploy wordpress && juju add-relation wordpress mysql && juju expose wordpress | 20:23 |
| thumper | fwereade: ping | 20:23 |
| fwereade | thumper, pong | 20:23 |
| thumper | fwereade: can we chat in about 10min? | 20:23 |
| fwereade | thumper, sure | 20:24 |
| fwereade | natefinch, I think you're right | 20:24 |
| fwereade | natefinch, is that the firewaller falling over? | 20:25 |
| natefinch | fwereade: sorry, in a meeting | 20:30 |
| fwereade | natefinch, np | 20:32 |
| thumper | fwereade: https://plus.google.com/hangouts/_/canonical.com/chat?authuser=1 | 20:37 |
| cherylj | wallyworld: you around? | 21:26 |
| wallyworld | cherylj: sorta | 21:27 |
| cherylj | wallyworld: could you ping me when you get a few minutes? I need some help with a bug | 21:27 |
| wallyworld | sure, give me 10 mins | 21:27 |
| cherylj | sounds good, thanks | 21:27 |
| === sinzui_ is now known as sinzui | ||
| wallyworld | cherylj: which bug? | 21:37 |
| cherylj | https://bugs.launchpad.net/juju-core/+bug/1512782 | 21:37 |
| mup | Bug #1512782: wget cert issues causing failure to create containers <cloud-installer> <juju-core:Triaged> <https://launchpad.net/bugs/1512782> | 21:37 |
| cherylj | They're doing some weird stuff with their lxcbr0 and nested kvm / lxc | 21:38 |
| wallyworld | cherylj: give me a minute to read bug info; that's wget/lxc stuff hasn't changed in ages so it's likely to be specific to their setup | 21:39 |
| cherylj | The last update is probably the most useful | 21:40 |
| cherylj | wallyworld: this line in the machine-0.log also looks really weird to me: DEBUG juju.worker.certupdater certupdater.go:191 new addresses [localhost juju-apiserver juju-mongodb anything 10.0.7.1] | 21:41 |
| wallyworld | cherylj: that's normal - we use those hard coded names plus machine IP addresses in the CA cert SAN | 21:42 |
| cherylj | ok | 21:42 |
| wallyworld | cherylj: what may be the issue though is that we add the IP address of juju managed maches to the cert SAN - that I address has to be the source of where the wget comes from | 21:43 |
| wallyworld | if they are using some weird network setup the cert addresses won't match | 21:43 |
| cherylj | wallyworld: I think that's the case | 21:43 |
| wallyworld | hmmm | 21:44 |
| wallyworld | off hand i'm not sure how to solve that | 21:44 |
| cherylj | the logs for the lxc-create show they're getting the image through 10.0.3.45, but the only ip in the certupdater is the 10.0.7.1 | 21:44 |
| wallyworld | yup | 21:44 |
| menn0 | thumper: wrt to that the juju-run/juju-dumplogs symlink issue, I've had a better idea | 21:44 |
| wallyworld | the cert updater wroks off listening to the addresses juju records for the machines | 21:44 |
| thumper | menn0: yeah? | 21:44 |
| menn0 | thumper: make the symlinks relative to the cmd.Context's Dir | 21:45 |
| thumper | will that always work? | 21:45 |
| wallyworld | cherylj: so we need to look at the address updater worker to see how to make it recognise and report the correct addresses | 21:45 |
| menn0 | thumper: in production this will be "/", but in tests (when using testing.RunCommand) it'll be a temp directory | 21:45 |
| wallyworld | cherylj: off hand, i'd need to look into how all that works | 21:45 |
| thumper | menn0: have you checked that production it is "/" ? | 21:46 |
| menn0 | thumper: yep, I just checked that | 21:46 |
| thumper | menn0: also consider the current local provider | 21:46 |
| menn0 | thumper: what's different with the local provider? | 21:46 |
| thumper | menn0: it is just "special" | 21:46 |
| thumper | datadir is ~/.juju/<env-name>/ | 21:46 |
| wallyworld | cherylj: actrually, looking at the logs, that 10.0.3.45 address is the state server address | 21:47 |
| menn0 | thumper: yep, that's not related to this. the source of the symlink doesn't change. it's just where the symlink gets put. | 21:47 |
| wallyworld | i'm talking about the machine on which the container is being created | 21:47 |
| thumper | menn0: k | 21:47 |
| cherylj | wallyworld: yeah... | 21:47 |
| menn0 | thumper: i'll make the change now as a separate PR so you can have a look | 21:47 |
| wallyworld | cherylj: so we need to ensure the CA cert records in its SAN list the IP addresses of all worker machines, the addresses from which wget requests originate | 21:48 |
| wallyworld | that means we need juju to record the correct thing in the machine address field | 21:49 |
| wallyworld | so we need the machine agent (i think it's the agent) to report the correct addresses | 21:50 |
| wallyworld | not sure how smart this all is with different nrtwork setups | 21:50 |
| cherylj | wallyworld: I see that it picks up 10.0.3.45 as a machine address. Not sure why the certupdater didn't include that one | 21:51 |
| wallyworld | cherylj: that's the state server address | 21:51 |
| wallyworld | we need to record the machine address from which the wget originates | 21:51 |
| wallyworld | cherylj: so we need each machine to correctly report to juju its address | 21:52 |
| wallyworld | or addresses | 21:52 |
| wallyworld | those are stored in the machine addresses field in state | 21:52 |
| wallyworld | that's what the cert updater listens to | 21:53 |
| wallyworld | i think its the machine agent on each machine which reports those addresses | 21:53 |
| cherylj | wallyworld: you mean like this? INFO juju.worker.machiner machiner.go:100 setting addresses for machine-0 to ["local-machine:127.0.0.1" "local-cloud:192.168.122.1" "local-cloud:10.0.3.45" "local-machine:::1"] | 21:53 |
| wallyworld | so my guess from memory is that we need to look at how the machine agent queries its host addresses | 21:53 |
| wallyworld | cherylj: are they running the lxc containers on machine 0? | 21:54 |
| wallyworld | i was assuming there'd be a machine 1 or 2 or whatever | 21:55 |
| cherylj | wallyworld: no, I think they're doing it on machine-1 | 21:56 |
| wallyworld | i think you said they were nesting lxc inside kvm? | 21:56 |
| wallyworld | cherylj: right ok. so whatever the source IP address of the wget request is has to be recorded in the SAN list | 21:56 |
| wallyworld | cherylj: that source address is the machine hosting the lxc containers | 21:57 |
| wallyworld | which is typically machine 1's address. or it needs to be the kvm address if lxc inside kvm | 21:57 |
| wallyworld | cherylj: so there needs to be a line in the logs like the above but which says setting addresses for machine-1 .... | 21:58 |
| cherylj | wallyworld: okay, I see what you mean now | 21:58 |
| wallyworld | cherylj: i have to relocate, will be afk for 20 minutes | 21:59 |
| cherylj | wallyworld: sure, np | 21:59 |
| ericsnow | natefinch: check out http://reviews.vapour.ws/r/3080/ | 22:24 |
| menn0 | thumper: this approach is working out pretty nicely | 22:26 |
| thumper | menn0: awesome | 22:26 |
| thumper | menn0: did the no tail bit land in master? | 22:26 |
| thumper | menn0: it may help me fix this problem :) | 22:26 |
| menn0 | thumper: yep, that's landed | 22:27 |
| menn0 | man there's lots of tests that use cmd.DefaultContext that should probably be using cmdtesting.Context | 22:29 |
| * menn0 ignores for now | 22:29 | |
| perrito666 | and then you froze | 22:32 |
| perrito666 | wwitzel3: | 22:32 |
| perrito666 | I meant wallyworld | 22:32 |
| axw_ | perrito666: what do you need a visa for? | 22:58 |
| perrito666 | axw_: enter australia | 22:59 |
| axw_ | perrito666: just for meetings? :/ | 22:59 |
| perrito666 | axw_: I got an e-visa | 23:02 |
| wallyworld | perrito666: we're here now | 23:16 |
| davechen1y | https://github.com/docker/docker/pull/17700 | 23:27 |
| davechen1y | ^ docker have removed support for lxc | 23:27 |
| mup | Bug #1513659 opened: 1.24.6 fails to bootstrap with "ERROR juju.cmd supercommand.go:430 upgrade in progress - Juju functionality is limited" <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1513659> | 23:31 |
| mup | Bug #1513659 changed: 1.24.6 fails to bootstrap with "ERROR juju.cmd supercommand.go:430 upgrade in progress - Juju functionality is limited" <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1513659> | 23:37 |
| mup | Bug #1513659 opened: 1.24.6 fails to bootstrap with "ERROR juju.cmd supercommand.go:430 upgrade in progress - Juju functionality is limited" <cloud-installer> <juju-core:New> <https://launchpad.net/bugs/1513659> | 23:40 |
| davechen1y | mwhudson: ding ding ding. | 23:44 |
| * mwhudson vibrates tunefully | 23:45 | |
| mwhudson | davechen1y: eh? | 23:45 |
| davechen1y | your ppc64 observation | 23:47 |
| ericsnow | katco: am I okay landing my fix of natefinch's patch? http://reviews.vapour.ws/r/3080/ | 23:48 |
| mwhudson | davechen1y: ah heh | 23:48 |
| ericsnow | katco: I'd like to land those LXD patches today | 23:49 |
| mwhudson | davechen1y: can you re-run test_shared on arm64 pls? | 23:49 |
| mwhudson | davechen1y: that was pretty mystically debugging from russ | 23:49 |
| davechen1y | he is the master to psychic debugging | 23:50 |
| davechen1y | mwhudson: arm64 -- will do | 23:50 |
| mwhudson | ta | 23:51 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
