| katco | perrito666: i think i see what's going on | 00:02 |
|---|---|---|
| katco | perrito666: http://play.golang.org/p/VzqUYG-D2y | 00:02 |
| katco | perrito666: the capacity fluctuates depending on how far into the slice you start your window, so you pretty much have to create a new slice and copy into it | 00:02 |
| perrito666 | it might be looking into the capacity and trying to get as many objects instead of lenght objects | 00:03 |
| perrito666 | man, I lova play.golang so much | 00:04 |
| katco | hehe | 00:04 |
| katco | it is nice | 00:04 |
| perrito666 | love* | 00:04 |
| perrito666 | its like a playable pastebin | 00:05 |
| katco | :) | 00:05 |
| katco | sandboxed too | 00:05 |
| katco | i think rob pike did an interesting write-up on it somewhere | 00:05 |
| katco | or maybe it was russ cox | 00:05 |
| wwitzel3 | wallyworld: got it figured out, the user-data needed to be base64 encoded to be properly stored on GCE | 00:06 |
| wwitzel3 | wallyworld: but the cloudinit script doesn't expect it to be base64 encoded (afaict), so that is the issue. | 00:06 |
| wallyworld | wwitzel3: it's been a while since i looked, isn't is base64 encoded for all providers? | 00:06 |
| wwitzel3 | wallyworld: it is, but that is actually handled in the provider libraries for most, not in juju | 00:07 |
| wwitzel3 | wallyworld: go's gce library doesn't enforce how you insert metadata like go amz | 00:08 |
| wallyworld | so we should update the gce provider then | 00:08 |
| wwitzel3 | sorry, api libraries, no the providers | 00:08 |
| wallyworld | i guess we don't have access to api lib | 00:08 |
| wwitzel3 | but yes, we updated the provider to base64 encode | 00:08 |
| wwitzel3 | but cloud-config is still nil, so we are looking at the cloudinit GCE datastore to see if it does indeed deal with the user-data being base64 | 00:09 |
| wwitzel3 | wallyworld: so we are on the right path, thanks for letting me bug you earlier :) | 00:12 |
| wallyworld | np, i didn't help much | 00:12 |
| wwitzel3 | I'm used to that when I talk to you | 00:13 |
| wwitzel3 | ;D | 00:13 |
| katco | wwitzel3: no one disrespects a tanzanite team member without disrespecting all of us! ;) | 00:13 |
| wwitzel3 | katco: haha, I'll remember that for next time | 00:14 |
| perrito666 | ok, being 9:20 pm I feel I should EOD cheers all | 00:21 |
| katco | perrito666: tc | 00:21 |
| katco | perrito666: i'll see you tomorrow yeah? | 00:21 |
| perrito666 | indeed | 00:22 |
| katco | ok, then no happy holidays for you yet :) | 00:23 |
| perrito666 | heh ah true down under people are in their friday | 00:23 |
| katco | if i'm getting this error: juju.worker exited "environ-provisioner": no state server machines with addresses found | 00:31 |
| katco | but i have created a machine with the factory and job of JobManageEnviron, what am i doing wrong? | 00:31 |
| menn0_ | wallyworld: I get this in any unit after upgrading from 1.20 to 1.22 | 01:27 |
| menn0_ | exited "uniter": failed to initialize uniter for "unit-mediawiki-0": cannot initialize hook commands in "1.22-alpha1.1-trusty-amd64": symlink 1.22-alpha1.1-trusty-amd64/jujud 1.22-alpha1.1-trusty-amd64/action-fail: no such file or directory | 01:28 |
| menn0_ | haven't we seen that before? | 01:28 |
| === menn0_ is now known as menn0 | ||
| thumper | yes | 01:31 |
| thumper | yes we have | 01:32 |
| thumper | menn0: hangout? | 01:32 |
| thumper | menn0: the 1.20 upgrade worker creates a relative symlink | 01:33 |
| thumper | I think the fix I did for 1.21 didn't get landed in master because "we didn't support 1.20 -> 1.22 upgrades" | 01:34 |
| thumper | new code expects absolute | 01:34 |
| thumper | the 1.21 upgrade worker makes an absolute symlink | 01:34 |
| menn0 | thumper: sorry, just saw this | 01:36 |
| menn0 | thumper: standup channel? | 01:36 |
| thumper | ack | 01:36 |
| * thumper turns off distractions... | 02:01 | |
| thumper | if people super need me, text | 02:01 |
| menn0 | wallyworld: are you looking at bug 1396099? | 02:38 |
| mup | Bug #1396099: AWS/Joyent/HP/manual/maas: juju deploy error "connection is shut down" <api> <ci> <deploy> <ec2-provider> <joyent-provider> <manual-provider> <regression> <juju-core:Triaged> <https://launchpad.net/bugs/1396099> | 02:38 |
| wallyworld | menn0: yes, but been in meetings for the past 2 hours | 02:38 |
| wallyworld | partly done, but i'm not happy with the soluton | 02:38 |
| wallyworld | as we can get address changes at any time | 02:39 |
| wallyworld | and there's potential for at least 2 restarts as the state server comes up | 02:39 |
| menn0 | wallyworld: is there a way to avoid restarting the API server and just have the new cert be used for new connections? | 02:39 |
| wallyworld | that's what i need to dig into | 02:40 |
| wallyworld | when i first looked, it didn't seem plausible | 02:40 |
| wallyworld | but i will have to look a bit harder | 02:40 |
| menn0 | wallyworld: that would be slickest solution (but I can see how it might be hard/impossible) | 02:40 |
| wallyworld | yes, agreed. i did initially want to do that but since it appeared hard, it was easier to restart | 02:41 |
| wallyworld | i didn't think it would matter | 02:41 |
| wallyworld | as there would only be a very short window when the state server comes up | 02:41 |
| menn0 | wallyworld: turns out computers are pretty fast :-p | 02:48 |
| wallyworld | menn0: scripting, yes | 02:49 |
| menn0 | wallyworld: that's what I mean | 02:49 |
| wallyworld | i didn't think we'd be polling state server to jump in and grab a connection | 02:49 |
| menn0 | wallyworld: I don't think anything is polling | 03:17 |
| menn0 | wallyworld: as soon as the bootstrap command returns the tests (and my script) attempt to deploy a charm | 03:17 |
| menn0 | wallyworld: and that's often the same time the API server gets restarted | 03:17 |
| wwitzel3 | so if I want to make a change to the cloudinit DataSourceGCE python, is there an easy way to test that with Juju? | 03:24 |
| thumper | ok, I have given up on getting this done today | 03:24 |
| wallyworld | menn0: yeah, sorry, i didn't mean polling in the strictly literal sense, just a client that connected immediately after the state server comes up. | 03:25 |
| thumper | however, I do have it to the state where I just need the tests | 03:25 |
| wallyworld | still in meeting s o i haven't looked anymore yet | 03:25 |
| thumper | and break it in half | 03:25 |
| thumper | found that I have some state methods needed | 03:25 |
| thumper | so I can land them independantly | 03:26 |
| katco | thumper: sorry, he's all mine! | 03:57 |
| thumper | he who? | 03:59 |
| katco | thumper: ian | 03:59 |
| thumper | katco: he's all yours, with my blessing | 03:59 |
| * thumper ditches him while he has the opportunity | 03:59 | |
| thumper | no backsies | 03:59 |
| katco | thumper: i thought you'd be more jealous | 03:59 |
| katco | rofl | 04:00 |
| thumper | happy holidays everyone | 04:01 |
| thumper | I'll be back Jan 5th | 04:01 |
| katco | thumper: happy holidays | 04:01 |
| wwitzel3 | see ya thumper, have a good one | 04:02 |
| katco | thumper: be safe, have fun | 04:02 |
| thumper | I foresee much hacking in the next two weeks | 04:02 |
| thumper | and none of it in go | 04:02 |
| thumper | python, javascript and C# | 04:02 |
| thumper | laters | 04:02 |
| katco | nice | 04:03 |
| katco | anastasiamac: oh and i sincerely apologize for monopolizing ian's time. sorry about that | 05:00 |
| katco | ok off to bed... tc and happy holidays to all of those for whom it is friday! | 05:05 |
| menn0 | katco: thanks... have a good break | 05:06 |
| katco | menn0: you too good sir! | 05:06 |
| menn0 | wallyworld: I think I see a way to swap out certificates on the fly | 05:06 |
| === kadams54 is now known as kadams54-away | ||
| === kadams54-away is now known as kadams54 | ||
| menn0 | wallyworld: if you look at the implementation of the listener returned by tls.NewListener | 05:08 |
| menn0 | wallyworld: you can see that the config (which contains the cert) is passed to Server for each inbound connection | 05:09 |
| menn0 | wallyworld: so if you can change the cert in the config, new connections will use the new cert | 05:09 |
| menn0 | wallyworld: of course the config is kept private so we'd have to implement our own tls listener | 05:10 |
| menn0 | wallyworld: but that's trivial (it's tiny) and the part that does the hard work (Server) is public | 05:10 |
| wallyworld | menn0: yep, i came to the same conclusion, but haven't been able to finish implementing because i've been ina meeting all day | 05:28 |
| wallyworld | will complete the fix tonight | 05:28 |
| wallyworld | i was wary of changing the tls config on the fly | 05:29 |
| wallyworld | but on reading the tsl code in the std lib i think it will be ok | 05:29 |
| wallyworld | it still feels a bit dirty, but hopfully will work | 05:29 |
| menn0 | wallyworld: there's a bit of restructuring to do to make the api server accessible to the certupdater isn't there? | 05:30 |
| wallyworld | yes, but i'm part way through doing it | 05:30 |
| anastasiamac | katco: np :) i got the whole 15 min of his time :) | 05:30 |
| menn0 | wallyworld: also, you'll probably need a mutex to protect updates to the config | 05:30 |
| wallyworld | probably yes | 05:30 |
| wallyworld | although i'm thinking of using a channel | 05:31 |
| menn0 | wallyworld: and I'd be tempted to take copies of the config so that existing connections keep their previous config | 05:31 |
| wallyworld | there's only one server though | 05:31 |
| wallyworld | the cert is only ever augmented with new addresses | 05:32 |
| wallyworld | so i should just be able to replace on the fly | 05:32 |
| menn0 | wallyworld: there's one server but each time a connection is accepted a new Server instance is created and it gets a pointer to the config | 05:32 |
| wallyworld | really? | 05:33 |
| menn0 | yep | 05:33 |
| menn0 | have a look at crypto/tls/tls.go in the stdlib | 05:33 |
| wallyworld | the worker (which is the server) is run and then stays running | 05:33 |
| menn0 | the listener.Accept method | 05:33 |
| wallyworld | it shouldn't matter as a new cert will accepts connections made previosuly | 05:34 |
| menn0 | wallyworld: you're probably right | 05:34 |
| wallyworld | as we are just adding to the address list | 05:34 |
| menn0 | wallyworld: if that's the only change made to the cert | 05:34 |
| wallyworld | that's the only change | 05:35 |
| menn0 | wallyworld: and the cert probably doesn't even get looked at once the connection is set up | 05:35 |
| wallyworld | i can always iterate if we find an issue | 05:35 |
| wallyworld | yes, that's my understanding | 05:35 |
| wallyworld | i gotta run to relocate, will finish the fix tonight now that all my meetings are done | 05:36 |
| menn0 | wallyworld: I am still worried about what happens if the cert gets accessed (for a new connection) just as it's being updated | 05:36 |
| wallyworld | me too, i have to think it through | 05:36 |
| menn0 | wallyworld: b/c I think the handling of connection happens in it's own goroutine | 05:36 |
| menn0 | wallyworld: np | 05:36 |
| menn0 | wallyworld: have a great xmas | 05:36 |
| menn0 | wallyworld: i'm about done | 05:36 |
| wallyworld | menn0: you too, thanks for thinking through this issue with me, good that we converged on a similar approach | 05:37 |
| wallyworld | have a good break | 05:37 |
| menn0 | wallyworld: yep that is a good sign | 05:37 |
| menn0 | wallyworld: (or we're both equally stupid) | 05:37 |
| wallyworld | lol, well maybe | 05:37 |
| menn0 | wallyworld: it was fun to think about | 05:37 |
| wallyworld | i will do it and test and see how it works | 05:37 |
| wallyworld | i just wish i had more time today out of meetings to actually do the fix | 05:38 |
| wallyworld | anyway, gotta run, be back later | 05:38 |
| menn0 | wallyworld: cool, speak later | 05:39 |
| fwereade | davecheney, was just looking at state/multiwatcher, and wondering what RelationUnitsChange was doing in there? | 09:10 |
| perrito666 | morning all | 10:17 |
| TheMue | morning perrito666 | 10:18 |
| frankban | fwereade: is there already a plan for importing the jenv files generated by "juju user add" to the environments directory of the juju home? perhaps something like "juju user import pat/to/jenv"? I am asking because we added support for using jenv files without a corresponding entry in environments.yaml in quickstart and that would complete the story. | 10:35 |
| frankban | path/to/jenv even | 10:36 |
| fwereade | frankban, yes, that's the heart of it, but I think we're calling it `juju connect` | 10:38 |
| fwereade | frankban, thumper will have a better idea of when we can expect that than I do, though | 10:39 |
| wwitzel3 | fwereade: you know who I should talk to about a cloud-init question? smoser is out already | 12:27 |
| fwereade | wwitzel3, hmm, not really :( | 12:27 |
| wwitzel3 | fwereade: dang, ok, thanks :) | 12:28 |
| fwereade | wwitzel3, possibly gsamfira_ might know about it, what with cloudbase-init? | 12:28 |
| wwitzel3 | fwereade: ok, cool. I'm trying to figure out why my stored user-data on GCE isn't unpacking in to cloud-config | 12:32 |
| fwereade | wwitzel3, sorry, nothing really springs to mind | 12:33 |
| wwitzel3 | fwereade: it is probably my fault, I think I might need package some headers with the base64 blob | 12:34 |
| wwitzel3 | fwereade: I will probably talk you throughout this process, you can ignore me, or just make unattentive bu supportive comments | 12:34 |
| wwitzel3 | talk at you that is | 12:35 |
| fwereade | wwitzel3, have you tried dephlogisticating the grommets? | 12:35 |
| fwereade | (sorry) | 12:35 |
| fwereade | (talk away :)) | 12:35 |
| wwitzel3 | fwereade: haha, no, that's perfect | 12:36 |
| wwitzel3 | fwereade: at least you weren't on the hangout with me yesterday, after doing nothing but hitting my desk and swearing for 5 minutes I told ericsnow "I should probably take a break" | 12:36 |
| fwereade | wwitzel3, ouch | 12:37 |
| wwitzel3 | fwereade: but, when I came back, in like 5 minutes, I sovled the problem | 12:37 |
| fwereade | wwitzel3, yeah, tuning those things is tricky though | 12:38 |
| fwereade | wwitzel3, give up too early and the frustration just comes right back when you return | 12:38 |
| fwereade | wwitzel3, wish I could tell what the optimal take-a-break moments are | 12:38 |
| wwitzel3 | fwereade: in my experience, you have to go until you break mentally *lolcry* .. that is how your brain resets and lets you look at the problem from a different view. | 12:40 |
| wwitzel3 | fwereade: or add booze ;) | 12:40 |
| fwereade | wwitzel3, haha, yeah | 12:40 |
| perrito666 | for me the trigger is when the thought "this cant be done" arises | 12:44 |
| === kadams54 is now known as kadams54-away | ||
| wwitzel3 | ericsnow: so looking at ec2's userdata, we send it up Base64 encoded, but it is stores decoded. | 13:15 |
| wwitzel3 | ericsnow: so I feel I've done proper due diligence in that we are doing the right thing in the GCE provider. | 13:16 |
| wwitzel3 | ericsnow: We need to update DataSourceGCE.py to b64decode the incoming raw user data from GCE. | 13:16 |
| wallyworld | fwereade: hi, i have a fix for the CI blocker, would love it if you could take a look | 13:30 |
| wallyworld | http://reviews.vapour.ws/r/667/ | 13:30 |
| wallyworld | perrito666: i didn't get a chance to look at the pr you mentioned sorry, spent all day in meetings and then had to fix a CI blocker | 13:32 |
| perrito666 | wallyworld: dont worry man | 13:32 |
| TheMue | wallyworld: lgtm | 13:44 |
| wallyworld | TheMue: awesome, tyvm | 13:44 |
| TheMue | wallyworld: yw, nice solution. only stumble first over the call of processCertChanges and the go statement afterwards, but then I saw that it starts an anonymous goroutine too | 13:45 |
| wallyworld | yes, sorry if it was unclear | 13:46 |
| TheMue | wallyworld: it's fine, I like it | 13:47 |
| fwereade | wallyworld, I'm not totally in love with the stop chan, wouldn't a tomb be a bit nicer? | 13:50 |
| wallyworld | guess so, seemed like overkill | 13:51 |
| wallyworld | the stop chan concept is use delsewhere | 13:51 |
| hazmat | rogpeppe, if you could switch your examples to not being the store charm, it would be nice to send that email to the public list | 13:51 |
| aznashwan1 | perrito666: are you by any chance talking about the failing test when you set /utils to the latest commit? | 13:52 |
| perrito666 | aznashwan1: I currently have no clue what you are talking about | 13:52 |
| fwereade | wallyworld, yeah, I know, but it's generally vulnerable to panic when double-closed, where tomb is fine with double-killing | 13:52 |
| rogpeppe | hazmat: yeah | 13:53 |
| aznashwan1 | perrito666: because I ran into that on my latest PR and fixed it | 13:53 |
| rogpeppe | hazmat: i need a decent example | 13:53 |
| wallyworld | fwereade: fair point, i was assuming that listener close would only be done once | 13:53 |
| wallyworld | fwereade: can i land to unblock and follow up? the listener is not closed during the normal operation of the state server | 13:54 |
| aznashwan1 | perrito666: there was a commit in /utils which changed a message and a test failed in core because of it, thus preventing you to get up to date | 13:54 |
| fwereade | wallyworld, yeah, you're probably right, but that feels to me like exactly the sort of invariant that it's easy to break accidentally | 13:54 |
| fwereade | wallyworld, yeah go for it | 13:55 |
| wallyworld | ok, ta. i need to tweak the lxc cache. so i'll fix in that branch | 13:55 |
| aznashwan1 | perrito666: nothing major, just an ErrorMatches failing because an error was slightly modified, but it's solved now, and my latest PR will have utils set to its current HEAD | 13:56 |
| dimitern | aznashwan1, that's *exactly* why we need to get utils and other subrepos under CI integration tests with core | 14:03 |
| aznashwan1 | dimitern: you make a very good point, /utils needing this most of all the others... | 14:10 |
| dimitern | aznashwan1, we're discussing it and I'll push for it to happen sooner rather than later | 14:11 |
| aznashwan1 | dimitern: that would be awesome and I wish you the best of luck towards it :D | 14:16 |
| dimitern | aznashwan1, thanks :) | 14:16 |
| perrito666 | natefinch: coming? | 14:34 |
| TheMue | wallyworld: seen, the changeCertListener.Close() panics | 14:47 |
| TheMue | wallyworld: ? | 14:47 |
| natefinch | wallyworld: need someone to pick up that PR and try to push it through? | 15:16 |
| natefinch | mgz_: looks like CI says no space left on device.... want to show me how to fix it so I can do it myself later? | 15:17 |
| mgz_ | you mean the gating job? | 15:33 |
| natefinch | mgz_: http://juju-ci.vapour.ws:8080/job/github-merge-juju/1691/console | 15:33 |
| mgz_ | that tends to just be a symptom of having got a bad instance on aws | 15:34 |
| natefinch | mgz_: interesting | 15:34 |
| mgz_ | I don't think there's anything to do but wait for a bit, then try relanding later | 15:35 |
| natefinch | mgz_: kk | 15:35 |
| mgz_ | it may also be the branch is bad | 15:35 |
| mgz_ | first panic is "runtime error: close of nil channel" | 15:36 |
| natefinch | hmm | 15:36 |
| natefinch | which is the cause and which is the effect? That's the question. | 15:37 |
| mgz_ | it's possible panics are going to a drive on the instance without much room | 15:37 |
| mgz_ | which would be why we see it sometimes | 15:37 |
| mgz_ | (aart from the known we got a crummy instance thing) | 15:37 |
| mgz_ | does go dump itself somewhere by default during tests? | 15:38 |
| natefinch | what do you mean dump itself? | 15:42 |
| mgz_ | stick anything on disk | 15:44 |
| mgz_ | a la segfault | 15:44 |
| natefinch | mgz_: nope, the panic output is all you get | 15:50 |
| katco | so it sounds like we're still working on getting CI unblocked? | 16:03 |
| natefinch | katco: alas, yes | 16:04 |
| katco | natefinch: Shower Thought: You and your wife might create a new human before we create the next version of Juju. | 16:05 |
| natefinch | katco: haha yep | 16:05 |
| perrito666 | katco: they will, next version of juju is due jan 9 | 16:09 |
| katco | perrito666: :) | 16:09 |
| natefinch | mgz_: I think I see the bug in wallyworld's code. I'll fix it and see if I can repropose. | 16:16 |
| mgz_ | natefinch: ace | 16:16 |
| natefinch | mgz_: he was closing a channel he never gave a value, which is why it was "close of a nil channel" | 16:20 |
| katco | natefinch: that's not an error is it? | 16:20 |
| natefinch | katco: sorry, when I say "never gave a value", I mean, he never initialized it to a valid channel | 16:20 |
| katco | natefinch: ah gotcha, sorry | 16:21 |
| natefinch | hmm... so my wife is having some contractions, not at all sure it's labor, but it's something. Can someone else pick this up? I made a comment at the bottom of the PR: https://github.com/juju/juju/pull/1346 | 16:25 |
| katco | good luck natefinch | 16:26 |
| natefinch | thanks... might be nothing, she's had them before, but I probably shouldn't be on the computer ;) | 16:26 |
| katco | :) | 16:26 |
| katco | someone will take care of it, go go | 16:26 |
| katco | mgz_: so catching up: will the tests fail on my machine for this bug? | 16:35 |
| mgz_ | katco: I expect the first panic to happen for you, the following fallout is mongo dependent, so may all be different | 16:37 |
| katco | boy this was a catastrophic failure lol | 16:37 |
| * katco is scrolling through the jenkins log | 16:38 | |
| mgz_ | katco: yeah, the only bit you really need to care about is the first panic | 16:39 |
| katco | yeah trying to figure out what the heck package failed first... looks like api? | 16:39 |
| mgz_ | the rest is pretty much just mongo falling over due to poor test isolation, which apparently also fills up our allocated /tmp space | 16:39 |
| mgz_ | yeah, api | 16:39 |
| mgz_ | the pr is 1346 | 16:39 |
| wwitzel3 | ericsnow: https://bugs.launchpad.net/cloud-init/+bug/1404311 | 16:39 |
| mup | Bug #1404311: gce metadata api doesn't properly stream binary data <cloud-init:New> <https://launchpad.net/bugs/1404311> | 16:39 |
| ericsnow | wwitzel3: nice! | 16:40 |
| katco | mgz_: blindly trying nate's suggestion | 16:41 |
| ericsnow | wwitzel3: you posting that to #cloud-init? | 16:41 |
| === kadams54 is now known as kadams54-away | ||
| katco | mgz_: found a few other issues. net.Serve closes the listener so we were also double-closing the channel, also our changeCertListener methods were operating on copies of the struct which meant they were locking copies of mutexes | 16:57 |
| katco | mgz_: api package tests look like they're succeeding; i'll submit a pr after that to let CI run the full suite | 16:57 |
| perrito666 | I have a gramatical issue on my code | 17:00 |
| katco | mgz_: PR incoming. | 17:10 |
| alexisb | alrighty all, today is the last day before our holiday shutdown. Please feel free to start you celebrations early once you get to a good stopping point on what you are working on | 17:22 |
| alexisb | and enjoy your time off with family and friends! | 17:22 |
| katco | alexisb: thank you! | 17:23 |
| ericsnow | alexisb: you too | 17:23 |
| katco | alexisb: happy holidays to you and your family :) | 17:23 |
| alexisb | I am very much looking forward to the great work we will be doing in 2015 | 17:23 |
| alexisb | it will be an exciting year! | 17:23 |
| TheMue | alexisb: thanks, also happy holidays to you and your family | 17:23 |
| TheMue | katco: you added line 162 in apiserver.go based on nates diagnosis? | 17:27 |
| katco | TheMue: correct | 17:27 |
| TheMue | katco: ok, then lgtm, seen the other parts earlier | 17:28 |
| katco | TheMue: as well as a few other things. look like a test in cmd/jujud/agent needs to be updated as well | 17:28 |
| katco | TheMue: ty | 17:28 |
| TheMue | katco: yw | 17:28 |
| katco | TheMue: so you might be more familiar with these changes. i need to pass a channel in cmd/jujud/agent/upgrade_test.go:493: not enough arguments in call to a.apiserverWorkerStarter | 17:29 |
| katco | TheMue: it doesn't look to me like the test cares, so i was just going to pass in a channel and do nothing with it. | 17:30 |
| TheMue | katco: looking | 17:30 |
| katco | mgz_: i'm in the process of fixing this, but why does it look like the CI run has frozen? http://juju-ci.vapour.ws:8080/job/github-merge-juju/1693/console | 17:37 |
| TheMue | katco: machine.go line 798 the method defined in line 838 is used, the channel created one line before | 17:48 |
| katco | TheMue: hm i see that. but the test that needed updating didn't look like it was trying to test cert upgrades at all, right? | 17:50 |
| katco | TheMue: so we don't really care if certs are changed, right? | 17:50 |
| TheMue | katco: we already cared before, by stoping and restarting the apiserver | 17:51 |
| TheMue | katco: the test for this isn't changed, but the way how it is realized, now w/o a restart | 17:52 |
| katco | TheMue: i'm sorry, i'm not understanding. i'm looking at TestLoginsDuringUpgrade in upgrade_test.go. I don't see any API server restarts in there? | 17:55 |
| katco | TheMue: and i'm completely missing why a test for logging in during an upgrade would be concerned with cert changes? | 17:56 |
| TheMue | katco: the important change is here: http://reviews.vapour.ws/r/667/diff/# machine.go line 802 | 18:01 |
| TheMue | katco: but I have to admit I don't know where it has been tested so far | 18:01 |
| TheMue | katco: the old version does the restart, the new one signals the cert change, so that the listener reacts | 18:02 |
| mgz_ | katco: looks like one hang test, probably the reason for no output for ages | 18:03 |
| TheMue | katco: that's in apiserver.go line 102, where the goroutine is started | 18:03 |
| katco | TheMue: i'm still not understanding what all of that has to do with this test though? the test is not compiling because it doesn't have enough arguments anymore, and as far as i can tell it doesn't need to receive anything on this channel to test what it's trying to | 18:06 |
| TheMue | katco: ah, I see, sorry, yes. This test hasn't been changed or the local change of wallyworld hasn't been checked in. so you have to change the test, sorry. | 18:10 |
| katco | TheMue: oh no worries. i just was missing what you were trying to communicate :) | 18:10 |
| TheMue | katco: no, you absolutely have been right, I missed the point regarding the test | 18:11 |
| katco | TheMue: so my original question is: can you see any reason not to just pass it a channel and then not do anything with that channel? trying that, the tests hang | 18:11 |
| katco | TheMue: trying to figure out why... | 18:11 |
| TheMue | katco: if you only pass in a channel and nobody listens a sending will block the system | 18:13 |
| katco | TheMue: i created a buffered channel | 18:14 |
| katco | make(<-chan params.StateServingInfo, 1) | 18:14 |
| TheMue | katco: ok, then I would have to digg deeper too | 18:15 |
| katco | TheMue: but conceptually, you don't see ignoring that channel as an issue? | 18:15 |
| TheMue | katco: based on the title of this tests it's focussing the login during upgrade, no cert change | 18:16 |
| TheMue | katco: so yes, this shouldn't be an issue here | 18:17 |
| katco | TheMue: great, ty... i'll continue trying to figure out why the test hands with a buffered channel =/ | 18:18 |
| TheMue | katco: I'm lurking here into the channel a bit longer, so feel free to ping me | 18:19 |
| katco | TheMue: thanks, i appreciate it :) | 18:19 |
| TheMue | katco: yw | 18:19 |
| katco | TheMue: if i end up not talking with you again, happy holidays! | 18:19 |
| TheMue | katco: thanks. also to you happy holidays. | 18:20 |
| katco | TheMue: ty! | 18:20 |
| === mup_ is now known as mup | ||
| === mup_ is now known as mup | ||
| katco | found the issue | 19:05 |
| katco | deleted the closing of the stop channel from the wrong place. re-running api tests now | 19:06 |
| === meetingology` is now known as meetingology | ||
| === ev_ is now known as ev | ||
| katco | any CI people around? if this fix works, i'd like to open the trunk | 19:11 |
| katco | mgz_: ^ | 19:12 |
| === mup_ is now known as mup | ||
| mgz_ | katco: go for it | 19:27 |
| katco | mgz_: i didn't realize i could open the trunk. how do i do that? | 19:28 |
| * perrito666 is back from 2 hs without internet nor power... gotta love summer | 19:28 | |
| katco | wb perrito666 | 19:28 |
| mgz_ | katco: you wait for the rev to get through gating, then go through the rest of the jobs (can track on jenkins or the report site), then mark bug as fix released when the revision is blessed | 19:30 |
| perrito666 | ericsnow: ping | 19:30 |
| perrito666 | katco: tx | 19:30 |
| katco | mgz_: oh cool. i don't think i've ever done that. | 19:30 |
| ericsnow | perrito666: hey | 19:30 |
| katco | mgz_: will you be around if i have questions? or is anyone else familiar with this process? | 19:31 |
| katco | i don't know what reports to look at. i'm guessing 1.22-alpha1 for a bless? | 19:32 |
| katco | oh i need to backport this to 1.21 don't i. | 19:33 |
| katco | for beta 5? | 19:33 |
| perrito666 | I tried to make a nap during the blackout but was recruited by my wife -> https://twitter.com/majomalnis/status/546021647632052225 you have no idea how hard is to make those with >30C | 19:33 |
| katco | perrito666: yum :) | 19:34 |
| katco | well, the bug is only marked for 1.22, so i guess i don't have to backport this | 19:34 |
| perrito666 | katco: oh, they are tasteless :p until you add the white stuff | 19:34 |
| katco | perrito666: are they sugar cookies? | 19:34 |
| perrito666 | katco: no, animal cookies actually | 19:35 |
| perrito666 | they must be glaced to have some taste | 19:35 |
| katco | perrito666: i don't think i know what that is lol | 19:35 |
| perrito666 | really? I thought they where fairly common its an english recipe | 19:35 |
| perrito666 | small animal shaped cookies | 19:36 |
| katco | ohhh | 19:36 |
| katco | like animal crackers perhaps? | 19:36 |
| perrito666 | http://books.google.com.ar/books?id=OlIOxYU2YWsC&pg=PT73&source=gbs_toc_r&cad=3#v=onepage&q&f=false | 19:38 |
| * perrito666 shares the source code :p | 19:38 | |
| katco | haha | 19:38 |
| TheMue | so, working with red wine surely is no good base for proper code. ;) working day is over. i wish you all a merry christmas and a happy new year. see you all freshly charged in the new year. o/ | 19:49 |
| perrito666 | TheMue: likewise, enjoy that wine | 19:51 |
| katco | TheMue: tc! happy holidays! | 19:51 |
| perrito666 | ok, attempt n 19831287319238 to finish today's work without being interrupted | 20:38 |
| * katco is trying to figure out where a test is hanging, and go test is not giving me any output | 20:39 | |
| bodie_ | I'm trying to figure out how to get a state testing unit's charm URL | 20:42 |
| perrito666 | ericsnow: still here? | 20:43 |
| ericsnow | perrito666: yep | 20:43 |
| perrito666 | metadata know about series? | 20:43 |
| bodie_ | it seems that the testing unit doesn't have the charm URL in its doc, but I'm not sure how to get that where it needs to go. I'm using state.AddCustomCharm | 20:46 |
| perrito666 | ericsnow: ? | 20:53 |
| ericsnow | hey | 20:53 |
| katco | is anyone familiar with the dummy provider? | 20:56 |
| ericsnow | katco: other than that we use it in our tests, sorry no | 20:58 |
| katco | bleh | 20:58 |
| katco | why is -gocheck.vv not scrolling output for me. it's making it impossible to diagnose this | 21:04 |
| perrito666 | katco: the other day nate said that I should also -check.v=true to get that working | 21:05 |
| katco | perrito666: sweet let me try that | 21:05 |
| katco | nothing =| | 21:09 |
| katco | go test github.com/juju/juju/cmd/jujud/agent/... -check.v=true -check.vv=true -gocheck.f=TestManageEnvironV | 21:09 |
| katco | (note that i modified the name of that test to include a V to differentiate between other tests) | 21:10 |
| natefinch | katco: I see things are going as well now as when I left earlier | 21:12 |
| katco | natefinch: hey welcome back | 21:13 |
| katco | natefinch: i fixed some things but there is a test hanging somewhere and i can't get any output from go test so i can put tracers in | 21:13 |
| katco | natefinch: if i pepper in panics, i can figure out what's hanging, but c.Log, fmt.Println, doesn't show anything. and no amount of v's coerce gocheck into giving me anything. | 21:14 |
| natefinch | katco: I think you need to give it test.v before anything shows | 21:14 |
| katco | natefinch: yeah i did | 21:14 |
| natefinch | hmm | 21:14 |
| katco | natefinch: here are the changes: https://github.com/katco-/juju/commit/9e7758f27625a139ee8f8c87827278c2f8b18559 | 21:14 |
| katco | natefinch: and something is hanging in MachineSuite in cmd/jujud/agent/machine_test.go; i think it's TestManageEnviron | 21:15 |
| katco | natefinch: i'm toying with the idea of landing my refactoring changes, because it makes it cleaner to get a machineagent which may help this problem along | 21:16 |
| katco | natefinch: b/c right now it appears (???) to be hanging on a newAgent | 21:17 |
| natefinch | hmm the agent tests pass for me on your branch | 21:22 |
| katco | http://juju-ci.vapour.ws:8080/job/github-merge-juju/1694/console | 21:22 |
| katco | i'll submit again in the off chance it was a fluke, but i can't get them to pass on my machine now | 21:23 |
| natefinch | katco: your comment says net.Server was closing the stop channel already? I don't see how that's possible | 21:25 |
| katco | natefinch: where did i see that... | 21:27 |
| katco | natefinch: http://golang.org/pkg/net/#Conn | 21:29 |
| katco | natefinch: https://github.com/katco-/juju/commit/9e7758f27625a139ee8f8c87827278c2f8b18559#diff-19f910c28876da8a8d94937e102b2ebeR96 | 21:29 |
| natefinch | katco: yeah but that doesn't close the stop channel, which is what kills that goroutine. It'll just stick around forever... but I think the close of the channel should be there. | 21:31 |
| bodie_ | code freeze is today, yes? | 21:31 |
| bodie_ | I had a branch which fwereade requested in rather strong terms to be in the release | 21:32 |
| bodie_ | it seems there might have been a miscommunication with TheMue, I was traveling yesterday and thought we were in sync to get it landed | 21:32 |
| bodie_ | https://github.com/juju/juju/pull/1341 | 21:32 |
| katco | natefinch: sorry trying to remember my reasoning. with that in there, we definitely get a double close error | 21:33 |
| bodie_ | this is important to me if it's still possible to land -- it's merely a few testing changes to master and the inclusion of a new charm version which includes a MUCH cleaner ux for Actions | 21:33 |
| natefinch | katco: it's probably the Close() method on changeCertListener getting called more than once. | 21:33 |
| * bodie_ nudges natefinch in the hopes of getting a pair of in-crowd eyes on this... | 21:33 | |
| katco | natefinch: tomb was calling it, and then i though the listener | 21:33 |
| natefinch | katco: but right now nothing is going to be closing the channel | 21:34 |
| katco | bodie_: trunk is blocked on this bug =/ | 21:34 |
| natefinch | and unfortunately I have to run.... | 21:34 |
| bodie_ | gotya, does that mean this can't land before freeze or what should I expect? | 21:34 |
| bodie_ | or who should I talk to? | 21:35 |
| natefinch | bodie_: no idea. If we want it in, it'll get im/ | 21:35 |
| natefinch | in | 21:35 |
| katco | bodie_: i've been trying my best to open trunk, don't know if i'll succeed. i'm running into issues | 21:35 |
| bodie_ | okey doke | 21:35 |
| natefinch | bodie_: I've never met a code freeze that wasn't more of a slushy | 21:35 |
| bodie_ | the addition of pressure lowers the freezing point of code :P | 21:36 |
| natefinch | lol exactly | 21:36 |
| natefinch | ok gotta run before I get divorced | 21:36 |
| bodie_ | lol XD | 21:36 |
| bodie_ | katco, lmk if I can pitch in anywhere -- I'm with my family, but if it's crucial I'm sure I can squeeze in a couple of hours | 21:37 |
| katco | bodie_: will do... reevaluating what nate said | 21:37 |
| katco | bodie_: do you know what to do when CI runs out of space? | 21:48 |
| katco | mgz_: ^^ on the off chance you're still on | 21:48 |
| perrito666 | ohh cmooon again? | 21:48 |
| perrito666 | CI was out of space this AM | 21:48 |
| katco | =/ | 21:48 |
| mgz_ | katco: if you mean the landing job, see the log | 21:51 |
| katco | perrito666: what fixes it? | 21:51 |
| mgz_ | it's an ephemeral image just for the tests | 21:51 |
| mgz_ | it doesn't run out of space unless the instance happens to be bad, or the tests fail horribly enough to fill /tmp | 21:51 |
| mgz_ | *instance | 21:51 |
| katco | mgz_: ah ok. so then a fresh tests gets a new image? | 21:52 |
| perrito666 | well that was failing this am already | 21:52 |
| mgz_ | katco: yes | 21:52 |
| katco | mgz_: ok that makes sense | 21:52 |
| katco | mgz_: ty :) | 21:52 |
| mgz_ | perrito666: the same branch has been failing | 21:52 |
| mgz_ | land anything else it's almost certainly fine | 21:52 |
| * katco is hoping she finally has it fixed, running tests locally again atm | 21:52 | |
| bodie_ | mgz_, I made a tweak to charm which fwereade was really keen on, and opened a PR to fix testing in master | 22:11 |
| bodie_ | mgz_, I thought TheMue and I were in sync to get it landed yesterday, but I think we had a misunderstanding since I landed the charm changes but not the ones in juju core | 22:11 |
| bodie_ | mgz_, I'm wondering how I should make sure this is in 1.22 since it's a user-facing issue we really want to get in | 22:12 |
| bodie_ | I don't know who to talk to or what to do about this and it's making my skin crawl.. :S | 22:12 |
| katco | bodie_: i'm submitting another test run. the test is hanging on my machine, but there's no reason not to try it while i fiddle with things. if you want to help out, you can try testing cmd/jujud/agent/... and find out where it's hanging | 22:15 |
| katco | bodie_: sorry on this PR: https://github.com/juju/juju/pull/1347 | 22:15 |
| katco | bodie_: i cannot for the life of me get gocheck to feed me verbose output. | 22:16 |
| bodie_ | sure, let me give that a spin. can I get a quick glance over RB 661? it's really quite trivial | 22:17 |
| katco | bodie_: sure | 22:17 |
| perrito666 | ok fine gentlemen and ladies, EOD | 22:20 |
| katco | bodie_: not exactly sure what i'm looking at. does the new version of charm change the outfile key to snapshot? | 22:20 |
| perrito666 | have all a nice end of year and overall holidays whatever you belief is | 22:20 |
| katco | perrito666: happy holidays, tc | 22:20 |
| bodie_ | katco, basically it just reduces the boilerplate for actions.yaml | 22:22 |
| bodie_ | katco, the change to do it is already landed in charm; this just updates master for the required testing changes | 22:22 |
| katco | bodie_: seems benign. lgtm | 22:23 |
| bodie_ | ^_^ | 22:23 |
| mgz_ | bodie_: does it have an associated bug on launchpad? | 22:23 |
| katco | bodie_: especially since it is tweaking the tests | 22:23 |
| mgz_ | bodie_: if so, target it to the milestone, otherwise you can file a bug to go along with maybe | 22:23 |
| bodie_ | that sounds like a good option | 22:23 |
| bodie_ | thanks mgz_ | 22:24 |
| bodie_ | there's not a bug | 22:24 |
| katco | mgz_: ty for being so helpful past your EOD the last day of the year | 22:24 |
| mgz_ | :) | 22:24 |
| bodie_ | indeed | 22:25 |
| fwereade | katco, driveby: `go test <whatever> -test.v -gocheck.vv` should give you enough? | 22:26 |
| katco | fwereade: ty, but this gives me nothing until the process times out: go test -v github.com/juju/juju/cmd/jujud/agent/... -check.v=true -check.vv=true | 22:26 |
| katco | bodie_: are you experiencing the same issue with the test hanging? | 22:32 |
| * bodie_ hails fwereade -- https://bugs.launchpad.net/juju-core/+bug/1404397 | 22:32 | |
| mup | Bug #1404397: charm actions.yaml simplification breaks master tests <actions> <charm> <juju-core:New> <https://launchpad.net/bugs/1404397> | 22:32 |
| bodie_ | katco, just taking a crack at it after opening the bug | 22:32 |
| katco | bodie_: no worries, thank you | 22:33 |
| bodie_ | katco, go version? | 22:37 |
| bodie_ | I don't suppose it makes a difference since CI isn't taking it | 22:37 |
| katco | bodie_: 1.3.3 | 22:38 |
| katco | bodie_: i made 1 change since i resubmitted; we'll see if this latest version works out | 22:38 |
| * katco sighs | 22:45 | |
| katco | the CI test is now failing in api/watcher | 22:46 |
| bodie_ | O.o | 22:47 |
| katco | that seems fairly random to me | 22:47 |
| katco | it timed out | 22:47 |
| bodie_ | katco, fetching but not seeing your change | 22:48 |
| katco | it's an ammendment | 22:49 |
| bodie_ | ah | 22:49 |
| bodie_ | katco, api/watcher/... passes for me on your branch | 22:52 |
| bodie_ | let's see what happens when I do a whole-hog juju test | 22:52 |
| katco | bodie_: i also resubmitted the job. again. | 22:53 |
| katco | bodie_: it hadn't failed in those places yet. provider/dummy also failed | 22:53 |
| katco | bodie_: which is currently hung, but with the output i couldn't get from cmd/jujud/agent | 22:54 |
| katco | either i really didn't get enough sleep last night or this is all fairly random | 22:56 |
| bodie_ | you just haven't sacrificed to the loa recently | 22:59 |
| katco | lol | 23:00 |
| bodie_ | fwiw, my laptop is just about exactly as fast as the CI server.... | 23:01 |
| katco | lol | 23:02 |
| katco | if this fails i'm going to EOD. i was on very late last night and i don't think i'll solve this | 23:02 |
| bodie_ | so, on that note, my anemic laptop is well past api/watcher | 23:06 |
| katco | and CI is still on the package after api/watcher | 23:07 |
| bodie_ | oh and there we go. something blew up | 23:07 |
| katco | i think it's going to time out | 23:07 |
| bodie_ | test ran too long on mine. | 23:08 |
| bodie_ | :/ | 23:08 |
| bodie_ | I'll give it a whack on the digitalocean 8-core instance | 23:08 |
| katco | where did you time-out? | 23:11 |
| bodie_ | still running, but here's the output so far | 23:11 |
| bodie_ | not sure why it's still running come to think of it | 23:11 |
| bodie_ | looks like apiserver, but not really obvious where or why | 23:13 |
| bodie_ | http://paste.ubuntu.com/9574300/ | 23:13 |
| katco | that's where CI is going to hang. there's a bug for sure. it's where the changes are centered | 23:13 |
| katco | yup CI has failed | 23:22 |
| bodie_ | katco, my hunch is that whatever was supposed to release the semaphore hold didn't | 23:24 |
| bodie_ | that might be obvious | 23:25 |
| katco | it's definitely something along those lines | 23:25 |
| katco | i am too fried to debug threading atm -.- | 23:25 |
| katco | so i'm going to EOD. sorry i couldn't get trunk unblocked | 23:26 |
| bodie_ | it seems there are many goroutines which were in a semacquire state at time of death | 23:26 |
| bodie_ | take care :) and merry christmas / happy festivus | 23:26 |
| katco | happy solstice in my case :) | 23:26 |
| katco | but happy holidays to you | 23:26 |
| bodie_ | perhaps it would be useful in debugging to keep track somehow of who has the lock | 23:30 |
| bodie_ | I see only three goroutines which were killed while waiting for lock in apiserver.(*changeCertListener).Close | 23:31 |
| bodie_ | which traces to http://golang.org/src/sync/mutex.go line 66 | 23:32 |
| bodie_ | fwiw | 23:32 |
| bodie_ | but that's not where the problem is; that's merely the symptom | 23:32 |
| bodie_ | whoever had the lock perhaps was waiting on io | 23:32 |
| bodie_ | and there are an enormous number of IO Wait-ing goroutines | 23:33 |
| bodie_ | perhaps the cert was never sent | 23:33 |
| bodie_ | I sense a perturbance in jujud/agent/upgrade_test.go line newApiserverWorker line 493 (nil channel) | 23:38 |
| bodie_ | katco ^ hope that helps | 23:38 |
| bodie_ | but I'm not certain what the intent is | 23:38 |
| bodie_ | EOD/holiday for me, take care all! | 23:38 |
| * bodie_ would like someone to vet bug 1404397 since it's a big Actions usability improvement | 23:40 | |
| mup | Bug #1404397: charm actions.yaml simplification breaks master tests <actions> <charm> <juju-core:New for binary132> <https://launchpad.net/bugs/1404397> | 23:40 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!