| mup | Bug #1517258 opened: juju 1.24.7 precise: container failed to start and was destroyed <oil> <juju-core:New> <https://launchpad.net/bugs/1517258> | 00:13 |
|---|---|---|
| anastasiamac | waigani: tyvm for review \o/ | 00:17 |
| anastasiamac | waigani: i'll prefix the PR with WIP and will add api/apiserver logic to it :D | 00:17 |
| alexisb | waigani, did you use your new tool for the review? | 00:37 |
| anastasiamac | alexisb: waigani: new tool? | 00:40 |
| wwitzel3 | wallyworld: ping | 00:58 |
| wallyworld | hey | 00:59 |
| wwitzel3 | hey, hangout? | 00:59 |
| wallyworld | ok | 00:59 |
| wwitzel3 | wallyworld: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1 | 00:59 |
| mgz | soooo... how do people actually feel about table tests these days? | 01:37 |
| mgz | I tried it as more-idiomatic go thing, but I might just be out of date | 01:38 |
| davecheney | mgz: i think table driven tests are ace | 01:42 |
| mgz | davecheney: do you normally have two different structs for error cases and non-error cases? | 01:44 |
| davecheney | depends | 01:44 |
| davecheney | i normally add an err error field to the test table | 01:45 |
| davecheney | and then you just test that the actual error and the erro int he test table is the same | 01:45 |
| davecheney | for most cases it lets me combine success and fail cases | 01:45 |
| mgz | hm, issue is I need ErrorMatches | 01:48 |
| mgz | as I'm testing an error that includes at the bottom level a url.Parse error string I don't want to assert on the specifics of | 01:48 |
| davecheney | do you care what error it is | 01:48 |
| davecheney | or just the presence of an error | 01:48 |
| mgz | nope. | 01:48 |
| davecheney | got, err :=f () | 01:49 |
| davecheney | if t.err != err { // oops } | 01:49 |
| davecheney | should do | 01:49 |
| mgz | well, I have two error paths, so I care a bot | 01:49 |
| mgz | *bit | 01:49 |
| mgz | davecheney: exact code in cinder_test.go http://reviews.vapour.ws/r/3170/ | 01:51 |
| davecheney | mgz: lgtm | 01:55 |
| davecheney | that's how id do it with gocheck | 01:55 |
| davecheney | who want's to look at the weirdest data race report | 02:02 |
| davecheney | http://paste.ubuntu.com/13322899/ | 02:02 |
| davecheney | func (fakeAssignCaller) BestFacadeVersion(facade string) int { return 1 | 02:03 |
| davecheney | } | 02:03 |
| === natefinch-afk is now known as natefinch | ||
| natefinch | davecheney: derp? | 02:04 |
| davecheney | sure, most things are human error | 02:05 |
| davecheney | but how can there be a data race on a function that never touches any data ? | 02:05 |
| natefinch | it even leaves out the receiver variable... | 02:06 |
| davecheney | why does it need the receiver, it alwasy returns 1 | 02:07 |
| perrito666 | davecheney: gocheck? | 02:07 |
| davecheney | nope | 02:07 |
| perrito666 | ok, I am curious now | 02:10 |
| davecheney | http://reviews.vapour.ws/r/3174/ | 02:10 |
| natefinch | some sort of pointer auto-dereference thing? | 02:10 |
| davecheney | have a look | 02:10 |
| davecheney | natefinch: if I have a *T and I need to call a method that takes a T, how does the compiler do that ? | 02:11 |
| perrito666 | interesting | 02:12 |
| natefinch | davecheney: it dereferences the pointer, which is what I was thinking... | 02:12 |
| davecheney | so why does that lead to a data race ? | 02:12 |
| davecheney | Read by goroutine 12: | 02:13 |
| davecheney | github.com/juju/juju/api/unitassigner.(*fakeWatchCaller).BestFacadeVersion() | 02:13 |
| davecheney | <autogenerated>:18 +0xa9 | 02:13 |
| davecheney | Previous write by goroutine 11: | 02:13 |
| davecheney | sync/atomic.CompareAndSwapInt32() | 02:13 |
| davecheney | /home/dfc/go1.4/src/runtime/race_amd64.s:281 +0xc | 02:13 |
| davecheney | sync.(*Mutex).Lock() | 02:13 |
| davecheney | i'm going to have to blog this one | 02:13 |
| natefinch | it's copying the value | 02:13 |
| natefinch | even though we're not even using the value | 02:14 |
| natefinch | and so that counts as being "read", and in other other goroutine it's changing that value... so what gets copied depends on the order in which those two things happen.... even though we don't care. | 02:15 |
| perrito666 | I would have guessed that it was compiled as a function rather than a method when the receiver is not assigned | 02:15 |
| natefinch | I definitely would have hoped that the compiler would figure out not to copy the value and therefore not trigger the race detector... however, it seems like it must not be that smart.... or the race detector's not that smart | 02:17 |
| natefinch | The interesting thing then is, it seems like you should _always_ use *T for methods that don't need the receiver, to avoid exactly this problem. | 02:18 |
| perrito666 | natefinch: methods that dont need a receiver are called functions :p | 02:18 |
| natefinch | perrito666: except it has to be a method to satisfy an interface | 02:19 |
| davecheney | natefinch: bingo | 02:19 |
| davecheney | natefinch: an observation in gopl was if you use pointer receivers for some methods, you shuld probably use them for all methods | 02:22 |
| davecheney | and vice versa | 02:22 |
| natefinch | davecheney: I've read that, and I wasn't sure if I agreed... sometimes value receivers are nice to both point out that the value isn't being modified, as well as enforce that fact. Sort of immutability-lite ... but certainly with gotchas like this, it certainly is problematic. | 02:24 |
| davecheney | natefinch: yeah, i'm still undecided | 02:34 |
| davecheney | http://play.golang.org/p/DTDzYu4b3C | 02:34 |
| davecheney | here is a very small repro that shows the problem | 02:34 |
| davecheney | no mutex | 02:34 |
| davecheney | no sync atomic | 02:35 |
| wallyworld | wwitzel3: where are the logs etc? i can't see any in /var/log/juju | 02:38 |
| natefinch | davecheney: nice simple repro. Do you know if it's the compiler's fault or the race detector's? | 02:56 |
| natefinch | fantasitc... when CI tests fail, they don't bring down all the machines they created :/ | 03:10 |
| natefinch | someday I'll get a clean run of the CI tests and maybe actually repo this bug | 03:13 |
| mgz | natefinch: this is an issue with the way this fails | 03:24 |
| mgz | natefinch: you call `juju destroy-environment` on an env you've just deployed machines for, but have the machines have not yet come up, juju leaks the machines | 03:24 |
| natefinch | mgz: even with --force? | 03:24 |
| natefinch | I guess --force shouldn't matter.... but still... ouch, damn. | 03:25 |
| natefinch | mgz: I think this is a pass? https://pastebin.canonical.com/144311/ | 03:28 |
| natefinch | would be nice if it said "PASS" or something at the end, rather than making me guess :/ | 03:29 |
| mgz | natefinch: yeah, looks okay, `echo $?` | 03:30 |
| natefinch | 0 | 03:31 |
| mgz | that's a pass. | 03:31 |
| natefinch | I didn't change anything past menn0's fix | 03:31 |
| natefinch | granted, it's on EC2 not joyent, but... | 03:32 |
| * natefinch shrugs | 03:33 | |
| natefinch | works on my machine :/ | 03:33 |
| mgz | natefinch: you can use the joyent creds from lp:cloud-city | 03:36 |
| natefinch | mgz: k, thanks | 03:36 |
| menn0 | natefinch, mgz: I found it work with the local provider and EC2 as well | 03:36 |
| menn0 | so it's beginning to look joyent specific which is weird | 03:37 |
| natefinch | menn0: probably a timing issue | 03:37 |
| natefinch | probably joyent is faster or slower in some specific part of the script | 03:37 |
| menn0 | natefinch: agreed, that's likely | 03:37 |
| mgz | I suspect you just aren't seeing a race, :12 at last deploy, :19 before status is first called in your log | 03:37 |
| mgz | probably long enough for EC2 to allocate instance ids for both machines | 03:38 |
| natefinch | yep | 03:38 |
| mgz | :09 at deploy on CI to :12 before first status | 03:39 |
| natefinch | mgz: the default-joyent creds in the environments.yaml in cloud-city? | 03:39 |
| natefinch | s/creds/environment | 03:39 |
| mgz | natefinch: yup | 03:40 |
| natefinch | mgz: btw, I hope you're not actually in the UK at this hour | 03:40 |
| mgz | or parallel-joyent and use the exact stream that the test did | 03:40 |
| mgz | natefinch: nah, NY | 03:40 |
| natefinch | mgz: ahh, good. Whatcha doing on this side of the pond? | 03:41 |
| mgz | sprint, talking about QA across all of CDO products | 03:42 |
| natefinch | neat | 03:42 |
| mgz | waigani_: updated r/3170 - thoughts welcome | 03:46 |
| natefinch | menn0: you said you hacked the script to do an upload tools, care to show me where I'd do that? | 03:48 |
| natefinch | or mgz would you know how I could best do that? ^ | 03:51 |
| mgz | natefinch: you can just append --upload-tools | 03:51 |
| mgz | oh wait, jes is still special? | 03:52 |
| mgz | --upload-tools doesn't work with jes, or at least didn't at the time the test was written | 03:53 |
| natefinch | wait, what, really? | 03:56 |
| natefinch | ffs | 03:57 |
| natefinch | how the hell do you test if you can't upload tools? | 03:57 |
| menn0 | mgz: i'm not aware of a limitation with upload-tools and jes | 04:03 |
| menn0 | but with joyent there is a routing problem | 04:03 |
| menn0 | new instances often can't get to the state server's internal IP address | 04:03 |
| menn0 | so they can't download the tools from the state server | 04:04 |
| menn0 | I ran into that with my testing | 04:06 |
| menn0 | wallyworld: where do instances get tools from during provisioning? always the state server or maybe directly from streams? | 04:07 |
| waigani_ | mgz: +1 to how you've got the tests now. LGTM | 04:09 |
| mgz | menn0: yeah, we have specialy hackery in CI to work around the fact the provider can't clean up networks | 04:15 |
| mgz | menn0: see http://juju-ci.vapour.ws/job/joyent-deploy-jes-trusty-amd64/configure | 04:16 |
| mgz | $RELEASE_TOOLS/joyent-curl.bash /cpcjoyentsupport/fwrules | sed -e 's/[\[\{]/\n\0/g;' | grep $JOB_NAME | sed -e 's/.*"id":"\([^"]*\)".*/\1/' | xargs -I{} $RELEASE_TOOLS/joyent-curl.bash /cpcjoyentsupport/fwrules/{} -X DELETE || true | 04:16 |
| mgz | waigani_: thanks | 04:16 |
| mgz | menn0: I'm failing to remember why --upload-tools didn't work, just remember I disabled it on abentley's say-so | 04:19 |
| mgz | natefinch: there's a one-line change you can make in utility.py that will enable --upload-tools again | 04:21 |
| mgz | well, and a dedent. | 04:22 |
| mup | Bug #1497316 changed: TestUniterSteadyStateUpgrade permission problem <ci> <intermittent-failure> <windows> <juju-core:Expired> <https://launchpad.net/bugs/1497316> | 04:23 |
| menn0 | mgz: what about changing the call to boot_context() in assess_jes_deploy.py | 04:30 |
| menn0 | it takes an uploadTools arg | 04:30 |
| menn0 | (hardcoded to false) | 04:30 |
| mgz | ah, pants, yeah, that too | 04:31 |
| mup | Bug #1497316 opened: TestUniterSteadyStateUpgrade permission problem <ci> <intermittent-failure> <windows> <juju-core:Expired> <https://launchpad.net/bugs/1497316> | 04:32 |
| natefinch | mgz: seems easy enough. I think I have it, with the additional change to make False args.upload_tools in boot_context | 04:38 |
| mup | Bug #1497316 changed: TestUniterSteadyStateUpgrade permission problem <ci> <intermittent-failure> <windows> <juju-core:Expired> <https://launchpad.net/bugs/1497316> | 04:38 |
| natefinch | er in the call to boot_context | 04:38 |
| * natefinch tries it out | 04:38 | |
| mgz | okay, http://reviews.vapour.ws/r/3172/ should be good to go | 04:38 |
| mup | Bug #1497316 opened: TestUniterSteadyStateUpgrade permission problem <ci> <intermittent-failure> <windows> <juju-core:Expired> <https://launchpad.net/bugs/1497316> | 04:41 |
| mup | Bug #1497316 changed: TestUniterSteadyStateUpgrade permission problem <ci> <intermittent-failure> <windows> <juju-core:Expired> <https://launchpad.net/bugs/1497316> | 04:47 |
| natefinch | hmm yeah, not working... getting no tools found when doing create-environment. | 04:52 |
| wallyworld | menn0: sorry, was out at 1:1. during provisioning we go to state server | 04:55 |
| wallyworld | menn0: cloud init is configured to try all the recorded api host addresses to till connections | 04:55 |
| menn0 | wallyworld: ok thanks | 04:56 |
| menn0 | mgz, natefinch: so I must have been running into joyent firewall issues then. I frequently saw new instances that couldn't connect to the state server to get tools. | 04:56 |
| menn0 | natefinch: you might want to make sure you remove the rules as per the CI job | 04:57 |
| natefinch | menn0: this is running the CI test... | 04:58 |
| menn0 | natefinch: yes, but the CI job that runs assess_jes_deploy.py does some stuff before and after it runs too | 04:58 |
| natefinch | ahh | 04:58 |
| menn0 | natefinch: see the job's config in jenkins | 04:58 |
| menn0 | natefinch: or the long command mgz mentioned earlier | 04:59 |
| menn0 | natefinch: that's from the CI job | 04:59 |
| natefinch | menn0: gross | 05:01 |
| menn0 | natefinch: it's not exactly beautfiul no | 05:01 |
| wwitzel3 | wallyworld: if you juju ssh 0 , you'll see all the logs | 05:02 |
| wallyworld | oh, that's not the bootstrap machine, doh | 05:03 |
| wwitzel3 | wallyworld: nope, that is the client machine, in the home directory there is the 1.26-alpah1 client too | 05:04 |
| wwitzel3 | wallyworld: the client on the path is 1.22 | 05:04 |
| wallyworld | ok | 05:04 |
| mgz | bug 1451104 could just be fixed in the joyent provider, then CI hackery goes away | 05:07 |
| mup | Bug #1451104: Joyent machines can fail to fetch tools <ci> <deploy> <joyent-provider> <reliability> <juju-core:Triaged> <https://launchpad.net/bugs/1451104> | 05:07 |
| mgz | bug 1485781 is more specific | 05:08 |
| mup | Bug #1485781: Juju is unreliable on Joyent <joyent-provider> <reliability> <repeatability> <juju-core:Triaged> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1485781> | 05:08 |
| natefinch | le sigh | 05:08 |
| davecheney | i read that as "joyent is unreliable on joyent" | 05:09 |
| davecheney | it had a nice ring to it | 05:10 |
| natefinch | no, it's "CI only really works in CI" | 05:11 |
| natefinch | there's just like 100 implicit dependencies that need to be perfectly correct to get things to run | 05:12 |
| davecheney | works on my cloud(tm) | 05:12 |
| wallyworld | wwitzel3: the data being logged does not match what is on streams.canonical.com | 05:19 |
| wallyworld | maybe they have squid or something serving stale data | 05:20 |
| wwitzel3 | wallyworld: but that shouldn't prevent upload-tools working? | 05:24 |
| wallyworld | true, i was looking to see why a normal upgrade fials | 05:24 |
| natefinch | mgz: what's $JOB_NAME? | 05:25 |
| mgz | natefinch: what you're passing as forth arg to assess_jes.py | 05:27 |
| mgz | +u | 05:27 |
| natefinch | mgz: so the copied environment name? | 05:31 |
| mgz | well, ideally something unique, the idea is jobs don't stomp on each other if running at the same time | 05:32 |
| wallyworld | wwitzel3: aha | 05:33 |
| wallyworld | the tools metadata is fucked | 05:33 |
| wallyworld | incorrectly generated | 05:33 |
| wallyworld | this has happened before :-( | 05:33 |
| wwitzel3 | wallyworld: can we use juju-metadata to generate new data? | 05:34 |
| wallyworld | no, it needs to be signed | 05:35 |
| wallyworld | it's a CPC issue | 05:35 |
| wwitzel3 | wallyworld: and this is what is impacting even with upload-tools? | 05:35 |
| wallyworld | not sure, that's next on the list to look at | 05:36 |
| wallyworld | this is a serious, serious issue | 05:36 |
| wwitzel3 | wallyworld: ah, ok | 05:36 |
| mgz | wallyworld: which streams? | 05:37 |
| wallyworld | devel, maybe others, haven't checked | 05:37 |
| natefinch | still no tools found... | 05:37 |
| natefinch | whelp. It's after midnight and well past bed time. I guess I won't be fixing this today. | 05:37 |
| wallyworld | mgz: all sreams seem affected | 05:39 |
| === mwenning is now known as mwenning-afk | ||
| mup | Bug #1482155 changed: lxc restriction on multiple state servers <ha> <lxc> <state-server> <juju-core:Invalid> <https://launchpad.net/bugs/1482155> | 05:47 |
| mup | Bug #1482155 opened: lxc restriction on multiple state servers <ha> <lxc> <state-server> <juju-core:Invalid> <https://launchpad.net/bugs/1482155> | 05:50 |
| mup | Bug #1482155 changed: lxc restriction on multiple state servers <ha> <lxc> <state-server> <juju-core:Invalid> <https://launchpad.net/bugs/1482155> | 05:53 |
| mgz | wallyworld: so... don't think we actually need cpc, probably from a lp:juju-release-tools change, so we're just sitting on their hardware | 06:01 |
| wallyworld | ok, so we can maybe fix? | 06:01 |
| wallyworld | or rollback | 06:02 |
| mgz | yeah, but it's also night in canada | 06:02 |
| wallyworld | SOP \o/ | 06:04 |
| wallyworld | SPOF | 06:04 |
| mup | Bug #1517344 opened: state: initially assigned units don't get storage attachments <juju-core:Triaged> <https://launchpad.net/bugs/1517344> | 07:35 |
| menn0 | everything is awesome | 08:57 |
| menn0 | everything is awesome when you're part of a team | 08:57 |
| dimitern | menn0, :) are you talking about the blocker bug? | 08:58 |
| menn0 | not specifically | 08:58 |
| menn0 | just slightly delirious | 08:58 |
| menn0 | and that song is stuck in my head | 08:58 |
| dimitern | menn0, you should get some rest then ;) | 08:59 |
| menn0 | dimitern: nah... perfect time to write some more code | 09:00 |
| menn0 | ;-) | 09:00 |
| dimitern | menn0, I know the feeling | 09:00 |
| menn0 | fwereade: review please: http://reviews.vapour.ws/r/3178/ | 09:13 |
| fwereade | menn0, ack | 09:14 |
| fwereade | menn0, and now that's stuck in my head too | 09:14 |
| menn0 | you're welcome | 09:14 |
| menn0 | ;-) | 09:14 |
| dimitern | voidspace, dooferlad, frobware, guys, please have a look at http://reviews.vapour.ws/r/3167/ when you can | 09:32 |
| voidspace | dimitern: looking | 09:36 |
| voidspace | dimitern: good to read this code as I don't (yet) the concepts fully | 09:36 |
| dimitern | voidspace, cheers - I'm glad to clarify things where needed | 09:39 |
| voidspace | dimitern: straightforward so far | 09:50 |
| frobware | dimitern, added some comments; also (for standup) why do we replace " " in space names? | 09:56 |
| dimitern | frobware, thanks! spaces are not valid in constraints | 09:58 |
| frobware | dimitern, so why do we collapse them? if you pass a name with " " isn't that just invalid? | 10:00 |
| voidspace | maas space names can have spaces | 10:01 |
| voidspace | we'll have to translate between juju space names and maas names | 10:01 |
| dimitern | but not juju space names | 10:01 |
| dimitern | fwereade, jam, standup? | 10:02 |
| mup | Bug #1517391 opened: MachineStorageIdsWatcher severely undertested <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1517391> | 10:09 |
| mattyw | anastasiamac, are you still awake? | 10:30 |
| mattyw | anastasiamac, I guess you shouldn't be really | 10:30 |
| anastasiamac | mattyw: hi o/ | 11:13 |
| mattyw | anastasiamac, hey hey. I see http://reviews.vapour.ws/r/3171/ is marked WIP - do you still want reviews or should I leave it for now? | 11:14 |
| anastasiamac | mattyw: tyvm for asking - plz leave it for now :D | 11:14 |
| mattyw | anastasiamac, will do | 11:15 |
| anastasiamac | mattyw: \o/ | 11:15 |
| mattyw | anastasiamac, feel free to ping me or anyone else from emerald squad with reviews for x-model relations | 11:16 |
| anastasiamac | mattyw: gr8 idea - will do :) | 11:17 |
| === blahdeblah_ is now known as blahdeblah | ||
| mup | Bug #1517428 opened: state depends on multiwatcher and/or params <tech-debt> <juju-core:Triaged> <https://launchpad.net/bugs/1517428> | 12:12 |
| voidspace | dimitern: ping | 12:18 |
| dimitern | voidspace, pong | 12:18 |
| voidspace | dimitern: your PR - parseDelimitedValues | 12:18 |
| voidspace | dimitern: where does rawValues come from? | 12:18 |
| voidspace | dimitern: is that from the original command line invocation? | 12:19 |
| voidspace | dimitern: I wonder why we're doing the " " stripping inside the provider | 12:19 |
| voidspace | dimitern: we should have already converted from juju space names to maas space names by here | 12:19 |
| voidspace | dimitern: so they *might* have valid spaces in them | 12:19 |
| dimitern | voidspace, the raw values come from constraints | 12:19 |
| voidspace | dimitern: if they're space names then they need converting to maas names | 12:20 |
| voidspace | dimitern: and the provider can't do that as it requires access to state | 12:20 |
| voidspace | dimitern: and constraints will use juju names | 12:20 |
| dimitern | voidspace, I guess I overengineered it a bit - I'll drop the space stripping and that will simplify it - there's no way we get invalid constraints that far into the provisioning process | 12:20 |
| voidspace | dimitern: ok | 12:20 |
| voidspace | dimitern: maas space names can have spaces in them - so stripping them is wrong | 12:21 |
| voidspace | dimitern: I'll add an issue | 12:21 |
| dimitern | voidspace, the conversion needs to happen, but not having it now won't block us from possibly doing the demo | 12:21 |
| voidspace | dimitern: but the conversion needs to happen in a different place, not in the provider code | 12:22 |
| dimitern | voidspace, yeah, that's true | 12:22 |
| voidspace | dimitern: the provider code should only be dealing with MAAS names (or ids) | 12:22 |
| dimitern | voidspace, I think it will happen in the provisioner | 12:22 |
| voidspace | cool | 12:23 |
| voidspace | dimitern: frobware: dooferlad: just to mention that I'm off tomorrow, it's in the calendar. | 12:31 |
| dimitern | voidspace, have a good one ;) | 12:31 |
| voidspace | dimitern: I'm spending the day with my mum, visiting stratford | 12:32 |
| voidspace | should be good | 12:32 |
| dimitern | after some minor but required CI surgery on the parallel streams job, I'm happy to report we now have a blessed master!! | 12:38 |
| dimitern | don't rush all at once to land your stuff :P | 12:39 |
| dimitern | voidspace, I've re-queued your PR #3692 for merging as well as all 3 of mine | 12:51 |
| mup | Bug #1516144 changed: Cannot deploy charms in jes envs <blocker> <charms> <ci> <regression> <juju-core:Fix Released by menno.smits> <https://launchpad.net/bugs/1516144> | 12:51 |
| dimitern | hopefully no conflicts will emerge | 12:52 |
| mattyw | dimitern, ping? | 13:02 |
| dimitern | mattyw, pong | 13:04 |
| mattyw | dimitern, you know maas, are you able to respond to this guy? https://github.com/juju/juju/issues/3627 | 13:04 |
| dimitern | mattyw, I have no idea why that happens unfortunately :/ | 13:05 |
| frobware | voidspace, ack | 13:27 |
| frobware | dimitern, you left mine out... sniff. :) | 13:28 |
| mup | Bug #1517474 opened: provider/ec2: don't artificially limit EBS volumes to xvdf-xvdp <juju-core:Triaged> <https://launchpad.net/bugs/1517474> | 13:30 |
| jam | fwereade: ping for planning meeting? | 13:33 |
| dimitern | frobware, sorry, should've checked :/ | 13:39 |
| dimitern | frobware, voidspace, I've updated http://reviews.vapour.ws/r/3167/ - can you have another look and approve it if it looks ok? | 13:40 |
| frobware | dimitern, taking a look | 13:40 |
| frobware | dimitern, glad we drooped the " " stripping | 13:40 |
| dimitern | frobware, yeah, now that I think about it, it was like this because the --networks argument to deploy (where these came from in an earlier PoC) was not well validated and/or rushed to a demo-able state | 13:42 |
| mgz | dimitern: I don't suppose you heard anything from wallyworld on the status of upgrade issue? he was concerned our streams were screwed, but I don't see any mailling list updates from him. | 13:42 |
| dimitern | mgz, which upgrade issue is that? | 13:45 |
| dimitern | frobware, once our PRs land, it's time to rebase maas-spaces onto master | 13:48 |
| frobware | dimitern, +1 | 13:49 |
| dimitern | mgz, was that about upgrade 1.20.x->1.24.4 not working unless with --version 1.24.4 ? | 13:50 |
| mgz | I think it was several layers of investigation into rt #85463 | 13:55 |
| mup | Bug #85463: [apport] evolution-exchange-storage crashed with SIGSEGV in e2k_restriction_unref() <evolution-exchange (Ubuntu):Invalid by desktop-bugs> <https://launchpad.net/bugs/85463> | 13:55 |
| mgz | not that mup | 13:55 |
| mattyw | katco, woohoow! | 13:56 |
| mattyw | katco, I've been denying myself the lxd provider till it lands in master | 13:57 |
| voidspace | dimitern: LGTM | 14:22 |
| dimitern | voidspace, cheers! | 14:22 |
| mup | Bug #1517499 opened: i/o timeout on bundle deployment <juju-core:New> <https://launchpad.net/bugs/1517499> | 15:01 |
| natefinch | axw: thanks for fixing that bug with unit assignment. Can't believe I forgot to remove the docs after successful assignment. I'm surprised that didn't show up more often in tests. | 15:13 |
| frobware | cherylj, having trouble with the bond0 interface and juju-br0 1516891. Will concentrate on landing 1512371 first | 15:28 |
| cherylj | frobware: okay, thanks for the update! | 15:29 |
| frobware | cherylj, this may be at the root of some problems - https://bugs.launchpad.net/ubuntu/+source/ifupdown/+bug/1337873 | 15:29 |
| mup | Bug #1337873: Precise, Trusty, Utopic - ifupdown initialization problems caused by race condition <cts> <patch> <ifupdown (Ubuntu):Fix Released by dgadomski> <ifupdown | 15:29 |
| mup | (Ubuntu Precise):Confirmed> <ifupdown (Ubuntu Trusty):Confirmed> <ifupdown (Ubuntu Vivid):Confirmed> <ifupdown (Debian):New> <https://launchpad.net/bugs/1337873> | 15:29 |
| cherylj | frobware: interesting. It's an old bug, but wasn't addressed until last month? | 15:38 |
| frobware | cherylj, I think there's a bit of work to address the bonding bug (for juju-br0) | 15:40 |
| cherylj | frobware: do you know if it's just a problem with MAAS 1.9? | 15:41 |
| frobware | cherylj, I don't. | 15:42 |
| cherylj | just re-read the bug, it looks like the support for bonding is introduced in 1.9 | 15:43 |
| cherylj | frobware: is there a workaround where users could manually edit /e/n/i? | 15:44 |
| frobware | cherylj, let me try. though I am getting distracted... :) | 15:44 |
| cherylj | frobware: thanks :) I'd feel more comfortable moving it to 1.25.2 if we could give people a manual workaround | 15:45 |
| natefinch | ericsnow: you around? | 15:58 |
| ericsnow | natefinch: briefly | 15:58 |
| frobware | cherylj, synthesizing what juju boostrap would do to /e/n/i and rebooting yields no working interfaces. \o/ ... :( | 15:59 |
| frobware | cherylj, so, not sure there's even a quick workaround right now. | 15:59 |
| cherylj | frobware: yikes!! | 15:59 |
| natefinch | ericsnow: I don't think your fix for the OOM error is actually changing the core behavior... you're running the waits in goroutines, but you're starting all the executables in what is effectively a tight loop... since Cmd.Start() just fires off a goroutine | 15:59 |
| natefinch | ericsnow: er, maybe not a goroutine per se... but we'll still be starting processes really fast and could easily have 100 fire off in a second | 16:01 |
| ericsnow | natefinch: fork doesn't take long | 16:02 |
| ericsnow | natefinch: fork+exec I mean | 16:02 |
| ericsnow | natefinch: by the time Start completes it should already be exec'ing | 16:03 |
| natefinch | ericsnow: I guess I don't know when linux stops "counting" the memory of the fork against juju... or when it decides the fork doesn't really need that much. You're going to have 100 ssh processes all running at the same time if there are 100 machines to run on. | 16:06 |
| ericsnow | natefinch: once it execs | 16:06 |
| katco | cherylj: got a sec? | 16:07 |
| cherylj | katco: sure, what's up? | 16:07 |
| natefinch | ericsnow: so, once Cmd.Start returns, we're no longer being hurt by the memory the fork uses? | 16:07 |
| katco | cherylj: can you hop in https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1 | 16:07 |
| ericsnow | natefinch: correct | 16:08 |
| natefinch | ericsnow: interesting. ok, thanks. | 16:09 |
| ericsnow | natefinch: np | 16:09 |
| wwitzel3 | mgz: what do you mean isolate? I know if I bootstrap with 1.22.8, set the agent-stream to devel, and issue update-juju --version 1.26-alpha1 I get: no matching tools available | 16:10 |
| wwitzel3 | s/update/upgrade | 16:10 |
| mgz | wwitzel3: I mean set agent-metadata-url to a specific different location | 16:13 |
| wwitzel3 | mgz: sure, what location? | 16:14 |
| ericsnow | natefinch: FWIW, I don't think the 100 ssh procs will be a big problem (~50k each) | 16:14 |
| mgz | abentley: do you have bandwidth to work with wwitzel3 to look at the upgrade from 1.22.8 to devel streams? | 16:14 |
| natefinch | ericsnow: only a problem due to the memory copying thing from fork | 16:14 |
| ericsnow | natefinch: yep | 16:15 |
| abentley | mgz, wwitzel3: Yes, I can help out. I was just dealing with a similar issue 1.20.x | 16:16 |
| abentley | wwitzel3: hangout? | 16:16 |
| wwitzel3 | abentley: https://plus.google.com/hangouts/_/canonical.com/moonstone?authuser=1 | 16:17 |
| mup | Bug #1517535 opened: Agent stuck in "allocating" state <juju-core:New> <https://launchpad.net/bugs/1517535> | 16:25 |
| natefinch | katco: in case you missed it, I talked with ericsnow and he convinced me that his code actually is doing the right thing... it was just my lack of knowledge of how linux's memory handling works during fork & exec. | 16:26 |
| mup | Bug #1517535 changed: Agent stuck in "allocating" state <juju-core:New> <https://launchpad.net/bugs/1517535> | 16:31 |
| mup | Bug #1517535 opened: Agent stuck in "allocating" state <juju-core:New> <https://launchpad.net/bugs/1517535> | 16:34 |
| katco | natefinch: cool, looking forward to seeing it land | 16:34 |
| katco | mattyw: jujubot merged commit de417eb into master 20 seconds ago. it's in master now :) alpha2! | 16:44 |
| katco | natefinch: ericsnow: wwitzel3: lxd provider is now in master. good job team :D | 16:44 |
| mattyw | katco, natefinch ericsnow wwitzel3 good work folks - just try to stop me using the lxd provider now! | 16:45 |
| ericsnow | katco: \o/ | 16:45 |
| mgz | cherylj: bug 1512399 fix landed on 1.25 | 16:53 |
| mup | Bug #1512399: ERROR environment destruction failed: destroying storage: listing volumes: Get https://x.x.x.x:8776/v2/<UUID>/volumes/detail: local error: record overflow <amulet> <bug-squad> <openstack> <sts> <uosci> <Go OpenStack Exchange:Fix Released by gz> <juju-core:In Progress by gz> <juju-core | 16:53 |
| mup | 1.25:In Progress by gz> <https://launchpad.net/bugs/1512399> | 16:53 |
| cherylj | yay! thanks, mgz! | 16:53 |
| === jamesmil_ is now known as jamesmillerio | ||
| wwitzel3 | abentley: https://bugs.launchpad.net/juju-core/+bug/1507867/comments/23 figured out the reproduction steps | 17:10 |
| mup | Bug #1507867: juju upgrade failures <canonical-bootstack> <upgrade-juju> <juju-core:In Progress by wwitzel3> <https://launchpad.net/bugs/1507867> | 17:10 |
| wwitzel3 | abentley: seems code related, but thought you might be interested | 17:11 |
| abentley | wwitzel3: Yes, using --upload-tools breaks future upgrades. That is a known bug. sinzui? | 17:11 |
| wwitzel3 | abentley: yeah, but I thought setting the tools-url manually worked around it? | 17:12 |
| abentley | wwitzel3: Yes, I thought so, too. Interesting. | 17:13 |
| abentley | wwitzel3: Can I see the log? | 17:15 |
| abentley | wwitzel3: Can you upgrade to a non-devel version like 1.25.0? | 17:16 |
| wwitzel3 | abentley: nope | 17:17 |
| wwitzel3 | abentley: the log doesn't have much in it, even at debug | 17:17 |
| abentley | wwitzel3: Sometimes the "reading tools with major.minor version 1.22" line is informative. | 17:19 |
| wwitzel3 | abentley: that line does not exist in the log | 17:19 |
| abentley | wwitzel3: for an upgrade, the machine-0 log may also be useful, but you have to read it very damn closely. | 17:19 |
| wwitzel3 | abentley: I dont see any streams activity, like it isn't trying at all | 17:19 |
| mgz | is the lxd-provider documented anywhere at present? | 17:20 |
| katco | mgz: in the release notes; no documentation as of yet | 17:22 |
| mgz | katco: ta | 17:22 |
| alexisb | the idea of having both bundle deploy support and a lxd provider on a devel ppa juju makes me giddy :) | 17:25 |
| katco | alexisb: i know right? those 2 features together = love | 17:26 |
| abentley | wwitzel3: Here is the machine-0 log of a successful upgrade from 1.22 to master, if it is useful: http://data.vapour.ws/juju-ci/products/version-3327/aws-upgrade-22-trusty-amd64/build-376/machine-0.log.gz | 17:29 |
| natefinch | ericsnow: I see you're still posting on the juu-run bug. Do you think you'll have time to get that landed today? I was going to try to do the work to get it landed, but if you're doing it, I can move on to something else. | 17:42 |
| ericsnow | natefinch: I'm trying but quickly running out of time | 17:43 |
| ericsnow | natefinch: I'll let you know and hand it off if needed | 17:43 |
| stokachu | kadams54: got a PR for you for theblues pacakge | 17:43 |
| natefinch | ericsnow: ok. yeah, I had assumed you weren't going to have any time today, so was prepared to do it. Let me know when you know. | 17:44 |
| ericsnow | natefinch: k | 17:44 |
| ericsnow | natefinch: will be soon | 17:44 |
| stokachu | 21 | 17:45 |
| kadams54 | stokachu: Taking a look | 17:48 |
| natefinch | fwereade: btw, really like your point about passing an abort channel rather than a timeout value. way more flexible, and still trivial to use as a timeout. | 18:11 |
| fwereade | natefinch, cool, thanks :) | 18:11 |
| wwitzel3 | abentley: was that after using upload tools? | 18:23 |
| abentley | wwitzel3: no, that was our standard test. http://reports.vapour.ws/releases/3327/job/aws-upgrade-22-trusty-amd64/attempt/376 | 18:34 |
| wwitzel3 | abentley: yeah, the standard test works for me as well | 18:35 |
| natefinch | katco: I'm going to go back to bug #1491688 for now, since ericsnow seems to still be working on his bug | 18:51 |
| mup | Bug #1491688: all-machine logging stopped, x509: certificate signed by unknown authority <bug-squad> <landscape> <logging> <rsyslog> <sts> <juju-core:Triaged> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1491688> | 18:51 |
| katco | natefinch: oh, ok. sorry, didn't think ericsnow would be around today | 18:51 |
| natefinch | katco: me either :) | 18:52 |
| ericsnow | katco: leaving in a few minutes | 18:52 |
| katco | ericsnow: ok. do you think you'll get this in by EOD? | 18:53 |
| ericsnow | katco: was going to hand it off to natefinch soon | 18:53 |
| katco | ericsnow: natefinch: is it possible for you two to pair for the remaining time? | 18:54 |
| ericsnow | katco: made the mistake of reading fwereade's review comments :) | 18:54 |
| katco | natefinch: i don't want you to get whiplash | 18:54 |
| natefinch | katco: I gotta pick up my daughter from preschool now, unfortunately | 18:56 |
| katco | natefinch: ok | 18:57 |
| katco | ericsnow: please send nate an email with all the info he needs for the hand-off. we need this landed by tonight | 18:57 |
| natefinch | katco: I'm very familiar with eric's changes and william's comments, though, so I don't think there will be many rpoblems | 18:57 |
| katco | natefinch: ok | 18:57 |
| ericsnow | natefinch: I'm going to update my branch and then you can take it from there | 18:57 |
| natefinch | ericsnow: cool | 18:57 |
| ericsnow | natefinch: thansk | 18:57 |
| natefinch | katco: I spent a lot of time looking into it before I realized eric was working on it, so I guess that works out :) | 18:57 |
| katco | =| | 18:58 |
| katco | communication people! | 18:58 |
| katco | ericsnow: how dare you not actually be out today | 18:58 |
| natefinch | exactly! | 18:58 |
| natefinch | ;) | 18:58 |
| natefinch | ok, gotta run | 18:58 |
| === natefinch is now known as natefinch-afk | ||
| ericsnow | katco, natefinch-afk: okay, I'm out | 19:06 |
| katco | ericsnow: gl | 19:06 |
| === ericsnow is now known as ericsnow-afk | ||
| === _thumper_ is now known as thumper | ||
| lazypower | o/ allo core devs. Is there ever a reason to use a scope: container qualifier on a relationship outside of subordinate services? I've been noodling this for a while and dont see a use-case for it outside of subordinate relations. | 20:08 |
| === natefinch-afk is now known as natefinch | ||
| thumper | davecheney: isn't the reason the structs moved into the multiwatcher was because state depends on multiwatcher... | 20:25 |
| thumper | davecheney: could the dependency change the other way around? | 20:25 |
| thumper | so state/multiwatcher just depends on state? | 20:25 |
| thumper | I'm not sure I'm remembering correctly | 20:26 |
| stokachu | kadams54: is there a way to do a globbed like search with theblues library? | 20:28 |
| stokachu | using search('nova') only pulls in nova-compute and not nova-cloud-controller etc | 20:28 |
| kadams54 | stokachu: I don't rightly know. We use ElasticSearch on the backend, so it would be worth trying ES' query string syntax: https://www.elastic.co/guide/en/elasticsearch/reference/1.3/query-dsl-query-string-query.html#query-string-syntax | 20:41 |
| stokachu | kadams54: ok thanks | 20:42 |
| mup | Bug #1517611 opened: TestFilesystemInfo race condition in 1.25 <ci> <intermittent-failure> <regression> <juju-core:Incomplete> <juju-core 1.25:Triaged> <https://launchpad.net/bugs/1517611> | 20:50 |
| stokachu | kadams54: find any issues with that PR? | 21:01 |
| * thumper sighs | 21:01 | |
| thumper | I merged the blessed master into my feature branch | 21:01 |
| thumper | and I have so many failures with go 1.5 | 21:01 |
| thumper | why are go 1.5 test runs not voting? | 21:02 |
| natefinch | doh | 21:02 |
| natefinch | thumper: because they weren't passing :/ | 21:02 |
| urulama | stokachu: the way to search anything mentioning "nova" is https://api.jujucharms.com/charmstore/v4/search?text=nova&autocomplete=1&limit=100 | 21:02 |
| thumper | clearly the fix then is to fix them | 21:02 |
| thumper | not just ignore them | 21:02 |
| natefinch | agreed | 21:02 |
| thumper | FFS | 21:03 |
| stokachu | urulama: perfect thank you! | 21:03 |
| natefinch | the lxd stuff is behind build tags for go 1.3+, (since it requires go 1.3+) Hopefully you're not seeing problems with that code | 21:03 |
| urulama | stokachu: and it's ranked | 21:04 |
| stokachu | urulama: yea it looks like it return tursty/precise first | 21:04 |
| stokachu | urulama: so can i assume those are the blessed ones at the top each time? | 21:04 |
| urulama | stokachu: yes | 21:05 |
| stokachu | urulama: perfect much appreciated | 21:05 |
| urulama | stokachu: if you want to see just the blessed ones (no user namespace charms) do this https://api.jujucharms.com/charmstore/v4/search?text=nova&autocomplete=1&limit=100&owner= | 21:06 |
| stokachu | even better | 21:07 |
| kadams54 | stokachu: QA checks out. I'm going to merge it. | 21:08 |
| stokachu | kadams54: sweet thanks! | 21:09 |
| thumper | natefinch: github.com/juju/juju/worker/provisioner, github.com/juju/juju/payload/api/private, github.com/juju/juju/cmd/jujud/agent (messy db cleanup), github.com/juju/juju/cmd/juju/commands, github.com/juju/juju/api/unitassigner , and some fallout that appears to be due to a clash with my current work | 21:11 |
| stokachu | urulama: so i noticed some of the results return charms like glance and neutron, is there a way to just search if the 'term' is in the id? | 21:11 |
| natefinch | thumper: hmm weird, yeah, none of that is lxd code. not sure why it would fail for go 1.5 | 21:12 |
| thumper | natefinch: most likely the runtime changes in 1.5 | 21:13 |
| thumper | w.r.t. GOMAXPROCS default | 21:13 |
| stokachu | kadams54: ew owned by docs? | 21:13 |
| thumper | causes more races in tests | 21:13 |
| natefinch | thumper: I don't think so. I always run GOMAXPROCS=8 | 21:13 |
| * thumper shrugs | 21:13 | |
| thumper | something else then | 21:13 |
| natefinch | sumthin | 21:13 |
| natefinch | katco: I have no idea how to write a test to show that eric's code uses goroutines in exactly the right way and not in the wrong way :/ | 21:14 |
| kadams54 | stokachu: Yeah, I'll get it straightened out. | 21:14 |
| stokachu | urulama: not a big deal if not, i can filter it out | 21:14 |
| stokachu | kadams54: :) | 21:14 |
| katco | natefinch: let me take a peek at his code again | 21:14 |
| katco | natefinch: what function are you trying to test? | 21:15 |
| natefinch | katco: the meat of the change is in startSerialWaitParallel | 21:15 |
| natefinch | katco: so, before we were effectively doing the whole inside of the loop in a goroutine, and now we're only doing the second half in a goroutine | 21:16 |
| katco | natefinch: i think the trouble is because the wait func should be passed into startSerialWaitParallel | 21:17 |
| urulama | stokachu: it should be with something like https://api.jujucharms.com/charmstore/v4/search?name=mongodb&limit=100 | 21:18 |
| katco | natefinch: pull that out, test it separately, and then test that startSerialWaitParallel calls it properly | 21:18 |
| urulama | stokachu: however, some names don't return any results, which makes me wonder why :S | 21:18 |
| katco | natefinch: i.e. startSerialWaitParallel calls whatever is passed in properly | 21:18 |
| natefinch | katco: yeah, that can work. I already didn't like that wait was getting half its args from closing over them and half not, so that would fix that problem. | 21:19 |
| katco | natefinch: isn't the only var it's closing over, "wg"? | 21:20 |
| natefinch | katco: 1 is half of three when you assign it to an int ;) | 21:21 |
| katco | lol | 21:21 |
| natefinch | techically correct is the best kind of correct! | 21:21 |
| natefinch | yeah, I though there was something else, but I guess not :) | 21:22 |
| katco | does anyone recall seeing a bug open recently around not being able to "upgrade-juju" after using "--upload-tools" ? | 21:24 |
| natefinch | ah hah, the cmd too | 21:24 |
| natefinch | katco: upload tools is bad juju, so to speak. | 21:24 |
| katco | natefinch: so not the point ;p | 21:24 |
| natefinch | I'm not really clear on the rules when you use upload-tools vis a vis upgrades | 21:25 |
| thumper | davecheney: ping when you are around | 21:27 |
| perrito666 | I hate replicaset sometimes... and some other too | 21:28 |
| * perrito666 bbl bike time | 21:28 | |
| stokachu | urulama: so would https://api.jujucharms.com/charmstore/v4/search?name=nova&limit=100 return everything with 'nova' in the name? | 21:28 |
| urulama | stokachu: it should, but doesn't :S so, stick with the text=nova&owner= for now and filter out the ones you don't need | 21:31 |
| stokachu | urulama: ok cool | 21:31 |
| urulama | stokachu: i'll give a look why name queries fails | 21:32 |
| stokachu | urulama: thanks again :) | 21:32 |
| urulama | stokachu: np | 21:32 |
| natefinch | god I hate using external tests. It's like every time I go to write a nice little unit test, someone is actively trying to make it more difficult :/ | 21:50 |
| natefinch | katco: well, refactored, but now to write the tests: https://github.com/natefinch/juju/commit/5f5e26441bc250c221e8e8b688432db8c5e8beec#diff-813c65409327242bb7df0d66ac593b91R195 | 21:52 |
| natefinch | katco: gonna require some... tricks, I think. | 21:52 |
| katco | =| | 21:53 |
| natefinch | just difficult to detect when you're running in a separate goroutine or not | 21:53 |
| natefinch | actually, I think I can do it. | 21:55 |
| natefinch | hmm.. | 21:55 |
| natefinch | ahh, I have it | 21:56 |
| natefinch | some creative locks in the start and wait functions can prove that start is always called in order, and wait gets called all at the same time | 21:57 |
| mup | Bug #1517632 opened: juju upgrade-juju after upload-tools fails <juju-core:New> <https://launchpad.net/bugs/1517632> | 21:59 |
| === natefinch is now known as natefinch-afk | ||
| perrito666 | seriously this country.... a bit of summer and people have barbecue parties ever single night... and my desk is next to the windows | 22:59 |
| alexisb | thumper, axw, anastasiamac: release call is running over | 23:00 |
| thumper | k | 23:00 |
| alexisb | be there shortly | 23:00 |
| anastasiamac | k :D | 23:01 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!