/srv/irclogs.ubuntu.com/2015/04/02/#juju-dev.txt

ericsnowaxw: PTAL http://reviews.vapour.ws/r/1373/ (should be quick)00:00
jw4anyone familiar with upgrades, willing to answer a couple questions?00:02
axwericsnow: done00:05
* axw goes for breakfast00:05
ericsnowaxw: thanks!00:05
axwwallyworld: can you please link me the status spec?01:13
wallyworldaxw: https://docs.google.com/a/canonical.com/document/d/1SsHDxi58xgrim6VTKorqMn4GlzxPV9EFvu1ueBgnMlc01:13
axwta01:13
axwwallyworld: so, a charm can never status-set error. what if the software is irreparably broken?01:17
axwi.e. the charmed service, not the charm itself01:18
wallyworldaxw: it then sets blocked01:18
wallyworldblocked means it can't run and a user has to intervene01:18
wallyworldfor whatever reason01:18
wallyworlderror is reserved to report hook errors only01:18
axwwallyworld: mk. that implies to me that it can be repaired, which may be optimistic01:19
* axw continues01:19
wallyworldaxw: wrt storage add, did we want the additional storage (eg a new volme) to be able to get different sizing, tags etc, or did we want to enforce storage add must create the new volume exactly the same as the first one01:23
axwwallyworld: I think it should take the same args as "--storage" in "juju deploy"01:24
axwwallyworld: i.e. pool,size,count01:24
wallyworldaxw: awesome, that was my tpught also01:24
wallyworldjust wanted to confirm01:24
wallyworldty01:25
axwnp01:25
wwitzel3can I get one more set of eyes on http://reviews.vapour.ws/r/1371/01:45
=== natefinch-dinner is now known as natefinch
axwwallyworld: reviewed02:02
wallyworldaxw: ty, will look02:02
axwwwitzel3: will take a look soon02:02
wwitzel3axw: thanks, appreciate it02:04
natefinchaxw: my stuff needs to be in for 1.23 that is getting cut basically ASAP... so if you could prioritize it, I could get it merged (barring major problems, of course) - http://reviews.vapour.ws/r/1299/02:04
axwnatefinch: ok02:04
natefinchaxw: thanks, I appreciate the help.02:05
=== ericsnow is now known as ericsnow_afk
=== ericsnow_afk is now known as ericsnow
wallyworldaxw: i've updated the pr for that health branch, after you finished your current review02:47
axwwallyworld: ok02:47
wallyworldaxw: i have to go out for lunch, if when you look at the pr you are happy with it, could you $$merge$$ for me and i'll propose the next one when i get back03:14
axwwallyworld: np03:15
wallyworldty03:15
=== ericsnow is now known as ericsnow_afk
=== ericsnow_afk is now known as ericsnow
axwwwitzel3: reviewed, sorry for the delay03:48
wwitzel3axw: np, thank you03:49
wwitzel3axw: it requires work, so I won't get to it until tomorrow anyway ;)03:51
axwwwitzel3: sorry :~(03:51
* axw is creating work for lots of people today03:52
wwitzel3haha03:52
wwitzel3that's probably a good thing03:52
marcoceppi_okay, 1.23-beta2 has been really flaky with local provider05:00
marcoceppi_seems about every third machine doesnt' get the agent running on it05:00
marcoceppi_I realize that's not helpful, I'm gathering log files05:05
marcoceppi_re-ran cloud-init, I think this is a problem with the seed data, ssh keys aren't getting installed05:12
marcoceppi_I also can't seem to connect to the API with python-jujuclient anymore. I noticed that the api server is only listening on ipv605:23
marcoceppi_is this a new configuration?05:23
marcoceppi_prefer-ipv6: false is in my jenv, not sure why the api-server is only binding to the ipv6 address locally, this may be a redherring05:26
marcoceppi_also noticed that the user changed to admin where it was user-admin previously05:26
axwwallyworld: if you can take a look at http://reviews.vapour.ws/r/1360/ and http://reviews.vapour.ws/r/1361/ some time today, I'd be most grateful05:52
wallyworldsure, np. thanks for landing my stuff05:52
wallyworldi'm just doing a pr for the next bit05:52
axwwallyworld: nps05:52
axwcool, ping me and I'll review05:52
wallyworldaxw: http://reviews.vapour.ws/r/1376/05:54
axwlooking05:54
wallyworldah i half looked at the managed filesystem source one last night, looked good05:55
jogwallyworld, Would you have time to read over bug 1435644, or suggest someone? It's a private cloud streams question and I know you've been able to give guidance on similar bugs.05:58
mupBug #1435644: private cloud:( environment is openstack )index file has no data for cloud <juju-core:New> <https://launchpad.net/bugs/1435644>05:58
wallyworldjog: ok, sre05:58
jogwallyworld, thank you, it needs another set of eyes, as I'm not sure what to suggest.05:59
wallyworldjog: i just need to finish a few things, will comment on the bug when i can06:00
jogwallyworld, np I'm EOD06:01
wallyworldok06:01
jamaxw: http://reviews.vapour.ws/r/1377/ can you give it a quick look? It is the same patch as katco but it just fixes a test that was also using the leader-election feature flag (which is now removed)06:05
axwjam: okey dokey06:05
jamthx06:05
jamhmmm... I think it is supposed to be targetting 1.23 I'll have to check on that06:05
axwjam: shipit06:08
axwah yeah, probably06:08
jamaxw: I have to rebase it, but I'll propose another one for 1.2306:08
axwjam: nps, I expect it'll be exactly the same given the size :) just merge if so06:09
jamaxw: sgtm06:09
marcoceppi_where does juju cache it's api endpoints?06:29
jamaxw: so it looks to be an entirely bigger patch. I have the feeling master is not up-to-date with the leader-election code that is in 1.2306:30
marcoceppi_I changed values in the api endpoints in .jenv but it says loading from cache06:30
jamtbh I was surprised the patch was so small06:31
jamthe flag in master only changes the client side (essentially), but all of the internal workers were running behind that flag in the 1.23 branch.06:31
axwjam: as was I. I'll take a look06:34
jamkatco: ping06:35
axwjam: LGTM06:37
marcoceppi_what is the api user in juju? user-admin or admin?06:37
jamaxw: http://reviews.vapour.ws/r/1378/ is the newer proposal, but I'm running through tests, etc since it kept growing.06:37
axwsure06:38
jammarcoceppi_: the name of the user is "admin" the *tag* is "user-admin" (vs machine-0 etc)06:38
marcoceppi_jam: so what should I use in the websocket? user-admin?06:38
jammarcoceppi_: so in most of the API we talk in terms of tags. so Login probably wants 'user-admin', but know that in the future we're looking to be able to name those users and they will then be "user-joe" etc.06:40
jamI believe the agents login as "machine-0" etc.06:40
marcoceppi_sure, thanks06:40
marcoceppi_I'm having a hell of a time getting python-jujuclient to connect to 1.23-beta2 bootstrapped environment06:41
axwwallyworld: reviewed06:42
wallyworldty06:42
jammarcoceppi_: I believe thumper has been making changes to Login which is supposed to be compatible, but I thought he had to submit some patches recently.06:42
marcoceppi_let me try rolling back to 1.2206:43
jammarcoceppi_: if something is broken wrt compatibility, we really appreciate that feedback06:44
jamwell, *I* appreciate it, hopefully the people who have to fix it do too :)06:44
marcoceppi_jam: I need to do more testing, I'm having a hard time locking this down06:44
marcoceppi_other than "it's not working"06:44
jammarcoceppi_: sure. if it doesn't "just work" and it used to, then we certainly have a  bug06:45
marcoceppi_too many moving pieces06:45
axwwallyworld: BTW, I saw later that the time is set by the state server. you can probably safely ignore my comment, I was just thinking it would be nice to show relative times in the UI06:45
axwwallyworld: but that involves quite a bit of complexity, in order to ensure timestamps are synchronised between state servers and clients06:45
wallyworldaxw: Agreed, this is one area where I'm hoping for feedback from users. The spec says timestamp but I suspect we'll be asked for relative times instead/as well as.06:46
wallyworldi just want to get *something* reasonable out there so we can then see what real users ask for and complain about06:46
* axw nods06:46
jamwallyworld: axw: you probably need to use the universal "time since epoch GMT" timestamp, and then on the client turn that into localtimes. It seems the most portable way06:49
jamrelative times (as in 1hr ago) seem like a fair amount more work.06:49
wallyworldjam: it does store epoch06:50
wallyworldon client it formats as RFC82206:50
axwjam: that still assumes the clocks are all correct. I suppose it's fair to say that if they're not, you're on your own06:50
wallyworldi can't recall if that includes tz conversion06:50
jamaxw: if you're clocks aren't reasonably close to reality there isn't really much we can do is there?06:50
axwwallyworld: it just displays the timezone it's in. you could concert to local TZ06:50
wallyworldah, i did mean to do that, but it got late06:51
axwjam: well, you can implement poor-man's NTP, but it's probably not worthwhile :)06:51
marcoceppi_jam: okay, there's an issue in 1.23-beta206:51
wallyworldaxw: i am counting on our deployents using NTP OOTB06:51
jammarcoceppi_: a bug is appreciated, targetting the 1.23 series, as this seems pretty critical. It *might* already be fixed but I wouldn't hold my breath on it.06:52
marcoceppi_jam: just writing repro steps06:52
marcoceppi_jam: would it be worth compiling the 1.23 branch of the repo to verify as well?06:52
axwjam: in my former life, I worked on a distributed system where the servers (owned by customers) were *constantly* skewed and they would never run NTP. I think it's okay to rely on NTP, I just had flashback nightmares06:52
axwwallyworld: SGTM06:52
jamaxw: :)06:52
jammarcoceppi_: if you want to take the time, its nice to have the info, but its mostly a "do what you can" sort of thing.06:53
jamit's not your responsibility to fix all our bugs, but we're happy to have your support06:53
marcoceppi_it's 3am and this has blocked me, I don't have much else to do06:53
jamI'm sorry to hear that, and sorry that we broke stuff, but certainly appreciate you discovering it.06:56
jammuch better to find out in a beta release.06:56
marcoceppi_jam: http://pad.lv/143953507:00
marcoceppi_I'll try compiling 1.23 from github then try trunk as well07:00
voidspacemorning all07:04
jammarcoceppi_: thanks for investigating, that looks much weirder than I expected07:04
marcoceppi_jam: fwiw, this works with 1.23 branch (juju version shows 1.23-beta3)07:09
mupBug #1439535 was opened: 1.23-beta2 websocket incompatibility <juju-core:New> <python-jujuclient:New> <https://launchpad.net/bugs/1439535>07:09
dimiternvoidspace, morning07:09
dimiternvoidspace, there's a new bug about the addresser I've assigned to you07:10
dimiternvoidspace, trivial to fix - just don't start the worker if environs.SupportsNetworking() returns false07:11
voidspacedimitern: ah, ok07:12
voidspacedimitern: easy enough :-)07:12
voidspacedimitern: is it a problem to have an addresser around otherwise?07:13
dimiternvoidspace, well, I don't think so - but it should not spam the logs and restart every 3 seconds :)07:13
voidspacedimitern: ah, yes it is a problem07:14
voidspacedimitern: just reading the bug :-)07:14
dimiternvoidspace, I'd prefer to have it running in all environments TBH07:14
dimiternvoidspace, but without a netEnviron interface it can't do its job anyway07:15
jammarcoceppi_: I don't see anything specifically in "git log --first-parent juju-1.23-beta2..1.23" that makes it clear why your API issues would have been fixed07:15
jammarcoceppi_: is this local provider?07:16
jamyeah "switch local"07:16
marcoceppi_jam: aye, it is07:16
wallyworldaxw: i'm going blind, i can't see what calls func processPending(...)07:17
axwwallyworld: the main loop in storageprovisioner.go07:17
axwat the top07:17
jammarcoceppi_: which makes me worry that it is more of a packaging issue.07:17
jamdimitern: voidspace: do you know if the backport of "addressable containers" into 1.23 would have affected local provider?07:17
marcoceppi_jam: really? seems odd it would be in packaging07:18
jammarcoceppi_: to quote sherlock holmes, "when you have eliminated the impossible, whatever remains, however improbable, must be the truth?"07:18
jammarcoceppi_: I certainly wouldn't expect it there either.07:18
dimiternjam, it's not a backport - it was first implemented in 1.23, but it shouldn't affect the local provider07:18
jamdimitern: I just see addressing changes in 1.23 beta 307:19
jamwhich *could* be something that would allow local provider to work/not work.07:19
jammarcoceppi_: I don't see anything about login, etc changing07:19
axwwallyworld: found it?07:19
wallyworldaxw: ffs, looked straight past it. the process functions aren't thread safe, is it worth a comment that they're being called from a single threaded loop, or is that asking for an unnecessary comment07:19
jammarcoceppi_: just to eliminate variables, can you "git co juju-1.23-beta2" and see if it is broken for you ?07:19
dimiternjam, it's possible, however the local provider does not use the same brokers as other providers, where most of the changes are07:19
axwwallyworld: like the uniter, the whole thing is not goroutine-safe07:20
marcoceppi_jam: sure07:20
axwwallyworld: I can put a comment, but I'm not sure where it'd go ...07:20
wallyworldfair enough07:20
jamI see changes to GCE, but those should also not effect local07:20
wallyworlddon't wory07:20
jammarcoceppi_: I do see a change to how local provider was getting its IP address. I wonder if it was suffering from the "claim I'm on the LXC bri07:22
jambridge bug in 1.23-beta2"07:23
marcoceppi_jam: possibly? about to compile07:23
wallyworldaxw: reviewed, just a few small comments/suggestions07:25
axwwallyworld: cheers07:25
jamanyone have thoughts on expired cert issues?07:25
jam1.23 points to go.googlesource.com07:26
jamwhich has a bad cert07:26
marcoceppi_yup, I give up jam, 1.23-beta2 from source works07:26
jamnm. I was testing in a VM that had been suspended for 2 weeks, and the time hadn't updated yet.07:27
jamso their cert was renewed 2 weeks ago07:27
jammarcoceppi_: le sigh....07:28
jammarcoceppi_: but you can reliably reproduce when running /usr/bin/juju, right?07:28
anastasiamacjam: another fix that went in into 1.23 related to http- and apt- addresses that may affect marcoceppi_ is the check for loopback address07:28
marcoceppi_jam: everytime, going to spin up another vm to verify07:28
jamanastasiamac: yeah, but in theory he just tested rolling back his local build to 1.23-beta2 and it still passed07:28
anastasiamacjam: however m not sure why there would b a diff btw 1.23-beta2 and 1.23-beta3 with respect to this fix..07:28
anastasiamacjam: yep..07:29
jammarcoceppi_: the other thing is to make sure you do "godeps" and all that, just in case there is some depedency we were using in 1.23-beta2 that had something wrong.07:29
marcoceppi_jam: I run godeps everytime I build07:29
marcoceppi_following this workflow: http://marcoceppi.com/2014/11/compiling-juju-core-from-source/07:29
jammarcoceppi_: that looks decent to me07:30
jamhaving various versions of juju in $PATH could confuse things in the past, did you check if there were build errors?07:30
jam(as that would leave the other 1.23 around in $GOPATH/bin)07:31
jamaxw: can you try a quick test for me? "cd worker/uniter; go test -check.v -check.f Leader"07:31
jamI'm still sorting out things in this VM, but on other machines it was failing for me. I'm wondering if the tests are just poorly isolated (they use some other test's setup to be able to work properly)07:32
jamor maybe I'm just insane :)07:32
marcoceppi_jam: juju version reported 1.23-beta2 and `which juju` was the one in $GOPATH/bin07:32
jammarcoceppi_: were you running "juju foo" or $GOPATH/bin/juju foo ?07:33
jamIIRC, there was a really old bug at one point where because juju was running sudo07:33
marcoceppi_just `juju foo`07:33
jamit would find the wrong juju because root's PATH was not user's PATH07:33
marcoceppi_I can try again with full path07:33
jammarcoceppi_: I don't expect that's the problem, but just trying to eliminate possibilities07:33
marcoceppi_sure07:33
axwjam: will do, with your 1.23 branch you mean?07:37
jamaxw: i'm having it fail on master as well as stock 1.2307:37
jamthere it passed07:38
jamat least that gives me a baseline07:38
jamthe VM is happy07:38
marcoceppi_jam: yeah, I'm unable to reproduce on a clean vm now07:38
axwjam: works on master for me07:38
marcoceppi_there's something buggered with my setup, this was just a big 'ol snipe hunt07:39
jamaxw: thanks for confirming. I'll just move forward :)07:40
voidspacedimitern: for bug 143868307:58
mupBug #1438683: Containers stuck allocating, interface not up <add-machine> <cloud-installer> <landscape> <maas-provider> <network> <juju-core:Triaged by mfoord> <https://launchpad.net/bugs/1438683>07:58
dimiternvoidspace, yeah?07:58
voidspacedimitern: can you point me to where in the logs or config that indicates eth1 is being used07:58
voidspacedimitern: I can't find eth1 mentioned by name or MAC address like this07:59
voidspacedimitern: and the reference to primary network in the logs is for eth007:59
dimiternvoidspace, in machine-0.log around returning the result from PrepareContainerInterfaceInfo07:59
voidspacedimitern: ok, thanks07:59
voidspacedimitern: and there it is...08:00
dimiternvoidspace, as for eth0 being primary - there are 2 places (also in that log) - before starting but after acquiring the instance, and before starting the container08:00
voidspacedimitern: I grepped for "eth1", not sure how I missed that :-/08:00
dimiternvoidspace, :)08:00
voidspacedimitern: ok, so easy08:02
voidspacedimitern: we iterate over all subnets and map subnets to interfaces08:02
voidspacedimitern: we don't check if the interface is disabled - and where the same subnet appears more than once we'll overwrite08:02
dimiternvoidspace, yeah - both of these are problems08:02
voidspacedimitern: so we *do* need to skip disabled interfaces there08:03
voidspacedimitern: that will be the problem08:03
dimiternvoidspace, where?08:03
voidspacedimitern: prepareAllocationNetwork08:03
voidspacedimitern: lines 1021 and 105108:03
voidspacecoffee08:04
dimiternvoidspace, so the result of NetworkInterfaces should include all NICs we can find, but with correct device index and disabled flag08:04
voidspacedimitern: yep, I already have a fix for that - although "correct device index" is a bit arbitrary if lshw doesn't give it to us08:05
voidspacedimitern: and when we iterate over interfaces to pick we should be skipping disabled ones08:05
dimiternvoidspace, hmm I'm not quite sure08:06
voidspacedimitern: this is in prepareAllocationNetwork I'm talking bout08:06
voidspace*about08:06
dimiternvoidspace, even if we skip them, we still shouldn't overwrite the subnet info in case more than one nic is on the same subnet08:07
voidspacedimitern: do we ever want to return interface info for a dsiabled nic?08:07
voidspacedimitern: agreed, but that's a different issue08:07
voidspacealthough08:07
voidspacewe need to pick *an* interface, so if we have picked a subnet and allocated an address on it - we only need one non-disabled nic08:07
voidspaceso overwriting isn't really an issue (so long as we skip disabled ones)08:08
dimiternvoidspace, if by "return" you mean from NetworkInterfaces() - yes, we want that; in prepareAllocationNetwork though - that's different08:08
voidspaceit's prepareAllocationNetwork I'm talking about08:08
dimiternvoidspace, ok08:08
voidspacethat's where the specific problem is08:08
voidspacedimitern: in prepareAllocationNetwork do we want to always prefer the primary network, unless it's disabled?08:08
voidspaceand always try that first (try to allocate an IP on the subnet associated with the primary interface)08:09
dimiternvoidspace, just keep in mind that we'll most likely start allocating  addresses (and creating NICs) for containers such that we have a 1:1 correspondence with the host enabled nics08:10
voidspaceok08:11
dimiternvoidspace, so in the long term it doesn't matter which NIC is primary08:11
voidspaceso I don't think overwriting actually matters, as we only want to pick one nic anyway08:11
voidspacebut we *do* need to skip disabled networks in prepareAllocationNetwork08:11
dimiternvoidspace, but for now we only want 1 address/nic per container - connected to the host's primary nic08:12
voidspacewhich is a 3 line fix plus test... (along with the device index fix I already have for NetworkInterfaces)08:12
dimiternvoidspace, yes, but we shouldn't pass [id1, id1] to Subnets when we have 2 nics on the same subnet id108:12
dimiternvoidspace, (or if we do, we should only fetch the info once per unique id)08:13
voidspacedimitern: ok, that's also true08:13
voidspace:-)08:13
dimitern:)08:15
voidspacedimitern: so the singular runner doesn't have an obvious way to not start a worker08:44
dimiternvoidspace, it won't start it if you don't add it to the list of workers to run :)08:45
voidspacedimitern: right - so in which case the agent needs to construct an environ08:45
voidspacedimitern: *or* we can start the runner and Handle can be a no-op if networking isn't supported08:45
voidspacedimitern: either is fine, I'd just like to know which is preffered08:46
voidspacedimitern: startEnvWorkers (cmd/jujud/agent/machine.go) doesn't currently have an env08:47
dimiternvoidspace, I think we should start it08:51
dimiternvoidspace, but then internally it should handle "networking not supported" properly, i.e. logging that it won't do anything, but not returning an error08:52
voidspacedimitern: that's fine - and easier :-)08:53
dimiternvoidspace, exactly :)08:55
TheMuedimitern: shit, exactly this moment the garage comes to fetch my car. will report afterwards, got an idea after your hint08:58
dimiternTheMue, ok, no worries08:59
axwwallyworld: in your email you wrote "juju-get" and "juju-set". I think s/juju/status/ ?09:24
TheMuedimitern: thinking about I would say your theory is correct, I'll talk to rvba. when watching the node during boot and it's wanting to fetch metadata from the controller the return code is 404.09:42
=== mgz is now known as mgz_
TheMuedimitern: this sounds like "yes, I can connect the server, but it doesn't has the info (here the lshw) that I want."09:43
dimiternTheMue, yeah, something like this - perhaps it's not because lshw is missing, but something else09:45
TheMuedimitern: maybe, I'll discuss on #maas. but it seem to be able to at least reach the maas controller. and that's why pings also work.09:47
dimiternTheMue, yeah, ok09:48
TheMuebtw, my car is now gone *sniff* less than a year old09:48
axwdimitern: not sure if it's due to networking stuff going on, but I cannot "juju ssh" to ec2 machines on master. works fine if I disable proxying through machine-009:57
axwdimitern: I get "nc: getaddrinfo: Name or service not known"09:57
dimiternaxw, hmm odd I'll try here with a fresh tip of master09:58
dimiternaxw, but it will be in about half an hour09:58
axwdimitern: no rush, just thought I'd let you know in case you knew of somethign related09:58
dimiternaxw, do you have containers deployed on machine 0 ?09:59
axwdimitern: could be unrelated, I see a bunch of network failures due to network resolution failing09:59
axwdimitern: nope09:59
dimiternaxw, if you paste me some logs I'll investigate10:00
axwI just destroyed, I'll recreate10:00
axwdimitern: never mind, must've been an intermittent DNS error.10:15
dimiternaxw, *whew* good! :)10:36
* dimitern goes to get his car back10:58
rogpeppeaxw: i was just looking at NewClient in apiserver/client11:01
rogpeppeaxw: ISTM that you could save a mongo round trip in every single API call by using common.NewToolsURLGetter(st.EnvironTag().Id(), st)11:02
rogpeppeaxw: instead of fetching the env from mongo every time11:02
rogpeppeaxw: (unless there's some reason that the env uuid might change, i guess)11:03
axwrogpeppe: yeah, agreed. that possibly predates EnvironTag caching... pretty sure that value is fixed11:06
rogpeppeaxw: cool. might be worth bearing in mind.11:07
jamaxw: *if* you're still around I have a new update for the branch. Unfortunately, we still have 1 test that fails because it is nondeterministic. I haven't figured out how to fix that yet.12:43
jamkatco: are you around?12:43
jamkatco: if/when you are, I've been looking at enabling leader-election always in http://reviews.vapour.ws/r/1378/12:43
jamit is almost complete except for one big TODO. If you have time, I'd appreciate a look at it.12:43
jamI think we could enable this in production, because the test is failing because we're triggering a hook that should be triggered, but at a random time so I can't just add it to a fixed spot in the test expectation.12:44
dimiternback12:49
abentleyalexisb: I don't have a blessed revision, and there's at least one GCE bug unfixed.  Do you want to meet anyway?13:11
abentleyalexisb: Correction, I see a bless for 16bed49 in build 250813:12
abentleyalexisb: Correction 9710c29 / 250913:13
mgzyeah, the ppc tests managed to get through on that one13:13
abentleymgz: local-deploy-vivid-amd64 was stuck for most of yesterday, and is stuck in the same way today.  I am disabling it.13:14
mgzabentley: fair enough13:21
dimiterndooferlad, voidspace, ping13:31
dimiternI feel like I'm talking to myself >_<13:32
voidspacedimitern: heh13:32
mgzkatco: are you around? I'm trying to test cn and the v4 signing code on 1.23-beta2 is not playing ball with me it seems13:32
dooferladdimitern: howmy I be o service?13:35
TheMuedimiter: btw, I'll file a bug against maas, it seems to fail during cloudinit13:40
dimiternTheMue, ok, that's what it looked like to me13:43
TheMueyeah, to rvba also13:43
dimiternTheMue, but we should still find a way to reproduce and fix that bug13:43
TheMueto repoduce it is no problem, but we cannot fix the booting bug13:44
TheMuewe can fix our bug to stop processing, that's what I've done13:45
TheMuenow need tests for it13:46
=== kadams54 is now known as kadams54-away
hazmatmarcoceppi_: ping13:58
alexisbericsnow, wwitzel3, what is the latest on the GCE bugs?14:00
ericsnowalexisb: wrapping up (all but 1 merged)14:02
marcoceppi_hazmat: otp14:03
hazmatmarcoceppi_: just curious about that issue w/ py juju client and 1.23.. just looked up the bug report and it looks like its already resolved though.14:04
aznashwanericsnow, axw: could I bother you guys with a closer lookthrough of http://reviews.vapour.ws/r/1218/ and, if you're feeling really adventurous, http://reviews.vapour.ws/r/1330/? :D14:06
ericsnowaznashwan: sure thing14:06
dimiternI'm is off for now - might be back later14:12
katcojam: mgz: BoD14:16
katcojam: one of the tests is failing then?14:16
wwitzel3alexisb: I should have the last patch up in 20 minutes or so14:19
mgzkatco: bod? beginning of day?14:21
katcomgz: yep!14:21
mgzkatco: >_<14:21
katcomgz: ?14:21
mgzkatco: so, we expect 1.23-beta2 to work in china right? it has all your code in it.14:22
mgzkatco: I get enough silly acronyms from xwwt14:22
katcomgz: as much as we can expect untested fixes to work =/14:22
wwitzel3lol14:22
xwwtmgz: Keeps you on your toes.  ;)14:22
katcomgz: we never had a cn north instance to test against for legal reasons14:23
mgzkatco: how should I go about debugging it? I have flipped the debug flag in goamz, and it basically confirms we are using the v4 path (doesn't print the headers, I could also enable that extra dumping in the signing file), but aws complains at me that the access key query param (I presume) is not supplied14:24
mgzwhich is non-sensical as that's not how v4 signing works14:24
mgzI will paste you my junk for your curiousity14:24
katcomgz: hmmm... yes ty14:24
mgzkatco: https://pastebin.canonical.com/128882/14:26
* katco looking14:26
mgzI can try flipping the juju code to always using v4 and running it against us-east-1?14:27
katcomgz: that might be informative14:28
mgzwill do that, and turn on the signing debug output14:28
katcoi'm doing some research on that param14:29
mgzso, the v2 version does all the query params as normal, plus the access key, plus a hashed-with-private-key param14:32
katcomgz: right, v4 uses headers primarily14:32
mgzyup14:33
katcomgz: do you know what request was being made? e.g. what api is failing?14:33
katcolooks like https://github.com/go-amz/amz/blob/v1/ec2/ec2.go#L1080 ?14:34
mgzkatco: DescribeAccountAttributes14:36
cheryljericsnow:  when you get a minute, could you review this:  https://github.com/juju/replicaset/pull/114:37
mgzkatco: hm, I didn't get more printed from the signing flipping to os.Stdout at the top14:37
cheryljericsnow: I found another repo that needed updating14:37
ericsnowcherylj: yeah14:37
mgzkatco: same issue with us-east-114:37
ericsnowcherylj: thanks for being so thorough14:37
katcomgz: i think that debugging is only for v414:37
ericsnowcherylj: I need to add a GH webhook for that repo too14:37
mgzit's using v4 now14:38
mgzI flipped it in juju14:38
cheryljericsnow: thanks!  I think that will be the last one14:38
ericsnowcherylj: did you mean to remove the summary comment that was on line 114:38
ericsnow?14:38
cheryljericsnow: yes, I did for consistency with the other repos.14:38
cheryljericsnow: I debated it, but figured I should err on the side of consistency.  That, and the doc that was put together didn't mention the summary line14:39
ericsnowcherylj: k14:39
mgzkatco: I'm rebuilding from the beta2 tag in git, for line number references14:39
ericsnowcherylj: if I recall correctly, that summary line was a suggestion from the license text but is much less relevant for our use :)14:40
cheryljericsnow: yeah, I think you're right.14:40
ericsnowcherylj: LGTM14:41
cheryljericsnow: thanks!14:41
katcomgz: so you're passing in SignV4Factory(...)?14:42
katcomgz: because if you've switched the debug logger to stdout, and it's using v4, i have no explanation for why you're not getting output.14:42
mgzkatco: https://pastebin.canonical.com/128887/14:45
mgzkatco: I am also mystified14:46
katcomgz: i can never remember... does --debug collect the logs from the disparate machines?14:46
katcomgz: bc i think this would be in a jujud instance?14:47
jamkatco: so the patch you proposed to remove leader-election feature flag was strictly against master, which doesn't actually have all the code that is in 1.2314:48
jamkatco: so I picked it up and started fixing it for 1.23, which involved a fair bit more work14:49
katcojam: ahhh i see. not a straightforward backport14:49
jambut now there is 1 more test to fix14:49
katcojam: doh :(14:49
mgzkatco: https://pastebin.canonical.com/128889/14:49
jamwhich has a race condition I don't have an immediate answer for14:49
mgzkatco: this is all just from the client14:49
jamkatco: http://reviews.vapour.ws/r/1378/ should be my work so far, proposed against 1.23, and I should have a big TODO in there. I'd appreciate it if you could look at it, since I'm way past EOD here14:50
katcojam: sure thing14:50
katcojam: fyi juggling a few things here. i've been out for a week, and am behind on some storage work for tanzanite14:50
katcomgz: i see your v4 signing is definitely running... and you're sure godeps or gopkg.in isn't fooling you on which version you're compiling against?14:52
mgzkatco: it may be, I will double check14:55
mgzonly thing I get is:14:55
mgzgodeps: "/home/ubuntu/go/src/gopkg.in/amz.v3" is not clean; will not update14:55
mgzwhich implies I changed the right junk14:56
katcomgz: that's... good i think14:56
katcoyeah14:56
=== kadams54-away is now known as kadams54
katcomaybe try a go clean github.com/juju/...?14:56
katco"go clean github.com/juju/..."14:57
katcoif that wasn't clear14:57
mgzwill `rm -rf ~/go/pkg/linux_amd64/*` to be sure14:58
katcok14:58
katcomgz: out of curiosity -- and i don't know much about this -- are these "temporary credentials" by any chance?14:58
katcomgz: this came up in my research indicating there may be some inconsistency: https://github.com/mitchellh/goamz/pull/15414:59
wwitzel3can I get a review for http://reviews.vapour.ws/r/1371/, thanks :)14:59
mgzkatco: the cn ones may be, but the us-east-1 is just our normal account15:00
mgzkatco: DEBUG: 2015/04/02 15:12:04 GZ: requestTime err Could not retrieve a request date. Please provide one in either "x-amz-date", or "date".15:12
mgzthat error happens, but gets swallowed15:12
katcomgz: o.0 swallowed?15:12
katcomgz: i see... in ec2.go15:13
mgzkatco: yeah, Signer returns error, but query in ec2.go doesn't check for it15:14
katcomgz: well that's the smoking gun, it's not actually signing the request15:14
katcomgz: it looks like ec2.go's query(...) is providing a timestamp. i wonder if that's a valid attribute to look at15:15
wwitzel3ericsnow: that gce review is up when you get a chance15:15
ericsnowwwitzel3: looking now15:15
mgzkatco: I don't understand the requestTime code really15:15
mgzhow does it think it's getting a header set if we're *sending* our first request?15:15
katcomgz: the headers are set by the code15:16
katcomgz: when building the http.Request15:16
katcomgz: the signing documentation (i believe) states that the user can either specify x-amz-date or date. i'm not sure if Timestamp is something to look for15:16
mgzwell, we can make the ec2 code do that15:18
mgzbut currently it can just never work?15:18
katcomgz: it looks that way... v4 needs a timestamp15:18
mgzI will bug a file15:19
katcomgz: http://pastebin.ubuntu.com/10724611/15:19
katcomgz: try that rq, just to spike and see if we can get cn working15:19
mgzkatco: doing15:20
marcoceppi_hazmat: yeah, I'm not sure what happened15:21
katco(and handle the error if you hadn't already)15:21
marcoceppi_but for a good several hours I couldn't get juju-local to repsond on the websocket15:21
marcoceppi_just getting "websocket closed"15:21
katcomarcoceppi_: no websocket for you!15:21
marcoceppi_Ukraine is not game15:23
mgzkatco: auth failure, but I have output15:23
katcomgz: auth failure as in the signing failed?15:23
mgzkatco: https://pastebin.canonical.com/128890/15:24
katcomgz: ah... so that indicates that signing is fine, b/c it usually complains loudly about signing15:26
katcomgz: is that the us-east-1? or cn?15:26
mgzkatco: us-east-115:27
katcostupid me just looked at log lol15:27
mgzI can switch back as needed15:27
katcomgz: well, i think at least that gets you past the signing issue15:27
katcomgz: in terms of credentials... not too sure what causes that. http://docs.aws.amazon.com/AWSEC2/latest/APIReference/errors-overview.html15:28
katcomgz: just indicates to check your creds15:28
natefinchman I wish our code had more comments about why the code is doing what it's doing15:28
wwitzel3natefinch: but it is self documenting (said every person writing code ever) ;)15:29
mgzkatco: yeah, signing errors are deliberately obscure15:29
katcomgz: well i don't think it's a signing error... usually it will say something like "the hash you sent does not match the hash we calculated" with some debugging info15:30
katcomgz: i think it's something genuinely amiss with the creds15:30
natefinchwwitzel3: yeah, that's my main problem with the "no comments" people... sure, I can tell *what* the code is doing, but I can't tell *why* they're doing it.15:31
katconatefinch: but thank god all those New(...) methods have comments!15:31
katco// NewFoo returns a new Foo.15:31
natefinchI'd give my external keyboard for a  // hide the not found error with ErrPerm so we don't expose when the machine is not allowed to be accessed vs. doesn't exist15:32
mgzkatco: I suspect you could repo this locally with any aws account15:32
mgzkatco: anyway, will file bug and we'll work out what's up15:33
katcomgz: k, ty for troubleshooting with me.15:33
ericsnowwwitzel3: review done (with a couple minor comments)15:38
wwitzel3ericsnow: I think we can just toss out the replacement based on what I am seeing15:41
ericsnowwwitzel3: yeah, I think so too15:41
ericsnowwwitzel3: but it's worth a comment because if our assumptions are wrong it would effectively be a silenced error15:42
wwitzel3ericsnow: well, looking at this again, I think we should just check for deprecated, and log a warning about it being skipped (and any replacement)15:43
ericsnowwwitzel3: perfect!15:44
wwitzel3as for NewZone, I didn't see it used outside of the tests at all15:44
ericsnowwwitzel3: so it's even less of an issue :)15:51
=== ericsnow is now known as ericsnow_afk
mgzkatco: filed bug, also seems I reviewed your original branch so we've no one to blame but ourselves for missing that err return :)16:02
katcomgz: lol16:02
natefinchmgz: why is the Machine api called Machiner?  It's not a Go interface :/16:02
katcomgz: is that a 1.23 blocker?16:02
rogpeppedimitern: ping16:07
rogpeppeanyone out there have some decent knowledge of the ins and outs of the juju deploy command implementation?16:07
mgzkatco: I believe so16:10
katcomgz: ok16:10
mgznatefinch: it's one of the ancient ones16:11
katcomgz: have a link to the bug handy?16:11
mgzah, we haven't got the little bot in here, I thought it'd pop up in a minute16:11
mgzbug 143976116:11
mupBug #1439761: AWS V4 signing does not work <ec2-provider> <juju-core:Triaged> <juju-core 1.23:Triaged> <https://launchpad.net/bugs/1439761>16:11
mgzdoesn't add anything to what we've been over in here currently16:11
katcomgz: so, us-east bootstrapping with v4 isn't working, right?16:13
katcomgz: b/c i will have no way of closing this bug for china without someone's help16:13
katcomgz: (no test instance)16:13
mgzkatco: indeed, I believe v4 signing should work on us-east-1 - and doesn't, even with the date fix atm16:13
mgzkatco: I can give you ec2 creds if you need them as well16:14
katcomgz: well, again, i think signing is working fine :)16:14
katcomgz: it will complain if not16:14
katcomgz: it's something regarding the request16:14
katcomgz: ty, i will let you know. i have some i can try16:14
mgzkatco: my assuming, given the same creds work when v2 is used, is that the v4 code must be at fault16:15
mgzis there anything else I need to be supplying that I'm not, past the access and secret key (and region/endpoint)16:15
katcomgz: possibly the ec2 code is at fault for not providing some requisite in the request16:15
mgzkatco: sure, that also sounds possible16:16
mgzI tried removing the Timestamp query param as well as an experiment16:16
katcomgz: um, i don't think so. let me have a go and i'll let you know16:16
mgzkatco: hm, I have another idea16:19
alexisbvoidspace, hehe, I will have to make sure we have Fritos and tab at the sprint ;)16:19
katcomgz: i was using s3cmd when i was working on s3, is there a way to control ec2 instances in a similar way since i don't have access to a web console for this account?16:19
wwitzel3lol Fritos and Tab16:20
voidspacealexisb: good call :-)16:20
wwitzel3I'd like to opt-out16:20
voidspacehah16:21
voidspacewwitzel3: not allowed :-)16:21
mgzkatco: yeah, `apt-get install euca2ools`16:21
katcomgz: ty16:21
katcomgz: what's your other idea?16:21
mgzthen `euca-describe-instances` etc16:21
voidspacewwitzel3: I just linked alexisb to Code Monkey by Jonathan Coulton...16:22
wwitzel3ahhh, haha16:22
katcomgz: ty works16:22
mgzkatco: using x-amz-date only, as date is a standard http field and may get overwritten later16:23
mgzno joy yet16:23
katcomgz: good point16:23
mgzformat is probably wrong16:23
voidspaceand interestingly, Jonathan Coulton is how dooferlad and I met a few years before working together...16:24
voidspacein a cafe in London before a Jonathan Coulton concert16:24
mupBug #1439761 was opened: AWS V4 signing does not work <ec2-provider> <juju-core:In Progress by cox-katherine-e> <juju-core 1.23:Triaged> <https://launchpad.net/bugs/1439761>16:28
=== natefinch is now known as natefinch-lunch
katcomgz: i have a reproducible live test in amz16:46
katcomgz: same error16:46
mgzkatco: ace, I'm just having another go at date formats16:48
katcohttp://docs.aws.amazon.com/AmazonRDS/latest/APIReference/CommonParametersV4.html16:49
katcomgz: it request ISO 860116:49
katcomgz: but i believe the signing takes care of that conversion16:50
mgzkatco: it does, but I'm trying to understand the docs over what the headers are expected to me16:51
mgz*be16:51
mgzimplies only ISO8601, but switching to that didn't help16:52
mgzmessing with the date header at all seems suspect16:52
katcomgz: well, the signing will convert whatever you feed it to ISO 8601 Basic Format16:53
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
=== ericsnow_afk is now known as ericsnow
mgzkatco: for comparison, https://pastebin.canonical.com/128903/17:05
katcomgz: huh... we do ec2.us-east-1, and they do us-east-1.ec2.17:09
=== natefinch-lunch is now known as natefinch
* natefinch opens a window because even though there's still 8"(20cm) of snow on most of the ground... it's 75°f (24°C) inside17:58
* natefinch needs an imperial->metric units bot17:58
natefinchericsnow: core team meeting18:08
wwitzel3ericsnow: PTAL http://reviews.vapour.ws/r/1371/18:37
=== kadams54 is now known as kadams54-away
mupBug #1439813 was opened: Destroying lxc environment sometimes throws Go error <juju-core:New> <https://launchpad.net/bugs/1439813>19:02
=== kadams54-away is now known as kadams54
=== mgz_ is now known as mgz
ericsnowwwitzel3: LGTM19:19
=== lazyPower is now known as lp|away
natefinchhazmat: you had some magic with btrfs to make the local provider lightning fast.... do you have that documented somewhere?19:41
natefinchanyone on happen to know how why I might be getting "connection is shut down" when trying to use the machiner API in a watcher's Handle method?19:53
rick_h_natefinch: https://pastebin.canonical.com/128921/ is a copy of an email on this a while back19:58
rick_h_heh, april 2014 to be exact so a year ago19:59
natefinchrick_h_: awesome thanks!20:00
rick_h_natefinch: np, fulltext search ftw20:00
natefinchrick_h_: heh, I hadn't thought to just do a mail search, couldn't remember if he's actually emailed it out or not20:06
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
ericsnowcould I get a review on http://reviews.vapour.ws/r/1380/?20:22
ericsnowit adds a --file option to relation-set20:22
=== kadams54 is now known as kadams54-away
=== meetingology` is now known as meetingology
hazmatnatefinch: documented?.. just the code itself.. nutshell is if /var/lib/lxc is a btrfs mount, lxc can use btrfs cow snapshots for lxc-clone'd containers ( -s -B btrfs), so that and a template container per series.21:38
hazmatthe rest was just manual registration / with user data gathered from machinescripts api, so its async init21:39
ericsnowvoidspace: I thought you were long gone :)21:54
mupBug #1439880 was opened: Container's interfaces are all on private networks instead of host's eth0 network <oil> <juju-core:New> <https://launchpad.net/bugs/1439880>22:30
katcomgz: you still around by any chance?22:47
katcoaxw: did i misunderstand? are you all off today in AUS?23:18
anastasiamackatco: it's good friday today - public holiday... so yes :D23:58
anastasiamackatco: Monday is easter monday - another public holiday :D23:58
anastasiamackatco:23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!