[00:20] <sinzui> thumper: yes, I will jigger it now
[00:44] <mwhudson_> davechen1y: https://launchpad.net/ubuntu/+source/golang/2:1.5~rc1-0ubuntu1 \o/
[00:49] <sinzui> thumper: http://reports.vapour.ws/releases/2985/job/joyent-deploy-jes-trusty-amd64/attempt/117 used --debug
[00:51] <davechen1y> mwhudson_: /me pops cork
[01:01] <sinzui> thumper: menn0: joyent's networking non-sense is often a cause of not downloading agents from the state-server. That is the main reason for the reties on joyent jobs. Even when agents are started, the units fail. I think you can see thie in the 117 ^ attempt.
[01:46] <thumper> sinzui: the unit agent failure is almost certainly a problem with JES and the new debug log writer
[01:46] <thumper> menn0: ^^
[01:46] <thumper> sinzui: the failing to download tools in the machine agents cloud init is something else
[01:47] <thumper> menn0: I'm guessing the new manifold bollocks for the log sender is a problem
[01:47] <thumper> menn0: the people making defining the dependencies almost certainly got it wrong
[01:47] <sinzui> thumper: yeah, joyent has 72.* and 165.* addresses. We often see the private nets cannot see each other when joyent gives us differing public addresses
[01:48] <thumper> ah...
[01:48] <thumper> sinzui: what is our solution there?
[01:49] <menn0> thumper: sorry, playing catch up. where can I see the unit agent failure?
[01:49] <sinzui> thumper: retry, or deploy lots of machines to hold the 72.* addresses, release the 165.* addresses
[01:49] <thumper> menn0: http://data.vapour.ws/juju-ci/products/version-2985/joyent-deploy-jes-trusty-amd64/build-117/unit-dummy-source-0.log
[01:49] <sinzui> thumper: we tried prefer-ipv6, but we still see agents downloads using ip4
[01:49] <menn0> sinzui, thumper: i made a joyent specific routing fix a long time ago which allowed the various private nets they allocate to talk to each other
[01:49] <thumper> sinzui: hmm...
[01:50] <menn0> sinzui, thumper: is that not working any more?
[01:50]  * thumper shrugs
[01:50] <sinzui> menn0: I don't think it is that reliable, but maybe the firewall issue i describe in another bug is the root cause. I haven't escallated the other issue because I am still gathering evidence
[01:51] <menn0> sinzui: or maybe joyent has changed the way their network works
[01:51] <sinzui> menn0: they changed firewal rules recently :)
[01:52] <menn0> sinzui: ok right
[01:52] <menn0> thumper: should I take a look at this panic?
[01:53] <thumper> menn0: if you have the bandwidth, yes please
[01:54] <menn0> thumper: well there's also this bootstrap issue and the State pool work. tell me my priorites and I'll work to that :)
[01:54] <thumper> ugh
[01:54] <thumper> menn0: I'll poke the panic, you continue with bootstrap issue
[01:56] <menn0> thumper: k
[02:20]  * thumper sighs
[02:20] <thumper> at least the panic is entirely reproducable with the local provider
[02:20]  * thumper goes to make coffee
[02:40] <menn0> thumper: that's good to know. should be easy enough to track down then.
[02:40] <thumper> well, you'd think that
[02:40] <menn0> :)
[02:40]  * thumper pokes more
[02:41] <menn0> thumper: here's the bootstrap fix: http://reviews.vapour.ws/r/2407/
[02:42] <thumper> menn0: the connection is null in sendLogRecord
[02:42]  * menn0 pulls up current master
[02:42] <thumper> bah humbug
[02:44]  * thumper takes a breath
[02:44] <thumper> menn0: if you have time
[02:44] <thumper> would love a hangout to talk this through as I read the code
[02:44] <menn0> thumper: sure
[02:44]  * menn0 close the office door
[04:05] <thumper> menn0: I'm just testing, but I *think* that the one line error fix will make things work
[04:05] <thumper> just not as cleanly as we'd like
[04:05] <menn0> thumper: b/c it retries
[04:05] <thumper> ack
[04:05] <menn0> thumper: and at some point the api info will be correct....
[04:05] <menn0> nasty
[04:06] <thumper> ack
[04:06] <thumper> yeah
[04:06] <thumper> I'd still like to represent the dependencies correctly, but this will get us over the hump
[04:06]  * thumper is deploying now
[04:06]  * menn0 nods
[04:06] <menn0> it would definitely be nice to do it right
[04:07] <thumper> yep, that works
[04:20] <huwshimi> thumper: Hey. I've got a branch ready to land for jujulib. Is there a process for landing branches yet?
[04:21] <thumper> nope, but I can land it for you
[04:21] <thumper> if it is reviewed
[04:22] <huwshimi> thumper: It's this one. Has a couple of plus ones. https://github.com/markramm/jujulib/pull/3
[04:23]  * thumper takes a look at the actual diff too
[04:27] <thumper> huwshimi: merged
[04:27] <thumper> huwshimi: thanks, looks real good
[04:27] <huwshimi> thumper: Brilliant, thanks!
[04:27] <thumper> now that I've dealt with the bug I was looking at, I'm going to fix my review comments that didn't get addressed before mramm2 merged my branch
[04:31] <thumper> huwshimi: hmm...
[04:31] <thumper> huwshimi: found a bug in the makefile :)
[04:31] <huwshimi> thumper: oh...
[04:31] <thumper> calling make check when there is no .env setup fails
[04:31]  * thumper will fix
[04:34] <thumper> huwshimi: "rm .file" fails if .file doesn't exist
[04:34] <thumper> "rm -f .file" does not fail if it doesn't exist
[04:35] <huwshimi> thumper: Ah, nicely spotted. Thanks
[04:35] <thumper> now I get a lot of lint
[04:35]  * thumper sighs
[04:35] <huwshimi> thumper: Yes, yes you do.
[04:36] <huwshimi> thumper: and an 'assert False' :)
[04:37] <thumper> huwshimi: that was from mramm
[04:37]  * thumper passes the buck
[04:37] <huwshimi> haha
[05:20]  * thumper is done
[05:20] <thumper> laters
[08:44] <dimitern> morning
[08:47] <dimitern> fwereade, hey
[08:48] <fwereade> dimitern, o/
[08:48] <dimitern> fwereade, when you have a moment, I'd like you to review this http://reviews.vapour.ws/r/2406/ please
[08:48] <dimitern> fwereade, it should do everything we discussed about setting/getting constraints
[08:49] <fwereade> dimitern, cool, I will try to get to that, I see menn0 has covered a few things already?
[08:50] <dimitern> fwereade, yeah, and I really appreciate that, but since it's changes core functions a second look will be nice :)
[09:19] <Mmike> Hi, lads. What is juju using for leader election, which algo/protocl?
[09:54] <urulama> just to verify: can openstack provider deal with user tokens from Keystone instead of usernames and passwords? In case it does, how is refreshing done and where are they stored?
[10:46] <dimitern> dooferlad, hey, so my constraints branch is landing now
[10:46] <dimitern> dooferlad, after it lands it's a good time to sync up net-cli with master
[13:05] <mup> Bug #1486553 opened: i/o timeout errors can cause non-atomic service deploys <juju-core:New> <https://launchpad.net/bugs/1486553>
[13:27] <mattyw> fwereade, http://reviews.vapour.ws/r/2415/
[13:32] <jogarret6204> hi all,  I have an upgrade stuck, causing many errors such as: ..."blocked because upgrade in progress".  Any ideas how to unstick it?
[13:35] <frankban> hi ocr, could you please take a look at https://github.com/juju/juju/pull/3035 ? (this is a MP against a feature branch for the GUI embedded story). thank you!
[13:35] <natefinch> jogarret6204: upgrading what version to what version?
[13:35] <frankban> katco: ^^^
[13:37] <jogarret6204> natefinch:i think i am 1.24.3 now...  not sure how to tell what it is targeting..
[13:38] <jogarret6204> agent-version: 1.24.3.1
[13:39] <natefinch> jogarret6204: likely going to 1.24.5, if you started the upgrade today or yesterday.
[13:39] <jogarret6204> any way to kick it along?  seeing other issues now
[13:39] <jogarret6204> message: agent is lost, sorry! See 'juju status-history all-in-one/34'
[13:40] <jogarret6204> but can't check that, that command returns: ERROR upgrade in progress - Juju functionality is limited
[13:40] <natefinch> jogarret6204: is it just one machine or a bunch of machines?
[13:42] <jogarret6204> bunch.  I have maas on baremetal, it uses a juju VM on same box for state server.  then about 10 of these machings in issue right now
[13:43] <jogarret6204> I opened a juju-deployer bug - this may actually be the cause of that
[13:44] <jogarret6204> http://bit.ly/1KvMybk
[13:48] <natefinch> jogarret6204: interesting.  I don't really know the deployer code, so can't comment on what it does or does not do.  But certainly the unit number should not be used as the count of units.
[13:49] <natefinch> frankban: katco is going to be in late this morning, btw
[13:50] <frankban> natefinch: ok thanks, there is no rush
[13:51] <jogarret6204> natefinch:  i'm thinking that it may not be a bug if I get this upgrade issue fixed.
[13:56] <wwitzel3> ericsnow, natefinch: can we delay standup maybe 10 minutes? I lost track of time and have bacon on the stove
[13:57] <wwitzel3> well, actually, in the oven, but same thing
[13:58] <natefinch> wwitzel3: I can wait for bacon
[14:09] <ericsnow> fwereade, cmars: how do I "uninstall" a worker from an engine?  (Runner has StopWorker...)
[14:09] <fwereade> ericsnow, you can't, yet; was waiting for a direct need to do so
[14:10] <fwereade> ericsnow, what's your use case?
[14:11] <ericsnow> fwereade: I have per-workload-process workers with a definite lifetime
[14:11] <ericsnow> fwereade: when Juju stops tracking such a workload process then the worker must be stopped and forgotten
[14:12] <ericsnow> fwereade: for now I am resorting to using runners but would rather use dependency engine
[14:12] <fwereade> ericsnow, that sounds sane for the short term
[14:12] <ericsnow> fwereade: yeah, I figured we'd sort it out later :)
[14:13] <fwereade> ericsnow, sorry, I wasn't expecting to need it until I got relatively deeply stuck into the machine agent
[14:13] <ericsnow> fwereade: no worries :)
[14:13] <fwereade> ericsnow, (out of interest, what are the dependencies of your process workers?)
[14:14] <fwereade> ericsnow, (and their responsibilities?)
[14:16] <ericsnow> fwereade: deps - mostly API client; responsibilities - e.g. periodically update status from the underlying technology (e.g. docker)
[14:17] <fwereade> ericsnow, are the workers expected to, e.g., restart the processes if they fail?
[14:17] <fwereade> ericsnow, or are they just observers?
[14:17] <ericsnow> fwereade: not yet
[14:18] <ericsnow> fwereade: for now just observing
[14:18] <ericsnow> fwereade: later potentially starting and stopping them
[14:18] <fwereade> ericsnow, cool, thanks
[14:18] <ericsnow> fwereade: np
[14:18] <fwereade> ericsnow, a thought re responsibilities, not sure if it's good
[14:19] <fwereade> ericsnow, how hard would it be for one such worker to know when it was finished, itself, and return something like ErrUninstallMePlease?
[14:20] <ericsnow> fwereade: I'll think about it (OTP)
[14:20] <fwereade> ericsnow, I'm not sure that even covers all the machine-agent use cases tbh, it might just be a bad idea, but let me know if you think of anything
[14:21] <ericsnow> fwereade: will do
[14:41] <mup> Bug #1486297 opened: Action doesn't correctly translate unit name into tag if hyphen present <juju-core:New> <python-jujuclient:New> <https://launchpad.net/bugs/1486297>
[14:44] <mup> Bug #1486297 changed: Action doesn't correctly translate unit name into tag if hyphen present <juju-core:New> <python-jujuclient:New> <https://launchpad.net/bugs/1486297>
[14:56] <mup> Bug #1486297 opened: Action doesn't correctly translate unit name into tag if hyphen present <juju-core:New> <python-jujuclient:New> <https://launchpad.net/bugs/1486297>
[14:59] <marcoceppi> alexisb: health checks, are those on the roadmap?
[14:59] <alexisb> heh marcoceppi we were just chatting about that
[15:00]  * marcoceppi mind melds
[15:24] <mup> Bug #1486297 changed: Action doesn't correctly translate unit name into tag if hyphen present <juju-core:New> <python-jujuclient:New> <https://launchpad.net/bugs/1486297>
[15:31] <fwereade> dimitern, I'm feeling dense, would you explain: I think we need some way to distinguish between the "fallback to env spaces constraint" and "explicitly clear spaces constraint" cases
[15:31] <fwereade> dimitern, do we not need it; or do we do it but I just don't see it?
[15:33] <dimitern> fwereade, I'll try at least :)
[15:33] <marcoceppi> How do I tell juju what agent to bootstrap with?
[15:33] <marcoceppi> I have 1.24.4 isntalled but it bootstraps 1.24.5 trying to validate a regression
[15:33] <dimitern> fwereade, so explicitly empty values always override matching fallbacks (i.e. "mem=" overrides "" and "mem=4G"), but only when doing resolution from deployment to provisioning constraints
[15:34] <dimitern> fwereade, that's what now happens after my changes
[15:34] <dimitern> (soon to be available in master as well, when we merge net-cli)
[15:35] <wwitzel3> ericsnow, natefinch: https://github.com/juju/charm/pull/143
[15:36] <ericsnow> wwitzel3: ship-it
[15:36] <natefinch> officially used loggo's per-package logging adjustments for the first time in 2 years: juju set-env logging-config="juju.worker.leadership=WARNING"
[15:38] <natefinch> fwereade: any chance we were going to de-spam the leadership logging sometime soon?
[15:38] <fwereade> natefinch, huh, I thought we had
[15:38] <fwereade> natefinch, I thought iit was mostly at trace level
[15:38] <fwereade> natefinch, hmm, maybe we didn't do tracker?
[15:39] <natefinch> fwereade: yeah, the log lines I see are all from tracker
[15:40] <fwereade> natefinch, damn, sorry
[15:41] <natefinch> fwereade: not the end of the world.  Fixable via loggo (as long as you don't need to see anything under warning from leadership)
[15:42] <fwereade> dimitern, ah, ok, and if we had "spaces=foo" and replaced it with "mem=4G" we'd get the fallback spaces; but "mem=4G spaces=" would ensure no spaces constraints? or have I completely confused myself?
[15:43] <dimitern> fwereade, exactly right
[15:43] <fwereade> dimitern, cool
[15:43] <fwereade> dimitern, ok, then, in state -- how do we store the distinction between those two cases?
[15:43] <fwereade> dimitern, we seem to have lost the pointers in that struct
[15:44] <dimitern> fwereade, FWIW resolution was broken in a few places, e.g. adding a machine does resolution, SetConstraints on it before deployment doesn't and takes whatever you give it
[15:44] <fwereade> dimitern, ahh, machine constraints weren't including env fallbacks?
[15:44] <dimitern> fwereade, when set, but when added it worked as expected
[15:45] <dimitern> fwereade, I *think* it's only important to store non-empty values (after doing resolution)
[15:45] <fwereade> dimitern, sorry, lost again, how can you set constraints on a machine?
[15:45] <dimitern> fwereade, before provisioning
[15:45] <dimitern> fwereade, m.SetConstraints()
[15:45] <fwereade> dimitern, can users do that?
[15:45] <fwereade> dimitern, (other than when adding?)
[15:46] <fwereade> dimitern, I think those values should just be coming from the resolved env+service constraints for the unit whose addition triggered machine addition
[15:47] <dimitern> fwereade, I don't think so
[15:47] <dimitern> fwereade, but it's still a bug
[15:47] <fwereade> dimitern, but either way, when we're storing service constraints in state we shouldn't resolve them
[15:47] <dimitern> fwereade, I agree, and I made it so it's definitely like this in both cases
[15:48] <dimitern> fwereade, we're not
[15:48] <dimitern> fwereade, we only resolve unit constraints when asked
[15:49] <fwereade> dimitern, how can we do that correctly if we're throwing away the distinction between "fallback" and "clear" in the service constraints we store in mongo?
[15:50] <dimitern> fwereade, let me look at the code
[15:51] <fwereade> dimitern, np, sorry it's taken me so long to start looking
[15:52] <dimitern> fwereade, ok, that's a good catch sir
[15:52] <fwereade> dimitern, yay, my brain still works :)
[15:52] <dimitern> fwereade, so we should store them when empty, at least for the services
[15:52] <fwereade> dimitern, I think so, yeah
[15:52] <dimitern> fwereade, good, it should be easy to fix
[15:52] <fwereade> dimitern, and probably across the board, even if the distinction is academic once resolved
[15:52] <fwereade> dimitern, cool
[16:08] <TheMue> dimitern: just before riding home by bike, http://reviews.vapour.ws/r/2419/ contains the changes we talked about yesterday. hints regarding the testing of the finishedWorker are welcome. I'll take a look when I'm at home
[16:15] <wwitzel3> cmars: ping
[16:24] <mup> Bug #1486640 opened: Typos in help  <juju-core:New> <https://launchpad.net/bugs/1486640>
[16:30] <dimitern> TheMue, sure, will have a look in a bit
[16:46] <jogarret6204> just checking back in and seeing a team call here.. am I in wrong group for "general noob help"?  Sorry if so.  where is general help?
[16:47] <lazyPower> jogarret6204: general noob help is in #juju :)
[16:47] <lazyPower> #juju-dev is primarly for those hacking on juju core
[16:47] <lazyPower> you're more than welcome to hang in both places though, we welcome all feedback
[16:51] <jogarret6204> don't want to slow you down here, so I'll hit the other.  thanks LazyPower and natefinch.
[16:59] <natefinch> jog: sorry, yeah, #juju is more the general help channel :)
[17:04] <natefinch> jog: sorry, obv not meant for you
[17:28] <sinzui> katco: dimitern : can either of you arrange fix for bug 1486675. I think the test needs more smarts, juju is fine
[17:28] <mup> Bug #1486675: supportedSeriesWindowsSuite.TestSupportedSeries fails <blocker> <ci> <regression> <unit-tests> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1486675>
[17:28]  * perrito666 is connected though his phone and loving his isp
[17:29] <perrito666> well creating a simple stream over 1/2 3g really makes me save on heating, the phone is enough for the whole office
[17:30] <natefinch> heh
[17:31] <natefinch> wow, this is a dumb error: ERROR environment destruction failed: destroying environment: container "nate-local-machine-1" is not yet created
[17:32] <perrito666> well, that was actually possible
[17:32] <perrito666> I think I recall davechen1y or thumper talking about a test that tried to reproduce that by being a race condition
[17:33] <mup> Bug #1486166 changed: JES deploy fails <ci> <jes> <regression> <juju-core:Fix Released by thumper> <https://launchpad.net/bugs/1486166>
[17:33] <mup> Bug #1486675 opened: supportedSeriesWindowsSuite.TestSupportedSeries fails <blocker> <ci> <regression> <unit-tests> <windows> <juju-core:Triaged> <https://launchpad.net/bugs/1486675>
[17:34] <katco> sinzui: dimitern's team is on bug-squad, so i'll leave it to him
[17:38] <natefinch> perrito666: my point was rather, if I am destroying the environment, I don't care that there's a container that isn't created yet, that's just one less thing to tear down.
[17:39] <dimitern> TheMue, you have a review
[17:40] <dimitern> sinzui, katco, sure, let me have a look
[17:41] <dimitern> sinzui, this looks like a fallout of a recent change I saw
[17:41] <sinzui> dimitern: yes
[17:43] <dimitern> sinzui, this one most likely https://github.com/juju/juju/pull/2981
[17:43] <dimitern> bogdanteleaga, are you around?
[17:43] <sinzui> yep, that is what I saw
[17:44] <bogdanteleaga> dimitern, looking into it
[17:44] <dimitern> sinzui, katco, so in these case what - is a revert in order?
[17:44] <dimitern> bogdanteleaga, thanks!
[17:44] <sinzui> dimitern: a revert will unblock. otherwise race to land a fix
[17:47] <bogdanteleaga> dimitern, sinzui https://github.com/juju/juju/pull/3040
[17:48] <bogdanteleaga> dimitern, sinzui I g2g now, merge it once it gets reviewed please
[17:48] <sinzui> thank you bogdanteleaga
[17:54] <dimitern> bogdanteleaga, awesome, ta!
[17:56] <dimitern> sinzui, setting it to merge
[17:56] <sinzui> you rock dimitern
[17:58]  * dimitern wishes all bugs where like this :)
[17:59] <natefinch> OMG.... just realized what the problem I've been fighting with all day..... printing a value out with %v was causing a panic from inside fmt somewhere
[18:00] <perrito666> aghh gce is saying me I am not authenticated... only for one particular operation wth
[18:06] <dimitern> sinzui, btw I'm not sure if you're monitoring feature branches for trends in failures like for the main branches, but if there is some data about "net-cli", which we're planning to merge tomorrow, will be awesome
[18:07] <sinzui> dimitern: you cannot merge it because it has never passed http://reports.vapour.ws/releases#net-cli
[18:07] <sinzui> dimitern: merge tip, When CI blesses it, we can merge it into master
[18:07] <dimitern> sinzui, hmm that's useful to know
[18:07] <dimitern> sinzui, we just did that today
[18:08] <sinzui> dimitern: I shall try to force ci to retest net-cli.
[18:08] <dimitern> sinzui, it's currently only 4 commits behind
[18:08]  * sinzui is trying ton retest 1.24 today as well
[18:08] <dimitern> sinzui, great, thanks! it will be nice to have some early feedback
[18:09] <sinzui> dimitern: maybe this issues are fixed already https://bugs.launchpad.net/juju-core/net-cli
[18:10] <dimitern> sinzui, I hope so, however the windows one is a bit worrying
[18:10] <dimitern> sinzui, or you mean because it's gone from master?
[18:13] <sinzui> dimitern: I hope the issue was really in master, and your merge fixed it
[18:15] <dimitern> sinzui, I'll give it a try now as I'm changing that list command
[18:16] <sinzui> alexisb: I think we have enough evidence to say Joyent's firewall changes did hurt Juju, and that deleting them when Juju destroys the environment fixes the issue: I want bug 1485781 fixes in 1.25 and 1.24 (maybe 1.22 if we ever plan a release)
[18:16] <mup> Bug #1485781: Juju is unreliable on Joyent <joyent-provider> <reliability> <repeatability> <juju-core:Triaged> <juju-core 1.22:Triaged> <juju-core 1.24:Triaged> <https://launchpad.net/bugs/1485781>
[18:38] <katco> ericsnow: natefinch-afk: wwitzel3: sorry i missed the stand-up this morning. anything i can help with?
[18:39] <ericsnow> katco: review http://reviews.vapour.ws/r/2405/ ?
[18:39] <ericsnow> katco: (and don't sweat missing standup :)
[18:39] <katco> ericsnow: tal
[19:20] <alexisb> sinzui, thank you for getting the info on joyent and opening the bug
[19:20] <alexisb> that is good stuff
[19:24] <mup> Bug #1486712 opened: Race on uniter-hook-execution, prevents to resolve unit. <sts> <juju-core:New> <https://launchpad.net/bugs/1486712>
[19:53] <katco> ericsnow: reviewed
[19:53] <ericsnow> katco: thanks!
[21:37] <mup> Bug #1474885 changed: juju deploy fails with ERROR EOF <local-provider> <precise> <juju-core:Fix Released> <https://launchpad.net/bugs/1474885>
[21:52] <mup> Bug #1486749 opened: juju backups create should fail earlier for hosted environments <juju-core:In Progress by cherylj> <https://launchpad.net/bugs/1486749>
[21:58] <mup> Bug #1486749 changed: juju backups create should fail earlier for hosted environments <juju-core:In Progress by cherylj> <https://launchpad.net/bugs/1486749>
[22:07] <mup> Bug #1486749 opened: juju backups create should fail earlier for hosted environments <juju-core:In Progress by cherylj> <https://launchpad.net/bugs/1486749>
[22:36] <davecheney> alexisb: ping
[23:31] <menn0> waigani: would you mind taking a look at http://reviews.vapour.ws/r/2425/ pls?
[23:51] <perrito666> ericsnow: still here?
[23:51] <ericsnow> perrito666: barely
[23:51]  * perrito666 sees ericsnow fading
[23:52] <perrito666> ericsnow: go use gce with the fields instead of the json file, is it enough to just copy the values of the json?
[23:53] <ericsnow> perrito666: pretty much
[23:53] <perrito666> ericsnow: I believe the issue I told you about earlier might be because storage provisioner tries to do stuff in a machine and finds itself lacking these creds
[23:53] <ericsnow> perrito666: the PK might not copy-and-paste quite right so you have to watch that
[23:54] <ericsnow> perrito666: does it do more than make calls on the provider?
[23:54] <perrito666> ericsnow: calls that require auth
[23:55] <ericsnow> perrito666: the provider has all the auth it needs
[23:55] <perrito666> the provider?
[23:56] <ericsnow> perrito666: provider/gce/...
[23:56] <perrito666> ericsnow: well I am getting auth errors from one of the machines
[23:56] <perrito666> so :) something is wrong
[23:56] <ericsnow> :)
[23:57] <ericsnow> perrito666: you're adding new methods to the gceConnection interface (in environ.go), right?
[23:57] <perrito666> ericsnow: yep
[23:58] <ericsnow> perrito666: then I would definitely not expect auth issues :/
[23:58] <perrito666> well it is only happening with one machine i think
[23:58] <perrito666> that I added with add-machine
[23:58] <ericsnow> perrito666: do you have to enable some extra permissions in the GCE developer console?
[23:58] <perrito666> so I am looking it up
[23:58] <ericsnow> (manually)
[23:59] <perrito666> ericsnow: where is that stored on the server?
[23:59] <ericsnow> perrito666: where is what stored?
[23:59] <perrito666> ericsnow: the oauth token