/srv/irclogs.ubuntu.com/2015/02/24/#juju-dev.txt

beisnersinzui, so in the openstack provider attempt, juju fires up a vivid instance, and i can see the nova console log that it's booted and ready with whatever userdata was passed to it.  but there is a key mismatch.00:00
beisnersinzui, with the maas deploy, i get the same symptom (key issue), and it's much harder to get console output.00:00
beisnerer rather, maybe not a key mismatch, but definitely a key issue.00:01
sinzuibeisner, I don't think key issues are series issues00:01
beisnersinzui, all those woes go away when I set  default-series:  utopic   or  trusty   in the environments.yaml.00:01
sinzuibeisner, I think you found a bug :)00:03
beisnersinzui, ok, what can i do/collect to raise a helpful/meaningful bug on this?00:04
beisnerie.  is there a way to get more verbosity from juju bootstrap?00:04
sinzuibeisner, the cloud-init-output log and maybe a machine-0 log if it gets that far00:05
beisnersinzui, unit 0 never comes alive according to juju00:05
beisnerand juju debug-log is no help at that stage00:05
sinzuibeisner, I often ssh into the machine the moment I see the ip is open and tail the /var/log/cloud-init-output.log00:06
beisnersinzui, ah cool.  i'll dive in a bit more.  appreciate the guidance.00:07
wwitzel3ericsnow: just realized I forgot to hit publish on that review, you have a review from me now.00:07
ericsnowwwitzel3: thanks00:07
wwitzel3ericsnow: mostly minor, I found it all easy to follow, no comments about the service stuff since it is pretty generic boiler platey and we talkeda bout it before00:09
ericsnowwwitzel3: cool00:09
wallyworldthumper: you around?00:20
wallyworldsinzui: so we should be able to get 1.22 out now right? since the precise upgrade issue is only 1.23?00:28
sinzuiwallyworld, no, because we never got a pass for 1.2200:32
wallyworldsinzui: assuming ci becomes happy00:32
sinzuiwallyworld, 1.22-beta4 has NEVER passed CI00:32
wallyworld:-(00:32
sinzuiwallyworld, if it does pass, then I release00:32
wallyworldlet's hope this one works00:33
davecheneythumper: ping00:41
* thumper is here now00:44
wallyworldthumper: can i grab 5 when you are free?00:47
thumperwallyworld: sure, I'll be done chugging lunch in about 5 minutes00:48
wallyworldnp00:48
thumperit's a banana protein smoothy00:48
wallyworldmmmmm00:48
anastasiamac5min for smoothie?00:48
anastasiamacthy*00:49
davecheneythumper: shall we meet in the 1:1 hangout ?00:49
thumperdavecheney: yep, how about in 11 minutes?00:49
davecheneythumper: go talk to wallyworld first00:49
thumperta00:49
davecheneyi'll see ou in the hangout in whenever00:49
thumperwallyworld: our 1:1?00:53
wallyworldyup00:53
wallyworldsinzui: i just looked at the dashboard and the latest 1.23 run had local-upgrade-precise-amd64 passing00:58
wallyworldhave you upped the timeout already?00:58
sinzuiwallyworld, I did, that was my proof01:13
wallyworldsinzui: logs are needed to help see where the time is going01:14
wallyworldi can't see any linekd to the dashboard01:15
sinzuiwallyworld, I can give you the failures, but the passes might be more informative since I also gave the slaves more time to collect01:15
sinzuis/slave/services01:15
wallyworldsinzui: could you attach both to the bug for me?01:16
sinzuiwallyworld, I will see what I can do. I don't want to make them public if the contain credentials01:16
wallyworldsinzui: make the bug private?01:16
sinzuiwallyworld, I cannot because that will hide the critical blockers01:18
wallyworldsinzui: oh, maybe send privately?01:18
sinzuiwallyworld, this wouldn't be awkward if the credentials for reports.vapour.ws had not also failed this weekend too01:18
=== kadams54-away is now known as kadams54
sinzuiwallyworld, did you see smoser's ruling on apt-get dist-upgrade01:25
wallyworldsinzui: one sec, just finishing standup01:25
wallyworldsinzui: read scott's bug comments, we should be ok - we don't use proposed pocket with precise01:43
wallyworldthat i know of01:43
alexisbmenn0, thumper is this doc up to day??:01:49
alexisbhttps://docs.google.com/a/canonical.com/document/d/1jsuoTbXZbj3wtoXCpc5MGFVvwuofYx3hLw2eNoXEmE0/edit#heading=h.aby6yid7wq2d01:49
* menn0 looks01:49
menn0alexisb: yes, except that now we have a working proof of concept of logging to MongoDB and have run scaling tests (see your email)01:50
sinzuiwallyworld, we never use proposed01:50
menn0alexisb: I have been working on turning the POC into production code but whether it gets merged is somewhat dependent on whether the performance hit is deemed ok01:51
alexisbmenn0, yep understood, I just need a way to capture the work for logging that can be shared with those that are interested01:51
sinzuiwallyworld, but juju does do apt-get upgrade, which didn't give us an updated cloud-utils. apt-get install  then did a remove01:51
menn0alexisb: ok. let me know if you need to know more.01:52
alexisbie answer the question for "why is logging for JES important and requires work"01:52
alexisbnope I think that gets me what i need01:52
alexisbthanks01:52
wallyworldsinzui: so maybe the fix to put cloud-utils and cloud-image-utils on the one line is not required anymore01:52
sinzuiwallyworld, it is required because we haven't changed anything else to ensure removals cannot happen01:53
wallyworldok01:53
menn0wallyworld, sinzui: you guys seem to be discussing the same part of the code that I just filed a bug about. (bug 1424892)01:53
mupBug #1424892: rsyslog-gnutls is not installed when enable-os-refresh-update is false <cloud-init> <juju-core:New> <https://launchpad.net/bugs/1424892>01:53
wallyworldmenn0: nah, different01:54
sinzuimenn0, yes01:54
wallyworldthis is the deb packaging issue whcich affected cloud-utils01:54
* thumper sees that menn0 has answered all of alexisb's questions01:54
thumpercoffee time then...01:54
wallyworldmenn0: your issue is the behaviour of the flags to disable apt01:54
sinzuimenn0, apt-get update finds the new packages (in cloud-* example) but apt-get update will not install them because update is not permitted to install new deps!01:55
sinzuimenn0, but apt-get dist-upgrade can install new deps01:55
wallyworldmenn0: when upgrading, do you recall if the state server rejects connections until all nodes in the env are deemed to have upgraded?01:55
menn0wallyworld: not quite ... let me quickly look at the code01:56
menn0wallyworld: a state server will accept API connections once it itself has upgraded01:59
menn0wallyworld: but state servers always upgrade first, before other nodes01:59
wallyworldmenn0: ok, ta. i'm looking into the CI blocker where precise upgrades tie out01:59
wallyworldthere's a bucket load of mongo connection failures that go on and on01:59
wallyworldbut state server probably not at fault if it accepts connections after it has finished02:01
menn0wallyworld: do you have some logs handy?02:01
wallyworldmenn0: i'll forward an email. sinzui had to increase timeout from 10 mins to 20 to make CI local precise upgrades pass. but just precise02:02
menn0wallyworld: that is certainly odd.02:02
wallyworldindeed02:02
wallyworldprecise runs a slightly different mgo version02:02
wallyworldthat's all i can think of off hand02:02
sinzuiwallyworld, hp's swift isn't publish failed. I am getting out the hammers02:03
wallyworldwhat's that about hp?02:03
sinzuiwallyworld, timouts uploading02:04
wallyworldoh joy02:04
sinzuiwallyworld, I switch the job to not rebuild. just try to publish what the previous job made02:05
wallyworldok02:05
=== kadams54 is now known as kadams54-away
beisnerwallyworld, sinzui - back for a bit, raised a bug on that vivid thing.   should be readily reproducible but holler if there are any ?s.    https://bugs.launchpad.net/juju-core/+bug/142490002:14
mupBug #1424900: Bootstrapping Vivid:  ERROR failed to bootstrap environment, Permission denied (publickey), ci-info: no authorized ssh keys fingerprints found for user ubuntu <openstack> <uosci> <juju-core:New> <https://launchpad.net/bugs/1424900>02:14
wallyworldthank you02:14
wallyworldsinzui: menn0: with those logs, i see the 1st 4 minutes spinning up the state server and machines 1,2, then the state server upgrade completes in about  minute, then we see a tonne of connection terminated errors lasting several mintes, so it seems there's an issue wit the worker nodes upgrading02:19
menn0wallyworld: yep, i'm looking at those logs now02:19
menn0wallyworld: the machine-0 logs are all perfectly normal02:19
menn0wallyworld: but the machine-1 logs indicate that the agent never restarted into the new version02:20
wallyworldmenn0: in machine 1 lo, i see a 3 minute gap fetching tools02:20
menn0wallyworld: it sees the need to upgrade but never seems to reboot02:20
wallyworldmenn0: the test may have timed out before machine 1 could restart02:21
wallyworldtake off the 3 minutes to fetch tools02:21
sinzuimenn0, its about timing. I was on the machine when machine 1 was shutdown because of a timeout. it was upgrading. so I hacked the job on the precise slave to give it 20 minutes to see a pass.02:21
wallyworldand it probably would have been ok02:21
menn0but 20 mins is just crazy02:22
sinzuimenn0, all machines normally upgrade in less that 60 seconds. so 18 minutes is scarry02:22
wallyworldsinzui: this is what i see as the issue02:22
wallyworld2015-02-23 18:36:07 INFO juju.worker.upgrader upgrader.go:201 fetching tools from "https://10.0.0.191:17070/environment/329350b1-edf2-4d62-8156-7338a12d3808/tools/1.23-alpha1.1-precise-amd64"02:22
wallyworld2015-02-23 18:39:52 INFO juju.utils http.go:66 hostname SSL verification disabled02:22
wallyworldthat almost 4 minute gap02:22
menn0looking at the timestamps, the machine was going VERY slowly02:22
menn030s between seeing the need to upgrade and then /starting/ to download the tools02:23
sinzuimenn0, yep. I rebooted the machine too. It is fast when I use it02:23
menn0sinzui, wallyworld: the agent is still starting up when it sees the need to upgrade. the timings between the various workers starting up is rather wide, like the system was crawling.02:24
wallyworldyes02:25
wallyworldit just seems machine 1 is very, very slow02:25
sinzuimenn0, it wasn't/isn't02:25
menn0sinzui: it might not be, but it just looks that way based on the logs02:27
sinzuimenn0, I also cleared /var/cache/lxc/cloud-precise02:28
wallyworldi'm not sure what to do now to diagnose further - if it really is just precise, that makes it very hard to reason about02:28
sinzuiwe are still waiting for the same job to run with 1.22 to compare02:28
wallyworldi guess we see how 1.22comes out02:29
menn0wallyworld: we could spin up a precise instance on ec202:29
wallyworldwe could, i might do that02:30
menn0wallyworld, sinzui: as an example, you can see the difference the "slow down" when you look at the deployer worker in the logs02:33
menn0wallyworld, sinzui: before the API disconnection (due to the state server upgrading itself), the deployer worker starts 1s after it's parent worker (api-post-upgrade)02:34
menn0wallyworld, sinzui: at the bottom of the logs it starts 5.5 mins after it's parent worker02:35
sinzuiwow02:35
menn0wallyworld, sinzui: that's pretty strange02:35
sinzuithat is indeed the difference we feel watching it02:36
sinzuimenn0, the upgrade jub just ran with 1.22.02:37
wallyworldhmmm. maybe something is trashing the disk? or eating to cpu?02:37
menn0wallyworld: that's what i'm thinking. and that thing could even be something in Juju itself.02:37
sinzui2015-02-24 02:35:45 INFO juju.cmd.juju upgradejuju.go:214 started upgrade to 1.22-beta4.102:37
sinzuiand status shows it complete at02:37
sinzui2015-02-24 02:36:0902:37
menn0sinzui: what instance spec is being used for the precise tests?02:37
menn0sinzui: that's the kind of timing I would have expected02:38
wallyworldbut is the above for the state server? or a machine?02:38
wallyworldah02:38
wallyworldthe cmd02:39
sinzuimenn0, the precise slave has 8G ram with 4 vcpu02:39
sinzuimenn0, at this moment it has lots of resources free, but it was busy last hour building and testing02:40
sinzuimenn0, there was nothing for us to cleanup when we started investigating. the machine was fast for us02:40
wallyworldsinzui: the precise slave is dedicated to running the juju test in question?02:41
sinzuiwallyworld, it is02:41
menn0sinzui: i'm looking at jenkins. are the latest successes because of the extended timeout?02:43
sinzuimenn0, the 2 master ones are02:43
menn0sinzui: kk02:44
sinzuimenn0, the 1.22 passed normally02:44
menn0sinzui: 18mins is just nuts02:46
sinzuimenn0, sure, but not for maas. I wondered if recent changes needs a new deps and extra work for precise02:46
menn0wallyworld: are you spinning up a precise instance? i'd like to poke around as the upgrade happens02:47
wallyworldmenn0: just resetting by source tree02:47
wallyworldmy02:47
menn0sinzui: perhaps bit I can't imagine what would cause this02:47
menn0but02:47
menn0sinzui: one thing that could be helpful is if the env had debug logging turned on.02:55
menn0sinzui: it starts of in debug but switches to info for the root logger02:56
menn02015-02-23 18:32:05 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=INFO;unit=DEBUG"02:56
sinzuimenn0,  I can do that now that ci is locked down02:56
sinzuimenn0, I can switch it now, then we wait for 1.23 to test02:56
menn0wallyworld: meant to say... all those API disconnect messages in the machine-0 logs are just due to the repeated "juju status" polling that the test script does. that's fairly normal.02:57
sinzuimenn0, debug is in place02:57
menn0sinzui: awesome02:57
wallyworldmenn0: i have a bootstrapped precise instance - did you want me to add your ssh public key?02:58
menn0wallyworld: please02:58
* thumper tries for focus for an hour02:59
thumperif it is urgent, text me02:59
* thumper doesn't expect anything urgent02:59
wallyworldmenn0: 54.82.35.127  i'll start a precise worker machine03:00
wallyworldmenn0: i'm a tool - i bootstrapped 1.23 not 1.2103:02
wallyworldffs03:02
menn0it's so hard to get good help these days...03:02
wallyworldforgot to type /usr/bin/juju03:03
wallyworldsigh03:03
menn0wallyworld: has that host gone away? I can't connect to it now.03:14
wallyworldmenn0: yeah, almost done starting a 1.21 host, sorry03:15
wallyworldit's been slow to come up03:15
wallyworldmenn0: machine 0 is 54.158.193.4003:17
wallyworldmachine 1 is 54.204.193.5303:17
menn0wallyworld: I thought you were just going to test with the local provider on a single precise machine03:19
wallyworldmenn0: i was curious to see how precise in general went03:19
wallyworldbut we can do both03:20
menn0wallyworld: is my key there?03:20
wallyworldyep03:20
wallyworlddo you want to use machine as host for a local env03:20
wallyworldmachine 003:20
wallyworldmongo would need to be stopped03:21
wallyworldi should just fire up a new machine03:21
menn0wallyworld: I was using the wrong key... i have a personal one and a canonical one. fixed now03:21
menn0wallyworld: how about I fire up the machine for the local test03:23
wallyworldok03:23
wallyworldmenn0: upgrade on aws precise was fast, so it has to be a resource contention issue03:29
menn0wallyworld: ok. a useful data point. i'm just setting up this other instance now.03:29
menn0sinzui: SSD or magentic storage?03:29
sinzuimenn0, I think the latter. the machine is in Hp03:31
menn0sinzui: cool. i'll go with that.03:31
menn0wallyworld: it's 54.190.88.226. installing 1.21 now.03:41
wallyworldok, does it have my ssh key imported?03:41
sinzuiwallyworld, all the mass deploy jobs are still failing.03:42
wallyworldhmmm03:42
wallyworldi wonder what's different compared with clouds03:43
wallyworldthe same scripts should work on both03:43
wallyworldis there a url with logs?03:43
sinzuiwallyworld, no, we cannot get logs for maas because they have unresolvable dns03:46
wallyworldoh joy, this will be fun to solve03:46
sinzuiwallyworld, I am attempting it get into the maas to find the names, for the unit, them use virt-viewer to connect directly to the console03:46
sinzuiI dont' have creds though for maas 1.7 and 1.8 though03:47
menn0wallyworld: well i'm seeing a problem with the upgrade on precise but it doesn't look the same as what happen in the logs from sinzui04:28
menn0wallyworld: machine-0 fails to upgrade because of:04:29
menn0"set AvailZone in instanceData" failed: failed verification of local provider prerequisites:04:29
menn0cloud-image-utils must be installed to enable the local provider:04:29
menn0    sudo apt-get install cloud-image-utils04:29
menn0wallyworld: shouldn't the upgrade step handle that itself?04:29
wallyworldthat's expected04:30
wallyworldthat package has to be on the host machine04:30
wallyworldas do one or two others04:30
wallyworldhmmm, maybe04:30
wallyworldbut in general04:30
wallyworldthe local provider checks for needed packages and prints those messages if they are not there04:30
menn0I installed juju-local so shouldn't this already have been taken care of04:30
menn0?04:31
menn0wallyworld: maybe it got uninstalled by another operation?04:31
wallyworldcloud-image-utils isn't a prereq of juju local, maybe it should be?04:31
menn0wallyworld: i'll work around this for now so that I can get to the actual problem that CI is seeing04:32
wallyworldi think in trusty cloud-utils includes cloud-image-utils04:32
wallyworldi can't wait for precise to go away, but still 2 years keft04:32
sinzuiwallyworld, I cannot copy this crap gui's cloud-init-output04:33
wallyworldsigh04:33
wallyworldis there an obvious error?04:34
sinzuiwallyworld, I can say it is the same error as before where the apt-line is garbled04:35
wallyworldi'm trying to spin up maas locally but the bootstrap node refuses to transition from Deploying04:35
wallyworldthat doesn't make sense as there should be no quotes there now04:35
wallyworldand why just maas and now AWS or HP etc04:35
sinzuiand it died as I almost got the log04:37
sinzuiwallyworld, as per the bug04:37
sinzuiutil.py[WARNING]: Failed to install packages: ['bridge-utils', 'curl cpu-checker bridge-utils rsyslog-gnutls cloud-utils cloud-image-utils']04:37
wallyworldsigh04:38
* sinzui double checks commit 04:38
wallyworldseems bridge-utils is added twice which is messing it up04:38
sinzuiwallyworld, are are testing your commit04:38
wallyworldyes on aws04:39
wallyworldworks perfectly04:39
sinzuiwallyworld, apt doesn't care about duplicates on the command line04:39
wallyworldsure, but adding twice seems to mess up juju's rendering of the cmd line04:39
wallyworldsinzui: can i get access to your maas?04:40
wallyworldif i can't reproduce, i can't fix04:40
sinzuiwallyworld, sure but it so poorly documented I cannot offer much04:40
wallyworldmaybe pm or email the ip of the controller and auth key04:41
menn0wallyworld, sinzui: i haven't been able to repro this precise upgrade timeout issue yet but I still have a few ideas04:48
wallyworldsinzui: ah, i see where bridge-utils is added twice - it's hard coded in the maas provider05:01
wallyworldthat's why it is failing on maas05:02
sinzuiwallyworld, really>05:02
sinzuiwallyworld, jog hack the local machine to capture logs from machine-0. This isn't too helpful since the failures are machines 1 and 305:03
wallyworldsinzui: yes, on machine 0 we run the scripts ourselves, not cloud init05:03
menn0sinzui, wallyworld: ok, i'm stumped on how to reproduce this precise issue05:07
wallyworldsinzui: it may be the quickest thing to do is to go back to installing one package at a time now that cloud-image has been moved into the correct repo05:07
menn0sinzui, wallyworld: every upgrade works quickly for me05:07
sinzuiwallyworld, yes, with the caveat that we must also install clout-utils to force the upgrade05:08
wallyworldsinzui: yep, i'll install cloud-utils along with the other ones we expect05:08
sinzuiwallyworld, or we do dist-upgrade (but I think that is risky for today)05:09
wallyworldyeah, i'll keep it simple05:09
wallyworldmenn0: i think juju might be ruled out - something must be thrashing the machine though05:09
menn0wallyworld: either that or i'm missing some aspect of the test setup05:10
menn0i'm about to EOD05:10
wallyworldok, thanks for looking05:11
wallyworldsinzui: do i need to do apt-get install --target-release precise-updates/cloud-tools cloud-utils   ?05:15
wallyworldor can i leave off the --target-release bit05:16
wallyworldi think i need it right? for precise?05:16
menn0sinzui, wallyworld: of course I just realised that I was testing with a 1.23 that was a little behind the times and didn't include some of the recent commits that might be contributing to the issue05:25
* menn0 facepalms05:25
menn0sinzui, wallyworld: it might be worth repeating what i've done...05:25
wallyworldhard to get good help :-)05:25
menn0touche05:25
wallyworldi gotta fix this other one first :-(05:25
menn0but I really need to EOD or my wife is going to get pissed05:26
wallyworldnp, leave it to us05:29
joghi wallyworld05:50
jogwallyworld, I was just looking at the MaaS test results for 1.22 revision 79e5ea8a and still see the failure mentioned in bug 1424695.05:53
mupBug #1424695: maas cloud-init cannot download agent from state-server <ci> <maas-provider> <network> <regression> <trusty> <juju-core:In Progress by wallyworld> <juju-core 1.22:In Progress by wallyworld> <https://launchpad.net/bugs/1424695>05:53
jogwallyworld, it's looks like you through that commit should have fixed that bug?05:54
jogthought even05:54
jogwallyworld, looks like maybe you dropped for a bit, did you see my comment above?05:58
wallyworldjog: no05:58
jogwallyworld, I was just looking at the MaaS test results for 1.22 revision 79e5ea8a and still see the failure mentioned in bug 1424695.05:58
mupBug #1424695: maas cloud-init cannot download agent from state-server <ci> <maas-provider> <network> <regression> <trusty> <juju-core:In Progress by wallyworld> <juju-core 1.22:In Progress by wallyworld> <https://launchpad.net/bugs/1424695>05:58
joglooks like you expected that commit to resolve that quoted package string issue?05:59
wallyworldjog: yes, sadly maas fails where the other clouds are ok, so i've marked bug as in progress again. i don't have a working maas to test with05:59
jogwallyworld, you have access to finfolk.internal ?06:00
wallyworldno, i tried and couldn't get in06:00
* jog wonders what happened to core vmaas setup on gremlin.internal that was setup during the Brussels sprint.06:16
dimiternjog, well, we were supposed to use it for ipv6 work of maas and juju, but we didn't need to06:22
dimiternjog, IIRC the maas guys used it for qa stuff (or was it finfolk?)06:22
jogdimitern, finfolk is used by juju-qa but I have an extra env setup for debugging for anyone on core that needs it06:23
dimiternjog, right, that's good to know then :)06:24
wallyworlddimitern: hi, i'm a little concerned that the vet warnings can be suppressed. shouldn't we be fixing the warnings? those provisioner ones are annoying :-)06:42
dimiternwallyworld, they should've been fixed yesterday by jw406:43
dimiternwallyworld, but maybe it bounced due to the blocker06:43
dimiternwallyworld, nope - it did bounce - https://github.com/juju/juju/pull/165406:44
dimiternwallyworld, and these appear for go1.4 only, which we don't officially support yet ;)06:45
dimiternwallyworld, go vet in 1.4 is dumber than 1.2 - "%q" is reported for a type with String() method06:45
dimiternwallyworld, I'm not sure how much you've seen of my comments06:56
wallyworlddimitern: my stupid connection to free node keeps dropping, so none sorry :-(06:56
dimiternwallyworld, I thought so :)06:57
dimiternwallyworld, basically go vet 1.4 is dumber than go vet 1.2 (which is still the official go version we're using) - "%q" for a type with String() is reported06:57
dimiternwallyworld, and there's this PR which bounced due to the blocker - https://github.com/juju/juju/pull/1654 which fixes the warnings06:58
wallyworldwtf, go vet go dumber06:58
wallyworldwhat are they thinking06:58
wallyworldi'm on go 1.3.x06:58
dimiternthey were *not* thinking :)06:58
wallyworlddimitern: with the blockers - we tried and couldn't reproduce the precise uprade one today06:59
wallyworldwe contend it's a machine thrashing issue on the test vm06:59
dimiternwallyworld, the slowdown or the quoted packages?06:59
wallyworldslowdown'07:00
wallyworldwith the packages one, i changed the behaviour but maas still failed (only maas, not aw etc)07:00
wallyworldso i'm looking to revert the apt behaviour to be as per 1.21 since as of today or yesterday the cloud archive has been updated07:01
dimiternwallyworld, hmm.. maas took longer than usual to upgrade according to the bug reports07:01
wallyworldand we don't need to install cloud-utils and cloud-image-utils together07:01
wallyworldoh, we were working on the assumption of local taking too long, that's what cutis told us07:01
dimiternone thing occurred to me - for precise, are we making sure we add the cloud-tools pocket?07:02
wallyworlddimitern: yeah, that's added07:03
dimiternthere's the MaybeAddCloudToolsWhatever in cloudinit that gets called for a given series - but IIRC it was put in an if block with enableAotUpdate07:03
wallyworlddimitern: and i just confirmed, doing apt-get install cloud-utils by itself breaks07:03
wallyworldbut apt-get install cloud-utils cloud-image-utils works07:03
wallyworldbut setting that up breaks on maas07:03
dimiternwallyworld, due to cloud-init getting removed?07:03
wallyworldyeah07:03
dimiterncan you confirm cloud-image-utils comes from the cloud-tools pocket on precise?07:04
dimiternit has to be fixed there07:04
wallyworlddimitern: actually, it may be that we are only adding the cloud-tools pocket when bootstrapping07:05
wallyworldthat would explain things07:05
dimiternwallyworld, that's wrong if we do07:05
dimiternwallyworld, I'll have a look what triggers it07:05
wallyworlddimitern:  i have a branch which i was going to propose pn gh to revert the apt behaviour to be more like 1.21, but with cloud-utils added (it needs to be installed to get the right version). bootstrap is fine, but adding a new machine fails.  it may be fue to cloud-tools pcoket not being added except for at bootstrap. but i have to go out to a Foo Fighters concert, will be back later. here's the branch. https://github.07:09
wallyworldcom/wallyworld/juju/tree/revert-apt-install-method07:09
dimiternwallyworld, sure, I'll look into it07:09
wallyworldthe above branch reverts the behaviour of multiple packags on the one apt-get line07:09
dimiternwallyworld, enjoy the concert ;)07:09
wallyworldty07:09
wallyworlddimitern: if you run up a machine, apt-cache policy cloud-utils should show 0.2707:10
wallyworldon bootstrap and a worer node07:10
wallyworldworker07:10
wallyworldand cloud-image-utils should be there too07:10
dimiternwallyworld, ok, will check both07:10
wallyworldi'll check back later in a few ours07:10
wallyworldtyvm07:11
dimiternnp07:11
hazmatis gwacl's primary project site still launchpad.net/gwacl?08:58
jamdimitern: did you see william today?09:10
dimiternjam, not yet09:10
jamwallyworld: did you want to discuss ensure-ha --to ?09:20
jamnatefinch: /wave10:32
natefinchjam: howdy10:32
natefinchjam: gimme 2 minutes to go get my coffee?10:32
jamk10:33
jamnatefinch: I don't hear you10:40
perrito666morning11:20
perrito666natefinch_: still up?11:59
natefinch_perrito666: I am here... got up for an early meeting11:59
natefinch_perrito666:  probably will have to go soon as the kids are stirring12:00
perrito666natefinch_: I cannot help but notice you are OCR today and I have a very small change here http://reviews.vapour.ws/r/995/12:01
natefinch_perrito666: ship it!12:03
perrito666tx12:04
perrito666I should have known that the trick was to find you half asleep12:04
perrito6663 blocking bugs? oh what have I done to deserve this12:05
natefinch_doh12:05
* dimitern nailed the cause of the blocker12:10
* dimitern hates cloud-init more than before now12:11
jamfwereade: greetings12:22
fwereadejam, heyhey12:22
jamperrito666: doesn't removing restore break compat with older servers?12:23
jamfwereade: so I commented on your last request. I was wondering if we would still want the server to give the official time.12:27
fwereadejam, so the server would always return what you asked for?12:28
fwereadejam, I'm open to being convinced, but I can't see what we'd use it for today12:29
fwereadejam, and if we need it in the future it's a new api version anyway surely?12:29
jamfwereade: the client could request an amount, the server may upgrade that to a longer amount and replies with the actual amount12:31
jamfwereade: has better future compat if we change the minimum timeout, clients that were asking for something short still work12:31
fwereadejam, surely even if we're just changing that behaviour we'd be adding a version anyway?12:35
jamfwereade: then why worry about it at all? I thought the point of client requests was to allow flexing it12:36
jamI don't think we'd *have* to change the version just because the boundary ranges change, if we make it clear12:36
jamthat said, we don't need to spend hours on this12:36
fwereadejam, because 30s was a magic number embedded deep inside the server that I suspect is more than usually vulnerable to changes without proper foresight12:37
fwereadejam, it's the client that needs 30s, and it will still be the client who needs more or less time in future, I think?12:38
fwereadejam, boundary values perhaps, so long as we're making them looser, I suppose, but that's still a bit risky for my tastes12:38
perrito666jam: mm, you have a good point I think I can make it use both ways according to the server version12:39
fwereadejam, "may I be leader for the next X seconds [yes|no]" STM to be simple and not simplistic -- and, hmm, does not actually preclude creating a longer lease internaly -- it just means that the client couldn't take advantage of that knowledge to space out its requests more12:41
jamfwereade: so I don't think any client is going to make hard time constraints. They might be able to say "it will take approx X" but I doubt anyone can guarantee that they won't take longer than that12:42
fwereadejam, sorry, what won't take longer?12:42
fwereadejam, time to renew the lease?12:43
jamfwereade: whatever thing they are running that they are expecting to hold the lease12:43
fwereadejam, hence putting it in the client's control, and having them request a 60s and renew it in 30s, independent of what else is happening12:43
fwereadejam, based on user feedback what is desired/required is a guarantee that a successful is-leader call give you 30s grace in which you're sure you're the leader12:44
fwereadejam, worst-case you request *just* before a failing renew, and you *still* get 30s of grace from that success12:45
jamfwereade: as in we won't nominate a new leader for 30s after the last one expired ?12:45
fwereadejam, no? we won't nominate a new one until immediately afterthe last one expires12:46
fwereadejam, that's why I ask for 60s and guarantee 3012:46
fwereadejam, and refresh every 3012:46
fwereadejam, even when you fail your nth request, your n-1th one is still good, leaving you time to react12:47
fwereadejam, while you are still leader and nobody else is stepping on your toes12:47
jamfwereade: what is triggering this polling ?12:48
jamif it's juju calling a leader hook, can't we easily be blocked waiting for some other hook to finish?12:48
jamis it *while* running a given hook we expect them to split off a thread/process to call back into us to ensure that they're still active?12:49
fwereadejam, the leadership tracker will be running anyway, independently of anything else -- the hook tool asks the context which asks the tracker if it can be leader12:49
fwereadejam, nobody seems to want that12:50
fwereadejam, best-effort leader-deposed execution once we're outside the reported grace period seems sufficient12:50
jamfwereade: it *feels* to me like someone would write "ok, I need to reconfigure my workload, ask for leader for X seconds, start reconfiguring, oh reconfiguring took too long, but I'm stuck in that process"12:51
jamwho/what would actually refresh the leadership request12:51
fwereadejam, the refreshing is continuous anyway while you're leader whether you're running a hook or not12:52
jamfwereade: k, then I don't see any need to set any time, Juju refreshes at an interval of X12:52
fwereadejam, I'm not so certain that we'll never need to tweak that...12:54
jamfwereade: you just said that if we need to tweak it, it would be a version bump (IMO)12:54
jammy point was, if we want to make it variable, make it easy to be variable and still compatible12:54
fwereadejam, I thought that was what I was doing :)12:54
jamor just make it fixed and we bump the API version when we need to change ti12:54
jamit12:55
jamfwereade: you made it variable from one side, but not the other12:55
jamfwereade: if this is being confusing or feeling like I'm antagonizing you, I'm not trying to, I'm happy with a JFDI here.12:56
jambut it seemed odd to say that only one side would get to change without an API version bump12:56
jamwhen it isn't hard to make it graceful either way12:56
fwereadejam, I know you're not, but I must admit I am a little confused so there's probably something worth figuring out12:57
jamfwereade: I think my gut reaction is "why is this an error, and why can't we just request 0s and get told what the lease time is"12:57
fwereadejam, my contention is that it is hard to make it graceful if we allow the server any more than a yes/no :)12:57
jamasking for too long I could accept, asking for too short seems like "nope, just take this longer one"12:58
jamfwereade: but I was thinking it was being exposed to the charm itself12:58
jamand that charmers were going to have to say "I'm goingto run this op, I think it will take 3 minutes, give me a 3min lease"12:58
jamwhich lead to the other problems12:58
fwereadejam, if I ask to be leader for 60s and the server makes me leader for 300s, I'm still definitely leader for 60s12:58
jam(who can actually refresh, etc)12:58
fwereadejam, as far as I'm aware that was never on the table12:59
fwereadejam, nobody asked for it, and it adds complexity and temptation to ask for infuriatingly long lease times that then lead to poor experiences when those units fail13:00
jamfwereade: that and missed guestimates that then lead to a need to have a separate process that is refreshing. I think the point is Juju is doing all the refreshing (which actually has its own problem of Juju being up and happy, but the charm code stuck in an infinite loop)13:00
jamfwereade: but that's probably still a decent place to be.13:01
wallyworlddimitern: looks like you found a fix for the cloud-utils issue. i didn't think to set apt-get update = true in cloud init for precise. how does that interact with the apt disable update settings? did you use GetPreparePackages() to selectively include target-series for precise?13:23
=== Murali_ is now known as Murali
dimiternwallyworld, it overrides the apt-get update disable flag13:27
dimiternwallyworld, it has to otherwise it won't work13:27
wallyworlddimitern: ah ok, and we can't add the cloud-tools repo without updating i assume13:27
dimiternwallyworld, I'm using your approach with GetPreparePackages from utils/apt, but rather than joining all with " ", I'm calling AddPackage for each one13:27
wallyworlddimitern: yeah, that's exactly what i did in my latest branch i pushed before i left13:28
dimiternwallyworld, yes - it update is off, no apt-sources or packages are installed13:28
dimiternwallyworld, there's a quirk though due to cloud-init13:28
wallyworlddimitern: i discovered you had to do the packages one by one otherwise it would be sad13:28
wallyworldFoo Fifgters were farking awesome btw, bloody excellent concert13:29
wallyworldgad, can't type13:29
wallyworldFoo Fighters13:29
dimiternwallyworld, sweet! :)13:29
wallyworldby ears are ringing :-)13:29
wallyworldmy13:29
dimiternwallyworld, btw - this is the meat of my patch http://paste.ubuntu.com/10389087/ (apart from a similar if (in the beginning) in the templateUserData func in the lxc-broker)13:30
wallyworldlooking13:30
wallyworlddimitern: yeah, that matches my understanding also13:31
wallyworlddimitern: i can review once you propose13:32
dimiternwallyworld, sure, I'll propose soon, but I wanted to test on a precise host just to be sure13:32
wallyworldnp13:32
dooferladdimitern: hangout?14:01
dooferladdimitern: MAAS+Juju Network interlock14:02
dimiterndooferlad, uh, yeah - omw, thanks!14:02
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1424695 1424777
=== kadams54-away is now known as kadams54
mgzgsamfira: are you around? I've got a fix for windows testing stuff I'd like you to look at14:20
dimiternmgz, hey14:50
dimiternmgz, you'd probably guess what I'll ask about :)14:50
mgzdimitern: I hope to have an update, various things are borked14:53
mgzdimitern: who's ocrs today?14:53
dimiternmgz, natefinch_ and anastasiamac according to the calendar14:53
dimiternmgz, ok, thanks14:53
mgzhm, I wonder if I'm in the middle of those two14:54
mgznatefinch_: can I have a review plz? https://github.com/juju/testing/pull/5214:54
natefinch_mgz: looking15:05
=== natefinch_ is now known as natefinch
alexisbdimitern, howdy, sorry I was late for our 1x1 but I am on the hangout now whenever you are ready15:06
dimiternalexisb, oh, omw15:06
natefinchmgz: this is a lot more complex than it seems to need to be.  If we know we don't need a whitelist on anything other than windows, why not just put a if runtime.GOOS != "windows" { return }  at the top, and let the rest of the function be windows only?15:08
mgzlook at the context above, there's also a JUJU_MONGOD variable that's whitelisted everywhere15:09
mgz(and potential for more I guess)15:10
natefinchmgz: oops, missed the append of the testingVariables, sorry15:10
mgzI could make it shorter by just checking that, but it's written in such as way as it wanted to be extensible15:10
natefinchyeah... it would probably be a lot easier if I weren't looking at it in a diff15:17
katcodimitern: ping15:34
dimiternkatco, pong15:35
katcodimitern: it looks like v2 S3 signing is slightly different than standard v2 signing15:35
katcodimitern: amazon states: "Amazon S3 now supports the latest Signature Version 4. This latest signature version is supported in all regions and any new regions after January 30, 2014 will support only Signature Version 4."15:35
ericsnownatefinch: 1-on-1?15:36
katcodimitern: are you ok with shifting s3 to only using v4? porting the special v2 signing stuff into our standard v2 signing method would be a bit of a pain15:36
dimiternkatco, let me assimilate this for a moment :)15:36
katcodimitern: not a problem, please take your time :)15:36
dimiternkatco, so for v2 we can do that later I guess (drop the special signing and leave the post-jan-2014 one only)15:38
dimiternkatco, for v3 we need to make it work as needed, because we'll be switching to v3 pretty soon15:38
katcodimitern: you're talking goose versions now?15:38
dimiternkatco, no - about goamz15:39
katcodimitern: ack sorry... wires crossed. that's what i meant :)15:39
dimiternkatco, ok, sgtm then15:39
dimiternkatco, what's your immediate plan?15:39
katcodimitern: so some clarification15:39
katcodimitern: we want v3 of goamz in juju by this friday for the feature freeze15:39
katcodimitern: this is to support efforts in the china region15:40
katcodimitern: so goamz v3 would drop s3 signing v2 in favor of standard v4, and we would put that into juju by friday15:40
katcodimitern: are we on the same page?15:40
dimiternkatco, sgtm, however I have one request15:40
katcodimitern: sure thing15:41
dimiternkatco, I've already ported what was sensible to port from v1 to v215:41
dimiternkatco, would you be so kind to do the same for v2 to v3?15:41
katcodimitern: with some guidance, sure. are the change-sets fairly self contained?15:41
dimiternkatco, it shouldn't be a lot anyway, and I can help15:41
dimiternkatco, I think so15:42
katcodimitern: yeah sure thing then. let me get the live tests working and then i'll work on that15:42
dimiternkatco, cheers!15:42
katcodimitern: same to you! tyvm o/15:42
katcodimitern: i would like to buy you a beer in nuremberg :)15:43
dimiternkatco, \o/ I'll hold you on to this though :P15:44
katcodimitern: it will be my pleasure! :D15:44
dimitern;)15:44
katcoi have a stein that my mom got me... i'm wondering if i should bring it15:45
jw4dimitern, wallyworld ; fwiw it seems that go vet in 1.4.2 is saner than 1.4.115:45
jw4dimitern, wallyworld but in any case the PR that *did* land allows you to set the environment variable IGNORE_VET_WARNINGS="some non-zero string" which will cause the pre-push hook to report but not fail on go vet warnings15:46
dimiternjw4, that's fine - I'd prefer the flexibility of being able to ignore it, if needed15:47
jw4dimitern: yep15:48
natefinchericsnow: yeah, we can pop into moonstone15:51
ericsnownatefinch: k15:51
stokachuif I wanted to change the hostname during juju kvm creation do i need to modify the cloud-init config for that or is there an easier way15:51
jw4fwereade: still working on upgrade steps, but I wanted to get this wip in front of you sooner rather than later: http://reviews.vapour.ws/r/997/15:52
dimiternsinzui, I've marked bug 1424695 as duplicate of bug 1424777 as it's caused by the same issue15:56
mupBug #1424695: maas cloud-init cannot download agent from state-server <ci> <maas-provider> <network> <regression> <trusty> <juju-core:In Progress by wallyworld> <juju-core 1.22:In Progress by wallyworld> <https://launchpad.net/bugs/1424695>15:56
mupBug #1424777: local-provider precise failed to upgrade <ci> <local-provider> <precise> <regression> <upgrade-juju> <juju-core:In Progress by dimitern> <juju-core 1.22:In Progress by dimitern> <https://launchpad.net/bugs/1424777>15:56
sinzuidimitern, hurray of sorts15:57
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1424695
dimiternsinzui, indeed :)15:57
dimiternit's like this I believe16:03
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs: 1424777
stokachudimitern: is there a way to customize the hostname during a kvm CreateMachine?16:04
dimiternstokachu, I don't know for sure, but I doubt it16:05
dimiternstokachu, we're not setting anything specific in cloud-init for hostname16:05
stokachudimitern: is that file generated dynamically?16:06
stokachuthe cloud-init file16:06
dimiternstokachu, well, yes16:06
dimiternstokachu, before provisioning a machine16:06
stokachuproblem is all KVM creations have a hostname of 'ubuntu'16:06
dimiternstokachu, that comes from the ubuntu-cloud image IIRC16:07
stokachuyea i was hoping there was a way to change that prior to provisioning16:07
dimiternstokachu, that's the first time I've heard someone wanting this btw - it's good you've filed a bug for it16:08
stokachuit only affects local provider, trying to deploy ceph units, it fails b/c all 3 units have same hostname16:09
natefinchmgz: gave you a review, just one minor tweak requested, but otherwise LGTM16:09
stokachudimitern: https://bugs.launchpad.net/juju-core/+bug/132609116:11
mupBug #1326091: deploying into kvm with local provider, hostnames are not unique <kvm> <local-provider> <juju-core:Triaged> <https://launchpad.net/bugs/1326091>16:11
dimiternstokachu, cheers, if it's a blocker for you guys, I'd suggest pinging alexisb16:11
alexisbyes stokachu can you plese send me mail16:13
mgznatefinch: thanks! I'll adjust and land16:14
stokachualexisb: will do thanks16:15
stokachudimitern: thank you too16:15
dimiternstokachu, no worries16:20
dimiternwallyworld, if you're still here - http://reviews.vapour.ws/r/998/16:20
dimitern^ fixes the blocker bug16:20
dimiternnatefinch, axw, others? ^^16:20
cmarshi dimitern, updated http://reviews.vapour.ws/r/974/, does it still look good to land?16:23
natefinchdimitern: looking16:24
dimiterncmars, hey! yes indeed - I've checked it already yesterday16:24
dimiternnatefinch, thanks16:24
cmarsdimitern, awesome, thanks for the review!16:24
dimiterncmars, np16:24
perrito666brb16:25
=== kadams54 is now known as kadams54-away
=== kadams54-away is now known as kadams54
natefinchdimitern: is it possible to have --target-release series package1 package2 package3?   And if so, will that still do the right thing with this code (i.e., they'll be joined by a space as the package name)?16:38
dimiternnatefinch, no it's not16:38
dimiternnatefinch, as far as I could understand from apt-get docs16:39
dimiternnatefinch, and even if it was, it still won't work due to the way cloud-init 0.6.3 passes them to apt-get16:40
dimiternnatefinch, e.g. "--target-release rel pkg1 pkg2 pkg3"16:40
dimiternnatefinch, well, actually - sorry, it *is* possible16:41
dimiternnatefinch, apt-get accepts that, but not cloud-init16:41
natefinchdimitern: gah, what a pain in the ass.16:42
dimiternnatefinch, in reality, --target-release is just a hint to the apt policy engine to decide which candidates to prefer16:42
dimiternnatefinch, oh tell me about it :) I've been on it since 8 am16:43
natefinchdimitern:  I wonder if we should be checking to make sure that they don't do --target-release series pkg1 pkg2  ... but maybe that's being too smart?16:47
dimiternnatefinch, I have a couple of panics in place if for that reason16:48
dimiterns/if//16:48
dimiternnatefinch, all tests pass - both make check and the live ones I've described16:49
dimiternnatefinch, I suppose you could argue if 99% of the existing tests pass, it's a bad thing and we need better ones16:49
dimiternnatefinch, but I'd rather land this and unblock everyone in a few hours, then I'm happy to improve the tests as a follow-up16:50
dimiternnatefinch, I'm porting the same fix to trunk and fully intend to retry the same live tests before proposing it16:51
=== kadams54 is now known as kadams54-away
* dimitern is sick of blockers already - let's not add to that :)16:51
dimiternkatco, still around?16:52
dimiternkatco, is this ready to land https://github.com/go-amz/amz/pull/27/ ? it looks so to me - if it is, i'll go ahead and merge it16:53
katcodimitern: no not quite yet16:55
dimiternkatco, ok, np16:56
katcodimitern: almost there16:56
* dimitern steps out for a while - will be back soon16:56
natefinchdimitern: gave you a ship it16:58
=== kadams54-away is now known as kadams54
alexisbnatefinch, can you edit the HA doc now?18:11
dimiternnatefinch, thanks!18:15
mgzman, I got review 999?18:32
mgzone off, one off18:32
jw4mgz: quick think of something else to PR18:32
mgzreview please: reviews.vapour.ws/r/999/18:32
mgznatefinch: ^18:39
cmarsis landing still blocked?19:02
cmarsok, nvm19:03
natefinchalexisb: still view only19:08
natefinchmgz: ship it19:08
alexisbnatefinch, ack, I dont have permission to give you write access we will have to bug wallyworld19:09
natefinchalexisb: yeah, figured.  No problem. I meant to ask for write access from him last night and forgot.19:11
dimiterncmars, it is, I'm still testing the port of the fix for trunk, but 1.22 should get unblocked at least (not that it matters I guess)19:19
cmarsdimitern, got it, thanks for fixing!19:20
dimiterncmars, np19:20
dimiternnatefinch, FYI, there's the tech-debt bug 1425245 I filed for your suggestion19:21
mupBug #1425245: improve cloud-init tests after the fix for bug #1424777 unblocks CI <tech-debt> <juju-core:Triaged by dimitern> <juju-core 1.22:Triaged by dimitern> <https://launchpad.net/bugs/1425245>19:21
natefinchdimitern: thanks :)19:21
dimitern:)19:23
hazmataxw: llgoi ever get announced?19:50
* hazmat switches to pm19:54
natefinchthumper: do you have a link to the doc about CLI 2.0?20:01
natefinch(asking for a friend)20:01
thumpernatefinch: it isn't written yet :)20:01
natefinchthumper: heh20:01
natefinchthumper: thought it was already being worked on20:01
natefinch(ish)20:02
=== kadams54 is now known as kadams54-away
dimiternnatefinch, PTAL http://reviews.vapour.ws/r/1001/ - fix for the blocker for trunk20:27
dimiternthumper, ^^20:45
thumperdimitern: ack20:46
dimiternthumper, I'm waay pas EOD, so I'm going - please add $$fixes-1424777$$ if it's ok20:46
thumperdimitern: do you have the link for the 1.22 version?20:46
thumperdimitern: and thanks for the fix20:47
dimiternthumper, https://github.com/juju/juju/pull/1670/20:47
thumperta20:47
dimiternthumper, np20:47
perrito666every time I unplug my external monitor my computer hangs... how sad21:39
thumperperrito666: tell Trevinho in #ubuntu-desktop21:52
thumperperrito666: tell him I sent you :-)21:52
thumperperrito666: although I'm not sure he'd be on right now as he lives in Italy21:52
thumperperrito666: but is is known to work weird hours21:52
perrito666thumper: why that makes me think that I will get something thrown over my head21:52
thumper:-)21:52
thumperhe's a good guy, and one of the current maintainers of unity 721:53
perrito666thumper: anyway I am using vivid,so that is most likely my fault21:53
thumperperrito666: assuming you're using unity21:53
perrito666thumper: it is unity, version is a mistery, whatever is shipped with vivid21:53
cmarswhy am I still getting "Build failed: Does not match ['fixes-1424777']"22:25
perrito666cmars: can I be a smartass?22:30
perrito666:p22:30
perrito666cmars: jokes aside, that bug must be marked as critical and not fix commited?22:31
cmarsperrito666, it's fix committed though22:32
cmarshttps://bugs.launchpad.net/juju-core/+bug/142477722:32
mupBug #1424777: local-provider precise failed to upgrade <ci> <local-provider> <precise> <regression> <upgrade-juju> <juju-core:Fix Committed by dimitern> <juju-core 1.22:Fix Committed by dimitern> <https://launchpad.net/bugs/1424777>22:32
ericsnowcmars: it has to be "fixed released" before it's unblocked22:33
jw4marcoceppi made this recently: http://juju.fail/status.json22:33
thumpercmars: we need to make sure the ci test that it is supposed to fix is actually fixed22:33
thumpersinzui: ping22:33
thumpersinzui: can we see if dimiter's patch has fixed the precise upgrade?22:33
cmarsjw4, marcoceppi that's pretty neat22:35
jw4cmars: I believe it actually uses the same mechanism the CI bot uses22:36
marcoceppicmars: yeah, I was just about to put a small webpage in front of it, but status.json will always be available for consumption22:36
marcoceppijw4: it does, about to push the code up which generates this page22:36
jw4marcoceppi: cool22:36
sinzuithumper, it does, but aws is ill, preventing an official blessing, but we are switching everything we can off of aws22:40
thumpersinzui: cool22:41
katcowallyworld: you around yet?23:32
wallyworldyup23:32
katcowallyworld: got time for a quick hangout? have a series of s3 questions23:32
wallyworldsure23:32
katcowallyworld: sweet... 1:1?23:32
wallyworldyup23:32
perrito666ahaa I believe it chrome that goes crazy when x is resized23:34

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!