mrammfwereade: I'll look into the details, but am not able to get clarity on that right at this moment because people are not around00:04
fwereademramm, np, I'm heading off to sleep soon00:04
mrammfwereade: understood00:04
mrammtake care of yourself -- it's late your time!00:04
fwereademramm, cheers :)00:05
thumperah ffs00:17
davecheneym_3: ping00:18
thumperdavecheney: I have bootstrap problems with raring with the 1.9.14 package, and tip of trunk00:19
thumperdavecheney: I'm in dive mode, but breaking for lunch now00:19
thumperas in diving in with logging to try and figure out wtf is going on00:20
=== thumper is now known as thumper-afk
davecheneythumper-afk: ok, i'm doing the same00:21
davecheneyit could be that we have too many tools in the public bucket00:21
davecheneyin fact, that is probably it00:21
davecheneyrelease mode uses best fit, so it iterates over the tools00:22
davecheneydev mode uses exact fit, so it's a striaght hit and run00:22
fwereadethumper-afk, davecheney: sleep00:26
fwereade(me sleep, not you)00:26
m_3davecheney: pong01:33
davecheneym_3: wazzup ?01:39
m_3back home01:40
davecheneyi'm going to make one more attempt to figure out what is going on with hp cloud01:40
davecheneyi'm going to allocate 100 machines01:40
davecheneymanually remove their public addresses01:40
davecheneyanother 10001:40
davecheneysee if that gets us over the 2^8 hump01:41
m_3yeah, did you ever try it from _outside_?01:41
m_3that's what I was gonna do next01:41
davecheneym_3: i was able to launch 100 extra isnances from inside via the nova command01:41
m_3basically spin up what we have from outside of hp01:41
m_3but I've seen this problem before, a bunch01:41
m_3intermittent inability to resolve the enpoint urls01:42
davecheneymaybe it will work from outside01:42
davecheneyit's like when we spin up two many machines inside an openstack tennant01:42
m_3this kills charmtests against hp sometimes (w/ juju-0.6)01:42
davecheneyit runs out of ip's to respond _to_ dns queries01:42
m_3I really don't understand it01:42
davecheneyi can sort of explain it01:42
m_3couldn't work with it yesterday cause of the talk01:42
davecheneygiven the ways that each tennant probably has the _same_ 10/8 address space01:43
m_3dude, there were >300 people in our talk yesterday!01:43
davecheneyso there is probably shittones of nat going on01:43
m_3they mostly stayed01:43
davecheneym_3: i saw the photo01:43
davecheneythat is fucking amaizing !?!01:43
m_3but /8's frickin huge01:43
davecheneysure, but it's the same 10/8 for each customer01:44
m_3you're thinking that's natted out using a limited pool of outsides?01:44
davecheneyjust like your router is using 192.168.0/1601:44
davecheneymy suspicion is because we're asking for so many public addresses, we're sort of choking off our own air surply01:44
davecheneyanyway, going to explore that theory today01:45
m_3hmmm... it really looks like the same as wehn I hit it from ec201:45
davecheneym_3: is this at all related to the stuff mgz was saying about security groups and screwing ourselves by making too many stupid requests ?01:46
m_3dude... dunno01:46
m_3I plan to sort of wrap my head around all of this again tomorrow01:47
davecheneyyou've got to uncompress from ODS01:47
* davecheney waves his wand01:47
m_3I'll start from scratch and grok what you and mgz's done this week01:47
davecheney"thou shall never have to say cloud again"01:47
davecheney"i've got a cloud in my pants, and everyone is invited"01:47
* m_3 groan01:48
davecheneytoo soon ?01:49
davecheneym_3: fyi, just bringing the code on juju-hpgoctrl2-machine-001:49
davecheneyup to date with the overnight changes01:49
m_3davecheney: awesome... thanks man01:50
davecheneym_3: no worries, we fixed some good issues this week01:50
davecheneythe deploy logs from the load test are much cleaner01:50
davecheneythey actually tell you what is going on01:50
davecheneystatus is now usable while you are doing a big deploy01:50
davecheneyalso, when hp cloud wants too, it is nearly twice as fast to bring up an instance than ec201:51
davecheneywhich doesnt' suck01:51
davecheney2013/04/19 01:59:49 ERROR worker/provisioner: cannot start instance for machine "16": cannot set up groups: failed to create a rule for the sec02:00
davecheneyurity group with id: <nil>02:00
davecheneycaused by: Maximum number of attempts (3) reached sending request to https://az-2.region-a.geo-1.compute.hpcloudsvc.com/v1.1/17031369947864/os-02:00
thumper-afkdavecheney: &http.Response{Status:"200 OK", StatusCode:200, Proto:"HTTP/1.1", ProtoMajor:1, ProtoMinor:1, Header:http.Header{"X-Amz-Request-Id":[]string{"90D56A3D6895C07E"}, "X-Amz-Id-2":[]string{"6lThSRAi5lMeq9oe8oSeibO7fjvZQLjgKGYG0Gs7vRMBZrQ6Z0xVlIfyILAoWO4A"}, "Date":[]string{"Fri, 19 Apr 2013 02:38:03 GMT"}, "Content-Type":[]string{"application/xml"}, "Server":[]string{"AmazonS3"}}, Body:(*http.bodyEOFSignal)(0xf84022ec60),02:39
thumper-afkContentLength:-1, TransferEncoding:[]string{"chunked"}, Close:false, Trailer:http.Header(nil), Request:(*http.Request)(0xf8403ac600)}02:39
thumper-afkdavecheney: this is the response from the http request inside goamz/s3 for the request to list the public bucket02:39
thumper-afkseems like body is: Body:(*http.bodyEOFSignal)(0xf84022ec60)02:39
=== thumper-afk is now known as thumper
thumperhate it when I forget to reset the nick02:41
davecheneythat it because it is chunked02:42
davecheneylength is unknown from the server02:42
thumperok, what does that mean?02:42
davecheneyrfc 2616 chunked transfer encoing02:42
davecheneyit's not a problen02:43
davecheneyits a way of sending the http body without having to specify the length first02:43
davecheneyvery common if you are streaming a response02:43
thumperdavecheney: this is the line that is failing: err = xml.NewDecoder(hresp.Body).Decode(resp)02:47
thumperdavecheney: I wonder if we have moved into a chunked response now, which the decoder can't handle due to number of tools...02:54
* thumper writes up findings to the list02:58
davecheneythumper: nah, it just gets a Reader03:06
davecheneywhat implements the reader is not important03:06
thumperdavecheney: so what would it be then?03:15
* thumper has put the heating up03:16
thumperdavecheney: ping04:29
rogpeppemornin' all06:02
davecheneym_3: https://bugs.launchpad.net/juju-core/+bug/117059507:06
davecheneythis is why we're having problems in load test07:07
davecheney2013/04/19 07:07:20 INFO rpc: discarding obtainer method reflect.Method{Name:"Kill", PkgPath:"", Type:(*reflect.commonType)(0x7468a8), Func:reflect.Value{typ:(*reflect.07:07
davecheneycommonType)(0x7468a8), val:(unsafe.Pointer)(0x4d6359), flag:0x130}, Index:4}07:07
davecheney2013/04/19 07:07:20 INFO rpc: discarding obtainer method reflect.Method{Name:"requireAgent", PkgPath:"launchpad.net/juju-core/state/apiserver", Type:(*reflect.commonTyp07:07
davecheneye)(0x767768), Func:reflect.Value{typ:(*reflect.commonType)(0x767768), val:(unsafe.Pointer)(0x4d63e7), flag:0x131}, Index:8}07:08
davecheney2013/04/19 07:07:20 INFO rpc: discarding obtainer method reflect.Method{Name:"requireClient", PkgPath:"launchpad.net/juju-core/state/apiserver", Type:(*reflect.commonTy07:08
davecheneype)(0x767768), Func:reflect.Value{typ:(*reflect.commonType)(0x767768), val:(unsafe.Pointer)(0x4d64ac), flag:0x131}, Index:9}07:08
davecheney^ rogpeppe1 is this a problem ?07:08
rogpeppe1davecheney: no07:08
davecheneywas spotted on pa restart07:08
rogpeppe1davecheney: that's expected behaviour07:09
rogpeppe1davecheney: the warnings are useful when developing07:09
rogpeppe1davecheney: i know they're annoying otherwise07:09
davecheneyok, nm07:09
rogpeppe1davecheney: i guess we should probably move those methods off the rpc root object to stifle the warnings.07:10
davecheneyrogpeppe1: if they aren't bugs then I wouldn't worry about it for the moment07:11
rogpeppe1davecheney: do you know if anything's happened about 1.10 yet?07:13
rogpeppe1davecheney: 'cos i have a couple of minor bugs (i already have the fixes for them) that it would be great to sort out if there was a moment or two more.07:14
rogpeppe1davecheney: it seems nobody has ever used juju get.07:14
davecheneyrogpeppe1: yeah, i saw that bug07:14
davecheneyi think you are right07:14
davecheneynoone ever did use it07:14
rogpeppe1davecheney: i had a fun time yesterday starting up a juju env, making some weirdish relations, upgrading charms, resolving hooks, etc07:15
rogpeppe1davecheney: it actually seemed to work pretty well07:15
davecheneyrogpeppe1: i don't doubt that, we have excellent charm compatibility07:16
rogpeppe1davecheney: i've just had an idea for a way to make it easy to write little charms that exercise particular functionality; trying to knock something together today07:18
davecheneyjujud   8613 root    1w   REG  253,1    71378 131869 /var/log/juju/machine-0.log07:18
davecheneyjujud   8613 root    2w   REG  253,1    71378 131869 /var/log/juju/machine-0.log07:18
davecheneyjujud   8613 root    3r   CHR    1,9      0t0   5786 /dev/urandom07:18
davecheneyjujud   8613 root    4w   REG  253,1    71378 131869 /var/log/juju/machine-0.log07:18
davecheneydo I even want to ask why we have 3 fd's pointing to the same log file ...07:18
davecheneyrogpeppe1: sweet07:19
rogpeppe1davecheney: stdout and stderr are expected07:19
rogpeppe1davecheney: not sure about 407:19
davecheneythat isn't as important as# lsof -p $(pgrep jujud) | grep -c ESTABLISHED07:19
davecheneythat is why we can't provision more than about 200 machines in a run07:20
rogpeppe1have you found the source of the leak?07:20
davecheneylooking now07:20
davecheneyshouldn't take long07:20
davecheneygiven the number of times this problem turns up07:20
davecheneyi'm smacking myself it wasn't the first thing I looked for07:20
rogpeppe1davecheney: this was the status from one of yesterday's environments http://paste.ubuntu.com/5719234/07:22
rogpeppe1davecheney: note the interesting relationship between mongo and logging there07:23
* davecheney isn't quite sure what is wrong there07:24
davecheneyare they circular ?07:24
rogpeppe1davecheney: nope07:24
rogpeppe1davecheney: there's nothing wrong07:24
rogpeppe1davecheney: it's just quite cool that you can do it07:24
rogpeppe1davecheney: basically, logging requires mongo to store its logs. but we also want to store the log files produced by mongo itself, so the logger is subordinate to mongo as well as being related to it.07:25
rogpeppe1davecheney: i set it up deliberately like that, thinking it might not work07:26
rogpeppe1davecheney: but it seems to work fine (at least on the surface! i haven't *actually* looked at the logs in mongo)07:26
rogpeppe1anyone know if there's an easy way for a charm to find out its service name?08:28
davecheneyyou'd think that would be straight forward08:29
rogpeppe1currently the only thing i can think of is `pwd | sed blahblah`08:29
rogpeppe1which is a hack08:29
davecheneyit isn't a config property ?08:29
rogpeppe1davecheney: no08:30
* davecheney gives up08:30
rogpeppe1davecheney: i'm not sure it should be a config property08:30
davecheneymaybe i used the wrong word08:30
davecheneysetting might be appropriate08:30
rogpeppe1davecheney: it could easily be an env var though08:30
rogpeppe1davecheney: settings can change08:31
rogpeppe1davecheney: this is immutable08:31
davecheneyagain i'm using the wrong word08:31
davecheneysurely we have a class of setting which are immutable08:31
rogpeppe1davecheney: ah, ok08:31
rogpeppe1davecheney: i don't *think* so08:31
rogpeppe1davecheney: there might be a special case for public-address i suppose08:32
rogpeppe1davecheney: ah, but that's relation setting anyway08:32
rogpeppe1davecheney: currently service settings map exactly to the config defined in the charm08:32
rogpeppe1davecheney: which seems good to me08:32
rogpeppe1davecheney: i think just an env var JUJU_SERVICE to go along with JUJU_ENV_UUID would be good08:33
davecheneyi agree08:33
davecheneysounds like something very useful08:33
rogpeppe1davecheney: and i'd add JUJU_SERVERS too08:33
rogpeppe1davecheney: yeah, it's very useful because it's an easy and predictable disambiguation mechanism08:34
rogpeppe1davecheney: so i can create a directory that has a predictable name but is guaranteed not to clash with similar names chosen by other colocated charms08:35
rogpeppe1davecheney: JUJU_SERVICE is a one-line change08:36
rogpeppe1davecheney, fwereade, dimitern: do you know if trunk is still frozen?08:37
fwereaderogpeppe1, davecheney, dimitern: I have heard nothing from mramm re the deadline ambiguity alluded to my mgz08:38
=== rogpeppe1 is now known as rogpeppe
fwereaderogpeppe1, davecheney, dimitern: whoops, I did actually, had missed that mail08:39
rogpeppefwereade: to you only? i don't think i saw anything08:39
fwereaderogpeppe, davecheney, dimitern: I think we should revert the 1.10 version for now08:39
rogpeppefwereade: ok - what's the situation?08:39
fwereaderogpeppe, apparently the *real* deadline is EOD monday08:39
rogpeppefwereade: oh, that's great! i'll propose a couple of bug fixes then, if that's ok.08:40
fwereaderogpeppe, so I think we should revert the version and keep going on low-risk/high-impact bugfixes for today at least08:40
rogpeppefwereade: 1130149 and 1170425 are both easy and worth doing08:41
fwereaderogpeppe, although, tbh, today at *most* also applies ;p08:41
rogpeppefwereade: agreed entirely08:41
rogpeppefwereade: BTW what do you think about a $JUJU_SERVICE env var?08:42
rogpeppefwereade: so a charm can know what service it's running as08:42
fwereaderogpeppe, use case?08:42
rogpeppefwereade: to go along with JUJU_ENV_UUID08:42
rogpeppefwereade: it gives an easy way for a charm to create a predictable directory that won't clash08:43
rogpeppefwereade: also it provides a reliable way for a knowledgable charm to find the unit config (although tbh i think we should provide JUJU_SERVERS or something like that instead of needing to do that)08:43
fwereaderogpeppe, I think service name is too coarse, and you really want unit name08:44
fwereaderogpeppe, sorry, what's the unit config?08:44
rogpeppefwereade: the uniter agent config08:44
rogpeppefwereade: yeah, unit name would be good08:44
rogpeppefwereade: currently you *can* find it out, but only by mangling pwd, which is dreary and nasty.08:45
rogpeppefwereade: mind you i'm not sure it's currently possible to have two units of the same service in the same container, is it?08:46
fwereaderogpeppe, yeah, but hitting the agent conf at all is dreary and nasty -- we should be explicitly making the API server addresses available if hooks need them08:46
fwereaderogpeppe, nothing stopping you doing that08:47
rogpeppefwereade: yeah, i think we should; but i think the unit name is useful info too.08:47
fwereaderogpeppe, JUJU_UNIT_NAME is already there, isn't it?08:47
rogpeppefwereade: for my particular use case, i'm wanted to write a charm that made it easy to test pwd08:48
rogpeppefwereade: ah, i missed that08:48
fwereaderogpeppe, sorry, I'm being slow, test what about pwd?08:48
rogpeppefwereade: for my particular use case, i'm wanting to write a charm that made it easy to test aspects of charm behaviour08:49
rogpeppefwereade: $JUJU_UNIT_NAME is great08:49
fwereaderogpeppe, sweet08:49
rogpeppefwereade: although perhaps $JUJU_SERVICE might be useful too, i dunno08:49
fwereaderogpeppe, I *am* wondering about the juju gui charm though08:50
rogpeppefwereade: yeah08:50
rogpeppefwereade: i really think we should provide server address info08:50
fwereaderogpeppe, I'm not sure the juju gui should be bound to the juju that deployed it08:50
rogpeppefwereade: ah, that's an interesting point08:50
fwereaderogpeppe, I suspect that API information should just be service config08:50
fwereaderogpeppe, even if it's a little less convenient to set it up08:51
rogpeppefwereade: the problem with that is that in a HA world that info changes08:51
rogpeppefwereade: i could see that it might be good to allow both ways actualy08:52
rogpeppefwereade: use the local server unless a config option is set08:52
fwereaderogpeppe, maybe, I need to think about this for a bit08:53
rogpeppefwereade: then we can potentially have something that watches some environment and makes config changes when the set of server addresses changes08:53
fwereaderogpeppe, it kinda feels like the same old service-output problem08:53
rogpeppefwereade: anyway, i don't think there's a good reason to make it hard for a charm to access its own API server08:54
rogpeppefwereade: ?08:54
rogpeppefwereade: which problem was that?08:54
fwereaderogpeppe, that we'd kinda like to be able to get information back out of services08:55
rogpeppefwereade: ah yes. we really really do08:55
fwereaderogpeppe, it should ideally always be possible to deploy a service with default configuration and have it work nicely08:55
rogpeppefwereade: i think that's one of the most crucial missing juju features. that and allowing a charm to change things asynchrously.08:56
rogpeppefwereade: i agree.08:56
fwereaderogpeppe, in the case of a password a default password is painfully insecure, and generating one on the fly should be perfectly possible, but there's no way to get it out08:56
fwereaderogpeppe, for the async stuff you mean juju-run basically?08:56
rogpeppefwereade: yeah08:56
fwereaderogpeppe, agreed on both points08:57
fwereaderogpeppe, anyway, those bugs08:57
fwereaderogpeppe, 1130149, +10008:57
rogpeppefwereade: because currently there's no way for a unit to *say* anything other than in response to something else08:57
fwereaderogpeppe, 1170425, I'll take quite a lot of convincing08:57
fwereaderogpeppe, yep, definitely08:58
rogpeppefwereade: are you suggesting that juju get shouldn't work on a subordinate service?08:58
fwereaderogpeppe, I'm suggesting that calling Constraints on a subordinate service is DIW08:58
rogpeppefwereade: did i suggest otherwise?08:58
fwereaderogpeppe, last night, I think you did ;p08:59
rogpeppefwereade: ah, i didn't know you'd seen that :-)08:59
rogpeppefwereade: i knew you'd be -1 on that suggestion08:59
fwereaderogpeppe, so long as it's done by skipping the Constraints call I'm fine, I guess, but I'm a bit surprised that the gui always wants to get constraints alongside config09:00
rogpeppefwereade: i can't really see a down side, but there y'go.09:00
rogpeppefwereade: it gets all the service info in one call09:00
rogpeppefwereade: the fix i made just tested IsPrincipal09:01
fwereaderogpeppe, ok, that's fair enough in the current context09:01
davecheneywow, hp cloud is so much faster than ec209:02
rogpeppedavecheney: it wouldn't take much :-)09:03
davecheneybootstraps take < 2 mins on hp cloud09:03
rogpeppefwereade: your password use-case is an interesting one.09:03
rogpeppefwereade: and highlights one particular issue with getting stuff out of a service - can the service somehow choose a "shared" value that all units agree on, or can you just see a set of values for each unit? i think probably just the latter actually.09:05
fwereaderogpeppe, that's just a matter of exposing stuff we already have, so it would certainly be simpler09:05
rogpeppefwereade: in a way you could think of the service config the relation settings of the juju client09:06
rogpeppes/the relation/as the relation/09:06
rogpeppefwereade: so a similar model could apply - a charm could run config-set to set its own config settings that could be seen by the client.09:07
fwereaderogpeppe, that is a *very* nice way of looking at it09:07
fwereaderogpeppe, but it does ring up interesting race possibilities, I think09:08
rogpeppefwereade: really? each unit would have its own set of config settings09:08
fwereaderogpeppe, if I deploy 3 units of something, which one gets to pick the output password for the service administrator?09:08
rogpeppefwereade: they all pick their own passwords09:08
fwereaderogpeppe, I don't think it necessarily makes sense at a unit level but go on09:09
rogpeppefwereade: as a client i have to choose which unit to get the password from09:09
rogpeppefwereade: that's why the relation analogy is nice - with relations, there's one group of settings for each unit, and each unit can set its *own* settings, but can only read the remote settings.09:10
rogpeppefwereade: if you have a service where all units must agree on a password, they can work it out together and present a unified front09:10
rogpeppefwereade: doing leader election perhaps through a peer relation09:11
rogpeppefwereade: shared read-write settings are a no-no i think09:11
davecheney^ fixes openstack connection leak09:15
rogpeppedavecheney: yay!09:15
davecheneydoing a 300 node test now09:16
davecheneyit's not leaking09:16
davecheneyso sayeth lsof09:16
davecheneybut i'll leave it running and get some dinner09:17
rogpeppedavecheney: i'm not sure the fix is quite right09:17
rogpeppedavecheney: it can still potentially leak, i think09:17
davecheneyrogpeppe: oh realy ?09:17
rogpeppedavecheney: if retryAfter == 0 we leak09:17
davecheneyoh fuck, i didn't see all those stupid returns09:17
davecheneyright, will fix some more09:17
rogpeppedavecheney: i'd be tempted to put it into its own function09:18
davecheneyrogpeppe: ohhh09:18
davecheneyi have many many refactors to this package09:18
rogpeppedavecheney: with a deferred "if err != nil {resp.Close()}09:18
davecheneyrogpeppe: the body closing is all over the shop in that package09:19
davecheneyi have a branch for fixing that as well09:19
* rogpeppe is not greatly surprised09:19
davecheneyPTAL https://codereview.appspot.com/866804809:20
rogpeppedavecheney: i think that's wrong too, probably09:21
davecheneywell fuck09:21
davecheneywhere do you think it goes ?09:21
rogpeppedavecheney: does nothing read the resp body returned from sendRateLimitedRequest ?09:22
rogpeppeoops sorry!09:22
rogpeppeyou're good, i think09:22
davecheneyit is hard to understand when it is read and not read09:23
davecheneyand there are other potential places where the connection can leak09:23
davecheneycheck out client.BinaryRequest09:23
davecheneyi've patched all those in my other branch, but they didn't appear to be the problem09:23
rogpeppedavecheney: LGTM09:23
davecheneyno rush on the review, I have something similar bodged into the load testing machine09:25
davecheneyand it's doing the job09:25
fwereadedavecheney, LGTM also09:33
davecheneyfwereade: what is the story with patches to trunk ?09:34
davecheneyyes ? no ? please ? maybe ?09:34
fwereadedavecheney, I'm going to revert the version right now09:34
fwereadedavecheney, low-risk/high-value changes to trunk are fine for today I think09:34
davecheneyAHHH SHIT09:35
davecheneythis is too goose09:35
davecheneyand jon's bot is fucked09:35
fwereadedavecheney, hell-damn -- I think the juju-core revert still stands09:38
fwereadedavecheney, rogpeppe: https://codereview.appspot.com/885504409:38
fwereadedavecheney, but jam's not around today is he?09:38
fwereadedimitern, do you know how we can land goose fixes ATM?09:38
* davecheney grumbles about things 09:38
rogpeppefwereade: LGTM trivial09:38
jamfwereade: I'm definitely not here and responding to davecheney's request09:39
jamdefinitely not right now.09:39
davecheneyjam: good to know09:39
davecheneyor not09:39
davecheneyi think09:39
fwereadejam, well, that is very lazy and irresponsible of you ;P09:39
davecheneyfwereade: LGTM, just commit it09:39
fwereadejam, sorry, I though this was your day off09:39
fwereadedavecheney, don;t worry already happening ;09:39
jamfwereade: it is09:40
jamwhich is why I'm definitely *not* doing it exactly right now.09:40
jamand it should be done in as long as it takes to confirm it doesn't break juju-core's test suite09:41
davecheneyjam: thanks for fixing gz's one as well09:41
davecheneyto opine for a second09:41
davecheneythe http package it trail by fire for everyone09:41
davecheneysurely there must be a better way to write a http client that doesn't mame anyone who touches it09:41
jamdavecheney: so why doesn't gc close the resp.Body stuff? Or it does, but may take a while. Or it doesn't because underlying it all is a shared http connection that keeps a reference?09:42
davecheneyjam: there is no finaliser on the response body09:42
davecheneythis is part of the connection reuse logic09:42
davecheneya very questionable decision09:42
davecheneyeventually if every refreence to the response, and hence the net.Conn was freed09:43
davecheneythe finaliser on the fd would close it09:43
davecheneybut because of the way the connection reuse logic works, a response (and hence the body) is 'checked out' until you close it09:43
rvbafwereade: can I land the maas provider constraints stuff today?09:43
fwereadervba, ...honestly I can't think why not, if it works, let me go review that right away09:44
fwereadervba, I don;t think it's likely to be destabilizing09:44
rvbafwereade: it should be pretty safe09:45
fwereadervba, but I seem to be being dense, because I don't see a review09:45
fwereadervba, MP09:46
rvbafwereade: https://codereview.appspot.com/8842045/09:47
fwereaderogpeppe, btw, are you planning to look at both those bugs you linked before?09:47
rvbafwereade: it has been reviewed by dimitern already.09:48
rogpeppefwereade: yeah, i'm doing them09:48
fwereaderogpeppe, <309:48
fwereadervba, we try to have 2 reviews (except for the truly trivial), may I take a quick look before I approve?09:49
rvbafwereade: sure, please do.09:49
fwereadervba, that's approved09:52
fwereadervba, tyvm09:52
rvbafwereade: ta09:52
fwereadervba, I am dense, I found it in LP and reviewed there09:53
fwereadervba, close enough09:53
rogpeppefwereade: BTW the old "// Breaks compatibility with py/juju" comment in statecmd/get.go - do you know anything more about that? i'm presuming that py juju printed the actual value and the compatibility breakage is just because we're returning null10:10
davecheneyrogpeppe: i think I wrote that10:13
rogpeppedavecheney: do you remember what the issue was?10:14
fwereaderogpeppe, I'm afraid I'm almost 100% ignorant of get, but, well, we should avoid compatibility breaks where possible10:16
davecheneyrogpeppe: this was probbly lisbon II10:16
davecheneyand gustavo said do it this way10:16
rogpeppefwereade: agreed totally.10:16
davecheneyi think it was something he felt was an improvement over python10:16
rogpeppedavecheney: hmm, interestin10:16
rogpeppedavecheney: surely the fact that it never prints default values isn't right though...10:17
davecheneyrogpeppe: it was certainly this issue surounding the difficulty in diferentiating between the default value10:20
davecheneyand a value which was set, but set to the default10:20
davecheney^ i've broken HP Cloud, where is my medal10:20
rogpeppedavecheney: yeah. maybe py juju didn't have the "default" bool10:20
davecheneyfrom emmory10:21
rvbafwereade: the change to the MAAS provider is merged now.  Make sure you have the last version of the gomaasapi lib otherwise some tests in environs/maas will fail.10:21
davecheneyit was the issue of telling the default value, ie, nothing set, from the value which was set, but was set to the default10:21
fwereadervba, cool, thanks10:21
fwereadedavecheney, rogpeppe: isn't it mportant that we differentiate between those cases?10:22
rogpeppefwereade: yeah, i'm thinking that10:22
rogpeppefwereade: i think our DeepEquals there is wrong10:22
davecheneyfwereade: i think so as well10:22
davecheneyit is a tricky problem10:22
davecheneyand gets orders of magnitude more complicated when you consiuder upgrade charm10:23
davecheneymay supply a default value where one was previous set10:23
fwereadedavecheney, the upgrade-charm logic is that values left default change to new defaults; values set and coincidentally matching the old defaults should not change10:24
rogpeppefwereade: that seems right to me10:24
rogpeppefwereade: which would indicate that if the service config has an entry, then default == false10:24
fwereaderogpeppe, agreed10:24
rogpeppefwereade: so no need for an equality comparison10:25
fwereaderogpeppe, and in that case we just poke in the value from the charm default, and I think we're done there10:25
davecheneyfwereade: doesn't that mean if I set a config value, then upgrade my charm, then unset that config value, I may find that the default value then makes it look like nothing happened ?10:25
fwereaderogpeppe, +110:25
rogpeppedavecheney: wouldn't that be the correct behaviour10:25
davecheneyrogpeppe: i'm trying not to make a judgement here, at the time this problem sounded NP hard10:25
rogpeppedavecheney: because in fact the value of that setting from the charm's pov has not changed10:25
davecheneyrogpeppe: i'm talking about the human using the tool10:26
davecheneydefaults appear to work for the charm, not the user10:26
rogpeppedavecheney: ah, the user *would* see that something changed10:26
rogpeppedavecheney: they'd see the "default" attribute switch to true10:26
rogpeppedavecheney: although the "value" entry would stay unchanged10:26
davecheneyrogpeppe: it's is clear I'm tlaking out my rectum10:27
davecheneyi don't have anything more useful to add at this point :)10:27
rogpeppedavecheney: well you *are* down under10:27
davecheneyrogpeppe: fuckit, everthing is upside down here10:27
davecheneymy favorite part of doing juju destroy-environment is the way all the ssh connections to your HP tenant stall10:28
davecheneyi'm sure they are doing some network reconfiguration as machines leave your tenant vlan10:28
fwereadedavecheney, btw, do you know the latest status of the mongodb in raring?10:39
fwereaderogpeppe, davecheney, dimitern: has anyone been able to bootstrap onto raring today?10:47
dimiternfwereade: haven't tried on raring10:48
rogpeppefwereade: i haven't tried - will do10:48
TheMuefwereade: just doing it with quantal, but not yet raring (have just update my test image to quantal, raring will follow)10:55
fwereadeTheMue, you've been testing bootstrap *to* not just bootstrap *from*, though, right?11:00
TheMuefwereade: yep, set in environments.yaml, i only wanted to have a matching image ;)11:01
fwereadeTheMue, so you've tested bootstrap to precise/quantal/raring from precise in a bunch of ec2 regions?11:02
TheMuefwereade: from quantal11:04
fwereadeTheMue, I has a confuse, I though you only just started working with the quantal image?11:04
TheMuefwereade: yesterday i tested precise from precise, now i want to test quantal from quantal11:06
fwereadeTheMue, ok, so everything in 1.9.14 works to/from precise? or not?11:07
TheMuefwereade: yes, for me with a clean image (no dev stuff lying around) and installed juju from the ppa e'thing worked fine11:08
TheMuefwereade: now it complains, charm not found11:13
TheMuefwereade: but bootstrap worked11:14
jamallenap: I just posted the reason why 'go build' still really needs you to be in GOPATH: https://code.launchpad.net/~jtv/juju-core/makefile/+merge/15864011:15
jamLet me know if I can clarify anything for you.11:15
davecheneyfwereade: re mongo in raring11:17
davecheneyas far as I am concerned, it works11:17
fwereadeTheMue, ofc it complains charm not found, there are hardly any charms for quantal ;p -- you'll need to deploy precise/mysql (or env-set default-series=precise, deploy mysql)11:18
fwereadedavecheney, I guess you talked to thumper about it last night? he seemed to be having problems iirc11:19
davecheneyTheMue: yes, the only series which is worth bootstrapping is precise11:19
davecheneythere are precious little charms for Q and R11:19
davecheneynot even the ubuntu charm11:19
fwereadedavecheney, well, there should in theory be no reason not to bootstrap into other series11:20
davecheneysure, you can bootstrap into Q11:20
davecheneythen deploy cs:precise/mysql11:20
fwereadedavecheney, but raring does seem to be acting pretty weird11:21
fwereadedavecheney, hey, btw, what would go wrong if we did start building everything for i386 as well?11:21
davecheneyfwereade: no, we can start doing that straght away11:21
davecheneyfwereade: on11:22
davecheneyone thing11:22
davecheneyif I bootstrap from a 386 machine11:22
davecheneyare the tools going to look for 386 or amd64 versions ?11:22
fwereadedavecheney, they should default to amd6411:22
davecheneyif the arch is clamped to amd64, then there will be no problem11:22
TheMuefwereade, davecheney: feels somehow funny, bootstrapping quantal and deploying precise11:22
davecheneygood, then there will be no problem11:22
fwereadedavecheney, client series should not affect chosen tools11:22
davecheneyTheMue: i agree, i think cross series environments are the work of the devil11:23
davecheneyfwereade: cool, i've adjusted the recipes to build amd64 and 38611:23
fwereadedavecheney, brilliant -- and the mongodb one too?11:23
davecheneythat is alreaedy done11:23
fwereadedavecheney, if it's not a problem that will enable developer to upload-tools from i38611:23
davecheneyand remember, you don't need mongo on the client11:23
fwereadedavecheney, <311:23
davecheneyonly on the server, and as we discussed that will always be amd6411:24
fwereadedavecheney, I had thought that was due to actual problems with i386?11:24
davecheneyfwereade: no, their problem is apt-get install juju-core on a 386 machine is a noop11:25
fwereadedavecheney, not being able to upload tools from i386 is I think also a problem, but I agree it is not a critical one that should seriously delay us11:26
davecheneyfwereade: you can run 386 tools on amd6411:26
davecheneybut the version won't match so the bootstrap won't work11:26
davecheneyif the arch on the uploaded tools' were clamped to amd64, this would solve that problem11:27
dimiterni'm installing a fresh vbox with raring daily to try bootstrapping11:28
fwereadedavecheney, heh, I had thought that should work, someone assured me it wouldn't and I never tried it11:29
fwereadedavecheney, but I think there's something I still don't get: what is the problem with running i386 servers?11:30
fwereadedavecheney, it seems like all our tools and dependencies *can* be built for i38611:31
davecheneyfwereade: there are two problems11:31
davecheney1. we don't have any released tools for 386 (that is being fixed, we build them from the packages in PPA)11:32
davecheney2. for most of the ec2 machines, amd64 is the deftault11:32
davecheneythe t1.micro is the only machine that runs 38611:32
davecheneyso the answer to both is, it should work, but we haven't tried11:32
davecheneymainly because upload tools was such an arse before you fixed it11:33
davecheneyto follow series11:33
fwereadedavecheney, also m1.small, m1.medium, c1.medium11:33
davecheneym1 small is amd6411:33
davecheneyanyway, it doesn't matter11:33
davecheneywe can fix it11:33
davecheneyit was just never a priority before11:34
davecheneydoing a test build now11:34
fwereadedavecheney, http://aws.amazon.com/ec2/instance-types/ says otherwise11:34
fwereadedavecheney, i386 is not my highest priority but it'd be great to have it as a possibility11:34
davecheneyfwereade: not really interested in arguing about this, the m1.small we always bootstrap for the state server is amd6411:34
davecheneyand this argument is impinging on my personal dislike for i38611:35
davecheneylet me get back to you when I have the build recipe straightened out11:35
fwereadedavecheney, sorry, I wasn't trying to argue with you -- but yeah, I am not helping your productivity11:36
davecheneynp probs11:36
davecheneyi don't want to argue about this -- it's trivial11:36
davecheneyone thing, i don't think the tests pass on 38611:36
davecheneycertainly not for all series11:36
davecheneybecause we don't have the right ec2 cloud data service fixtures11:37
fwereadedavecheney, hmm, that's interesting, sounds like it may partially be a test isolation issue though11:37
rogpeppedavecheney, fwereade: my raring bootstrap failed11:38
fwereadedavecheney, excluding upload-tools, client arch should not mater11:38
fwereaderogpeppe, Processing triggers for ureadahead ... forever?11:38
rogpeppefwereade: just looking11:38
* rogpeppe typed "juju looking" there initially11:38
davecheneyrogpeppe: occupational hazzard11:39
rogpeppefwereade: yiup11:39
rogpeppefwereade: wtf is ureadahead?11:40
fwereaderogpeppe, it speeds up boot stuff AIUI11:40
fwereaderogpeppe, I have *no* idea what is going on there or why it changed11:40
rogpeppefwereade: just looking at the ps output. what is whoopsie? http://paste.ubuntu.com/5721320/11:41
davecheneyjesus fuck people11:42
davecheneycan everyone stop asking about mongo/38611:42
* davecheney was referring to the mailing list11:42
davecheneywhich trailed the IRC channel by 20 mins11:42
rogpeppefwereade: and the pstree output which makes it clearer http://paste.ubuntu.com/5721323/11:43
davecheneywhoopsie is the thing that catches any SIGSEGV's and sends a report to ubuntu11:43
davecheneyrogpeppe: what does /var/log/cloud-init-output.log say11:44
davecheneyi bet it couldn't find the tools11:44
rogpeppedavecheney: i don't think it got that far11:44
davecheneyor possibly there was an error in bootstrapping which your set -xe change caused bootstrapping to quit before running its full course11:44
fwereadedavecheney, that's just a wget, but it gets stuck just after installing mongo11:44
davecheneyrogpeppe: it installed mongo, it has done at least some of cloud init11:44
rogpeppedavecheney: http://paste.ubuntu.com/5721330/11:45
rogpeppedavecheney: it's "processing triggers for ureadahead"11:45
davecheneysounds like a bug in raring11:45
rogpeppedavecheney: indeed11:45
davecheneyi couldn't get it to intsall in a vm on tuesday11:45
dimiternfwereade: danilos reports successful bootstrap and all on raring us-east-111:45
fwereadedimitern, normally I would be happy about that sort of news11:46
fwereadedimitern, today I just WTF even harder11:46
davecheneyfwereade: different regions may have different version or raring11:46
dimiternfwereade: once my raring vbox finally installs, i'd be trying out some other regions11:46
fwereadedavecheney, ah, yes, maybe they haven't been updated anywhere else yet11:47
danilosdimitern, fwereade: package version I am using: http://pastebin.ubuntu.com/5721342/11:49
fwereadedanilos, if it's still up, would you let me know the AMI you're running?11:50
danilosdimitern, juju status: http://pastebin.ubuntu.com/5721346/11:50
danilosfwereade, sure, looking11:51
dimiternwallyworld_: will you joining us on mumble?11:52
danilosfwereade, ami-d0f89fb911:52
fwereadedanilos, golly11:53
fwereadedanilos, I wonder how that one got there11:53
fwereadedanilos, ah, ok, that's bootstrapping into precise from raring11:54
davecheneyfwereade: each ami differs per region11:54
rogpeppefwereade, dimitern, davecheney: https://codereview.appspot.com/885104511:54
fwereadedavecheney, yeah, I was surprised that it wasn't a raring AMI11:54
fwereadedavecheney, then I realised default-series11:54
rogpeppefwereade: it's nice that i can now trivially start a raring bootstrap instance11:55
rogpeppefwereade: that CL fixes those bugs with juju get BTW11:56
fwereaderogpeppe, yeah, just a shame that it doesn't work ;p11:56
rogpeppefwereade: not our fault :-)11:56
rogpeppefwereade: apart from "we" is really canonical, so of course it's our fault...11:56
rogpeppedanilos: as on call reviewer, could i ask for a review of https://codereview.appspot.com/8851045 please?11:57
rogpeppetime for lunch11:57
dimiternfwereade: so ap-southeast-1 gives the "use of closed network connection" R->P12:01
danilosdimitern, fwereade: with ap-southeast-1 region it fails: http://pastebin.ubuntu.com/5721370/12:01
fwereadedimitern, danilos: I think that is the chunked encoding business that tim found12:02
dimiternfwereade: so how to go about fixing it?12:02
fwereadedimitern, danilos: would one of you try hacking up goamz/s3/s3.go to ReadAll of the response before trying to decode the XML, and see whether that helps?12:02
fwereadedimitern, danilos: it's in List IIRC12:03
dimiternfwereade: I can take a look, but let me first reproduce it12:06
fwereaderogpeppe, the raring bootstrap failure is resolved by dropping set -xe12:14
fwereaderogpeppe, ie the cannot bootstrap *onto* raring, vs *from* raring that dimitern's poking at12:15
dimiternfwereade: i.e. removing set -xe allows you to bootstrap to raring?12:16
fwereadedimitern, *onto* not *from*12:16
fwereadedimitern, yes12:17
rogpeppefwereade: that's odd - is the scope of that set -xe greater than the cloudinit scripts that we use?12:17
dimiternfwereade: what's the use - there are not charms for raring?12:17
fwereaderogpeppe, I reckon the ureadahead is connected12:17
fwereadedimitern, charms don't tend to run on the bootstrap machine anyway12:18
rogpeppefwereade: i'm surprised that any of our scripts have run by that stage, including the set -xe12:18
rogpeppefwereade: i'd have expected to see some output12:18
rogpeppefwereade: from the initial mkdir at any rate12:19
fwereaderogpeppe, yeah, it makes little sense12:23
rogpeppefwereade: i'm just having a look at the cloud-init sources12:24
fwereaderogpeppe, bah, status output ordering has changed12:24
rogpeppefwereade: hmm, where's that an issue?12:25
fwereaderogpeppe, seems a bit arbitrary, surprised my eye12:26
fwereaderogpeppe, not saying it's significant  to automatic consumers of that data12:26
rogpeppefwereade: ok. it's trivial to fix. do you want alphabetic ordering again?12:26
rogpeppefwereade: i hadn't realised it was an issue, sorry12:26
fwereaderogpeppe, yeah, would be nice, was just starting to do it myself12:26
fwereaderogpeppe, np, nor had I12:27
fwereaderogpeppe, stick with what you're doing, I'll propose in a mo12:27
rogpeppefwereade: if you could do it, that would be great. please leave Err at the top. otherwise, just pipe the struct fields through sort, and the tests should remain identical12:27
dimiternrogpeppe: reviewed12:31
rogpeppedimitern: ta!12:31
fwereadervba, ping12:31
rvbafwereade: pong12:33
fwereadervba, I think I just answered my own question actually but it could probably use some discussion12:33
TheMuerogpeppe: and another review12:34
rogpeppeTheMue: thanks12:35
fwereadervba, sorry,trying to marshal my thoughts12:35
rogpeppefwereade: i'd like your input too if that's ok.12:35
fwereaderogpeppe, sorry, on which?12:35
rogpeppefwereade: on https://codereview.appspot.com/885104512:35
rogpeppefwereade: 'cos it's a last-minute change that may well be crackful :-)12:36
fwereaderog, fuck, never sent my comment12:36
rogpeppefwereade: ah, np12:36
fwereaderogpeppe, I think we always want to output the actual value12:37
rogpeppefwereade: we always do, i think12:37
rogpeppefwereade: or do you mean that it should print nil when it's unset?12:37
fwereaderogpeppe, it should print the actual value12:37
fwereaderogpeppe, whatever the default happens to be, or maybe nil if there's no default12:37
rogpeppefwereade: doesn't it do that?12:37
fwereade+ "outlook": map[string]interface{}{12:38
fwereade+ "description": "No default outlook.",12:38
fwereade+ "type": "string",12:38
fwereade+ "default": true,12:38
fwereade+ },12:38
rogpeppefwereade: in that case there is no default12:38
rogpeppefwereade: i thought that omission was better than saying "nil" explicitly12:38
rogpeppefwereade: then the value is *always* of the correct type12:38
fwereaderogpeppe, I thought we were aiming for compatibility12:39
fwereaderogpeppe, python always outputs a value, I think12:39
* rogpeppe looks back at the python code12:39
TheMuefwereade: so "None" would be the correct value?12:40
fwereadeTheMue, well, nil I think12:40
TheMuefwereade: nil in Py is None, isn't it?12:41
rogpeppefwereade: ok; i don't like it, but i accept the compatibility argument.12:41
wallyworld_fwereade: dimitern: i've finally finished the openstack constrants work. i got bitten badly by a stupid Go gotcha regarding for loop variables, took me ages to find the cause of my test falures but it's finally done.12:41
fwereadeTheMue, ah sorry I thought you meant the string "None"12:41
dimiternwallyworld_: awesome12:41
fwereadewallyworld_, excellent12:41
rogpeppewallyworld_: using a loop variable in a closure, by any chance? :-)12:41
TheMuefwereade: ok, nil or None, how is it represented in yaml?12:42
wallyworld_rogpeppe: i was assigning the address of the for loop variable to another variable12:42
fwereadeTheMue, IIRC it's nil12:42
TheMuefwereade: I thought it would be the string None12:42
fwereadeTheMue, sorry, "null"12:43
rogpeppewallyworld_: ah yes. in a for range, presumably. i argued strongly that it should be in its own scope, but failed to persuade.12:43
TheMuefwereade: yep, just found it on yaml.org, null12:44
wallyworld_rogpeppe: yes, that is it. i am disappointed that Go behaves like that. it is so unintuitive and no other language suffers from that12:44
rogpeppewallyworld_: C behaves like that12:44
rogpeppewallyworld_: and C++12:44
dimiternwallyworld_: have you live tested this on both ec2 and canonistack or hp?12:44
TheMuerogpeppe: :D12:44
wallyworld_rogpeppe: hmm. i've never been bitten by the issue in those langiages12:44
wallyworld_dimitern: not yet. i wanted feedback in parallel with that testing. it works with the doubles etc.12:45
rogpeppewallyworld_: that's because one generally doesn't take the address of local variables. but i've been bitten by that kind of thing many times in C.12:45
wallyworld_fwereade: dimitern: you guys are most familiar with the required logic, so if you could look closely that would be great. not straight away, but at your convenience12:45
dimiternwallyworld_: sure, i'll look into it12:46
fwereadewallyworld_, cheers12:46
wallyworld_dimitern: there still needs to be a followup branch to rework some of the default image id stuff used in the live tests. but this branch is waaaaaay big enough already12:48
dimiternwallyworld_: what would be the nett gains from the follow-up?12:48
daniloswallyworld_, hey, you should be sleeping or drinking right now, not putting out huge branches while I am OCR! :)12:48
dimiternwallyworld_: considering the release is nigh, etc.12:48
wallyworld_danilos: sorry, i got back from soccer and really want to get this stuff done12:49
wallyworld_dimitern: the followup branch simply removes the need to specify default instance type and image id for the live tests12:49
daniloswallyworld_, no worries, it's going to be an interesting exercise for me, I am sure others will review it much faster though :)12:49
wallyworld_dimitern:  i thought i had already missed the deadline12:49
dimiternwallyworld_: ok, so istm we can postpone the follow-up post release on monday?12:50
wallyworld_danilos: it's a very big branch sorry. but a lot is deleted and/or moved code12:50
dimiternwallyworld_: the abs deadline is now eod monday12:50
daniloswallyworld_, yeah, I can see that (and you said as much in the MP)12:50
wallyworld_dimitern: yes, we can postpone, although the followup will just be test changes12:50
dimiternwallyworld_: sweet12:51
wallyworld_dimitern: hopefully when i do the live tests with this everything will work ok, otherwise i'll need to tweak a bit12:51
rogpeppefwereade: PTAL https://codereview.appspot.com/8851045/12:51
wallyworld_danilos: the idea is that the logic used to live in ec2, but should be common to ec2 and openstack etc. the only ec2 and openstack specific bits is the logic to select what instance types to consider and where the image metadata comes from12:52
rogpeppewallyworld_: i'm wondering if the moved logic would work better in its own package12:55
TheMuegna, bootstrap and later destroy works, but status and scp any log not (local: quantal / remote: raring)12:55
rogpeppewallyworld_: (this is only after a tiny peek BTW)12:55
rogpeppewallyworld_: perhaps environs/instances ?12:55
wallyworld_rogpeppe: i'd have no objection to that12:55
fwereaderogpeppe, LGTM with one tedious request12:56
rogpeppewallyworld_: i'd just like to try to avoid cluttering environs with lots of logic that isn't truly universal to all providers12:56
wallyworld_sure, np12:56
rogpeppefwereade: ooo kkkk12:56
fwereadeTheMue, when you say "bootstrap works", did you log in and check for a running agent?12:57
fwereaderogpeppe, wallyworld_: +1 on environs/instances12:57
allenapjam: That was a useful explanation, thank you :)12:57
rogpeppefwereade, wallyworld_: actually, how about environs/instance and environs/image ?12:57
wallyworld_dimitern: fwereade: danilos: one thing i forgot to add but will do so before landing is logging when the fallback instance choosing logic is invoked so the user knows their chosen instance type is not being used, but a "best guess" is12:58
rogpeppefwereade: what's your objection to "  if s, ok := serviceCfg[k]; ok {" BTW?12:58
dimiternwallyworld_: oh yeah, sgtm, thanks12:58
fwereaderogpeppe, that was danilos12:58
rogpeppefwereade: oh yeah12:58
rogpeppefwereade: sorry12:58
fwereaderogpeppe, np12:58
wallyworld_rogpeppe: i think that's a bit too far? the logic is conceptually about choosing an instance to bootstrap so it sort of all belongs together12:59
rogpeppewallyworld_: ok, seems reasonable. i'd keep it singular though probably, though YMMV12:59
* wallyworld_ goes to get an alcoholic drink :-) or three13:00
dimiternwallyworld_: have phun :)13:01
TheMuefwereade: will do13:02
fwereaderogpeppe, dimitern: https://codereview.appspot.com/8834047 (trivial probably)13:03
danilosfwereade, is that just sorting entries in a struct?13:04
rogpeppefwereade: LGTM13:04
dimiternfwereade: me too13:04
rogpeppefwereade: trivial13:05
fwereadedanilos, yep13:05
fwereadecheers guys13:05
danilosfwereade, I was going to LGTM, but I am too late I suppose :)13:05
danilosanyway all, I am OCR, so feel free to ping me for reviews13:05
fwereadedanilos, if you do it quickly you'll beat the submit ;p13:05
danilosfwereade, heh, nah, you've got 3 already, that's plenty enough ;)13:06
rogpeppefwereade: i think i'll just delete that "breaks compatibility" comment13:06
dimiterndanilos: when you're getting into the code all all is new, don't hesitate to review even stuff that has 2 LGTMs; asking questions always helps13:06
rogpeppefwereade: i agree with danilos' remark13:06
danilosdimitern, sure thing13:07
fwereaderogpeppe, sorry, which?13:07
rogpeppefwereade: the on13:07
rogpeppe// This breaks compatibility with py/juju, which will set13:07
rogpeppe// default to whether the value matches, not whether13:07
rogpeppe// it is set in the service confguration.13:07
fwereaderogpeppe, that we ought to collect that stuff for the release notes? yeah :/13:08
rogpeppefwereade: that is true13:08
rogpeppefwereade: i suppose i should email dave13:09
fwereaderogpeppe, stick it in a Done card maybe13:09
fwereadedimitern, any interesting results with ReadAll?13:10
dimiternfwereade: not really13:10
dimiternfwereade: my raring vm is misbehaving still13:11
fwereadedimitern, so ReadAll gets a closed connection?13:11
fwereadedimitern, bah, ok13:11
fwereadedanilos, are you set up on raring atm?13:11
danilosfwereade, yeah13:11
dimiternfwereade: i'll let you know if i make a breakthrough13:11
fwereadedanilos, ok, can I ask you to investigate the public-tools issue please?13:12
fwereadedanilos, you'll want to try bootstrapping without --upload-tools13:12
fwereadedanilos, and observing a "cannot find tools: connection closed" or something13:12
danilosfwereade, sure, on any region specifically?13:12
fwereadedanilos, have you seen those at all?13:13
danilosfwereade, seen it with ap-southeast-1, not with the default us-east113:13
fwereadedanilos, or have you just not been bootstrapping without --upload-tools?13:13
danilosfwereade, I haven't tried with --upload-tools, no13:13
fwereadedanilos, ok, cool, I seem to see it all the time in ap-southeast-2, but wherever you can repro it reliably13:14
fwereadedanilos, we don't want --upload-tools here I think13:14
danilosfwereade, right, understood13:14
fwereadedanilos, if you look in goamz/s3/s3.go13:14
danilosfwereade, should I use 1.9.14 package or trunk?13:15
fwereadedanilos, in the List method IIRC13:15
fwereadedanilos, trunk please13:15
fwereadedanilos, not worried about actually bootstrapping successfully, just listing the tools ok13:15
fwereadedanilos, there's a line with xml.NewDecoder(hresp.Body) or something13:16
fwereadedanilos, try to ReadAll the  body into a buffer and see whether we can decode that ok13:16
rogpeppefwereade: FWIW the next lines after "Processing triggers for ureadahead ..." are:13:16
rogpeppeSetting up multiarch-support (2.17-0ubuntu5) ...13:16
rogpeppe(Reading database ... 52136 files and directories currently installed.)13:16
danilosfwereade, yeah, that's in S3.run()13:16
fwereaderogpeppe, sorry, ECONTEXT -- this is bootstrapping into raring from tip?13:17
fwereaderogpeppe, maybe I just never gave the ureadahead trigger long enough?13:17
rogpeppefwereade: into raring from precise13:18
rogpeppefwereade: you did - mine is still there.13:18
rogpeppefwereade: after some hours13:18
fwereaderogpeppe, yeah, I thought I'd given it long enough... but I never saw anythng after the ureadahead triggers13:19
fwereaderogpeppe, am I blithering?13:19
rogpeppefwereade: only if i am13:19
rogpeppefwereade: so probably :-)13:19
fwereaderogpeppe, ISTM that dropping `set -xe` makes it all work, can you confirm/deny?13:20
rogpeppefwereade: i will try13:20
=== wedgwood_away is now known as wedgwood
rogpeppefwereade: submitting the config get branch first13:21
fwereaderogpeppe, go for it13:21
rogpeppefwereade: done13:22
fwereaderogpeppe, ok, hum, now it seems not to work any more13:22
fwereaderogpeppe, which is sort of good, because it was obviously an insane "fix"13:23
rogpeppefwereade: yup. although i could kind of imagine a way in which it might possibly have been a fix in some moderately insane kind of way13:23
fwereaderogpeppe, but... now I don't know at all what is going on :(13:23
rogpeppefwereade: i really looks like a raring problem13:23
rogpeppefwereade: you know, i think cloud-init isn't hung up there - i think it's probably finished13:24
rogpeppefwereade: maybe the runcmd thing doesn't work in raring13:25
rogpeppefwereade: but... surely cloud-init can't be broken that badly?13:25
fwereaderogpeppe, hmm, that is surprising to me13:25
fwereadervba, ok, I know my question now13:25
rogpeppefwereade: i can't currently see anything in ps alxw that says "python" or "cloud"13:25
rogpeppefwereade: and the final line in cloudinit.log is "Apr 19 10:58:16 ip-10-4-50-223 [CLOUDINIT] cloud-init[DEBUG]: Ran 18 modules with 0 failures"13:25
fwereadervba, once the node is acquired, how do we find out what arch it has?13:26
danilosfwereade, does log.Printf have a length limit?13:26
fwereadedanilos, not that I am *aware* of, but...13:26
fwereaderogpeppe, huh, that is most upsetting13:27
danilosfwereade, it's cut-off XML that I get13:27
danilosfwereade, can't pastebin it since it thinks it's PHP or other web scripts13:27
fwereadedanilos, bah13:27
fwereadedanilos, this works: http://paste.ubuntu.com/5721579/ ..?13:29
rogpeppefwereade: the set -xe can't be anything to do with it - the scripts really are in their own #!/bin/sh file.13:29
danilosfwereade, http://people.canonical.com/~danilo/list-xml.txt13:29
fwereadedanilos, cheers13:29
danilosfwereade, in general it does, but pastebin heuristic is probably bad here ;)13:29
danilosfwereade, I'll try this with a region I had it working with just to make sure it's not Printf problem13:30
rogpeppehmm 7882 bytes; i wonder if that's an 8K block with a 310 byte html header or something13:31
danilosfwereade, without the region set it works, but the output is much shorter: http://pastebin.ubuntu.com/5721586/13:32
fwereaderogpeppe, are you aware of any magic necessary to deal with chunked transfer?13:32
danilosdoing any of this from gdb is not very useful it seems :/13:32
rogpeppefwereade: there is definitely magic in that area, but whether it's relevant here i dunno13:32
danilosfwereade, I'll take a peek at Content-Length as well13:33
rogpeppedanilos: that's actually longer (8138 bytes)13:33
fwereadedanilos, tyvm13:33
danilosrogpeppe, is it? it seemed shorter ;)13:33
rogpeppedanilos: istm that a truncation at 8192 might be possible13:34
rogpeppedanilos: wc is your friend :-)13:34
danilosrogpeppe, that would require me to get out of gdb (which is not a bad idea, considering how "useful" it is :))13:34
* rogpeppe doesn't like gdb much13:35
rogpeppedanilos: does this only happen with the released binary?13:38
danilosrogpeppe, nope, I am using trunk now13:38
danilosrogpeppe, and region ap-southeast-113:38
rogpeppedanilos: and you can reproduce the issue? fantastic.13:38
danilosrogpeppe, yeah, it seems so13:44
rogpeppedanilos: i'm just trying myself, from the raring instance that failed to bootstrap correctly :-)13:44
danilosrogpeppe, fwereade: ContentLength is -1 fwiw (indicates "unknown" according to http://godoc.org/net/http#Response)13:44
danilosrogpeppe, cool13:45
rvbafwereade: something like this should work: http://paste.ubuntu.com/5721626/13:48
rogpeppeblasted goyaml requires gcc :-)13:48
fwereadervba, ok, cool -- and are the possible values "amd64", "i386", "arm"?13:49
fwereadervba, or should there be a translation layer?13:50
* rogpeppe finally has a juju binary built on raring13:50
rvbafwereade: "i386" / "amd64" / "armhf/highbank"13:51
rvbafwereade: wait, no: 'i386/generic' / 'amd64/generic' / 'armhf/highbank'13:51
fwereadervba, this is I guess the point at which I need to start understanding something about arm ;p13:52
fwereadervba, anyway the thing on my mind is the arbitrary tools choice13:52
fwereadervba, I guess you only have amd64 machines available currently?13:52
rvbaamd64 and arm machines actually.13:53
rvba(in the MAAS lab)13:53
fwereadervba, hmm, how is it that we never accidentaly pick and arm machine?13:53
rvbafwereade: when I test things in the lab (with Go juju), I disable the arm nodes.13:54
fwereadervba, (the issue is just that acquireNode doesn't pay attention to possibleTools, and it ought to be constraining the arch of the machine chosen to one we have tools for)13:54
fwereadervba, I guess we can always just loop over acquires until we get one we have tools for13:55
fwereadervba, but that feels a bit crap13:55
rvbaIt does.13:55
rogpeppedanilos: right, i've replicated the same problem13:55
fwereadervba, is there any way to say "a machine with one of these architectures"?13:55
fwereadervba, because possibleTools.Arches() will give you the necessary input for that13:56
rvbafwereade: that's already what the constraints do IIRC.13:56
rvbaOh, you mean one of these archs as opposed to just this arch right?13:57
fwereadervba, yeah13:57
rvbalet me checkā€¦13:57
fwereadervba, actually maybe we don't want exactly that13:59
danilosrogpeppe, it seems to fail at reading "https://s3.amazonaws.com/juju-dist/?delimiter=&prefix=tools%2Fjuju-&marker=" for me, which comes in just fine in a browser13:59
rvbafwereade: this would require a change in MAAS.  Right now, it expects one or zero value for the architecture constraint.14:00
rogpeppedanilos: i'm just trying to see if i get the same failure when compiling against go tip14:00
fwereadervba, ok I think I know what we should do then14:00
fwereadervba, of the arches from possibleTools (which have already taken constraints into account), sort by preference, and construct a fresh constraints for each arch14:01
fwereadervba, if we can't acquire a node matching the first arch, try the others in order before giving up14:02
fwereadervba, sane?14:02
rvbafwereade: sounds sensible14:02
fwereadeoh blast meeting14:02
danilosrogpeppe, now I am getting the same even without the region set: http://pastebin.ubuntu.com/5721653/ (I am not sure if it's related to reusing control-buckets or not since that's in the URLs it tries to get a list off)14:05
rvbafwereade: changing the maas side to accept a list of arch constraints is simpler though, and completely backward compatible.14:05
rogpeppefwereade: was just kicked off14:05
fwereadervba, will have to think about that, not sure if there's some sort of preference ordering we can/should assume14:06
danilosrogpeppe, fwiw, I am printing URLs it sends requests to in there14:07
rogpeppedanilos: in a call currently14:08
danilosrogpeppe, sure, I suppose you won't need me anymore if you can reproduce it yourself anyway ;)14:08
rogpeppedanilos: you may well find a more promising line of enquiry14:09
danilosrogpeppe, oh, this was probably failing because I consumed the entire response body with ReadAll()14:12
rogpeppedanilos: sorry, what was probably failing?14:12
danilosrogpeppe, never mind, brain blip14:12
danilosrogpeppe, fwereade: however, reusing the same HTTP connection (re-enabling keep-alive by commenting out "Close:true" in the request) made it work for me with ap-southeast-1, or at least not fail in the same spot14:14
danilosrogpeppe, fwereade: so it seems that amazon decides to kill off connections on some zones earlier than on others14:14
hazmatso what i was saying is that juju doesn't specify a key pair when launching an instance14:17
hazmatan aws key pair that is14:17
danilosrogpeppe, fwereade: yeah, this patch was sufficient to get latest juju core to work even with ap-southeast-1 for me: http://pastebin.ubuntu.com/5721681/; juju status at http://pastebin.ubuntu.com/5721680/14:21
fwereadedanilos, yay!14:21
danilosfwereade, rogpeppe: should I leave that with you guys since I am not sure what I could be breaking by switching to keep-alive HTTP? :)14:22
dimiterndanilos: awesome!14:23
rogpeppedanilos: good work14:23
mrammrogpeppe: can you land that one?14:23
fwereadedanilos, I'm not sure we know either, but I think rogpeppe has access to goamz14:23
rogpeppedimitern: i want to find out *why* it makes that difference14:23
mrammrogpeppe: I think you have goamz commit14:23
fwereaderogpeppe, ofc14:23
danilosrogpeppe, I assume it's amazon deciding to kill off if you do 10-15 HTTP requests in separate connections in space of 1-2s14:25
danilosrogpeppe, perhaps a firewall setting on their side to defend against DoS or bad API clients or similar...14:26
rogpeppedanilos: i suppose so; seems weird. why from raring only?14:26
danilosrogpeppe, good point, I don't know :)14:26
fwereaderogpeppe, not from raring only I think actually14:27
rogpeppedanilos: no? it works ok from precise14:27
fwereaderogpeppe, it looked for a while as if it were14:27
danilosrogpeppe, fwereade: want me to try something else before I destroy-environment?14:27
fwereaderogpeppe, not always and not for everybody14:27
rogpeppefwereade: the s3 package has, i think had quite a lot of use.14:27
rogpeppefwereade: ah!14:28
fwereaderogpeppe, but our public-bucket has only recently started bumping up against 8k of tools info14:28
rogpeppei think it's probably an old go bug14:28
rogpeppefixed in tip14:28
fwereaderogpeppe, that sounds encouraging14:28
rogpeppebecause i just compiled against go 1.1 beta and it works14:28
fwereaderogpeppe, that is less encouraging because I have no desire whatsoever to switch language version14:29
rogpeppei just successfully kicked off an instance from a raring-built juju ex14:29
rogpeppefwereade: yeah i know14:29
fwereaderogpeppe, dimitern, mramm: what's the worst thing that could happen if we just trash everything in the bucket older than, say, 1.9.10?14:30
rogpeppefwereade: kittens die?14:30
rogpeppefwereade: do it!14:30
fwereaderogpeppe, dimitern, mramm: if it is, as it seems it may be, size-related, that would give us some breathing room14:30
fwereaderogpeppe, shit, I don't think I have keys for that bucket14:31
rogpeppefwereade: i think i might have14:31
TheMuestrange, even ssh-keygen for the host doesn't help14:32
dimiternfwereade: go for it14:32
fwereadeTheMue, which authorized-keys are you using?14:32
rogpeppefwereade: i just PM'd you some that might work14:32
mrammI'm fine with trashing old tools now14:33
TheMuefwereade: i've done a ssh-keygen -R ec2-... for the dns name14:33
mrammas long as fwereade thinks it makes sense ;)14:34
rogpeppeTheMue: this is what i do to ssh in to an instance: ssh -i $home/.ec2/rog.pem ubuntu@$114:34
fwereademramm, I *think* it does, but I am drawing a blank on what fricking tools I should be using14:34
dimiternso bootstrapping from Q to R gives me the cloud-init-output.log up to "processing triggers for ureadahead"14:36
fwereadedimitern, and apparently no scripts run, right?14:36
mrammit would be good to make copies of the older tools somewhere before deleting them all14:36
dimiternfwereade: how can I tell - mongo seems installed and running14:36
mrammthat said, I *doubt* we've hit the maximum size limit on s314:36
fwereadedimitern, anythig starting with "juju" in /etc/init?14:37
dimiternfwereade: but status is failing with "2013/04/19 16:37:05 ERROR state: connection failed, paused for 2s: dial tcp connection refused"14:37
fwereademramm, just an 8k chunk size for that particular response14:37
fwereadedimitern, we shouldn't be starting that mongo at all actually14:37
fwereadedimitern, it should be "juju-db" or something14:37
dimiternfwereade: no juju* in /etc/init/14:37
dimiternfwereade: wanna see the full c-i-o.log?14:38
TheMuedimitern: that's what i got too when doing juju status14:38
fwereadedimitern, I've seen it plenty of times14:39
fwereadedimitern, TheMue: there is no controversy about what happens, we just want to figure out why ;p14:39
dimiternfwereade: how should it look like when it works?14:39
mrammfwereade: I'm fine with moving the old ones out, and testing to see if that makes a difference14:40
dimiternfwereade: I mean, what fails to run exactly?14:40
fwereadedimitern, AFAICT, none of the scripts we set ourselves14:40
fwereadedimitern, it's just the packages14:40
fwereadedimitern, hmm, maybe poke around in the actual userdata from the metadat service14:41
fwereadedimitern, just to verify that we do have sane input data14:41
fwereadedimitern, not that I really doubt it14:41
fwereadedimitern, the stuff we want to run is all in environs/cloudinit/cloudinit.go14:41
dimiternfwereade: i'll look14:41
fwereadedimitern, there's an addScripts func that's called about 100 times :)14:41
hazmatdimitern, ec2metadata is installed for looking at the raw metadata service14:42
hazmatalso in /var/lib/cloud/instance/user-data.txt14:42
fwereadedimitern, if the userdata has stuff that looks familiar from there, we can start getting serious about blaming cloudinit ;p14:42
hazmatdimitern, its not particularly important.. but because juju doesn't specify an amz keypair name, its not actuall installed. amz does not install a default key pair on instances, unless one is specified. its cloudinit running and dropping in the key thats working. it can be seen14:42
hazmatfwereade, unlikely14:42
* hazmat is trying it out on raring too14:43
fwereadehazmat, well, indeed, so hopefully we *will* be finding that we have fucked out input data somehow14:43
hazmatfwereade, what was the fix for 2013/04/19 07:43:31 ERROR command failed: use of closed network connection14:43
hazmatjust apply the pastebin patch?14:43
fwereadehazmat, I think so -- but I haven't actually verified that one myself14:44
dimiternthere are 2 files in /var/lib/cloud/instance with similar names: user-data.txt (9616) and user-data.txt.i (60748)14:44
hazmati'm not able to bootstrap on ec2 atm  because of it..14:44
hazmatdimitern, the first one14:44
hazmatits compressed with juju-core14:44
dimiternhazmat: what's the .i one?14:45
dimiternhazmat: it looks weird - like a mail message dump14:46
dimiternthere's a bug reported about raring ignoring runcmd in cloudinit: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/115872414:47
rogpeppethat might just have something to do with it14:49
fwereadewell, fuck me14:49
dimiternand several others related: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1103881 (marked as dupe of the one above)14:49
rogpeppedimitern: nice one14:49
dimiternso raring's cloudinit's fucked then14:49
hazmatdimitern, thats not correct14:49
hazmatdimitern, you can see it trying to run in the cloudinit output14:49
hazmatits getting confused by the cert stuff14:50
hazmatcloud init output .. http://paste.ubuntu.com/5721757/14:50
rogpeppehazmat: it doesn't get that far for me14:51
hazmatrogpeppe, this is with fresh trunk14:51
dimiternhazmat: in my case it never got that far14:51
hazmatand upload-tools14:51
hazmatwithout upload-tools i can't bootstrap because of the closed network conn error14:51
rogpeppehazmat: it stops just after line 176 on your paste there14:52
dimiternhazmat: try danilos's patch above14:52
rogpeppehazmat: with a "Processing triggers for ureadahead ..." line14:52
dimiternrogpeppe: exactly like here14:53
rogpeppehazmat: the "closed network conn" error is, i'm pretty sure, a go1.0.2 bug. it's fixed in tip. it may even be fixed in 1.03 - i'll just try that.14:53
danilosrogpeppe, fwereade: fwiw, I've tested go 1.0.3 from https://launchpad.net/~gophers/+archive/go/+build/3851809 and based on test case from http://code.google.com/p/go/issues/detail?id=4704 I've created my own at http://pastebin.ubuntu.com/5721759/ which consistently fails with raring go 1.0.2 and succeeds with this 1.0.3 from niemeyer's PPA14:53
danilosrogpeppe, just as you were saying that :)14:53
hazmatwe should really have 1.0.3 in raring..14:53
rogpeppehazmat: +10014:54
rogpeppedanilos: ok, so it was fixed in 1.0.3, the actual go release14:54
fwereadeaw hell, 1.0.2 -> 1.0.3 is a theoretically unscary sort of change I guess14:54
dimiternfwereade: right now before the release of raring? :)14:54
fwereadebut there is always the famous difference between theory and practice14:55
rogpeppefwereade: the only problem is 1.0.3 does actually have a bug that affects http retries14:55
hazmatDaviey, arosales, can you have someone push golang 1.0.3 for raring?14:55
rogpeppefwereade: i don't *think* it affects us, but niemeyer knows much more14:55
fwereaderogpeppe, where does that hit us?14:55
fwereaderogpeppe, ah ok14:55
hazmatrogpeppe, so both versions are busted?14:55
rogpeppehazmat: there have been shitloads of bugs fixed since 1.0.314:56
hazmatrogpeppe, and no 1.0.4 in site?14:56
mrammbut only in trunk :/14:56
mramm1.1 is coming14:56
mrammsoon and very soon14:56
rogpeppehazmat: 1.1 is on feature freeze now14:56
fwereadeTheMue, any luck so far?14:56
niemeyerdanilos, rogpeppe: Probably related to issue 491414:56
TheMuefwereade: just trying from a different machine14:57
mrammso for this one, can we just add the workaround?14:57
niemeyerIt's not about 1.0.2 vs 1.0.3.. it's about a patch someone cowboyed on the Debian package14:57
rogpeppeniemeyer: ah14:57
mrammchanging go versions at this point will be counterproductive I think14:57
fwereademramm, agreed14:57
hazmatarosales, Daviey pls ignore 1.0.3 is apparently broken and we have workarounds for 1.0.214:57
niemeyerand AFAIK it's still there, despite me trying to find a new maintainer for the package on the ML14:57
hazmatniemeyer, oh.. can we yank that patch14:58
mrammniemeyer: that is no good14:58
niemeyerif there are any proud Debian package maintainers around, taht'd be a great time :)14:58
niemeyerhazmat: We should really update to 1.0.3 instead14:58
rogpeppeniemeyer: there's that problem with 1.0.3 that you encountered. is that going to a problem for us?14:59
niemeyerrogpeppe: That was about rietveld, IIRC14:59
niemeyerrogpeppe: We don't have to build lbox with 1.0.314:59
rogpeppeniemeyer: that's what i thought14:59
rogpeppeniemeyer: but you might have used similar techniques in goamz, i thought15:00
niemeyerrogpeppe: I don't *think* so..15:00
rogpeppeniemeyer: if you haven't then we're all good :-)15:00
mrammwhat was the bug though?   Have we tested on 1.0.3?  Are we going to hit it somewhere else?15:00
niemeyerrogpeppe: The missing feature in 1.0.3 is the ability to break redirections15:00
niemeyerrogpeppe: Which we use with Rietveld to catch a cookie in-flight15:01
dimiternniemeyer: yeah, that's why I had to patch lpad for lbox to work for me on 1.0.315:01
niemeyerrogpeppe: 1.0.3 bogusly broke the ability to that with the http package15:01
rogpeppeniemeyer: yeah, i had vague recollections of that15:01
fwereaderogpeppe, mramm, dimitern, hazmat: I can confirm that cloudinit does the right thing if we switch off apt upgrade :/15:01
* rogpeppe wishes niemeyer had pushed harder at the time for a patch to 1.0.315:01
dimiternfwereade: i'll try15:01
niemeyerrogpeppe: We survived fine, though15:02
rogpeppefwereade: do we need apt upgrade?15:02
rogpeppeniemeyer: true. it's been itchy at times though15:02
fwereaderogpeppe, well, it seems in general like a sensible thing to do15:02
dimiternfwereade: so just comment out this line: c.SetAptUpgrade(true)15:02
fwereadedimitern, that's all I did15:03
fwereaderogpeppe, maybe it is less important just after a series has been released though ;p15:03
niemeyermramm,hazmat: It's worse than that..15:05
niemeyerThere's nothing to import from Debian either15:05
hazmatfwereade,  interesting.. for some reason it works for me.. in terms of getting to runcmd (us-east-1, trunk w/ upload-tools) http://paste.ubuntu.com/5721791/15:05
hazmatone odd thing in that cloud-init .. its adding in the experimental ppa15:05
fwereadehazmat, ISTM you're bootstrapping into precise15:06
hazmatmaybe that's  for mongodb15:06
dimiternfwereade: i can confirm it works for me with that line commented out15:06
hazmatfwereade, doh.. if that's the default then yes15:06
fwereadehazmat, which it will if you don't specify otherwise -- that's what default-series now defaults to15:06
hazmati thought the default was the client series15:06
fwereadehazmat, it was, but 99% of charms are for precise15:07
fwereadehazmat, the ideal would maybe be to separate bootstrap-series from default-series15:07
fwereadehazmat, but in practice people who want to `juju deploy wordpress` want precise as a default15:07
rogpeppenow that we've isolated the issues, i'm going for the rest of my lunch break :-)15:07
fwereaderogpeppe, enjoy15:07
fwereaderogpeppe, tyvm15:08
mrammfwereade: but I think this is fine15:08
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: - | Bugs: 2 Critical, 61 High - https://bugs.launchpad.net/juju-core/
dimiternfwereade: we can add a script calling apt-get upgrade at the end maybe?15:08
hazmatfwereade, makes sense.. sorry for the confusion15:08
fwereadedimitern, ha, yes, we could -- that's nice15:08
mrammand I think it would even be fine to *always* start the bootstrap machine on the LTS15:08
dimiternfwereade: i'll try it out and if it works will propose it15:09
mrammdimitern: sounds like a good move15:09
mrammniemeyer: do you know who has keys to the public tools bucket in amazon?15:10
fwereademramm, I think I would prefer to stick with the existing behaviour if we can get it working right15:11
niemeyermramm: I was hoping that only David would do, but I think he gave to someone else as well15:11
mrammfwereade: agreed15:11
niemeyermramm: i have them as well, obviously, as I created the bucket15:11
niemeyermramm: I mean, not his keys, but access to the bucket15:11
mrammthe underlying upgrade issue is: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/112438415:11
hazmatthat's a dup of https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/110388115:14
mrammniemeyer: can you share permissions on the bucket to fwereade?15:16
dimiternso adding "apt-get upgrade -y" at the end of scripts seems to work15:19
fwereadedimitern, I don't think it should be at the end really, better to have it at the beginning15:19
dimiternfwereade: the main problem is upstart, if we upgrade it too early, the same issue will happen15:20
dimiternfwereade: and i don't think it matters when, all relevant stuff will be restarted anyway after15:20
fwereadedimitern, unless it's still running while a charm tries to use apt itself15:20
fwereadedimitern, academic for now but a potential landmine all the same15:21
dimiternfwereade: I can not do it at all, if you think it's riskier to have it15:21
dimiternfwereade: just leave the commented out part, but remember this will affect all series, not just raring15:22
fwereadedimitern, I think we should definitely be special-casing it for raring15:22
fwereadedimitern, if cfg.Tools.Series == "raring"15:22
arosaleshazmat, ack15:23
dimiternfwereade: ah, good point, will do15:24
fwereadedimitern, and in other cases just do a normal SetAptUpgrade15:24
dimiternfwereade: alas cfg.Series is unknown15:25
fwereadedimitern, look up15:25
fwereadedimitern, cfg.Tools.Series15:26
dimiternfwereade: there's cfg.Tools.Series instead - should I use that?15:26
dimiternfwereade: ok\15:26
fwereadedimitern, that's what I said to begin wth ;15:26
dimiternfwereade: sorry :)15:27
fwereadeno worries :)15:27
dimiternfwereade: switching too fast between sessions15:27
fwereadedimitern, I know the feeling ;P15:28
* danilos is off for the day and week, enjoy it everyone15:28
dimiterndanilos: happy weekend! and thanks for debugging!15:29
fwereadedanilos, enjoy, and thank yu very much15:29
TheMuefwereade: just a short intermediate information, it looks like i've got a ssh config problem. :( i only can reach one of my two private hosts, no other one. have to look why :/15:30
fwereadedimitern, ok, so, doing it at the end is probably ok15:31
TheMuefwereade: so it's no wonder i can't reach any ec2 host15:31
dimiternfwereade: i already did it in the beginning and testing now15:31
fwereadedimitern, well if that works I think it'd be best15:31
fwereadedimitern, but you made a good point ;)15:32
dimiternfwereade: cheers15:32
fwereadeTheMue, which authorized-keys are you using?15:32
fwereadeTheMue, surely you can usually ssh to your machines, right?15:32
dimiternit'll be a bitch to test it - i have to clone a bunch of cloudinit outputs and force series to raring just for these 2 lines of code15:32
fwereadedimitern, shouldn't be *too* bad though15:33
TheMuefwereade: to my machine?15:33
TheMuefwereade: ah, wrong read15:33
fwereadeTheMue, when you run juju, can you usually `juju ssh 0`?15:33
TheMuefwereade: it has been possible, but not now anymore. so i tried two private hosts. the one is working, the other not. i'm puzzled.15:34
dimiternaaaaand it works!15:34
* fwereade cheers at dimitern15:35
TheMuedimitern: applause15:35
fwereadeTheMue, ok, we *know* it's not going to work on raring without dimitern's fix15:35
TheMuefwereade: great15:35
fwereadeTheMue, none of the juju commands will15:35
fwereadeTheMue, because no state servers or agents or anything are started15:36
TheMuefwereade: but i just also tested precise, just to make sure that this is not the reason, and my box still fails15:36
TheMuefwereade: running without a state server is a bit, hmm, useless :D15:36
fwereadeTheMue, ok, so, for the 3rd time of asking, which public key are you authorizing?15:36
fwereadeTheMue, and can you ssh to it directly if you `ssh -i appropriate-private-key ubuntu@blahblahblah`?15:38
TheMuefwereade: i've tried with the private key in my .ssh folder15:39
TheMuefwereade: i've got to admit i've never directly connected to ec2 before, never needed it15:40
TheMuefwereade: and that missing experience now is my problem15:40
fwereadeTheMue, do you maybe have a strange authorized-keys, or authorized-keys-path, configured?15:41
dimiternTheMue: can you please go here: https://portal.aws.amazon.com/gp/aws/securityCredentials15:42
TheMuefwereade: can't remember, but i'll take a deeper look. normally i use everything as standardized as possible15:43
dimiternTheMue: authenticate beforehand, then go to Key Pairs tab15:43
dimiternTheMue: create a new key pair, download the private key, save it to your ~/.ssh/, chmod it to 600, then add the snipped I pasted to the kanban meeting into your ~/.ssh/config15:44
dimiternTheMue: after that it should work and you will be able to ssh without problems15:44
TheMuedimitern: i'm doing15:44
TheMuedimitern: could you please paste that snipped again?15:45
fwereadedimitern, what happens to you if you comment out that line from your config and just `ssh ubuntu@blah`?15:46
dimiternfwereade: without the ssh config like this it fails (pubkey auth error), I have to use ssh ubuntu@blah -I ~/path/to/key15:47
fwereadedimitern, so it doesn't automatically pick the right key? do you have loads of them set up or something? :)15:47
dimiternfwereade: because i hate typing on the console more than i should, i added the ssh config to save me some typing, that way i can do just ssh blah and it works, as long as the dns name ends with .computeaws.com15:48
dimiternfwereade: i have like 10 keys in there15:48
fwereadedimitern, so, hmm, maybe we are picking a first choice that ssh doesn't?15:48
dimiternfwereade: most likely, yeah15:49
fwereadedimitern, ok, cool, that makes sense15:50
TheMuehmm, error changes, now i have a timeout. but eu-west-1 is slow the whole day.15:53
dimiternTheMue: it usually takes me 2-3 mins to connect with ssh, after a successful bootstrap on eu-west-115:54
TheMuedimitern: bootstrap has been a longer time ago ;)15:54
dimiternfwereade, rvba: I'm seeing test failures related to maas in trunk now - any clue?15:54
fwereadedimitern, update maaslib15:55
fwereadedimitern, or whatever it's called15:55
dimiternfwereade: ok, cheers15:55
dimiternrogpeppe: what about danilos's fix to goamz?15:56
rogpeppedimitern: it's the wrong fix15:56
rogpeppedimitern: we really need to use a non-broken version of go15:57
rogpeppedimitern: i think that danilos' fix will probably break other things15:57
dimiternrogpeppe: but won't the fix help us with the current release at least?15:57
rogpeppedimitern: there's a good reason why Close is set to true15:57
dimiternrogpeppe: not really - HTTP/1.1 + Keep-Alive has been around since forever now15:58
dimiternrogpeppe: if go implementation is crack, that might be a reason15:58
rogpeppedimitern: i don't think you can reuse connections to an S3 server.15:58
rogpeppedimitern: i may be wrong - niemeyer will know why Close is true there.15:58
* rogpeppe wonders if there's any chance of go1.0.3 going into raring15:59
dimiternrogpeppe: you can, but up to 100 reqs on the same connection, according to the official docs15:59
dimiternrogpeppe: https://forums.aws.amazon.com/thread.jspa?threadID=9140215:59
dimiternrogpeppe: we should ask Daviey and/or arosales perhaps?16:00
* arosales reads backscroll16:00
dimiternarosales: basically what's the chance of including go 1.0.3 instead of 1.0.2 in raring?16:00
arosalesdimitern, that definitely an ubuntu dev uploader question16:02
arosalesdimitern, but what is the delta?16:02
rogpeppedimitern: so if we don't set Close, then it will die randomly after maybe 10 requests16:02
rogpeppearosales: do you mean how big is the difference between 1.0.2 and 1.0.3 ?16:02
arosalesrogpeppe, correct16:03
rogpeppearosales: there's no difference really apart from bugs fixed16:03
rogpeppearosales: that's not *strictly* true, but for our purposes it is16:04
arosalesrogpeppe, how big are the bug's patches?16:04
dimiternfwereade, rogpeppe: https://codereview.appspot.com/8648047 - raring fix16:04
rogpeppearosales: for this particular bug?16:04
arosalesrogpeppe, for the bug fixes between 1.0.2 and 1.0.316:05
arosalesrogpeppe, the main issue to the package will be the delta in the changes.16:05
dimiternarosales: and 1.0.3 has been mainstream for quite some time now (months)16:05
arosalesreason I am asking16:05
arosalesdimitern, gotcha I am trying to just grasp the package delta from .2 to .3 to give better input on the package upload question16:06
dimiternrogpeppe: can you prepare a delta easily?16:07
arosalesgiven an ubuntu dev with upload rights, such as Daviey, would need to weigh in. But I think he would have similar questions.16:07
niemeyerarosales: http://code.google.com/p/go/source/list?name=release-branch.go116:07
dimiternarosales: I assume you ask for the diff between 1.0.2 and 1.0.3. releases?16:07
rogpeppearosales: the delta in the go source tree between 1.0.2 and 1.0.3 is 22484 lines16:07
arosalesniemeyer, thanks16:07
arosalesdimitern, yes16:08
fwereadeniemeyer, that bucket is still yours, right?16:08
niemeyerfwereade: It is16:08
fwereadeniemeyer, because I think that if we just delete all the tools older than, say, 1.9.10 (a month ago) we will cut the XML down comfortably below 8k16:09
rogpeppearosales: well, that's the context diff anyway16:09
arosalesdimitern, mramm, do you know if Daviey had sponsored the upload yet?16:09
arosalesrogpeppe, gotcha16:09
niemeyerfwereade: XML?16:09
dimiternarosales: not really, no16:09
fwereadeniemeyer, and buy ourselves some breathing room without messing with either last-minute cowboy hacks to x3, or changing the platform we build on16:09
mrammarosales: not yet I don't think16:09
fwereadeniemeyer, it only started happening with the last release16:10
fwereadeniemeyer, AFAIWCT the relevant code has not changed16:10
niemeyerfwereade: Sorry, I'm out of context16:10
fwereadeniemeyer, but the response that gives us trouble got to ~8k at that point16:10
fwereadeniemeyer, ah sorry16:10
fwereadeniemeyer, when we  list the juju-dist bucket, we get this "connection closed" error when trying to decode the XML16:11
rogpeppeniemeyer: the XML in the LIST response16:12
arosalesmramm, ok so then it may not be as big of an issue to get .3 uploaded over .2. The next question would be stability/testing which I am guessing is better in .316:12
fwereadeniemeyer, and if we ReadAll to see what we get before trying to decode, we see that it cuts off suspiciously close to 8k16:12
rogpeppearosales: .3 has generally been used considerably more than .2 by my understanding16:12
niemeyerfwereade: I see16:13
fwereadeniemeyer, if we set Close: false on the request we do get all the data, but I'm not sure what other consequences might be lurking there16:13
dimiternfwereade: limit on s3 connection reuse, for one16:13
dimiternfwereade: https://forums.aws.amazon.com/thread.jspa?threadID=9140216:13
fwereadeniemeyer, IMO source hacks, and platform changes, are both much riskier and more potentially destabilizing than just trashing some old tools16:13
niemeyerarosales: Some of these patches will need to be yanked as well16:13
niemeyerarosales: The offending one, mainly16:14
niemeyerarosales: But possibly others16:14
niemeyerfwereade: I don't mind trashing the old tools, but I disagree with the overall principle16:14
niemeyerfwereade: This is putting smoke around the actual problem16:14
arosalesniemeyer, ok and that could be a SRU if needed too16:14
fwereadeniemeyer, I'm not holding this up as a good solution :)16:14
niemeyerarosales: Right, very much think so16:15
fwereadeniemeyer, I am proposing it as the least risky way to give ourselves the breathing space to resolve the actual problem16:15
niemeyerarosales: There's some useful background here too: http://code.google.com/p/go/issues/detail?id=491416:15
niemeyerarosales: Which describes how the bogus patch came to get into the package, and never leave for whatever reason16:15
niemeyerarosales: The Debian package is quite poorly maintained right now16:15
arosalesniemeyer, thanks for additional info16:16
niemeyerfwereade: Understood.. I'm saying it doesn't sound less risky16:16
niemeyerfwereade: Saying "we think it breaks around 8k so let's reduce the payload" is a total guess, and doesn't address or describe the real cause of the issue16:17
fwereadeniemeyer, we could be screwed tomorrow by s3 sending smaller chunks, you mean? or something more subtle?16:17
dimiternbump: https://codereview.appspot.com/8648047/16:17
niemeyerfwereade: This is the real bug: http://code.google.com/p/go/issues/detail?id=491416:17
niemeyerfwereade: If it's not addressed, the bug is still there16:17
fwereadeniemeyer, agreed16:18
fwereadeniemeyer, but I am pretty sure that switching go version is riskier, and hacking goamz is... *slightly* hackier16:19
fwereadeniemeyer, smarter solutions accepted with joy and gratitude, ofc16:20
rogpeppefwereade: i don't think switching go version is risky16:21
* fwereade raises an eyebrow16:22
rogpeppefwereade: we've always been testing with different go versions16:22
rogpeppefwereade: i'm pretty sure we're robust in that regard16:23
* dimitern everybody seems to be ignoring my fix, and i though we're in a hurry16:23
mrammrogpeppe: have we been testing with 1.0.3?16:26
niemeyerfwereade: I think not fixing the bug is riskier than fixing it16:26
mrammI thought we tested with 1.0.2 and tip mostly16:26
niemeyerfwereade: In either case, I'll remove the old tools as requester after lunch16:26
rogpeppemramm: i've been testing with 1.0.3 and tip interchangeably16:26
mrammcan we patch just that bug and release
dimiternmramm: i'm only using
mrammdimitern: rogpeppe: sounds like we are testing16:26
mrammgood deal16:27
dimiternthe only issue i had with 1.0.3. is the "redirect blocked" error with lbox/lpad16:27
fwereadedimitern, sent a couple of comments16:27
fwereaderogpeppe, I am concerned that just running the tests, and maybe a simple env or two, is not enough to say it's not risky16:28
rogpeppefwereade: i've bootstrapped --upload-tools and deployed etc16:28
fwereaderogpeppe, but perhaps I mischaracterise the effort you have been putting into his16:28
rogpeppefwereade: and also a lot on tip, which is considerably different again, and we work there fine16:29
fwereaderogpeppe, I just don't think that we have time to reasonably verify a change of that sort16:29
dimiternfwereade: i did test both cases - there was a raring specific test already, which i changed, and the other is non-raring specific16:29
fwereadedimitern, sweet, sorry16:29
fwereadedimitern, that's much nicer than I feared then16:30
mrammI think upgrading go in the archive is unlikely16:30
dimiternfwereade: surprisingly, me too :)16:30
mrammgiven timeline16:30
fwereadedimitern, ok, LGTM with trivial rearrangement of apt-related settings16:31
dimiternfwereade: cheers16:31
mrammupgrading juju after feature-freeze is one thing16:31
mrammit is an applicaiton16:31
fwereademramm, regardless of timeline I have never upgraded a framework version, let alone a language version, without encountering... surprises16:31
mrammbut tools like go really *should* be frozen16:31
mrammfwereade: understood16:32
dimiternmramm: it's not like any other project in the archive is using go, right?16:33
mrammdimitern: I don't know16:33
dimiternmramm: and the users would likely want the latest stable go version, which was released 12 sept 201216:34
mrammdimitern: if we wanted to do that, it would have been good16:34
dimiternmramm: it can be checked trivially by finding packages that depend on it16:34
mrammdimitern: but doing it after feature freeze, and then after final freeze -- that is not the time16:34
dimiternrogpeppe: can you take a look too please? https://codereview.appspot.com/8648047/16:35
fwereademramm, dimitern, niemeyer, rogpeppe, et al: I believe the safest path is to stick with 1.0.2, paper over the problems, and get onto 1.0.3 as soon as we can after the release, so we can get an update that works properly into our users' hands ASAP after that. I am well aware that if this fucks up, it is on my head16:35
mrammI agree that those processes are there to serve users, so if nobody else uses go from the archive.....16:35
mrammdimitern: also it is not just about packages in the archive, it's about applications our users build with the "released version" of go16:36
fwereadebut I have done the last-minute-cowboy thing in the past, and it has not had the success ratio I might have hoped for16:36
* mramm admits that most go users are probably not using a packaged version of go16:36
rogpeppefwereade: tbh i think it's ridiculous that raring isn't shipping with the latest version of go anyway16:37
dimiternmramm: yeah, my point was the users will likely want a better version of go with more fixes (if they're not already using it by manually upgrading the one in the archive)16:37
fwereadeso I do not believe that I can in good conscience approve this change16:37
fwereaderogpeppe, agreed, but IMO orthogonal16:37
mrammdimitern: I doubt that matters to many users16:37
dimiternwe're pulling straws here..16:37
mrammrogpeppe: that is very true, but we can't fix that anymore16:37
dimitern("grasping at" apparently) :)16:38
rogpeppedimitern: i agree with you16:39
rogpeppedimitern: and i think that juju has had so little live testing that it makes no significant difference at this stage16:40
rogpeppefwereade: ^16:40
dimiternrogpeppe: +1016:40
rogpeppefwereade: we're gonna hoof loads of bugs out anyway16:40
rogpeppefwereade: and at least we won't be doing it against an old and buggy version of Go16:40
niemeyerfwereade: I don't think it's orthogonal.. the version of Go in the archive is *broken*16:44
niemeyerfwereade: and juju is being directly affected by it16:45
niemeyerfwereade: Fixing this isn't cowboying.. it's using the freeze process for what it's meant to be used for16:45
fwereaderogpeppe, niemeyer: I cannot recall a single case in which a significant framework or library upgrade has been free of unpleasant surprises16:46
fwereaderogpeppe, niemeyer: my experience overwhelmingly directs me to do these just *after* a release, not just *before*16:46
niemeyerIt's so ironic that I'm involved in fixing this now, when my request to be the package maintainer in Ubuntu was declined16:46
niemeyerfwereade: I can recall many of those16:47
rogpeppefwereade: we have already tested against this upgrade many times16:47
niemeyerfwereade: In the case of minor release updates16:47
niemeyerfwereade: Patch release updates, in fact16:47
niemeyerfwereade: Happens all the time in Ubuntu16:47
niemeyerfwereade: It's 1.0.2 to 1.0.3.. it's not 1.0 to 1.1, or 1.0 to 2.016:48
fwereadeniemeyer, maybe I do the golang guys a disservice -- I probably do -- but even if I had perfect knowledge of the consequences, I'm concerned that it's impractical16:50
niemeyerfwereade: I can't address your psychological feelings I'm afraid16:51
fwereadeniemeyer, and, sure, it happens all the time in ubuntu; but the professionally paranoid stick to LTSs for that reason16:51
niemeyerfwereade: It happens in LTSs as well16:53
niemeyerfwereade: That's why LTSs exist, in fact16:53
fwereadeof a series of unappealing options, I choose the one that least perturbs all the other things we don't know we depend upon16:54
fwereadeand I must now be away, lest marital strife come to pass16:54
fwereadeI'm sorry to disappoint16:54
rogpeppefwereade: what's your solution?16:55
niemeyerI have no idea, but apparently he doesn't want to see the real bug fixed16:55
fwereaderogpeppe, fix it ASAP *after* the release16:55
fwereadeniemeyer, please do not mischaracterise my position like that16:55
dimiternrogpeppe: ping16:55
niemeyerfwereade: I'm not, by all means16:55
rogpeppefwereade: what do you think is the worst can happen?16:56
niemeyerfwereade: I want to see the bug fixed in raring16:56
rogpeppedimitern: pong16:56
niemeyerfwereade: You're suggesting we don't do that16:56
niemeyerfwereade: It's as simple as that, I think16:56
dimiternrogpeppe: sorry being a pest - https://codereview.appspot.com/16:56
dimiternrogpeppe: *for being*16:56
fwereadeniemeyer, I don't think it's remotely acceptable to swap out the go version *after* final freeze16:56
fwereadeniemeyer, if it wasn't a big enough deal before, it's surely not now16:57
rogpeppedimitern: is there a CL number i should be interested in?16:57
niemeyerfwereade: What's final freeze for, if we can't fix real bugs in that period?16:57
niemeyerfwereade: Huh.. okay16:57
dimiternrogpeppe:  :) oops - https://codereview.appspot.com/8648047/16:57
* niemeyer moves on to other things then16:57
fwereadeniemeyer, I *wish* we had encountered this 2 weeks ago16:57
fwereadeniemeyer, but we did not16:58
fwereadeniemeyer, thank you for your forbearance16:58
rogpeppefwereade: so what test are you worried might fail?16:58
rogpeppefwereade: if our live tests pass, we can deploy and add relations etc, all with go1.0.3, what's not tested there that we have tested with the other stuff?16:59
niemeyerrogpeppe: Should we mention that we actually use tip?16:59
* niemeyer ducks16:59
rogpeppeniemeyer: i just might have already mentioned that a few times16:59
rogpeppefwereade: tip is hundred times as different as 1.0.3 from 1.0.2 and it works fine16:59
rogpeppefwereade: i really think your paranoia is unwarranted here17:00
rogpeppefwereade: because noone is going to use this version in production *anyway*17:00
rogpeppedimitern: LGTM assuming live tests pass against raring17:03
dimiternrogpeppe: yeah, tested 3 times17:03
dimiternrogpeppe: with default-series: raring17:05
rogpeppedimitern: cool17:05
TheMuedimitern: great17:05
dimiternTheMue: did you manage to fix your ssh issues?17:09
TheMuedimitern: i'm in progress, cleaning up a bit during this evening.17:10
* dimitern this is ridiculous! i have to reboot..17:13
TheMuedimitern: can't believe *lol*17:13
dimiternTheMue: for the past 2-3h whatever I do the machine freezes for a second about every 5s or so17:14
rogpepperight, i'm done here17:15
dimiternwell, not everywhere, but mostly in emacs and terminal17:15
rogpeppesee y'all monday17:15
rogpeppesunny evening, yay!17:15
dimiternrogpeppe: have a good weekend!17:15
rogpeppedimitern: and you17:15
TheMuerogpeppe: have a nice weekend17:15
rogpeppeTheMue: and you too17:15
rogpeppefwereade: and you also17:15
TheMuerogpeppe: thx17:15
dimiternyeah.. i'm off as well17:16
dimiternsee you guys and take care17:16
TheMuedimitern: have a nice weekend too17:16
dimiternTheMue: same to you :)17:16
TheMuedimitern: thx17:16
mramm2internet here is pretty terrible18:09
mramm2so IRC is totally unreliable18:09
=== wedgwood is now known as wedgwood_away
=== wedgwood_away is now known as wedgwood
=== wedgwood is now known as wedgwood_away

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!