=== TheMue_ is now known as TheMue [06:53] fwereade_, TheMue: morning [06:53] heya wrtp [06:53] wrtp, fwereade_ : moin [06:53] fwereade_: i'd quite like a chat about deploying Go juju some time [06:55] wrtp, sounds good; 5 mins? [06:55] fwereade_: cool [07:00] wrtp, invite out [07:01] fwereade_: ah, actually i can't do a verbal chat now, 'cos i'll wake up the sleeping angel [07:02] wrtp, np, ready all the same :) [07:02] wrtp, (you mean I got dressed for nothing? :p) [07:02] fwereade_: lol [07:03] fwereade_: so... [07:03] fwereade_: as far as i can see we've got two different problems (or maybe three) [07:03] fwereade_: 1) how do we deploy an appropriate binary at development time [07:04] fwereade_: 2) how do we deploy binaries that aren't for the platform we're currently on [07:04] 3) (?) how do we make sure that we're deploying the right version? [07:05] wrtp, yeah, I think all 3 are real, but niemeyer's plan seems to cover (3) pretty well [07:06] 1) is difficult unless we can assume that we're only testing on a compatible platform to the dev platform, which maybe isn't a bad assumption [07:06] fwereade_: remind me of niemeyer's plan again [07:06] wrtp, I think that assumption is the right one for dev mode at least to start out [07:06] wrtp, the versioning thread I was being obtuse in yesterday [07:07] wrtp, not sure I can summarise it very well [07:07] fwereade_: on the mailing list? [07:07] wrtp, yeah [07:08] fwereade_: ah, i totally missed that; "Bootstrap scheme for Go port" [07:08] wrtp, but basially the initial email covers it, with the addition of the odd-number-anywhere-denotes-dev-version bit [07:09] * wrtp is reading [07:12] fwereade_: by "semantic versioning" he's referring to this: http://semver.org/ ? [07:12] wrtp, yes [07:13] wrtp, the odd-number dev versions break what's suggested there AFAICT, but it's pretty clear all the same [07:24] fwereade_: isn't the odd-number proposal using odd numbers for the same kind of thing that "pre-release" versions are used in the original sem. ver. spec? [07:24] wrtp, it is indeed, so it's pretty clearly comprehensible [07:25] fwereade_: so... why not just use prerelease versions so we'll be fully semver compatible? [07:25] wrtp, but it's still an exception that is not obvious if you just say "we're using semver" [07:25] wrtp, well, say we've released 4.0.0 [07:26] wrtp, for testing major upgrades of dev versions sanely we'll want to be using 5.x.x [07:27] fwereade_: 5.x.x-alpha.0 ? [07:27] wrtp, but assuming 2 breaking changes in a dev cycle, we don't want to make users go 4->8 next release, it would be weird [07:27] fwereade_: i don't see the problem. we can do 5.x.x-alpha.0, 5.x.x-alpha.1 etc etc [07:28] 5.0.0-alpha.0, 5.0.0-alpha.1 actually [07:28] wrtp, "Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API. It MAY include minor and patch level changes. Patch and minor version MUST be reset to 0 when major version is incremented." [07:29] wrtp, I think we break it one way or another whatever we do [07:30] fwereade_: i'm not sure that pre-release versions need to be strictly compatible in the same way [07:31] wrtp, but if we're using the version for upgrade/compatibility logic we need to be able to have sane versions when we're testing that we can actually upgrade, don't we? [07:31] wrtp, it's really just extending the definition of pre-release [07:33] wrtp, maybe I'm wrong and the existing provisions satisfy it [07:33] fwereade_: am just trying to sort it out in my head :-) [07:33] wrtp, I just recoil a little from worrying about them in the context of the upgrade logic [07:37] fwereade_: i still think i don't see the problem. we can use exactly the same logic that's described on the semver page. [07:38] fwereade_: but i'd suggest that in "development mode" the "pre-release" logic switches to giving pre-release a higher precedence than the normal version. [07:39] fwereade_: as it is, the comparison logic with odd numbers will be weird. [07:39] s/as it is/as proposed/ [07:39] wrtp, the core of the problem is that we can't signal a backwards-incompatible change between two differentr pre-release versions with the same major version... can we? [07:40] wrtp, how's it weird? I *think* the logic will be just the same [07:40] wrtp, we just don't release odd-numbered versions to the canonical location [07:41] fwereade_: ah, i'd missed that twist [07:42] fwereade_: i'm not sure why we'd need to signal a backwards-incompatible change between two different pre-release versions [07:43] fwereade_: we don't care about breaking dev deployments [07:43] wrtp, yeah, but to fulfil semver we ought to [07:44] wrtp, ...right? [07:44] wrtp, I don't see it carving out an incompatibility exception for pre-release versions, but maybe I'm just blind [07:45] fwereade_: i'm not sure. it doesn't talk about prerelease versions much. and why would you have a prerelease version if it wasn't to find problems and fix them *before* releasing the one that does it "right". [07:45] ? [07:45] i think if we use pre-release versions in this way, we don't need two repositories and we can just use the normal semver logic [07:47] * fwereade_ thinks [07:47] in fact, we don't even need a "development mode" switch, i realise, because we can just up the client version to, say 5.0.0, which is satisfied by 5.0.0-alpha1 until 5.0.0 real version is released. [07:48] fwereade_: semver doesn't talk about any compatibility requirements *between* pre-release versions AFAICS [07:49] wrtp, hm, don't we need a dev repo for that all the same? otherwise we have to have a speculative version bump every time we start a new cycle [07:49] fwereade_: that sounds right to me [07:49] wrtp, the speculative version bump? [07:50] fwereade_: yeah [07:50] wrtp, not convinced [07:50] fwereade_: but if it turns out the version bump isn't needed, we can release as a lower version. [07:50] wrtp, I think we should be shooting for releases that *are* backward compatible in general [07:50] fwereade_: definitely [07:50] wrtp, that feels like it *also* breaks semver [07:51] hmm, yeah, probably. [07:52] fwereade_: but won't we have to increment speculatively with the odd-numbering thing too? [07:52] wrtp, hmm [07:53] wrtp, yeah :( [07:54] fwereade_: i don't think i mind the speculative versioning. i think it maps quite well to what we're actually doing - building a version that is planned to become a real version at some point. [07:54] fwereade_: and if we decide the version is wrong, we can delete the wrongly-speculatively-versioned prerelease versions [07:55] wrtp, wait a mo, with the odd-numbered-versions we *don't* have to [07:56] wrtp, release 4.0.0; dev work starts on 4.0.1; we release 4.0.2 from latest state of 4.0.1 when we're ready [07:56] fwereade_: no? [07:56] wrtp, we only have to bump the major version at the point we introduce something that actually is incompatible [07:57] fwereade_: how's that different from: release 4.0.0; dev work starts on 4.0.1-alpha.0; we release 4.0.1 from latest pre-release version of 4.0.1 ? [07:59] wrtp, ok, I think you're right :) [07:59] wrtp, don't think we can avoid the separate repos though [07:59] fwereade_: why's that? [08:01] wrtp, because 4.0.1-alpha.0 will appear to be a valid version for 4.0.0 to deploy on the server [08:01] wrtp, I think we may be making too much of this really -- the public repo should only have the stuff we've actually released [08:02] wrtp, so we need private repos when we're testing [08:02] fwereade_: that's probably true [08:02] fwereade_: so the public repo is the dev repo without the prerelease versions [08:03] fwereade_: alternatively the comparison logic could be configurable to make it not consider pre-release versions. [08:03] wrtp, yeah, that just crossed my mind too [08:05] wrtp, ok, tell you what, we can get around all this by spinning up parallel universes to do dev work in and just merge the timelines later [08:05] wrtp, I'll go find a physicist ;) [08:05] SGTM [08:06] fwereade_: might get a few weird effects when we get conflicts tho' [08:06] wrtp, more seriously, I hadn't really been thinking about a central dev repo [08:06] wrtp, details details ;) [08:07] wrtp, as devs the only versions we care about are released ones and whatever we're working on at the moment [08:07] wrtp, there may be a preview repo somewhere for other people to test against, with the no-backward-compatibility-guarantees exception [08:08] fwereade_: there's another issue if devs are sharing the same repo: you'll have to explicitly set your client version number to your "own" pre-release version, otherwise you'll get someone else's (unless yours happens to compare greater than others') [08:08] wrtp, yeah, indeed, hence my private-repo assumptions [08:09] fwereade_: i think it could work ok though. [08:09] fwereade_: even with a public repo [08:10] fwereade_: we could accept an environment variable "JUJU_PRERELEASE" which would give the prerelease version that you're working on. [08:10] e.g. JUJU-PRERELEASE=rog.0 [08:11] wrtp, I'd really prefer to have the repo location be the only moving part [08:12] fwereade_: yeah, you're probably right. [08:39] re [11:15] fwereade_: http://gopkgdoc.appspot.com/pkg/launchpad.net/~rogpeppe/+junk/version [11:19] fwereade_: that implements the "must be at latest version to use pre-release versions" rule, as well as the rest of the semantic versioning rules as far as i could make them out. [11:56] wrtp, that seems pretty neat really [11:56] wrtp, let's see what niemeyer thinks :) [11:59] fwereade_: cool. yeah, let's see. [12:27] fwereade_: oh yes, i forgot, gustavo's not around today [12:49] wrtp: Could you please add more doc for the structs fields? Especially the semantics of prerelease and build and how they are handled in Less(). [13:09] fwereade_: time for a talk about "force upgrade" for units? [13:09] TheMue, ofc :) [13:09] TheMue, I'm no expert but I'll help if I can [13:10] TheMue, re wrtp's stuff, http://semver.org/ shoudl cover most of it [13:10] fwereade_: The current watcher implementation only observes if the node in ZK is created. And you wrote a comment that this isn't enough. [13:11] fwereade_: Oh, after a quick look an interesting doc. Thx, will read it later. [13:12] TheMue, juju/agents/unit.py:197 is probably a good place to start looking [13:12] fwereade_: Will take a look. [13:12] TheMue, basically people want to be able to upgrade broken units [13:12] TheMue, and the default implementation only does an upgrade if it's in the "running" state [13:15] TheMue: what fwereade_ says. [13:16] fwereade_: IC, so there already has been an initial error implementing NeedsUpgrade() and SetNeedsUpgrade() without viewing the node content [13:16] TheMue: i could probably describe the correspondence of the fields with the version components as described at semver.org, i guess [13:16] TheMue, I'm not sure it was an error at the time, the --force flag is relatively recent [13:16] fwereade_: Is this a newer change or already longer in code? [13:17] fwereade_: Ah, ic, this is an explain (you say so?). [13:17] TheMue, sorry, cannot parse [13:17] TheMue: it was a very quick coding job this morning in response to a discussion with fwereade_ - it's not fully documented yet... [13:18] wrtp: I only wanted to understand those two fields better, but the doc mentioned above already looks good. [13:18] TheMue: the interaction between prerelease and build doesn't really describe the interaction between those two fields, unfortunately. [13:18] fwereade_: Argh, yes, hadn't the word. Should read: this is an explanation. [13:19] fwereade_: So I will handle the content now too. [13:21] TheMue, cool, thanks (it was merged less than a month ago, so it was before I switched back to python, and I'm sure I remember seeing the first upgrade stuff in go while I was still on go) [13:25] fwereade_: I've got to thank you for the hint. Otherwise it would have been wrong from the beginning. [13:25] TheMue, I think this is one of the biggest risks we face tbh [13:26] TheMue, and being the only person on the team with semi-current knowledge of the python makes me a little nervous, but I'll do my best ;) [13:27] fwereade_: Your the official Knowledge Transfer Agent. *lol* [13:27] haha :) [14:01] fwereade_: do you have a preference for what kind of provider storage we should use for juju binaries? [14:02] fwereade_: perhaps we should just use S3 on EC2 [14:02] wrtp, not really; I know there's been quiet talk of decoupling storage provider from machine provider but that feels like a distraction at this point [14:02] wrtp, so, yeah, whatever storage we already have for the appropriate provider [14:11] fwereade_: i wonder if Environ.Bootstrap should take the client version as an argument, or if the environs package should fetch the version number from somewhere on its own. [14:12] fwereade_: my inclination is towards the former [14:12] wrtp, likewise, I think [14:12] wrtp, this may not be python but explicit still beats implicit in general ;) [14:15] fwereade_: cool. and then i guess i'd store the location of the binaries in the state, so agents can pull them out using Environ.GetFile [14:17] wrtp, ...I *think* so but most of my cycles are still concentrated on figuring out what the hell I was oing a months ago in the followup branches I hadn't proposed yet [14:17] fwereade_: :-) [14:17] fwereade_: that can be educational... [14:17] wrtp, also on not being able to type, it seems [15:28] fwereade_: what's the most universally available command-line tool for downloading from a web server? wget? (i realise that the code to download the binaries from S3 cannot be written in Go itself... doh!) [15:30] wrtp, ha, I guess so :) [15:30] wrtp, but really you can install what you want in the cloud-init, can't you? [15:30] fwereade_: how do i get the cloud-init to install a binary from S3? [15:31] wrtp, (not that I'm advocating a separate juju-getter package or anything :p) [15:31] fwereade_: sure, i could apt-get another package. but if a couple of lines of shell will do it, i think that'd be better. [15:31] wrtp, you can run arbitrary scripts, so... however you want ;p [15:32] wrtp, indeed [15:32] wrtp, wget a signed URL seems sensible [15:32] fwereade_: you mean a URL containing signed content? [15:35] wrtp, I was just thinking something like juju/providers/ec2/files.py:40 [15:35] fwereade_: yeah, i was planning to provide that [15:36] fwereade_: another question: how can i tell what tools are in a basic ubuntu release? (for instance, i'm wondering if unzip is there by default. the less apt-gets the better, i think) [15:36] wrtp, hmm, I have no idea I'm afraid :( [15:36] fwereade_: guess i'll just try it and see :-) [15:37] wrtp, sounds good :) [15:38] fwereade_: i *thought* the useful thing about zip vs tar is that it's supported within Go. but tar is also supported, so i think that's probably the better option. [15:39] fwereade_: and tar and gzip will most definitely be part of the stock distro [15:39] wrtp, so I would assume ;) [15:40] fwereade_: they darn well should be [15:42] fwereade_: i guess we should probably sign the content too. [15:43] wrtp, probably sensible [15:47] fwereade_: yet another thing: have you got an opinion on where we should put the binaries? [15:47] wrtp, not really :) [15:48] fwereade_: /tmp then :-) [16:30] fwereade_, TheMue: https://codereview.appspot.com/6081044 [16:32] wrtp, LGTM [17:14] fwereade, TheMue: https://codereview.appspot.com/6082044 [17:16] i should probably change it to use gocheck though, i suppose [17:17] fwereade, TheMue: right, that's me for the week. see ya monday. have a great weekend! [17:18] wrtp: Have a nice weekend. gocheck idea is good. I'm already off, only looking at tjose two last proposals. [17:20] TheMue: one reason for not using gocheck is that it's a highly independent package. i quite like not depending on very much, and the testing package works fine here. [17:22] wrtp: Yes, you don't need it. It's only that a different maintainer later may ask himself why testing here is different than for other packages. [17:23] TheMue: the tomb package doesn't use gocheck, for example [17:23] wrtp: It's an external package, not inside the project. [17:24] TheMue: yeah. this might become external too. [17:24] wrtp: But indeed, you don't need it so far. [17:24] wrtp: In this case it's ok. [17:25] wrtp, basically LGTM, don't worry about it until monday :) [17:47] fwereade: minor versions can add features [18:24] So, last proposal done, have a nice weekend.