| jpds | Anyone awake? | 05:41 |
|---|---|---|
| axw | jpds: I am awake, something I can help with? | 05:56 |
| achandra | hello. I appear to be having the toughest time trying to connect to my swift object-store on devstack via juju | 06:05 |
| achandra | ive pulled up the url of the object-store via keystone catalog | 06:05 |
| achandra | however, once i create the juju-dist and upload the meta-data | 06:06 |
| achandra | i still get an un-authorised connection | 06:06 |
| achandra | ideas? | 06:06 |
| rogpeppe | mornin' all | 07:16 |
| rogpeppe | hmm, i left juju running the local provider over friday and saturday and ended up with a 2.1GB log file | 07:18 |
| rogpeppe | ha, it's all my fault | 07:20 |
| rogpeppe | it panicked 47717 times before i killed it yesterday | 07:21 |
| axw | morning rogpeppe | 07:22 |
| axw | heh :) | 07:22 |
| rogpeppe | axw: hiya | 07:22 |
| rogpeppe | axw: i was just glancing at https://codereview.appspot.com/14465045/ | 07:26 |
| rogpeppe | axw: where are the sftp urls coming from? | 07:26 |
| axw | rogpeppe: environs/sshstorage | 07:26 |
| axw | its URL method returns sftp:// URLs | 07:26 |
| rogpeppe | axw: ah, because the URLs were only expected to be used from a shell script? | 07:26 |
| axw | rogpeppe: I was expecting them to not be used at all (or only for logging) | 07:26 |
| rogpeppe | axw: ah, i see | 07:26 |
| axw | but yeah, they get used by wget or http.Get in some cases | 07:26 |
| rogpeppe | axw: also, not quite sure what you mean by "problematic for the trunk implementation" | 07:26 |
| rogpeppe | axw: do you just mean the trunk branch of juju? | 07:26 |
| axw | rogpeppe: yeah, sorry, badly worded | 07:26 |
| axw | I just mean that the current code is broken | 07:26 |
| axw | and this change fixes it by wrapping it in httpstorage | 07:26 |
| rogpeppe | axw: np, i *thought* so, but i wasn't sure | 07:26 |
| axw | thus hiding the invalid URL method away | 07:26 |
| rogpeppe | axw: right, now i think i have enough info to review the change properly :-) | 07:27 |
| axw | :) thanks | 07:27 |
| rogpeppe | axw: hmm, i'm getting a chunk mismatch error | 07:28 |
| rogpeppe | axw: could you repropose please? | 07:28 |
| axw | yup, just a moment | 07:28 |
| axw | rogpeppe: do you think it'd be worthwhile using gotype instead of "go build" in .lbox.check? might speed things up a bit ... can you think of any downsides? | 07:34 |
| axw | maybe it's just my crappy laptop's the problem | 07:34 |
| rogpeppe | axw: is there a gotype command? | 07:34 |
| axw | (repropose is still chugging along) | 07:34 |
| axw | rogpeppe: yeah, somewhere in go.tools | 07:34 |
| rogpeppe | axw: you're running go tip then, right? | 07:35 |
| axw | code.google.com/p/go.tools/cmd/gotype | 07:35 |
| axw | rogpeppe: nope, why? | 07:35 |
| axw | go/types works on 1.1 | 07:35 |
| axw | it didn't for a bit, but rsc fixed that | 07:35 |
| rogpeppe | axw: hmm, what go version are you using? | 07:35 |
| axw | rogpeppe: 1.1.1 | 07:35 |
| rogpeppe | axw: axw: i'm pretty sure it's not go build that takes all the time | 07:35 |
| rogpeppe | axw: hmm | 07:35 |
| rogpeppe | axw: on my machine it's go vet that takes forever | 07:36 |
| axw | rogpeppe: I didn't even verify. okay, probably not worthwhile then | 07:36 |
| rogpeppe | axw: but... | 07:37 |
| rogpeppe | axw: i thought that was only since go vet became external (since go 1.1.1) and started using go types | 07:37 |
| rogpeppe | axw: it takes about 30s to run on my machine | 07:37 |
| axw | rogpeppe: minutes here, but I have a crappy laptop | 07:38 |
| axw | rogpeppe: propose finished | 07:38 |
| rogpeppe | axw: i mean "go vet" takes 30s | 07:38 |
| rogpeppe | axw: propose can take minutes for me too | 07:38 |
| axw | hmmk | 07:38 |
| axw | I suppose it doesn't matter too much, forces me to think a bit harder before proposing junk ;) | 07:39 |
| rogpeppe | axw: i really wish it was faster | 07:44 |
| rogpeppe | axw: BTW i'm *still* seeing a chunk mismatch :-| | 07:44 |
| axw | ack | 07:45 |
| axw | :( | 07:45 |
| axw | hmm | 07:45 |
| TheMue | morning | 07:45 |
| axw | rogpeppe: where does this chunk mismatch show up? | 07:45 |
| axw | morning TheMue | 07:46 |
| rogpeppe | axw: https://codereview.appspot.com/14465045/diff/1/provider/null/environ.go | 07:46 |
| rogpeppe | axw: one mo, maybe i didn't reload the main CL page | 07:46 |
| rogpeppe | axw: ignore me | 07:46 |
| axw | rogpeppe: I think so | 07:46 |
| axw | nps | 07:46 |
| rogpeppe | axw: it works fine | 07:46 |
| rogpeppe | axw: does anything use sshstorage other than the null provider? | 07:47 |
| axw | rogpeppe: nope | 07:47 |
| axw | only the null provider, only for bootstrap | 07:47 |
| rogpeppe | axw: oh yes, one thing that's been niggling at me for a while: how do you ensure that the bootstrap storage object is eventually closed? | 07:51 |
| rogpeppe | axw: as doesn't it contain a live ssh connection? | 07:51 |
| axw | rogpeppe: you don't/can't at the moment | 07:51 |
| axw | yep | 07:51 |
| rogpeppe | axw: ah, so it just leaks | 07:52 |
| axw | rogpeppe: yes, but this is only ever done from the command line | 07:52 |
| rogpeppe | axw: currently... | 07:52 |
| axw | hmm | 07:53 |
| rogpeppe | axw: we shouldn't assume that though - it's ok for people to use this stuff directly from Go. | 07:53 |
| axw | rogpeppe: if this is moved to something non-interactive, I think we'll have bigger problems | 07:53 |
| axw | like, how to input sudo passwords | 07:53 |
| rogpeppe | axw: ha, yes | 07:53 |
| axw | but yeah I understand | 07:53 |
| rogpeppe | axw: although that's not a problem if it's already running as root, i guess | 07:53 |
| axw | true | 07:54 |
| axw | actually | 07:54 |
| axw | not true | 07:54 |
| axw | it's the target machine that matters | 07:54 |
| axw | and you typically wouldn't allow root ssh | 07:54 |
| rogpeppe | axw: how does the target machine know you're running as root? | 07:55 |
| axw | rogpeppe: anyway, I guess this would be solved by adding a Close method to Environ | 07:55 |
| axw | rogpeppe: ? | 07:55 |
| rogpeppe | axw: yes, but that's actually a very big change | 07:55 |
| axw | rogpeppe: null provider bootstrap takes two parameters: host, and login user | 07:56 |
| rogpeppe | axw: well, actually, it would probably need a close method on both Environ and Storage | 07:56 |
| axw | rogpeppe: login user is the current user on the source machine by default, but you'd probably need to change that to something non-root | 07:56 |
| rogpeppe | axw: sure | 07:57 |
| rogpeppe | axw: we could probably use a comment in the code just to let people know that this has been thought about | 07:59 |
| axw | rogpeppe: okey dokey, will add one | 08:00 |
| mattyw | morning folks, any reason why the maximum length of a summary using lbox is so low? | 08:05 |
| rogpeppe | mattyw: i think it's an lp restriction | 08:06 |
| rogpeppe | mattyw: ridiculous, ain't it? | 08:06 |
| rogpeppe | axw: just to clarify: what code is actually using the storage URL? | 08:07 |
| axw | rogpeppe: I didn't follow all the way to the end, but it's related to generating simplestreams metadata | 08:08 |
| axw | if the tools exist on the target, but the metadata doesn't, it grabs the tools off the target and generates the metadata from them | 08:08 |
| rogpeppe | axw: ah! so it uses the URLs inside the metadata | 08:09 |
| axw | rogpeppe: it uses them to obtain the tools, to generate the metadata | 08:10 |
| rogpeppe | axw: probably the other way around, no? it doesn't need to obtain the tools to generate the metadata, but the metadata will contain URLs which need to point to the tools | 08:10 |
| rogpeppe | axw: (but i haven't looked at the code...) | 08:10 |
| axw | rogpeppe: I think the juju CLI is fetching them off the target, generating the metadata at the CLI end, then uploading the metadata | 08:11 |
| axw | because there's nothing running on the target necessarily | 08:11 |
| axw | (in this case there isn't) | 08:11 |
| axw | the metadata includes information about the tools like file size and SHA256 sums | 08:11 |
| axw | so the tools need to be obtained to generate it | 08:11 |
| rogpeppe | axw: hmm, so how can tools exist on the target without metadata? i thought we'd moved away from legacy tools storage... | 08:14 |
| axw | rogpeppe: I could only surmise that it was a botched bootstrap | 08:15 |
| axw | it got part way through downloading the tools, but didn't get as far as generating/uploading the metadata | 08:16 |
| axw | rogpeppe: then a second bootstrap found the tools on the remote machine, but no metadata | 08:16 |
| axw | so it decided to fetch them and generate the metadata then (and failed) | 08:16 |
| rogpeppe | axw: hmm | 08:16 |
| rogpeppe | axw: perhaps that's the logic that needs to be fixed | 08:17 |
| axw | rogpeppe: perhaps, though I think it's prudent to hide invalid URLs away anyway | 08:17 |
| axw | something else may come along wanting to use it | 08:18 |
| rogpeppe | axw: well, the difficulty i have with this is that URLs are supposed to be usable from other machines | 08:18 |
| axw | though TBH, it's still flawed - they're only valid for the lifetime of the current process | 08:18 |
| rogpeppe | axw: and this only makes sense if the URL is only used locally | 08:18 |
| * axw nods | 08:18 | |
| mattyw | fwereade, I'm about to set that mp as needing review, are there any bugs or blueprints I should link it to? | 08:37 |
| fwereade | mattyw, nothing for that work specifically -- IIRC there's an "identity" blueprint somewhere that the followups will probably apply to though | 08:38 |
| fwereade | mattyw, if you'd like to capture your requirements as a bug and link that it can't hurt though | 08:39 |
| fwereade | mattyw, always nice to be able to look back and figure out why we write the code in the first place ;) | 08:39 |
| mattyw | fwereade, ok will do | 08:40 |
| rogpeppe | axw: i'm having difficulty finding the logic that fetches the tools off the target, generating the metadata at the CLI end | 08:41 |
| rogpeppe | axw: istm that to do that it would need to List the storage, but there are very few places that do that | 08:42 |
| axw | rogpeppe: I'll have a hunt | 08:43 |
| axw | rogpeppe: in the mean time, https://codereview.appspot.com/14483043 | 08:43 |
| rogpeppe | axw: argh, darn chunk mismatch again | 08:44 |
| axw | ah wtf | 08:44 |
| axw | rogpeppe: the copying/listing/etc. is in environs/sync | 08:45 |
| axw | brb | 08:45 |
| mattyw | fwereade, https://codereview.appspot.com/14389043/ thanks :) | 08:51 |
| axw | rogpeppe: in environs/tools/simplestreams.go, generateMetadata | 08:51 |
| axw | " if fetch && t.Size == 0 {" | 08:51 |
| rogpeppe | axw: yeah, i'm seeing it now | 08:52 |
| rogpeppe | axw: in sync.SyncTools | 08:52 |
| fwereade | mattyw, tyvm | 08:53 |
| mattyw | fwereade, no, thank you | 08:53 |
| axw | rogpeppe: ideally that code should use the Storage | 08:53 |
| axw | rogpeppe: but that would require a means of getting from a URL back to a storage name | 08:53 |
| rogpeppe | axw: hmm, yeah | 08:54 |
| rogpeppe | axw: unless a tools.List contained an expanded version on Tools which also held the original name, i guess | 08:55 |
| axw | yeah, or that | 08:56 |
| axw | rogpeppe: I reproposed that CL | 08:57 |
| * fwereade breakfast | 08:57 | |
| rogpeppe | axw: istm that part of the problem is that we're in a half-way house between two schemes - we've got two forms of metadata (the simple streams stuff and the storage list) and the latter doesn't have enough information to fulfil the former's requirements | 09:11 |
| rogpeppe | axw: that is, the real problem here is that toosl.ReadList isn't returning information on the hashes & sizes of the contents of the storage | 09:12 |
| axw | rogpeppe: yeah, because that would incur quite a lot of overhead for the usual case where the metadata already exists | 09:13 |
| axw | but yes, there's information loss there | 09:13 |
| axw | the storage names in particular | 09:14 |
| rogpeppe | axw: what would happen if we just passed in fetch==false ? | 09:18 |
| axw | rogpeppe: the metadata would be written out with Size=0, and no hash. I *think* jujud gets unhappy about that | 09:20 |
| rogpeppe | axw: that's what i was wondering | 09:20 |
| rogpeppe | axw: it's good to think through stuff in this area, i think, especially because we want to solve lp:#1219582 | 09:23 |
| _mup_ | Bug #1219582: bootstrap is slow, lookups tools multiple times <canonistack> <juju-core:Triaged> <https://launchpad.net/bugs/1219582> | 09:23 |
| rogpeppe | it's kind of ironic that s3 does actually provide hash and size info, although i think it's only the md5 hash | 09:25 |
| rogpeppe | axw: does it actually make a difference if the metadata exists on the target machine? | 09:30 |
| rogpeppe | axw: it looks to me as if the logic in SyncTools ignores any metadata in the target | 09:31 |
| axw_ | rogpeppe: it ignores it for copying the tools over, but it's used to decide which tools tarball to install into an instance... not sure what happens if it's not there | 09:32 |
| rogpeppe | axw: which means that we'll always read all the tools in both the target and source storage; i may have missed something though. | 09:32 |
| * axw_ tries | 09:32 | |
| === axw_ is now known as axw | ||
| axw | rogpeppe: that doesn't sound right, let me check | 09:32 |
| mattyw | fwereade, ping? | 09:35 |
| axw | rogpeppe: SyncTools only does a storage.List on the source and target, and copies across what's missing from source. It grabs metadata from target if it exists, then generates metadata for what's missing by fetching the tools back off the target | 09:35 |
| rogpeppe | axw: ah, no WriteMetadata does it, i see | 09:35 |
| rogpeppe | axw: we really need some sort of local storage cache | 09:35 |
| fwereade | mattyw, pong, sorry, got distrcted half way through | 09:36 |
| axw | rogpeppe: yeah, that could certainly help | 09:36 |
| mattyw | fwereade, no problem, just going through the next stage of the id stuff you said about adding the owner as an argument to AddService | 09:36 |
| fwereade | mattyw, ah, yes | 09:37 |
| mattyw | fwereade, that seems to break A LOT of stuff (all just mechanical changes because now there's a new argument) so I just wanted to check that this was what you wanted | 09:37 |
| fwereade | mattyw, I can't think of another way to do it, I'm afraid | 09:39 |
| mattyw | fwereade, ok no problem, just wanted to check we had the right idea before we fix all the things we broke | 09:39 |
| fwereade | mattyw, yeah, thanks for checking :) | 09:39 |
| mattyw | fwereade, I guess this is one reason for it being the next step - the number of lines in the diff is likely to be huge | 09:40 |
| fwereade | mattyw, indeed, probably best to do as little as possible beyond that | 09:41 |
| rogpeppe | mattyw, fwereade: this may be a silly idea, but i just thought i'd mention the possibility: | 09:48 |
| rogpeppe | mattyw, fwereade: rather than add "owner" arguments to lots of methods in State, we could potentially have a facade in front of State that knows the current owner | 09:48 |
| fwereade | rogpeppe, we're not planning to give anything except services owners at the moment | 09:49 |
| rogpeppe | fwereade: even still - i just wonder if it might be a more economical approach, given that any given API client will be acting on behalf of a particular user | 09:50 |
| fwereade | rogpeppe, and fwiw that facade is really the api server, anyway, isn't it? | 09:51 |
| rogpeppe | fwereade: sure, but we're about to change many many calls into state.State. | 09:51 |
| fwereade | rogpeppe, expand a little maybe? ISTM that it's going to be even harder to do that | 09:51 |
| rogpeppe | fwereade: type StateWithOwner {*state.State; User string} | 09:52 |
| rogpeppe | fwereade: then StateWithOwner.AddService calls State.AddServiceOwnedBy | 09:53 |
| fwereade | rogpeppe, so we s/State/StateWithOwner/ everywhere? | 09:53 |
| fwereade | rogpeppe, not 100% sure that wins us anything | 09:54 |
| rogpeppe | fwereade: i'm not sure either; worth thinking through though, i think. | 09:54 |
| fwereade | rogpeppe, ISTM that the only benefit of what you propose is that we need to make fewer test changes | 09:55 |
| fwereade | rogpeppe, but that each of those tests gets a bit more abstracted from what it's testing | 09:55 |
| rogpeppe | fwereade: test changes are a big cause of churn and time taken | 09:55 |
| fwereade | rogpeppe, I think that's an argument for a better way to define test fixtures, not an argument for messing with the code itself | 09:57 |
| rogpeppe | fwereade: there are 231 calls to AddService | 09:57 |
| fwereade | rogpeppe, how many places is AddService really used? once? twice? | 09:57 |
| rogpeppe | fwereade: once | 09:58 |
| rogpeppe | :-) | 09:58 |
| fwereade | rogpeppe, ensuckening the code to accommodate shitty tests is not a win IMO | 09:58 |
| fwereade | rogpeppe, I think the actual answer is in an AddService params struct | 10:00 |
| mgz | jam: HEY | 10:00 |
| mgz | er, caps | 10:00 |
| jam | hey mgz | 10:00 |
| mgz | mumble? | 10:01 |
| fwereade | rogpeppe, because I don't think this will be the only change like this | 10:01 |
| fwereade | rogpeppe, eg it's crazy that we create a service and set its config in different txns | 10:01 |
| fwereade | mattyw, ^^ - since you have to hit them all anyway, a params struct might not be a bad idea | 10:02 |
| mattyw | fwereade, yeah just thinking about that, most of the changes seem to be in tests that don't need to know about an owner, this would be one way of getting around that | 10:02 |
| rogpeppe | fwereade: a params struct seems reasonable | 10:06 |
| fwereade | mattyw, mmmmm I worry a little that we actually *do* want an owner on every service | 10:06 |
| fwereade | mattyw, the other possibility is to replace all of those with a helper in state/testing that adds a service with an implicit user-admin owner | 10:06 |
| mattyw | fwereade, but in the tests where we don't need to test against an owner it seems odd to pass one in | 10:06 |
| rogpeppe | fwereade: i'm thinking along those lines | 10:07 |
| rogpeppe | fwereade: i'm wondering about something like testing.Deploy | 10:07 |
| rogpeppe | fwereade: which would do the AddTestingCharm, add some number of units and return the results. | 10:07 |
| rogpeppe | fwereade: which would hopefully match reasonably well to the requirements of lots of tests | 10:08 |
| TheMue | fwereade: you pushed the ec2 and openstack Prepare() in. shall i continue now with the others? | 10:08 |
| fwereade | mattyw, rogpeppe: I'd really not like to use the Deploy terminology anywhere we don't have to, it's pretty much crack | 10:08 |
| fwereade | TheMue, that would be awesome, yes please | 10:09 |
| rogpeppe | fwereade: really? | 10:09 |
| TheMue | fwereade: ok | 10:09 |
| fwereade | rogpeppe, yeah -- AddService and AddUnit are sane, promoting AddService+AddUnits to a primitive of its own is just silly at the model level | 10:09 |
| fwereade | rogpeppe, fine, it's nice at the UI level | 10:10 |
| rogpeppe | fwereade: istm that it's one of the best know juju commands and has reasonably well known semantics, so when we see it in a test we know what's happening | 10:10 |
| fwereade | rogpeppe, but the command itself is about the user experience, not the underlying model | 10:10 |
| fwereade | rogpeppe, when we're testing the model I would prefer that we not mix in concepts from higher levels | 10:11 |
| rogpeppe | fwereade: i'm interested in making our test code smaller here | 10:11 |
| rogpeppe | fwereade: how many testing AddService calls are *not* followed by an AddUnit ? | 10:11 |
| rogpeppe | fwereade: and how many are not preceded by an AddTestingCharm call? | 10:12 |
| fwereade | rogpeppe, the charm case is somewhat stronger than the unit one, I think | 10:13 |
| rogpeppe | fwereade: i sometimes toy with the idea of allowing tests to set up stuff with some kind of textual DSL | 10:14 |
| fwereade | rogpeppe, and I would prefer that mattyw implement a small step in a helpful direction than that he get lumbered with the burden of analysing and fixing additional context 230 tims | 10:14 |
| rogpeppe | fwereade: sure, sorry, i've diverted here | 10:14 |
| fwereade | rogpeppe, something like that would be great, I agree | 10:14 |
| rogpeppe | axw: how have you been reproducing lp#1235717 ? | 10:17 |
| _mup_ | Bug #1235717: null provider bootstrap fails with error about sftp scheme <juju-core:In Progress by axwalk> <https://launchpad.net/bugs/1235717> | 10:17 |
| mattyw | fwereade, rogpeppe sorry - internet went down | 10:18 |
| fwereade | mattyw, no worries | 10:19 |
| mattyw | (maybe not the whole thing - but certainly my view of it) | 10:19 |
| rogpeppe | mattyw: np - you only missed my ramblin | 10:19 |
| rogpeppe | g | 10:19 |
| rogpeppe | mattyw: i think the conclusion is: add a params struct | 10:19 |
| fwereade | rogpeppe, mattyw: wait a mo | 10:19 |
| fwereade | rogpeppe, mattyw: a params struct is, independently a somewhat good idea, but I think I'd rather see a trivial AddService helper in state/testing, that takes the same args and just sets user-admin | 10:20 |
| rogpeppe | fwereade: seems reasonable - then we can make the call itself panic if it gets an empty owner, perhaps | 10:21 |
| fwereade | rogpeppe, mattyw: I wouldn't say no to a params struct as well ofc | 10:21 |
| fwereade | rogpeppe, yeah, something like that | 10:21 |
| fwereade | rogpeppe, mattyw: we *will* need to make sure juju still works on services with empty owner fields | 10:21 |
| fwereade | rogpeppe, mattyw: but the StateSuite(?) has references to all those collections anyway, so we can mess them up in the DB and check they still work | 10:22 |
| fwereade | rogpeppe, mattyw: however once service owners are part of the model I think it would be foolish to allow them to remain unset in normal use | 10:22 |
| fwereade | rogpeppe, matty: probably not a panic though, this stuff will be happening in the api server | 10:23 |
| rogpeppe | fwereade: unless, for backward compatibility purposes, we say that blank means admin | 10:23 |
| fwereade | rogpeppe, internally we do | 10:23 |
| fwereade | rogpeppe, no excuse for muddying up the interface | 10:23 |
| rogpeppe | fwereade: agree | 10:24 |
| mattyw | fwereade, rogpeppe ok, just want to make sure I understand this right: First step is to make a helper in state/testing for addservice, this will call the addservice function as is but also set the admin user inside the service? | 10:27 |
| rogpeppe | mattyw: i think so, yes | 10:27 |
| mattyw | and all the calls to addservice in the tests should use this new helper? or only the ones that need the owner set? | 10:28 |
| rogpeppe | mattyw: all the calls, i think | 10:34 |
| rogpeppe | fwereade: looking at the null provider, it's a pity that "null" needs to be quoted in environments.yaml. i wonder if we should've gone with the "nil" provider instead... | 10:35 |
| axw | rogpeppe: sorry, was afk | 10:39 |
| axw | yes, I have reproduced | 10:39 |
| rogpeppe | fwereade: do you think we should print admin-secret in the output of juju init? | 10:39 |
| fwereade | rogpeppe, ha, we should be dropping those really, shouldn't we | 10:39 |
| rogpeppe | axw: i wanted to reproduce for myself - was wondering how you did | 10:40 |
| axw | 1. bootstrap. 2. stop juju processes; remove /etc/init/juju*, everything in /var/lib/juju *except* storage/tools/releases/*; bootstrap again | 10:40 |
| rogpeppe | fwereade: well, it's potentially useful | 10:40 |
| fwereade | rogpeppe, who for?> | 10:40 |
| rogpeppe | fwereade: but then again there are probably other sources for info on provider settings | 10:40 |
| axw | rogpeppe: going to make/have dinner, I'll write down the steps properly a bit later | 10:40 |
| rogpeppe | fwereade: anyone that wants to have a known admin secret login | 10:40 |
| rogpeppe | axw: thanks | 10:40 |
| rogpeppe | fwereade: for example if i want to use the web GUI interface on one of my environments currently, i have to type in something like c1c4cb509044b524b35585f0fb259646 which isn't ideal | 10:42 |
| fwereade | rogpeppe, pasting it in isn't so hard though :) | 10:44 |
| jam | dimitern: TheMue: standup? | 10:45 |
| jam | https://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig | 10:45 |
| mattyw | rogpeppe, fwereade does the helper want to go in state/testing or in juju/testing in the JujuConnSuite (alongside addtestingcharm for example) | 11:02 |
| rogpeppe | mattyw: the latter | 11:03 |
| mattyw | rogpeppe, make more sense, ok thanks | 11:03 |
| dimitern | fwereade, rogpeppe: https://codereview.appspot.com/14486043 | 11:07 |
| * TheMue => lunch | 11:15 | |
| natefinch | so, are the dev releases of juju released to the public or are they internal only? | 11:16 |
| jam | natefinch: dev releases are public | 11:16 |
| jam | natefinch: 1.15.1 is in the public s3 bucket, the hp storage container, tec. | 11:17 |
| jam | etc | 11:17 |
| natefinch | jam: interesting.... I haven't built a windows installer for it, so I'm assuming the installer on the website is still 1.14 | 11:17 |
| dimitern | natefinch, that's still the "stable" release | 11:18 |
| natefinch | dimitern: ahh, ok, so the dev releases are public, but don't replace the latest stable release | 11:19 |
| dimitern | natefinch, I think so yeah | 11:19 |
| natefinch | dimitern: basically wanted to make sure I wasn't falling down on the job by not creating an installer for 1.15. | 11:20 |
| dimitern | natefinch, I think there still should be an installer for every release, even dev ones | 11:20 |
| dimitern | jam, fwereade ? ^^ | 11:20 |
| natefinch | dimitern: that would seem to make sense. | 11:20 |
| jam | natefinch: we would still want a 1.15 installer, otherwise we can't test that it is suitable for being a stable release. | 11:27 |
| jam | ideally it would just be part of the release process | 11:28 |
| jam | but the need for a Windows machine makes it slightly tricky | 11:28 |
| jam | natefinch: if you could easily set it up with an automated "ec2" sort of script, then someone like curtis could kick it off after cutting the tarball | 11:28 |
| natefinch | jam: You mean a script to create the installer? I certainly could, but I don't know about "easily" :) It's sort of an 8 step process if you 're starting from a bare machine and need to build both the juju client and the installer from it. | 11:31 |
| jam | natefinch: so the idea would be you *could* do a Windows ec2 instance so that anyone on the team could have access to it. | 11:31 |
| jam | It would be possible to bundle the final image | 11:31 |
| jam | so you don't have to do all the setup yet-again | 11:31 |
| jam | I used that in the past for other projects and it was ~ok, not great, but reasonable | 11:32 |
| natefinch | jam: ahh, yeah that would certainly be easier than scripting the install of Go and all that :) | 11:32 |
| natefinch | jam: dang, I was thinking I should just write a charm to do it... but of course that won't work on windows :) | 11:34 |
| rogpeppe | hmm, | 11:43 |
| rogpeppe | axw: one problem is: tools.WriteMetadata actually ignores any metadata that it finds that corresponds with tools already in the target storage | 11:44 |
| rogpeppe | axw: if we fixed that, the problem would disappear for almost all the time | 11:45 |
| rogpeppe | axw: and make sync tools faster too | 11:45 |
| dimitern | rogpeppe, fwereade, review poke | 11:50 |
| rogpeppe | dimitern: sorry, looking | 11:51 |
| axw | rogpeppe: hmm yeah, I guess it should check if size/sha256 are initialised or not, and use that to decide to blat over the top | 12:06 |
| rogpeppe | axw: it only appends to toolsList | 12:07 |
| dimitern | axw, size/sha256 are never initialized if you upgrade from 1.14 btw | 12:07 |
| rogpeppe | axw: whereas it actually want to update it if there are some entries in existingMetadata that are also in toolsList already | 12:08 |
| rogpeppe | s/want/wants/ | 12:08 |
| rogpeppe | axw: "Merge in existing records" is a bit of an overstatement | 12:09 |
| rogpeppe | axw: i wonder if things might work fine if we just ignored the List entirely. | 12:12 |
| rogpeppe | axw: and just merged the metadata | 12:12 |
| axw | rogpeppe: I don't really get it. In the scenario I've been investigating, len(existingMetadata) == 0 | 12:13 |
| rogpeppe | axw: really? interesting. | 12:13 |
| axw | rogpeppe: i.e. the tools tarballs are in storage, but no the metadata json | 12:14 |
| axw | not* | 12:14 |
| rogpeppe | axw: i tried a scenario where there *is* existing metadata, but we still have the same problem | 12:14 |
| axw | ah | 12:14 |
| axw | rogpeppe: can you please describe it? | 12:14 |
| rogpeppe | axw: i bootstrapped with the null provider (to localhost) | 12:15 |
| rogpeppe | axw: (the bootstrap failed because i'm already running the local provider, but it doesn't matter in this case) | 12:15 |
| rogpeppe | axw: then i tried bootstrapping again, and saw the "sftp:" problem | 12:15 |
| rogpeppe | axw: and i can verify that the metadata is all there | 12:16 |
| axw | ah ok | 12:16 |
| axw | rogpeppe: that's probably a more plausible explanation for what Bjorn reported :) | 12:16 |
| rogpeppe | axw: and looking at WriteMetadata, i can't see how any hash/size metadata for existing tools in the target storage would get into the metadata without re-reading them | 12:17 |
| rogpeppe | axw: because the toolsList argument won't have that metadata and the "Merge in existing records" loop doesn't change any entries in that argument | 12:18 |
| rogpeppe | axw: yes - it'll happen every time | 12:18 |
| axw | rogpeppe: ahh, because it's appending, not replacing... | 12:18 |
| rogpeppe | axw: so i'm wondering whether a better approach is just to ignore tools in the target bucket if they don't have metadata | 12:18 |
| rogpeppe | axw: yes | 12:19 |
| axw | ok | 12:19 |
| axw | I understand now, thanks | 12:19 |
| rogpeppe | axw: it appends only if it doesn't find any existing entries | 12:19 |
| rogpeppe | axw: the only down side i can currently see about ignoring tools with no metadata is that we may redo work, re-copying tools that were already copied | 12:20 |
| rogpeppe | axw: which will happen if the upload process was aborted half way through | 12:20 |
| hazmat | g'morning (relatively) i think the recent simplestreams/tools rejigger broke what used to be a correct stackoverflow answer that's still being referenced/used, if anyone has a minute it would be worth correcting http://askubuntu.com/questions/327177/cannot-bootstrap-due-to-precise-images-in-regionone-with-arches-amd64-i386 | 12:23 |
| rogpeppe | dimitern: reviewed | 12:29 |
| dimitern | rogpeppe, cheers | 12:29 |
| dimitern | fwereade, poke | 12:29 |
| dimitern | rogpeppe, I wouldn't mix tags into state - they're an api concept | 12:31 |
| axw | rogpeppe: seems to me that SyncTools should be calling WriteMetadata with only "missing", not targetTools+missing | 12:40 |
| axw | and whatever's where already should be assumed to be correct already | 12:41 |
| axw | s/where/there/ | 12:41 |
| axw | i.e. only update metadata for things that are copied | 12:41 |
| === gary_poster|away is now known as gary_poster | ||
| rogpeppe | dimitern: Tag is implemented on State objects | 12:45 |
| dimitern | rogpeppe, but not used otherwise for queries and such | 12:45 |
| rogpeppe | dimitern: does that matter here? this an error message for users | 12:46 |
| rogpeppe | dimitern: i guess users don't really see tags either though | 12:46 |
| dimitern | rogpeppe, well | 12:48 |
| dimitern | rogpeppe, yeah I suppose it'll reduce the code a bit | 12:48 |
| rogpeppe | dimitern: yeah, i think just a simple list of agents would work just as well | 12:49 |
| rogpeppe | axw: i've been wondering about the "abort half way through problem" and perhaps this is a stupid idea, but what if we took the metadata as primary, and always copied that *first*? | 12:50 |
| rogpeppe | axw: and avoid the use of Storage.List entirely | 12:51 |
| rogpeppe | axw: well, we couldn't avoid it entirely | 12:51 |
| rogpeppe | axw: but we'd only use it to find tools that the metadata mentions that don't show up in the list | 12:51 |
| * dimitern => lunch | 12:51 | |
| rogpeppe | axw: if we do things that way, the first thing SyncTools would do would be to merge the metadata | 12:52 |
| rogpeppe | axw: and then it would copy across the actual tools data as necessary | 12:53 |
| axw | rogpeppe: if it then fails while copying tools, we're going to have a bad time, no? | 12:53 |
| axw | next time we do a sync | 12:54 |
| axw | it'll think the target has everything | 12:54 |
| rogpeppe | axw: no, because we make sync actually check the List | 12:54 |
| rogpeppe | axw: at least, i *think* it could work ok | 12:57 |
| rogpeppe | axw: it'll potentially be a problem if we *haven't* done the sync | 12:57 |
| axw | rogpeppe: it sounds viable to me. I'd probably want to talk to wallyworld before doing it tho | 13:00 |
| rogpeppe | axw: for the time being, i think we should just ignore the target tools list | 13:01 |
| rogpeppe | axw: i'm really not very keen on the localhost storage wrapper solution | 13:01 |
| axw | rogpeppe: I'm not keen on the fact that URL() is there at all | 13:02 |
| rogpeppe | axw: it's there because we need it | 13:02 |
| rogpeppe | axw: it's our way of telling a remote cloud-init shell script how to get the juju tools | 13:03 |
| axw | rogpeppe: yeah I know the general use case, but I mean, it doesn't make sense for sshstorage | 13:03 |
| axw | since it's transient | 13:03 |
| rogpeppe | axw: perhaps you could make the method return an error? | 13:03 |
| axw | so I suppose you're right | 13:04 |
| axw | yeah | 13:04 |
| axw | it should return an error, and we should change this stuff to never call it | 13:04 |
| rogpeppe | axw: yeah | 13:04 |
| rogpeppe | fwereade: do you know if there's still a case where we do any kind of "best-fit" matching to the available tools? even juju bootstrap now picks an exact match, no? | 13:06 |
| axw | rogpeppe: by "ignore the target tools list", do you mean only update metadata for copied tools? | 13:08 |
| rogpeppe | axw: i mean that we make our copy decisions based on the source and target metadata only | 13:09 |
| rogpeppe | s/we make/we should make/ | 13:09 |
| rogpeppe | axw: that way we ensure that we always have the hash and size metadata | 13:10 |
| === gary_poster is now known as gary_poster|away | ||
| fwereade | rogpeppe, I don't think there's any ore guessing,no | 13:15 |
| rogpeppe | fwereade: eh? | 13:16 |
| rogpeppe | fwereade: ah, i see | 13:16 |
| rogpeppe | more | 13:16 |
| fwereade | rogpeppe, ah, yeah, sorry | 13:16 |
| rogpeppe | fwereade: i'm just pondering the implications of the change i was discussing with axw above | 13:16 |
| axw | rogpeppe: I'm clearly missing something (possibly wine related). I think we should just make this change: in SyncTools, in the call to WriteMetadata, pass "missing" instead of "targetTools" | 13:16 |
| rogpeppe | fwereade: to copy metadata *first*, then the tools themselves | 13:17 |
| fwereade | rogpeppe, that sounds bad? | 13:17 |
| axw | rogpeppe: because WriteMetadata seems to expect toolsList to contain only changed tools | 13:17 |
| axw | rogpeppe: as in, "newToolsVersions" being one-to-one with toolsList | 13:18 |
| rogpeppe | axw: looking again | 13:18 |
| fwereade | rogpeppe, offhand it seems that we'd be able to read metadata for missing tools if we did that | 13:18 |
| rogpeppe | fwereade: we would, but i'm not *sure* that's a bad thing | 13:18 |
| rogpeppe | fwereade: i'll just look for axw, back in a bit | 13:19 |
| * fwereade is becoming vexed by these blasted daywalking mosquitos | 13:20 | |
| rogpeppe | axw: i don't think that works | 13:22 |
| rogpeppe | axw: because missing is derived from Storage.List, and hence contains no metadata | 13:23 |
| rogpeppe | axw: i mean, no hash/size metadata | 13:23 |
| axw | rogpeppe: no, missing is copied, and the copy generates the size/hash | 13:24 |
| rogpeppe | axw: oh, that's sneaky and underdocumented :-) | 13:25 |
| axw | ;) | 13:25 |
| rogpeppe | axw: hmm, that *might* have the effect i'm after; still thinking | 13:26 |
| rogpeppe | axw: it means the argument to WriteMetadata now means something quite different | 13:27 |
| axw | rogpeppe: I'm not so sure - WriteMetadata seems to treat it as "changed" tools | 13:27 |
| axw | I think the name is just a bit generic | 13:27 |
| rogpeppe | axw: "The metadata from toolsList is already present" | 13:27 |
| rogpeppe | axw: hmm, but i guess it is present *now* | 13:28 |
| rogpeppe | axw: um, no | 13:28 |
| rogpeppe | axw: it's *some* of what's present now | 13:28 |
| axw | rogpeppe: yeah, some is present, some is new | 13:28 |
| rogpeppe | axw: it's what we've just copied, but none of what was there originally | 13:29 |
| axw | but going down the function, it then treats the whole lot as being new? | 13:29 |
| axw | rogpeppe: it currently includes what's there already | 13:29 |
| axw | SyncTools calls WriteMetadata(targetTools+missing) | 13:29 |
| rogpeppe | axw: yes - and that's very deliberate | 13:29 |
| === gary_poster|away is now known as gary_poster | ||
| rogpeppe | axw: the logic is trying to create a metadata file that includes both what we've just copied *and* what was in the bucket before | 13:30 |
| rogpeppe | axw: and our problem is that it can't know what was in the bucket before without reading it | 13:30 |
| rogpeppe | axw: rather, it can't know the metadata for previous bucket contents without reading them | 13:31 |
| rogpeppe | axw: heh, istm that WriteMetadata could pass metadataStore to generateMetadata and then we wouldn't have any of this problem | 13:34 |
| rogpeppe | axw: you said earlier that we don't know the tools storage name given the Tools struct | 13:35 |
| rogpeppe | axw: but of course that's not true - we do! | 13:35 |
| axw | rogpeppe: assuming the URLs are all relative to the same tools dir we do | 13:35 |
| rogpeppe | axw: we don't need to use the URL | 13:36 |
| axw | ah | 13:36 |
| axw | right | 13:36 |
| axw | it's statically defined | 13:36 |
| axw | :) | 13:36 |
| rogpeppe | axw: it's tools.StorageName(tools.Version) | 13:36 |
| axw | StorageName | 13:36 |
| axw | forgot about that | 13:36 |
| axw | rogpeppe: I'm still convinced that WriteMetadata should only be called with missing. "The metadata from toolsList is already present" means, to me, size/sha256 is precomputed for the tools in the list | 13:37 |
| axw | which is clearly not the case for the existing tools | 13:37 |
| rogpeppe | axw: and we should probably change sync.copyTools to avoid using URL too | 13:38 |
| rogpeppe | axw: if that's the case, then the current logic ignores any tools that are present before sync-tools is called, AFAICS | 13:39 |
| rogpeppe | axw: the question is "already present"... where? | 13:39 |
| axw | rogpeppe: howso? newToolsVersions will only contain things that were copied; existingMetadata may contain additional things | 13:40 |
| rogpeppe | axw: i'm thinking about the case where we've aborted a bootstrap half way through | 13:40 |
| axw | rogpeppe: that currently won't copy over again on the second sync, right? | 13:41 |
| rogpeppe | axw: (and i've also just realised why it's a really bad idea to re-read the target storage to work out its hash and size, but that's another story) | 13:41 |
| rogpeppe | axw: currently it won't copy over again on the second sync, but it *will* generate metadata for the files it copied across last time | 13:42 |
| axw | yeah. if "missing" were passed through, that'll still happen due to the "fetch" | 13:42 |
| rogpeppe | axw: really? | 13:43 |
| * rogpeppe looks again | 13:43 | |
| rogpeppe | axw: istm that fetch will only affect stuff that's in toolsList already | 13:44 |
| axw | rogpeppe: but toolsList is appended to | 13:44 |
| axw | with things in existingMetadata | 13:44 |
| rogpeppe | axw: yes, but i was talking about tools that do not currently exist in the target storage metadata | 13:45 |
| rogpeppe | axw: because we copied the tools but didn't get around to updating the metadata | 13:45 |
| rogpeppe | axw: and currently we *will* generate metadata for those tools, even though they don't have metadata, i think | 13:45 |
| axw | rogpeppe: yes, we do | 13:46 |
| axw | rogpeppe: in the copy | 13:46 |
| rogpeppe | axw: sorry, we do what? | 13:46 |
| axw | rogpeppe: we do generate metadata for them | 13:46 |
| axw | as you said | 13:46 |
| rogpeppe | axw: and changing to use missing will mean that we don't, right? | 13:48 |
| axw | rogpeppe: it will if it's not in the target storage's metadata | 13:48 |
| rogpeppe | axw: really? | 13:48 |
| axw | ah wait | 13:49 |
| axw | no | 13:49 |
| axw | I get it now | 13:49 |
| * axw sighs | 13:49 | |
| rogpeppe | axw: i think part of the difficulty with analysing this is that WriteMetadata mixes concerns quite a bit | 13:50 |
| rogpeppe | axw: did you just lose contact? | 13:54 |
| axw_ | rogpeppe: I think, instead, we should change newToolsVersions to map[string]*tools.Tools, and store each item from toolsList in that | 13:54 |
| axw_ | yeah | 13:54 |
| rogpeppe | axw_: last thing i saw from you was "axw sighs" | 13:54 |
| axw_ | rogpeppe: then replace any item in it if Size==0 | 13:54 |
| axw_ | <rogpeppe> axw: i think part of the difficulty with analysing this is that WriteMetadata mixes concerns quite a bit | 13:55 |
| axw_ | <axw> rogpeppe: and the wine | 13:55 |
| axw_ | <axw> but yes | 13:55 |
| rogpeppe | lol | 13:55 |
| === axw_ is now known as axw | ||
| axw | rogpeppe: something like this: http://paste.ubuntu.com/6205035/ | 14:02 |
| rogpeppe | axw: BTW it's fine to use a version.Binary as a map key | 14:04 |
| rogpeppe | axw: (i know the original didn't, but just saying) | 14:04 |
| axw | rogpeppe: thanks, will keep it in mind | 14:05 |
| rogpeppe | axw: that looks plausible, but i had in mind a slightly more radical solution; one mo | 14:06 |
| rogpeppe | axw: something like this (as primitives) http://paste.ubuntu.com/6205057/ | 14:08 |
| rogpeppe | axw: then most of the awkward logic in WriteMetadata could be abstracted out to where it belongs, in SyncTools | 14:09 |
| rogpeppe | axw: because there is no reason AFAICS why WriteMetadata should be so bound up with merging stuff | 14:10 |
| axw | rogpeppe: that sounds like a good idea to me | 14:12 |
| axw | it's a little complicated at the moment | 14:13 |
| rogpeppe | axw: here's a version with some doc comments, to give a slightly better idea of what i'm thinking of: http://paste.ubuntu.com/6205080/ | 14:14 |
| rogpeppe | axw: FromList could probably be MetadataFromList | 14:16 |
| natefinch | jam, rogpeppe, mgz, fwereade, dimitern, TheRealMue: anyone up for an easy one? https://codereview.appspot.com/14326044/ | 14:16 |
| rogpeppe | natefinch: looking | 14:17 |
| natefinch | arosales: ping | 14:18 |
| rogpeppe | axw: would you mind going forward with that? i know it's a serious diversion from your original fix :-) | 14:19 |
| axw | rogpeppe: yep, I'm happy with that | 14:20 |
| axw | getting pretty late now, I'll pick it up in the morning | 14:20 |
| rogpeppe | fwereade: i'd appreciate it if you took a glance at my above paste, as applied to tools.WriteMetadata | 14:20 |
| rogpeppe | axw: thanks | 14:20 |
| rogpeppe | axw: it was useful you were able to stay on | 14:20 |
| axw | rogpeppe: thanks for enduring my nonsense :) | 14:21 |
| rogpeppe | axw: not at all | 14:21 |
| rogpeppe | axw: i was nonsensical too | 14:21 |
| arosales | natefinch, pong | 14:22 |
| axw | alrighty, I'm off to prod llgo a bit. ttyl | 14:22 |
| natefinch | arosales: it has come to my attention that's there.s a 1.15.1 release for Juju but no one has asked me for a windows installer for it... that seems like something we should fix | 14:23 |
| natefinch | rogpeppe: I actually should update dependencies.tsv as well, since that change to ec2 requires a change to goamz (which is already in goamz) | 14:25 |
| rogpeppe | natefinch: you should :-) | 14:25 |
| natefinch | rogpeppe: done, committed, proposed | 14:26 |
| arosales | natefinch, I think we are only doing osx and windows installs for stable releases | 14:27 |
| arosales | s/installs/clients/ | 14:27 |
| natefinch | arosales: as jam pointed out to me this morning, we can't know if it's stable if we don't test the dev release :) | 14:27 |
| arosales | natefinch, good point :-) | 14:28 |
| arosales | natefinch, could you sync with sinzui on the process for creating windows clients and testing them? | 14:29 |
| natefinch | arosales: sure thing | 14:29 |
| arosales | natefinch, thanks. | 14:29 |
| sinzui | natefinch, we can talk later today. I know how you build it. | 14:30 |
| natefinch | sinzui: sure thing. Just wanted to make sure it didn't fall through the cracks, especially given we've occasionally had windows-only problems. | 14:30 |
| arosales | natefinch, sinzui I expect it to be a little ad hoc atm until sinzui's team can get some time to build out the testing matrix with tooling. | 14:31 |
| rogpeppe | natefinch: reviewed | 14:32 |
| sinzui | arosales, I got Lion + xcode 5 + 1.14.0 to install on my mac, but I cannot get 1.14.1 to install. I do reproduce the same error seen in my untested pull request. We need to look at vars.go and learn what changed between 1.14.0 and 1.14.1 I also confirm 1.15.1 fails the same way for me: https://github.com/mxcl/homebrew/pull/22762 | 14:33 |
| sinzui | natefinch, I did test azure deploys and upgrades. That is why the release went out at 3AM my time | 14:34 |
| arosales | sinzui, thanks for working on osx. are you able to build locally? | 14:35 |
| arosales | juju-core devs any hint on the error, "src/launchpad.net/juju-core/juju/osenv/vars.go:40: undefined: Home" | 14:36 |
| arosales | when building juju-core | 14:36 |
| === amithkk_ is now known as amithkk | ||
| sinzui | arosales, 1.14.0 can. 1.14.1+ cannot. | 14:37 |
| === amithkk is now known as Guest99228 | ||
| arosales | sinzui, same error when building locally? | 14:37 |
| fwereade | rogpeppe, sorry, is that paste just about primitives with which to rewrite WriteMetadata? | 14:37 |
| mgz | arosales: you need vars_linux.go in that directory? | 14:37 |
| rogpeppe | fwereade: yes, pretty much | 14:37 |
| rogpeppe | fwereade: and in the process fix a current problem with it | 14:38 |
| mgz | arosales: should have a func Home() in it | 14:38 |
| rogpeppe | fwereade: (as well as the fact that it's very hard to reason about currently, because it does about 4 different things at once) | 14:38 |
| fwereade | rogpeppe, yeah, I am having that problem as I try to parse exactly how they'd be used | 14:39 |
| fwereade | rogpeppe, they do look sensible at first blush though :) | 14:39 |
| rogpeppe | fwereade: they'd be called from SyncTools, which would use MergeMetadata as appropriate | 14:40 |
| rogpeppe | fwereade: at the moment a significant portion of WriteMetadata is really about syncing tools | 14:40 |
| fwereade | rogpeppe, that sounds like a good way to go offhand | 14:41 |
| rogpeppe | fwereade: thanks. i think it's preferable to papering over the problem, cf. https://codereview.appspot.com/14465045/ | 14:42 |
| fwereade | rogpeppe, heh, the description there surely raises an eyebrow or two | 14:43 |
| fwereade | rogpeppe, don't we have a filesystem storage implementation anyway if we're just on localhost? | 14:43 |
| rogpeppe | fwereade: indeed | 14:43 |
| rogpeppe | fwereade: my brain hurts | 14:44 |
| rogpeppe | fwereade: erm, no we don't have a filesystem storage implementation for the bootstrap storage on the remote machine | 14:46 |
| rogpeppe | fwereade: that CL uses a local HTTP listener to talk to the remote storage via ssh | 14:47 |
| fwereade | rogpeppe, heh, so presumably we don't have one for the local provider either? | 14:47 |
| rogpeppe | fwereade: i think we do | 14:48 |
| fwereade | rogpeppe, we should probably be using that where it makes sense then tbh | 14:48 |
| rogpeppe | fwereade: this is only an issue when bootstrapping the null provider, i think | 14:48 |
| fwereade | rogpeppe, hmm, I could have sworn we added file:// url handling for exactly that case | 14:49 |
| rogpeppe | fwereade: sorry, not quite sure where you're coming from here | 14:49 |
| rogpeppe | fwereade: how would that help when the files we're referring to are the other end of an ssh connection with no juju binaries at the other end? | 14:49 |
| rogpeppe | s/are the/are at the/ | 14:50 |
| mattyw | fwereade, I'm wondering If I should track the juju id stuff as a blueprint to help us keep track of it? | 14:50 |
| mattyw | (sorry to interupt) | 14:50 |
| fwereade | rogpeppe, I was just thinking of what happens purely at the far end -- so we do surely need the SSH storage to write to it, but unless it's actively *easier* we should probably just be using the filesystem to get them when we're actually running on the remote machine | 14:51 |
| rogpeppe | fwereade: that's what we do, i think | 14:51 |
| fwereade | mattyw, just a sec let me have a quick look, I couldhave sworn there wasone already | 14:51 |
| fwereade | rogpeppe, ok, then I completely misunderstood something | 14:52 |
| rogpeppe | fwereade: the URL in this case is only used locally as a result of WriteMetadata and SyncTools being a bit crackful | 14:52 |
| rogpeppe | fwereade: my suggestion is to make SSHStorage.URL return an error | 14:53 |
| rogpeppe | fwereade: (to make it clear that there really is no URL) | 14:53 |
| rogpeppe | fwereade: and to change the SyncTools logic so it never uses the storage URL (it really doesn't need to) | 14:53 |
| fwereade | rogpeppe, ah, ok, if getting URLs out of SyncTools solves the problem I'm 100% behind that | 14:54 |
| rogpeppe | fwereade: i'm pretty sure it does, yes | 14:54 |
| fwereade | rogpeppe, and indeed it seems there really is no sensible URL | 14:54 |
| fwereade | rogpeppe, great, thanks | 14:55 |
| rogpeppe | fwereade: hence the refactoring suggestion above. axw had a suggestion for tweaking the existing logic, but i really would prefer to clean things up in this area | 14:55 |
| rogpeppe | fwereade: as it's taken both of us about a day to understand the problem, and i know that i won't understand the code when i go back to look at it again tomorrow | 14:56 |
| fwereade | rogpeppe, that sounds smart to me -- I guess axw will be looking into it tonight? | 14:56 |
| rogpeppe | fwereade: yes, he said he'd take it forward | 14:57 |
| sinzui | Does anyone have a 1.14.1 and 1.15.1+ setup that they can test upgrade-juju on openstack. My testing of this always fails | 14:58 |
| arosales | mgz, thanks for the suggestion. Is this a manual add or something we are missing in out builds (re vars_linux.go) | 15:04 |
| mgz | no. I'm just trying to help you diagnose why your tree is broken | 15:04 |
| jamespage | sinzui, I'm trying to test that; but I'm getting confused about how 'juju-tools' endpoint should be configured in our keystone instance | 15:07 |
| jamespage | 1.14.1 works with a juju-tools URL "http://10.98.191.34:8080/v1/AUTH_79699f6f71e245b186720f1e2bc03cf0" | 15:08 |
| jamespage | 1.15.1 does not like that | 15:08 |
| sinzui | jamespage, I tried tools-url: https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/tools | 15:09 |
| sinzui | ^ that is per wallyworld. | 15:09 |
| sinzui | I can see the released tools are found on canonistack and hp by appending juju-dist/tools to the public-bucket url | 15:10 |
| jamespage | hmm | 15:10 |
| natefinch | mgz, arosales: about the broken code.... he's talking about building on OSX - which I think we don't have an implementation of Home for | 15:11 |
| jamespage | sinzui, nope | 15:11 |
| arosales | mgz, ah gotcha | 15:11 |
| jamespage | juju fails to find them and tries to revert back to s3 | 15:11 |
| jamespage | which is not accesible in my env | 15:12 |
| mgz | natefinch: seems unlikely, unless there's some weird build stuff, the logic is just if "windows" else linux | 15:12 |
| arosales | natefinch, mgz, I thought sinzui said he also saw the error locally . . . | 15:12 |
| arosales | mgz, natefinch I would have to defer to sinzui on the error, but sinzui may be busy atm with juju-tools | 15:13 |
| natefinch | mgz: There's a vars_windows.go and a vars_linux.go .... there's no var_darwin.go | 15:13 |
| mgz | there's no reason it should just use linux though | 15:14 |
| natefinch | mgz: the implementation is the same, yes. It's a bug in the code | 15:14 |
| arosales | mgz, sorry for losing the context | 15:14 |
| arosales | mgz, here is the background https://github.com/mxcl/homebrew/pull/22762 | 15:14 |
| arosales | mgz, roughly trying to build on osx via brew. | 15:15 |
| mgz | seems like that just needs fixing then | 15:15 |
| arosales | natefinch, do you see a similar error for windows client builds? | 15:15 |
| sinzui | jamespage, Does this look like a sane listing? I think it is. https://pastebin.canonical.com/98610/ | 15:15 |
| mgz | easy enough for anyone with a mac to test on | 15:15 |
| natefinch | arosales: no, it's OSX-only | 15:15 |
| arosales | huh that is interesting | 15:16 |
| jamespage | sinzui, yeah - thats what mine looks like | 15:16 |
| natefinch | arosales: it's a very clear cut problem... we have code for this Home function that compile sfor Windows and for OSX, but we forgot about OSX | 15:16 |
| arosales | natefinch, ah | 15:16 |
| natefinch | arosales: where we == me | 15:16 |
| arosales | :-) | 15:17 |
| sinzui | jamespage, I places the tools there with "juju sync-tools -e public-tools-canonistack --dev" | 15:17 |
| arosales | natefinch, mgz sinzui or myself can test on osx if you have a something you would like us to try | 15:17 |
| sinzui | jamespage, maybe we need a trailing / in the tools url? | 15:17 |
| natefinch | arosales: I can make a fix... should I make it based on 1.14? Or should I just make it based on 1.15 and we can make sure it gets into 1.16? | 15:19 |
| arosales | natefinch, my preference would be 1.15.1 based so we can target 1.16 | 15:20 |
| natefinch | arosales: mine too :) | 15:20 |
| * arosales guesses it would need to fit trunk and then also be applied to the 1.15.1 branch | 15:20 | |
| sinzui | mgz, sorry, just caught up to want you are looking at . I just annotated the module. I can test, but I need to reboot :( The change does explain why 1.14.1+ is broken, though why osx/darwin/bsd/posix doesn't behave like gnu/posix is puzzling | 15:21 |
| arosales | sinzui, I think natefinch is going to work on a fix | 15:25 |
| natefinch | sinzui: it's just a build tag on one of the files that says "compile this file only for linux" when it should have said "compile this file for any OS that isn't Windows" | 15:26 |
| sinzui | natefinch, fab, I am schooled. | 15:27 |
| natefinch | sinzui: I managed to add support for windows while simultaneously effectively removing support for OSX ;) | 15:27 |
| natefinch | mgz: is there magic I need to do to rename a file so bzr keeps the revision history? | 15:28 |
| sinzui | natefinch, I suspected that since win is the main difference between 1.14.0 and 1.14.1. | 15:28 |
| mgz | natefinch: `bzr mv`? | 15:28 |
| natefinch | mgz: ahh of course | 15:30 |
| natefinch | mgz: https://codereview.appspot.com/14496043 | 15:33 |
| mgz | looking | 15:34 |
| mgz | what is that magic... | 15:35 |
| mgz | natefinch: so, why was it failing, given the two files didn't have build constraints before | 15:37 |
| natefinch | mgz: the _<os>.go is effectively a build tag... so before we had _linux and _windows | 15:38 |
| natefinch | mgz: now I replaces the _linux one with an in-file build tag that specifies !windows | 15:38 |
| mgz | that's all to magic | 15:38 |
| mgz | I am not liking the build package docs I'm reading... | 15:38 |
| mgz | *too | 15:39 |
| natefinch | mgz: standard go way of doing build tags ... I actually really like the _<os>.go because it makes it very clear from just the name of the file where it'll build | 15:39 |
| natefinch | mgz: like _test.go | 15:39 |
| natefinch | natefinch: I sorta wish _!windows.go would work | 15:39 |
| mgz | ugh | 15:40 |
| natefinch | mgz: ^^ heh, tagging myself | 15:40 |
| natefinch | mgz: not really... but almost | 15:40 |
| natefinch | mgz: it would also be more clear if rietveld made the name change more obvious | 15:44 |
| mgz | the launchpad diff is clear enough | 15:45 |
| mgz | (I checked that) | 15:45 |
| natefinch | mgz: yeah, that one's much better in this case | 15:45 |
| sinzui | rogpeppe, Is the status of this issue correct? Are we committing to fix it by 1.16.0? I think the landing implies it is no longer critical https://bugs.launchpad.net/juju-core/+bug/1233278 | 15:59 |
| _mup_ | Bug #1233278: juju test suite is not isolated <juju-core:In Progress by rogpeppe> <https://launchpad.net/bugs/1233278> | 15:59 |
| rogpeppe | sinzui: i don't think we're committing to fix it by 1.16 - it no longer seems to be causing quite so many problems | 16:00 |
| rogpeppe | sinzui: i think we can lower the pri to High, but please check with fwereade | 16:00 |
| sinzui | ^ fwereade what say you? ^ | 16:01 |
| mattyw | rogpeppe, fwereade I'm nowhere near done but is this the kind of thing you guys had in mind for the addservice helper? https://code.launchpad.net/~mattyw/juju-core/add-owner-to-service/+merge/189654 | 16:18 |
| rogpeppe | mattyw: yeah, looks reasonable. i *think* i'd be happier with the owner last in the params, as it seems the least important thing, but that's all | 16:21 |
| mattyw | rogpeppe, in the actual call to addservice? | 16:21 |
| rogpeppe | mattyw: yeah | 16:21 |
| mattyw | rogpeppe, probably a good idea, I've just shoved it there at the moment so I can get the compiler to generate a list of things I need to fix :) | 16:22 |
| rogpeppe | mattyw: i think it would do that anyway | 16:23 |
| rogpeppe | mattyw: as you're adding an extra arg | 16:23 |
| mattyw | rogpeppe, I just meant I just put it anywhere in the list without thinking about order | 16:24 |
| rogpeppe | mattyw: ah, no probs | 16:24 |
| jamespage | sinzui, http://paste.ubuntu.com/6205614/ | 16:26 |
| jamespage | I can't get my tools to pickup using tools-url | 16:26 |
| jamespage | sinzui, I can't get the tools-url to override anything afaict | 16:33 |
| sinzui | jamespage, thank you for confirming it sin't just me. I reported a bug and will continue to look for a configuration that is accepted | 16:33 |
| jamespage | sinzui, the other problem is that a keystone juju-tools entry that was compat with 1.14.1 is not compat with 1.15.1 | 16:33 |
| jamespage | and vica versa | 16:34 |
| sinzui | jamespage, When I tested this *before* release. I uploaded tools to a testing area. There were no aws tools at the time. I could deploy using the deb I built. You can still see my testing files at https://region-a.geo-1.objects.hpcloudsvc.com/v1/60502529753910/juju-dist | 16:35 |
| jamespage | sinzui, I'm wondering if because there is a juju-tools in my keystone catalog, the local client won't use the tools-url in the environments.yaml | 16:36 |
| * sinzui ponders :443 as a factor | 16:36 | |
| jamespage | sinzui, yeah - I don't use https for swift for this openstack deployment | 16:36 |
| jamespage | sinzui, actually my streams data looks like fud one second | 16:39 |
| jamespage | sinzui, hmm | 16:41 |
| jamespage | sinzui, well thats interesting | 16:44 |
| jamespage | sinzui, I juju destroy-environment'ed again | 16:44 |
| jamespage | and it was OK | 16:44 |
| jamespage | odd | 16:44 |
| sinzui | jamespage, you can upgrade-juju on an openstack? | 16:44 |
| jamespage | sinzui, I'll have to rollback to confirm that - one second | 16:45 |
| jamespage | sinzui, I can upgrade a bootstrap node | 16:48 |
| jamespage | my bug from the other day is back where the bootstrap node does not start any more units... | 16:49 |
| jamespage | still don't know how exactly that resolved itself | 16:49 |
| sinzui | jamespage, I have seen just the bootstrap node upgrade. The agents failed though. | 16:49 |
| * sinzui looks for log and updated bug about that | 16:50 | |
| natefinch | fwereade: care to talk about your comments on my rootdisk branch? | 16:54 |
| natefinch | dimitern, rogpeppe, mgz, fwereade: just got what looks to be a sporadic test failure on the bot - FilterSuite.TestUnitRemoval from inside the uniter | 17:00 |
| mgz | natefinch: can you file a bug and requeue? | 17:00 |
| natefinch | will do | 17:00 |
| dimitern | natefinch, I think I've seen that before, please make sure there's no existing bug about it | 17:08 |
| natefinch | dimitern: I saw another bug about a sporadic failure in uniter, but the tests and logs seemed different, so I filed a new bug. | 17:18 |
| dimitern | natefinch, ok, thanks | 17:19 |
| natefinch | Anyone know if we're allowed to expense parking fees at the airport for travel to the sprint? | 17:20 |
| rogpeppe | fwereade, natefinch, dimitern: https://codereview.appspot.com/14395043 | 18:07 |
| rogpeppe | natefinch: sorry, i don't know | 18:08 |
| rogpeppe | that's me for the night | 18:08 |
| rogpeppe | g'night all | 18:08 |
| natefinch | rogpeppe: I'll take a look. g'night | 18:08 |
| rogpeppe | natefinch: thanks | 18:51 |
| === hatch is now known as lbox | ||
| === lbox is now known as hatch | ||
| fwereade | natefinch, heyhey | 20:03 |
| natefinch | fwereade: howdy | 20:05 |
| fwereade | natefinch, about root-disk | 20:06 |
| fwereade | natefinch, it's not a problem with your branch per se, just a problem I noticed while I was looking at it ;) | 20:06 |
| natefinch | fwereade: I gathered as much. I'm fine with fixing it, was just unclear what the fix was | 20:07 |
| fwereade | natefinch, I *think* it's just to drop the fields that aren't always set, and move the hardwareCharacteristics method out into a func that acceptsinstanceType, rootDisk, (more?) | 20:08 |
| fwereade | natefinch, just so it doesn't look like hardwareCharacteristics is a meaningful operation on ec2Instance | 20:08 |
| fwereade | natefinch, the only place we create them is in StartInstance anyway, so it shouldn't be onerous | 20:09 |
| * fwereade waits to hear what he's missed :) | 20:09 | |
| fwereade | natefinch, am I blithering? | 20:10 |
| natefinch | fwereade: ha, I missed the fact that hardwareCharacteristics is only called from inside StartInstance. You're right, it's crack to store that data on the ec2Instance only to return it one line down in the same function | 20:10 |
| fwereade | natefinch, I think it's a hangover from when we thought it would be reasonable to ask an instance for the details of its hardware, but it's fiddlier than one might hoe | 20:11 |
| fwereade | er, hope | 20:12 |
| natefinch | fwereade: totally makes sense. | 20:13 |
| natefinch | fwereade: well that's an easy fix | 20:13 |
| fwereade | natefinch, cool, consider that LGTM with the change then | 20:13 |
| natefinch | fwereade: cool | 20:14 |
| fwereade | natefinch, oh, a thought | 20:14 |
| fwereade | natefinch, possibly we should keep the existing minimum size in play? | 20:14 |
| natefinch | fwereade: like don't let people set disk less than 8G? | 20:15 |
| fwereade | natefinch, I'm in no way wedded to the idea, but I'm a bit worried people might use =0 as shorthand for "I don't really care" without meaning "I *really* don;t care" | 20:15 |
| fwereade | iyswim | 20:15 |
| fwereade | natefinch, we guarantee we'll do at least as well as they ask | 20:16 |
| fwereade | natefinch, so an 8G result does match anything less than that | 20:16 |
| fwereade | natefinch, but actively giving them much less might cause avoidable problems | 20:17 |
| natefinch | fwereade: I can see someone wanting to keep their root disk as small as possible and then using the spare for a different partition (or something... ).. The image itself only uses like 1.5G or so from what I've seen | 20:17 |
| fwereade | natefinch, OTOH, if someone *does* ask for 4G they probably mean it | 20:17 |
| fwereade | indeed | 20:17 |
| natefinch | fwereade: I think treating 0 as 8G is fine | 20:18 |
| fwereade | natefinch, great, sgtm | 20:18 |
| fwereade | thanks | 20:18 |
| natefinch | fwereade: but if they ask for 1G... we should do our best and fall over if that turns out to be impossible | 20:18 |
| natefinch | fwereade: cool | 20:18 |
| fwereade | +1 | 20:18 |
| * fwereade is happy about the new place... it has a CRT TV,and it emerges that my dreamcast still works, as does the lightgun | 20:19 | |
| natefinch | haha awesome. Loved the dreamcast :) | 20:19 |
| fwereade | house of the dead 2 remains flat-out awesome :) | 20:20 |
| fwereade | sometime I will see if MSR really is better than any of the project gotham games | 20:21 |
| fwereade | I'm a bit afraid in case it isn't | 20:21 |
| natefinch | I don't think I ever played it. Played one of the gotham ones... forget which | 20:22 |
| fwereade | basically the same deal | 20:22 |
| fwereade | except they left the traffic islands in where they took them out of the gotham games | 20:22 |
| natefinch | huh funny | 20:23 |
| fwereade | I thought they were great, wasn't the same without them | 20:24 |
| natefinch | we played a ton of NHL2k*, NFL2k* and soul calibur | 20:27 |
| natefinch | which is sorta funny, since I don't even particularly like watching sports | 20:28 |
| fwereade | natefinch, the only sports game I ever really got into was... nhlpa on the megadrive maybe? | 20:52 |
| natefinch | fwereade: nice. | 20:53 |
| natefinch | fwereade: that was pretty much the last time I played sports games. American Football is surprisingly interesting to play, though in video games it often comes down to "give the ball to the guy who can run faster than anyone else" | 20:55 |
| natefinch | fwereade: though probably games have come a long way in the past decade ;) | 20:55 |
| fwereade | natefinch, heh, I am always a little bit embarrassed that I've never remotely figured out american football | 20:56 |
| fwereade | natefinch, probably just takes the right videogame though | 20:56 |
| natefinch | fwereade: there's a ton to it, but most of it is "Get the ball past the other line of guys and dance around in their home base" | 20:57 |
| fwereade | natefinch, clear goals are important ;) | 20:57 |
| natefinch | fwereade: I'm getting a problem with livetests.go .. can't tell if it's my fault or something external: | 20:58 |
| natefinch | [LOG] 85.45774 DEBUG juju.environs.simplestreams fetchData failed for "http://127.0.0.1:56349/test-bucket/tools/streams/v1/index.sjson?AWSAccessKeyId=x&Expires=1696711806&Signature=6T43LEGnZYvE04zTygifGAi5uxA%3D": The specified key does not exist. | 20:58 |
| natefinch | [LOG] 85.45776 DEBUG juju.environs.simplestreams cannot load index "http://127.0.0.1:56349/test-bucket/tools/streams/v1/index.sjson?AWSAccessKeyId=x&Expires=1696711806&Signature=6T43LEGnZYvE04zTygifGAi5uxA%3D": invalid URL "http://127.0.0.1:56349/test-bucket/tools/streams/v1/index.sjson?AWSAccessKeyId=x&Expires=1696711806&Signature=6T43LEGnZYvE04zTygifGAi5uxA%3D" not found | 20:58 |
| natefinch | /home/nate/code/src/launchpad.net/juju-core/environs/jujutest/livetests.go:874: | 20:58 |
| natefinch | c.Assert(err, gc.ErrorMatches, ".*missing machine nonce") | 20:58 |
| natefinch | ... error string = "no \"raring\" images in test with arches [amd64]" | 20:58 |
| natefinch | ... regex string = ".*missing machine nonce" | 20:58 |
| natefinch | OOPS: 36 passed, 6 skipped, 1 FAILED | 20:58 |
| natefinch | --- FAIL: TestEC2 (8.01 seconds) | 20:58 |
| fwereade | natefinch, heh, I have a horrible feeling that those were a casualty of the simplestreams change... are they all doing UploadFakeTools? | 20:59 |
| fwereade | natefinch, except waitno | 20:59 |
| fwereade | natefinch, they still ought to "work" with fake tools, they'll just go wrong on the instances | 21:00 |
| fwereade | natefinch, regardless I'm a bit surprised it's looking for raring... does the test specify that explicitly? | 21:01 |
| natefinch | fwereade: trunk works, must be something my change broke | 21:03 |
| fwereade | natefinch, ok, so, those log messages shouldn't be the problem | 21:04 |
| fwereade | natefinch, we don't expect .sjson in this context | 21:04 |
| fwereade | natefinch, yeah, some confirmation: ec2.TestImagesData does indeed lack raring-amd64 | 21:10 |
| fwereade | natefinch, but I am growing more and more confused about where raring is coming from | 21:13 |
| natefinch | fwereade: merging code in from trunk fixes the problem | 21:13 |
| fwereade | natefinch, ah cool :) | 21:13 |
| fwereade | natefinch, bbs | 21:13 |
| natefinch | fwereade: interesting: ./juju bootstrap --constraints root-disk=4G | 21:21 |
| natefinch | ERROR cannot start bootstrap instance: cannot run instances: Volume of size 4GB is smaller than snapshot 'snap-5c0cb05f', expect size >= 8GB (InvalidBlockDeviceMapping) | 21:21 |
| fwereade | natefinch, ha | 21:22 |
| natefinch | fwereade: I guess that answers that | 21:23 |
| * fwereade pretends he was all foresighted and clever earlier | 21:23 | |
| natefinch | haha | 21:23 |
| adam_g | hi.. the release notes for 1.15.0.1 WRT logging are a bit unclear. where is the correct place to look for logs on a unit now? or are hooks no longer logged locally if no log config has been set pre-deploy? | 21:45 |
| natefinch | adam_g: you're on at an awkward time for Juju devs. Our New Zealand guy who is usually on now is on vacation this week. I'm about to head out, and I'm not sure about the answer to your question. There's a couple Australian guys that should come on in the next few hours to answer your question - axw and davecheney | 21:58 |
| natefinch | adam_g: fwereade is european time, so it's pretty late for him, but he might be able to answer that question... he was afk last I knew, though | 21:59 |
| adam_g | natefinch, thanks. found this: https://lists.ubuntu.com/archives/juju/2013-September/002998.html which explained it well. might be worth linking that from any release notes out there | 21:59 |
| natefinch | adam_g: ahh, cool | 21:59 |
| natefinch | adam_g: heh, that's from the New Zealand guy who is on vacation. I must have missed that email. | 22:00 |
| adam_g | right | 22:01 |
| adam_g | me too :) | 22:01 |
| natefinch | Gotta run, dinner's ready. Glad you found the answer to the question. | 22:04 |
| bigjools | morning! | 22:26 |
| davecheney | yay end of daylight savings | 22:51 |
| davecheney | now we get a sensible crossover with the US | 22:51 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!