[00:00] wallyworld: quick call then [00:00] wallyworld: 1:1 ? [00:00] thumper: didn't you notice the sarcasm? [00:00] no [00:00] ok [00:00] wallyworld: I actually thought that you were looking forward to working [00:17] wallyworld: :p [00:18] rick_h_: living the dream once again, wheeeeee [00:21] hey, you know you missed it! can't miss the finish line [00:21] wallyworld: ^ [00:22] rick_h_: indeed, wasn't my first choice to disappear this last little while [00:25] isn't it one of the relay strategy to put strongest sprinters last? :D [00:25] hah [00:25] good times while away wallyworld ? [00:26] i'm heading out next week. leave to the best of the best at the end :p [00:26] rick_h_: i wish, spent the entire time working my fingers to the bone - ended up having to buy a chain saw to get everything done :-D [00:27] wallyworld: oooh new tools ftw! [00:27] rick_h_: you camping or something next week? [00:27] yeah, new tools :-) [00:27] wallyworld: i wish. snowing today [00:27] i wish we had snow [00:28] wife and i celebrating 10yrs in hawaii [00:28] rick_h_: oh congrats [00:28] so i'll be closer where i can keep an eye on you :) [00:28] i'll wave from my balcony [00:29] new camper is done may 11 so after these may sprints i'll live from the woods for a while [00:31] with no wifi :-( [00:32] nope have a booster antenna for the mifi device [00:32] fatal error: concurrent map read and map write [00:32] goroutine 1660 [running]: [00:32] runtime.throw(0x18945e0, 0x21) [00:32] bzzt, cannot land my 1.25 fix wihtout fixing gomanta [00:32] fairy nuf [00:32] davecheney: that seems ungood [00:40] rick_h_: thumper cherylj menn0 http://reviews.vapour.ws/r/4507/ [00:40] fix is here [00:40] i need to land that before I can land that other fix [00:46] davecheney: fwiw, in master joyent doesn't use manta anymore, still may need to clean up some old tests [00:52] wallyworld: thanks for that [00:52] i need to land this on master to backport it [00:52] then i'll raise a tech debt care to remove the manta library as a dep [00:52] +1, less things to depend on, wheee! [00:52] davecheney: sure, np. the change to not use manta only landed recently, still scrambling to clean everything up [00:53] but yes, less deps is good. and we now don't need provider storage so long as the provider supports tagging [00:55] cherylj, wallyworld, davecheney, sinzui: launchpad is down so the check blockers script is dying at the start of merge attempts [00:55] really? [00:55] looks like i picked a bad day to quit sniffing glue [00:55] lol [00:56] menn0 cherylj wallyworld sinzui: we're working on it [00:56] ty [00:56] did the hamster die [00:56] blahdeblah: ok great [00:57] * menn0 starts following @launchpadstatus [01:01] you must construct more pylons [01:02] menn0: I can comment out the check [01:03] sinzui: pleaes [01:03] sinzui: that would be good but it might not be enough. juju has deps on packages hosted on launchpad. godeps might complain. [01:03] sinzui: it's worth a try though [01:08] menn0: bzr is working so disabling the check might be enough [01:09] sinzui: great [01:09] menn0: http://juju-ci.vapour.ws:8080/view/Juju%20Ecosystem/job/github-merge-juju/7283/console is retying now [01:10] sinzui: looks good so far [01:10] spoke too soon [01:10] :/ [01:10] no I just got godeps [01:11] * menn0 nods [01:11] seems to be stuck at godeps [01:11] progress! :) [01:11] sinzui: it seems to be moving... i'll keep an eye on it [01:11] sinzui: thanks! [01:12] np menn0 [01:13] launchpad.net appears to be back now anyway [01:21] Bug #1568643 opened: RebootSuite times out if unrelated lxd containers exist [01:25] sinzui, I am pretty sure you are a machine that never sleeps [01:25] thanks for the 1.6 work [01:25] alexisb: I don't sleep well :) [01:26] wallyworld: https://github.com/juju/juju/pull/5065 [01:26] looking [01:26] ^ backport to 1.25 which unblocks my other change landing [01:27] lgtm [01:30] danku! [01:40] anastasiamac: reviewed your branch [01:41] axw: thnx \o/ [02:21] grrr... why has lxd suddenly stopped working? [02:24] menn0, we have lots of lxd bugs going around atm [02:25] there are about 3 criticals against juju-core that are impacting lxd/lxd provider [02:26] axw: did you want to take one more look: http://reviews.vapour.ws/r/4502/ ? [02:26] alexisb: yep I know... lxd just stopped working for me in the middle of a complex test reproduction - annoying [02:26] menn0: what happened? [02:27] cherylj: looks good, thanks [02:27] thanks, axw! [02:27] cherylj: I was deploying the dummy charm various models under one controller [02:27] menn0: what were you seeing? [02:27] the first machine came up as normal [02:27] the other were stuck in pending [02:27] no activity [02:28] nothing in juju's logs about it [02:28] nothing in the lxd logs that I could find [02:28] oh fun, I've seen that too [02:28] the machines didn't exen exist when I ran "lxc list" [02:28] menn0: do the machines exist in the database? [02:28] (mongo) [02:28] yes [02:28] they were showing in status so had to have been in the DB [02:29] it's like juju didn't ask lxd to create the instances, or lxd didn't create them for some reason [02:29] ah, I've hit an issue where the services show up, but no machines to host them were ever created [02:29] that sounds like what I just saw [02:29] I mean, they didn't even exist in mongo [02:31] oh right... in my case the machines did show up in status [02:31] so they must have been in the DB [02:32] cherylj: ^ [03:05] https://github.com/juju/juju/pull/5064#issuecomment-208119816 [03:05] it's been a while since tht one failed [03:05] good to see it's as unreliable as ever [03:11] cherylj: are you happy with the response to https://bugs.launchpad.net/bugs/1568602 [03:11] Bug #1568602: Cannot build OS X client with stock Go 1.6

[03:11] we see that report a lot [03:11] and it's always a crapped up go install [03:11] usually by not removing the old version before unpacking the new version [03:12] davecheney: thanks for looking at it. sinzui mentioned he got it working now [03:13] Bug #1568602 changed: Cannot build OS X client with stock Go 1.6

[03:13] Bug #1568654 opened: ec2: AllInstances and Instances improvement [03:14] cool, let me know [03:15] I can provide some suggestios for how to use the upstream tarball concurrently [03:15] if needed [03:15] it's pretty straight forward [03:16] not sure if we'll need that. sinzui ? ^^ [03:16] wallyworld: can you take another look? http://reviews.vapour.ws/r/4504/ [03:16] sure [03:16] build times go up, build times go down. you cannot explain that [03:16] wallyworld: I had to move things around to avoid circular dependencies [03:18] cherylj: lgtm, ty [03:18] wallyworld_: thanks! [03:19] davecheney: getting it working was easy for me but not CI. You were right about more than one tool chain on the host. CI in an effort to purge the env I setup for, disocvered the wrong go tool chain. It found the go I used to compile the go 1.6 we place in a special dir away from the system. All that is resolved now [03:20] okie dokes [03:20] this might be the one time that i actually say "you should set GOROOT" [03:21] but please, don't tell anyone I said thta [03:30] Can I get another review? http://reviews.vapour.ws/r/4510/ [03:30] (pretty easy) [03:33] cherylj: LGTM, ship it [03:33] thanks, davecheney! [03:34] I'll have to get in line... lots of merges lined up :) [03:34] menn0: model-migration merge completed! [03:35] boom! [03:36] cherylj: *\o/* [03:39] wallyworld_: provider/joyent/local_test.go: [03:39] 10: lm "github.com/joyent/gomanta/localservices/manta" [03:39] ^ ja'cuse [03:39] still some code using gomanta [03:40] should I kill it with spite ? [03:40] davecheney: yes, please, all the manta stuff needs to go away [03:40] davecheney: 2.0 should not import gomanta at all [03:41] hulk smash! [03:42] so what does joyent use now ? [03:44] wallyworld_ or axw: can I get some help with a charm storage / GridFS related issue? [03:44] menn0: sure, what's up? [03:44] depends what it is :-) [03:45] so I fixed this: https://bugs.launchpad.net/juju-core/+bug/1541482 [03:45] Bug #1541482: unable to download local: charm due to hash mismatch in multi-model deployment <2.0-count>

[03:45] it turns out the api server had a local cache of charms, which wasn't model specific [03:45] it was easy to fix [03:46] but fixing it has exposed another problem [03:46] if you deploy a local charm to one model [03:46] and then deploy the same charm to another model in the same controller, the unit in the 2nd model can't download the charm [03:47] if the local charm is the same but with slight modification it works [03:47] but not if it's exactly the same charm [03:47] menn0: was this cache added by something in charmrepo? [03:47] i've added lots of debug logging and everything looks fine [03:47] wallyworld_: no the cache is in the charm download API handler [03:48] ok, there's also one in charmrepo from memory [03:48] wallyworld_: it uses a directory to download charms out of storage into [03:49] wallyworld_: the cache assumes that the contents of a charm with a given charm URL is always the same [03:49] wallyworld_: but that isn't true for local charms across models [03:49] menn0 wallyworld_: I think the "binarystorage" index might not be model-specific, checking... [03:49] menn0: i'm not familair with the code - why does the api server need a cache of stuff from mongo? [03:50] wallyworld_: good question. the endpoint supports returning file listings and downloading particular files out of the charm archive [03:50] wallyworld_: I guess the expectation is that there will be many API calls for the one charm archive [03:50] right, so a local cache adds very little that i can see [03:51] i guess so [03:51] wallyworld_: at any rate, the cache isn't the problem here [03:51] is the index model specific? [03:51] wallyworld_: the cache was hiding a deeper bug in charm storage [03:51] menn0: sorry had my wires crossed, that's just for tools... [03:52] axw, wallyworld_: the way charms are stored looks fine but there's obviously a problem [03:52] * menn0 prepares a paste [03:54] menn0: the charns collection should be model aware [03:54] wallyworld_: yep it is [03:54] wallyworld_: the problem isn't there I don't think [03:55] that's where the charm doc goes - the url is not model aware but that shouldn't matter [03:55] wallyworld_: it's in the GridFS put/get code I think [03:55] there's a PutForBucket api [03:55] so long as we set the bucket id to be the model uuid that should be enough [03:56] wallyworld_, axw : http://paste.ubuntu.com/15752687/ [03:56] wallyworld_, axw : that's the output from a bunch of debug logging I added [03:57] wallyworld_, axw : does that help? [03:58] so for the second charm post, the GridFS entry appears to get reused [03:58] oh poo [03:58] but during the download attempt for the second model it can't be found [03:58] my maas update seemed to loose all the power settings for the machines [03:58] menn0: that looks like it's using the old blobstore [03:58] should be blobstore.v2 [03:58] with PutForBucket [03:59] this is blobstore.v2 [03:59] not that it should matter i guess in terms of this bug [03:59] ah [03:59] some internal methods were not renamed [03:59] sigh [04:01] Bug #1568666 opened: provider/lxd: non-juju resource tags are ignored [04:01] Bug #1568668 opened: landing bot uses go 1.2 for pre build checkout and go 1.6 for tets [04:01] Bug #1568669 opened: juju 2.0 must not depend on gomanta [04:04] menn0: it doesn't make sense at first glance - the resource path i think from memory is the raw access to the blob - that should not be affected by wthat happens above with bucket paths etc [04:05] so the resource path should be there - unless something is deleting it [04:05] is it worth putthing debug in the remove methods [04:06] wallyworld_: ok, i'll try that. I have put some in some of the cleanup-on-error defers already but they're not firing [04:06] wallyworld_: i'm also checking how things look in the DB [04:07] menn0: so to be sure i understand - the issue is that the second upload correctly creates resource metadata for the new model uuid etc, and it rightly shares a de-deuped blob with the first upload, but the attempt to access that blob fails [04:07] and it is failing at the point at which the blob itself is retrieved [04:07] wallyworld_: spot on. that's my best understanding at the moment [04:07] yes [04:07] hmmm, so yeah, that blob should be there unless deleted [04:08] this should all be orhtogonal to any bucket uuid stuff [04:08] yeah, it seems to be passing the right path to gridfs .. [04:08] based on the error [04:09] looking at the storedResources collection and the blobstore db directly, everything looks ok [04:10] jeez, well that sucks [04:10] i'll add more debug logging in the lookup side of things [04:11] at least the storage side looks correct - it is using the model uuid correctly [04:11] and de-duping across models [04:15] wallyworld_: it certainly *looks* like it's doing the right hting [04:15] still could be a subtle issue i guess [04:16] yep [04:16] i've just added a lot more logging on the get side of things [04:25] wallyworld_, axw: I think I've found it [04:26] wallyworld_, axw: the GridFS instance is created with the model UUID as the prefix [04:26] wallyworld_, axw: so even though the charm is being requested with the correct de-duped UUID [04:27] wallyworld_, axw: the GridFS prefix prevents it being found [04:27] the namespace? [04:27] wallyworld_: yes [04:27] does that seem right? [04:27] so maybe the namespace should be the controller uuid [04:27] i think so [04:28] this was all done pre-multi model [04:28] Hi! Am I in the right place to ask about filing a bug against Juju? I'm kind of new to the whole FOSS community thing and I have *no idea* how to use Launchpad. [04:28] arcticlight: sure, ask away [04:29] and welcome [04:30] menn0: does each controller get a different uuid in HA? [04:30] wallyworld_: naively fixing this by changing the charms code to use a statestorage with the controller's UUID will no doubt break existing deployments [04:30] i think so right? [04:30] menn0: it will break existing yes [04:30] wallyworld_: It's actually really simple... Juju doesn't seem to depend on `curl` but goes ahead and uses it anyway. But on Ubuntu Server 14.04 LTS it's not installed by default, or at least it wasn't on my system (I have a fresh install) and `juju bootstrap` blew up. Thought I'd mention it [04:31] wallyworld_: in HA the controller is really the whole cluster [04:32] wallyworld_: there is only one controller model and one UUID [04:32] wallyworld_: so that's not a problem [04:32] arcticlight: yes, it does use curl - to retrieve the tools binaries at bootstrap. i'm surprised 14.04 lts doesn't have it out of the box, but i'm not a packaging guy [04:32] arcticlight: thanks for mentioning, i'll followup with someone who knows more about ubuntu default packaging and see if we need to fix anything [04:32] wallyworld_: Yeah. I fixed it by installing curl, but it did blow up on a fresh install. I figured someone should know so they can add a dependency on curl. [04:33] wallyworld_: Welcome! [04:33] arcticlight: just ask if there's anything else you get stuck on. lots of folks here who can help [04:34] menn0: so the "easy" option is to use the controller uuid with gridfs namespace. i'm ok with release notes telling poeple they need to re bootstrap between betasa, but that's IMHO [04:34] wallyworld_: OK! Good to know. I've been fooling around with MAAS/Juju since around 2013. I'm just really shy lol~ Figured this was simple enough to just pop on and ask about tho [04:35] we don't bite [04:35] unless really provoked :-) [04:36] wallyworld_: what about 1.25 systems? [04:36] menn0: we have sooooooo many tings to fix with upgrades there [04:36] this is just one more [04:37] menn0: upgrading from 1.25 will be really, really difficult [04:37] already [04:37] menn0: i suspect we'll considering migrayions :-) [04:37] migrations [04:37] might be easier in the long run [04:38] wallyworld_: you realise that would mean backporting migrations to 1.25? [04:38] yes, or a form thereof [04:38] that will likely be so much easier that the alternative [04:38] jeez wallyworld_, u made thumper quit with all this upgrades talk [04:38] wallyworld_: I've realised for some time that this was on the cards :-/ [04:38] na, i just bit him hard [04:39] menn0: i think we all have :-) sort of a slowly increasing dread [04:40] dread != anticipation [04:43] dread is more accurate :) [04:43] * menn0 tries the naive fix [04:47] sure... but when u dread problems become harder and motivation disappears... when u anticipate, besides being prepared, there is always something to look forward to.. [04:58] wallyworld_: were you proposing we pass the controller UUID to charm related calls to state/storage.NewStorage() or were you proposing that we use the controller UUID in the call that stateStorage makes to blobstore.NewGridFS()? [04:59] menn0: I think the latter [04:59] menn0: i think(?) just the latter is all we need [04:59] that would be my preference [04:59] so they're all sharing a common gridfs namespace, and we have model-specific catalogues that point into it [04:59] yup [05:01] wallyworld_: cool that makes sense (the latter, not the former). I misunderstood what you meant the first time around and then realised it didn't make sense as I started to change all the calls to NewStorage() [05:01] oh sorry :-) [05:02] i should have been more clear [05:02] wallyworld_: no it was my bad [05:03] wallyworld_: use the controller UUID over a fixed value? the controller UUID isn't readily available in stateStorage [05:03] menn0: isn't there a helper method on state? you don'r have access to that in stateStorage? [05:03] wallyworld_: no it gets a MongoSession, not a State [05:04] wallyworld_: I can rejig things [05:04] where does it get the model uuid from then? [05:04] wallyworld_: or use a fixed value "juju" or "state" or "jujustate" [05:04] the model UUID is passed in with the MongoSession [05:04] 2 args [05:04] so can't we change that to controller uuid? [05:05] menn0 wallyworld_: may as well just use a constant, it doesn't really need to be the controller UUID [05:06] it just needs to be the same for all views on the db [05:06] except if we want to share a mongo instance between controllers [05:25] Anyone know if there are moves afoot to introduce DNS as a Service support into either juju itself or the charm store? [05:26] wallyworld: i've just tried using a fixed GridFS namespace ("juju") in state/storage.NewStorage() and that fixes the problem [05:27] menn0: great. if it's easy to get controller uuid in there that would be good [05:28] wallyworld: your reasoning for that is to avoid any chance of collision in case of a mongodb instance being shared by multiple controllers? [05:28] menn0: yeah [05:28] we just don't know what might need to be supported in the future [05:28] wallyworld_: but that would break anyway... our main db is called "juju" [05:29] and the collection names in it are fixed [05:29] well, we use model uuids though [05:29] i guess "juju" is ok for gridfs [05:30] wallyworld_: I was about to allow my mind to be changed :) [05:30] quick win for now [05:31] menn0: i guess there's pros and cons. maybe if it's easy to do.... [05:31] wallyworld_: if we were to do it then it would probably make sense to have NewStorage just take a *State [05:31] it could then pull the model uuid, controller uuid and session off that [05:31] sound ok? [05:32] it looks like all the call sites have a *State [05:33] menn0: let's use a bespoke interface that just declares the state methods it needs [05:33] pass in the concrete state.State from the caller [05:33] but declare the NewGridFS() method to use a smaller interface [05:34] or just add a controller uuid param [05:34] wallyworld_: ok sounds good. i'll pass in state but via an interface [05:35] blahdeblah: i don't think there's anything coming in core, but i thought the ecosystems/charm guys had something for juju itself they use [05:35] wallyworld_: state doesn't currently have a ControllerUUID method but i'll add one [05:35] menn0: i though it did? [05:35] wallyworld_: nope. there's ControllerModel and you can get it from there but that's messy [05:35] wallyworld_: there's a unexported controllerTag field so I'll just expose the UUID via that [05:36] like we do for ModelUUID() [05:36] * menn0 bids [05:36] nods even [05:44] wallyworld_: thanks - will ping them [05:50] wallyworld_: I just noticed that state/imagesstore uses a fixed "osimages" prefix [05:51] menn0: ah yes. i think the thinking there was that it is ok to share cached generic lxc images between controllers [05:51] wallyworld_: and state.ToolStorage uses the model UUID [05:52] hmmm, so tools storage could break the same way maybe? [05:52] * menn0 wonders if ToolsStorage could end up with the same problem where an entry is inaccessible [05:52] haha [05:52] yes I think so [05:52] wallyworld_: it's probably hard to trigger at the moment [05:52] menn0: do we even need a namespace? [05:52] wallyworld_: can you upload tools for a hosted machine [05:53] ? [05:53] um [05:53] i think the controller stores all the tools tarballs for the hosted models, would need to check [05:53] wallyworld_: I guess if you do juju upgrade-juju --upload-tools for 2 hosted models you might hit the same problem [05:54] not sure, but sounds plausible doesn't it [05:54] if you used exactly the same tools [05:54] yes, that's they key - need to be the same [05:54] wallyworld_: does upload-tools recompile the tools [05:54] ? [05:54] depends - if it finds binaries in the path, then no [05:54] is my recollection [05:55] wallyworld_: ok. if it did recompile then the odds of the binary being exactly the same is slim [05:55] yes [05:55] but let's not count on it [05:55] wallyworld_: regarding whether we need a namespace, I think you need to provide soemthing. [05:56] wallyworld_: the docs say the convention when there's only one namespace required is to use "fs" [05:56] wallyworld_: I think it still makes sense to use a namespace for different uses of gridfs [05:56] menn0: so maybe we just use a const "osimages", "tools", "charms" etc? [05:58] wallyworld_: yep. [05:58] wallyworld_: or we use the controller UUID... [05:59] wallyworld_: no a prefixes per use seems better [05:59] menn0: i think not looking at the different usages - namespaces for images, charms, tools etc seems ok [06:00] we still differentiate by model uuid inside each namespace [06:00] wallyworld_, axw: why does ToolsStorage and GUIStorage not use state.NewStorage()? [06:00] what's different about binaryStorage? [06:01] menn0: not sure, i know someone renamed binary storage recently, but can't recall the details [06:02] i can recall the specifics ottomh [06:02] wallyworld_: ok. well I'll avoid fixing the world. [06:02] not in this pr :-) [06:02] next one [06:04] wallyworld_: thanks for your help and input [06:04] * menn0 is gone for now [06:04] menn0: np, didn't do much :-) [06:04] thanks for fixing [06:04]