=== gary_poster is now known as gary_poster|away [02:23] * thumper misses list comprehension in go [03:02] thumper: mr ocr, i have a branch which hooks up simplestreams mirrors support for tools https://codereview.appspot.com/13952043 [03:07] for fuck sake, out landing bot has been shot down [03:07] shut [03:07] ah maintenance i think [03:13] hi wallyworld_ [03:13] hi [03:13] I'll look shortly [03:13] i hope canonistack is back soon [03:13] np [03:20] wallyworld_: I thought it was back already? [03:20] bradm: when i nova list it says the instances are shutdown [03:20] +--------------------------------------+----------------------+---------+-------------------------+ [03:20] | ID | Name | Status | Networks | [03:20] +--------------------------------------+----------------------+---------+-------------------------+ [03:20] | 4829b364-72ad-4ee7-a21c-3ba640f28854 | juju-gobot-machine-0 | SHUTOFF | canonistack=10.55.32.55 | [03:20] | 97a7c226-a195-4014-9df5-c998bba3a491 | juju-gobot-machine-3 | SHUTOFF | canonistack=10.55.32.52 | [03:20] +--------------------------------------+----------------------+---------+-------------------------+ [03:21] wallyworld_: yeah, the compute node being rebooted will do that [03:21] bradm: it would not have been in the procedures to restart stuff that was running? [03:22] wallyworld_: I wasn't directly involved, but it would seem not to be the case [03:22] :-( [03:22] this is the second time our instances have been broken :-( [03:22] you can't just power it on? [03:22] i'm not sure how [03:23] i assume thee's a nova command [03:23] i'll take a look [03:23] nova start [03:23] yeah, trying that now [03:24] bradm: back running, seems quicker perhaps [03:25] wallyworld_: probably, there's likely hardly anyone elses instances going :) [03:25] \o/ [03:26] the compute nodes are pretty beefy machines [03:26] they're just being overcommitted by a lot [03:29] I'll chase up what happened internally, the announcements did say things would be restarted [03:29] but that definately appears not to be the case, or at least not consistantly [03:31] thanks :-) [03:33] wallyworld_: something is wrong with the gobot [03:33] no mongod [03:33] :-( [03:34] i'm not familiar with how it is set up sadly [03:34] we need more monday gods [03:34] mon-god [03:34] yeah [03:34] although stopping and starting should not have affected it you'd think [03:36] fwiw with my dinky little juju test env on lcy02 the reboot didn't break it, its back up and going [03:37] oh good [03:38] thumper: i had a quick look - mongod is in /usr/local/bin and /usr/local/bin is in the path so i'm not sure [03:41] * thumper -> haircut [03:50] thumper, testing saucy local fwiw [03:52] thumper, is there a particular version of interest? trunk i assume? [04:16] thumper, fails for me.. although looks like diff issue, namely the upstart job needs a wait between dropping an upstart template to disk and starting till inotify triggers and register with upstart. [04:32] this is very interesting, a default juju bootstrap on lcy02 fails, since the instance type isn't big enough [04:34] mongodb shuts itself down saying there's not enough space [04:53] bradm: that started happening about a week ago for some reason, i think folks are looking into it [04:53] wallyworld_: I can tell you why [04:54] wallyworld_: the default instance is a m1.tiny, which has a 2G / [04:54] ok :-) [04:54] juju used to be ok in 1G [04:54] or even 512 [04:54] wallyworld_: I just bootstrapped with more, mongodb alone uses 3G [04:54] wallyworld_: its disk thats the issue, not memory [04:54] serious? the landing bot bootstrap machine used to be a 512M instance [04:54] ah disk [04:55] i thought you were talking about ram [04:55] still, juju should not pick tiny on canonistack [04:55] I can't say why mongodb suddenly wants all your disk, but that seems to be the c ase [04:56] bootstrap it with a m1.tiny and you'll see, check the logs in /var/log/mongodb [04:56] it pretty clearly says it needs mor disk [04:56] there's some issue in how juju is choosing the instance [04:56] it used to work. it should be picking small [04:56] i'm not sure of the current status though but it is being looked at [04:56] yeah, not sure where it changed, but thats the fix, to bootstrap with contraints that give you a bigger disk [04:59] is that whats happening with your gobot? needs more disk for mongodb? [05:01] I wonder if mongodb should be using the smallfiles option [05:02] bradm: could be, but it was running fine before the shutdown [05:04] wallyworld_: /var/log/mongodb/mongodb.log should make it pretty clear [05:05] bradm: yeah, true. i'm tied up trying to get some coding finished, but i'll look soon [05:15] wallyworld_: cool, I can do some more testing myself once I've gotten this charm done [05:16] bradm: ok. i'm flat out right now as i'm off from tomorrow for a week and am trying to get everything done before i go. i'll hopefully be ale to look a bit later [05:17] wallyworld_: actually, I'm off next week too :) [05:17] \o/ [05:17] going anywhere? [05:18] yeah, my parents have taken our son for a holiday, we're driving up there to pick him up and spend some time with them [05:18] its one of our first times (outside of hospital) that we've been away from him, its interesting [05:19] how old is he? [05:19] 6 [05:19] yeah, we didn't spent time away from out son for a few years also [05:20] there are medical issues with him too, so we're probably a bit more protective than normal [05:20] yeah, i can understand that [05:20] he had 2 open heart surgeries before he was 5 [05:25] wow [05:25] glad he's ok [05:25] yeah, he's pretty good given what he's had to go thru [05:25] how about you? going anywhere interesting? [05:26] hervey bay to watch whales, then to frazer island for a few days [05:26] looking forward to it [05:27] ahh, nice - I've been to hervey bay whale watching before, lots of fun [05:28] yeah, me too about 10 years ago [05:28] with kid #1. now with kid #2 [05:29] we're starting to think along those lines for holidays as the boy gets older, he might actually get a bit more out of it [05:29] yep. we took kid #1 to nz when he was 4 and he remembers nothing. what a waste [05:30] it'd be pointless for us before now, we always seemed to spend a good portion of the year with him in and out of hospitals [05:32] that's a shame, i hope he gets well asap [05:32] he's been really good this year [05:32] usually a flu would mean a trip to hospital, this year so far things have been good [05:34] \o/ [05:35] ohh, there's two mongodb running in my juju env [05:35] and its the non juju one taking up all the space [05:37] the juju started one has --smallfiles, the other one doesn't [05:38] ah === thumper is now known as thumper-afk [06:13] wallyworld_: as I'm working through some other things, I came across this question. How does "juju bootstrap --upload-tools" work today w/ openstack. Doesn't it put the tools in your private bucket, which should *not* be world readable? [06:13] jam: yes [06:13] it puts them in private bucket [06:14] wallyworld_: right, and both cloud-init and Upgrader just use a "wget" to get the tools [06:14] no Auth [06:14] jam: bot is down since the canonistack maintenance. i haven't had a chance to look deeply, but running tests says it can't find mongod in path, but mongod is in /usr/local/bin and that dir is in the path from what i can see [06:15] jam: it uses a temp url [06:15] which is publically readable [06:15] jam: i've tested bootstrapping with upload tools and simplestreams and it works fine [06:16] unless i've missed smething [06:17] jam: when getting to tools URL, it does a storage.URL() which for environs storage returns a url from which anyone can read [06:17] wallyworld_: we don't have temp urls on canonistakc, IIRC. I'm worried we're actually making our private containers world readable [06:17] wallyworld_: sounds like the bot is using the old tarball [06:17] it should use mongodb from ppa:juju/stable [06:17] wallyworld_: when working it out originally, we decided it was ok that the "public-bucket-url" had to be world readable [06:18] I don't expect it to be any different with tools-url [06:18] but I'm seriously suspecting that we should be able to "juju bootstrap --upload-tools" on Canonistack [06:18] jam: i'll have to check but it all seemed to work ok [06:18] upload tools is now automatic [06:18] wallyworld_: you mean sync-tools ? [06:18] no, upload [06:18] again, I think if it *is* working, we have a security hole [06:19] i don't recall explicitly setting permissions on the control buket [06:21] wallyworld_: "swift stat $PRIVATE-BUCKET" has ".r:*,.rlistings" [06:21] wallyworld_: :( [06:21] hmmm. the tool stuff doesn't set that i'mpretty sure [06:21] wallyworld_: I don't know *who* is setting it, but it is wrong, and it means private tools won't work when we "fix" it. [06:22] wallyworld_: I think the "auto-upload-tools" stuff creates a bucket and sets it world readable [06:22] i'll have to check [06:23] jam: [06:23] containerName: ecfg.controlBucket(), [06:23] // this is possibly just a hack - if the ACL is swift.Private, [06:23] // the machine won't be able to get the tools (401 error) [06:23] containerACL: swift.PublicRead, [06:23] this was put in in january [06:23] by dimiter i think [06:23] wallyworld_: with nobody realizing "you can't get the tools, but you're exposing all of your secrets to the world" ? [06:24] I'm not 100% sure what goes in the private bucket [06:24] as I don't think we put creds there. [06:24] So it *might* be ok [06:24] we put the state file there [06:24] wallyworld_: which is just the IP address, right? [06:24] i'd have to check but i don't think creds go there [06:24] i *think* so [06:29] wallyworld_: I think the only actually private thing is potentially private charms [06:30] As I'm pretty sure we put the charm data in there [06:30] however [06:30] that *also* needs to be accessible via "wget" because of how we removed Environ creds from the Uniter agents. [06:30] so it could be worse i guess [06:31] fwereade: I need to chat with you about this. [06:31] wallyworld_, fwereade: security bug #1231278 [06:31] ok [06:31] (mup won't find it because it is private) [06:32] wallyworld_, jam, reading back [06:32] We just need a discussion, because there is certainly a "vulnerability vs not working at all" that we have to sort through. [06:32] fwereade: G+ might be appropriate [06:33] wallyworld_: bigjools seems to be enjoying himself without you so far :) [06:33] jam, well, I'm here, if the sight of a dressing gown will not be damaging to your sensibilities [06:33] jam: how do you know? [06:33] wallyworld_: he's posting pics of the great barrier reef on G+ [06:33] fwereade: my eyes, my poor, poor eyes [06:33] fwereade: started: https://plus.google.com/hangouts/_/26fdcf993421ca83a1cf0b1a3ddd35772695e493 [06:33] jam: ah ok. that social networking thing i ignore [06:34] fwereade: you could just turn the camera off :) [06:44] https://code.launchpad.net/~dave-cheney/juju-core/158-lp-1210407/+merge/187675 [06:56] axw: thanks for your review, see my comments [06:56] looking [07:04] davecheney: will lgtm, just curious about this: "we don't reboot machines" -- it doesn't work? [07:05] I get your point though - it doesn't really matter [07:05] axw: if you reboot a machine it gets a new ephemeral ip [07:05] and at that point, nothing works [07:09] axw: why do you say twice ? [07:09] I get your point. It just feels wrong to do it twice when it only ought [07:09] to be done once. But, given that's not really possible... LGTM. [07:09] " [07:09] davecheney: *if* it were able to reboot [07:10] it's idempotent though, so doesn't matter. [07:10] axw: fair point [07:11] also, bootcmd http://cloudinit.readthedocs.org/en/latest/topics/examples.html [07:11] does what runcmd does [07:11] it has the same firstboot properties [07:12] davecheney: ok, then the doc comment on juju-core/cloudinit/Config.AddBootCmd is wrong :) [07:12] axw: ok, i'll fix that in a followup [07:13] davecheney: in that page, the comment for bootcmd has a hidden gem: " * bootcmd will run on every boot" [07:14] urgh [07:14] oh well [07:14] care factor, quite small [07:14] this may fix the azure disk suckage [07:15] but while reading that page [07:15] where does it say runcmd is only rnu once ? [07:15] * axw shrugs [07:16] i'm glad we've arrived at this place [07:16] it says it in juju-core, but that's maybe not authoritative [07:17] axw, best I can tell, we've never tried [07:17] everyone knows rebooting an ec2 intsance will screw it [07:18] no worries, it's not a big deal [07:18] fwereade: when you have a moment, would you mind expanding on your comments here? https://codereview.appspot.com/13832045/ [07:19] axw, sure [07:19] I've changed the authentication stuff around a bit to allow HTTP GETs & HTTPS PUTs; wanted to know what you meant first, though, in case I was expending too much effort on this... [07:21] axw, I was just ruminating that *if* cert distribution prove to be some sort of hassle (mainly because the CLI still needs direct storage access for deploy/upgrade-charm) we *could* use ssh storage for the manual provider and filesystem storage for the local one, because the clients that need write access should already have the information needed to set up the appropriate Storage types [07:22] ah, right [07:22] fwereade: does anything other than CLI need write access? [07:22] axw, the API server itself may do [07:23] axw, but (assuming non-HA, anyway) that's doable via the filesystem [07:23] fwereade: ok. it can write directly, given it's local, so that's fine [07:23] yep [07:23] ok, so yes I did expend too much effort [07:23] oh well [07:23] axw, well, you expended it too early, at least [07:23] :) [07:23] axw, but no real harm done, I think [07:25] axw, we would ideally like to not depend on provider storage at all but that's not an immediate plan [07:26] ok [07:27] axw, what we will need to do soon, though, is start exposing storage access via the API, so that an API-only CLI can still upload charms from local repos [07:27] fwereade: also, when you're not busy, would you please look at my latest replies on these two: https://code.launchpad.net/~axwalk/+activereviews [07:27] fwereade: I was wondering if/when storage would be API based [07:27] axw, will do [07:28] thanks [07:34] fwereade: actually, my changes to httpstorage aren't for naught [07:34] they'll allow GETs to not require a self-signed cert [07:34] forgot taht important bit :) [07:34] axw, I don't think they are, indeed [07:34] fwereade: I mean, changes I haven't pushed yet [07:34] axw, ah right -- cool then :) [07:34] I've been changing things today [07:38] axw, https://codereview.appspot.com/13632046/ LGTM [07:39] fwereade: thanks. the error is tested in jujutest/livetests [07:39] axw, I was just thinking of direct tests for New/Is [07:40] fwereade: ok, then no. I'll add some before landing [07:40] axw, cheers [07:42] eh that package has no tests... time to add some [07:48] mornin' all [07:59] morning rogpeppe1 [07:59] axw: hiy [08:00] a [08:00] :-) [08:04] axw, https://codereview.appspot.com/13255051/ nearly LGTM, take a look and let me know your thoughts [08:04] rogpeppe1, morning [08:04] fwereade: yo! [08:04] fwereade: thanks, reading [08:12] fwereade: replying now, but yeah, there is currently no handling of destruction for bootstrap nodes [08:12] the others can be destroyed as usual [08:13] I wasn't really sure where to draw the line with "null" :) [08:24] fwereade: i'd don't quite understand this comment: https://codereview.appspot.com/13912043/diff/1/environs/configstore/disk.go#newcode134 [08:25] Hi jam, hi mgz… would any of you have time to talk about what seems to be a serious bug in the MAAS provider (bug 1229275). [08:25] fwereade: i *think* the only time we add attributes is when we call Prepare [08:25] <_mup_> Bug #1229275: juju destroy-environment also destroys nodes that are not controlled by juju [08:25] rvba: oops! [08:25] rvba: yup, but I'll need to get on a bus in a sec [08:27] rogpeppe1, it's not really actionable, especially in the light of our later discussions [08:27] fwereade: ok, cool [08:27] rogpeppe1, prepare chooses bootstrap-state and writes it; bootstrap uses exactly that [08:27] fwereade: yup [08:27] rogpeppe1, it may involve some light massage of bootstrap responsibilities vs prepare responsibilitie [08:27] rogpeppe1, but nbd [08:27] So basically, juju destroys all the instances he gets back from the provider's instances() method, and that is basically all the instances. [08:28] rvba, that looks like a critical to me [08:28] Critical indeed. [08:28] rvba, how does the maas providers markinstances as controlled by itself? [08:28] rvba: the provider's Instances method should not be returning instances it didn't itself create [08:28] fwereade: it doesn't [08:28] rogpeppe1: that's the problem indeed. [08:28] rvba: the other providers take care to avoid that [08:29] rvba, well, crap -- as someone maasy, how would you recommend we do so? [08:30] fwereade: if this needs to be addressed on the MAAS side, then the easiest way is probably to set a tag on the nodes. A tag identifying the juju environment. [08:31] Out of curiosity, how do the other providers do it? [08:32] rvba: either by looking at the security groups or the name attached to the instances, I believe [08:33] instances that env controls are given names juju-ENVNAME-* [08:36] Right… that's how the Azure provider works too now that I think of it. [08:38] mgz, rvba: fwiw envname is bad [08:39] mgz, rbva: long-term, envname can only ever be a local alias for the actual environment uuid [08:39] it's all a little dodgy, but I don't like a the alternatives much [08:39] mgz, rvba: and we've already had problems with two people using the same env name and same provider credentials [08:40] it's easy to say "don't do that then" [08:40] well, we should check for that on bootstrap and blow up [08:40] right, I need to get on bus [08:40] but that's not as helpful as designing things such that we don't have to do so in the first place [08:41] rvba, I need to take a break for a bit, but... actually just a mo [08:41] rvba, how does juju not destroy those other instances first? [08:42] rvba, the provisioner will be asking for all instances and culling those it doesn't recognise [08:42] rvba, so *starting* a juju environment should also kill everything else [08:42] rvba, as should upgrading it [08:42] rvba, do you know if that's the case? [08:43] fwereade: I just tested it, that's not what happens. [08:43] (I'm testing with the latest trunk) [08:44] rvba, ok, that's weird [08:45] rvba, if it's not culling unknown instances it implies that actually AllInstances is reporting the right ones [08:46] fwereade: that should happen during bootstrap right? [08:48] fwereade: well, you also have to run "juju status" first or it can poll the Provider at all yet [08:48] we've had many "run status and everything dies" bugs :) [08:50] fwereade: I simply tested running "juju bootstrap", is the culling supposed to happen there or later, for instance when the bootstrap node comes up? [08:50] rvba, jam speaks truth, you need to connect once before the bad things will happeb [08:50] rvba, it'll happen when the proviioner starts running [08:50] Okay, testing that now. [08:50] (node is installing) [08:51] rvba, which will happen just after the first command that connects [08:51] rvba, cheers [08:51] axw, responded, let me know what you think of the Destroy error question [08:51] bbiab [08:53] fwereade: yes sorry, I agree Destroy should return an error for now [08:56] fwereade: you were right, culling did happen. [09:02] rvba, well, ok, the good thing here is we don't have to worry about backward compatibility then, because nothing (sensible) we do can make the situation any worse [09:05] axw, this right here ^ is a reason for an EnvironUpgrader that acts directly on the environment (independent of the should-it-hit-state discussion) [09:06] * axw reads back [09:07] axw, short version: maas instances are not tied back to their environment, and getting instances from maas gets *all* instances, not all instances in the *environment* [09:07] * axw nods [09:07] and destroys them all [09:07] fwereade: where's the link to EnvironUpgrader? [09:08] I didn't get much sleep last night, so a little slower than usual today [09:08] axw, sorry, we were chatting about it in the state-upgrades thread [09:09] axw, your contention was that it should connect to state [09:09] axw, I think that's the wrong way round [09:10] axw, *but* that adding an optional upgrade method to environ might be a good idea for other reasons [09:10] fwereade: for example, so you could add a tag to the maas nodes that you control? [09:10] axw, exactly so :) [09:11] axw, or indeed so we could correct the envname problem (above) for the other providers [09:11] fwereade: your latest reply clarifies things for me, and yes, much nicer to not manipulate state from environ [09:11] axw, great [09:15] fwereade: I've updated https://codereview.appspot.com/13255051/ [09:16] okay if I handle Destroy properly in a followup? [09:19] axw, absolutely [09:20] axw, LGTM [09:20] thanks [09:20] fwereade: I'll get the last of the httpstorage stuff in next, then get onto Destroy [09:21] axw, perfect, tyvm [09:21] fwereade: and then Prechecker wireup [09:22] great -- that one's going to be a bit interesting, I think, we should plan how we get it in there ahead of time [09:22] * fwereade bbiab again, see you all atthe meeting [09:23] me too, I need a break. bbl === thumper-afk is now known as thumper [09:48] rogpeppe1: I've realized that I really don't like mornings [09:49] thumper: that's taken you a while :-) [09:49] thumper: i've realised that i forgot (again!) about our chat last night [09:49] My head just isn't in it that early [09:50] I should go check the agenda [10:36] https://bugs.launchpad.net/bugs/1229275 is that actually Critical ? [10:36] <_mup_> Bug #1229275: juju destroy-environment also destroys nodes that are not controlled by juju [10:36] seems High at best [10:36] especially given "nobody is assigned to it" [10:58] fwereade, there it is https://codereview.appspot.com/13963043 - first part, the secrets blanking will follow [11:02] dimitern: the other way around === gary_poster|away is now known as gary_poster [11:18] dimitern, would you take a really quick look at https://bugs.launchpad.net/juju-core/+bug/1229286 ? it feels somewhat likely to be unitery [11:18] <_mup_> Bug #1229286: debug-log and boolean options are broken in trunk [11:19] fwereade, looking [11:19] dimitern, the config bits specifically [11:20] dimitern, may be helpful to confer with TheMue, he was touching config recently [11:20] fwereade, I haven't tried juju set when live testing the api uniter [11:21] fwereade, just debug-hooks and relation-set/get [11:21] dimitern, yeah, I should have thought of that [11:21] dimitern, in fact the stuff you're doing is as critical as this regardless [11:21] TheMue, is there any likelihood you'll be able to look into it this pm? [11:22] fwereade: yep, will do [11:23] fwereade: lunch in a few moments, but then [11:23] fwereade, did debug-log show the hooks output before? [11:23] frankban, hey [11:23] TheMue, cool, thanks, please just verify what's happening with set vs config-changed [11:24] frankban, about that bug ^^ [11:24] frankban, have you tried using debug-hooks instead? [11:24] dimitern: no [11:24] dimitern, frankban: re logging you need to enable that logging in env config now [11:24] dimitern, frankban: thumper knows exactly [11:25] frankban, debug-hooks will show you if config-changed got fired [11:25] dimitern: as I mentioned in the bug description, I am pretty sure that config-changed is called [11:25] dimitern, frankban: it is due to logging changes that were made recently to make things more "productiony" [11:25] bootstrap with --debug [11:26] or --log-config==DEBUG [11:26] thumper: cool, good to know [11:26] or whatever you want [11:26] this log config then propagates to all the agents [11:26] thumper: so, by default, hooks output is not displayed in the debug log, correct? [11:26] ah, good to know [11:26] can be updated using "juju set-env log-config=blah" [11:26] frankban: correct [11:26] only warning and errors [11:27] used to be debug for everything [11:27] I'll write an email for juju-dev tomorrow to explain the changes [11:27] and hooks [11:27] not juju hooks [11:27] but how to do other logging stuff [11:28] dimitern: so, the real bug is about boolean options: it seems they are always set to false [11:28] thumper: thanks for the clarification [11:29] np [11:29] frankban, hmm.. TheMue, can this be relevant to your recent config changes? [11:30] dimitern: should not, only empty settings have been touched [11:30] dimitern: will will take a look after lunch [11:30] * TheMue => lunch [11:35] fwereade, did you have a chance to look at https://codereview.appspot.com/13963043 ? [11:35] dimitern, been in meetings I'm afraid, i'll try to fit in in before Igo forlunch [11:36] fwereade, ok [11:41] dimitern, did we not have an implementation for Upgrader that swapped out 127.0.0.1? [11:41] dimitern, erDeployer [11:41]