[00:54] anyone here want to help debgging a manual provider issue ? [00:54] http://paste.ubuntu.com/7144037/ [00:54] wedged here [00:58] hmm, wedged waiting on mongodb [00:58] but mongodb isn't running [00:58] .... [00:58] juju-db start/running, process 4156 [00:58] 2014-03-24 00:52:01 DEBUG juju.environs.bootstrap state.go:73 loading "provider-state" from "file:///var/lib/juju/storage/provider-state" [00:58] 2014-03-24 00:52:01 DEBUG juju.agent agent.go:318 read agent config, format "1.18" [00:59] 2014-03-24 00:52:01 DEBUG juju.agent bootstrap.go:63 initializing address [localhost:37017] [00:59] 2014-03-24 00:52:01 INFO juju.state open.go:79 opening state; mongo addresses: ["localhost:37017"]; entity "" [00:59] 2014-03-24 00:52:01 DEBUG juju.state open.go:99 connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused [00:59] the first line is a lie [01:03] root@winton-06:/etc/init# status juju-db [01:03] juju-db stop/waiting [01:03] what ? [01:03] why did it stop ? [01:05] davecheney: is juju-mongodb installed? [01:05] yes it is [01:05] root@winton-06:/etc/init# stat /usr/lib/juju/bin/mongod File: ‘/usr/lib/juju/bin/mongod’ [01:06] * davecheney goes spelunking for log files [01:07] 2014-03-24 01:02:01 DEBUG juju.utils.ssh ssh_openssh.go:147 running: ssh -o "StrictHostKeyChecking no" -o "PasswordAuthentication no" -i "/home/ubuntu/.juju/ssh/juju_id_rsa" "ubuntu@10.245.67.6" 'sudo pkill -6 jujud' [01:07] ^ why sudo pkill [01:07] -6 is an odd signal [01:07] sudo killall jujud ? [01:14] https://bugs.launchpad.net/juju-core/+bug/1296485 [01:14] <_mup_> Bug #1296485: manual provider fails to remove itself if bootstrapping fails

[01:16] ok, here is theprobme [01:17] root@winton-06:/etc/init# start juju-db [01:17] juju-db start/running, process 4660 [01:17] root@winton-06:/etc/init# status juju-db [01:17] juju-db stop/waiting [01:17] juju-db is shutting down [01:18] davecheney: should probably use killall, but we use a particular signal to tell the machine agent to uninstall itself [01:18] axw: http://paste.ubuntu.com/7144083/ [01:18] oh crap [01:18] root@winton-06:/var/log/upstart# getconf PAGESIZE [01:18] 65536 [01:18] FUCK [01:19] :\ [01:19] is someone working on a patch to mongo? [01:20] * davecheney looks [01:20] ill go an ask in #ibm [01:21] seems kinda important [01:21] no thumper ? [01:22] * axw shrugs === Guest84660 is now known as bodie_ [01:25] tim is away today - public holiday [01:27] (╯°□°）╯︵ ┻━┻ [01:31] wallyworld: it'll be a lonely standup - got anything to talk about? [01:32] axw: touch base on that upstart logging thing? [01:32] wallyworld: sure [02:09] wallyworld: I was just thinking we could bind mount /var/log/juju- in the containers, and symlink the whole thing as ~/.juju/local/log. What do you think? [02:09] * wallyworld thinks [02:10] i like that all the files or kept together that way [02:10] maybe I should check with thumper before making that change [02:10] it's not that big a change to implement [02:10] yeah [02:10] i'd be inclided to do it [02:11] i think it makes sense [02:11] I'll have a poke in a bit [02:11] i don't like how some files are symlinked and others aren't [02:15] wallyworld: reviewed. sorry, you'll have a few files to merge [02:15] mine just landed [02:15] np at all [02:15] thanks for reviewing [02:57] davecheney: hi, thanks for reviewing. that SSLHostnameVerification code is copied across from another package so it not new to this mp. it's a pattern people introduced to avoid passing true/false as params which give no indication as to what's happening [02:57] wallyworld: fair enough [02:57] right, but that doesn't mean you need a new type [02:57] just a well named constant [02:57] i think it was done to stop people passing in true/false [02:57] it doens't prevent that [02:57] ie force them to use the const [02:58] won't work [02:58] doesn't work like that [02:58] i'm just guessing, i didn't write the code [02:58] maybe it was done for clarity? [02:58] see previous comment about constant name [02:58] anyway [02:58] feel free to ignore that suggestion [02:58] it was a minor nit [02:59] that change is a signficant improvement [02:59] np, thanks, i'll read the comments and see if i can improve things [03:00] the hostnameVerification variable name tries to hide the bool nature of the values and reinforce the use of the constants [03:00] yes, but it doens't work [03:01] it's meant as a visual aide to reading the code [03:01] feel free to ignore that suggestion [04:09] axw: wallyworld: Is thumper still around? [04:09] jam1: public holiday I'm told [04:09] axw: ok, thanks. we had a phone call scheduled, but I'm happy to go have breakfast instead. :) [04:11] I see the holiday on the calendar now. [04:12] jam1: how do holidays end up on there? [04:12] axw: people add them [04:12] ok [04:12] This is the Juju team calendar, but it looks like you aren't set with edit rights [04:12] I'll fix that now [04:12] jam1: thanks [04:13] axw: your @canonical account can add and modify things in the calendar now [04:13] cool, ta [04:15] jam1: wallyworld says you were able to reproduce #1295501. after breakfast, would you mind running bootstrap with strace so I can try to get to the bottom of it? [04:15] <_mup_> Bug #1295501: local provider upstart script broken

[04:15] (assuming you can still repro) [04:15] axw: I saw the file come up pointing at a /var/log that I didn't have. [04:16] jam1: yeah, the file should be there - something must've gone wrong early on in bootstrap [04:16] axw: *bootstrap* seemed to work, but restart failed [04:16] silently wrong I mean [04:16] ah, but you think it "failed" but didn't actually stop [04:16] yup [04:49] https://bugs.launchpad.net/juju-core/+bug/1235529 [04:49] <_mup_> Bug #1235529: Local provider does not respect tools-url

[04:49] this can be closed, right ? [04:49] we've changed that logic a bazillion times [04:49] that bug report can't be accurate anymore [04:50] ubuntu@winton-02:~/src/launchpad.net/juju-core$ uname -a [04:50] Linux winton-02 3.13.0-8-generic #28-Ubuntu SMP Mon Feb 17 08:22:39 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux [04:50] crap, ignore [04:52] davecheney: I don't think so, the local provider makes assumptions that it's bootstrapping tools with version.Current with buildnumber++, IIRC [04:53] fairy nuf [05:53] jam: thanks for the email. copy&paste last line of my reply: "Looking at your log dirs, though, I see they both have the files (machine-0.log). So is the restart bug still occurring...?" [05:54] I haven't seen that the restart problem is still happening, I got stuck noticing that it didn't clean up after itself, and that strace -f isn't working. [05:56] axw: I'll bootstrap and then poke machine-0 real quick [05:56] thanks [06:06] how the FUCK do I godeps [06:07] fucking every time have to fight with this tool [06:07] why is the head of launchpad/golxc broken ?!? [06:08] davecheney: godeps -u dependencies.tsv [06:09] I think it doesn't like it when repos are dirty though [06:09] in that case I sometimes just bzr update -r [06:09] axw: manually killing jujud w/ SIGTERM, and using restart juju-agent-jameinel-local all seem to bring the agent back up. [06:09] why should it care if the repo is dirty [06:10] the repo I care about isn't the one that is dirty [06:10] jam: bugger, no repro then :/ [06:10] davecheney: welcome to golangs EXCELLENT dependency management :-( [06:10] wallyworld_: i walked into that one didn't i [06:10] yep :-) [06:10] davecheney: yeah I know, it would be nice if you could specify just one [06:10] or cat them in or something [06:11] wallyworld_ jam: in any case, this CL should make local provider bootstrap a little simpler and more robust: https://codereview.appspot.com/79310043/ [06:12] looking [06:17] incidentally, I have a feeling this fixes #1294776 [06:17] <_mup_> Bug #1294776: No debug-log with MAAS, 14.04 and juju-core 1.17.5

[06:25] wallyworld_: I did at a test... was there something in particular you think was worth testing? === vladk|offline is now known as vladk [06:26] axw: ah sorry, totally missed it. [06:27] nps [06:27] i could have sworn i looked at allt he files [06:28] wallyworld_: good to land then? [06:52] axw: sorry, went to pick up kid. yeah [06:53] axw: one thing i didn't look at closely - will it handle existing local dirs with root ownership etc? or maybe we just require a destroy-env first? === vladk is now known as vladk|offline === vladk|offline is now known as vladk === vladk is now known as vladk|offline [07:17] wallyworld_: there are some cases where it's necessary to destroy-env --force, but there's no change there [07:18] can't remove them if you're not root [07:19] ok [07:19] just thining out loud [07:31] morning jam, 1-1 ? === vladk|offline is now known as vladk [07:57] good morning [08:01] mornin' all [08:01] vladk, dimitern, wallyworld_, axw, jam: hiya [08:01] rogpeppe: hello [08:02] morning vlad, rogpeppe [08:02] rogpeppe: morning [08:02] and vladk [08:12] rogpeppe, hey [08:13] morning vladk [08:13] dimitern: morning [08:22] rogpeppe: g'day [08:22] wallyworld_: yo! [08:22] how are they hangin'? [08:22] i'm looking for a review of this, if anyone has a little time: https://codereview.appspot.com/78890043/ [08:23] * wallyworld_ is about to eat dinner sorry [08:24] wallyworld_: pretty good, except i'd worked out a great pair programming workflow with voidspace and now he's gone and got sick :-( [08:24] you'll just have to do it with yourself [08:24] * wallyworld_ has never tried pair programming over irc before [08:25] wallyworld_: google hangouts, each person with two screens, each sharing their own screen [08:25] wallyworld_: and each with their own branch; you constantly push to your own branch and merge the other person's [08:26] wallyworld_: makes it very easy to move concurrently on micro-tasks, which seemed to be working very well [08:27] it wouldn't work well on every kind of task though [08:28] rogpeppe, have you tried termbeamer or tmate for console sharing ? [08:32] sounds like a good workflow for sure [08:38] dimitern: it wouldn't work great for me, as i don't use a terminal emulator [08:38] rogpeppe, oh, acme, right [08:39] dimitern: or any other editor that's not rooted in the 1970s :-) [08:39] rogpeppe, yeah :), well emacs is from 1970s too [08:40] dimitern: indeed so [08:40] dimitern: well, written in the '80s i think [08:42] dimitern: no, it seems it really did originate in the '70s [08:43] rogpeppe, 1976 is the first version [08:44] dimitern: i'd appreciate it if you could review this (some new API additions and client refactoring): https://codereview.appspot.com/78890043/ [08:47] rogpeppe, sure, looking [08:48] dimitern: ta! [09:00] jam1: 1-1 ? [09:01] rogpeppe: brt, switching machines, taking the dog outside [09:01] jam: k [09:03] jam, how's the dog doing? :) [09:04] dimitern: pretty good, but I can't quite leave her by herself downstairs while I'm upstairs on the phone call [09:05] :) [09:07] she knows to not pee inside, but I can't quite trust that I'll notice she needs to go out in time. [09:16] rogpeppe, reviewed [09:43] dimitern: "Why using a const only here? I'd pick one - always use a const for a facade name or never." [09:44] dimitern: which places are you looking at that that aren't using a const for the facade name? [09:44] rogpeppe, deployer i think [09:45] dimitern: actually, there are quite a few. i'd decided that i wasn't going to change all of them, because it makes the CL even bigger. [09:45] dimitern: i think some consolidation in a subsequent CL might be appropriate [09:55] rogpeppe, i'm fine with that [09:59] morning all [10:00] morning mgz [10:00] hello [10:02] morning mgz [10:02] mgz, you have LGTM on your networks-in-state branch [10:02] dimitern: ta [10:06] dimitern: I'll land that and do follow up fast with test and the function params change [10:06] mgz, cheers! [10:20] axw: nice job figuring out bug 1294776, I was able to reproduce it finally making that change to my environments.yaml [10:20] <_mup_> Bug #1294776: No debug-log with MAAS, 14.04 and juju-core 1.17.5

[10:26] wwitzel3: cool. sorry for stealing all your bugs - I just stumbled across this one [10:26] axw: no it is great, I couldn't even reproduce it, didn't even think to edit the yaml file. [10:33] mgz, dimitern: were there more tests on that branch that I didn't see? [10:37] fwereade, what tests? [10:37] dimitern, unit tests for services without network docs [10:38] fwereade, good point - mgz could add these in the follow-up changing AddService perhaps? [10:38] mgz, you could do this by adding a service document in state directly and then fetch it with the state public methods [10:39] dimitern, mgz: they're only necessary once there are clients that actually ask for the networks, I guess [10:39] dimitern, mgz: they're not optional but I guess they don't have to land with that branch [10:42] fwereade, +1 [10:43] mgz, take a look at state/compat_test.go:TestEnvironAssertAlive for an example how to add something directly bypassing the public state methods [10:45] mgz, fwereade, jam1, rogpeppe, wallyworld_, standup? [10:46] vladk, ^^ [10:46] fwereade: I am missing the test you said, but I do know how to write it and will include it in the followup [10:47] mgz, great, tyvm [10:47] fwereade: so landing that bit of code was blocking dimitern getting the next step done, so I said to go ahead and land it, and we can fix in post [10:47] fwereade: standup ? [10:47] jam1, np, it's safe until that method has a client [10:47] :) [10:48] jam1, hangouts are being difficult, will join as soon as I can [10:48] fwereade: np, we'll start without you === dimitern is now known as dimitern|lunch === vladk is now known as vladk|lunch [12:10] fwereade: re "Taking DG into account when choosing pre-existing instances" - I wasn't planning on doing that until I get to ec2 AZs, is that okay? [12:11] i.e. I'd implement Azure AS support first without it [12:11] oh wait [12:12] I think we did agree on using it for Azure too... it would just always say create me a new machine === dimitern|lunch is now known as dimitern [12:21] rogpeppe: can you please take a quick look over https://codereview.appspot.com/75250043/patch/100001/110003 ? [12:21] axw: looking [12:22] axw: what's the ManagerMachines call actually used for? [12:23] rogpeppe: for grouping manager machines into an availability set [12:23] or spreading across availability zones [12:23] axw: manager instances. presumably? [12:23] rogpeppe: we need to know where the existing ones [12:23] rogpeppe: yep [12:24] rogpeppe: this is to avoid AllMachines + IsManager for each, which isn't so scalable [12:25] axw: axw: so... i've recently added an APIHostPorts call, which gets information on the API addresses of the current state servers. [12:26] axw: but i've realised that [12:26] axw: i think i want the instance ids in there too [12:26] axw: so istm that perhaps that info is pretty much what you need [12:26] axw: and it would save you an O(n) scan through the machines [12:27] rogpeppe: the code that uses ManagerMachines does different things depending on the input entity - so I really don't want to use another API [12:27] rogpeppe: or is it another method on state? [12:27] axw: it's another method on state [12:27] ah ok [12:27] axw: but... [12:28] axw: yeah, i think it could work actually [12:29] rogpeppe: that sounds fine, I will change over to using that before I land then [12:29] rogpeppe: thanks [12:30] rogpeppe: will you be adding instance IDs for HA stuff, or will I need to do that? [12:30] axw: how about we have a call that returns []struct {InstanceId instance.Id; APIHostPorts []instance.HostPort; StateHostPorts []instance.HostPort} ? [12:30] axw: hmm, there might be an issue actually [12:30] axw: because the address info isn't set immediately [12:31] ah, yeah, that would be a problem [12:31] axw: but presumably you need to know immediately what instance ids correspond to state servers [12:31] rogpeppe: yep [12:34] axw: ok. but, i just realised - i think you can use the existing StateServerInfo call as is [12:34] * axw looks === vladk|lunch is now known as vladk [12:36] rogpeppe: heh yeah, looks like it. thanks :) [12:36] bbl, I'll have abetter look later [12:40] natefinch: you around? [12:40] wwitzel3: he just said he had to go deal with kids for a bit [12:41] jam1: thanks [12:50] axw, that's fine [12:50] axw, I'm just obsessively reminding you that it's part of the stream of work :) [12:51] rogpeppe: it does require additional state.Machine() calls for each... but I suppose that's not so bad, since there can only be 7 state servers max [12:51] fwereade: cool [12:52] axw: i wonder if it might be nice to add a State.Machines(ids []string) ([]*Machine, error) call [13:06] rogpeppe: maybe, though I'm less inclined to expand state now that I know the upper bound is so small [13:06] axw: yeah [13:07] morning all [13:07] bodie_: hiya [13:30] wwitzel3: back now [13:31] natefinch: k [13:48] rogpeppe, mgz, fwereade, a quick review? machine networks in state https://codereview.appspot.com/79250044 [13:49] dimitern: in a little bit [13:49] rogpeppe, ta [13:56] dimitern, rogpeppe, fwereade: how do we log into the mongo db for localhost? Is it special? I swear just using the SSL key on AWS worked, but I can't get it to work on local [13:56] dimitern, I'm having some difficulty [13:56] dimitern: looking [13:56] dimitern, Machine.Networks calls readNetworks [13:56] natefinch: there are two levels of authentication [13:56] dimitern, which NotFounds if the doc is not there [13:56] dimitern, right? [13:57] dimitern, but the tests imply that doesn't happen [13:57] natefinch: i'm not sure what you mean by the ssl key though [13:57] dimitern, I am 100% for returning {no networks, no error} if the doc is not there [13:57] dimitern, but I don't see where that's tested [13:58] pwd [13:58] fwereade, right [13:58] fwereade, the doc is always created [13:58] dimitern: I'm not sure about the past tense for those variable names [13:58] fwereade, at machine creation time [13:58] dimitern, in upgraded environments that didn't happen [13:58] mgz, we should pick one and use it throughout [13:58] dimitern, we need tests for that behaviour on machine and service [13:58] fwereade: yeah, that's on my plate [13:58] fwereade, will add them, ok [13:59] rogpeppe: I had thought the SSLPEMKeyFile was for authorization, but now that I reread it, I guess it's probably not [13:59] dimitern, mgz: only one of you needs to do it -- would be ideal for dimitern to since he consolidates the fields [13:59] okay [13:59] er funcs [13:59] dimitern, ok? [14:00] fwereade, will do [14:00] dimitern, <3 [14:15] dimitern: further state/api changes, as suggested by yourself: https://codereview.appspot.com/79470043/ [14:15] * rogpeppe needs some lunch [14:29] natefinch: how are you getting on? shall i come and join you? [14:29] does he get to share your lunch? :) [14:30] mgz: nae chance [14:30] not that he would like it anyway, i reckon [14:32] rogpeppe: heh. I like most things. What are you having? Also, I think we're doing ok right now... it looks like part of the problem is that for replica sets you're supposed to use --keyfile instead of --auth for authentication (according to mongo's docs) [14:32] natefinch: you have to use both [14:32] rogpeppe, LGTM [14:32] dimitern: ta [14:33] rogpeppe: you're welcome to join us. The docs say --keyfile implies --auth, so I guess it is using both. [14:33] rogpeppe: https://plus.google.com/hangouts/_/76cpjfeph92ud9qfcn92ant4io?hl=en [14:34] natefinch: thickly sliced wholemeal bread, oliver oil, encona chilli sauce, extra mature cheddar, sliced tomatoes, as a sandwich [14:34] rogpeppe: that sounds awesome === hatch__ is now known as hatch [15:42] fwereade, mgz, next in line - networks in status (via api server) https://codereview.appspot.com/76400049 [15:43] fwereade, can you please expand on https://codereview.appspot.com/79250044/diff/1/state/networks.go#oldcode17 I'm not sure I follow you [15:46] dimitern, so, those docs will be stored in mongo with {"networkstoinclude": [blah, blah], "networkstoexclude": [blah, blah]} [15:46] dimitern, and all those field names will be in every document [15:46] fwereade, so how about `bson:"included"` and `bson:"excluded"` ? [15:47] dimitern: will look [15:47] dimitern, and while I wouldn't normally be one to jealously count every byte... -- yeah, that's perfect [15:47] fwereade, ok, will do [15:47] dimitern, I don't think we want those machine networks in status [15:48] dimitern, similarly to how we don't have machine constraints in status [15:48] dimitern, we'll want the networks the machine is actually on in reality, but we only know that post-provisioning [15:48] dimitern, we do want included/excluded networks in *service* status though [15:49] fwereade, well, the card was about machines [15:49] dimitern, so, we do want to show machine networks [15:49] dimitern, but not those networks [15:49] fwereade, and we don't have a similar card for services [15:50] fwereade, so what then? [15:50] dimitern, those are the requested networks (like constraints) and we want to show the actual ones (like hardware characteristics) [15:50] fwereade, a separate couple of fields on machineDoc the provisioner sets? [15:50] dimitern, I think we decided it'd be the machiner facade [15:51] dimitern, er *not* the machiner facade [15:51] dimitern, but somemachine-agent-specific one [15:51] dimitern, ie the machine agent comes up and asks the state server what networks it should set up [15:51] dimitern, the state server asks the provider what vlans are there, and caches that info on the machine in state somehow [15:52] dimitern, (*probably* another little document rather than tacking it onto the machine, usual forces still apply there) [15:52] fwereade, ok then, so we need to talk about this some more tomorrow [15:52] dimitern, the impact of that is (1) the machine agent sets up the desired networks [15:52] fwereade, i'll finish the rest today [15:52] dimitern, and (2) those networks are stored in state ready to be included in status [15:53] dimitern, if you would do exactly that CL for the service status, though, I would be happy [15:53] dimitern, if it needs a card, add one, sorry we missed it before [15:54] in juju-core trunk are there any known bugs with ec2 `juju add-machine --series saucy`? [15:54] it's been sitting as 'pending' in `juju status` for a good 15 minutes now [15:55] fwereade, adding one now [15:55] which is long even for ec2 to provision a new instance :) [15:57] hatch, not aware of any -- anything in machine-0.log? [15:57] hatch, (does the machine have an instance id in status?) [15:58] "2": [15:58] instance-id: pending [15:58] series: saucy [15:58] nope [15:58] just checking the log now [15:58] hatch, cheers [15:59] fwereade the messages from around when I provisioned the machine https://gist.github.com/hatched/3ef622f05e1a1b1249c4 [16:00] hatch, and messages with "provisioner" in them? [16:00] hatch, they might only trigger a bit after you added the machine [16:01] hatch, (I only see 5 lines there) [16:02] updated with a bunch of lines which were just dumped out [16:03] it dumped pages and pages of those errors [16:04] updated again [16:04] this is using fresh trunk [16:04] a fresh [16:04] on precise [16:08] sinzui, got ack for 1.17.6 for upload - building now [16:09] jamespage, thank you for warning [16:15] is juju --upload-tools currently working? [16:15] i'm failing to bootstrap under ec2 using --upload-tools [16:16] rogpeppe I just did that using the most recent trunk [16:16] on ec2 [16:16] i'm seeing a "error: flag provided but not defined: --instance-id" [16:16] hatch: and it worked? [16:17] yeah - add-machine didn't [16:17] but normal deploying did [16:22] fwereade, updated https://codereview.appspot.com/79250044/ as suggested [16:23]