[04:08] axw: ping [04:08] davechen1y: pong [04:08] axw: i'll find the bug [04:08] but we have a prolem when we use deploy --to 0 [04:09] which causes debug-log to go crazy [04:09] okey dokey [04:09] i'd really like to be able to fix it [04:10] because we're leaning heavily on the gui deployed to machine 0 [04:10] the workaround is don't use --to [04:10] axw: sorry, i don't know why I'm making this your problem [04:10] you didn't write --to [04:10] heh that's okay, I can look into it [04:11] there's no consolidated log on local, so may take me a bit to get set up [04:11] yeah [04:11] that is why we didn't hit it til now [04:11] bug 1211147 [04:12] <_mup_> Bug #1211147: Deploying service to bootstrap node causes debug-log to spew messages [04:13] if it's too hard to fix easily I can stop useing --to [04:13] but we're struggling to keep the number of machines deployed down to a reasonable level [04:37] hello [04:37] question from the field [04:37] does juju upgrade-charm --switch work [04:37] to switch from cs: to local: ? [04:38] hazmat: ping [04:40] davecheney: don't know that, but to fix your logging problem temporarily... [04:40] delete /etc/rsyslog.d/26-juju*, restart rsyslogd [04:40] (I think) [04:41] axw: thanks, i'll work up a one line juju ssh fix [04:41] on the state server that is [04:41] davecheney: I haven't tested it. [04:41] I don't have canonistack set up atm [04:44] axw: testing now [04:45] hmm, close, but there is still something else going on [04:47] axw: on positive news [04:47] debug hooks is working like a chapm [04:47] champ [04:53] davecheney: sorry went to get a sandwich [04:53] glad to know debug hooks is working. sorry about the logging... I'll keep looking [05:05] axw: it should work, maybe rsyslog is holding the config somewhere else ... [05:06] axw: ahh, it could be that debug-log is brokwn [05:06] totally [05:07] davecheney: how's that? [05:07] made a new environment without using --to [05:07] debug log still spews output [05:07] hmmkay [05:07] which is a bit of letdown [05:10] oh [05:10] i think the problem is all the \n's have been stripped from the debug log file [05:10] sorry [05:10] all-machines.log [05:10] so tail looks insane [05:12] morning axw and davecheney, how are things in AU? [05:12] jam: morning! not too shabby thank you [05:12] davecheney: is your debug-log problem bug #1211147? [05:12] <_mup_> Bug #1211147: Deploying service to bootstrap node causes debug-log to spew messages [05:13] jam: yes, and it is actually our log format [05:13] struggling to get back onto canonistack tho.. [05:13] axw: canonistack is fail [05:13] just get ec2 or hp cloud credentials [05:13] jam: the problem with debug-log is the log format [05:13] it's stripping the \n [05:13] so wc -l on all-machines says there are 0 lines [05:13] and tail prints everything fromthe beginnning of time [05:14] davecheney: so "juju debug-log" is just doing "juju ssh 0 tail -nX /var/log/juju/all-machines.log" but you're saying that something in how we are doing rsyslog is causing us to strip all \n chars? [05:14] jam: that is my diagnosise so far [05:14] davecheney: that certainly sounds like a new problem [05:15] jam: try it [05:15] davecheney: I guess there are company accounts for EC2/HPCloud? who do I need to ask? [05:15] axw: no [05:15] you shold obtain your own credentials and expense them [05:15] ah ok [05:15] i'm sorry nobody explained this too you [05:15] this really should be day one induction stuff [05:15] the company has resources, canonistack [05:15] actually I think rog mentioned we could expense EC2.. [05:15] but it's frequently not up to the task [05:16] davecheney: actually, we do have a company account with arosales for hp [05:16] that's a bit sad :( [05:16] axw: yeah, there is even a line item on the expense system for those type of expenses [05:16] ok [05:16] thanks [05:18] jam: I'm not sure it's a company account, more like an account that we have for certain testing [05:18] and now the occasional emergency charm school [05:19] marcoceppi: right. For "testing" which is what axw is doing. vs hosting your own project. [05:19] * marcoceppi doesn't know what anyone does. Not even meself [05:19] axw: which Canonical will pay for you to host your own project (equivalent of up to 3 m1.tiny (?)) as long as you manage it with Juju to give us feedback on the tool. [05:20] cool. [05:20] I better think up some interesting projects then [05:20] jam: it is quite strange, the output format of the rsyslog config hasn't changed since wallyworld_ landed the fix [05:20] sorry the original support [05:20] i wonder if it is a hp cloud thing [05:20] * davecheney tries ec2 === tasdomas_afk is now known as tasdomas [05:22] indeed it is [05:22] it works perfectly in ec2 ... [05:22] i wonder if it is the hostname [05:26] jam: i think there are two problems [05:26] hulk smash causes rsyslog to feed back to itself [05:26] but also the output format + hp cloud does something strange [05:27] davecheney: right, hulk-smash is something I feel we have an understanding of the bug, we just need to fix it. hp cloud stripping '\n' is something new we haven't seen before. [05:27] jam: understood [05:27] will treat as a second issue [05:28] just trying axw's fix for hulk smash [05:29] jam: axw confirm that reminging /etc/rsyslog/26-juju* and restarting rsyslog works [05:29] goodo [05:29] thanks [05:30] davecheney: so that fixes the \n thing as well? or just the syslog feeding into itself? [05:32] jam: only 1211147 [05:32] i'll dig into the log format thing on hp cloud [05:32] and open a new issue [05:45] axw: is it possible to do [05:45] juju status 1 [05:45] and show only machine 1 [05:45] davecheney: nope [05:45] only unit or service [05:45] axw: roger [05:46] question, what is the presence ping delay ? [05:46] 60 seconds ? [05:48] there's a var in presence/presence.go that says 30s, tho I don't claim to understand the workings of that code [05:48] ^ davecheney [06:05] So, juju deploy mysql mysql-1 (using mysql-1 as an alias) throws an error, is this expected [06:22] axw: thanks [06:22] i was able to demo that [06:22] does anyone know whre the missing and unknown states show up [06:22] is that only for machine records ? [06:33] erk [07:06] jam: I reproduced that line-ending bug on lcy02 [07:06] I'm happy it was reproducible, less so that it exists :) [07:07] I'll look into it after fixing the original problem [07:10] Is StoreSuite known to fail sometimes? http://paste.ubuntu.com/5984006/ [07:14] noodles775: yeah I think I've seen TestBlitzKey fail before, randomly [07:15] noodles775: we are supposed to be rigorous about not having casually failing tests in the test suite, though it seems being away for 3 weeks caused this rigor to diminish a bit. [07:15] (to land code, all tests must pass on the bot, for example) [07:19] axw, fwereade_, jam, davechen1y, mgz, wallyworld_: morning! [07:20] jam: Yep, although it can be v. hard to fix them when they're hard to reproduce. [07:20] axw: ack, txs. [07:20] rogpeppe: good morning [07:21] axw: fancy coming to the standup today? [07:21] axw: i don't know what time it is for you then, mind [07:21] rogpeppe: about 7:30 (dinner time ;)) - but I'll come today [07:22] axw: don't feel obliged - i just thought it might be nice if you could make it [07:23] rogpeppe: no I would like to come, and I haven't been to one yet [07:23] axw: cool [07:23] axw: how's it going anyway? [07:24] rogpeppe: not too shabby. working on manual deployment, and fixing a debug-log bug atm [07:24] axw: cool [07:26] axw: some time i hope we can have a way of changing the debug level in an environment dynamically [07:26] rogpeppe: I believe thumper has a branch in the works [07:26] it would be nice [07:26] axw: ah, that's good to know [08:27] jam1: if it interests you: https://bugs.launchpad.net/juju-core/+bug/1212148 [08:27] <_mup_> Bug #1212148: rsyslog accumulator strips newlines on openstack [09:10] rogpeppe: hey [09:10] dimitern: yo! [09:10] rogpeppe: a relation is not an entity, right? we cannot reuse the LifeGetter for it [09:11] dimitern: hmm, yeah [09:11] rogpeppe: ok [09:14] dimitern: we could make it an entity, i guess [09:17] rogpeppe: I don't think this is a good idea [09:17] rogpeppe: it's not used as other entities are [09:17] dimitern: the AllWatcher reports on changes to it, not too differently from other entities [09:18] dimitern: entities are not all used the same [09:18] rogpeppe: yeah, but there's no agent or client that uses it as its main entity [09:18] dimitern: that's not true of Service either [09:18] dimitern: or Environment [09:18] I'm trying to reproduce bug 1208504, but get "Cannot find requested image 81078" when trying to bootstrap on HP Cloud. If anyone has any experience with HP Clound and knows what I'm doing wrong, please let me know :) http://paste.ubuntu.com/5984281/ [09:18] <_mup_> Bug #1208504: "error: no instances found" Post bootstrap on HP Cloud

[09:19] rogpeppe: hmm [09:20] rogpeppe: we can do that later, if we decide to do it anyway [09:22] rogpeppe: and what will be its key? there are 2 keys that are used - id int and a key string (created from the endpoints) [09:22] rogpeppe: the string key is _id in the doc, but is some places the id is also used [09:22] dimitern: i'd presume the latter, but there might well be reasons why that's a bad idea [09:22] fwereade_: you around? [09:29] and it's even less clear about endpoints [09:33] relationUnit seems it can be implemented at client-side with only a few methods on the server-side [09:36] fwereade_: when you have some time, we need to have a chat about how best to implement certain uniter objects in the api [09:47] does anyone else see this test failure on trunk? http://paste.ubuntu.com/5984415/ [09:48] rogpeppe: you need to update goamz [09:48] dimitern: ah [09:49] dimitern: wow, that's an unexpected way for it to fail (updating goamz did fix it) [09:49] dimitern: do you know what the issue was? [09:50] rogpeppe: yes, it stems from the changes to support instance-state in juju status [09:50] rogpeppe: sorry, I should have sent an email [09:50] axw: did you update the dependencies file? [09:50] rogpeppe: uhmm. I forget. I think so... [09:50] axw: cool [09:51] rogpeppe: the issue fixed in goamz was in the testing server, to pass through the instance state [09:52] previously it was always blank, hence the test failure [09:56] back later for the standup [10:13] * rogpeppe has just discovered how to speed up ec2 local tests from 3 minutes to 6 seconds [10:13] rogpeppe: how? [10:14] dimitern: change s3.attempts [10:14] rogpeppe: and still all pass? [10:14] dimitern: yeah [10:17] rogpeppe: cool! [10:17] rogpeppe: is that in juju or goamz? [10:17] dimitern: goamz [10:17] rogpeppe: ah [10:23] yeah, the ec2test local bits are... fun in many ways [10:23] mgz: apologies for that :-) [10:24] :) [10:24] mgz: i did the bare minimum required to get something working. [10:24] mgz: what issues are you having, BTW? [10:25] general lack of flexibility mostly [10:26] mgz: right; it wasn't designed to be flexible [10:26] tests in juju-core rely on hardcoded values that are in goamz, and aren't otherwise exposed or chnageable from inside a test, for instance [10:27] mgz: the instance address == instance id thing? [10:28] like that, yes [10:28] mgz: i don't think it should be too hard to interject flexibility in some places [10:36] dimitern, mgz: https://codereview.appspot.com/12918044 [10:36] looking [10:36] me too [10:37] i'd like to have a separate project containing AttemptStrategy, so that both juju-core and goamz can use it, but i'm not sure how gustavo would feel about that [10:38] rogpeppe: lgtm [10:38] dimitern: thanks [10:39] also. I still wonder if this is really the right way of dealing with eventual consistency things. we poll-wait in a bunch of cases where actually it doesn't matter and operations could just continue [10:39] I guess it's simpler like this at least [10:49] mgz: eventual consistency is very hard to deal with [10:49] mgz: can you think of a better plan for dealing with it? [10:50] rogpeppe: I don't know if you remember me mentioning the s3 thing about 2 months ago, and getting pushback from exposing it as a knob from goamz. I think there was some buy-in that we could have a flag for "disable retries entirely" [10:50] jam1: ah, ok [10:50] jam1: i wondered about that === jam1 is now known as jam [10:50] well, there are a bunch of places where pushing the handling up a few layers would mean we could make better choices [10:50] jam1: environs/ec2 is ridiculously slow to test [10:50] rogpeppe: yep [10:51] but again, I think it's pretty set that it needs to be handled inside goamz, which limits things [10:51] It triggers because we check to see if you have any local tools/images/etc that we should be using, and the s3 eventual-consistency code kicks in saying "I don't have anything in your private bucket yet, but I might in 3 seconds" [10:51] mgz: what places are you thinking of that we are poll-waiting when we don't need to? [10:52] jam: yeah. that sounds reasonable - that may well happen for real when we're uploading tools [10:55] rogpeppe: so it is fine that it triggers in real life, adding 3-5s on every bootstrap isn't a big deal. In the case of the test suite, it is detrimental. [10:55] jam: absolutely [10:55] goamz's double *isn't* eventually consistent. :) [10:55] jam: indeed [10:56] I was also quite surprised that we were spending 3+min in time.Sleep() [10:56] jam: almost exactly three minutes in the case of environs/ec2 [11:19] Could someone please set the following MP to Approved so it lands? https://code.launchpad.net/~michael.nelson/juju-core/1183159-ssh-options/+merge/179422 [11:20] noodles775: done [11:20] ta [11:22] jam, mgz, dimitern: alternative formulation that might be more niemeyer-acceptable [11:22] https://codereview.appspot.com/12740044 [11:24] rogpeppe: reviewed [11:25] dimitern: the other one is just as unclean and should have the same warning really [11:25] dimitern: well, it's a bit less clean, i suppose as it's less direct [11:26] dimitern: (the new CL that is) [11:26] rogpeppe: what about putting the Retry boolean into the s3 constructor? [11:26] AutoRetry perhaps? [11:26] jam: that's ok if the tests have access to where the S3 instance is being constructed [11:27] rogpeppe: well, at least it would be exposed from codebase A into codebase B, and codebase B can expose it up the stack as necesasry. [11:27] jam: there are many possibilities here [11:27] I'm a *big* fan of having as little mutable global state as possible [11:28] dimitern: IIRC the specific objective Gustavo had for the attempts logic was that "real" code shouldn't ever have to tweak this setting because we should make sure it Just Works. [11:28] I'm not sure that I agree 100%, because site-specific tweaking is always a possilibity (what if your ping time means it takes longer to see a consistent state?) [11:28] For our particular use case, the boolean fits better what we actually need [11:29] and is nice an minimalist [11:29] well, we can have "just works" and better tests in the same time [11:29] jam: the sense of the boolean should be the other way if it's exposed in S3 [11:30] jam: although [11:30] jam: i hope nobody is initialising the S3 struct themselves [11:31] fwereade_, rogpeppe, dimitern, axw, mgz, wallyworld_, natefinch: standup time: [11:31] https://plus.google.com/hangouts/_/f497381ca4d154890227b3b35a85a985b894b471 [11:41] jamespage, re mulitple networks and juju (bug 1188126) , is there any order to how quantum attaches the nics, or is quantum api usage the only way to resolve the ambiguity? [11:41] <_mup_> Bug #1188126: Juju unable to interact consistently with an openstack deployment where tenant has multiple networks configured

[11:42] hazmat, the order quantum attaches nics is unhelpful [11:42] hazmat, its attaches then in alpha numeric order for the uuid of the networks [11:43] hazmat, i'd prefer to see some sort of option to specific which network to use in juju - then the instances can be connected to the quantum networks in a deterministic order - i.e. a preferred admin network type arrangement [11:45] hazmat, that might also fit well with using LXC containers when adding extra network interfaces for each container [12:02] axw: I did say you could get back to work, not go back to your personal life :) [12:02] have a great evening [12:02] natefinch: I'll catch up with you after lunch here to do some review/code things [12:03] mgz: cool. probably going to have to be in a few hours, due to my daughter's doctor's appointment. Not ideal for your schedule, I know. I'll ping you when I'm back. [12:04] sure! [12:05] jamespage, ala vpc_id/quantum_net_id for an environment? [12:06] jamespage is this primarily a hosts issue when building a cloud, ie the baremetal gets networks for each of the guest tenants its hosting? [12:07] hazmat, no - I hit this purely with openvswitch overlay networking [12:07] where I can have any number of L2 networks attached to each instance [12:09] jamespage, does any of that mapping of nic to net id get exposed in instance metadata, or is quantum the only way to resolve? [12:10] jamespage, you'd want to spec a net id for private net usage via an env setting? or have juju resolve the entire net topology via quantum [12:11] hazmat, not sure about metadata (it might - I'll check) [12:12] its all exposed in quantum of course [12:12] hazmat, specing a net-id for the private network as an env setting works for me [12:17] jam: hah, thanks :) [12:44] can i get a pair of eyes on https://codereview.appspot.com/12859043/ ? [13:09] noodles775: did you get the tests to run to completion? [13:12] time for lunch [13:14] sidnei: which tests? [13:15] noodles775: all of them *wink* [13:15] sidnei: sorry - you've lost me. Are you talking about a juju-core test run for which branch? [13:16] noodles775: for any branch, iirc you were having trouble getting a clean test run of the whole suite, and i told you i was running only tests for specific packages. [13:17] sidnei: yes, complete test run works fine. I don't remember the context last week where I was having issues. Perhaps before I had the correct mongedb version? [13:17] noodles775: could be. and you're in raring not saucy right? [13:18] sidnei: saucy locally, but all my dev is on a precise canonistack instance. [13:18] noodles775: ok, so you run the tests on precise. [13:19] yep. [13:20] * sidnei tries on a precise lxc [14:05] rogpeppe, jam, mgz: https://codereview.appspot.com/12906044 - endpoints refactoring as we discussed [14:37] rogpeppe, jam: thanks! [14:41] rogpeppe, jam, mgz: next one https://codereview.appspot.com/12748044 - relations as entities [14:44] rogpeppe: I might have missed something important there [14:44] dimitern: looking [14:56] rogpeppe: if you get a chance, https://codereview.appspot.com/12859043/ [14:56] * sidnei lunches [15:07] anyone ever see this on juju bootstrap ... 'juju open.go:89 state: connection failed, will retry: dial tcp 127.0.0.1:37017: connection refused' [15:08] full verbose details @ http://pastebin.ubuntu.com/5985297/ [15:09] cruejones: why sudo ? is that the local provider? [15:09] yup [15:10] cruejones: do you have mongo running on 127.0.0.1:37017 ? [15:11] mgz: ping [15:12] dimitern: does not look like anything is running at that port [15:12] cruejones: does it run on something like 10.0.3.x ? [15:13] cruejones: that's the lxc container address where it should run [15:14] rogpeppe: thanks for the review - updated with your suggestions [15:14] dimitern: nope [15:15] cruejones: are you using the tarball version of mongo, as described in the README? [15:16] cruejones: ah, sorry you probably followed the procedure there and did make install-dependencies ? [15:16] dimitern: hey [15:16] dimitern: ha, no [15:16] mgz: https://codereview.appspot.com/12748044/ please? [15:17] cruejones: did you get juju-core from source? [15:17] dimitern: just updated to latest juju-core, setup my environment.yaml and ran 'sudo juju bootstrap' === tasdomas is now known as tasdomas_afk [15:17] dimitern: okay [15:17] cruejones: if you can run the tests successfully, that's the first step: "go build ./... && go test ./..." in juju-core/ [15:18] dimitern: ok, having to go there is not so user friendly though [15:19] mramm: ^ [15:19] cruejones: well, I haven't seen this error, sorry, but then again I haven't used the local provider yet [15:20] cruejones: most likely it's an issue with the correct version of mongo and/or lxc prerequisities - that's why we've updated the README to streamline the process [15:21] cruejones: we are doing two things to make this better 1) adding error messages when the packages aren't installed correctly [15:21] and 2) adding a juju-local-provider package that installs both juju and the local provider dependencies [15:22] dimitern: sorry, don't get me wrong I really appreciate the help - trying to point out where things could and do go wrong with first impressions [15:22] or I am just clueless and/or bad luck ... probably a both ;) [15:23] cruejones: you're right, it's not user friendly, but we're trying to make it better and appreciate sharing the problems you encounter [15:23] cruejones: definitely, we have to fix this so the process feels seamless and we very much value anybody pointing out where it is not [15:24] cruejones: can you try getting juju-core from source and retry the same, possibly following the steps in the README to see if it helps? 1.12 was some time ago and some changes were already made for the better [15:26] dimitern: will do after some mid-morning meetings, let you know how it goes .. thx [15:36] fwereade_: ping [15:36] henninge: he's out today [15:36] Ah, that's why. ;-) [15:36] dimitern: Will he be back tomorrow? [15:37] henninge: I think so [15:37] cheers [15:44] mgz: does it look ok? [15:47] sidnei: you've got a review [15:48] cool, not as bad as i expected ;) [15:48] rogpeppe: do the changes to instanceData doc need some sort of schema migration between versions? [15:49] dimitern: fine I think, though I'm not sure I get all the implications [15:49] sidnei: i don't *think* so [15:49] mgz: since nothing is using relation as entities yet, there are no implications I can see [15:50] fair enough [15:50] mgz: the only issue might be that now you can attach annotations to relations as well [15:50] rogpeppe, mgz: which might be a good thing [15:51] but there's no api for that anyway [15:51] rogpeppe: on a related topic, i think this is wrong: http://paste.ubuntu.com/5985486/ [15:51] rogpeppe: i think the append should be inside the if s != ""? [15:52] sidnei: i don't *think* so [15:52] sidnei: i think you're allowed to say "mem= disk=10M" [15:53] sidnei: to say "no constraint on memory, but i want to specify that" [15:53] sidnei: i think that's true for any constraint attribute [15:55] rogpeppe: but that's on HardwareCharacteristics.String() [15:55] dimitern: i don't think you can now attach annotations to relations [15:55] rogpeppe: so it ends up as 'mem= ' in juju status [15:55] sidnei: ha, yes [15:55] rogpeppe: while looking at the code it seemed so, but I might be wrong [15:55] sidnei: only if you've got zero memory of course [15:56] dimitern: i think you can only set annotations on entities that embed annotator [15:56] rogpeppe: right, which is never the case. the one i hit was one case with OsDisk being nil, but it should never be nil anyway. [15:56] sidnei: i think the hardward characteristics stuff was probably just copied naively from constraints [15:57] it definitely looks so [15:57] sidnei: they're so similar (and they share some identical code) that i think they could both sit usefully in the same package [15:57] rogpeppe: ah, right! [15:58] sidnei: i don't think the h/w characteristics stuff should use uintStr at all, which is just there for that constraint logic [15:59] if hc.Mem != nil {strs = append(strs, fmt.Sprintf("mem=%M", hc.Mem))} [15:59] rogpeppe: %M? is there an extra char missing there? [16:00] sidnei: ha ha, %d i meant [16:00] k, will make that change. [16:00] %dM [16:00] in fact [16:04] rogpeppe: in the comment 'Just "return" should be do ok here.' that's meant for the 'return nil' line? [16:05] sidnei: it's meant for the if and the return [16:06] mgz: back finally [16:06] natefinch: ace [16:06] rogpeppe: oh, i guess i could just 'return err' without the if you mean? since err will be nil anyway? [16:07] ah, not even that since it assigns to err [16:07] got it now [16:07] sidnei: yeah, and since you've already declared the err return variable, you can use a bare return [16:10] rogpeppe: what's our stance on bare returns? In general I try to avoid them, since they sometimes can return things you didn't mean to return [16:11] natefinch: i think in very short functions where it's easy to see everything at a glance, they're ok [16:11] natefinch: that said, i try to avoid them on most occasions too [16:12] natefinch: in this case, the function's only 3 lines long - it seems reasonable there [16:14] rogpeppe: yeah, for short functions it doesn't matter much... I just usually get to the return and think "really? I'm trying to save 3 (or whatever) characters?" so in practice I never actually use bare returns even though I'm not fundamentally against them [16:15] natefinch: yeah, i see that pov [16:16] rogpeppe: I sorta wish I didn't even have the option, then I wouldn't have to argue with myself about it ;) [16:16] natefinch: yeah. adg shares your view there, i think. [16:21] allenap: around? [16:24] or jtv [16:24] sidnei: Hi there. [16:25] allenap: i assume you are somewhat knowledgeable about azure things. im looking at this table http://msdn.microsoft.com/en-us/library/windowsazure/dn197896.aspx and wondering which of the two columns for 'os disk-space' i should use to fill in some values [16:27] allenap: or maybe that already comes from roleSize? [16:27] oh, it seems to be defined there already, just need to pick one of the two [16:29]