[00:30] thumper: ping, i'm free to talk whenever you are [00:31] wallyworld: https://github.com/juju/juju/pull/513 :) [00:31] wallyworld: going to eat dinner, i'll check for a LGTM in a bit and hopefully land it! [01:00] katco: awesome. for the 1.20 update, you will need to ensure you use the 1.20 branches of the relevant sub repos ie you will need to update their imports and commit the changes and then use the relevant rev shas [01:06] o/ wallyworld [01:06] how was nuermberg? [01:07] nuern nurem... equal edit distance [01:18] axw: hey ya. good but busy. we gots a lot to do. got time for a chat? [01:18] wallyworld: sure [01:19] see you in standup meeting [01:43] davecheney: around? [01:48] yeah [01:48] thumper: sup [01:48] chat? [01:49] thumper: sure, lets meet in the 1-1 hangout [01:49] kk [01:51] * thumper is there waiting for davecheney [02:06] thumper: did you want to delay our 1:1 [02:06] wallyworld: yes [02:06] ok, just ping me [02:36] wallyworld: now? [02:36] sure [02:44] lucky(~/src/github.com/juju/juju) % gb ./... │················ [02:45] juju/sockets/sockets.go:6:2: no buildable Go source files in /home/dfc/src/gopkg.in/natefinch/npipe.v2 [02:45] anyone else getting this error ? [02:49] wallyworld: thumper axw https://github.com/juju/juju/pull/514 [02:49] review please [02:49] this is blocking my build [02:50] davecheney: otp, will look soon [02:50] eh, I thought it was updated to add a file for linux [02:50] * axw check [02:50] s [02:51] i ran godeps and it didn't complain [02:51] i belive I have the right rev [02:51] seriously [02:51] davecheney: I think you just need to merge trunk again, I think there's a new rev [02:51] this isn't the right way to fix the problem [02:51] axw: trunk of which repo ? [02:52] npipe [02:52] um [02:52] merge trunk of juju [02:52] there's a new rev of npipe in dependencies.tsv [02:53] gopkg.in/natefinch/npipe.v2 git e562d4ae5c2f838f9e7e406f7d9890d5b02467a9 [02:53] that is what it says [02:53] and that is the rev I have [02:54] I FUCKING HATE GOPKG>IN AND THE INTERACTION WITH GODEPS [02:54] I HATE IT SO MUCH [02:54] I MAKES ME MISERABLE [02:54] * thumper chuckles [02:55] davecheney: what's the problem exactly? I'm not having any problems with godeps [02:55] we have two tools that are trying to do the same thing, pin a dependency of a go package [02:56] i think it is a serious error to combine those two tools [02:56] davecheney: I don't think it is too terrible, just annoying [02:57] one seems to pin (vaguely) the api, the other pins to commits [02:59] davecheney: did you see all the syslog stuff being done? are you happy with it? i thought at one point you may have had an opinion on it? maybe i am misremembering? [03:00] wallyworld: as in moving away from rsyslog and doing file rolling inside the application ? [03:00] davecheney: that and there was talk of using Go's syslog, not sure if that went ahead or not [03:00] don't use that package [03:00] it's fucked [03:00] we can't remove it [03:00] but it's still fucked [03:00] yup [03:01] i also am not in favor of doing file rolling inside the applicatoin [03:01] from a decade of sysadmin experience [03:01] yep [03:01] that is a sure fire way to loose logs [03:01] davecheney: can you replay to the thread on juju-dev, or email nate directly with your concerns? [03:01] reply [03:01] wallyworld: will my comments be given serious consideration [03:02] i've not seen a lot of that in the past [03:02] i know that is a snotty thing to say [03:02] but i'm not going to kick up a stink unless there is actual possiblity of chance [03:02] change [03:02] otherwise i'll save everyone the discomfort of hearing me rant [03:02] they will be taken seriously by me, and whomelese else i talk to i'll tell them :-) [03:02] don't rant as such, just provide facts [03:03] and i'll make sure nate and whoever else takes note [03:03] here are the facts [03:03] 1. the state/api/database/mongo whatever runs on linux [03:03] it will continuye to run on linux for the forseable future even if we have windows workloads [03:03] we should continuye to use rsyslog for log collection [03:03] as it decouples the application from log rolling [03:03] applications write to stdout [03:04] somethign else takes care of where that stdout goes [03:04] if you make the application responsible for rolling it's own logs [03:04] you'll generally find that you end up writing to a log file that has been unlinked [03:04] I did comment on the PR about the approach to rolling being a bit broken, but I don't see any change in nate's new PR [03:05] i agree. can you put that in an email to the list? feel free to mention your previous sysadmin experience to add weight to your arguments [03:05] wallyworld: http://12factor.net/logs [03:05] ^ this is what we shold be aiming for [03:05] wallyworld: i honestly don't think i can be objective at thispoint [03:05] i've given you what I can [03:06] i don't want to take on the heartache of another loosing battle [03:06] hmmm, ok. i would prefer you provided the raw material (based on yuor experience) - we can run with it from there [03:06] ie fight the battle [03:06] but i can do it also [03:07] wallyworld: which thread ? [03:07] * wallyworld looks [03:07] i can't find it [03:07] if it's on juju-team [03:07] i'm not subscribed to that list [03:07] or i should say, my subscription is still pending [03:08] juju-dev - getting rid of all-machines.log [03:08] but we could do a new post referencing that thread [03:09] also: https://github.com/juju/juju/pull/375, superseded by https://github.com/juju/juju/pull/512 [03:23] wallyworld: that big test was just moved. I'd rather not change existing things too much at this point [03:23] ah you noticed :) [03:23] axw: yeah, just made a followup comment :-) [03:25] * axw tests it once more for good luck [03:38] wallyworld: can you please review https://github.com/juju/testing/pull/28 too? [03:38] sure [03:41] wallyworld: btw, to test with mongo 2.6 you can just do "JUJU_MONGOD=path/to/mongod go test ./..." [03:41] will not work in CI, though; that would require additional changes [03:41] axw: np, thanks. there was a test CI job I started a while back [03:42] cool [03:42] but i never finished it [07:39] morning [07:41] morning TheMue [07:42] I thought jam is here today.. [07:52] dimitern: IMHO he is on holiday the whole week [07:52] dimitern: oh, no, just took a look into the calendar [07:54] TheMue, yeah, maybe he'll pop in at some point today [07:56] dimitern: yep [07:57] dimitern: so comming back to my change of SupportsNetwork and its usage [07:57] TheMue, how's that capability branch for the networker going? [07:57] TheMue, :) [07:57] dimitern: bingo [07:57] axw: https://github.com/juju/juju/pull/518 [07:58] dimitern: the capability is currently already implemented, imho only with the wrong naming [07:58] dimitern: but is this enough [07:59] dimitern: and then use it where today we test if the providerType is MAAS? [08:04] waigani_: looking [08:06] waigani_: :x a much simpler solution has just dawned on me [08:07] waigani_: we could have just got the bootstrap instance and used its Addresses method... [08:07] TheMue, there's a new capability needed, let's call it AllowOnlySafeNetworker(machineId string) bool [08:07] * axw continues looking anyway [08:08] TheMue, which will return false for all providers, except local [08:09] TheMue, and for local it will check internally if the machineId == bootstrapMachineId (or however it's called) and return true (i.e. do not allow unsafe networker on the bootstrap node) or false (for all other machines) [08:10] dimitern: ah, already wondered. this description better matches what I’ve seen in the discussion, thanks. [08:11] dimitern: as we call them mostly Supports… (reads better) I would call it SupportsUnsafeNetworker(machineId) [08:12] TheMue, I'm fine with that (in this case the meaning is inverted - true for most, false for local bootstrap node) [08:13] dimitern: I’m also not happy with Networker and SafeNetworker, which implies the first one is „unsafe“ ;) [08:13] waigani_: commented on the PR [08:14] TheMue, well it is kinda unsafe :) [08:14] waigani_, the bootstrap node can have and does have more than one address usually [08:20] axw: you horrible man [08:20] waigani: :( [08:21] waigani: on the plus side, the alternative is really quick to implement :p [08:21] axw: lol, no it's a good solution - I raised an eyebrow when I came across Addresses, but assumed we could not use it for all providers [08:41] axw: will the bootstrap instance have Id() == "0" ? [08:42] waigani: no [08:42] that's machine ID [08:42] instance ID you cannot predict [08:42] waigani: just call AllInstances() and get the one and only element out of it [08:43] axw: okay, so we only bootstrap one instance? When to the HA instances get bootstrapped? [08:43] waigani: bootstrap = start the first instance [08:43] HA comes later, when you do "juju ensure-availability" [08:44] right [08:47] dimitern: TheMue: just noticed your conversation [08:47] dimitern: TheMue: shouldn't "safe networking" be true for manual provider too [08:48] voidspac_, yes [08:48] dimitern: TheMue: and I agree "safe" is not a brilliant name [08:48] but I dislike burning too much energy on name bikeshedding [08:48] pick something and use it [08:48] voidspac_, for all machines with the manual provider, and only for the bootstrap node with local [08:48] yep [08:49] don't forget machines can be manually added to a non-manual provider environment [08:49] in which case provider != "manual" [08:49] axw: oh, ouch [08:49] probably don't want to muck with their networking either [08:49] dimitern: the unit test failures are all uniter tests [08:49] dimitern: are you / is anyone looking at them? [08:50] voidspac_, which failures? [08:51] dimitern: the ones jam emailed us about yesterday [08:51] dimitern: CI failures [08:51] axw, dimitern: so manual || local bootstrap || (non-manual && manually added)? [08:51] dimitern: you said you would look at them :-) [08:51] http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-amd64/1459/console [08:51] axw, good point; TheMue you should check for manually provisioned machines and not allow unsafe networker there as well [08:51] http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-amd64/1458/console [08:51] http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-amd64/1457/console [08:51] http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-amd64/1456/console [08:51] axw, what's the best way to tell if a machine is manually provisioned by its id? [08:52] voidspac_, aw, sorry, I'm struggling with networker refactoring, will have a look before the standup [08:52] dimitern: from what context? you can tell from the state.Machine [08:52] dimitern: I can have a look, although maybe not before standup [08:53] axw, ah, so the machine id is not special [08:53] axw: idea is to have an environ capability able to say if unsafe networkers are supported by passing the machine id [08:53] dimitern: nope. the nonce is special [08:53] axw, we'll need to lookup the machine in state via the api to be able to tell [08:53] dimitern: that would probably be best [08:53] axw, is the nonce constant? [08:54] dimitern: yup [08:54] set by the instance broker [08:54] well, in this case it's set by the manual provisioning code [08:54] axw, alright, thanks [08:54] dimitern: TheMue: state.Machine.IsManual() [08:54] axw: nice [08:55] * TheMue takes some notes [09:14] Hmm, just found „SetHasVote()“. Looks strange, would have named it „GrantVote()“. [09:21] axw: I'm hitting the same problem that I hit when I connected to the api after bootstrap: something is hanging. I've set state-server to true on tests [09:22] axw: here's what I've done: http://pastebin.ubuntu.com/8043738/ [09:22] waigani: *shouldn't* need to do that... there should be no interaction with the state server to obtain the instances or their provider addresses [09:23] axw: I wonder if it is this: c.ConnectionEndpoint(false) [09:23] waigani: I have nfi what ConnectionEndpoint is [09:30] fwereade_, ping? [09:31] fwereade_, ^^ I'm going to rebase the metric branch before landing - I think most of the review was us working around what should be implemented so I'm going to cleanup the history to make it easier to understand [09:33] morning [09:45] fwereade_: ping me when you arearound plz === rogpeppe1 is now known as rogpeppe [09:59] axw: okay I got it working. Do we also only want to grab the first address of the first instance? [10:00] waigani: I think we should just get them all, like the existing api address caching does [10:00] axw: cool, that is how I have it now [10:18] https://bugs.launchpad.net/juju-core/+bug/1356806 [10:18] Bug #1356806: cmd/juju: juju takes 3 seconds to do nothing [10:29] davecheney: it's a pity that parsing rsa keys is so slow. ISTR that it's because it does a "probably prime" check when it parses the key. [10:29] davecheney: although i thought that issue was fixed. hmm. [10:34] rogpeppe: this is on ppc64 [10:34] where there is no asm math library [10:34] juju is just doing something dumb [10:34] it doesn't _always_ need to read the private key [10:34] davecheney: i agree it's dumb, but it should not take 3s to parse an RSA key on any platform [10:34] i don't know how optimised the ppc port is at the moment [10:34] one would suggest, not at all [10:34] davecheney: even if not optimised [10:35] davecheney: parsing an RSA key should not require many math operations [10:35] davecheney: it's essentially just reading a couple of big numbers [10:35] rogpeppe: i think we're in agreement here [10:36] yet here we are, waiting 3 seconds for nothing to happen [10:36] davecheney: i'd like to fix both places [10:36] davecheney: both the Go library (to make parsing RSA keys more efficient) and juju-core (to avoid doing that when it doesn't need to) [10:36] alexisb, ping [10:46] voidspac_, standup? [10:47] dimitern: I'm there... [10:58] dimitern: connection lost :-( [10:58] dimitern: overlapping subnets are not allowed in the new model - we have one network [10:58] voidspac_, yes, but where have you seen the overlapping subnets in the model? [10:59] dimitern: but isn't it possible that subnets on different nics will look like they overlap [10:59] but don't because they're on separate networks [10:59] but we can't represent that in the new model [10:59] cidr is the primary key [10:59] voidspac_, I'm not sure I understand [11:00] or should the different nics be bound to different subnets anyway [11:00] voidspac_, cidr is the pk for a subnet, yes [11:00] voidspac_, nics can be bound to one or more subnets [11:00] dimitern: two different networks can have the same subnets allocated to different machines though, right? [11:00] dimitern: so the same cidr can be duplicated on different networks [11:00] voidspac_, the confusion I think comes from the term "network" [11:01] heh [11:01] voidspac_, in juju, per the new model, all subnets are in the same (virtual/logical?) network [11:02] voidspac_, in provider-space, if networks and subnets are supported, there is a possibility to have 2 networks with overlapping subnets [11:02] dimitern: right [11:02] dimitern: how do we cope with that in the juju model? [11:02] voidspac_, but I don't think this is a valid case [11:02] dimitern: ok [11:03] dimitern: because they won't be routable to each other and so you can't use them for anything useful [11:03] voidspac_, it a weird setup, and we should not allow that, aiui [11:03] voidspac_, yes [11:03] dimitern: but we currently don't "disallow it", we pretend it can't happen [11:03] voidspac_, but good point about bringing this up [11:03] you can still have unroutable subnets, and just whichever one is added first wins [11:04] dimitern: I think in summary, networking is hard [11:04] voidspac_, another possible case is having overlapping vlan subnets with different tags [11:04] right [11:04] :) oh yeah [11:04] voidspac_, you can't do this in maas I think [11:05] dimitern: TheMue: also I spent a while talking to perrito666 yesterday - current restore does not *need* direct mongo access [11:05] voidspac_, so juju shouldn't take it either [11:05] dimitern: TheMue: it just uses it after doing the restore to update some data, and can be fixed to not do that [11:05] dimitern: right [11:05] voidspac_, great! [11:05] dimitern: fair enough [11:05] dimitern: TheMue: so that's what I've been on, I was going to swap and look at the bugs [11:05] dimitern: I have a precise VM, should I try and repro these bugs? [11:07] voidspac_, yes please, I'll do the same; hopefully different setups can lead to reproduction of at least some bugs [11:07] dimitern: cool, coffee first then I'll spin up my VM [11:17] fwereade_: team leads? [12:03] wallyworld, ppc64el unittests are failing in many different ways. [12:03] oh joy [12:03] wallyworld, I am going to increase the number of retries, and also look for cruft left on the machine [12:04] I am not sure juju is really bad on ppc [12:04] sinzui: could it be new tests with ordering issues, or is it existing tests that have started failing? i'll have to look at the logs. currently in standup [12:32] voidspac_, managed to reproduce some of the ci test failures on a precise vm using rev 9b2d2106476bad0ac528256db23bad073257d4bf [12:32] dimitern: cool [12:35] dimitern: any ideas on the cause? [12:35] dimitern: I took a break and am now still updating the vm [12:35] voidspac_, actually I'm having issues with mongo now [12:35] juju won't build for me [12:35] think i need to update go [12:35] voidspac_, I installed mongodb-server from the ppa:juju/experimental [12:35] godeps completed but still build errors [12:35] ah [12:35] ls [12:35] voidspac_, you need to install mongodb-server and golang from there [12:36] dimitern: I can't just build golang from source? [12:37] where Go comes from is certainly not the problem [12:37] voidspac_, you can ofc, but to mimic the test environment as close as possible, I'd use the ppa [12:37] ok [12:37] just added the ppa and doing an update [12:37] go is building anyway [12:44] natefinch: actually I think it was the problem (which Go I was using) [12:44] natefinch: I think we're no longer compatible with the golang in precise [12:44] which I was using before [12:45] but building go from source works fine [12:45] voidspac_: oh, yeah, sorry, I thought you were using something reasonable [12:45] :D [12:45] although I now have golang from the experimental ppa as well [12:45] voidspac_, interestingly, I can run (most of) state/ tests without any mongo issues, but for agent/ or worker/uniter/ tests I get the same failures "cannot set admin password: need to login"} ("cannot set admin password: need to login" [12:45] natefinch: I'd moved it out of the way to try something a while ago and not moved it back [12:46] voidspac_: I think precise has like 1.1 or something, and we need ~1.2.1 [12:46] dimitern: running worker/uniter now [12:46] natefinch: right [12:46] voidspac_: newer versions of Go always work with older code. But older versions of Go don't always work with newer code [12:46] natefinch: the actual build error was something nice and helpful like [12:47] ../../../code.google.com/p/go.crypto/openpgp/packet/encrypted_key.go:8: import /home/michael/canonical/pkg/linux_amd64/code.google.com/p/go.crypto/openpgp/elgamal.a: object is [linux amd64 go1.2.2 X:none] expected [linux amd64 go1.1.2 X:none] [12:47] which looked like a version error to me... [12:47] does anybody have any clue about that error? "cannot set admin password: need to login" [12:47] natefinch: right, in theory anyway [12:47] dimitern: axw has done a lot of work in that area recently [12:47] ahh yeah.... that's actually just old binaries in your pkg directory [12:47] the "need to login" part happens in several unrelated tests, but not all of them fail [12:48] voidspac_, this is actually on trunk HEAD [12:48] voidspac_: they're actually really really really careful to make sure they maintain backwards compatibility. [12:48] voidspac_, I had too many issues with rev 9b2d2106476bad0ac528256db23bad073257d4bf so decided to try trunk first [12:48] dimitern: right [12:52] *sigh* [12:53] adding upstream repo and actually pulling HEAD [12:53] now re-running [12:53] I had some failures, but I wasn't on HEAD [12:53] although I was on master, so failures still od [12:53] *odd [12:53] but different [12:54] filter_test.go:305 c.Fatalf("unexpected config event") [12:55] voidspac_, no failures with worker/uniter on HEAD; now re-running on 9b2d2106476bad0ac528256db23bad073257d4bf, after I wiped out $GOPATH/pkg/, just in case [12:56] dimitern: I get one failure on HEAD [12:56] the same as above [12:56] voidspac_, lots of failures, all of them stemming from "need to login" [12:57] dimitern: so it looks to you like that issue has been fixed [12:57] if it's on 9b2d2106476bad0ac528256db23bad073257d4bf but not on head [12:58] voidspac_, I think something's wrong with my mongo there, I want to fix this first, so I can have a reliable setup [12:58] ok [12:59] I'm trying worker/uniter with 9b2d2106476bad0ac528256db23bad073257d4bf [13:04] dimitern: I get no worker/uniter failures on 9b2d2106476bad0ac528256db23bad073257d4bf [13:04] voidspac_, hmm how about for agent/ ? [13:05] will try [13:11] dimitern: I re-ran the worker / uniter tests and got the failure I got previously [13:11] ran agent/... tests fine [13:11] no failures [13:12] but I didn't have this failure on worker/uniter before [13:12] so looks like it's flaky rather than consistent [13:12] but it's still a different failure to the one on CI [13:12] or the ones you have [13:12] right [13:12] so, fark [13:12] not helpful at all [13:39] hmm.. recent yaml changes seem to cause some compat issues.. $ juju status -> error unmarshalling "/opt/juju/environments/ocean.jenv": YAML error: resolveTable item not yet handled: < (with =DEBUG;unit=DEBUG) [13:58] voidspac_, what mongodb are you using? [13:58] voidspac_, $ `which mongod` --version ? [14:01] voidspac_, sorry, I meant $ which mongod && mongod --version [14:01] natefinch: wwitzel3 standup [14:02] voidspac_, the version from the ppa is 2.2.4, but this is too old for what we're doing around state.Initialize(), more specifically, SetAdminMongoPassword uses admin.UpsertUser(), which is available in 2.4+ [14:02] sinzui, ping [14:02] hi dimitern [14:03] hey, I think I discovered a potential regression re precise + mongodb [14:03] :( [14:03] sinzui, but it's funny how it doesn't show on the CI tests - is the mongod there 2.2.4 from ppa:juju/experimental ? [14:04] sinzui, because I can reproduce this issue at will with both HEAD of trunk and rev 9b2d2106476bad0ac528256db23bad073257d4bf (from http://juju-ci.vapour.ws:8080/job/run-unit-tests-precise-amd64/1456/consoleFull) [14:04] dimitern, CI uses 2.4.6 to match cloud-tools [14:05] sinzui, from where? [14:05] http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/m/mongodb/ [14:05] dimitern, ^ looks like I should switch to 2.4.9 to match [14:06] sinzui, how can I install mongodb-server from there? add-apt-repo ...? [14:07] sinzui, is it know that we no longer support mongodb 2.2.x, even on precise? [14:07] known* [14:07] sinzui, is manual provider in our functional tests? [14:07] i'm encountering quite a few regressions.. just trying to determine sanity [14:08] hazmat, yes, deploys an upgrades to instance in aws and real metal [14:09] ah, found it - ppa:juju/stable [14:10] sudo add-apt-repository cloud-archive:tools [14:10] sinzui, hmm.. ic. okay. i'll keep filing bugs then [14:10] dimitern, https://wiki.ubuntu.com/ServerTeam/CloudToolsArchive [14:10] thanks sinzui [14:10] dimitern, I copy packages to juju stable because many people fail to install that archive when using local precise === bloodearnest_ is now known as bloodearnest [14:12] sinzui, right, I'm trying the same tests now with 2.4.6 from the archive [14:15] sinzui, I can't reproduce any of the failures now :( [14:17] sinzui, looks like manual provider add-machine but oddly not bootstrap is failing if there isn't an ubuntu user in the base image. [14:17] ah [14:17] sinzui, but I have a suggestion how to add debug logging to mgo, so we can see what's going on in more detail [14:18] hazmat, we have a bug for a cases where ubuntu was removed from a server image and lxc fails [14:18] sinzui, also wierdly bootstrap but not add-machine uses ssh-agent === jw4 is now known as jcw4 [14:20] dimitern: /usr/bin/mongod [14:20] db version v2.4.6 [14:20] Thu Aug 14 17:19:47.712 git version: nogitversion [14:20] dimitern: I added the ppa and installed mongodb-server [14:20] voidspac_, right I suspected that much - it'll never work with mongodb-server 2.2.4 from ppa:juju/experimental [14:20] dimitern: but it must be preferring a newer version from elsewhere [14:21] dimitern: but that's not the same error that CI has, right? [14:21] voidspac_, with mongodb-server 2.4.6 from cloud-archive:tools - no failures [14:21] they just have uniter test failures [14:21] jcastro: cross-team call today? [14:21] ah [14:21] dimitern: what about CI with HEAD? [14:21] I guess I can check... [14:21] voidspac_, I'll try HEAD next, just running uniter tests once more to be sure [14:22] hazmat, I re added ubuntu user to the kvm/maas machine I am provisioning. And I provisioned it with manual provider [14:22]