[00:07] cherylj: looks like CI is still running 1.2 [00:07] at least on the 1.25 branch [00:08] does sinzui 's fix need to be backported to 1.25 ? [00:08] >was it just that the go compiler sucks compared to c++? [00:08] ^ i literally cannot even [00:09] wallyworld: axw: i think this is ready to go http://reviews.vapour.ws/r/4573/ could someone plz stamp it? :D [00:09] davecheney: What is happening here http://reports.vapour.ws/releases/3933/job/run-unit-tests-race/attempt/1387 [00:10] ta [00:10] sinzui: are all the juju tests now run on go 1.6? [00:10] for 2.0? [00:10] the race detector does not support ppc64 [00:10] hang on [00:10] is this ppc64 [00:10] wallyworld: no windows and centos are tested on both 1.2 and 1,6 because the 1.6 do not pass [00:10] no, this is amazon [00:11] davecheney: That is exactly my wtf moment. a trusty containter that thinks it is ppc [00:11] sinzui: that's the unit tests right? [00:11] wallyworld: yes [00:11] ok, so the race deector in the trusty golang1.6 deb is busted [00:11] :9 [00:11] :( [00:11] paging mwhudson [00:12] sinzui: i thought horatio fixed the windows tests on 1.6, i'll need to check [00:12] davecheney: ah yes [00:12] wallyworld: no his last fix was for go 1.2 regressions [00:12] wallyworld all packages have been built with go 1.6 for several weeks [00:12] davecheney: try https://launchpad.net/~mwhudson/+archive/ubuntu/trusty-race-detector/+packages ? [00:13] sinzui: the issue right now is there's some upstream azure sdk changes we need but those require go 1.6 to build, so we need 1.6 everywhere first [00:13] so those tests need fixing [00:13] wallyworld: IU ask every day in meeting for them to be fixed [00:14] yeah, everyone is busy :-( [00:15] davecheney: I am inclined to make the race tests non-voting until we sort out what happened [00:17] wallyworld: in the case of windows, I see bugs we already see, but more often, and some that seem to have been revealed by recent fixes http://reports.vapour.ws/releases/3933/job/run-unit-tests-win2012-amd64-go1_6/attempt/51 [00:19] sinzui: oh the fix for beta6 for ODS - the pinger shutdown fix - may have introduced a new window failure [00:19] wallyworld: :( [00:19] well just just a guess [00:19] wallyworld: I am on unknowns tomorrow. I will get every bug files for the remaining Go 1.6 tests [00:20] it seems go 1.6 on windows slowness may be exaserbating some of these failures [00:20] wallyworld: and as for slowness. I think we need to extend timeouts for centos and windows [00:20] if that helps short term, that would be nice so we can get the azure fixes in [00:23] wallyworld: I will also get that sorted out tomorrow [00:23] ok, ty [00:36] davecheney: I think we need to force the race tests to run with go1.2 or quickly solve the golang 1.6 ld issue [00:37] mwhudson: what say you to that ? [00:37] afaik the race detector in go 1.6 works fine [00:37] but something is wrong with how it's pacakged in trusty [00:37] ah [00:37] * redir is eod later #juju-dev [00:37] sinzui, davecheney: there is a ppa with the support files needed [00:37] https://launchpad.net/~mwhudson/+archive/ubuntu/trusty-race-detector/+packages [00:37] sinzui: is using that ppa an option ? [00:38] sinzui, davecheney: i should get this into trusty too [00:38] sinzui, davecheney: alternatively, can you run the race tests on xenial? [00:38] davecheney: yes, yes I think it is [00:38] cool [00:39] mwhudson: I just switched the tests to xenial. got the same error though [00:39] sinzui: oh that's more surprising [00:39] sinzui: do you have apt/apt-get installing recommends by default? [00:41] mwhudson: no, but we support installing a different set of packages instead of using the makefile (tht is how we test s390x and arm64) [00:41] shouldn't matter [00:41] sinzui: can you link me to the log of it failing on xenial? [00:42] mwhudson: It is happening right now at http://juju-ci.vapour.ws:8080/job/run-unit-tests-race/1388/console. You need to log in as a dveloper to see it [00:44] can someone send me the developer credentials [00:44] my browser has forgotten them [00:47] sinzui: how does the golang-1.6 package get installed there? [00:47] all i see is golang-1.6 is already the newest version (1.6.1-0ubuntu1). [00:49] sinzui: anyway, somehow the golang-1.6-race-detector-runtime package needs to get installed [00:49] sinzui: usually that gets installed when golang-1.6 does, because golang-1.6 Recommends: it [00:49] er golang-1.6-go Recommends it, to be more precise [00:50] sinzui: but if golang-1.6-go is somehow installed when recommends are disabled, it won't, so that's my guess as to what's happening here [00:53] mwhudson: understood. The makefile forces no-recommends. the makefile got an update today to ensure golang 1.6 or higher is preffered. I will reconfigure the job to install the required packages. [00:57] sinzui: ah makes sense [00:59] anastasiamac_: LGTM [01:00] axw: \o/ [01:01] thans not fair. the rules to install extra packages conflict with the rules run with --race [01:05] oh, all is good. thank you mwhudson and davecheney . I can see golang-1.6-race-detector-runtime is installed and the tests are running with --race on xenial [01:06] sinzui: \o/ [01:20] woot [02:41] menn0, axw, wallyworld: anyone care for a quick review? It's fairly trivial, but for a critical bug: http://reviews.vapour.ws/r/4735/ [02:42] sure [02:43] I'd gotten eric to review, but he had some questions, so I wanted another pair of eyes [02:50] natefinch: if we have test coverage for proper use of ssl via other means, then we probably can remove that special case in the test [02:50] wallyworld: well, my static analysis tests are stuck in committee... so right now, we don't. [02:51] natefinch: ok, so no harm leaving it there for now. i think the pr is lgtm [02:51] wallyworld: ok... i think that's basically where eric landed too, just wanted a second opinion. Thanks. [02:51] np [03:03] wallyworld: do you know if the apt mirrors thing is *only* applicable to MAAS? [03:04] natefinch: the general principal is applicable anywhere, but as axw mentioned to me, maas allows apt mirroe to be configured for it so juju could make special use of that to set the juju apt mirror setting off of what maas uses [03:05] instead of making the user to specify manuall in 2 places [03:05] natefinch wallyworld: I'm not sure if it does now ... it has configuration for proxies for sure [03:06] wallyworld: does juju's apt mirror get propagated to containers? [03:06] natefinch: no, that's the whole point of this work :-) [03:06] it is supposed to [03:06] so we need to figure out why not and fix [03:07] that's my understanding anyway [03:07] wallyworld: ok... just making sure I understand the problem correctly. Sounds like it's two things: 1.) juju apt mirrors don't get propagated to containers. 2.) (bonus points) juju should pick up maas's apt mirror and pass it into containers juju creates on maas nodes [03:07] yes, assuming maas has such a setting [03:07] yup [03:07] given all this, we need to audit what's there in juju already [03:08] so we understand the current actual behaviour [03:08] and figure out the gap between that and the to be [03:08] and document the to be in that spec [03:10] wallyworld: I meant to ask, when you've got a spare few minutes, can you please have a look over my openstack PR? [03:11] sure [03:12] axw: looks like you already have a +1? [03:13] wallyworld: yeah, but babbageclunk is not a graduated reviewer AFAIK [03:13] ah, i didn't check who did it [03:13] just saw the tick [03:24] Bug #1334481 changed: juju should not record ssh certificates of ephemeral hosts

[03:27] where did all the help go? [03:27] do we not have help on the providers in the CLI anymore? [03:27] axw: lgtm with a drive by request [03:28] natefinch: i think the help topics are under construction, not sure [03:28] wallyworld: ta [03:28] wallyworld: ok, that gives me hope. Right now, juju bootstrap on a new cloud basically just gives me the middle finger with no way to figure out what I'm supposed to do. [03:28] what do you mean? [03:29] $ juju bootstrap google google [03:29] ERROR detecting credentials for "google" cloud provider: gce credentials not found [03:30] and as far as I can tell, none of our help actually tells you how to enter those credentials [03:30] juju help add-credentials [03:30] $ juju help add-credentials [03:30] ERROR unknown command or topic for add-credentials [03:31] sorry, no s [03:31] juju help add-credential [03:31] oh man... bad one to not have a plural alias [03:31] well, it is only a single credential [03:32] it is very often used in the plural, though [03:32] natefinch: juju help bootstrap tells you what you need anyway [03:32] but spot the sypo [03:32] typo [03:33] yeah, IWas gonna say, it even says to see "add-credentials" [03:33] so i'm not sure how the current help is deficient [03:33] bootstrap failed, juju help bootstrap has the answer [03:33] albeit with a typo [03:34] juju help bootstrap is a wall of text. it woudl be nice if the error about "credentials not found" also said "see juju help add-credential" [03:35] hopefully that stuff is being addressed [03:35] I hope so :) [03:35] we do still need juju help google/gce [03:35] i'm not 100% sure of the status of all that [03:36] understood [03:57] gah [03:58] spend 15 minutes figuring out gce's credentials only to hit another bootstrap error: [03:58] cannot start bootstrap instance: sending new instance request: sending new instance request: googleapi: Error 400: Invalid value for field 'resource.networkInterfaces[0].network': 'global/networks/default'. The referenced network resource cannot be found., invalid [04:00] davecheney: small utils/ssh change to allow StrictHostKeyChecking to be turned on: http://reviews.vapour.ws/r/4738/ [04:01] axw: or you? ^^ [04:02] menn0: looking [04:02] cheers [04:02] whelp, can't bootstrap gce... guess it's time to file a bug [04:06] natefinch: :/ how are we not finding that in CI ... [04:07] pretty sure I bootstrapped gce last week, must be something new or maybe region or account specific [04:07] axw: there was some networking stuff landed today... maybe it hasn't made it through CI yet [04:07] ah [04:07] axw: or at least, I heard them talking about networking stuff... I don't know for sure if anything landed [04:09] hmm.. the commit history on master does not look like anything interesting landed lately [04:10] yeah, weird, even if I do it from master weeks ago, still fails same way [04:10] I must be special [04:11] or I have to configure my GCE account in some way that I haven't [04:15] axw: thanks for the review. I think strict should be the default too but I didn't want to break existing consumers of utils/ssh [04:15] axw: FWIW it will soon default to on for juju ssh/scp" [04:15] it's working on my machine [04:16] pulling the host keys from the API server etc [04:16] but needs tidying up and better test coverage [04:16] natefinch: did it fail for you immediately? it seems to be working for me, but hasn't finished yet [04:16] menn0: fair enough [04:16] menn0: cool :) [04:17] axw: failed in 10 seconds [04:17] natefinch: I guess it's something about your account/project [04:17] axw: yeah, I'll try creating a new project [04:18] yes, my gce is bootstrapping fine too from master tip.. [04:19] this was a fairly old project... I'll see if a new one works better [04:19] still weird [04:22] that seems to be working better. [04:25] menn0: lookin [04:25] davecheney: it's all good. axw already reviewed. [04:28] axw: http://reports.vapour.ws/releases/3933 <- note gce [05:00] Bug #1576503 opened: Agents stuck in failed state [05:01] mgz: interesting, seems to be different tho [05:06] davecheney: screw you fslock... [05:06] axw: possibly over quota and bad message? [05:07] axw: we had passes since then [05:08] mgz: hmm dunno, not much to go on apart from failing to get public addresses... [05:08] sounds plausible tho [05:09] mgz: ah yes, GCE operation error: (QUOTA_EXCEEDED) Quota 'CPUS' exceeded. Limit: 96.0 [05:19] Bug #1576509 opened: Race in macaroon-bakery

[05:19] Bug #1576510 opened: Race in macaroon-bakery

=== urulama|____ is now known as urulama [05:39] axw: i've just realised that this SSH host key work has some interestsing corner cases with "juju scp" [05:39] axw: do we care much about "juju scp 0:some.file 1:" [05:39] ? [05:39] axw: it doesn't work by default because the keys aren't in place but it could [05:40] menn0: does scp let you do that? I thought it didn't [05:40] axw: it definitely does... I've just been reading up on it [05:40] so it does [05:40] menn0: I must be living in the past [05:40] uummm [05:41] menn0: what do you mean "keys aren't in place"? [05:41] why is it different to making two ssh connections, and writing to known_hosts twice? [05:44] axw: when you do "scp host1:some.file host2:" there's a connection from your local machine to host1, and then host1 connects to host2 [05:44] with juju there's no keys in place to allow connections between host1 and host2 [05:44] menn0: ah, I see. [05:44] menn0: I would say that's up to the user to manage [05:45] axw: that said, with "scp -3 host1:... host2:" the connections are local -> host1 and local -> host2 [05:45] so that's more likely to work [05:45] except I can't make it work right now [05:46] menn0: I don't really think it's worth worrying about [05:46] axw: with a bit of work I could have juju scp generate a known hosts with the keys for both hosts [05:46] axw: but it does make things a bit more complex [05:47] axw: for now I'll generate a known hosts for the first host [05:47] axw: that covers the majority of cases [05:47] menn0: that will still cover "juju scp /local/file 0:remote/path" right? [05:48] axw: of course. "0:" would be the first (and only host) in that command line [05:48] menn0: ok, just making sure I understood you. sounds good [05:48] cool [05:49] Bug #1576503 changed: Agents stuck in failed state [05:49] axw: a small one for friday? http://reviews.vapour.ws/r/4739/ [05:50] wallyworld: ok [05:52] wallyworld: how does that satisfyPrerequisites function work on ubuntu < trusty? [05:53] wallyworld: oh, this is only for series where mongo-3.2 is available [05:53] never mind [05:53] yeah [05:54] separate command, was thinking it was run by the machine agent [05:54] wallyworld: LGTM [05:54] ty [06:15] have u seen this before? I have just destroyed a controller (had 2 models: admin and default)... [06:15] when i list-controllers, i get empty list (just the headers) as expected [06:15] but when i list-models, i get "error: controller local.volumes not found" [06:15] i would have expected (and have in the past) to see empty list with just headers [06:15] it seems that maybe we are no longer cleaning current-controller ?.. is it bug-worthy?.. [06:22] Bug #1576527 opened: listSuite.TestListJSON got null (showSuite too)

[06:22] Bug #1576528 opened: current controller not cleared on destroy [06:31] Bug #1576534 opened: upgrade-charm refuses to upgrade [06:49] hello everyone [06:49] is 'juju deploy --to lxd:' supported by the maas provider in 2.0? [07:19] bdx: are you talking about MAAS 2.0, or Juju 2.0? Currently supported for Juju 2.0. For MAAS 2.0 & Juju 2.0 currently WIP and only available behind a feature flag and building from tip.... [07:22] frobware: nice, whats the flag? [07:22] frobware: maas2? [07:23] bdx: yep [07:23] frobware: I also must build master? [07:24] will this be dropping into beta7 then? [07:24] bdx: yep, and a moving target... currently very active for maas2 work [07:24] bdx: tbd [07:24] frobware: yea, I've been following along ... entirely. thx [07:25] bdx: we haven't done a CI run yet with this flag, so YMMV [07:25] I have a lab up with juju2.0 + MAAS2.0 + the flag [07:25] just not a build of master :-( [07:27] I'll give tip a build and see what gives [07:28] bdx: I would initially try the sinlge NIC setup, just a hint. :) [07:29] frobware: thanks, by single nic, you mean enlisted maas nodes only have 1 nic ? [07:30] bdx: yep. though things could have moved on since wed/thu when I was last tracking this. [07:30] that way no one has to decide what interface to put the bridge on [07:30] ? [07:31] bdx: we bridge all interfaces now [07:32] bdx: I just recall see a comment from voidspace that he was trying/testing single NIC setup to begin with [07:32] nice, ok [07:33] bdx: single NIC, with multiple VLANs, all end up with their own bridge: http://pastebin.ubuntu.com/16123783/ [08:00] anyone know why aws-china is a separate cloud rather than just an aws region ? [08:06] rogpeppe1, something something firewalls I think [08:06] wallyworld, you around mate? [08:08] frobware: bdx: I was testing containers deployed on machines with a single nic first [08:08] frobware: bdx: that worked and then we got multiple nics working too [08:09] voidspace: ooh nice [08:09] frobware: maas2 work is now feature complete - as soon as we get it passing CI we'll remove the flag [08:09] frobware: so still *technically* experimental, but hopefully not for much longer [08:13] rogpeppe1: different clouds for aws, aws-china and aws-gov because they require different authentication types or secret information for different regions [08:14] anastasiamac_: ok, so you can't use the same credentials across those clouds? that makes sense. [08:15] frobware: we're still getting maas schema failures though (json doesn't match what we expect) [08:15] frobware: this is the latest CI run: http://reports.vapour.ws/releases/3934/job/maas-2_0-deploy-xenial-amd64/attempt/5 [08:15] frobware: I'll fix that [08:17] rogpeppe1: essentially, yes... quoting the spec for more details :D..."There can only be one default credential for a given cloud. If that credential does not work on a particular region then you will need to specify the credential used to deploy in that region. There can only be one default region for a cloud." [08:17] anastasiamac_: thanks [08:18] rogpeppe1: i hope this helps \o/ [08:18] rogpeppe1: there is a similar separation for azure :D [09:01] frobware: standup [09:01] voidspace: arriving... now! [09:06] frobware: babbageclunk: https://github.com/juju/gomaasapi/pull/52 [10:20] fwereade: ping [10:43] hello [10:54] frobware: maas is up and running, but my nodes won't pxe boot to enlist. I've turned on DHCP and downloaded a boot image. What have I missed? [10:55] babbageclunk: boot order on VM nodes? [10:55] frobware: the nodes have their nics first in the boot order. [10:55] frobware: ha ha [10:56] babbageclunk: images imported? [10:56] babbageclunk: eth0 configured to be managed (and dns and dchp)? [10:56] yup - do I still need a 14.04 image imported? [10:56] babbageclunk: you have 16.04? [10:57] frobware: yup [10:57] babbageclunk: what does commisionning node say in your maas settings? (the drop-down in the UI) [10:57] frobware: xenial [10:58] babbageclunk: HO? [10:58] yes [10:59] babbageclunk: https://plus.google.com/hangouts/_/canonical.com/juju-sapphire [11:02] rogpeppe1, pong [11:03] fwereade: np, IS just saw an occurrence of this bug after upgrading to 1.25. fixed by a manual JS script https://bugs.launchpad.net/juju-core/+bug/1516989 (but not quite sure why the upgrade migration hadn't worked) [11:03] Bug #1516989: juju status broken

[11:04] fwereade: so, incident fixed, but not sure why it happened. [11:06] rogpeppe1, huh [11:06] rogpeppe1, sorry, I have only the vaguest memories of writing that script, and I don't recall tracking the fix in 1.25 [11:07] fwereade: np :) [11:07] fwereade: just thought you might be able to provide some insight at the time [11:08] rogpeppe1, PR3860 went into 1.25 on dec 1, it seems -- is it remotely possible this was somehow an upgrade to an unfixed 1.25? [11:08] fwereade: i don't think so; let me check [11:09] fwereade: the version was 1.25.5 [11:09] rogpeppe1, yeah, that should be fine [11:10] rogpeppe1, no further insights here, I'm afraid :( [11:10] fwereade: thanks [11:25] babbageclunk: if you do try my scripts you want to invoke as ... $ VIRT_RAM=1024 ./add-node ... [11:25] babbageclunk: because the default is 4GB, which may cause you a few problems [11:39] babbageclunk: how's your branch doing? [11:40] voidspace: finally got maas set up again, just setting up the private space node so I can have some constraints. [11:40] babbageclunk: ok [11:42] voidspace: it occurs to me I could test the change with storage constraints rather than space ones, sorry. [11:42] voidspace: did you try adding a machine without a constraint? [11:43] babbageclunk: I've added machines, yes [11:44] voidspace: so should I just kick off the merge anyway? Not making anything worse if it doesn't work (although I think it will) and then unblocking you. [11:44] babbageclunk: go for it [11:44] babbageclunk: you might still need to JFDI it [11:45] voidspace: oh, also dimiter made some comments that I should address. [11:46] babbageclunk: cool [11:47] babbageclunk: let me know when it's landed please [11:50] voidspace: wilco [11:51] voidspace: will $$jfdi$$ still run the tests? [11:52] babbageclunk: yes [11:52] ^^^^ [11:53] voidspace, frobware: so it just bypasses blocking? Or it runs them and then merges regardless of the result? [11:53] babbageclunk: bypasses blocking [11:53] voidspace: thanks [12:00] voidspace: building now, also successfully added a machine with a space constraint. [12:01] babbageclunk: awesome! [12:01] great news [12:24] frobware: could you take a look at http://reviews.vapour.ws/r/4741/ please? It is a rather important fix for the last bridge script mods. [12:24] dooferlad: looking [12:25] * dooferlad goes for lunch [12:27] * babbageclunk is lunching [12:43] dooferlad: reviewed [12:47] Bug #1576670 opened: undefined: tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

[13:17] Bug #1576510 changed: Race in macaroon-bakery

[13:17] Bug #1576674 opened: 2.0 beta6: only able to access LXD containers (on maas deployed host) from the maas network

[13:41] Bug #1576686 opened: bundle repository/bundles-lxd.yaml is invalid [13:43] voidspace, frobware: is there any way to get maas-cli 2 installed on wily? [13:43] experimental3 only has xenial. [13:43] babbageclunk: is this because your laptop is running wily? [13:43] frobware: yeah [13:44] frobware: can't try to use/adapt your scripts without it [13:53]