[00:01] alexisb: "cannot determine machine endpoint bindings: endpoint bindings [00:01] for "s#dummy-source" not found'" [00:01] alexisb: not sure what that means in terms of their code change, though [00:02] ok [00:50] 2016-02-02 00:47:47 ERROR juju.worker runner.go:229 exited "toolsversionchecker": cannot update tools information: cannot get latest version: cannot find available tools: cannot read product data, invalid URL "https://juju-dist.s3.amazonaws.com/tools/streams/v1/com.ubuntu.juju:released:tools.sjson" not found [00:51] 2016-02-02 00:47:55 ERROR juju.worker runner.go:229 exited "toolsversionchecker": cannot update tools information: cannot get latest version: cannot find available tools: cannot read product data, invalid URL "https://juju-dist.s3.amazonaws.com/tools/streams/v1/com.ubuntu.juju:released:tools.sjson" not found [00:51] is it _really_ necessary to poll for new tools every 8 seconds [00:55] juju has OCD [01:17] honestly ,checking once and hour would be overkill [01:40] thumper: https://github.com/juju/juju/pull/4259 [01:40] pprof PR for review [01:45] i wish the juju deployer hadn't stopped working [01:45] i know people tell me that i'm using an outdated versin [01:45] but frankly, if we (juju) have to live in a world where we have to keep making the versino in trusty work for the next 3 years [01:45] then the same should apply to the deployer [01:51] axw: do you know things about azure? [01:55] File "/usr/lib/python2.7/dist-packages/deployer/env/go.py", line 96, in get_cli_status [01:55] status = super(GoEnvironment, self).get_cli_status() [01:56] cherylj: thumper who changed the default juju status to be _not_ yaml [01:56] that has broken the juju-deployer shipped in trusty [01:58] yup, that's the problem [01:58] cherylj: where should I log this bug ? against juju [01:58] or against the deployer that it needs to call juju status with an explicit formatter ? [02:00] https://bugs.launchpad.net/juju-core/+bug/1540697 [02:00] Bug #1540697: Recent change to juju status has broken compatibility with trusty juju-deployer [02:05] mwhudson: I do, sorry, missed your message [02:05] axw: heh you answered the thread anyway [02:07] davecheney: the (informal) plan for 2.0 is that anyone who needs structured output needs to explicitly specify the format [02:07] davecheney: otherwise you get tabular, and its format might change over time [02:08] wallyworld, axw: git blame says you guys might know about the implementation of deploying bundles with juju core? [02:08] so long as that holds, deployer needs to be changed [02:08] natefinch: I know some, I added storage support [02:08] i guess deployer in trusty needs to be sru-ed then? [02:08] natefinch: francesco did the work, we tweaked as needed [02:09] wallyworld: is it me, or do we just drop all the flags (except storage) on the floor if you target a bundle? [02:09] Bug #1540697 opened: Recent change to juju status has broken compatibility with trusty juju-deployer [02:09] davecheney: deployer in trusty will work with 1.25 [02:09] tht is our mandate [02:10] natefinch: which flags? [02:10] 2.0 will be co-installable with 1.25 andf 1.25 wil be supported for 2 years [02:10] axw: juju deploy flags [02:10] 2.0 will not be backwards compatible [02:10] we will support juju in trusty via 1.25 [02:11] axw: like --constraints -n --networks etc ... looking at the code, it seems like we woudl totally let you 'juju deploy cs:/bundle/foo/bar --constraints=blah' and not complain, but completely ignore the constraints [02:12] axw: it's the "not complain" part that bothers me [02:12] natefinch: looks a bit like it. I'd say because it kind of changes the meaning of deploy entirely. not one service, but multiple [02:12] natefinch: yeah [02:12] we should error if you specify them [02:12] axw: exactly my thinking [02:13] natefinch: I added a bundle-specific syntax to the --storage command that allows you to specify for services in a bundle. we probably should do that for other things, but erroring would be a good start [02:14] axw: yep. ok, just wanted to make sure I wasn't missing anything obvious before filing a bug [02:25] wallyworld: ahh, ok [02:25] thumper: http://paste.ubuntu.com/14854594/ [02:26] davecheney: we shoudl try and make deployer work though if possible i think. 2.0 has deployer built in but i don't think it is yet fully functionally complete [02:26] this is on a modest sized deployment [02:26] wallyworld: it's a trivial chang to the deployer [02:26] but that has to be backported to trusty [02:26] which is not trivial [02:26] davecheney: nice [02:26] davecheney: it is - my other concern is that in 2.0 old yaml fields like agent0state are gone [02:26] thumper: i'm going to commit this and backport it [02:26] so we can investigate that failure [02:26] ack [02:27] so deployer may need subsequent tweaking [02:29] precise environments still have two mongodbs installed ... [02:29] root@ip-10-250-24-28:/tmp# ps auxww | grep mongo [02:29] mongodb 5462 0.6 0.9 372940 38264 ? Ssl 01:48 0:15 /usr/bin/mongod --config /etc/mongodb.conf [02:29] root 6410 3.2 1.2 3026932 46808 ? Ssl 02:28 0:01 /usr/bin/mongod --auth --dbpath /var/lib/juju/db --sslOnNormalPorts --sslPEMKeyFile /var/lib/juju/server.pem --sslPEMKeyPassword xxxxxxx --port 37017 --noprealloc --syslog --smallfiles --journal --keyFile /var/lib/juju/shared-secret --replSet juju --ipv6 --oplogSize 512 [02:29] root 6511 0.0 0.0 8100 932 pts/1 S+ 02:29 0:00 grep --color=auto mongo [02:29] our mongob does not restart if the process dies for any reason :( [02:30] really? [02:30] isn't that what upstart or systemd does? [02:30] kill 6410 [02:30] process does not restart [02:32] root@ip-10-250-24-28:/tmp# initctl list | grep juju [02:32] juju-db stop/waiting [02:32] juju-clean-shutdown stop/waiting [02:32] jujud-machine-0 start/running, process 5548 [02:32] process does not restart [02:39] Bug #1512718 changed: Leader election fails with "leadership failure: leadership manager stopped"

[02:39] Bug #1540697 changed: Recent change to juju status has broken compatibility with trusty juju-deployer

[02:39] Bug #1540701 opened: juju deploy silently ignores most CLI flags

[02:40] wallyworld: under which situation will a service not have a status doc? [02:40] thumper: never; there ws a bug in an early 1.24(?) release where that could have been the case [02:41] wallyworld: so I can assume that there will always be a status doc for services? [02:41] for migration? [02:41] yep [02:41] awesome [02:41] ta [02:41] the status doc is written first [02:41] in the []txn.Ops slice [02:42] kk [02:55] thumper: master is blocekd [02:55] can I have your permission to land this change ? [03:12] thumper: and environment with 11 machines, has 51 firewall watchers running [03:12] 51? [03:12] wow [03:12] no, sorry, [03:12] 20 @ 0x437c2a 0x446482 0x445472 0xa3f46f 0xa40234 0xd53b36 0x464e41 [03:12] # 0xa3f46f github.com/juju/juju/worker/firewaller.(*machineData).watchLoop+0x43f /home/dfc/src/github.com/juju/juju/worker/firewaller/firewaller.go:767 [03:12] # 0xa40234 github.com/juju/juju/worker/firewaller.(*Firewaller).startMachine.func1+0x44 /home/dfc/src/github.com/juju/juju/worker/firewaller/firewaller.go:243 [03:12] # 0xd53b36 github.com/juju/juju/worker/catacomb.Invoke.func3+0x76 /home/dfc/src/github.com/juju/juju/worker/catacomb/catacomb.go:115 [03:12] 20 [03:13] we do have 58 moingodb conections in use [03:13] http://paste.ubuntu.com/14854643/ [03:13] it's just a bloodbatch [03:13] it's just a bloodbath [03:13] and this environment is idle [03:15] davecheney: I've added my mark on the github PR [03:16] davecheney: try to land it now [03:34] ta [03:42] cmd/pprof/pprof.go:61:2: cannot find package "runtime/trace" in any of: /usr/lib/go/src/pkg/runtime/trace (from $GOROOT) /mnt/jenkins/workspace/github-merge-juju/tmp.f4uCbWbBZu/RELEASE/src/runtime/trace (from $GOPATH) [03:42] shake's fist at Go 1.2 [03:43] * davecheney deletes things til go 1.2 stops complaining [04:03] Bug #1540316 changed: provider/maas: bridgescript should support vlans

[04:06] Bug #1540316 opened: provider/maas: bridgescript should support vlans

[04:15] Bug #1540316 changed: provider/maas: bridgescript should support vlans

[04:25] wallyworld: not quite as big as your diffs, but sorry all the same: http://reviews.vapour.ws/r/3698/diff/# [04:26] revenge is sweet [04:26] on a positive note, net reduction of ~1600 lines [04:49] axw: reviewed, i love seeing code deleted [04:49] wallyworld: thanks [04:53] wallyworld: there's a card for creating the second model already, is a TODO necessary? [04:53] nah [05:26] thumper, each client of a 1.25 api server consumes 50 goroutines === bradm_ is now known as bradm [06:17] wallyworld: how do you feel about this? https://github.com/juju/juju/compare/cloud-credentials...axw:cloud-credentials-detect-regions [06:23] axw: love the name! detector :D [06:24] axw: looking [06:29] axw: it does give a way to avoid having a personal clouds file for openstack, if standard env vars are set via sourcing a credentials file or whatever [06:29] i think it makes sense [06:29] and we can use an lxd env var for the lxd host [06:30] wallyworld: might not even need to, you could list the remotes [06:30] wallyworld: e.g. "lxc remote list" [06:30] wallyworld: and report each of those as a region [06:30] true [06:31] wallyworld: I'll tidy this up and propose then. it's a bit less hacky than what was there, and supports openstack. we can revisit if there's dissent [06:33] yup [09:09] dimitern: thanks for the review, I replied to all your comments. [09:10] dimitern: is your WIP branch ready yet? [09:11] voidspace, it is ready - the ci run last night revealed the main issue, which I'm fixing now: don't assume bindings always exist [09:12] voidspace, so I'll be pushing a few more commits on top to fix that and re-enable the tests I disabled [09:12] dimitern: cool [09:28] voidspace, responded to your replies [09:33] dimitern: I think you're actually wrong about the test channel - the test really is the owner as it closes the channel. [09:33] dimitern: and the "code path" is still the same. [09:33] dimitern: the other two issues - ok. [09:35] voidspace, I don't mind that much if you leave the chan creation outside of the patched func [10:02] dimitern, standup [10:08] frobware: dimitern: dooferlad: PR for merge of latest master changes back to maas-spaces. http://reviews.vapour.ws/r/3703/ [10:12] voidspace, LGTM - you'll likely manage to land this before my branch which also includes it, so there shouldn't be a conflict [10:13] Bug #1540832 opened: Hard wired bridge prevents the use of the fan overlay network [10:16] Bug #1540832 changed: Hard wired bridge prevents the use of the fan overlay network [10:19] Bug #1540832 opened: Hard wired bridge prevents the use of the fan overlay network [10:51] dimitern, dooferlad, voidspace, jam: the trusty/series issue we talked about in the standup is being tracked here atm: https://bugs.launchpad.net/juju-core/+bug/1540771 [10:51] Bug #1540771: Bootstrap fails with trusty is not a valid distro_series

[10:53] mgz, looking for help with bug 1540771. dimitern plans to update his branch again (RSN) and we'll want another CI run but little point if we keep running into said bug. [10:53] Bug #1540771: Bootstrap fails with trusty is not a valid distro_series

[11:22] frobware/dimitern/voidspace: Updated bind syntax: http://reviews.vapour.ws/r/3569/diff [11:23] dooferlad, cheers, looking [11:34] dooferlad, reviewed [12:07] dimitern: http://reviews.vapour.ws/r/3569 addressed those issues. [12:28] dooferlad, thanks! let's get it in then I guess [12:38] Bug #1540900 opened: juju deploy ignores model default-series [12:46] fwereade_: ping [12:47] Bug #1540900 changed: juju deploy ignores model default-series [12:47] frobware: what's the release notes link again please [12:48] frobware: never mind, found it [12:48] mfoord, thought so as you show up as already viewing. :) [12:53] frobware: heh [12:56] Bug #1540900 opened: juju deploy ignores model default-series [12:56] fwereade_: I had a question about "connects", and another one about some lxd configuration stuff [13:17] jam, heyhey [14:01] cherylj: dimitern: frobware: alexisb: given that 2.0 is dropping maas 1.8 support we should probably not mention improved support for maas 1.8 in the release notes (there's a maas 1.8 compatibility section) [14:01] bug 1534636 [14:01] Bug #1534636: Destroying a hosted model in the local provider leaves the controller unusable

[14:01] jam: saw your doc, but punting until I get the maas-spaces unit tests to work on ppc64 [14:02] cherylj: dimitern: frobware: alexisb: we can probably also drop the mention of bug 1534636 as a known issue, as the local provider won't [14:02] *won't be there in 2.0 [14:02] * voidspace lunches [14:02] * frobware follows suit [14:06] cherylj, frobware, alexisb, voidspace, I think that section was copied from 1.25 rel. notes, so we should indeed drop it for the 2.0 notes [14:16] fwereade_: hey, still around? [14:36] bbl [14:41] jam, ha, yes, but apparently notification-blind [14:42] jam, any reason why we shouldn't do $$merge$$ on this https://github.com/juju/juju/pull/4131 ? [14:44] mattyw: yes, we are trying to get some branches landed and we'd rather not have master be a moving target [14:44] cherylj, ok cool thanks [14:45] frobware: I see now that maas-spaces has some new commits. What's the expectation now for testing? are we still testing dimitern's branch? [14:47] cherylj, ok - judging by the comment lxd provider is broken in xenail so it probably needs to land before the release - but I don't really know [14:47] cherylj, I'm not involved - I'm just an interested party [14:48] mattyw: we can look at getting it into alpha2, but really, if it misses, alpha3 is next week [14:48] cherylj, ack [14:59] cherylj, yes, new commits for blocking API and bootstrap whilst discovery taking place (this is the one we talked about last night). bind syntax change for '@' -> '=' which came out of CT. and merge current master into maas-spaces. [14:59] cherylj, dimitern is updating and fixing tests based on the CI run. [14:59] frobware: is dimitern's branch up to date with maas-spaces again? [14:59] cherylj, dimitern: do you plan to rebase maas-spaces onto your branch? [15:00] otherwise we need to get a bless on dimitern's, then merge into maas-spaces, then get a bless on maas-spaces before it can all land in master [15:00] cherylj, ack. [15:00] cherylj, I would advocate that dimiter rebases so we only do test once [15:01] frobware: agreed [15:01] and don't forget to pull in master too. There were some changes that came in [15:01] cherylj, we did yesterday - need to check if we're talking about the same change. [15:01] frobware: probably same one. the pprof thing [15:02] cherylj, yep [15:02] cherylj, 3 things to fix in dimiter's branch: CI test failures surrounding bindings, skipped unit tests (though for expediency we could skip these), and ppc64 unit test failures. [15:02] frobware: wallyworld has a PR up for a different bug that's only been seen on his branch (although the underlying problem exists on master too). I think it should go into his branch rather than master so we can demonstrate in CI that it fixes the problem [15:03] frobware: what's the confidence level in getting a bless on dimitern's branch today [15:05] cherylj, ppc64 might still fail. looking at that now. need to hear from dimiter about progress on the other problems. also bug 1540771 might cause a cursed build again. [15:05] Bug #1540771: Bootstrap fails with trusty is not a valid distro_series

[15:06] frobware: is that bug a problem with the MAAS setup? or a problem on the branch? [15:07] cherylj, I would say MAAS setup and/or CI setup. [15:07] frobware: is there someone who can work with jog_ to track it down so we don't hit it again? [15:09] cherylj, put me down, though I have a hard commitment 7:30pm. [15:11] frobware: ok, jog_ should be in before then [15:11] cherylj, I'm not entirely sure how quick a CI run can be but I would advocate we have a stripped down run that initially only runs the failures; we need to iterate faster [15:12] frobware: yes, but your changes could cause some of the previously passing tests to fail, so we need to get a complete run before a merge. We could see about doing some initial tests for verification before a complete run [15:12] sinzui: is there a way we could just run selected tests for maas-spaces? [15:12] mgz: ^^ [15:12] cherylj, agreed. [15:13] cherylj, I just don't want to wait $N hours to find out we haven't fixed the existing failures [15:14] frobware: yeah, I understand [15:18] cherylj: we can re-run selected tests for previously tested revisions. We cna do this beause we saved the built packages and streams. We cannot run tests with a branch. [15:21] cherylj, do you know if we have access to ppc64? [15:22] I'm futzing trying to fix this unit test failure [15:22] I have a qemu ppc64 running but it's not terribly efficient or productive. [15:22] actually, I cannot build juju atm (on ppc64) [15:24] mgz, sinzui, is it possible to get frobware access to the CI ppc machines? [15:26] cherylj: yes, though I am informed that ports.ubuntu.com is down and that limits what we can do. No installs [15:26] sinzui: should be okay, it's just a unit test run [15:26] sinzui, cherylj: ah, that explains some of my problems over the last 20 mins. [15:26] d'oh [15:34] frobware, dimitern, I've copied over the charms that CI uses for the basic deploy tests here: https://private-fileshare.canonical.com/~cherylj/dummy-charms/ [15:34] there's also a txt file with the steps used to deploy [15:35] you can start with testing on AWS, since that's failing on the maas-spaces-controller-space-config [15:37] dooferlad, if you're between tasks please could you help out on the test failures ^^ [15:37] frobware: I am not quite, but will be soon. [15:40] ericsnow: it occurs to me that we need to have the resource download code on the server regardless, to support the GUI [15:41] natefinch: I wasn't aware of any GUI-related requirements [15:42] ericsnow: I think rick_h__ would probably be upset if we couldn't deploy charms from the gui that used resources in the charmstore [15:42] natefinch: regardless, you make a good point about what the GUI might need [15:42] ericsnow: but you're right that there was nothing in the spec about it [15:42] fwereade_: would you be able to review https://github.com/juju/juju/pull/4266 ? It's related to that bug I talked to you about yesterday [15:42] natefinch: I can follow up with the GUI team === jog_ is now known as jog [15:45] natefinch: can you email the context there? [15:47] rick_h__: sure [15:49] rick_h__, natefinch: I just send an email [15:49] natefinch: ty on my phobe aylt the conference and want to better follow what's up [15:51] cherylj, alexisb: ack [15:53] natefinch: let's get back together some time after lunch to put together the demo for last iteration [15:53] ericsnow: ok [16:13] hi frobware, I'm here if you have questions about bug 1540771 [16:13] Bug #1540771: Bootstrap fails with trusty is not a valid distro_series

[16:15] jog, immediate question is if we run again will this just repeat. is this at related to not running against MAAS 1.7 or 1.8? [16:18] frobware, it failed several time in a row in the same way. Master had run before maas-spaces and passed. The bug is open for failures against MAAS 1.9 [16:19] jog, but new against 1.9? [16:19] frobware, not sure what you mean? [16:22] jog, trying to understand whether it is our branch that causes the failure or a separate 1.9 issue [16:24] well I just reran the last revision of master (1cb8f03) and don't see the failure there. I'm running the maas-spaces revision that failed again now, just to be sure there was no other issue. [16:27] dimitern, ^^ [16:28] jog, thx - will keep an eye out for the result. [16:35] jog, is your "reran" http://reports.vapour.ws/releases/3564 [16:35] jog, because that show as cursed [16:36] jog, ah, but only for ppc64. [17:01] man, the function that actually does charm deployments from the client is a hairy mess [17:02] 100 lines of if statements [17:03] how sad [17:04] and weird implicit meanings to error values... good lord [17:07] perrito666: https://github.com/juju/juju/blob/master/cmd/juju/commands/deploy.go#L249 [17:08] deploy charm Or bundle, just in case [17:08] is there a reason why that is not the at least 3 methods it should be? [17:09] perrito666: the problem is that our deploy command does way too much and has way too much encoded into the string we pass it [17:09] "we might have been given a bundle" <-- that is one. " if not a bundle then maybe a local charm" <- there is another, [17:10] the fact that this code is a nightmare is just an indication that the UX is a nightmare... [17:10] I mean, the code could be better even given the bad UX [17:10] that is not true, code can be bad and ux be decent [17:11] I have glanced at that function and already know it can be split [17:11] I do have the "fresh pair of eyes" advantage [17:11] yeah, it absolutely should be refactored [17:11] but cmon, a bit of good will might make things better [17:11] I think someone added bundles and just didn't want to refactor what was there [17:12] well mcdonalds lunch is not the best alternative to stay awake after lunch [17:12] heh [17:21] bbl [17:24] oh how nice, we're throwing away an error and assuming we know what was in it [17:25] oh hey, someone fixed it on master... ok, I gotta rebase. [18:08] dimitern: we have 19 code branches checking for the AddressAllocation feature flag [18:16] voidspace, so next step is to prepare a PR that drops them I guess [18:24] dimitern: did we ever get devices by default on master? [18:24] dimitern: it looks to me, from skimming the code, that if we lose AddressAllocation we lose containers as devices too [18:25] dimitern: and we go back to leaking DHCP leases [18:39] bq [18:53] voidspace, that's ok - we'll redo it properly anyway === StoneTable is now known as aisrael === TheRealMue is now known as TheMue [19:23] gah, why do we have a client api function that no one calls? (ServiceDeployWithNetworks) [19:25] .....maybe we keep it so the tests can test the server's backwards compatibility [19:26] hmm, in which case it should be in test code only. [19:27] Bug #1540469 changed: [ARM64][LXD Provider][2.0-alpha1-0ubuntu1~16.04.1~juju1] juju run command ERROR fork/exec /usr/bin/ssh: cannot allocate memory

[19:30] Bug #1540469 opened: [ARM64][LXD Provider][2.0-alpha1-0ubuntu1~16.04.1~juju1] juju run command ERROR fork/exec /usr/bin/ssh: cannot allocate memory

[19:32] natefinch: are you sure no one calls it? [19:32] perrito666: grep says it's only called in tests [19:32] I do recall that there is a cascade of calls for service deploy [19:32] ServiceDeploy -> serviceDeployWithX -> ServiceDeployWithY and so on [19:33] natefinch: take a look a few versions back, perhaps its legacy and asking to go [19:34] perrito666: yeah, almost certainly... like I said, my guess is that it needs to stick around for testing that we keep backwards compatibility with old clients... but that means it should be moved to test code, so it's more obviously something you should let rot in a corner [19:34] actually, if you are working in 2, that is a death sentence [19:36] Bug #1540469 changed: [ARM64][LXD Provider][2.0-alpha1-0ubuntu1~16.04.1~juju1] juju run command ERROR fork/exec /usr/bin/ssh: cannot allocate memory

[19:38] perrito666: *shrug* I'll file a bug [19:39] Bug #1540469 opened: [ARM64][LXD Provider][2.0-alpha1-0ubuntu1~16.04.1~juju1] juju run command ERROR fork/exec /usr/bin/ssh: cannot allocate memory

[19:42] Bug #1540469 changed: [ARM64][LXD Provider][2.0-alpha1-0ubuntu1~16.04.1~juju1] juju run command ERROR fork/exec /usr/bin/ssh: cannot allocate memory

=== natefinch_ is now known as natefinch [20:03] dammit... something is throttleing the hell out of my laptop lately [20:03] natefinch: use htop? [20:04] aghh, sms spam, just what I needed [20:05] perrito666: htop is very pretty and I have no idea what it means [20:05] natefinch: use the man page [20:06] if the upper graphs for microprocessor are full (or the ram one) something is behaving wrongly, you can use f6 iirc to sort things by cpu or ram [20:06] and that would put the culprit at the top [20:09] perrito666: I'm running tests, and expect my processors to all be pegged, but they're all at like 50%, and my tests are taking a lot longer than usual to run [20:10] natefinch: try iotop then [20:10] perhaps something is hammering your hd [20:13] hmm disk read is always 0, I presume that's a bug, or my computer would not be working ;) [20:28] that weird feeling when you change the signature of like 3 very important functions.... and no tests fail [20:28] ...I'm sure it's fine [20:29] natefinch: http://www.quickmeme.com/img/a7/a7eb19ae524e1d066fae16e5e7c0438c86fa321468cd3e3d0764731cf86f869c.jpg [20:29] lol indeed [21:04] no menn0 today ? [21:04] paging the on call reviewer: http://reviews.vapour.ws/r/3700/ [21:04] to the OR, STAT! [21:05] that is me [21:05] * perrito666 reviews [21:05] stat? [21:06] perrito666: means "right now" [21:06]