[00:04] thumper: nice work [00:04] thumper: and I knew we'd hit subnets pretty soon [00:04] voidspace: it is coming along nicely [00:04] thumper: that should be very straightforward to do [00:05] voidspace: I'm doing blockdevice, filesystem, and partition in gomaasapi [00:05] thumper: anyway, goodnight - see you tomorrow if you're around [00:05] thumper: cool, thanks [00:05] voidspace: night [00:05] o/ [00:20] axw_: does bug 1539684 ring any bells for you? [00:20] Bug #1539684: storage-get unable to access previously attached devices

[00:25] wallyworld: do you have a moment to review http://reviews.vapour.ws/r/4605/ [00:25] i do [00:26] sinzui: ths is a merge of next into master right? [00:26] wallyworld: yesl sorry. I seem to have missed a whole sentence. [00:26] np, thought it was, just checking [00:34] sinzui: i've eyeballed the changes, looks ok [00:35] and i am landing [00:36] * menn0 is back (had a visitor) [00:37] wallyworld: this is what's in the logs of the controller hosts added after enable-ha: [00:37] 2016-04-14 10:22:55 INFO juju.mongo open.go:125 dialled mongo successfully on address "127.0.0.1:37017" [00:37] 2016-04-14 10:22:55 DEBUG juju.worker.dependency engine.go:479 "state" manifold worker stopped: cannot connect to mongodb: no reachable servers [00:37] 2016-04-14 10:22:55 ERROR juju.worker.dependency engine.go:526 "state" manifold worker returned unexpected error: cannot connect to mongodb: no reachable servers [00:39] menn0: yeah, mongo is great [00:40] wallyworld: we really need those mongodb logs [00:40] * menn0 updates the bug [00:40] yes [00:42] alexisb: you still working on bug 1506225 ? [00:42] Bug #1506225: Failed bootstrap does not clean up failed environment w/o --force and error message is unhelpful

[00:44] wallyworld, nope I haven't loocke dat it in ages [00:44] ok, np, will update [00:51] Bug # opened: 1570651, 1570654, 1570657, 1570660 [01:03] once master has next merged [01:03] we need to remove the next branch [01:29] thumper: i am rebooting to try and fix my camera, will be a little late [01:29] wallyworld: ack [01:38] wallyworld: sorry didn't see message before. I think I've seen the storage bug in LP before, but haven't witnessed the bug first hand [01:38] np [01:39] axw_: i am just triaging bugs so was curious === axw_ is now known as axw [01:59] axw: running late, otp, be a minute or 2 [01:59] wallyworld: sure, ping when ready [02:02] sinzui: can the beta4 stabilisation bug be closed now? [02:03] axw: soon, I need to bump the version. CI will reject all branches that claim to be 2.0-beta4 [02:03] sinzui: ah ok [02:06] wallyworld: can you review http://reviews.vapour.ws/r/4606/ [02:06] sure [02:06] sinzui: lgtm. i have been naughty and snuck in a landing prior to the stampede [02:07] wallyworld: I better make sure CI is paused then because it will fail that revision [02:07] sinzui: sorry [02:07] i can abort [02:07] wallyworld: no need [02:08] wallyworld: Since CI makes version we release like agents. It assumes something terrible has happened if it is asked to makes and test a version it has made and released. [02:09] ah [02:09] wallyworld: CI needs to prevent someone tampering with the agents in streams [02:09] makes sense [02:09] i was being impatient [02:09] axw: ready now [02:10] wallyworld: all is fine. ci is paused, when all is merged I will unpause and rmove the blockl [02:10] tyvm [02:20] wallyworld: and everyone, looks like I need to find some disk space fix merges. [02:20] ah bollocks [02:24] why is mongo on this host === menn0 is now known as menn0-afk [03:13] Bug #1568943 changed: Juju 2.0-beta4 stabilization

[03:22] Bug #1541536 changed: Deployer and Quickstart failed setting annotations because of socket or json parsing

[03:22] Bug #1564791 changed: 2.0-beta3: LXD provider, jujud architecture mismatch

[03:34] Bug #1541536 opened: Deployer and Quickstart failed setting annotations because of socket or json parsing

[03:34] Bug #1564791 opened: 2.0-beta3: LXD provider, jujud architecture mismatch

[03:43] Bug #1541536 changed: Deployer and Quickstart failed setting annotations because of socket or json parsing

[03:43] Bug #1564791 changed: 2.0-beta3: LXD provider, jujud architecture mismatch

=== thumper is now known as thumper-afk === menn0-afk is now known as menn0 [04:16] wallyworld: got a few minutes to talk? [04:17] natefinch: sure, am outside, give me a sec to move [04:17] wallyworld: np [04:18] natefinch: https://plus.google.com/hangouts/_/canonical.com/tanzanite-stand [06:04] should I be able to bootstrap juju2 on ppc64el? [06:05] I get an error about "ERROR invalid constraint value: arch=ppc64el ; valid values are: [amd64 arm64]" [06:08] bradm: I have done so in the past when looking at various pp64el bugs. [06:10] ah, I see my issue now that I say that [06:11] my images didn't sync right for some reason [06:12] yup, booting fine after kicking the sync a bit harder [07:23] wallyworld: would you please take a look at the last diff here, http://reviews.vapour.ws/r/4533/diff/5-6/ [07:23] sure [07:23] wallyworld: the rest has been reviewed, the last rev is to remove models immediately [07:23] awesome [07:32] axw: a couple of questions [07:34] wallyworld: thanks [07:40] wallyworld: I'll add some more tests around the refcount, please see my reply to your first issue [07:40] sure [07:41] axw: ah sorry, i misread the command [07:45] wallyworld: the increment for controller model would have always silently failed, because it was being run before the document is created [07:45] without txn.DocExists [07:45] woohoo [07:45] win! [07:45] lukcily we are adding a test :-) [07:46] naughty thumper :-) [08:45] thumper-afk: frobware: dimitern: a really difficult one to start the morning off http://reviews.vapour.ws/r/4611/ [08:47] voidspace: shipit [08:47] frobware: thanks :-) [08:47] voidspace: +1 [08:47] dimitern: o/ [08:48] frobware: it turned out surprisingly difficult to get rid of AC only in MAAS [08:49] it would've been a lot easier to get rid of all bits of AC [08:49] dimitern: patience... :) [08:50] frobware: yeah, but still - I'm not quite happy with what I came up with eventually [08:51] dimitern: not just removing the branches where we check for the flag? [08:52] voidspace: removing the code in MAAS was easy, but convincing the provisioner/brokers to still work with and w/o AC on AWS, while not breaking MAAS at the same time, has proven to be nasty [08:53] dimitern: ah [08:54] dimitern: so sometimes use the flag but sometimes not [08:54] I see [08:54] the price we pay for dirty PoC-style hacks .. [08:55] voidspace: we do, but a lot of places in the code assumes having the feature flag on is sufficient to use the legacy approach AC (iptables, allocateaddress, etc.) [08:55] dimitern: oh yes, I was just understanding - that does sound much harder than "just remove some code" [08:55] whereas now just the flag is not sufficient, as we also need to check if SupportsAddressAllocation returns true or NotSupported [08:57] frobware: have you been to Vision Express? [08:57] not yet [08:57] ok [08:57] voidspace: going to go around lunchtime [08:57] frobware: ah, cool [08:58] frobware: couldn't bear to miss our daily banter :-) [08:58] voidspace: makes your eyes bleed :-D [08:58] frobware: but on the upside, I figured out how to untangle the contention around discoverspaces still going on when trying to add a container to the bootstrap node [09:03] babbageclunk, voidspace: meeting... === thumper-afk is now known as thumper [09:03] voidspace: standup [09:03] kk [09:30] dimitern, voidspace, dooferlad, babbageclunk: PTAL @ http://reviews.vapour.ws/r/4609/ [09:30] MAAS2 bootstrap and deploy seems to work with my branch!!! [09:31] Bug #1570759 opened: apt-get install juju does not install /usr/bin/juju [09:34] voidspace: congrats all round !! [09:56] frobware: you've got a review [10:35] dimitern: Do you think I can put a $$merge$$ on that gomaasapi PR? [10:35] https://github.com/juju/gomaasapi/pull/40 [10:35] dimitern: I can see the extra bit that needs adding in controller.AllocateMachine after that. [10:36] babbageclunk: let me have a look [10:36] dimitern: you already did! :) [10:37] babbageclunk: ah, yes - I'm ok with landing this - esp. if the storage tests pass in maas? [10:37] dimitern: ? [10:38] babbageclunk: make check passed I presume? [10:38] dimitern: Ok - I'll check it out and run them [10:38] babbageclunk: cheers! [10:40] babbageclunk: :) I'm not trying to be difficult, but folks familiar with storage are not around, so hopefully it has good tests that can show possible regressions [10:40] dimitern: No, that makes sense" [10:43] dimitern: I think its tests are mostly in the canned JSON mould, so there's not anything that would really show regressions, unfortunately. [10:44] babbageclunk: I see, well - we'll take what we can get [10:46] dimitern: :) [10:46] Bug #1570791 opened: ERROR wait: no child processes with juju run on ppc64el [10:47] dimitern: I think we're really the only clients of this part of the api, so if it's wrong it only hurts us anyway. [10:52] babbageclunk: yeah, the feature flag gives at least some piece of mind to fix stuff [11:04] dimitern: thanks. got sidetracked by GCE, but can now bootstrap there too. [11:04] Bug #1570796 opened: container startup issue when juju network management disabled [11:06] frobware: nice! [11:10] voidspace, dimitern: trying to work out what the storage stuff will look like in the constraint map JSON coming back from the MAAS api. [11:10] Is there any way to specify storage constraints from the CLI? [11:11] babbageclunk: yeah, sure - although I haven't done it with MAAS 2.0 CLI [11:13] babbageclunk: in 1.0 I'd use `maas 19-root nodes acquire storage='...' dry_run=True verbose=True` [11:13] or something like that [11:13] dimitern: Ooh, dry_run is handy! [11:13] as for the format of the storage argument, have a look at how's it constructed in maas/constraints.go [11:14] dimitern: ok, will do - thanks [11:33] dimitern: I can get it to refuse to give me machines if I ask for too much storage, but it doesn't seem to matter what I put fror networks (or not_networks), it always allocates me the node. [11:33] dimitern: Am I specifying this right? [11:33] networks=ip:192.168.200.5 [11:35] dimitern: argh - looks like I'm being bitten by docs not being up to date again - I think this is interfaces now. [11:38] dimitern: ok, if I specify interfaces=default:space=0 that works. [11:38] dimitern: I don't really understand what that means, but it seems to do something. [11:49] babbageclunk: 'interfaces' takes a list of items, separated by ; - each item can have a ":" and a list of key-value comma-separated attributes, like space=0 (0 is the ID; but the name should work as well there) [11:51] babbageclunk: so 'interfaces=default:space=0;admin:space=admin-api' means pick a machine with 2 or more NICs, which have addresses from the subnets in 'space-0' and 'admin-api' [11:52] babbageclunk: the not_networks on the other hand apply to the machine as a whole - i.e. not_networks=cidr:10.20.30.0/24 will mean none of the NICs on the machine have access to the subnet with cidr 10.20.30.0/24 [11:52] * dimitern needs to step out for ~30m [11:54] dimitern: thanks, that helps a lot! [11:56] dimitern: And the labels in the interfaces specifications - what do they do? [11:57] dimitern: Oh - do those become the labels in the constraints_by_type map that come back? So you can tie the ids back to the constraints you specified? [11:58] dimitern: (I know you're not there, just asking while it occurs to me.) [11:59] babbageclunk: labels are user-defined, in juju we use either binding names as labels (i.e. when you do `juju deploy mysql --bind 'server=db-space cluster=internal-api'`, juju will construct 'interfaces=server:space=42;cluster:space-62', assuming 42 and 62 are the maas provider ids of the 'db-space' and 'internal-api' spaces [11:59] babbageclunk: I was almost out, but couldn't resist :) [11:59] dimitern: gotcha. [12:00] babbageclunk: anyway - bbs; we could get on a HO later if you want [12:01] dimitern: o/ === babbageclunk is now known as babbageclunch [12:35] Bug #1466514 opened: apiserver has a race in the using of the port number

[12:48] * dimitern is back [12:59] frobware: I just privmsgd you [13:00] and by frobware I meant fwereade [13:17] Bug #1547741 changed: Cannot build on armhf with go1.2 <2.0-count>

[13:35] frankban: any chance of a small review to fix an issue with local charm series detection https://github.com/juju/bundlechanges/pull/21 [13:43] sinzui: I've got the updated public-clouds.syaml up, but I'm thinking I need to delete clouds.yaml before I'm done done. [13:44] abentley: agreed [13:46] wallyworld: lgtm [13:46] frankban: tyvm [13:47] Bug #1567594 changed: upgrade-gui command isn't in the juju tab complete

[13:47] Bug #1567938 changed: juju bootstrap requires network ID as config option on command line although it's specified in clouds.yaml

[14:05] Bug #1570883 opened: imageSuite.TestEnsureImageExistsCallbackIncludesSourceURL fails on centos go 1.6

[14:06] dimitern, voidspace: Can you please look at this? https://github.com/juju/gomaasapi/pull/41 [14:07] babbageclunk: sure, looking [14:08] dimitern: thanks! Was pretty easy to implement by just following what thumper had already done, but took a while for me to understand the constraints and response structure first. [14:19] sinzui, mgz_: I've got my keystone3 test passing using a fake juju client, but I think I'm at the stage where I need a real maas to test with. Would it make sense to use parallel-maas17? [14:20] frobware: I think you are in the best place to review http://reviews.vapour.ws/r/4613/ [14:20] abentley: no maas 1.7 is unsupported and juju2 doesn't know 1.7 or 1.8 [14:21] sinzui: Should I wait for munna to be idle, then? [14:21] abentley: I have run small concurrent loads [14:22] sinzui: I don't think the openstack bundle counts as a small load, does it? [14:23] dooferlad: looking [14:23] abentley: munna would be best, but can you use 1.9 on finfolk as well? [14:23] mgz_: I'm not picky. [14:25] mgz_: Here's what I'm doing: https://pastebin.canonical.com/154458/ [14:25] babbageclunk: LGTM [14:25] dimitern: sweet [14:25] (Yes, I know I'm using the 2.0 endpoint) [14:26] abentley: that all looks reasonable [14:27] mgz_: Okay, I'll try it on finfolk. [14:27] abentley: 1. no, and liberty has never deplpued on ur vmass [14:27] sinzui: Sorry, I don't understand. [14:29] abentley: some OS budles don't deploy on our vmass 1.9. You will learn if yours does and if it does, we gain new OS testing [14:29] sinzui: Okay. [14:30] uh, if you have a bundle that deploys in vmaas I would like to have it too [14:30] * fwereade just had an interesting new life experience: rescuing a cat from a tree. only one scratch drew blood. [14:31] fwereade: :) === rogpeppe3 is now known as rogpeppe [14:31] fwereade: you're now one step on the road to becoming a burly fireman [14:32] perrito666: I will pass it on when we have one [14:32] fwereade: hint for the future, if the cat is mature, just tip it and it will fall ok :p [14:33] fwereade: random question: i just did: "juju bootstrap --upload-tools ec2 aws"; go install github.com/juju/juju/...; juju upgrade-juju --upload-tools; should that have worked OK or is it something you shouldn't do? [14:33] perrito666: We have a landscape and wikimedia bundlles that work with native deploy on maas 1.9 [14:34] fwereade: 'cos i tried to deploy a unit and it's failing because it can't fetch the tools ("no matching tools available") [14:35] anyone else know about --upload-tools vs upgrade-juju ? [14:36] rogpeppe: that should be fine. In fact, I think that's the only way you can upgrade after doing the initial upload-tools on bootstrap [14:36] natefinch: ok, well then it looks like a bug [14:38] natefinch: FWIW upgrade-juju appeared to work (zero exit status) but it did print something about "available-tools...\nbest version...\n" which confused me a but [14:38] bit [14:39] dooferlad: ignore my github comments, doing it in RB [14:39] frobware: ack [14:39] rogpeppe: I haven't upgraded a juju environment.... uh, possibly ever. So, no idea what it's supposed to actually look like. [14:39] natefinch: i used to do it all the time - very useful if you don't wanna wait for ec2 instances to start up again [14:39] natefinch: pity it seems to be broken now [14:40] mgz_: Do we have a config for maas19 on finfolk? [14:40] alexisb: bug 1570035 is cause master failures We need an engineer to fix the test or juju. [14:40] Bug #1570035: Race in api/watcher/watcher.go

[14:41] rogpeppe: well, it really should work, so we should fix that [14:46] rogpeppe: don't forget that you must use --upgrade-tools to upgrade the admin model and then separately upgrade any other models, explicitly providing the version of your newly upgraded tools [14:46] ericsnow: rogpeppe: you can no longer upgrade anything but the admin model with --upload-tools [14:47] katco: so if i use upgrade-juju on the admin model, it breaks all my models? [14:48] rogpeppe: you can upgrade the others separately but not using --upgrade-tools [14:48] rogpeppe: it shouldn't? why do you think that's what would happen? [14:48] katco: that's what happened for me [14:48] rogpeppe: how did they break? [14:49] katco: i did upgrade-juju, then deployed a unit in the default model and it failed 'cos it couldn't get tools [14:49] katco: am just writing up a bug report now [14:49] rogpeppe: kk [14:49] rogpeppe, yeah, I thought that should work [14:51] katco: tbh i didn't know that different models in the same controller *could* have different sets of tools. seems like it might be easy to introduce problems there. [14:52] rogpeppe: well if you're using --upload-tools, you're off a production path anyway. all bets are off [14:53] katco: well, even if you're not [14:53] rogpeppe: i think different models having different agents satisfies the dev, test, prod use-case [14:55] sinzui, ack [14:56] katco, fwereade, ericsnow, natefinch: https://bugs.launchpad.net/juju-core/+bug/1570917 [14:57] Bug #1570917: upgrade-juju: success but then deploy fails [14:57] rogpeppe: what's interesting is that every shop i've been a part of would test the entire installation of juju -- controller and all -- in a dev, test, prod [14:57] rogpeppe: ta for the bug report [14:57] katco: yes, because the controller is an integral part of the behaviour of the system [14:58] rogpeppe: yep [14:58] katco: you can't really usefully test a dev version without testing the server part too [14:59] rogpeppe: i suppose in a multi-user world, you could look at the controller as a separately managed thing [14:59] rogpeppe: to draw an analogy, it is like the OS upon which your app (another model) runs [14:59] rogpeppe: ops people would test the controller in a separate env. app-devs would use dev, test, models, to test their workloads [15:00] mgz_, sinzui: http://10.0.30.100/MAAS/ is giving me "Service unavailable" even after a reboot. I think it's forgotten that it was our parallel-maas19 [15:00] rogpeppe: app-devs might not have access to nor care about the admin model [15:00] rogpeppe: uncharted waters here i suppose [15:00] katco: it depends what reasons one might have for using different juju versions [15:01] rogpeppe: btw upgrade-juju has a -m flag; no need to switch first [15:01] katco: ah [15:01] rogpeppe: If you're interested in an alternative to --upload-tools that uses actual simplestreams, I'd be happy to work with you. I've got lots of experience with simplestreams. [15:02] abentley: tbh this is all overhead on top of what i'm trying to do atm [15:02] abentley: but i'd be interested to see your alternative [15:03] ericsnow: standup time [15:04] rogpeppe: What I have right now is code that generates simplestreams json that can then be formatted as actual simplestreams metadata. lp:juju-release-tools make_agent_json.py [15:05] rogpeppe: I think it would be good to make this as convenient as --upload-tools, so that devs don't need to use --upload-tools any longer. [15:05] abentley: if it was, i'd use it [15:05] abentley: if only simplestreams was simple :) [15:06] how do we configure apt-http-proxy in juju2 now by default for a specific cloud? [15:07] rogpeppe: If you're up for working with me, I'm happy to try. I need dev input to get it to the point where it does what everyone needs. [15:07] frobware, dooferlad, dimitern: spaces sync? Am I in the wrong place, or is it not happening? [15:08] abentley: it's all a bit fraught around here currently, but i'll let you know if i get a few minutes free :) [15:08] babbageclunk: I don't remember that meeting happening recently... [15:08] I mean, I just see meeting notifications and click on them, I don't know nothing. [15:09] sinzui, alexisb, I have to go out in a sec so I can't test in detail but I'm reasonably confident that http://paste.ubuntu.com/15850768/ will fix lp:1570035 [15:09] fwereade, you still around? [15:09] lol [15:09] alexisb, ha :) [15:09] fwereade, you read my mind [15:09] I leave you to it [15:09] thanks! [15:09] that was serendipitous :) [15:09] I will assign the bug to you [15:10] alexisb, sinzui: yeah, I think that's crack actually, I misread something [15:10] alexisb, sinzui: will poke at it while I can and update with what I know [15:11] fwereade, assume, if you can take it great,if you need to hand it off please unassign yourself and send me a note [15:11] alexisb, will do [15:11] alexisb: I can bootstrap and deploy with MAAS2! [15:12] alexisb: actually, to be fair I *could*, I've completed the implementation of Subnets but not yet tested, and I'm about to try a manual check that I can *still* bootstrap and deploy... [15:12] SWEET!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! [15:13] alexisb: if it works I'll point you at the branch [15:13] voidspace, awesome thank you [15:23] Bug #1067213 changed: race: concurrent deployments corrupt secret token

[15:23] Bug #1086236 changed: environs/ec2: concurrent deployments fail creating the s3 bucket.

[15:23] Bug #1570917 opened: upgrade-juju: success but then deploy fails === Spads_ is now known as Spads [15:43] alexisb, I have to stop; proposed a quick solution in the bug [15:43] and unassigned myself [15:44] fwereade, awesome thank you [15:44] katco, ^^^ given this is a blocker it is high priority for th eteam [15:47] well with katco out :) [15:47] natefinch, perrito666, ericsnow what are you guys up to atm?? [15:48] alexisb: finishing up a bugfix.. should be proposing soon [15:48] alexisb: knee deep in 2 bugs [15:48] natefinch, can you pick up the bug fwereade was working on [15:48] lp:1570035 [15:48]