axw_thumper: is hosted env destruction going to be merged into master before the jes flag is removed? and CreateEnvironment added?01:11
=== axw_ is now known as axw
thumperaxw: CreateEnvironment is already there01:12
thumperaxw: and I hope so01:12
axwthumper: oh, sweet, thanks01:12
thumperI've just found that some of my assumptions about how bootstrap works are wrong01:12
axwthumper: which branch has CreateEnvironment (not PrepareForCreateEnvironment)?01:13
thumpermaster does...01:13
thumperyou mean api, cli, ?01:14
axwthumper: I mean in EnvironProvider01:14
thumperI'm not working on that01:14
axwthumper: remember that discussion about creating resources for azure?01:15
axwthumper: ok, I thought it would be part of multi-env01:15
thumperit has fallen off my todo list01:15
thumperit should be01:15
thumpermy todo list gained weeks of work yesterday01:15
thumperthat floated to the top01:15
axwthumper: heh :)01:15
axwthumper: BTW, I don't know if it'll be useful for other providers, but in Azure it's useful to know the UUID of the controller model01:16
axwthumper: s/useful/critical/01:16
axwthumper: might be worth adding to the env config at some point01:17
axwatm I'm just adding it to the azure config01:17
axwas an internal thing01:17
* axw wishes we had somewhere to store internal data that's not really config01:18
wallyworld_axw: with Offer and ServiceOffer - Offer was from anatasia's branch and it's going away but there' stuff that depends on it. i've added a todo in my next branch but didn't add the todo in the previous branch02:04
wallyworld_sorry for confusion02:05
axwwallyworld_: ok, so long as it dies a quick death02:05
wallyworld_it will02:05
bradmany ideas on why a juju deploy to a container would use the wrong uncompress option?  its trying to use xz on a .tar.gz cloud image tarball02:05
wallyworld_bradm: it's the lxc scripts which uncompress the image02:07
bradmhttps://pastebin.canonical.com/143978/ is the slightly truncated output02:07
wallyworld_juju merely downloads the image for lxc to then use as it sees fit02:08
bradmits definately pulling the tarball from the right looking location02:09
wallyworld_bradm: i know there was recent lxc breakage in wily due to upstream issues, but am not across the details02:09
rick_h__wallyworld_: that was more around the networking02:10
bradmwallyworld_: this is using trusty though?02:10
wallyworld_rick_h__: ah yes, you are right02:10
rick_h__wallyworld_: nothing with the image compression formats that I'm aware of02:10
wallyworld_yeah, correct, i forgot02:10
wallyworld_bradm: all i can suggest is to look at the lxc ubuntu-cloud template which is a bash script to see what it is doing02:11
wallyworld_i think it's in /etc/lxc somewhere02:11
bradmwallyworld_: would you expect juju to be providing a .tar.gz or a .tar.xz ?02:11
bradmhmm, no, not in /etc/lxc02:12
wallyworld_bradm: .tar.gz - we simply download from cloud-images.ubuntu.com. we use the clould-image-query script to find out what to download from a series02:12
bradmI'll find it.02:12
wallyworld_bradm: tl;dr; juju relies on upstream utils02:12
wallyworld_bradm: /usr/share/lxc/templates02:13
bradmwallyworld_: right, but this is all on trusty, I'm a bit concern it just broke02:13
wallyworld_yeah me too02:13
bradmhah, thats exactly where I'm looking02:13
wallyworld_we need to fix obviously :-)02:13
wallyworld_but we need to look upstream to diagnose02:13
bradmtrying to work out if its lxc-ubuntu or lxc-ubuntu-cloud02:13
wallyworld_bradm: i suspect lxc-ubuntu-cloud02:14
wallyworld_the lxc-ctrea script chooses i think02:14
rick_h__bradm: this smells like upstream moved to xz for better compression but juju grabbed the tar.gz image02:15
bradmyeah, its definately hard coded to using xz, according to the script02:15
wallyworlddamn, i'm o sick of this kernel bug killing my network02:16
rick_h__bradm: right https://github.com/lxc/lxc/commit/27c278a76931bfc4660caa85d1942ca91c86e0bf02:16
rick_h__bradm: line 334 in the diff seems about the right place02:17
bradm"lxc-ubuntu-cloud: Replace .tar.gz by .tar.xz and don't auto-generate missing tarballs"02:17
bradmfrom the release notes02:17
bradmrick_h__: hah, snap. :)02:17
bradmsame thing, slightly different direction02:17
rick_h__bradm: yea, can you file a bug on that please and copy myself and cherylj into the bug please?02:18
rick_h__bradm: including your version of lxc, juju, etc?02:18
bradmrick_h__: sure.  bug on where though?02:18
bradmie juju-core or lxc?02:18
rick_h__bradm: and we'll have to see if that needs to be made more flexible (there's an auto detect the format flag we use in juju-gui tarball for xz) or something else02:18
rick_h__bradm: on lxc02:19
rick_h__bradm: we can't get a release of juju out to fix this tomorrow02:19
rick_h__bradm: so weneed to file a backward incompatible bug with them or something. Maybe it'll end up a bug in how juju is getting the image that it's not getting the new ones?02:19
rick_h__bradm: but let's start there and we'll start working together on it please02:19
bradmrick_h__: for sure, easy enough to move bugs around.02:20
rick_h__bradm: ty much and <3 for the catch02:20
wallyworldrick_h__: bradm: juju uses upstream cloud-images-query to get the image url02:20
rick_h__wallyworld: right, so something changed there in lxc that it's thinking a different image should be used? I'm not sure tbh.02:21
wallyworldso if that is telling juju the wrong image, that will ned to be fixed too02:21
wallyworldrick_h__: ubuntu-cloudimg-query trusty released amd64 --format %{url} on my system returns02:23
rick_h__wallyworld: rgr02:24
wallyworldbradm: ^^^^ what does the above return on yours02:24
bradmwallyworld: same.02:25
wallyworldthat's what i would have expected02:26
bradmwallyworld: apparently the lxc container creation template script didn't.  :)02:26
wallyworldthat's what juju downloads02:26
wallyworlddamn :-(02:26
wallyworldso ubuntu-clouldimg-query and lxc are out of sync02:27
wallyworldhow did this not show up in our testing02:27
rick_h__wallyworld: we must not have tested the latest lxc release. This just came out the other day. Is it in backports/etc?02:28
wallyworldah i see. not sure where it lives02:28
rick_h__wallyworld: so this was released 2 days ago02:28
rick_h__bradm: can you look where you get it from?02:28
wallyworldwell i guess our tests will break soon enough :-)02:28
rick_h__wallyworld: hah02:29
axwwallyworld: any particular reason why we need to model "remote endpoints", rather than just passing charm.Relations around?02:29
bradmwe have -proposed enabled02:30
bradmso it looks like it hasn't made it to the main archive yet02:30
wallyworldaxw: you mean for params across the wire?02:31
axwwallyworld: yes02:31
axwwallyworld: I'm reviewing anastasiamac's branch, just wondering whether we need params.RemoteEndpoint, or if we can just use charm.Relation02:32
wallyworldaxw: we want to model wire structs distinct from the domain model. we pass in domain objects to api layer and map to params.*02:32
bradmrick_h__: theres the answer then, its not out in the wild yet02:32
wallyworldaxw: and on the way out, we map params.* back to domain model02:32
wallyworldaxw: but we currently leak params.* eberywhere :-(02:33
wallyworldbecause much of our domain model is defined in state02:33
wallyworldnot is a model package02:33
anastasiamacaxw: this way we could also easily distinguish exported endpoints from native ones, i guess :(... at some stage... if we want.... \o/02:33
wallyworldbradm: glad you caught that before it escaped :-)02:33
axwwallyworld anastasiamac: we're using charm.Relation in apiserver/params for non-remote relations, I'm just trying to understand if there's a good reason to have separate ways of serialising them02:34
axwwallyworld anastasiamac: I'm wondering if it's ever going to be the case that they'll have different information02:35
wallyworldaxw: IMO what's there now then is a mistake, but i could be told otherwise02:35
axwwallyworld: yeah, I know fwereade doesn't like that we just pass charm.X over the wire, but I don't know that having two ways of doing it is a good thing either02:36
axwwallyworld: I was thinking the same thing about your argument for using the term "Endpoint" btw. it's true that "Relation" isn't a good name, but I think it will be confusing to have two ways to refer to the same thing in the codebase02:37
wallyworldthat's worthy of consideration for sure02:37
axwwallyworld: we'll be going from being consistently inconsistent to inconsistently inconsistent02:37
wallyworldaxw: at some point, we need to fix things02:37
wallyworldand with juju 2.0 we can vreak stuff02:37
wallyworldso perhaps this new work is a good place to start02:38
axwwallyworld: can't wait :)02:38
* axw sharpens the axe02:38
wallyworldlet's do it "the right way" now and fix the other stuff after 2.002:38
anastasiamacaxw: we live on the edge! what's wrong with " inconsistently inconsistent"? :D02:38
axwanastasiamac: cognitive overhead02:38
anastasiamacaxw: no sense of adventure :D02:39
axwanastasiamac: I like to get shit done instead of labouring over what something means :)02:39
anastasiamacaxw: wallyworld: i agree that we'd benefit from doing the right thing now... sorry for mudding the mud02:39
axwanastasiamac: no apologies required, I just wanted to check what we should be doing02:40
anastasiamacaxw: i prefer to code :D02:40
anastasiamacaxw: and it was not an apology :P02:40
bradmwallyworld: LP#1515463 if you want to poke at the bug for any reason02:43
wallyworldaxw: that service directory branch should be good to go. a few of the things you mentioned were prior issues with cleanup pending as stuff is glued together over the next day or two02:43
wallyworldbradm: ty02:43
bradmwallyworld: I hope I captured the bug appropriately.02:44
axwwallyworld: ok, looking02:47
rick_h_ouch, irc go boom02:48
wallyworldbradm: looks good. maybe a comment suggesting that ubuntu-cloudimg-query needs to be loooked at02:48
rick_h_bradm: ty you're my hero for catching it in proposed02:48
wallyworldbradm: +102:48
axwwallyworld: LGTM02:49
anastasiamacwallyworld: \o/ plz land!!! :D02:50
bradmah, do I have to make this against a particular lxc version?02:50
wallyworldbradm: the bug triager will allocate to the right version? maybe?02:52
bradmI'd just hate them to miss it and the package get out to the wild02:53
* rick_h_ runs for the night, evening all02:54
wallyworld_bradm: stehpane has commented on the bug. i've also commneted02:58
bradmwallyworld_: perfect.03:00
bradmwallyworld_: looks like its well in hand now then.03:01
wallyworld_bradm: yeah, it does. we'll keep an eye on it at our end also :-)03:02
bradmI really don't care who fixes it where, just that I can deploy containers again. :)03:02
wallyworld_yup :-)03:04
bradmwell, I can by dropping -proposed, thats easy enough for now.03:05
pjdci just noticed that "juju run" stopped working in a 1.24.7 environment: https://pastebin.canonical.com/143985/plain/03:48
pjdcthe unit log has the following: https://pastebin.canonical.com/143986/plain/03:48
pjdcmachine-0.log: https://pastebin.canonical.com/143988/plain/03:49
thumperpjdc: it seems the agent is wedged03:51
thumperif you restart the agent, it should fix it03:52
pjdcthumper: on machine 0?03:52
thumperwhich machine was the log from?03:52
pjdci already restarted the agent on the jenkins unit, which is the first log03:52
thumperthat should fix it03:52
pjdcit's still spewing leadership failure03:53
thumperwas the log before or after the restart?03:53
pjdchere's the restart: https://pastebin.canonical.com/143989/plain/03:54
pjdcand it's just been logging the dying/stopped lines ever since03:54
thumperis the environment HA?03:54
pjdcit's not03:55
thumpercan I see the logs from machine 0?03:55
pjdcfrom before or after the chunk in https://pastebin.canonical.com/143988/plain/ ?03:55
pjdcall i have in machine-0.log lining up with the jenkins unit agent restart is this: https://pastebin.canonical.com/143990/plain/03:56
thumperhow much do you have?03:56
pjdci have everything; the environment is onyl a few hours old03:57
thumperhow big is the environment?03:58
thumperI think the first step is to file a bug for this failure03:58
pjdcpretty small. three machines, one service each, and a few subordinates deployed to each03:58
thumperthen we'll work out how to fix03:58
thumper it03:58
thumpercan I get you to do this:  `juju set-env logging-config=juju=DEBUG` to change the logging level03:59
thumperthen restart the machine-0 agent03:59
pjdcdone and done03:59
thumperthis should give us more output03:59
thumperlet it settle for 20s or so03:59
thumperthen lets look at the logs03:59
thumperpjdc: did you want to open the bug or shall I?04:00
pjdci can open it04:00
pjdcwell, that's annoying04:02
pjdcthe restart seems to have made it work again04:02
thumperthe reason juju run was failing was due to the uniter bouncing04:04
thumperjuju run is executed through the uniter worker04:04
thumperthe uniter was bouncing due to leadership issues04:05
thumperit seems that bouncing the server fixed those issues...04:05
thumperwhich it shouldn't have had04:05
thumperpjdc: to get the logging back to the default, you can say `juju set-env logging-config=juju=WARNING`04:06
pjdcrighto, ta04:07
thumperrick_h_: ping04:07
rick_h_thumper: pong04:07
thumperrick_h_: you moved our meeting to a time I now have busy04:07
rick_h_thumper: was just about to ping you about moving that meeting04:07
rick_h_thumper: ah, sorry didn't show a conflict04:07
rick_h_thumper: what works for you?04:07
thumperrick_h_: if you can't make the earlier one, I'll change my one04:07
thumperrick_h_: it is a gym thing :)04:07
rick_h_thumper: yes sorry, I missed parent-teacher conferences on my personal calendar04:07
thumperthe team lead meeting was moved to my normal gym time04:07
rick_h_I need some way to combine the two better so I don't schedule work stuff over personal stuff04:07
thumperso I booked a personal trainer for a session04:07
rick_h_thumper: hah, ok04:08
pjdcfiled as #1515475, fwiw04:08
mupBug #1515475: "juju run" stopped working after a few hours (1.24.7, newly deployed) <juju-core:New> <https://launchpad.net/bugs/1515475>04:08
rick_h_thumper: well we can move or do tonight or ...04:08
thumperpjdc: ta04:08
thumperrick_h_: as in now?04:08
rick_h_thumper: if you're ok with it?04:08
thumpersure, I have some questions04:08
rick_h_thumper: or I can move it forward 4hrs from where it sits now?04:08
* thumper looks04:09
rick_h_thumper: earlier into the day, not sure if that's too early your time04:09
rick_h_thumper: around your standup time I guess04:09
thumperrick_h_: it is 13:30 now04:09
thumperfour hours earlier is 9:3004:09
thumperwhich is fine04:09
rick_h_what questions? want to do that now and still keep it tomorrow?04:09
rick_h_thumper: or what do we need from here?04:09
thumpertomorrow... as I think we'll need the hour :)04:10
mupBug #1515475 opened: "juju run" stopped working after a few hours (1.24.7, newly deployed) <juju-core:New> <https://launchpad.net/bugs/1515475>04:10
rick_h_thumper: ah I can't...wife is away. doh04:10
thumperrick_h_: can't tomorrow?04:10
thumperrick_h_: we can go fast now if you like04:10
rick_h_thumper: can we do 8:30am? and bump your standup 30min?04:10
thumpercould do 8am04:11
thumperand not bump standup04:11
rick_h_thumper: why fast now? if you're rnuning out are you heading back and we can do the full hour later tonight?04:11
thumperdinner date :)04:12
thumper15 minutes now and some monday?04:12
rick_h_thumper: maybe, will be in london for customer thing monday04:12
thumpermy questions aren't deep04:12
thumperreally? heading to london?04:12
rick_h_thumper: k, let's do that and I'll try to get something else04:12
thumperfor how long?04:12
rick_h_thumper: for 3 days, Tues customer meeting04:12
* thumper chuckles to himself04:13
rick_h_thumper: https://plus.google.com/hangouts/_/canonical.com/rick?authuser=104:13
thumperrick_h_: lets go!04:13
mupBug #1515475 changed: "juju run" stopped working after a few hours (1.24.7, newly deployed) <juju-core:New> <https://launchpad.net/bugs/1515475>04:28
mupBug #1515475 opened: "juju run" stopped working after a few hours (1.24.7, newly deployed) <juju-core:Triaged> <https://launchpad.net/bugs/1515475>04:34
bradmwallyworld, rick_h_: its probably totally pointless testing telling us what we already know, but I just downgraded lxc related packages, put it on held and then redeployed, and its looking good.04:41
bradmI'll mention it on the bug.04:42
bradmeven though it seems well in hand.04:42
wallyworldbradm: by testing, i mean that our CI testing should catch this issue also04:42
bradmwallyworld: right.  has it just not run on proposed, or is there something else going on there?04:43
wallyworldbradm: we don't test with proposed AFAIK. but i guess we should04:43
wallyworldbradm: there are so many combinations of series, substrate, juju version etc04:44
wallyworldadding in proposed adds a whole new axis04:44
bradmwallyworld: indeed.04:44
bradmwallyworld: its just another set of jenkins jobs, right? ;)04:45
wallyworldbradm: yeah, but we don't have enough hardware. hardware is currently on order AFAIK04:46
bradmwallyworld: I know that feeling.04:47
jamfrobware: looks like I'm going to have to miss our standup again... my wife needs me to take her to the Dr today.09:19
frobwarejam: ack. and for the record I might miss tomorrow's as I have a dental appointment.09:24
voidspacefrobware: I assume we're doing juju-core instead of standup?10:00
frobwarevoidspace, I vote we do standup10:01
voidspacefrobware: heh10:01
dimiternvoidspace, our call is cooler :)10:03
voidspacedimitern: our call is way cooler10:04
dimiternvoidspace, so you're coming?10:06
voidspacedimitern: eh, are you serious? we shouldn't miss juju-core should we?10:07
voidspaceeven though not much is happening10:07
dimiternvoidspace, oh c'mon :P10:07
anastasiamacdimitern: voidspace: :(10:08
voidspaceanastasiamac: o/ :-(10:10
dimiternnope, they're done10:29
frobwaredimitern, care to HO around maas/spaces?10:56
rogpeppewallyworld: ping10:57
perrito666life after breakfast is much better11:07
dimiternfrobware, hey, yeah - in 10m?11:07
frobwaredimitern, 30m11:08
dimiternfrobware, even better11:08
frobwaredimitern, any chance you could join this https://plus.google.com/hangouts/_/midokura.com/juju_openstack11:11
dimiternfrobware, ok, just a sec..11:11
frobwaredimitern, thanks11:30
dimiternfrobware, I hope it was useful :) should we make the spaces call now?11:30
frobwaredimitern, please11:31
voidspacedimitern: looks like we can get static ranges from the subnets api in maas 1.911:39
voidspacedimitern: we have to fetch all the subnets (1 api call) and then make an additional call per subnet to get the range11:39
voidspacedimitern: a trivial one that should have been part of my last branch12:14
voidspacedimitern: in terms of the ListSpaces implementation for the maas provider12:16
voidspacedimitern: will we make it part of the networking environ interface?12:16
voidspaceit will only be needed / used for maas12:17
voidspacebut I think it will have to be part of the interface for autodiscovery to use it12:17
voidspaceor I can provide a helper function that does it in the maas namespace, that casts a given provider to a maasEnviron and calls ListSpaces12:18
dimiternvoidspace, looking12:21
dimiternvoidspace, LGTM12:22
voidspacedimitern: thanks12:22
dimiternvoidspace, yes, let's add a Spaces() method, taking no arguments for now (until we need filtering)12:23
voidspacedimitern: add to the interface?12:23
dimiternvoidspace, yes, right after Subnets() should be a good place for it - don't you think?12:23
voidspacedimitern: well, we're extending the public interface for all providers solely for maas12:24
voidspacedimitern: so I quite liked the helper function idea12:24
dimiternvoidspace, nope, we'll use that for all providers eventually12:24
voidspaceif we ever get to shared spaces...12:24
dimiternvoidspace, shared or not doesn't matter12:25
dimiternvoidspace, EC2 can list your env spaces by looking at subnet tags + env uuid12:25
voidspacefor EC2 you just check the model12:25
dimiternvoidspace, or, it can list the global "shared" spaces, when we get there12:25
voidspacethe model is the source of truth12:25
voidspaceif we don't have shared spaces then juju is the source of truth about what spaces there are and you don't need to go to the provider12:25
voidspaceit's only once you have shared spaces (as maas does) that you need to ask the provider12:25
voidspacebut fair enough12:26
voidspaceinterface method it is12:26
voidspacein theory we might need it for other providers...12:26
dimiternvoidspace, we can't get away without shared spaces I'm afraid, we're just postponing the moment where we need to deal with them12:26
voidspaceI'm sceptical it will become the highest priority any time soon12:26
voidspacebut time will tell12:26
voidspacedimitern: my current card (subnet api) may take a bit longer than I imagined (maybe an extra day)12:36
voidspacedimitern: the code itself is simple, but I'll need to extend the gomaasapi test server again...12:36
dimiternvoidspace, sure12:39
dimiternvoidspace, it needs to work and be tested with both legacy and new APIs, and Subnets() is a big part of "maas spaces (basic) support" (the other main part is what dooferlad is doing after bootstrap)12:40
voidspacedimitern: gah, the "reserved_ip_ranges" operation on maas lists all the ip ranges *except* the static range12:43
voidspaceso you can deduce it12:43
voidspaceI might just take the whole cidr minus the dynamic range12:43
voidspacethe other ranges are single ips for the gateway and cluster12:43
dimiternvoidspace, that's not entirely correct12:43
voidspacedimitern: which bit12:43
voidspaceI guess it might not be correct12:44
dimiternvoidspace, static-range != cidr - dynamic-range12:44
dimiternvoidspace, as there might be IPs in neither of the ranges12:44
dimiternvoidspace, however, looking at http://maas.ubuntu.com/docs/api.html (development trunk version)12:47
voidspacedimitern: there's unreserved range too12:47
dimiternvoidspace, there's op=statistics, which claims to include "subnet ranges - the specific IP ranges present in ths subnet (if specified)"12:47
dimiternand even better:12:47
dimiternOptional arguments: include_ranges: if True, includes detailed information about the usage of this range12:47
dimiternvoidspace, I'll give it a go on my maas now as it has all ranges set12:48
voidspacedimitern: the static range there is included as "unused" and is the same range as returned by unused_ip_ranges12:48
voidspacedimitern: I'm going to try reducing the size of the static range (so there are genuinely unused portions of the cidr) and see what happens12:48
dimiternvoidspace, yeah, good idea12:49
voidspacedimitern: my static range is defined to start at - but the unused range starts at 312:50
dimiternvoidspace, perfect! see this: $ maas hw-root subnet statistics 2 include_ranges=True -> http://paste.ubuntu.com/13238168/12:51
dimiternsubnet 2 is my pxe subnet -
perrito666fwereade_: priv ping me when you are around plz12:51
voidspacedimitern: nope12:52
voidspacedimitern: on my maas the static range starts at
voidspacedimitern: however the "unused" range reported by statistics (and by unreserved_ip_range) starts at
voidspacedimitern: I've updated my bug report12:53
dimiternvoidspace, all of the "unused" ranges are part of the static range ( - as defined on the cluster interface; dhcp range is
voidspacedimitern: the cluster configuration has "Static IP range low value" set to
voidspacedimitern: are you saying that ignoring that is the correct behaviour?12:54
dimiternvoidspace, what do you mean?12:55
voidspacedimitern: I'm talking about my maas here12:55
voidspacedimitern: I have static IP range low value set to
dimiternvoidspace, ok12:55
voidspacedimitern: but the range reported as unused/unreserved returns the low value as
voidspacedimitern: it isn't returning the static range (as defined on the cluster interface)12:55
dimiternvoidspace, is used for anything?12:55
voidspacedimitern: but is returning the portion of the cidr unused by anything else12:55
voidspacedimitern: that's the low bounds of the static range12:56
voidspaceit isn't used, but I don't see that it's relevant12:56
dimiternvoidspace, check the web ui for the subnet - e.g. in my case12:56
dimiternvoidspace, ah, I see you point12:56
dimiternvoidspace, "unused" includes IPs not part of static range12:57
dimiternvoidspace, but not assigned, cluster, or dynamic ips12:57
voidspacedimitern: correct12:58
voidspacedimitern: in #juju on canonical, roaksox is saying that it doesn't matter and we should use the unreserved range anyway12:58
voidspacedimitern: and in 2.0 the static range is going away12:58
dimiternvoidspace, awesome news!12:58
voidspacepresumably it will just be implied from cidr - dynamic range12:58
dimiternvoidspace, the let's just do that and not use the node group interfaces12:59
voidspacedimitern: yep12:59
dimiternvoidspace, then the bug I asked you to file is moot and can be closed12:59
voidspacedimitern: it's done12:59
dimiternvoidspace, cheers13:00
voidspaceperrito666: are you moonstone?13:07
perrito666voidspace: I am not13:07
voidspaceperrito666: ok13:08
voidspacetests failed because "your quota allows for 0 more running instance(s). You requested at least 1"13:27
voidspacedimitern: hmmmm... the networks api we're currently using for subnets allows filtering by nodeId (which we use)13:29
voidspacedimitern: I don't think the new subnets api does13:29
dimiternvoidspace, well, provided you use static IPs for all your nodes, this seems to work for me: $ maas hw-juju subnet ip-addresses 2 with_nodes=True -> http://paste.ubuntu.com/13238364/13:32
voidspacedimitern: so, request all subnets, request all ip addresses with node information, then request the unreserved range for every subnet13:33
voidspacethat's hardly simpler than what we currently have :-D13:33
voidspacebut we'll have space information13:33
voidspaceand we do the filtering rather than have maas do it13:33
voidspacematching allocated addresses to subnets to do the node filtering13:34
voidspaceand that's a bunch of stuff to add to the test server as well :-/13:34
dimiternvoidspace, yeah :/ it seems we still need 3 API calls13:35
dimiternvoidspace, but not every time13:35
voidspacewell, we need 1 plus 1 per subnet13:35
voidspaceand if we're filtering by instance id an extra one13:35
dimiternvoidspace, yeah13:35
voidspacealthough for that case we can trim down the number of subnets we need to query13:36
voidspacedimitern: hmmm... ec2 provider allows subnetIds to be empty (meaning list all subnets)13:39
voidspacedimitern: and I remember a bug about that13:39
voidspacedimitern: maas doesn't allow that13:39
voidspacedimitern: apiserver/subnets/subnets.go calls netEnv.Subnets with an empty slice of subnet ids13:41
voidspacedimitern: that will fail on maas13:41
dimiternvoidspace, yeah, because we had no way of linking networks to nodes apart from going via the cluster interfaces13:41
voidspacedimitern: I guess it didn't matter when ec2 was the only platform supporting spaces13:42
voidspacedimitern: but that needs fixing too13:42
voidspaceI bet "juju subnets list" fails for maas13:43
dimiternvoidspace, well, can't it return an error with empty subnetIDs only for the new api?13:43
voidspacedimitern: other way round13:43
dimiternvoidspace, nope, it won't fail as it doesn't hit maas at all - just state13:43
voidspacedimitern: it currently returns an error for empty subnetIds13:43
dimiternvoidspace, yeah, you got me :)13:43
voidspaceapiserver/subnets/subnets.go calls netEnv.Subnets13:44
voidspacedimitern: in cacheSubnets13:44
dimiternvoidspace, that's for "subnet add" only13:44
voidspacedimitern: ah, fair enough13:45
voidspacemaybe not an issue then13:45
voidspaceI won't fix it until we need to13:45
dimiternvoidspace, +113:45
dimiternoh dear.. ci's f*cked again - euca-run-instances: error (InstanceLimitExceeded): Your quota allows for 0 more running instance(s). You requested at least 113:46
mgz_dimitern: hm, the gating job? I'll see what else is up in ec2.13:56
dimiternmgz_, yeah, and we were seeing some weired unit test failures from a parallel universe :) where state.machineDoc doesn't have Principals field (added my aram originally IIRC)13:59
mgz_are you sure the deps are correct? Ci beuilds a completely clean tarball, which is not the same thing as building out of a local GOPATH14:01
cheryljfrobware: I know it's a bit late, but I did verify that your fix resolved the EMPTYCONFIG problem I ran into on maas14:02
dimiternmgz_, well, something's fishy for sure, as machineDoc has "Principals []string" - no omitempty or anything, so it will be there, unless mongo returns bogus docs from the collection14:02
frobwarecherylj, which fix? setting static IP range, or the fix I committed yesterday?14:02
mgz_er, that's not good, the jenkins web ui just went down14:03
cheryljfrobware: I just checked with the latest master, since I saw you had already merged http://reviews.vapour.ws/r/3102/14:03
frobwarecherylj, result!14:03
cheryljfrobware: I think we're going to try to cut a 1.25.1 soon.  Should I move the 1.25 milestone for bug 1412621 to 1.25.2?  or do you think you'll get to make the fix for 1.25 in the next day or so?14:05
mupBug #1412621: replica set EMPTYCONFIG MAAS bootstrap <adoption> <bootstrap> <bug-squad> <charmers> <cpec> <cpp> <maas-provider> <mongodb> <oil> <juju-core:Fix Committed by frobware> <juju-core 1.24:Won't Fix> <juju-core 1.25:In Progress> <https://launchpad.net/bugs/1412621>14:05
frobwarecherylj, happening now/this afternoon. Was on my list.14:05
cheryljfrobware: oh awesome, thanks!14:05
cheryljmgz_: do you think you'll get bug 1512399 merged into 1.25 in the next day or so?14:06
mupBug #1512399: ERROR environment destruction failed: destroying storage: listing volumes: Get https://x.x.x.x:8776/v2/<UUID>/volumes/detail: local error: record overflow <amulet> <bug-squad> <openstack> <sts> <uosci> <Go OpenStack Exchange:In Progress by gz> <juju-core:Triaged> <juju-core14:06
mup1.25:Triaged> <https://launchpad.net/bugs/1512399>14:06
cheryljmgz_: because we should probably get that into 1.25.114:06
beisnercherylj, mgz - yes please :-)   bundletester + openstack provider is in always-false-fail mode atm.14:09
mgz_cherylj: yeah, I should have that finished this week14:10
cheryljok, thanks, mgz_ !14:10
frobwaredimitern, ok to close http://reviews.vapour.ws/r/3088/ as we're not doing 1.24?14:21
dimiternfrobware, yeah, I wanted to keep the branch around until I forward port it, but the PR and RB entries can be closed14:35
dimiternfrobware, done14:37
frobwaredimitern, thanks14:37
mupBug #1515647 opened: Upgrade from 1.20.11 to 1.24.7 fails after machine-0 jujud updates <juju-core:New> <https://launchpad.net/bugs/1515647>14:54
mupBug #1515647 changed: Upgrade from 1.20.11 to 1.24.7 fails after machine-0 jujud updates <juju-core:New> <https://launchpad.net/bugs/1515647>15:00
frobwaredimitern, could I leverage your expertise ... ?15:03
dimiternfrobware, can it wait for a while? trying to do a few things at once here..15:05
frobwaredimitern, ok I'll pester voidspace. Need some help with the vanguard issue ^^15:05
voidspacefrobware: shouldn't bug squad do it15:07
voidspaceI'm trying to do feature work after two weeks on bug squad15:08
mupBug #1515647 opened: Upgrade from 1.20.11 to 1.24.7 fails after machine-0 jujud updates <juju-core:New> <https://launchpad.net/bugs/1515647>15:15
katcoericsnow: natefinch: sorry ubuntu froze on me. it's going to be a bit of a day i can tell15:22
natefinchkatco: heh no problem, we're just bullshitting about providers15:23
frobwarevoidspace, bug squad picked it up; wasn't sure of the process.15:23
frobwarecherylj, replica set issue committed for 1.25 now - https://bugs.launchpad.net/juju-core/+bug/141262115:33
mupBug #1412621: replica set EMPTYCONFIG MAAS bootstrap <adoption> <bootstrap> <bug-squad> <charmers> <cpec> <cpp> <maas-provider> <mongodb> <oil> <juju-core:Fix Committed by frobware> <juju-core 1.24:Won't Fix> <juju-core 1.25:Fix Committed> <https://launchpad.net/bugs/1412621>15:33
cheryljawesome, thanks, frobware !15:33
voidspacefrobware: cool15:51
cheryljfwereade_: you around?16:01
fwereade_cherylj, heyhey16:23
fwereade_cherylj, sorry I missed you16:23
fwereade_cherylj, what can I do for you?16:23
cheryljfwereade_: thanks for the additional info on the instancepoller.  I think that will help simplify some of the work.16:30
cheryljfwereade_: but, how would we track the instance progress with lxc?16:31
cheryljfwereade_: do you have any thoughts on that?16:31
* fwereade_ scratches head vaguely -- not sure how granular the info we can get from lxd is -- is .Status() intrinsically limited there?16:32
fwereade_cherylj, if we use the cloudinit2 report-progress-back, would that help?16:32
cheryljfwereade_: the problem is that all the "interesting things" happen before we return an instance back from StartInstance16:32
fwereade_cherylj, ah damn yes ofc16:33
alexisbvoidspace, ping16:33
* fwereade_ reloading context a bit...16:34
fwereade_cherylj, I think that callback is the cleanest option...16:34
fwereade_cherylj, so I don't *think* we need additional workers16:34
cheryljfwereade_: what do you mean by callback?16:35
fwereade_cherylj, so StartInstanceParams gets something like `StatusCallback func(InstanceStatus, string)`16:36
voidspacealexisb: popng16:36
voidspace*pong even16:36
alexisbvoidspace, 1x1?16:37
voidspacealexisb: ah yes!16:37
fwereade_cherylj, if we need more special tracking after StartInstance I'd hope we could get it via InstancePoller like everything else (with an option on a cloudinit2 alternative/supplement to instancepoller one day)16:38
cheryljfwereade_: the alternative is that we make creating container asynchronous.  We do enough to get the instance Id, return it to the provisioner, then start a goroutine and go about our merry way16:39
fwereade_cherylj, I would prefer not to -- that'd imply that the lxd broker had to accept long-term responsibility for completing the deployment in the face of all possible weirdness16:41
cheryljfwereade_: it would move that retry logic into the container code :)16:42
cheryljfwereade_: I think even with a callback, we have a chicken and egg problem16:43
fwereade_cherylj, but also add a bunch of responsibility for maintaining local state, surely?16:43
cheryljfwereade_: if we report provisioning status on an instance, not a machine16:44
cheryljfwereade_: we still need that instance back from StartInstance before we can report its status16:44
fwereade_cherylj, I'm imagining a StatusCallback implementation will be something like16:44
mupBug #1515647 changed: Upgrade from 1.20.11 to 1.24.7 fails after machine-0 jujud updates <juju-core:Invalid by cox-katherine-e> <https://launchpad.net/bugs/1515647>16:45
fwereade_func(status InstanceStatus, info string) error {16:45
fwereade_if err := machine.SetInstanceStatus(status, info, nil); err != nil {16:45
fwereade_/ etc16:46
cheryljfwereade_: and we could do that before we associate an instance with the machine?16:46
fwereade_cherylj, I think so -- model-wise, instance data is just a satellite of the machine entity -- and so is machine status, and so I think can be instance status16:47
cheryljfwereade_: okay, I can dig more down that path.  Thanks!16:47
mupBug #1515647 opened: Upgrade from 1.20.11 to 1.24.7 fails after machine-0 jujud updates <juju-core:Invalid by cox-katherine-e> <https://launchpad.net/bugs/1515647>16:48
fwereade_so machine has .[Set]Status(), and [Set]InstanceStatus, and there's some AggregateStatus that takes the output of both to build the user-facing status doc16:48
fwereade_cherylj, (and that way we can represent doing-stuff-but-no-instance-id-yet in status)16:49
cheryljfwereade_: ahhh, nice16:49
fwereade_cherylj, a pleasure16:49
mupBug #1515647 changed: Upgrade from 1.20.11 to 1.24.7 fails after machine-0 jujud updates <juju-core:Invalid by cox-katherine-e> <https://launchpad.net/bugs/1515647>16:51
katcoericsnow: can you have a look at http://reviews.vapour.ws/r/3004/ when you have a chance?17:02
ericsnowkatco: will do17:04
katcoericsnow: ty17:06
=== sarnold_ is now known as sarnold
dooferladfrobware, voidspace: Tiny review if you have a moment: http://reviews.vapour.ws/r/3127/17:41
voidspacedooferlad: didn't I already do that...18:26
voidspacedooferlad: it's already on maas-spaces branch18:27
voidspacedooferlad: it shouldn't land on master18:28
mupBug #1515736 opened: juju storage filesystem list  panics and dumps stack trace <juju-core:New> <https://launchpad.net/bugs/1515736>19:15
mbruzekhi mup that is my bug.19:15
mbruzekWho is working on the storage feature?  I found a panic that mup just pointed out.19:16
mbruzekoh I see cherylj already triaged it.19:17
mbruzekThank you cherylj19:17
cheryljmbruzek: np.  I can fix that later this afternoon.   Super simple problem19:18
cheryljBut it is surprising that it landed.  Means no one tried to actually run the command19:19
cheryljI guess it didn't get hit because of the mocking that happens in our unit tests19:19
cheryljmbruzek: do you want me to give you a patched juju to run until the bug is fixed?19:20
mbruzekcherylj: no need19:20
thumperrick_h_: we don't need this meeting in 10 minutes do we?19:21
thumperrick_h_: although we do need to talk about environment users19:21
mbruzekcherylj:  I am just glad it is triage, no hurry on the fix.  I am trying to document the storage feature19:22
rick_h_thumper: up to you, I tried to make sure we had a space in case we did need it19:22
cheryljmbruzek: ah, okay.  Thanks for helping us find these issues ;)19:22
* thumper thinks19:22
thumperrick_h_: yeah, lets chat19:25
mbruzekcherylj: who wrote the storage feature?  I have questions.19:31
cheryljmbruzek: axw19:31
natefinchsometimes I forget how crazy slow amazon is19:55
perrito666natefinch: compared to what?19:56
natefinchperrito666: the local provider, lxd provider... any machine built in the past 5 years19:57
perrito666lol, well if you count the amount of time I spend fixing my machine after some local provider tests I wouldn't be so sure19:57
natefinchperrito666: that's why the lxd provider is so awesome.   I wish our providers were plugins, so I could just use the lxd provider on my current bug (which is on 1.24)20:00
=== akhavr1 is now known as akhavr
mupBug #1515401 changed: destroy-environment leaving jujud on manual machines <ci> <destroy-environment> <manual-provider> <juju-core:Triaged> <juju-core series-in-metadata:Triaged> <https://launchpad.net/bugs/1515401>20:18
mupBug #1515401 opened: destroy-environment leaving jujud on manual machines <ci> <destroy-environment> <manual-provider> <juju-core:Triaged> <juju-core series-in-metadata:Triaged> <https://launchpad.net/bugs/1515401>20:21
mupBug #1515401 changed: destroy-environment leaving jujud on manual machines <ci> <destroy-environment> <manual-provider> <juju-core:Triaged> <juju-core series-in-metadata:Triaged> <https://launchpad.net/bugs/1515401>20:24
* fwereade_ has that unique sinking feeling when he finally finds a strange-looking goroutine at the bottom of the timeout and it leads back to code that... I saw earlier today and annotated with an "I don't think this is right"21:55
cheryljwallyworld: release standup?22:31
katconatefinch: looks like master if open22:40
katconatefinch: kicked off a merge for you22:42
davecheneythumper: lucky(~/src/github.com/juju/juju/utils) % ls23:56
davecheneypackage_test.go  syslog  yaml.go23:56
davecheneygettting there23:56
davecheneyyaml.go is next on the chopping block23:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!