[00:00] Bug #1454697 changed: jujud leaking file handles

[00:05] ericsnow: we still got a vet issue [00:05] provider/vsphere/ova_import_manager.go:269: missing verb at end of format string in Debugf call [00:05] definitely in master and maybe 1.24 [00:06] next time a branch lands could be fixed as a driveby === wgrant_ is now known as wgrant [00:08] wallyworld: ericsnow was working an issue all day =/ [00:09] wallyworld: with any luck we can return to planned work tomorrow [00:09] katco: yeah, i know, just letting him know about an issue from the day before [00:09] there were 2 - looks one one got misssed [00:09] i'll fix as a driveby if it's not done before i next need to land somethong [00:10] wallyworld: kk ty === ahasenac` is now known as ahasenack [00:30] whee, juju find another runtime bug [00:30] https://github.com/golang/go/issues/10844 [00:50] davecheney: Juju test suite ~= golang test suite? ;) [01:03] Bug #1454891 was opened: 1.23.2.1, mongo: document is larger than capped size

[01:05] hatch: we find bugs in go every release [01:05] yay for us! [01:05] maybe we should run those tests before it gets released ;) [01:21] thumper: with your connection time change, does that mean you no longer need a mongo majority to login? [01:22] thumper: I stumbled across "juju status" hanging yesterday when 2/3 of my state servers were dead, I think it's the ~same issue as the one that triggered this? [01:23] actually it was "juju ssh", but same deal [01:23] oh... interesting [01:23] axw: 1.20 didn't have this issue because we weren't recording login time [01:23] but 1.22+ did [01:23] not sure about majority [01:23] but if it needed it, that would suck [01:23] menn0: ideas? [01:24] thumper: I was on 1.22. I think majority only comes in when you try to write, which is why login was getting stuck [01:24] I can test it out [01:24] axw: we still try to write, just not using txns [01:24] ah yes, of course [01:24] hm, bugger [01:24] perhaps we can change the connection requirement [01:25] to try and return straight away [01:25] it is no biggie if it fails [01:27] menn0: do you know why mgo/txn doesn't prune the txn collection? seems like an oversight to me. doesn't it mean in general people's dbs will grow unbounded? [01:27] thumper: think it's worth me raising a bug for that? [01:28] axw: nah, I'll try to do it with this change I have in progress [01:28] okey dokey [01:28] wallyworld: yes, IMO [01:28] thumper, axw: could be a majority thing... not sure [01:28] menn0: how can I change the write requirements for this single update? [01:28] wallyworld: it's an oversight [01:28] menn0: you have become our resident mgo expert now [01:29] menn0: will it be fixed in mgo? [01:29] menn0: thanks for taking that for the team [01:29] * menn0 had noticed [01:29] menn0: make sure when you brace you grip with both hands [01:29] hatch: the bugs we find get turned into tests in the main repo [01:30] ahh cool [01:30] as for running juju over go before its released [01:30] that's me, every day [01:30] haha :) [01:30] wallyworld: i've talked to niemeyer_ and we're going to see what the solution I come up with looks like and see it if makes sense to be in mgo/txn [01:30] ok [01:30] at any rate i'm confident we can fix it within juju [01:31] sure, but that doesn't help other mgo use cases :-) [01:31] juju is the biggest user of mgo/txn so we're hitting these things first [01:31] yup [01:32] wallyworld, thumper: so what I want to do might involve pulling in a golang package that implements a bloom filter [01:32] * wallyworld likes flowers [01:32] wallyworld, thumper: otherwise we either have to do huge numbers of DB lookups or risk eating up lots of memory with a large map [01:32] I haven't heard about bloom filters since bzr days [01:33] a bloom filter fits this problem really well [01:33] menn0: can you explain bloom filters in works of two syllables or less, and how it fits this problem? [01:33] s/works/words/ [01:33] this one looks appropriate: https://github.com/willf/bloom [01:34] thumper: probably not in 2 words :) [01:35] thumper: but we need to know all the transaction ids that are still be referenced by all docs in all collections, and then go through the txns collection and remove the ones that aren't there [01:35] thumper: that set of txn ids could be huge [01:35] thumper: an algorithm on large sets supporting queries returning either "possibly in set" or "definitely not in set" [01:35] thumper: a bloom filter could store that much more efficiently [01:35] os [01:35] thumper: and tells us with 100% certainty when a txn isn't in use [01:35] ok [01:35] geez, what is up with me today [01:36] thumper: it will occasionally indicate that a txn is in use when it actually isn't but that's ok [01:36] thumper: it can get pruned next time [01:36] * thumper nods [01:37] thumper: the alternative is to do multiple db hits for each txn doc to see if it's use [01:37] which is doable but will be slow and i/o intensive [01:37] hmm... [01:37] menn0: the license there says we must copy the copyright for the binary distribution [01:38] thumper: hmmm that's annoying. do we already do that for other packages? [01:39] thumper: i can also hunt around for other implementations... [01:42] thumper: i'm getting called for lunch. bbs [02:06] menn0: when you are back, I'd like to talk about testing [02:23] wallyworld: json schema? I wasn't planning to do a json schema, just using juju/schema to define types and defaults like we have in environs/config and the env providers [02:24] ah, sorry, typo [02:24] wallyworld: did you have something else in mind, or shall I reword the card [02:24] ok [02:24] wallyworld: renamed it [02:24] ty [02:24] thumper: back [02:49] menn0: hangout? [02:49] yep, 1:1? [02:50] kk [03:19] menn0: did you ever figure out what caused this bug? https://bugs.launchpad.net/juju-core/+bug/1420057 We're seeing the same thing with the customer we've all been working on so hard lately. [03:19] Bug #1420057: agents see "too many open files" errors after many failed API attempts

[03:22] Bug #1454919 was opened: destroy-environment should cleanup all .jenvs connecting to environ

[03:23] natefinch: cherylj is on the case (the ticket related to that customer has marked as a dup of the original ticket) [03:23] cherylj: has a test running overnight trying to simulate the issue and before she finished it seemed like she was seeing a growing number of leaking sockets [03:24] we should know more tomorrow [03:24] ahh, cool. I hadn't realized she had been able to repro it. I haven't (but I was using 1.20.14, not 1.20.11, which possibly maybe might have been the problem... but I also wasn't in HA at the time) === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [05:11] wallyworld: the bug links from tanzanite leankit don't work. there's a stray "Bug " in the URL [05:11] bah, will fix, thanks [05:12] axw: done [05:13] wallyworld: thanks === kadams54 is now known as kadams54-away [07:26] o/ [07:45] dimitern: welcome back [07:45] dimitern: I hope you had a good break [07:45] voidspace, thanks! :) yeah, it was much needed [07:45] * dimitern needs to catch up on a ton of mails [07:45] dimitern: :-) [08:01] dimitern: o/ [08:01] dooferlad, morning! [08:04] wallyworld: I've got a high fix for 1.24, the lp:1437266, but cannot merge it because it doesn't match the current restriction [08:05] morning btw [08:09] dimitern: welcome back btw [08:10] dimitern: I'm stepping out in a few minutes, would you retry merging https://github.com/juju/juju/pull/2320 into 1.24 for me? I'll also port the fix later to master [08:26] TheMue, hey, thanks! will do [08:26] dimitern: great, master is also almost done. sadly have to leave now, visiting parents in law [08:27] TheMue, np, see you later then [08:27] TheMue, ah, I see you're actually on leave today :) enjoy [08:27] yep [08:28] dimitern: got Ascension Day and Father's Day here in Germany today [08:28] TheMue, sounds nice - a looong weekend :) [08:29] dimitern: yeeeeeah, and from tomorrow till Sunday the first beer festival here in the city :D [08:29] ah, master runs [08:30] TheMue, oh boy! [08:34] dimitern: so, fix is in for review. you already know https://docs.google.com/spreadsheets/d/1T5RrTFviR4Gpjm5gxOCP5aB7L9BL7_2QzYqQm7fOPHQ/edit#gid=0? just a quick help to get a simple overview. dooferlad will create something automatic based on lp ;) [08:34] * TheMue is stepping out, cu [08:34] TheMue, yeah, I saw that spreadsheet and thanks for organizing this [08:44] dooferlad, hey, looking at https://github.com/juju/juju/pull/2281 it seems to imply only lxc connectivity is restored after a reboot, but it should also be true for kvm right? the forward rules.. [09:01] TheMue: dimitern: dooferlad: problems joining the hangout, I think I have to restart my browser [09:01] be there in a minute [09:01] voidspace, np [10:59] TheMue: sorry, was at soccer, thanks for fix [11:59] katco, you around? [11:59] mattyw: in a meeting and for 2 more minutes :) [12:05] jam: thanks for the feedback, I know you've had some other priorities so I appreciate you taking the time on the spec [12:16] wwitzel3: you're welcome, I did want to meet with you directly. Maybe next week sometime since sky has quieted down? [12:22] jam: yeah, that sounds great [12:38] what as annotations in the api that you can set/read? [12:38] does every "thing" have them or only a limited set? [12:41] jw4: can you set an annotation on an action? [12:42] marcoceppi: I don't think so [12:42] marcoceppi: when I initially wrote the back end I put in the globalKey stuff to use with annotations [12:43] marcoceppi: but then I vaguely recall a discussion with fwereade where he said they wouldn't be used for actions [12:43] marcoceppi: so I'm pretty sure that ability has been removed now [12:43] ah man :( I have a use case where I want to set annotations on a per action level [12:44] oh well, I'll just file a bug for feature then, thanks [12:44] marcoceppi: I don't think it's too difficult to put in - yeah feature request would be good [12:46] Okay, so I have a question about 1.24 juju status-history [12:47] from an api standpoint can I query a point in time and get the status of the environment in that time frame? [12:47] jw4, that was silly of me if I said that [12:47] jw4, I do remember saying we shouldn't *expose* the globalKey outside state [12:47] fwereade: I knew if I mentioned your name carelessly I'd get the real story [12:48] fwereade: you're probably right - I may have misinterpreted your feedback [12:48] fwereade: or it may have even been someone else [12:48] fwereade: it was a while ago [12:49] jw4, marcoceppi: anyway adding it should be trivial -- a version bump to the api and some internal notion of globalKey for actions [12:49] anastasiamac, was it you who did the annotations api change to add charms? [12:49] fwereade: cool, any idea on the whole status-history thing? I imagine it's in the API but I'm having a hard time figuring out where teh latest api docs live [12:49] fwereade: yes [12:50] doc/api.txt hasn't been touched for about a year [12:50] marcoceppi, yeah, I should probably just delete that, it's got so out of date :/ [12:50] marcoceppi, wallyworld will know for sure [12:51] anastasiamac, adding another type shouldn't be too hard, right? [12:51] fwereade: 1line of code and a test :D [12:51] anastasiamac, <3 [12:51] this sounds like something I could even patch :D [12:51] but /me won't [12:51] anastasiamac, (although technically that's an api version bump too, so a *little* more than that, I think?) [12:52] fwereade: m not sure why u'd bump version... [12:52] fwereade: signatures won't change [12:53] anastasiamac, so clients can tell the difference between a server that's refusing to annotate an action because it won't, or because it can't [12:54] fwereade: k and bump the version then :D [12:59] marcoceppi: fwereade: this should be rtreated as usual feature /improvement request, right? :D [12:59] marcoceppi: fwereade: as in, log as bug in lp?.. [13:02] marcoceppi, yes, I think so [13:02] anastasiamac: yes [13:05] jw4: tyvm :D i guess marcoceppi is the best person to describe use case in lp :) [13:05] anastasiamac: yep - he's probably got a spec all entered already [13:06] * jw4 out for a bit [13:06] jw4: :) yes, i think his efficiency is nothing short of legendary [13:06] hehe [13:07] * marcoceppi pings wallyworld about new status stuff [13:08] * wallyworld was thinkng about sleep [13:08] what would marcoceppi like to know? [13:08] * marcoceppi allows wallyworld to sleep [13:08] it's like wicked late, isnt it? [13:08] only 23:08 [13:08] oh, still "today" [13:08] ;) [13:09] yeah :-) [13:09] I'll be quick [13:09] long day though [13:09] sure [13:09] wallyworld: I'm trying to, in 1.24 query the API for the status history, Ideally I'd like to using a timestamp, recreate the status of that environment at that point in time [13:09] is something like that possible? and what API endpoint should I be using [13:09] and I'll start piecing that together from there [13:10] client facade, "UnitStatusHistory" is the method/api name [13:10] args are Kind, Size, Name [13:10] Kind = "combined" or "agent" or "workload" [13:11] Size is how many entries to fetch [13:11] i think default is 20 [13:11] Name is unit name [13:11] so right now you can only query 1 unit at a time [13:11] and there's no filtering as such (eg on time) [13:11] wallyworld: excellent, I'll start from there. Does the statushistory caputre things like configuration changes? or is this really just a history of the status of each unit [13:12] I suppose I'm actually digging in the wrong part of the API for what I want [13:12] it captures hook executions eg config-change [13:12] it also captures action execution [13:12] and juju-run execution [13:12] right, hum, I think I may be going down the wrong API channel. I'll let you capture sleep and just go back to asking random questions of the rest of the devs taht are still awake [13:13] ok, sorry, i wasn't sure exactly what you were after [13:13] but what i said above pretty much describes the history api as it stands today [13:13] we can add to it if it is not suitable as is [13:15] wallyworld: awesome [13:15] well, my goal is to recreate a bundle of a deployment from a point in time [13:15] which means I'd need to know if there's a history of the configuration values or if that's just a bucket, etc [13:16] yeah, so afaik, we don't snapshot state like that [13:16] and need to know how many units were there, so it seems I may just need to replay all the API calls made to juju from start of deployment to X time [13:17] we do log apis calls but the params are redacted [13:17] bleh [13:17] juju doesn't really have what you want as of right now [13:18] I will open a bug then to figure out what'd it take to do this [13:18] thanks wallyworld [13:18] sure [13:18] we could look at taking a copy of state at time X - the state model isn't that large, but there would be all sorts of niggly complications [13:19] not a quick fix for you sadly i don't think [13:23] anastasiamac fwereade: https://bugs.launchpad.net/juju-core/+bug/1455089 [13:23] Bug #1455089: Allow annotations on actions [13:40] gsamfira: bug 1394223 [13:40] Bug #1394223: panic: Session already closed in provisioner tests

[13:40] ...we had a bug with more details than that [13:43] mgz: thanks [13:44] Bug #1455089 was opened: Allow annotations on actions [13:48] * fwereade bbiab [13:48] marcoceppi: tyvm :D [13:53] oh boy, cloud-init userdata generation is broken on precise for cloud-tools packages since the CentOS support landed [13:56] doh [14:01] not sure why my browser is telling me I have to allow the hangout plugin .. [14:01] since it was working yesterday [14:04] natefinch: stand up [14:12] dimitern: in what way? we have precise deploy/upgrade jobs in ci [14:13] mgz, well, cloud-init fails to complete on a precise host, not container [14:14] mgz, still testing and will file a bug [14:18] mgz, any luck? [14:20] jcastro: yeah, should have instructions shortly [14:23] jcastro: I just need to get my own account now, with the trial thing, to see if the signup bit is what I expect [14:23] * mgz reluctanly gives credit card details to them [14:27] mgz, oh ok so if it works for you then I don't need to bring it up at the cross team in 3 minutes? [14:27] ie, it was a config issue Juju works fine? [14:31] jcastro: the auth worked with the details I had, I'll try with the new account in a sec, there may still need to be some code changes [14:31] jcastro: feel free to mention in the cross team [14:31] userpass with the api password (not the account password) should be fine [14:32] huh, I was using the API password [14:32] jcastro: what did you have set as the username? [14:34] there's both a user id and a tenant id [14:34] the one they generated for me [14:34] I am using the tenant-name though, not the tenant-id [14:42] jcastro: so, I think we need to explictly support OS_TENANT_ID [14:44] so like do I need to set the variable or is there a way to put it in environments.yaml? === ericsnow is now known as ericsnow_afk === kwmonroe_ is now known as kwmonroe [14:48] jcastro: I'm seeing if I can get the other value out of the keystone api [14:48] * jcastro nods [14:49] jcastro: I can't see that [14:49] jcastro: but the RC file that you can download on the "Access & Security" page in the dashboard, "API Access" tab, [14:50] has OS_TENANT_NAME in it as well, looks like dhc000000 [14:50] you can set that as your tenant-name in your environments.yaml [14:50] I just don't see it listed anywhere else === kadams54 is now known as kadams54-away [14:51] and our code ignores the alternative tenant-id which is what's displayed === kadams54-away is now known as kadams54 === ericsnow_afk is now known as ericsnow [15:02] yeah I'm using dhc-blah as my tenant-id [15:03] oh sorry, I mean as my tenant-name [15:56] katco: left some notes on your rich status spec. [15:58] hello everyone. has the use of t1.micros been deprecated? the same constraints I used are now trying to launch a t2.micro instead [16:08] jose: Amazon has been deploying fewer and fewer t1.* machines, so we're moving our defaults to t2 [16:08] natefinch: ty [16:09] natefinch: is there a way for me to force the use of a t1? t2s are basically... not useful [16:09] also, it throws errors when I set the constraints [16:12] jose: juju deploy --constraints "instance-type=t1.micro" *should* work [16:13] natefinch: it isn't, ERROR failed to bootstrap environment: cannot start bootstrap instance: no instance types in us-east-1 matching constraints "cpu-power=100 instance-type=t1.micro"" [16:15] jose: yeah, that the error message saying amazon just doesn't have any of those in that AZ. You can try a different availability zone... just add --to zone=us-central-1a [16:15] ah. gotcha, thanks [16:16] jose: also, it's probably never correct to specify instance-type and any other constraint... since the instance type determines all features of a machine [16:16] I didn't specify cpu-power [16:16] I guess it was grabbed by the instance-type [16:17] jose: oh, ok, nevermind then :) [16:17] and looks like that did it! [16:17] thanks [16:17] awesome! :) [16:18] jose: you're welcome. BTW, you can specify the default AZ in your environments.yaml, so you don't have to specify it each time... from my experience us-east is very often low on t1.* machines, so you might want to default to central [16:18] yeah, I was about to do that [16:18] there was an error with the az though, said az didn't exist [16:19] I'll double check on my side and see what I can do, but looks like it's a go for now [16:19] yeah, you might need to leave off the a, so like us-central-1 [16:20] just what I did [16:25] mgz: I have evidently forgotten everything about bzr.... why am I getting this when I bzr pull? bzr: ERROR: Not a branch: "bzr+ssh://bazaar.launchpad.net/~juju-qa/juju-core/ci-cd-scripts2/". [16:26] natefinch: branch changed name? [16:27] perrito666: certainly that branch does not seem to exist [16:27] natefinch: eh, nope. it's not letting me - tried with central and west, and says 'az doesn't exist'. [16:28] jose: hmm [16:28] natefinch: what's that branch got in it locally for you? [16:29] mgz: ci tests for juju... like jujupy.py etc [16:30] natefinch: that's lp:juju-ci-tools [16:30] jose: sorry, should be regions, not availability zones, which is a subtle difference: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html try us-west-1 [16:31] yep, did us-west-1 and us-west-2 and nope. tried setting the region on my environments.yaml to see if that helps [16:32] ok, this is starting to look pretty weird. I can't launch the instances from Juju but I can from the aws console [16:32] in both the regions I've tried to launch them. [16:33] It's not impossible that they put more restrictions on API access than they do on their own console... but I would kinda hope they wouldn't. [16:35] jose: I have to step out for an hour-ish to get my kid from school and make lunch. Hopefully someone else on here can help, like perrito666 or wwitzel3 or ericsnow [16:35] np, thanks [16:35] have a great day! [16:35] jose: you too, and good luck! [16:35] thanks! [16:35] Bug #1454658 changed: TestUseLumberjack fails on windows

=== kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 [17:17] ericsnow: ping [17:18] Bug #1455158 was opened: Destroying a machine with a placed unit errors then destroys anyway.

[17:18] wwitzel3: hi [17:18] ericsnow: my ping was premature, I just realized I hadn't eaten yet and got really hungry the moment I pinged you [17:19] wwitzel3: I have that effect :) [17:19] ericsnow: I'm moving all of the procmanager stuff in to it's own module, but then I'd love to take some of your time to walk through and maybe help flush out the rest of the interface it will be exposing to juju [17:20] wwitzel3: sure [17:20] ericsnow: ok, I'm going to grab some food and then I'll ping you and we can jump on a hangout [17:20] ericsnow: then after we jump, we can just relax and look at the code [17:21] wwitzel3: k [17:43] right, EOD [17:43] g'night all === kadams54 is now known as kadams54-away === kadams54-away is now known as kadams54 === kadams54 is now known as kadams54-away [18:29] https://twitter.com/MEIZU/status/598865451880415232 <-- just in case you are wondering what I want for my birthday :p [18:32] perrito666: looks quite nice [18:34] perrito666: when is your birthday... y'know just in case I'm feeling philanthropic [18:34] lol, may 25th [18:34] oh.. coming up soon :) [18:56] mgz, I'll grab you tomorrow to talk about chaosmonkey if that's ok? [18:58] natefinch: just going to take my dog out and i'll brt [19:00] katco: ok [19:02] mattyw: sure thing [19:30] Bug #1453280 changed: Juju machine service for Windows creates incorrect tools symlinks

[19:43] natefinch: did i freeze? [20:00] Bug #1455222 was opened: juju doesn't see MAAS' "failed deployment" state

[20:00] Bug #1455224 was opened: TestListBlockDevicesDeviceFiltering fails on ppc64el/gccgo

[20:06] Bug #1455224 changed: TestListBlockDevicesDeviceFiltering fails on ppc64el/gccgo

[20:16] is there an environment variable for setting where your .config dir is, or is it always assumed to be in $HOME/.config ? [20:53] natefinch, yes, there is a way to change it. let me think [20:54] natefinch, $XDG_CONFIG_HOME ? from http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html [20:55] sinzui: ahh, yes, now I remember. I thought there was a standard. Thanks. [21:31] sinzui: was the release standup cancelled today? [21:31] wallyworld, yes it was [21:32] ok, i see beta3 is getting released \o/ [21:32] wallyworld, in a few minutes that will be true [21:32] sinzui: did you have a minute to explain to me this upgrade issue? [21:33] wallyworld, sure, do you want to reuse the release meeting hangout? [21:33] yup [21:42]