[05:02] Bug #1597601 opened: ERROR cannot deploy bundle: cannot deploy application: i/o timeout

=== frankban|afk is now known as frankban [08:23] thumper: fix it then ship it [08:42] thumper: hey ... any estimate when this will be fixed and shipped? :) http://reviews.vapour.ws/r/5185/ [08:42] urulama: before lunch [08:43] * urulama lunches in 15min then ... [08:43] :) === wwitzel3_ is now known as wwitzel3 === ionutbalutoiu_ is now known as ionutbalutoiu [09:31] katco: ping [09:32] (oops, too early) [09:32] perrito666 or natefinch: http://reviews.vapour.ws/r/5186/ [09:32] just picking on you because you did the other one [09:37] thumper: that is how you pay us [09:40] penalty for helping [09:42] thumper: done [09:42] as usual, If menn0 reviews this he will find 120138019283019831023 more things than I did [09:43] perrito666: ta [09:49] axw: ping [09:51] perrito666: and wallyworld will find even more than me :) [09:54] So wallyworld should be the only reviewer, naturally [09:58] menn0: hiya [09:58] menn0: do you know what the status of the controller/model config split is, by any chance? [09:58] perrito666, wallyworld, axw: ^ [09:59] thumper: ^ [09:59] rogpeppe: axw is working on it [09:59] it's wip, partly done [09:59] I'm not sure [09:59] what is the specific question? [10:00] wallyworld: we're wondering whether the config is going to be split in the providers themselves [10:00] wallyworld: and if so, what's going to happen to Provider.Schema method? [10:01] rogpeppe: not sure what you mean - providers shouldn't need to know about controller config in general [10:01] don't quite follow sorry. by the time the providers are instantiated, they will see environ/config/Config [10:01] which will contain only model config [10:01] axw: ah, ok, cool. [10:01] rogpeppe: I guess we'll still have it, but it won't include controller config fields. we can have a separate controller config schema, but I don't think that's really useful for you? [10:01] there's a state api call to get controller congih [10:02] axw: so there will still only be one set of configuration attrs for a given model [10:02] axw: but... what about credentials? [10:02] rogpeppe: there's a separate schema for creds [10:02] rogpeppe: so, separate schemas for model, controller and credentials [10:03] tho controller is general, so doesn't really need a schema at all [10:03] axw: so environschema.AccountGroup becomes redundant? [10:03] rogpeppe: yep [10:03] axw: so Provider.Schema returns two values? [10:03] axw: can the attribute names overlap? [10:04] rogpeppe: maybe. EnvironProvider already has a CredentialSchemas method though [10:04] rogpeppe: credential and model attrs? [10:04] axw: Schemas plural? [10:04] rogpeppe: yeah, each provider can define multiple auth types [10:04] rogpeppe: e.g. access-key and userpass for openstack [10:05] axw: so where is the auth type specified? [10:05] rogpeppe: hypothetically the model and credentials could have overlapping attr names. they should be considered completely separate [10:05] wallyworld: I am trying to convert the GUI API client for the new ModelManager facade (without COnfigSkeleton). I am encountering credential errors on ec2: e.g. http://pastebin.ubuntu.com/18158931/ [10:05] rogpeppe: a credential is a key-value map, one of the keys has a special name "auth-type" [10:06] the value of which is interpreted in a provider-specific manner [10:06] (and must have a schema defined by the provider) [10:07] axw: how does the CredentialAttr.FilePath stuff work when credentials are passed across the network? [10:08] frankban: I think that's a small bug, can you try passing "user-admin@local" as the owner? [10:08] frankban: you need to specify a credential to use if you are not the controller admin [10:08] rogpeppe: they have to be converted at the client first [10:09] rogpeppe: that bit is a bit iffy at the moment, needs some rework when we come to updating environschema [10:09] axw: that works [10:09] axw: ISTM that most of this stuff isn't really compatible with the way environschema does things [10:10] frankban: thanks, I'll patch that shortly [10:10] axw: or could do things, even [10:10] axw: cool, so we don;t need to pass any config anymore, and no ssh keys correct> [10:10] ? [10:10] frankban: authorized-keys are still required at the moment, I have a patch up but can't land until the beta is out [10:10] frankban: i.e. the patch will make authorized-keys optional [10:11] axw: in particular, we've now got many overlapping attribute names with potentially different types depending on other attributes [10:11] axw: well, this worked without ssh keys... http://pastebin.ubuntu.com/18159325/ [10:12] frankban: ah, special case for admin user :) that will be going away [10:12] rogpeppe: example? [10:12] axw: so, is it ok for the GUI, for the time being to not set CloudCredentials and to still pass fake ssh keys? [10:13] axw: and to use @local in the onwner-tag? [10:13] axw: for example, AFAICS you could have two auth types, both of which define a credential field with the same name but a different type [10:13] frankban: that's fine [10:13] axw: cool thanks [10:13] axw: as returned from CredentialSchemas [10:13] rogpeppe: credentials are always strings [10:13] axw: just to confirm, "user-admin@local" will continue working after your fix, correct? [10:14] frankban: yes [10:14] axw: ok, i see [10:14] frankban: we may want to stop accepting invalid SSH keys at some point, but that should be easy to drop in the GUI [10:14] axw: but different descriptions and other attributes, which amounts to a similar thing [10:15] axw: different sets of allowed values too [10:16] rogpeppe: I don't really understand the problem. You could hypothetically, but they're still relative to the auth-type. In the GUI you would select an auth-type, and then the attr name should make sense in that context [10:16] axw: ok so params would be like this: params': {'config': {'authorized-keys': 'bad-wolf'}, [10:16] 'name': 'test-model', [10:16] 'owner-tag': 'user-admin@local'}, [10:16] correct? [10:16] frankban: yup [10:17] axw: the problem is that in our service, we have a unified view of "the model config" which includes all attributes. that's just become extremely special-case-y [10:17] PR seeks review: http://reviews.vapour.ws/r/5153/ [10:18] rogpeppe: credentials are not part of model config *at all* now, though. you manage credentials separately in juju, using the Cloud facade. then when you create a model, you specify a credential by name [10:18] axw: i guess i was hoping to see a mode general mechanism than just special-casing auth-type [10:18] axw: to be more future proof, should always include the first credential name returned by Cloud.Credentials? [10:18] axw: they're part of the configuration you need when creating a model [10:18] axw: are you saying you can't explicitly pass credentials attributes now? [10:19] rogpeppe: that's something I'm working on fixing. you currently have to duplicate the creds into model config due to things being half done [10:20] axw: if possible, i think we'd prefer to avoid the *necessity* to have a named set of credentials in order to use some credentials, as it's just another thing to manage. [10:20] frankban: https://github.com/juju/juju/pull/5704 <- this PR adds a Cloud.CloudDefaults method, which includes the default credential name [10:20] frankban: I guess for now yo ucould just use the first one in Cloud.Credentials [10:21] * rogpeppe drops of the network for a few seconds [10:21] off [10:22] back [10:22] axw: if I need to change it later, then maybe I'll just go with empty credential and therefore only allow admin to create models from the GUI for now. does it sound reasonable to you? and also could you please send me an email when everything is ready with a summary of the new strategy I should implement in the GUI? [10:23] rogpeppe: so credentials are managed separately from models now. you can have multiple models using the same credentials. to avoid duplication, and to support updating/revoking creds, we use a name for reference [10:23] rogpeppe: similar to how you specify credentials at bootstrap time [10:23] frankban: yes that sounds fine to me. will do [10:23] axw: thanks a lot [10:24] np [10:44] ericsnow: this is a quick and dirty pr just to get *something* done for the release deadline http://reviews.vapour.ws/r/5187/ [10:45] wallyworld: k [10:59] wallyworld: can you please review https://github.com/juju/juju/pull/5741 === jamespag` is now known as jamespge === jamespge is now known as jamespage [11:03] axw: looking [11:04] katco, wallyworld, I am seeing a *lot* of audit spam that looks like: machine-0: 2016-06-30 10:56:18 CRITICAL juju.cmd.jujud machine.go:1055 ModelUUID not valid [11:05] fwereade: i just found that myself :-( i've fixed it [11:05] wallyworld, <3 [11:06] wallyworld, katco: do we know how this happened? presumably we did run a controller with these changes before we landed them? [11:07] fwereade: i just found it running a controller to test the addition of an local audit log file sink [11:12] Bug #1385276 changed: juju leaves security groups behind in aws

[11:12] Bug #1597704 opened: juju status --format=tabular goes double space if one message is long <2.0>

[11:16] jam: http://reviews.vapour.ws/r/5186/ [11:22] cherylj: can I JFDI https://github.com/juju/juju/pull/5741, or would you like me to wait? [11:25] thumper: http://reviews.vapour.ws/r/5188/ [11:48] Bug #1597720 opened: It is not possible to refer to multiple models with the same name from the CLI [12:07] axw: https://github.com/juju/juju/pull/5743 [12:37] fwereade: diagram? [12:37] wallyworld, do you have a CL up for that logging yuck? it's... really quite inconvenient [12:38] redir, that sounds good, I thought we had a couple of sessions coming? [12:38] fwereade: OK. I am also good doing it by habgout with a screen large enough and a trackball [12:49] redir: https://github.com/natefinch/claymud/blob/master/util/query.go [12:55] * perrito666 wonders if his credit card would mind if he nuked it with a laptop [13:01] redir, face to face is better, though: can we do it after the escape analysis session? [13:02] sure thing fwereade [13:17] babbageclunk, please update when you have a moment: https://bugs.launchpad.net/juju-core/+bug/1567708 [13:17] Bug #1567708: unit tests fail with mongodb 3.2 [13:21] https://bugs.launchpad.net/juju-core/+bug/1579010 [13:21] Bug #1579010: state: removing model can generate huge transactions

[13:21] ^^ thumper [13:31] Bug #1555368 changed: Panic due to sending on closed channel

[13:31] perrito666: http://reviews.vapour.ws/r/5190/ === dimitern` is now known as dimitern [13:35] alexisb: yup, sorry - was afk [13:36] junaidali: You asked in a PM about `unit-get public-address` returning an IP instead of a FQDN for your deployment and whether there was any way to work around that; I'm not entirely certain, but I think that Juju just passes through what the provider gives it. Someone here might be able to give more insight [13:37] alexisb: and or cherylj: A partner at IBM has pointed out a possible bug with Resources here: https://bugs.launchpad.net/juju-core/+bug/1597354 This is a blocker for IBM and they are very concerned about it. Can someone triage it and respond in the bug? [13:37] Bug #1597354: Juju 2.0 Resource Error - cannot add resource failed to write data: read tcp : i/o timeout [13:38] mbruzek: we need /var/log/syslog and /var/log/juju/machine-0.log from the controller machine [13:38] mbruzek: I'll put that in the bug [13:40] cherylj: they are chatting with me on IRC. I told them what you need as well. [13:40] Thank you cherylj [13:41] sure thing. I might need to pull in katco or someone else who's done work on resources to help [14:03] katco, ericsnow - could either of you take a look at this bug comment and see if there's any workaround? https://bugs.launchpad.net/juju-core/+bug/1597354/comments/3 [14:03] Bug #1597354: Juju 2.0 Resource Error - cannot add resource failed to write data: read tcp : i/o timeout [14:07] cherylj: yep [14:09] thanks ericsnow! [14:14] cherylj: Do you know who'd be the point of contact for juju actions? [14:17] aisrael: bogdanteleaga might be able to help you out [14:17] cherylj: Thanks! [14:17] np! [14:30] fwereade: sorry about that. landed that change at close to midnight to try and unblock axw the following day. didn't have time to do any manual testing [14:36] bogdanteleaga: I think the issue w/the client I'm helping is resolved, so you're off the hook. I have some usability requests but I'll file bugs for those. [15:11] ericsnow: hey saw you pinged me earlier [15:11] katco: just wanted to catch up [15:11] katco: and see how things have gone with closing ;) [15:11] ericsnow: boring paperwork :) [15:12] katco: yep [15:23] jam, alexisb - https://bugs.launchpad.net/juju-core/+bug/1594720 [15:23] Bug #1594720: lxd containers not using configured proxy for downloading images

[16:01] urulama: all api breaks landed [16:01] as far as we are aware [16:02] frankban: ^^ [16:02] if I have missed any, I'll be grumpy but let me know [16:02] just emailed juju-dev list with details [16:02] thumper: cool thanks [16:02] some cribbed from frankban's email [16:02] :) [16:02] no worries [16:04] thumper: all api breaks *that we are aware of atm* landed ? :) [16:04] yeah [16:04] urulama: ah "DestroyModel has moved from the "Client" facade to the "ModelManager" facade" I was not aware of that, we need to handle that as well [16:04] I thought I got them the first time around [16:04] frankban: yeah, seen the list :-/ [16:05] and the CharmInfo call [16:05] frankban: but ATM, we don't destroy models in the gui as well [16:05] not sure if you were getting it from "Client" or "Charms" [16:05] urulama: I remember we do [16:05] the go api only called the "Client" [16:05] frankban: it was disabled [16:05] thumper: yes I am working on CharmInfo [16:06] urulama: cool, but let's just keep that in mind, the change should be trivial [16:06] however there were two, slightly different, implementations in the server [16:06] frankban: yeah, keeping the list visible on screen :) [16:08] thumper: I see we had a side effect of this API change that is not only related to consistency at least: we discovered quite a lot of internal structures sent over the wire [16:22] Bug #1597830 opened: agent restarted as part of machine jobs update

[16:37] Hi, anyone seen a situation where the bootstrap node constantly has *very* high load? I'm not sure if it's the cause or a symptom, but mongodb is hammering the logs === frankban is now known as frankban|afk [16:38] high as in - 400-500! restarting juju-db brings it down for a little while, but it goes back up pretty quickly [16:39] juju version is 1.25.5 [16:43] plars: how many nodes? [16:43] perrito666: it seems to be just the bootstrap node where this is happening [16:43] plars: yes, but how many nodes are there in your environment? [16:44] perrito666: not many, I think 5 or so? [16:44] perrito666: I didn't see this problem until recently [16:45] bogdanteleaga: are you around? [16:45] perrito666: we had a power outage, and things came back. But I had a lot of trouble reaching anything or getting juju status. After a few timed out attempts, I was finally able to juju ssh 0 and see that the load was so high [16:45] plars: ok, that strange behavior with so little nodes I have not seen before, what size is your db? (you can tell by the size of /var/lib/juju/db folder in the bootstrap node [16:46] cheryl, yup [16:46] hey hey bogdanteleaga [16:46] perrito666: all the files taken together? there are a lot of numerical extension files there, like rotated logs? [16:46] -what's up [16:46] all together [16:46] bogdanteleaga: I've been working more on bug 1577949 [16:46] Bug #1577949: windows services cannot upgrade to 1.25.6

[16:46] perrito666: 7.8G according to du [16:47] bogdanteleaga: and I'm seeing that when we restart the juju machine agent on the windows machine, and it's running 1.25.6, it thinks that the unit agent isn't running and that it has to start it again [16:47] bogdanteleaga: so the service.ListServices call isn't listing the unit agent [16:48] bogdanteleaga: I don't see that anything has changed in juju/juju/service/windows [16:48] bogdanteleaga: nor has the github.com/gabriel-samfira/sys dependency [16:48] BUT [16:49] we have had some problems some time ago with that particular function having weird behavior when compiled with different go versions [16:49] bogdanteleaga: we did change from using go 1.2 to using go 1.6 between 1.25.5 and 1.25.6 [16:49] thumper: fwereade: https://github.com/juju/juju/pull/5746 (for some reason rb is not picking it up) [16:50] iirc, it didn't work with 1.4, but it did work with 1.6 in our tests, so we ended up not changing it [16:51] you can just do a small main function that calls it and see if it works [16:51] in case you have access to the failing machine [16:51] plars: for now, I can advice you to get juju/juju-db running and, if the load can be sustained leave it be and it might clean up the db, otherwise perhaps tomorrow I could give you a better answer since we are working on a similar issue tryig to pinpoint what is going on [16:51] bogdanteleaga: I have access, but I would need some handholding as I don't know windows much at all [16:51] perrito666: it's running ok, but it's been in this state for almost a week [16:52] plars: a full week running at a 400 load? [16:52] plars: could you ping me tomorrow same time? [16:52] perrito666: I modified rsyslog to have it discard a lot of those messages and was able to get the load down to 15-100 most of the time, but still not great [16:52] I might have a solution for you [16:52] perrito666: sure [16:52] tx a lot, if you could put up some logs and perhaps report a bug I would be most grateful [16:53] perrito666: sure, will do [16:53] again, tx a lot [16:53] bogdanteleaga: I need to run soon (I'm sprinting with the team in London this week and I need to get out of the office) [16:53] cherylj, so is it the deployer that checks for the unit agents? [16:54] yeah, I g2g soon too [16:54] so my first advice would be to just write a small main function and create an executable that just calls ListServices and check how the output looks [16:55] (run the executable on the failing machine, and also make sure to use the same go version for compilation) [16:55] bogdanteleaga: yeah - worker/deployer [16:56] bogdanteleaga: if it doesn't work, do you guys already know what changes were needed for 1.4? [16:57] (in case they're still broken) [17:11] Odd_Bloke: /quot eod [17:11] Odd_Bloke: sorry, ignore me [18:02] Bug #1597860 opened: "juju machine remove" cmd throughs error in 2.0 [18:56] Bug #1597879 opened: jujud hangs on trusty arm64 [19:07] cherylj, we needed a new package that used another windows api to fetch the info, you should probably ask gabriel about i [19:08] it* === urulama is now known as urulama|___ [22:56] Bug #1597941 opened: juju2.0beta10: websockets API usability Application Deploy failure to inform of required addCharm pre-requisite

[23:09] Bug #1597941 changed: juju2.0beta10: websockets API usability Application Deploy failure to inform of required addCharm pre-requisite

[23:12] Bug #1597860 changed: "juju machine remove" cmd thows error in 2.0 [23:12] Bug #1597941 opened: juju2.0beta10: websockets API usability Application Deploy failure to inform of required addCharm pre-requisite