[00:04] wallyworld: this too https://github.com/juju/juju/pull/6186 [00:04] now lunch === thumper is now known as thumper-lunch [00:04] thumper-lunch: you can tell rb tp pick it up using rbt post [00:07] menn0: a whole lot of tests in api/controller/controller_test.go are no longer there with the various new migration tests being introduced. did you move them anywhere? [00:08] wallyworld: yep.. see legacy_test.go in the same package [00:08] legacy_test.go [00:08] ah righto thanks [00:08] weird ... the underscore doesn't show up for me :) [00:09] wallyworld: the migration tests couldn't work any more when using JujuConnSuite so I set things up for moving these tests to using a mock API [00:09] sgtm [00:14] Bug #1474607 changed: worker/uniter/relation: HookQueueSuite.TestAliveHookQueue failure

[00:19] you know where your day is going when you find yourself reading the code for mongo tools [00:33] wallyworld: could u please answer user question on https://bugs.launchpad.net/juju/+bug/1620886? :D [00:33] Bug #1620886: juju status returns ERROR connection shutdown [00:33] don't have time right now [00:33] maybe tomorrow [00:34] need to get beta ready [00:34] what's this mean? http://juju-ci.vapour.ws:8080/job/github-merge-juju/9120/console at the end [00:34] the bot failed to merge but it tried to? [00:36] sinzui: ^^ [00:36] if your around [00:37] axw: could u plz comment on the bug wallyworld cannot? ^^ [00:38] anastasiamac_: it will take analysis and investigation [00:38] can it wait till tomorrow? [00:38] we need to concentrate on the beta today [00:38] wallyworld: did u read the bug? it's a simple question... [00:39] "I sthere a way i can bring the model back" [00:39] that is a can of worms [00:39] he started on beta12 [00:39] we made incompatible changes in 13/14 [00:40] he will likely need to hand edit yaml files [00:40] i've read the bug. to me the answer is simple "no" he'll have to re-bootstrap. but u and andrew r more fmailiar with the area [00:40] i'll assign to u \o/ [00:40] no thanks :-) [00:40] don't have time right now [00:40] who does? [00:40] no one today [00:40] beta take priority [00:41] balloons: You need to follow up with http://juju-ci.vapour.ws:8080/job/github-merge-juju/9120/console the merge failed, but the tests passed and the job itself reports success. [00:41] let's see who's free tomorrow [00:41] redir: have you tried to force your branch to merge? [00:41] sinzui: don't know how [00:42] redir: okay. I am taking a guess I will pretend to be jenkins on that host and run the failing command [00:42] * redir checks wiki [00:43] redir: "Pull Request successfully merged" [00:43] thanks sinzui-bot [00:44] redir: ballons There must have been a network hiccup. running the command again took a second to complete [00:45] sinzui: balloons I've noticed some slow GH action for the last 12h or so === thumper-lunch is now known as thumper [01:26] wallyworld: I moved the api bits here: https://github.com/reedobrien/juju/tree/feature/move-model-defaults [01:26] righto [01:27] i'll look when i can [01:27] wallyworld: but haven't had time to make anything useful of moving the command bits yet [01:27] and I have to go RSN [01:27] sure, np, i can pick it up [01:27] ty [01:29] later juju-dev [01:47] wallyworld: please see replies on http://reviews.vapour.ws/r/5618/ [01:47] ok [01:48] axw: all good, i was thinking the calling code would test for ErrExpired, but wasn't sure how it would be used [01:49] wallyworld: we check for NotFound atm, but not ErrExpired. it will make sense to do so when we have browser logins. because the client would pop up a browser and then go to the /auth/wait page straight away [01:49] wallyworld: it would find the interaction, but it *would* be expired if the user doesn't hit login inside 2 minutes [01:49] yep [01:54] Whazzat? 2016-09-08 01:48:00 ERROR cmd supercommand.go:458 failed to initialize state: validating initialization args: mismatching uuid (2eb2f71a-00d1-43db-8575-2d4eacb2bf3f) and controller-uuid (4d5116b1-945a-46af-8a27-69e7269d2753) not valid [01:55] I saw some stuff go by on email/irc about UUIds... what's going on here? I'm bootstrapping with a client from master, but trying to deploy beta17 .... is that broken, or did I do something wrong? [01:55] menn0: is there anything in rsyslog on 1.25 that would have caused this? u/thumper were going to check but i have not seen an update... https://bugs.launchpad.net/juju-core/1.25/+bug/1616832 [01:55] Bug #1616832: manual environment juju-db timeout

[01:58] wallyworld: you ok with that branch landing then? [01:58] yep [01:59] natefinch: yep broken [01:59] thanks [01:59] natefinch: that's we we strictly enforce client and agent version match [02:00] wallyworld: I thought we explicitly didn't do that, so you can use one client with multiple controllers that might be at different versions.... or is this just unique to bootstrap itself? [02:00] just bootstrap [02:00] the version check is just for bootstrap [02:01] Is this unique just to this one point in time, where pre-beta18 servers won't work with post beta18 and later clients, or is this a new policy in general? [02:03] it happened between beta17 and 18 [02:03] ok [02:03] it's not a policy [02:03] ok [02:03] but we don't guarantee client N can bootstrap agent N+1 [02:03] it works 99% of the time [02:03] it betas, we are less strict about checking for such compatibility [02:04] yep [02:12] anastasiamac_: NFI what's going on with that rsyslog issue [02:12] anastasiamac_: digging some more [02:13] menn0: thnx :) [02:17] anastasiamac_: hmmm... I think I've found something [02:17] * menn0 digs more [02:18] very simple review: http://reviews.vapour.ws/r/5623/ only adds logging [02:19] menn0: \o/ [02:19] * anastasiamac_ hopes menn0 found the reason for cpu/mem spikes on 1.25 and will solve all world problems!! [02:20] anastasiamac_: I doubt this has anything to do with cpu/mem spikes :) [02:20] menn0: i know but m mentioning to see who else is watching :D [02:21] anastasiamac_: here's a helpful article: http://b.kl3in.com/2011/10/ubuntu-server-slowly-stops-responding/ [02:22] anastasiamac_: it seems that if syslogd/rsyslogd hangs and stops reading from /dev/log then the buffer fills up as processes keep writing out logs, and when that happens processes start hanging when they try to log [02:23] anastasiamac_: i'm trying that out now [02:23] menn0: this is so sad :( [02:23] yeah, that's pretty awful [02:24] menn0: definitely worth adding to the bug and marking as Invalid for juju... ther is nothing we can really do on our side :( [02:24] anastasiamac_: well there is... we can have something that checks if rsyslogd is wedged and restart it when it is [02:24] anastasiamac_: there's a suggested script in the blog article [02:25] anastasiamac_: it's also possible that juju generated an invalid rsyslog config (or cert) which was causing it not to be able to work [02:27] menn0: both of which are gr8 suggestions and should b mentioned in the bug.. [02:27] anastasiamac_: just doing a bit more research and then I'll update the bug [02:28] menn0: awesome \o/ if u could add how much effort would be involved in resolving these, it'll help power-that-be to plan. [02:28] ok [02:29] menn0: i'll also add an eda tag to it in hopes that QA may come up with functional test to reproduce it \o/ [02:39] wallyworld: external user login support: http://reviews.vapour.ws/r/5624/ [02:39] axw: awesome, am about to ptopose a PR and then i'll look at the remaining PRs of yours [02:39] thanks [02:44] Bug #1557769 changed: private-address returns name, not ip, under 1.25.4

[02:44] Bug #1577556 changed: unit failing to get unit-get private-address in the install hook

[02:58] wallyworld: I don't even seem to be able to juju status from current master client to beta17 server. is that expected? [02:59] yes [02:59] ok [02:59] controller and controller model uuids are now different [02:59] they used to be the same [02:59] hence not uuids by definition [02:59] well, they are uuids... just not of the same thing :) [03:00] natefinch: note, with tools etc, the bootstrap messages always tell you what has been done so there are no surprises [03:00] wallyworld: yes, but it's buried in there if you're using --debug [03:01] that is true, debug is a firehose [03:07] axw: got a few minutes? [03:08] thumper: yeah? [03:08] looking at a storage bug and I want to get ideas on how to test and if I'm doing it right [03:08] 5 min hangout? [03:08] thumper: sure [03:09] axw: https://hangouts.google.com/hangouts/_/canonical.com/storge?authuser=0 [03:18] axw: or thumper: or anyone, here's a PR I'd like to land today if anyone has time to review at some stage http://reviews.vapour.ws/r/5625/ [03:25] wallyworld: I can look at it [03:25] yay, ty [03:34] anastasiamac_ (or anyone else) - what should I mark a bug assigned to me that appears to have been fixed, though perhaps not intentionally, and not by me? Fix committed? [03:34] natefinch: of course \o/ [03:34] anastasiamac_: ok :) [03:35] natefinch: it'll help if u knew PR that fixed it but otherwise, unless u r happy to field questions around the bug, keep it assigned to u [03:35] anastasiamac_: yeah, I have no idea. I could repro in beta 17 reliably, can't in master. ¯\_(ツ)_/¯ [03:36] natefinch: best kind of bug :D [03:37] natefinch: well done \o/ [03:41] wallyworld: http://reviews.vapour.ws/r/5620/ has already been looked over by mhilton and ashipika, so I'll stick with just your review if that's ok? [03:42] axw: sure, np, just making sure [04:11] wallyworld: lol, took me a while to figure out what hoak meant [04:14] lol [04:14] i can tweak that [04:14] er haok [04:15] wallyworld: I didn't have a better suggestion, so I was going to ignore it, but it was kinda funny. [04:16] wallyworld: one thing - I kinda wonder if we'll make people worry if they see 3/4 machines for HA, when really, that fourth machine is hasvote: false wantsvote: false [04:17] wallyworld: I don't know if it's really fixable, it's a lot of info to cram into a single column [04:17] natefinch: wantsvote false is not supposed to show [04:18] wallyworld: oh, I misread the test I was looking at.. You're right. Nevermind [04:19] natefinch: the 2/3 format is the same as is used for the scale column in status [04:19] i just made it up as a compact way to display [04:19] this betas will get feedback on that [04:19] and we'll tweak as needed [04:22] cool [04:22] axw: testing our fix now on canonistack [04:22] cool [04:29] wallyworld: you have a review [04:30] natefinch: awesome, ty [04:30] natefinch: i had better fix any issues before you disappear [04:30] menn0: and since today is an rsyslog, just found this pearl :) https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1318378 [04:30] Bug #1318378: rsyslog starts dropping connections in large environments

[04:35] anastasiamac_: that one is probably fixable by tweaking rsyslogd's config or raising process limits [04:36] anastasiamac_: as for the first one, I've demonstrated that if rsyslogd is stuck, anything logging to it will get stuck. [04:36] anastasiamac_: this will be mongodb and in 1.25, all the agents in the model [04:36] \o/ [04:36] these bugs are crying for ur comments ;) [04:36] writing up now [04:40] natefinch: if u r still here, thumper: i *think* this has been fixed coz we do not run mond as sudo.. right? https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1208430 [04:40] Bug #1208430: mongodb runs as root user

[04:43] anastasiamac_: no idea... but uh... if they can 0 day the database.... whet else are we protecting? The log files? All the important data is *in* mongo :/ [04:46] ok, I gotta run, I'm passing out here. wallyworld - changes LGTM [04:46] natefinch: awesome, tnas again [04:46] thanks [04:50] Bug #1317909 changed: juju add-unit performance degrades in large environments

[04:50] Bug #1318148 changed: Unit.PublicAddress shouldn't treat no machine as an error

[04:50] Bug #1318378 changed: rsyslog starts dropping connections in large environments

[05:03] anastasiamac_: lots of detail added to bug 1616832 [05:03] Bug #1616832: manual environment juju-db timeout

[05:04] anastasiamac_: it should probably be reopened against 2.0 b/c it could happen there too since mongod still logs to syslog in 2.0 [05:05] anastasiamac_: it's probably less likely because we demand a lot less of rsyslogd in 2.0 though [05:18] axw: if you are free, here's a cut and paste review http://reviews.vapour.ws/r/5627/ [05:18] ok [05:23] Bug #1318378 opened: rsyslog starts dropping connections in large environments

[05:28] thanks axw [05:28] nps [05:29] axw: http://reviews.vapour.ws/r/5628/ [05:29] looking [05:33] I'm off now [05:34] axw: if you're happy, can you do the merge dance? [05:34] laters [05:34] thumper: will do. later [05:57] wallyworld: I missed something in my branch when I QAd before. need to make a small change, will ping you for a review soon. change-user-password isn't storing a macaroon, so you need to log in again after changing your password [05:58] sure np [05:58] axw: i assume you'l lupdate the release notes at EOD as well? [05:59] wallyworld: yup [06:24] wallyworld: http://reviews.vapour.ws/r/5620/diff/3-4/ [06:24] looking [06:26] wallyworld: I've just re-run through all the QA steps again, and also confirmed that "change-user-password && status" does not prompt you after the password is changed [06:26] great [06:28] axw: lgtm, good that you found it [06:29] wallyworld: thanks === frankban|afk is now known as frankban [07:17] Bug #1554436 changed: Juju adds any RFC1918 address it finds on any state servers to the apiaddresses list in agent.conf

[08:39] wallyworld: there's a problem with "juju logout" which I was planning to defer, but maybe shouldn't. if you "juju logout", your cookie remains in the jar. then "juju login " will use it again without you being prompted [08:40] wallyworld: fixing it is going to take a bit, because there's an issue with persistent-cookiejar [08:40] axw: we can document as a known issue for this beta imo [08:41] wallyworld: ok [08:58] voidspace: ping [09:03] axw: not sur eif you have a minute for a trivial format fix http://reviews.vapour.ws/r/5630/ [09:15] babbageclunk: or maybe you could take a peek at the above for me? it's trivial, and needed for the beta [09:19] wallyworld: sure, looking now [09:19] you rock ty [09:23] wallyworld: LGTM! [09:24] babbageclunk: yay, ty. btw, it seems you finally got the right person from maas to talk to? [09:24] jeez, what a saga [09:25] wallyworld: Yup - ripped out all my bodges and put in the right thing really easily, working on the provider changes now. [09:25] babbageclunk: sigh, how frustrating. seems like only a couple of people knew this secret knowledge [09:26] wallyworld: Well, it was there in the docs, but I never searched with a keyword that matched it. [09:27] well, i didn't know the magic word either [09:28] wallyworld: It's good though - the hacked version would've been nasty. [09:28] indeed, but it's all we thought we had :-) [09:33] axw: could you please take a look at http://reviews.vapour.ws/r/5631/ ? [09:34] frankban: looking [09:34] axw: ty [09:37] frankban: LGTM, thanks [09:37] axw: cool thanks [10:15] babbageclunk: pong, sorry [11:23] dimitern: around? [11:25] babbageclunk: yeah? [11:26] dimitern: I'm trying to test something on AWS, but I lost my credentials when my machine crashed - actually, might have just found an email, hangon. [11:27] dimitern: Remind me of the url for the AWS console? [11:27] babbageclunk: I remember sending the shared AWS account creds by mail a while ago, but might have missed you [11:28] babbageclunk: pm-ed you the link [11:28] dimitern: awesome, thanks [11:33] voidspace, dimitern, frobware: Could someone take a look at this PR? https://github.com/juju/gomaasapi/pull/57 [11:33] babbageclunk: looking [11:34] voidspace: thx! [11:38] babbageclunk: so if args.OwnerData is nil then the iteration in ownerDataMatches just drops through to true [11:38] babbageclunk: it seems a bit weird to call the filtering function even when there's no filtering [11:38] babbageclunk: it's not wrong, just feels like the code doesn't match the user intent [11:38] babbageclunk: only a slight niggle [11:38] boiler repair man here, have to let him in [11:40] voidspace: I think it does match the intent, though - a nil filter should match any owner data, right? [11:45] babbageclunk: to me a nil filter means "don't bother checking" [11:46] voidspace: I prefer not to have a different control flow (like an early exit) if the normal path through the block does the right thing. [11:48] voidspace: But I don't feel super-strongly about it in this case. I can add `args.OwnerData != nil &&` on the start. [11:49] babbageclunk: I don't feel strongly, it's a matter of taste and it's your code [11:49] babbageclunk: so fair enough :-) [11:50] voidspace: ok cool - I tried it and I don't like it. :) [11:50] hah [11:51] babbageclunk: sorry otp - looking though [11:51] dimitern: no rush - I'm not blocked on it. [11:51] dimitern: thanks though! [11:53] dimitern: Actually, I just realised I can't make myself a new access key for AWS - can you make me a new one and invalidate the old one? Or do I need jam or someone else? [11:55] babbageclunk: you don't explicitly test the nil case in new tests, is that because it's covered by the existing ones? [11:56] voidspace: Yeah - if it didn't work then they wouldn't get any machines back. [11:56] babbageclunk: cool, LGTM [11:56] * babbageclunk successkid [11:56] babbageclunk: sure, just give me a few minutes [12:12] babbageclunk: sent you new kets [12:12] keys [12:20] gah [12:20] juju deploy stuck in allocating/pending and nothing in the debug-log and can't ssh [12:20] need moar logging [12:20] voidspace: is this with tip? [12:21] voidspace: shooting you an email please see if thisis what you're seeing [12:21] rick_h_: it is, but it's also with a custom charm on lxd to test an interrupted sleep action [12:21] rick_h_: ah, does look a bit like it yes [12:22] rick_h_: I have an ipv6 address! but I can ssh into that [12:22] fwereade: ping [12:23] and no juju logs on the system [12:24] fwereade: important email your way please [12:58] voidspace: so looking more closely at things, that email is only when HA is turned on close to when a deploy is in progress. [12:59] voidspace: just testing a bootstrap/deploy without HA isn't an issue and I'm assuming you're not in HA for your testing [12:59] babbageclunk: reviewed [12:59] voidspace: so thinking there's different things goin on between your setup and the email thread there [12:59] dimitern: thanks! [13:12] frobware: Sorry, just realised I merged that without waiting for an all-clear from you - were you alright with that change other than the comment you made? [13:15] voidspace: ok, betting that marcoceppi's upcoming email to the juju list is your issue. [13:15] voidspace: try http://paste.ubuntu.com/23147632/ and see if that unblocks you there [13:16] voidspace: heh and as bac points out the commands there are moved in trunk so need to use set-model-config as model-config is an alias for get-model-config [13:17] dimitern: How can I get values into Config.ResourceTags() in a test? [13:18] babbageclunk: what's Config.ResourceTags()? [13:19] dimitern: It's a set of user-defined tags that get applied to instances when they're started by the provider. [13:20] babbageclunk: ah, so the juju one, not the maas one [13:20] babbageclunk: let me have a look [13:20] dimitern: Yeah - I'm hooking it up to maas owner data now. [13:22] dimitern: Ah, think I've found the answer in config.go: "Config holds an immutable environment configuration." [13:22] dimitern: I can't. Alright, I'll create it with them instead. [13:22] babbageclunk: yeah :) also config with "resource-tags" key is used to hold them [13:31] rick_h_: hey [13:31] rick_h_: I've noticed no HO link on the sync call [13:32] dimitern: oh my bad [13:32] rick_h_: and I'm sitting in the standup one [13:32] dimitern: k, I'll meet you there [13:34] dimitern: Thanks! [13:57] anyone seen alexisb this morning? [13:57] babbageclunk, I just sat down [13:57] omw [14:00] natefinch: ping for standup [14:01] jam, fwereade: ^^ [14:06] morning all [14:09] perrito666: o/ [14:22] ahasenack: btw, I couldn't repro https://bugs.launchpad.net/juju/+bug/1614635 with current master. I could repro easily with beta17, so hopefully it just got fixed by something else. [14:22] Bug #1614635: Deploy sometimes fails behind a proxy [14:45] natefinch: got it [14:45] thx [15:00] rick_h_: hah, just seen your IRC message telling me to try model-config instead of set-model-config :-) [15:00] voidspace: :) [15:00] rick_h_: so, thanks... [15:38] morning juju-dev [15:41] morning redir [15:59] is there a qa list? [15:59] redir: yes, juju-qa@lists.canonical.com [16:00] you can always send things to it and can be manually moderated through, or you can join the fun [16:00] thanks mgz [16:01] I certainly don't get enough email as it is [16:22] dimitern: frobware ping, CI is seeing this failure: https://bugs.launchpad.net/juju/+bug/1621538 can we confirm that this is part of ongoing issues/work [16:22]