[00:10] thumper: this is a quick brain dump re: the version bug I was talking about yesterday: http://pastebin.ubuntu.com/23235882/ note we have 2.0-rc2 available in the steams but starts upgrading to 2.0-rc2.1. [00:11] thumper: rats, I realise i'm running late for a thing in town, I'll chase up when I get back :-) sorry to ping and run [00:11] why is rc2 in streams? [00:11] OMG status on beta7 looks terrible compared to now [00:12] thumper: I think it's in testing streams, right? http://juju-dist.s3.amazonaws.com/parallel-testing/agents/... [00:32] perrito666: figure anything out? [00:33] yes, that today is ... I guess I would translate to clerks day? [00:33] and therefore only open businesses are those tended by their owners [00:33] annyway, deploying to a container as we speak [00:34] also juju deploy cs:mysql --to lxd/0 didnt have the effect I was hoping [01:12] thumper: You have a moment to consider the version bug with the details I linked? [01:12] (I'm back from lunch now :-) )] [01:12] I'm expecting a visitor any minute [01:12] but she isn't staying, just picking soemthing up [01:13] I do need to take broken coffee things down to hardly normal [01:13] and get another coffee [01:13] thumper: ack [01:13] how about a little later in the afternoon? [01:13] thumper: hopefully they can repair it quickly. They do have some nice Rockets at southern hospitality that aren't stupidly expensive (for some value of stupid) [01:13] thumper: sounds good [01:14] rockets are too manual for me [01:14] I prefer the push button [01:14] but I do like grinding my own beans [01:14] so not too push button [01:15] heh :-) [01:16] alexisb: changes to the state package since beta 7: 232 files changed, 25849 insertions(+), 13540 deletions(-) [01:17] k ppl, EOD [02:17] thumper, yep [02:44] axw: do you have a couple of moments to spare while I wait for thumper? :-) I'm waiting to further this bug here https://bugs.launchpad.net/juju/+bug/1626784 (which I spoke with you about a couple days back) [02:44] Bug #1626784: upgrade-juju --version increments supplied patch version [02:45] veebers: looking [02:46] axw: cheers, this is some follow up thoughts: http://pastebin.ubuntu.com/23235882/ (as mentioned above) [02:53] veebers: are you using *agent-metadata-url*? tools-metadata-url was renamed to agent-metadata-url. I don't think we handle the old name in juju 2.0 [02:54] * veebers double checks [02:55] axw: ah yes, we're using agent-metadata-url, the log output is out of date, but the command/config used is correct [02:55] * veebers updates the logging call [02:59] veebers: seems to me that you're doing all the right things. I'll try and repro locally [03:01] axw: awesome, thanks. Please take into consideration the details in that pastebin as it was pointed out to me that there was some missing details in the bug report itself [03:01] veebers: yup [03:01] (I will update the bug with those details shortly) [03:07] veebers: I'm around for about 10 minutes, then I have to go and collect a child [03:09] veebers: well, I can repro [03:09] need to finish something off then I can come back and have a deeper look [03:17] thumper: I bugged axw about it as well. Seems he's able to repro [03:18] axw awesome thanks [03:21] * thumper needs to collect kiddo [03:59] axw: please let me know if there is anything I can help with that bug, it's blocking the upgrade tests which we'll need for between rc to rc and onwards [04:00] veebers: ok, I will look again in a little while and let you know. I think we can take it from here though [04:01] sweet === rmcall_ is now known as rmcall [04:51] wow, weather just went from zero to torrential [05:23] axw: just noticed the bug status changed on that, I'm champing at the bit to see what you come up with ;-) [05:23] veebers: I was just about to comment :) [05:24] veebers: I'm not sure if I did exactly the same as you, but in my case, after bootstrap it's at version 2.0rc1.1, not 2.0rc1 [05:24] veebers: the reason being that the agent stream doesn't include 2.0rc1 agents [05:25] axw: does that .1 carry over to the rc2? (The upgrade attempt is to 2.0-rc2) [05:25] then because the controller is custom, it seems to not want to upgrade to a streams version [05:26] veebers: ^^ haven't gotten to the bottom of that bit yet, but the fact that the server is an unreleased build seems to force uploads [05:27] axw: oh, sorry you're still investigating :-) [06:11] veebers: https://github.com/juju/juju/pull/6327 [06:25] axw: sweet, cheers for sorting that :-) === frankban|afk is now known as frankban [07:41] redir: this si the "main" unreachbale server bug :D https://bugs.launchpad.net/juju/+bug/1605767 [07:41] Bug #1605767: MachineSuite.TearDownTest no reachable servers [07:41] "main" = one of many \o/ [07:43] dimitern: https://github.com/juju/juju/pull/6328 [07:43] macgreagoir: http://paste.ubuntu.com/23239901/ [07:47] mainy [08:26] frobware: babbageclunk: https://github.com/juju/juju/pull/6330 PTAL [10:55] frobware: here's what I have so far: https://github.com/juju/juju/compare/master...dimitern:maas-bridge-some?expand=1 [11:20] here's some cleanup in the apiserver cert changing code - a little simplification and a test: https://github.com/juju/juju/pull/6331 [11:20] dimitern: fancy a review? [11:21] rogpeppe1: I'll put it on my list, sure [11:21] dimitern: ta [11:21] rogpeppe1: might take an hour though, as I'm in crit-bug-fix-mode atm :) [11:21] dimitern: np [11:24] o/ just wanted to drop by and share some love. rc1 has been awesome from the UX fixes in juju status throughout === marcoceppi_ is now known as marcoceppi [11:30] dimitern: https://github.com/frobware/juju/tree/master-lp1627037 [11:31] dimitern: unit tests are broken but it now supports '--interfaces-to-bridge' which is space delimited [11:31] frobware: perfect! [11:36] babbageclunk: PTAL https://github.com/juju/juju/pull/6332 === dimitern` is now known as dimitern [11:57] rogpeppe1: you've got a review with 1 comment [12:00] dimitern: No rush, just another group https://github.com/juju/juju/pull/6333 [12:27] dimitern: ta! === rogpeppe1 is now known as rogpeppe [12:41] mgz: yt? [12:44] redir: yo? [12:53] hey mgz are patches in juju/pathces applied only at test time or also at build time for releases? [12:54] they are applied as part of building the tarball [12:55] which happens as the first stage of the revision test process [12:55] and before any unit test runs [12:55] and as part of the release [12:56] only time it won't happen is on a dev's machine unless they specifically include it [13:23] mgz: thanks [13:23] jam: per our discussion. easy peasy https://github.com/juju/testing/pull/112 PTAL [13:51] alexisb: do we need to talk today? I don't think I have anything to talk about. [13:53] babbageclunk, have you heard back from jhobbs? [14:00] voidspace: mgz natefinch ping for standup [14:10] rick_h_: crap, sorry - omw [14:32] rick_h_: status update; we are getting closer to a patch we can use / build a binary from but have just run into an issue with aliases. [14:32] frobware: ok, thanks for the heads up. What's the aliases issue? [14:33] rick_h_: two things; all the bridge script idempotent transformations fail [14:34] rick_h_: second is that the stuff that dimiter has done on the maas side doesn't consider aliases. [14:35] rick_h_: we could dispense with the second (unit test) transformation but would rather not... [14:35] rick_h_: so this issue is new-right-now. thinking about we do about it. bbiab. [14:37] alexisb: Only what was on that email - there was something wrong with the test setup, he's rerunning now. [14:38] babbageclunk, ok, can you send me a mail to your current PR/branch? [14:38] alexisb: Sure - sent! [14:39] thanks [14:41] alexisb: actually I hadn't sent it then but I have now. [15:04] hiya, when i bootstrap aws with rc1, i get a machine with 3.5G RAM, when i bootstrap azure with rc1, i get a machine with 1.7G. is that normal? [15:05] i ask because once i get up to 10ish machines in a mode, azure slows waaaay down (or rather, juju commands like 'status' and 'models' slow waaay down). [15:06] jam: PTAL https://github.com/juju/juju/pull/6334 [15:06] kwmonroe: so the instances we get up are based on cpu constraints I think. [15:07] kwmonroe: we're looking at tweaking them for 2.0 final but not sure yet on the final bit. I'd be surprised if 10 machines though slow down the status/etc. [15:07] kwmonroe: I'd be curious if it's network lag or something? I mean does it really time slower after reach machine? [15:11] definitely 10 machines should be perfectly fine with 1.7G of RAM. I think, honestly, it's just that the azure machine in general is a much slower machine than what you get from AWS [15:11] azure machines in general are pretty slow, and the AWS machine we get is one of their newer sets. [15:12] rick_h_: natefinch: mongo (1.2G) + jujud (150M) eat up just about all the rams: http://paste.ubuntu.com/23242577/ and kswapd0 is using 50% of my cpu trying to find swap space (which doesn't exist): [15:13] yeah rick_h_, it does time slower after reaching double digit machines... [15:13] $ time juju status [15:13] ... [15:13] real 15m31.609s [15:13] user 0m0.090s [15:13] sys 0m0.050s [15:13] 15 minutes? [15:14] that's crazy [15:14] but mongo using that much is pretty unusual too [15:20] natefinch: rick_h_: is there anything i can pull off this controller to make a useful bug report (like why mongo is using so much ram)? if not, i'm gonna try adding swap space to see if it's slow because it's out of ram, or slow because kswapd0 is eating so much cpu. [15:21] kwmonroe: not that I'm aware of but if you can replicate it pretty consistently it'd be good to know what region, deployment, etc you've got going that is causing mongo to chew up like that [15:21] ack rick_h_, i'll keep an eye out [15:27] dooferlad: also assigned the other related card your way as I think the two bugs are related and might be useful to keep both in mind. [15:27] rick_h_: thanks [15:30] rick_h_, kwmonroe: we should file a bug, so we can track it. [15:31] no one ever said "man, I wish I hadn't filed that bug" ;) [15:31] on it natefinch, i'll spin up a cuople different regions to see how consistent it is [15:31] kwmonroe: awesome, thanks :) [15:33] Bug #1628155 opened: cmd/juju: juju deploy "see also" refers to non-existent command === daniel1 is now known as Odd_Bloke [15:50] redir: babbageclunk: do u think that this failure will be improved with ur work? https://bugs.launchpad.net/juju/+bug/1625768 [15:50] Bug #1625768: github.com/juju/juju/state go test timeout [15:52] anastasiamac: potentially... [15:56] anastasiamac: PTAL https://github.com/juju/juju/pull/6335 [15:57] rick_h_: now live testing https://github.com/frobware/juju/tree/master-lp1627037 [15:59] frobware: <3 [16:07] redir: lgtm [16:13] dimitern: ping [16:13] voidspace: pon [16:14] g [16:14] :-) [16:14] dimitern: it's alright, I'm going to think a bit more and maybe come back to you [16:14] voidspace: sure :) [16:26] dimitern: you still there? [16:30] * rick_h_ goes for lunchables === hml_ is now known as hml === xnox_ is now known as xnox === wwitzel3_ is now known as wwitzel3 === beisner- is now known as beisner [17:04] dimitern: frobware ping please [17:04] rick_h_: pong [17:05] frobware: dimitern can you please join https://hangouts.google.com/hangouts/_/canonical.com/rick?authuser=1 [17:05] rick_h_: omw === akhavr1 is now known as akhavr === stokachu_ is now known as stokachu === wwitzel3_ is now known as wwitzel3 === aluria` is now known as aluria === kragniz1 is now known as kragniz [17:23] rick_h_: I think so? I'll hack the code to pretend we get this from the service and see what happens. [17:23] natefinch: k === Spads_ is now known as Spads === wwitzel3_ is now known as wwitzel3 [17:36] anyone know much about azure that could hop on a hangout real quick? [17:37] i see a failed deployment in the azure portal, but juju isn't catching it === tvansteenburgh1 is now known as tvansteenburgh [17:43] kwmonroe: what's up? [17:43] rick_h_: might be easier to show you.. can you join tvansteenburgh and i? https://hangouts.google.com/hangouts/_/canonical.com/eco-wx [17:44] kwmonroe: omw [17:47] rick_h_, do you want me to join, given asure is on our court atm [17:47] alexisb__: you can, getting folks to file a pair of bugs === frankban is now known as frankban|afk [18:10] uh, do we not have a way to enable trace logging on the client? [18:10] logging-config only affects the server and --debug only makes us output debug [18:52] alexisb__: rick_h_: tvansteenburgh: if there's anything else you might find helpful to debug the "pending" vs "error" machine in azure, lmk. we've still got the env for https://bugs.launchpad.net/juju/+bug/1628246 [18:52] Bug #1628246: juju not detecting azure provisioning failure [18:53] kwmonroe: thanks for filing that [18:53] kwmonroe: ty [18:53] kwmonroe: leave it up a few hours in case axw thinks of anything that he'd find particularly useful [18:53] ack === beisner- is now known as beisner [19:44] we need a sprint on mars [20:01] perrito666: +1 for mars [20:10] man simplestreams is a giant PITA === wwitzel3_ is now known as wwitzel3 === JoseeAntonioR is now known as jose === wwitzel3_ is now known as wwitzel3 [21:40] * katco` just realized she has been glancing at a disconnected #juju-dev screen all day [21:46] axw: ping [21:49] menn0: too early [21:49] katco`: emacs :p [21:49] perrito666: yeah I figured. just trying my luck :) [21:49] doesn't matter, I put my question on the review instead [21:50] perrito666: i know you're joking, but something's going on with my bouncer. the katco here is connected, but my bouncer won't let me connect to that session =| [21:50] menn0: the standup time is pretty much the earliest youll find andrew, that s why its that time [21:50] katco`: interesting, bouncer spitting any logs? [21:50] perrito666: yeah, but nothing that was interesting in 5m of looking [21:51] katco`: just kill it, it happened to me a couple of times when using bip [21:51] perrito666: i rebooted my headless this morning; i think the disk might be going out (don't know if that's related) [21:52] was getting segfaults running any command; inodes corrupt, etc. [21:52] watched fsck run; seemed to clean some things up (shrugs) [21:52] katco`: ouch, that depends too much on your bouncer implementation but make sense that it would need some sort of access to logs/cache to let you in [21:53] well the reboot seems to have cleared that up, that's why i'm not sure if it's related [21:53] plus... the session is connected [21:53] really no clue yet. i'll probably have to look at it tonight [21:53] katco`: well if the files are corrupt you might have that issue even if fsck attempted to repair the disk [21:54] i don't know why it would still connect the session though... === hatch_ is now known as hatch [22:20] katco`: what's the story with PRs against develop? as OCR should I ignore them for now? [22:21] menn0: yeah, working on a PR that encompasses that [22:21] menn0: ta for asking [22:23] katco`: I take it we will all soon be landing into develop instead of master? [22:24] menn0: hooefully [22:24] menn0: i think that's the idea. right now the goal was to begin double-pushing to develop to test out the automated testing, but seems that's happening on master already [22:24] menn0: i imagine that when we switch we'll just rebase develop against a fresher master [22:26] katco`: yea figure just delete/recreate develop/staging and block landing on master [22:27] rick_h_: cool [22:40] thumper, ping [23:17] veebers: standup? [23:17] thumper: I'm just about to head out the door, see ping to alexisb in #juju :-)