[02:30] fark!!!! [02:30] rsyslog is fiddly [02:31] indeed, and the config format is arcane [02:31] hmm... [02:31] I think I broke something [02:33] a bugger [02:33] ugh [02:33] copy and paste error [02:33] * thumper does the destroy, make, bootstrap dance [02:34] make? [02:34] oh, you use make to build? [02:34] make install [02:34] just does: go install ./... [02:34] but easier to type [02:35] make check does: go test ./... [02:35] axw: so... when is debug-log going to use the API [02:36] axw: I have a feeling we need this backchannel that wallyworld keeps talking about [02:36] thumper: it uses it now for getting the SSH address, but I guess that's not what you mean :) [02:36] axw: so the api server looks at all-machines.log and filters, streaming info back to the client [02:36] right [02:36] yeah, I don't know - still slogging through destroy-env atm [02:36] axw: when we get it working properly streaming data over the api, then it'll work with the local provider [02:36] what back channel? [02:36] I'm getting all-machines.log working for local [02:36] there isn't one [02:36] but we should have one [02:37] sorry, what does that mean in this context? [02:37] it means we could get meaningful informational stuff back to the user [02:37] probably easier to explain in a hangout :) [02:37] sure [02:37] \o/ [02:37] it works [02:38] IT'S ALIVE!!!1!11!! [02:38] :) [02:38] hangout now, or are you busy atm? [02:38] what a fucking mission that has been [02:38] about to go and make a coffee and have a quick sit-down with Rache [02:38] l [02:38] but I'll be back after that [02:38] okey dokey [02:38] and we can chat [02:38] sounds good, ttyl [02:38] I could also talk you through these two branches I'll propose [02:39] cool [02:39] one to make the rsyslog udp port configurable [02:39] and the other to get the local provider to use rsyslog and all-machines.log [02:39] I've also changed the output, so instead of just %HOSTNAME%, we get things like 'machine-1' and 'unit-ubuntu-0' [02:39] * thumper goes for a bit [03:08] * thumper goes to fix all the broken tests... [03:08] dumb script checking in cloudinit [03:38] thumper: is there a reason why we don't delete the bootstrap state file if bootstrap fails? [03:39] um... because we might be 'kinda set up' ?? [03:39] hmm I guess. it does stop the instance if it screws up (it's easier to tell that it failed once it's synchronous) [03:43] local provider isn't as good, but could be made so [03:47] axw: hangout? [03:47] sure, just a mo [03:49] thumper: https://plus.google.com/hangouts/_/76cpiq5hco79gtjcr755e8jsok?authuser=1&hl=en [05:08] * thumper is happy that he has figured out some live kvm tests [05:08] * thumper is done for now, back for the team meeting [07:36] fwereade: hiya. TheMue reviewed my state changes for environment Life, but I thought I'd better wait for you since you reviewed the parent branch [07:36] when you have some moments: https://codereview.appspot.com/28880043/ [08:14] morning all [08:14] axw_: using the uuid check for the id looks better now, thanks [08:14] TheMue: cool, thanks for the review [08:14] morning dimitern [08:15] morning [08:40] mornin' all [08:41] morning rogpeppe [08:43] morning [08:43] well, afternoon :) [09:29] mornin' [09:43] morning mgz [09:44] mgz: did you see the recent discussion about maintaining compat? [09:44] the API thread on the list? [09:44] mgz: yeah, abentley reasonably pointed out that Status is something we probably want multi-version compatible [09:44] so don't be too hasty to tear out the client side code yet [09:45] I didn't fully understand his point [09:46] mgz: for CLI API changes that require us to add a new API, we'd like to keep the old code around in the 1.18 client in case it connects to a 1.16 server that doesn't have the new API [09:46] but, I guess that goes for one upgrade step only? we need to remove the mongo access in two stages [09:47] mgz: right. I'm not 100% sure if we want a 1.16 client to be able to do status on a 1.18 but maybe [09:54] jam, mgz: I really don't think we can avoid it [09:54] jam, mgz: 2-way compatibility for minor+2 was always the plan [09:54] jam, mgz: it's just something that we seem to repeatedly fuck up [09:55] jam, mgz: and ofc once we hit 2.0 we need compatibilty all the way back to 2.0 until 3.0 (and even that needs compat back to the final 2.x, I think) [09:57] fwereade: so I know that we didn't do it for several commands so far (I don't know them offhand, but juju destroy-machine --force comes to mind :) [09:58] jam, indeed so, I definitely fucked that up for 1.16.4 -- but it actually just turned out to be early warning of the same fuckup for 1.18 [09:59] jam, casual glance indicates that we've just been doing straight replacement with no fallbacks across the board [09:59] fwereade: that has definitely been the pattern. I did bring it up to you before [09:59] it was fine for all the ones that didn't need a new API [09:59] but the others are all incompatible [10:00] jam, well, fuck, I'm sorry -- I don't recall that, and if I told you to do something stupid then that's all on me [10:01] fwereade: weekly standup, btw [10:05] TheMue: are you coming to the standup? [10:06] jam: oh, yep, forgot that it is now, sorry [10:06] https://plus.google.com/hangouts/_/calendar/bWFyay5yYW1tLWNocmlzdGVuc2VuQGNhbm9uaWNhbC5jb20.09gvki7lhmlucq76s2d0lns804 [10:06] TheMue: ^^ [10:16] can anyone help me with the impact and test case for https://bugs.launchpad.net/juju-core/+bug/1239508 [10:16] <_mup_> Bug #1239508: Empty constraint value lost during some cloud-init step

[10:16] (working the SRU for saucy/1.16.3) [10:17] committed by me? [10:17] * thumper looks [10:19] jamespage: look at my comment, I marked it as invalid [10:19] jamespage: the bug is erroneous [10:19] I filed it, but the bug itself is wrong [10:20] the fundamental problem was the local provider wasn't starting [10:20] and I thought that this was the problem [10:20] but it wasn't [10:20] thumper, ah - ok [10:20] the problem was that the startup process was changed [10:20] thumper, I thought that was the case but I was not sure [10:20] thanks [10:20] meaning that the local provider needed to set up the storage earlier [10:21] hence the branch connected to that bug just has a "setup bootstrap storage" method [10:21] np [10:34] OK - uploaded for SRU team review - lets see how this goes [10:34] MRE soonish if this goes all OK [10:38] ehw, ^^ [10:38] jamespage: \o/ [10:43] fwereade, who's a good person to talk to about what juju-core is doing CI wise? [10:43] I need to evidence sufficient upstream testing as part of the Minor Release Exception we need for Ubuntu [10:46] jamespage: sinzui and abentley seem to be driving that [10:47] jamespage: I'm the one that set up the trunk robot for passing the test suite before landing to truk [10:47] trunk [10:48] jam: unit testing right? or does that exercise with each provider directly? [10:48] jamespage: it tests each provider against its double, but not against a live service [10:48] (eg, not against Amazon) [10:48] testing against Amazon is on Curtis's team [10:49] jam: does the robot cover unit testing pre-landing for stable branches as well? [10:50] jamespage: currently es [10:50] yes [10:50] Ci tests the candidate works on LXC, AWS, HP, Azure, and Canonistack. It also nominally tests stable to next are compatible [10:50] so 1.16 passes the test suite before accepting patches, etc [10:50] thumper, sinzui: I marked bug 1239508 as invalid all round - I got confused so figured others would as well [10:50] <_mup_> Bug #1239508: Empty constraint value lost during some cloud-init step

[10:53] jamespage, Ci also verifies the tarball works with Ubuntu packaging. We are verifying the package upgrade (and downgrade) [10:54] sinzui, is this publically visible somewhere [10:54] useful for my email to the tech board [10:54] visible, but the results are not very intelligible at http://162.213.34.53:8080/ [10:55] jamespage, I just drafted this as a the kind of report we want managers and engineers to see https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AoY1kjOB7rrcdEl3dWl0NUM3RzE2dXFxcGxwbVZtUFE#gid=0 [10:55] g+ kicked me out for some reason and I can't reconnect [10:58] dimitern, not to worry, we'renearly deon [10:58] * fwereade can has typing? apparently not [10:58] fwereade, sticky keyboard? :) [11:03] TheMue: forgot to mention. your package shipped, Monday, I believe, so it should be there sometime in the next 3-30 days :) [11:18] natefinch: fantastic, have to thank you [11:21] TheMue: it's no problem :) [11:23] natefinch: will be paid at least with one (or some more) beer [11:24] TheMue: bring some coffee, I'm not a big beer guy :) [11:25] natefinch: yeah, already thought so. will do ;) [11:36] thumper, http://newraycom.com/how-to-set-up-google-hangout-lower-third/ [11:37] arosales: cheers [11:37] * thumper goes to bed [12:02] sinzui: I set up a 1.17.1 milestone, for things that I know we need before we can do a 1.18.0 final release. [12:02] sinzui: is that a reasonable way to do it ? [12:02] +1 [12:05] axw_: poke if you're still around [12:05] I think the summary for https://bugs.launchpad.net/juju-core/+bug/1246905 needs to be updated [12:05] <_mup_> Bug #1246905: status for manual provider hangs

[12:05] jam: yo, just making dinner - what's up? [12:05] ok [12:05] * axw_ looks [12:05] reading the text, it looks like it should be something about "juju should use netwokr address from ?" [12:06] jam: yes, agreed - I'll update it in a bit [12:07] axw_: thanks, I was a bit confused reading the summary and looking at our trajectory. I think we can unschedule it from "stuff we are working on right now" unless you're looking closer at it [12:07] (so unset the milestone) [12:07] natefinch: are you still looking at bug #1218616 ? [12:07] <_mup_> Bug #1218616: all-machines.log is oversized on juju node

[12:08] I thought I remembered a ping on your logrotate branch [12:08] but I don't know where that's at [12:08] jam: yeah, sorry, it should be unset [12:08] axw_: yeah no problem, I was just going through the list and figured I'd check it with you [12:10] jam: I should pick it back up.... all it really needs is a signal to juju to have it reopen the log file so it starts writing to a new one, rather than the old one. Not exactly sure how hard that'll be to implement, since I don't know the core logging code that well [12:10] natefinch: I'm assigning https://bugs.launchpad.net/juju-core/+bug/1248329 to you. Is that ok ? [12:10] <_mup_> Bug #1248329: juju destroy-environment does not accept -e

[12:11] it sounded like you were going to pick it up at the meeting today [12:11] jam: yep [12:12] jam: so we'll support -e but not JUJU_ENV ? [12:13] natefinch: and not ~/.juju/environments [12:13] at least, that is what I'm going with [12:14] as that will be "you can run juju destroy-environment -e YYYY" on 1.16 and 1.18 [12:14] natefinch: so you *have* a transition step. Also specifying "-e env" still gets us "you had to type it manually" [12:14] which is what the old bug was about [12:14] I can respect the "required arguments should be positional" but we still hold to the letter of the original request [12:14] (don't make it so easy to destroy the production env) [12:15] jam: right. Are we going to remove -e in 1.20 then, or keep it? [12:15] natefinch: *shrug* [12:16] If someone feels super strong that we must not have a required but flagged argument then we can remove it in 1.20 [12:16] natefinch: you can write a bug about it and target 1.19 if you like [12:16] jam: yeah, it does bug me to have required flags :) [12:17] natefinch: I think of them a bit like named arguments [12:17] I actually like them sometimes [12:17] especially vs a "cmd dosomething true" [12:17] cmd dosomething -tox=true [12:17] fwereade: do you feel bug #1233936 is very important for 1.18 ? [12:17] <_mup_> Bug #1233936: worker/uniter: uniter restarts when relation removed

[12:17] jam: there are cases where it makes more sense, like if you have several arguments, or the value of the argument is something non-obvious, like a boolean or a number [12:18] (iow, target 1.17.1 or leave it at 1.19?) [12:18] jam, 1.17.1 [12:18] done [12:19] jam, it's not even hard -- find everywhere in worker/ and cmd/jujud/ that checks for NotFound, and cause it to also check for Unauthorized [12:19] jam, (ok that's more than the bug, but that bug is just a symptom of the broader problem) [12:20] fwereade: sure, but doing that work as part of the bug is fine [12:20] you could even land that today *wink* :) [12:20] jam, but actually coding doesn't seem to be something I can get away with doing at the moment [12:20] :) [12:20] I understand the feeling [12:20] fwereade: it does feel like something that is really missing test coverage [12:20] somehow [12:21] fwereade: I suppose our test suite doesn't assert that services don't bounce ? [12:21] jam, it was something we agreed and planned but missed in the actual hurly-burly of developing stuff [12:22] jam, well, I spotted an unchecked error return in the uniter.filter test suite that would have caught it [12:22] jam, but yeah, I don't think we have adequate coverage of such situations [12:22] jam, otherwise I don't think there'd be any bare NotFound checks [12:45] dimitern: ping [12:46] sinzui: bug #1253628 *might* be a blocker for juju 1.17.0, we could sort of go either way here [12:46] <_mup_> Bug #1253628: juju upgrade-juju incompatible with 1.16

[12:46] but not being able to upgrade 1.16 to 1.17 is a bit bad [12:46] yeah [12:47] sinzui: I *think* we don't have to block. because people can still test if 1.17 works for new workloads, and we advocate that people *never* upgrade a production env to a dev release [12:47] so not being able to, hey we just helped you out, right ? :) [12:48] jam, they cannot upgrade without the --dev flag [12:48] though... [12:48] sinzui: right, but they *can't do it at all* with a 1.17 that we release today :) [12:49] yesterday I upgraded the charmworld staging site from 1.13.3 and was surprised to see 1.15.0 selected. That was the aborted release. I then specified the upgrade to choose 1.14.1 then 1.16.3 to upgrade along a path I know worked [12:50] sinzui: right, we've explicitly stated that we will support stable => stable, but we don't guarantee from/to dev releases. [12:50] sinzui: I' believe Dimiter was specifically doing upgrade work so that it wouldn't default dev => dev anymore [12:50] fab [12:50] (so 1.17 won't try to upgrade to anything but 1.17+ by default, not 1.19) [12:52] dimitern: I'd like you to pick up bug #1253628 if you can [12:52] <_mup_> Bug #1253628: juju upgrade-juju incompatible with 1.16

[12:55] jam, will take a look [12:55] jam, is it more important than the CLI stuff? [12:55] dimitern: it is the compatibility for the CLI stuff. [12:55] and yes [12:55] dimitern: essentially because you use Client.EnvironmentGet [12:55] you *can't* actually upgrade 1.16 to trunk [12:55] because that API doesn't *exist* in 1.16 [12:55] fwereade: not sure if you saw this earlier, because you got cut off shortly after I sent it: [12:55] fwereade: hiya. TheMue reviewed my state changes for environment Life, but I thought I'd better wait for you since you reviewed the parent branch [12:55] when you have some moments: https://codereview.appspot.com/28880043/ [12:56] jam, ok, will switch to it then [12:56] axw_, I did, and I started to make comments, and then meetings swept me away [12:56] no worries [13:04] axw_, sent [13:10] fwereade: thanks [13:11] sinzui: I'm upgrading bug #1250154 [13:11] <_mup_> Bug #1250154: updateSecrets in juju-1.18 uses Client.EnvironmentGet which is not in 1.16

[13:12] sinzui: it turns out that axw_'s patch to try to pass secrets on demand [13:12] broke *every* cli command against 1.16 [13:12] :o [13:13] Yay! I like the kinds of changes that are 100% something. Thats how I roll. [13:13] sinzui: no half-assed measures [13:13] we're gonna break it [13:13] and we're gonna do it right! [13:14] axw_: not particularly your fault, as we weren't being cautious and labeling what the new APIs were [13:14] axw_: can I assign it to you ? [13:14] jam: certainly [13:14] nope. Not one promulgated charm branch worked yesterday thanks to me [13:14] axw_: I *think* we can just fall back to finishing api.Open in that case [13:14] axw_: because we don't *have* to pass secrets. [13:15] (most of the time) [13:15] jam: ah, because it's existing [13:15] yep [13:15] axw_: so there is still the potential for someone bootstrapping with 1.16 and then we try to use 1.18 for everything else [13:15] I assume we don't have to cater for installed 1.16 without having ever connected to it? :) [13:15] but I think that window is small enough we can document it and WontFix [13:15] +1 [13:16] "delete and rebootstrap with 1.18" [13:16] axw_: worst case, yeah [13:16] as long as destroy-environment still works :) [13:16] hehe [13:17] axw_: that particular bug makes it hard for me to test what other commands we broke, becaues... they're all broken :) [13:18] jam: all the SSH-based ones, IIRC, use a new API [13:18] yep - I added the PublicAddress API [13:19] sigh [13:19] axw_: yeah, I'm just going throug hthe "bzr diff -r juju-1.16.3..trunk" [13:19] so I'll get there eventually [13:19] jam: so that's ssh, scp, debug-hooks, debug-log [13:19] fwereade: compatibility is turning into quite a bit of work [13:20] can we just unbreak upgrade-juju and leave the rest? ;) [13:20] axw_: well updateSecrets is a definite fix, and upgrade-juju is [13:20] status hasn't been done yet [13:20] axw_: for the rest... I wanted to scope the work and then decide [13:20] but it is looking pretty big [13:20] ok [13:21] jam, isn't the secrets-pushing just one piece of work? ie if the api isn't there, fall back to a state connection to push them using the previous mechanism? [13:21] fwereade: so *that* is one of the "we must do" bits [13:21] fwereade: well, I don't think we *have* to fallback, but we could [13:22] fwereade: the big scope of the work is falling out from my auditing [13:22] I'm up to about 10 commands and counting [13:22] jam, that just didn't have APIs at all in 1.16!? [13:22] jam, how did the gui get anything done? [13:23] fwereade: well they were missing a key api. [13:23] fwereade: like "juju set-constraints" didn't support EnvironmentConstraints [13:23] fwereade: so the GUI could change a service but not the global [13:23] but they didn't need to [13:23] I guess [13:23] add-machine wasn't done by the guy [13:24] debug-hooks, ssh, scp weren't done by the gui [13:24] fwereade: service-set and unset needed new APIs to handle the empty string vs null appropriately [13:24] destroy-machines, [13:24] environment-get/set [13:24] so basically, the GUI didn't do machines or environment level things [13:26] dimitern: just to keep you in the loop, bug #1250154 is blocking *all* of trunk talking to 1.16 so if you go testing manually that needs to be fixed first. axw will pick it up, but you can do a quick hack if you want to test it. [13:26] <_mup_> Bug #1250154: updateSecrets in juju trunk uses Client.EnvironmentGet which is not in 1.16

[13:29] jam, ok, so what's the fix we're looking for? [13:29] jam, migrate my set-/get-environment changes into 1.16? [13:29] dimitern: updateSecrets (in a first pass) should just keep going if it can't get EnvironmentGet [13:30] dimitern: because the 99% case is that we don't actually need to pass the secrets [13:30] (it is only needed if they bootstrapped with 1.16 and then never did anything but upgraded to 1.18) [13:30] i'm not sure I get you [13:30] in trunk we need to implement a workaround for using state.EnvironConfig if client.EnvironmentGet fails? [13:31] dimitern: no. The particular bug is that when doing NewAPIConnFromName we check if we need to pass secrets by calling Client.EnvironmentGet [13:31] but that API didn't exist in 1.16 [13:31] however, we *also* don't normally need to set secrets [13:31] so a quick fix [13:31] is that if we try to EnvironmentGet in updateSecrets, and it fails, just keep going. [13:31] jam, ah, so just skip it and ignore the error? [13:32] jam: did you mean to set the overall compat bug to me, or the upgradeSecrets one? [13:32] (you assigned the former) [13:32] axw_: just the upgradeSecrets one [13:32] I'll fix it [13:32] ok [13:32] ta [13:33] axw_: I'm making you responsible for all that is wrong in the world. Are you ok with that? y/Y ? [13:33] ;) [13:34] yay, DestroyRelation should be ok. Finally something that may not have broken (aside from the breaking of everything :) [13:34] jam, so I shouldn't look at bug 1250154, but at bug 1253628 only? [13:34] <_mup_> Bug #1250154: updateSecrets in juju trunk uses Client.EnvironmentGet which is not in 1.16

[13:34] <_mup_> Bug #1253628: juju upgrade-juju incompatible with 1.16

[13:35] dimitern: well, I think axw_ is officially done working for today. So if you want to pick up 1250154 before he gets to it. It is the more important one. But both are important and you have more context for upgrade-juju [13:35] I'm confused now :/ [13:36] what's the workaround for SetEnvironAgentVersion ? [13:36] dimitern: just focus on fixing juju upgrade-juju when running against 1.16 [13:36] dimitern: essentially juju upgrade-juju needs to fall back to poking the DB directly if it can't client.EnvironmentGet [13:36] 1.16 in the cloud when using cli from trunk? [13:36] since if it can't EnvironmentGet then it can't SetEnvironAgentVersion [13:37] dimitern: "juju-1.16 bootstrap; juju-1.16 deploy; juju-trunk upgrade-juju --dev" is broken [13:37] I've gotta go, I'll look at 1250154 first thing in the morning if it's still broken [13:37] axw_: have a good night [13:38] jam, upgrade-juju --dev is no longer present in trunk [13:38] cheers, good night jam, all [13:38] dimitern: "juju upgrade-juju" is just broken [13:38] axw_, g'night [13:38] later dimitern [13:38] dimitern: we *can't* upgrade a 1.16 installation to anything [13:38] because it doesn't have the API we added to do so [13:38] regardless --dev or whatever [13:38] I am uncertain about bug #1253576. I expect a relation error to eventually be shown in status if the hook didn't complete. [13:38] <_mup_> Bug #1253576: Juju does not show relation status

[13:39] jam, so the fix should be "try EnvironmentGet, if it fails, access the state directly?" [13:39] dimitern: right [13:39] jam, if it fails in a certain way that is - when it's missing [13:40] dimitern: well, we can just fallback and then worry about other reasons why it fails [13:40] jam, so we'll have 2 commands in 2 old upgrade-juju and the new one [13:40]