=== gary_poster is now known as gary_poster|away [02:59] thumper: yo [03:12] wallyworld__: hey [03:12] wallyworld__: how's it going today? [03:12] thumper: notice how i didn't ping :-) [03:12] going ok, deep into some refactoring [03:12] wallyworld__: as you may have seen, I've put the kvm broker review up [03:12] wallyworld__: yeah... [03:12] thumper: funny that, i have a question [03:12] wallyworld__: I'm now getting containers to return hardware characteristics [03:13] quick hangout? [03:13] https://plus.google.com/hangouts/_/76cpj9mtcok4cua6di82i0o3ms?hl=en [03:14] * thumper joins === Beret- is now known as Beret [04:39] wallyworld__: https://codereview.appspot.com/36980043/ [04:40] \o/ [04:41] wallyworld__: it's a start [04:41] yes [04:45] thanks for the review [04:45] I'll tweak and land tomorrow morning [04:45] night all === jcsackett_ is now known as jcsackett === meetingology` is now known as meetingology [05:28] gtg to my daughter's school orientation, bbl [06:43] back [08:14] mornin' all [08:16] morning === jam1 is now known as jam [08:24] smoser: there was a discussion about allowing charms to add constraints (like mem, etc), I could see that being extended to support stuff like "!lxc". However, that probably isn't on the roadmap for this cycle. [09:18] jam: I forget, do we need to support old CLI with new server? [09:18] jam: just wondering if I can remove secrets pushing API [09:19] server-side [09:19] axw: the old CLI didn't push secrets via PI [09:19] API [09:19] so yes, but we don't have to keep that bit [09:19] axw: we have to allow the old CLI direct DB access [09:19] jam: ah, it's only on trunk isn't it? [09:19] no.. I broke the last release [09:20] axw: ? you broke the new CLI connecting to the old server [09:20] IIRC [09:20] but we can drop that bit [09:20] right now we are trying and if it fails just continuing [09:20] ok [09:20] with synchronous bootstrap, we don't even have to try anymore :) [09:20] jam: well, we should still push secrets for old servers, right? [09:21] or do we not care? [09:21] assume they're already set up? [09:21] axw: even if there was a 1.17.0 that had async bootstrap and we were pushing via the API as a dev release wedon't have to push secrets to it in 1.17.1 [09:21] axw: we don't care if it is only a dev release [09:21] thats why we call them *dev* [09:21] jam: sorry, I mean, can we drop the code that pushes secrets to existing installations [09:21] it is our way of "lets get this out there, without committing to supporting migration to/from it" [09:21] of non-dev [09:22] axw: so for pushing secrets directly to the DB, I'm not sure if we can drop it. [09:22] "maybe" [09:22] we had talked about "if you bootstrap with 1.16 and then *never do anything with it* and then try to connect with 1.17" that might be broken, but we're not sure we care [09:22] I really don't think we should [09:23] (you can destroy-env & rebotstrap because you have nothing in your env, or you can connect with the 1.16 that you bootstrapped, etc) [09:23] indeed [09:23] axw: so *I* wouldn't immediately say "when connecting via direct DB access don't pass secrets" [09:23] axw: it isn't worth poking that code just to remove it if we don't have to touch it at all [09:23] axw: I'd much rather focus on "we never connect to the DB for a 1.18 client and server" [09:23] so that instead of having a "and now we don't push secrets" we end up with a "and now we never connect to the DB" [09:24] axw: so *probably* we could drop it, but I'd rather get to the point where we can drop NewConnFromName completely [09:24] understood [09:24] hmm [09:25] jam: the reason why I'd like to remove it altogether, is then we can get rid of the idea of secrets [09:25] but I can leave it for now I guess [09:27] axw: we can't. we still have to not put the secrets into cloud-init, and only pass them once we connect to the bootstrap node [09:27] axw: the reason we have them today is to not put them into cloud-init [09:27] jam: that bit's irrelevant, we never put any config into cloud-init anymore [09:27] (with synchronous bootstrap) [09:28] cloud-init now does ssh keys, and that's it [09:29] axw: so IMO, changing that isn't our first priority right now. I do like that, but I'd be fine if getting rid of the notion of secrets was in 1.20 [09:29] mmkay [09:29] axw: vs, we actually have to finish stuff like upgrade-charm and status or we can't remove direct DB access in 1.20 [09:31] axw: and, of course, time spent with sinzui to make sure CI is happy is time very well spent. [09:31] jam: indeed. not sure what's going on there :/ [09:31] I can't reproduce the issues on either canonistack or ec2 [09:32] axw: I haven't been able to either. there was a comment about connection flakiness for them. Certainly how did we get a machine up and running and then not have SSH configured. [09:32] axw: one thing I was wondering [09:32] how do we pick the ssh private key to connect with? [09:32] If I set "authorized-keys: foobar" in my environments.yaml how do you match that with eg ~/.ssh/id_rsa_ec2key [09:33] jam: by default I think it takes id_dsa.pub, id_rsa.pub and identity.pub from ~/.ssh [09:34] jam: now sure what you mean by "how do you match that" [09:34] I think ssh will just cycle through all the possible private keys on ~/.ssh? [09:35] axw: no [09:35] it cycles through everything in your ssh-agent [09:35] I think [09:36] but for direct keys you can configure it in ~/.ssh/config or supply it as "ssh -i PATHTOKEY" or a couple of other ways. [09:36] but I'm pretty sure it doesn't just try keys on disk [09:36] I could be wrong, and I've overspecified my ssh config [09:36] no actually I think you're right [09:36] but I was wondering if what sinzui and aaron were seeing was because of keys not getting picked up correctly. [09:36] it'll also try those defaults I mentioned [09:37] jam: well, that bit hasn't changed at all, so I don't understand how that'd be the case [09:37] axw: right, the specific 2/3 private bits [09:37] axw: they weren't having Juju drive SSH before [09:37] axw: in the test suite [09:37] ah, true. [09:37] :) [09:37] they were copying logs of via scp at one point [09:37] but that command *might* be configured specially [09:38] the fact that they get "Permission denied (publickey)" sometimes hints that *something* is doing tat. [09:38] axw: for Canonistack, I've heard another wrinkel [09:38] if you give a Canonistack instance a floating IP [09:38] that becomes a world-routable IP, but it is *not* routable within the cloud [09:38] so machine A on canonistack has to talk to machine B in canonistack via the private address, and *not* the floating ip [09:39] axw: so you might try "use-floating-ip: true" in your canonistack config and see if that breaks bootstrap for you [09:39] hurngh [09:39] it shouldn't from local [09:39] but it might from cstack => cstack [09:39] no, but maybe from jenkins [09:39] axw: ah, also [09:40] you can't directly connect to CStack machines [09:40] * jam => lightbulb [09:40] yeah, gotta sshuttle [09:40] anyone know anything about amazon request limits? [09:40] axw: actually most people configure their SSH to bounce via chinstrap [09:40] axw: ProxyCommand ssh chinstrap.canonical.com nc -q0 %h %p [09:40] i'm still trying to get this guy's environment up, and we're seeing "Request limit exceeded" errors [09:41] axw: so for CStack abentley and sinzui both probably have that, which would let them SSH to a machine, but would *not* let them Dial a machine [09:41] jam: doesn't explain the permission denied error tho [09:41] rogpeppe: more that 100/5s is going to trigger their limits, but I don't know what the actual values are (xplod charm was causing amazon request limit exceeded problems before we fixed it) [09:42] axw: It would explain not being able to connect, but yes, doesn't explain perm denied [09:42] I think you're probably onto something with the private keys/them not using ssh before [09:42] axw: connect as "ubuntu" user? [09:42] jam: yep [09:42] jam: we're still seeing this error after a night of inactivity [09:42] axw: I *think* user@host:port is openssh specific [09:43] rogpeppe: whose inactivity :) [09:43] rogpeppe: everything shut down? [09:43] jam: well, we stopped jujud-machine-0 [09:43] rogpeppe: i [09:43] jam: i wonder if the other instances are making ec2 requests [09:44] if this was 1.13.2 the User and Machine agents had provider creds, IIRC [09:44] jam: everything is on 1.16.3 AFAIK [09:44] so they could have been doing it [09:44] rogpeppe: did you actually get it all moved over? [09:44] jam: yes [09:44] jam: but now we're getting this problem [09:44] jam: one issue after another :-( [09:53] wow, that ping thread is getting on a bit [09:54] axw: which ping thread? [09:55] rogpeppe: warthogs [09:56] axw: ha fun [09:56] axw: i don't look at warthogs much [09:58] rogpeppe: any chance to take a look at my CL today? [09:59] rogpeppe: oh, and hello btw [09:59] TheMue: i started on it yesterday, will continue today [09:59] rogpeppe: ah, looking forward, thx [09:59] TheMue: the main issue i have with it so far is that it always reads from the very start of the file [10:00] TheMue: and for very big files (and log files can be very big) that's a significant waste of resources [10:01] rogpeppe: I have to admit I stole an algo of you of the newsgroup :) [10:01] TheMue: ha ha [10:02] rogpeppe: but doing the initial reading from the end indeed seems better [10:02] TheMue: when was that from? [10:02] rogpeppe: oh, pretty old, would have to look again [10:02] rogpeppe: but I liked the clean approach [10:02] TheMue, rogpeppe: one thought would be that the api could pass in a possible "bytes from start" to give some context as to where it was looking, which might be a negative number to mean from the end of the file ? [10:03] jam: i think it would be more intuitive if the api passed in number of lines of context [10:03] rogpeppe: it would, but it is also *very* hard to do efficiently, vs if you had a byte offset hint [10:03] it could even just be a hint [10:03] jam: or even a start date [10:03] jam: it's not too hard [10:04] jam: i've done it before, and tail(1) does it without too much difficulty [10:04] rogpeppe: but even then you would have to find it in the file first. ok, a binary search could hel. [10:04] TheMue: you can't do binary search [10:04] TheMue: but you can read backwards [10:05] TheMue: actually, you could binary search if you're looking for a start date [10:05] rogpeppe: that's what I meant [10:05] the difficulty with a start date is clock skew [10:06] log lines won't necessarily be in strict date order [10:06] but that might not be too much of a problem in practice [10:07] rogpeppe: will talk to frankban if that inaccuracy would be ok there [10:07] rogpeppe, fwereade_, mgz, jam, others interested: PutCharm proposal document for comments https://docs.google.com/a/canonical.com/document/d/1TxnOCLPDqG6y3kCzmUGIkDr0tywXk1XQnHx7G6gO5tI/edit# [10:07] and as for client vs server clocks, you could probably ask for a given duration before the last log message [10:08] rogpeppe: hey, that's a nice approach, like it [10:08] rogpeppe: with different operating modes I still can keep the full scan if wanted [10:09] dimitern: looking [10:11] TheMue: I think the GUI probably wants "as many old lines as is comfortable to put into the UI" so strictly restricting by date might be unwanted [10:11] consider "it failed 2 days ago" [10:11] or "it looks failed, when did it fail" ? [10:11] So *hinting* to make finding the right value sounds good, but assuming it is just a hint is probably worthwhile [10:12] jam: i think that's probably more of an argument for being able to move back in time [10:12] I guess you could just seed "estimated size of line" in the code, and then update that estimation for a given request as you read through stuff and filter it [10:12] rogpeppe: sure, but what estimate is 'good' for 100 lines [10:12] jam: i don't think we need to estimate in bytes [10:13] rogpeppe: I mean we don't have the API hint, but we use an internal hint [10:13] given that you might have a really noisy machine that *isn't* in the filter [10:13] or the machine you are reading is noisy itself [10:14] jam: are you suggesting this as an optimisation? [10:14] rogpeppe: right [10:14] I think we don't need to expose it to the UI, and we do already have "num lines" in both the CLI and in what the GUI would like [10:14] that said [10:14] jam: i'm not entirely sure i see how it helps [10:15] rogpeppe: I think we can land an unoptimized version as long as we can think of a way to make it better when we need to. [10:15] jam: could you explain? [10:15] jam: yep, wanted is number of lines [10:16] rogpeppe: so we want to get (eg) 100 lines of filtered context for the UI. We start at some place near the end, read and determine how many filtered lines are in that space, and then jump back an estimate based on the number of lines we've found so far. [10:16] you could make the default start 1MB from the end of the file, which might catch 90% of the actual cases [10:16] but the specific tweaks would all be things that we'd actually need real world testing to fine tune [10:16] so lets not optimize too much until we have evidence that it is a problem [10:16] jam: so you're suggesting that we might or might not return the number of lines requested by the user? [10:17] jam: personally i prefer to avoid heuristics when we can [10:17] rogpeppe: I'm saying we start with an estimate of where those lines might be, and keep looking until we find them, potentially hitting the beginning of the file [10:17] rogpeppe: seeking from the end ==> heuristic about how much you should read in one chunk, etc [10:18] one mo, am just going to check if this guy's environment is actually working [10:18] if we do "something" that is ~ reasonable, we get most of the benefit and can drive the rest of the work by actual content [10:19] TheMue: so I guess my point is, we know the log file can get to multiple GB, so a small amount of "try to get the answer near the end of the file" is worth implementing. But don't do a lot of work to optimize the code until we actually know it is a problem [10:19] jam: btw, how often do we log rotate? [10:20] TheMue: we don't yet [10:20] IIRC natefinch started on something, but he wasn't able to get all files so stopped trying [10:20] jam: ouch [10:21] (he could get all-machines.log to be better, but juju itself kept the log file handle open, so it just kept writing to the rotated place) [10:21] TheMue: bug #1191651 [10:21] <_mup_> Bug #1191651: Juju logs don't rotate.

[10:22] or bug #1078213 [10:22] <_mup_> Bug #1078213: juju-machine-agent.log/logs are not logrotated

[10:22] jam: ic, and needs indeed a solution, especially for larger environments [10:23] TheMue: sure, and it will probably also end up interfering with debug-log, which we'll want to sort out, but it can be an exercise in the future for now [10:23] though I think just rotating all-machine.log would be a big win today [10:23] jam: yep [10:23] even if we aren't rotating everything [10:24] all-machines.log is something we can do 'easily' because rsyslog already has hooks for rotating its log files [10:24] vs jujud that would need a SIGHUP or something to be added [10:27] jam: btw, did you looked at the CL so far? [10:28] TheMue: only in brief, I didn't review it. [10:31] jam: ah, ok [10:31] oh darn, another problem encountered. [10:31] 2013-12-04 10:19:20 ERROR juju.provisioner provisioner_task.go:342 cannot start instance for machine "77": cannot set up groups: cannot authorize securityGroup: The permission '36226792-3--1--1' has already been authorized on the specified group (InvalidPermission.Duplicate) [10:31] anyone seen the above error before? [10:32] rogpeppe: I saw it once. After I had a permission group and I added ICMP to it. Then destroyed and rebootstrapped [10:32] because destroy doesn't delete perm groups [10:32] it saw a group that already existed and tried to reconfigure it [10:32] apparently in a duplicate fashion [10:32] * rogpeppe goes to look at that logic [10:33] rogpeppe: I *fixed* it by deleting all the low-numbered juju-ENV-0,1,2 etc groups [10:33] these guys are really seeing the worst of juju [10:33] in this case, you'd want to delete juju-ENV-77 [10:33] juju-ENV-machine-77, I think [10:36] jam: hmm, i guess it might just be an eventual consistency issue [10:37] jam: we revoke first, but perhaps the authorize hasn't seen the initial revoke, so it gives the duplicate error [10:46] jam: FYI i just looked at all their security groups and there's no security group for machine 77 [10:46] mgz: rogpeppe: TheMue: standup ? [10:46] jam: i think it must be the global group [10:46] https://plus.google.com/hangouts/_/calendar/am9obi5tZWluZWxAY2Fub25pY2FsLmNvbQ.mf0d8r5pfb44m16v9b2n5i29ig [10:47] could be [10:48] there in a sec [11:14] * dimitern gives up on the hangout [11:24] mgz: I approved your https://code.launchpad.net/~gz/juju-core/1.16-juju-update-bootstrap-tweak/+merge/196950 but you might want to hold off for a tick [11:24] IsStateServer got renamed to IsManager in the old 1.16.4 stuff [11:24] so it conflicts there [11:25] wait, we moved the rename back to 1.16 in the end? [11:25] mgz: that is part of destroy-machine --force, which is still targetted to the 1.16 series (as in something that customers do need, so we're putting it into the current stable series) [11:25] >_< [11:26] I'll merge and fixup conflicts [11:26] mgz: it hasn't landed yet, I accidentally proposed against trukn [11:28] okay, so if I win the race, *you* have to fix it up? :0 [11:28] that was not the correct smilie... [11:31] mgz: that is true, but I just marked mine approved [11:31] :D [11:32] mgz: well, frank's stuff is in the queue now, so you have a chance [11:33] depending on what the bot finds first :) [11:48] mgz: mine landed first :) [11:49] I let you win :P [11:49] I didn't realize you could do that, what does it look like on your end (Insert horizontal rule) [11:50] mgz, jam: geeks :D [12:16] * TheMue => lunch [12:52] hey natefinch. got my xps 15 today - did you have any issues loading 13.10 onto it? [12:52] I'm getting weird udev issues at startup :\ [12:52] about to go back to 12.04 and then upgrade... [13:00] axw: I just transferred my hard drive from my old machine to this one [13:01] axw: but that worked fine :D [13:01] morning natefinch, I hope you're feeling better [13:02] jam: got a litle extra sleep, feel like a new man... well, not exactly, but not bad :) [13:02] a new-ish man [13:03] jam: certainly better than I would have if I'd gotten up at 5:20 [13:05] natefinch: aha. I was thinking of doing that as a last resort :) [13:08] axw: sorry, it never occurred to me that installation would be a problem [13:09] natefinch: no problems, I'm somewhat used to this :( [13:09] sad to say === gary_poster|away is now known as gary_poster [13:24] lunch [13:47] rogpeppe, jam, any comments on the PutCharm doc? [13:48] mgz, if you want to take a look as well? [13:48] dimitern: i will have; juggling a few things currently. [13:48] rogpeppe, sure, take your time [13:49] dimitern: sure, looking [13:50] geh, google doc links are annoying to transfer across... :) [14:01] dimitern: left a couple of notes [14:05] rogpeppe: Any news from that issue? [14:06] niemeyer: we got it all working, but they've decided that juju is not for them in the future, sadly [14:06] rogpeppe: Due to that issue, or did they bring up any other reason? [14:06] niemeyer: a number of issues [14:07] rogpeppe: Okay, well.. at least we have feedback to work on then [14:07] niemeyer: yes - i am putting together a summary [14:07] rogpeppe: Thanks a lot [14:07] niemeyer: i asked if they could summarise their juju experience for us too [14:07] niemeyer: they've put a lot of time into it [14:12] mgz, cheers [14:21] jamespage, I think devs are willing to forgo an SRU for saucy for the sake of 1.16.4. [14:36] sinzui, jamespage: so 1.16.4 is set in stone (IMO) because it is actually being used in the wild. If we decide we have to revert things/move it to 1.18/whatever we can do so as necessary [14:36] jam, sinzui: so I guess you would like me to upload it to trusty and do the backports dance right? [14:37] jamespage: I think we want to make binaries available, I'm not 100% sure how that has to happen [14:38] jamespage, yes please :) [14:40] sinzui, jam: OK _ but as we probably can't sru this point release, it can't go into cloud-tools [14:49] sinzui, jam: uploaded to trusty and backporting for the stable PPA now [14:50] Thank you jamespage [15:08] jam: It appears that bug #1257481 applies only when hyphen is used as a separator. Can you confirm? [15:08] <_mup_> Bug #1257481: juju destroy-environment destroys other environments

[15:08] abentley: yeah, that seems right [15:09] we match machines on juju-ENVNAME-* [15:09] so, be more creative in your naming :0 [15:09] abentley: yes [15:10] jam: we can just make the pattern better, right? [15:10] in fact, I think it *was* better in pyjuju [15:10] mgz: see the branch associated with the bug [15:10] jam: I see no associated branch [15:10] mgz: "juju-ENVNAME-machine-\d*" [15:11] mgz: because I fat-fingered it to 1256481 [15:11] ugh [15:11] jam: you li... right [15:11] reviewing, at any rate :) [15:12] mgz: I'm proposing one with the right associated bugs, but the code is the same [15:12] then again, it is *one* line :) [15:12] jam: really does seem like we want a test [15:13] mgz: how would you write such a test? [15:13] mgz: we do still have 1 test that calls AllMachines [15:13] and I did manual testing on HP and Canonistack [15:14] I can do a whitebox test of what the regex is [15:14] but that doesn't really help much [15:14] you really need a test in an environment, because the thing interpreting the regex is HP/Canonistack/Havana/etc [15:14] mgz: and it doesn't help that our "go test -live" is actually broken right now (something about not having AgentVersion in the config) [15:15] jam: can write a local live test, hook in some extra instances, and assert only sanely named ones are returned by Instances [15:15] doesn't need to be a live live test [15:16] ...but I do see why you want that [15:16] mgz: so we can, though it sounds very close to testing that I implemented exactly that, if you have a good way to phrase it that isn't *too* tied to implementaiton, I do think it is good "I didn't screw this up when I refactored code" [15:17] manually nova boot another server with a name that would have matched under the old pattern, but won't under this? [15:17] that way we have a test that fails on the actual bug reported [15:18] mgz: reasonable point, please leave it in the review so I remember to do it [15:19] jam: are you putting up a new -cr or should I use that one? [15:25] dimitern, fwereade_: i've added my original thoughts for how we might do charm uploads to the PutCharm proposal. [15:28] dimitern, fwereade_: it seems a bit simpler to me, but there's probably a good reason why it would not work [15:28] rogpeppe, thanks, i'm reading it now [15:29] rogpeppe, and commenting [15:32] jam: Thanks for the quick fix. Looks like it will be really hard to cause a bogus match from here on out. [15:35] niemeyer: got a second? Hopefully quick question about replica sets [15:36] natefinch: Yo, yep [15:38] niemeyer: I'm writing tests for my code that configures replica sets, so it brings up some mongo instances with mongod --replset foo etc, but I need a good way to test if they're fully up and ready before adding them to the replica set. Suggestions? Would Ping() work? [15:43] niemeyer: my thought was to put a direct dial and then a ping into a loop, and wait til it succeeds or passes a deadline... but it seems never to succeed [15:43] can I have stamp on codereview.appspot.com/37210044 merge of 1.16 into trunk please? [15:47] jam: It appears that when you create an instance on openstack, you can attach metadata to it. And when you list instances, you can retrieve that metadata. This could be a sure-fire way of ensuring you delete only the instances you created, e.g. by storing the env name as metadata. [15:48] niemeyer: nevermind, I think I just figured it out... forgot to use a monotonic session. [15:55] natefinch: Have you seen the LiveServers method? That's one way too [15:55] natefinch: Oh, *before* adding to the RS.. okay, nevermind [15:56] niemeyer: right, the problem is that I was trying to add them to the replicaset before they were fully up, so mongo would complain that it couldn't add a majority of the servers to the set [15:56] fwereade_, jam, dimitern: any idea how the problem mentioned by iri- in #juju might be happening? [15:56] natefinch: Understood [15:57] fwereade_, jam, dimitern: (iri- is peter waller, BTW, the person I've been trying to sort out the environment of) === natefinch is now known as natefinch-dentis === natefinch-dentis is now known as natefnch-dentist [16:29] rogpeppe: can I have a stamp on the trunk merge of 1.16 please? cr 37210044 [16:29] mgz: looking [16:30] mgz: LGTM [16:30] ta! [16:43] hi rogpeppe or fwereade_, when bootstrapping juju is there a way to get the same effect of 'default-series' in the environment file via the command-line? --series does not seem to be it. [16:44] bac: i think it has to be in the environment config [16:44] rogpeppe: that was my conclusion but i'd hoped i'd overlooked something. [16:48] bac, it's not ideal, but you can always change default-series with set-env after you've bootstrapped [16:49] fwereade_: and that will affect your bootstrap node? [16:49] bac, no, it won't [16:50] yeah, that would be scary. [16:50] bac, it'll affect what charm series is inferred when yu deploy, and what series machines added without --series get, but that should beit [16:50] ok, thanks [16:50] bac, cheers [16:52] sinzui, I have a community update on juju to stream today, looking at the milestone for 1.16.5 it's 1 bug away, is it safe for me to say that we'll have .5 out before the holidays? [16:52] NO way [16:53] jcastro, I reported that bug weeks ago and no one is working on it. I think 1.16.5 can never happen because it breaks cli compatability [16:54] ok so I can just reiterate that .4 is where it's at? [16:55] I think we should hope for 1.17 this month with 1.18 in January [16:59] damn it. abentley there is an extra leading / in the azure release files that is not in the testing files. All the files were pushed to the wrong location [16:59] * sinzui blows a gasket. [16:59] Crap! === teknico_ is now known as teknico [17:11] abentley, I am pushing the files to the correct location. We can talk while everything goes up [17:14] rogpeppe: you'll get a new push of my CL tomorrow, I'm now changing it to reading from the end first [17:14] TheMue: cool [17:14] TheMue: you got my email? [17:15] rogpeppe: oh, eh, yes (right now, didn't looked into mail before) :D [17:16] sinzui: relevant? http://162.213.35.54:8080/job/azure-upgrade-deploy/71/console [17:16] TheMue: quite fun that i was still able to find the code that i remembered... [17:16] rogpeppe: great, my approach looks initially simpler, but with less error control and less generic [17:17] rogpeppe: will combine both [17:17] rogpeppe: but have to step out now, visitors [17:17] TheMue: ok, see ya [17:18] abentley, yes, in fact, it answers what is in my head, [17:23] sinzui, all the juju-core 1.16.4 packages are built in the juju-packagers PPA btw [17:24] jamespage, I saw thanks [18:44] * rogpeppe is done for the day [18:44]