menn0_grrr functional-ha-recovery still fails00:46
* menn0_ rolls up his sleeves00:46
menn0_at least mongo didn't go down in a ball of flames00:51
menn0_it looks like the test perhaps didn't wait quite long enough00:52
davecheney\o/ add more timeouts00:52
menn0_hmmm.... it's already waiting up to 20mins00:57
menn0_20mins to bring up 2 state servers00:57
menn0_should be enough00:58
davecheneyyup, that is far too long00:58
menn0_davecheney: something definitely went wrong02:02
menn0_there are signs that the extra state servers where on their way up02:03
menn0_but they didn't get there02:03
menn0_and the logs just stop well before the test gave up02:03
menn0_in fact, the logs are a bit sparse throughout02:03
menn0_I don't see the EnsureAvailability API call for example02:03
menn0_but I do see messages from the provisioner that the extra machines came up as a result02:04
davecheneymenn0_: what I was seeing last night, even when mongo wasn't fucked is the _second_ we issue ensure-availabilityu02:04
davecheneythe primary api server goes offline02:04
davecheneynot stops02:04
davecheneyjust stops accepting new connections02:04
davecheneyso the two new state servers get through cloud init02:04
davecheneywe start their jujud upstart jobs02:04
menn0_I saw that yesterday too but this is different02:05
davecheneyand thye just sit their waiting for the api server02:05
menn0_there are no errors or warnings in the logs02:05
davecheneythey don't evne get to the point of establishing a replica set02:05
menn0_everything looks positively happy02:05
davecheneythye don't even have mongo installed02:05
davecheneyits like turning on journalling buggers the primary state server02:05
davecheneyand thus the api server connected to it02:05
menn0_the other thing is, I've manually run this CI test against EC2 and it worked02:05
menn0_so it could be that it'll work sometimes02:06
menn0_Curtis sent me the details for logging in to our CI infrastructure02:06
menn0_I might run the test in Jenkins again and get on the hosts that get created02:07
menn0_20 mins is hopefully enough time to nose around02:07
davecheneymenn0_: so you got some working ec2 creds ?02:11
menn0_John set me up02:11
menn0_there's one canonical funded account02:11
davecheneygiven how many ne starters we have, it might be valuable to email the internal list and let them know how to get the details02:11
davecheneyrather than everyone figuring it out on their own02:11
menn0_and he can create user accounts within that02:11
jcw4waigani: thanks for the review comments03:09
waiganimenn0_ what's the latest?04:12
menn0_waigani: well, the problem doesn't happen very often04:12
menn0_I've only seen the functional-ha-recovery jenkins job fail once out of 4 attempts04:13
waiganiah a heisenbug04:13
menn0_and the one time it failed the logs don't indicate anything04:13
menn0_the one suspicious thing is that the logs stop dead several minutes before the test gave up04:13
menn0_but no errors or warnings before then04:14
menn0_I'm working on a little script which slurps all the useful logs from all the machines as the test is running04:14
waiganiany idea what command / test is run just before blackout?04:14
menn0_I have a theory that something went wrong with rsyslog (which feeds all-machines.log, which is the one we have for the failed run)04:15
menn0_but that the machine-N.log may have continued on04:15
waiganihave you checked the machine-n.log?04:15
menn0_yes sure, but only for runs where the test succeeds :)04:16
menn0_for the failed one where I wasn't looking as closely04:16
menn0_Jenkins only archived all-machines.log04:16
menn0_from machine-004:16
waiganiah right04:16
menn0_and that's the one that seems truncated04:16
menn0_I might make a change to the test so that it archives all the logs from all the machines04:17
menn0_that's probably more sensible than what I'm doing now04:17
waiganiit's not already doing that? what is it doing now?04:17
menn0_it just archives all-machines.log and cloud-init.log from machine-004:17
menn0_which is perfectly sensible if logging to all-machines.log is working correctly04:18
waiganiright, i see04:18
menn0_but I have a feeling it isn't (or at least wasn't for the failed run)04:18
menn0_I think having all the logs for all machines when tests fail is probably a good thing to have at this stage04:19
menn0_the test just ran successfully again...04:19
menn0_which is great04:19
menn0_and certainly better than what we had before04:19
menn0_but it bugs me that we still might have a lurking issue04:20
menn0_emailing curtis with an update now04:21
=== HankM00dy is now known as thehe
=== allenap_ is now known as allenap
=== viperZ28_ is now known as viperZ28
=== uru_ is now known as urulama
=== stub` is now known as stub
dimiternmorning all06:16
voidspacemorning all06:37
dimiternhey voidspace06:53
voidspacedimitern: hwy06:53
voidspace*hey even06:53
dimiternvoidspace, i'll be running tests on MAAS as soon as the talk starting now is done (~1h from now)06:54
voidspacedimitern: cool06:54
dimiternjam, jam1, hey, I'm in juju-networking06:59
jam1dimitern: hi, we're not done with the lightning talks, but we'll be switching to the boardroom soon07:02
dimiternjam1, ah, ok07:02
* voidspace lurches08:48
TheMuevoidspace: morning, just opened your latest PR09:03
jam1TheMue: voidspace:09:13
jam1hey guys09:13
jam1how's IPv6 stuff looking?09:13
TheMuejam1: I’m in contact with Serge and Stéphane regarding LXC. Got a mail back with lots of data in it. ;) There seems to be an issue with the bridge for the containers, but I have to go deeper into the pasted command output in the mail.09:16
jam1TheMue: as in 'br0' doesn't actually support ipv6?09:16
TheMuejam1: Will do it after finishing voidspaces review.09:16
TheMuejam1: That could be the worst answer, yes. Don’t hope so.09:17
jam1TheMue: certainly I expect that the containers can't just come up on lxcbr0 as they default to09:17
jam1because that isn't actually a bridge onto the outer network09:17
lifelessjam1: allocating routable ipv6 will need a daemon offering address space to the bridge (and you can then route it onto the exterior net09:21
jam1lifeless: in this case, I believe we are just setting the addresses on the individual containers, the bridge itself needs to know the range?09:24
lifelessjam1: same as dnsmasq is used to offer IP addresses to containers for ipv4, you need something offering it for ipv609:26
lifelessjam1: autoconfig will get you local address space only of course09:26
jam1lifeless: sure, but you can set them with ip manually09:27
TheMuejam1, lifeless: yes, currently all addresses on host and in the containers are set manually09:28
jam1anyway, I have another meeting now, but I'd like to hear your thoughts lifeless, perhaps in about an hour?09:28
lifelessjam1: I'm at a python user group meetup, but I'll try to catch you later perhaps :)09:29
TheMueand as said, host A can reach host B, host A can reach container A1 and A2, container A1 and A2 can reach, only A1 cannot reach B109:29
jam1lifeless: sounds like a good time, have fun09:31
lifelessTheMue: is there an etherpad or a bug with details?09:32
TheMuelifeless: not yet, so far we haven’t seen it as bug, only as a result of my lack of knowledge about ipv6 :D09:33
lifelessTheMue: ok, so I'm happy to cross-check, if you have a diagram (or full prose) description of the topology (and ip -6 route output etc from the hosts and containers)09:35
TheMuelifeless: thanks, will add this to my „research doc“ on google and send you the link09:36
dimiternvoidspace, still in a call; upgrading the maas nucs, it takes some time - when done i'm running tests09:37
lifelessTheMue: cool09:57
waiganiI'm going to be late to hangout - my plugin has crashed, trying to sort it09:58
natefinchanyone getting on the juju-core team meeting?10:00
TheMuenatefinch: oh, goooood hint *iirks*10:01
=== perrito6` is now known as perrito666
voidspacedimitern: cool, I have some review comments from TheMue anyway10:14
voidspaceTheMue: thanks for the review, useful10:14
TheMuevoidspace: yw and just for info, it’s core meeting time10:16
voidspaceTheMue: ah! I always forget core meeting10:17
voidspacegrabbing coffee and will join10:17
TheMuevoidspace: hehe, me this time almost too10:17
natefinchsinzui: are you around?  CI might be unblocked10:27
natefinchmorning alexisb10:28
mattywdavecheney, http://i.imgur.com/TkWPd9o.jpg10:54
voidspaceTheMue: dimitern: I assume we're not doing standup as well?10:59
dimiternvoidspace, TheMue, let's skip it yeah, unless you need to talk about something specifically?11:00
voidspacedimitern: no, you both know where I'm at11:00
voidspacedimitern: I'd like confirmation from you at some point that my branch doesn't screw MAAS11:00
dimiternvoidspace, certainly, I'm trying my best to get my local maas in a usable state - almost there i hope11:01
voidspacedimitern: haha, ok11:01
voidspacedimitern: I have minor cleanups to do on that branch anyway11:01
voidspacedimitern: so I'm not blocked11:01
dimiternvoidspace, sweet!11:02
waiganigit config --global rerere.enabled true11:02
waiganithat's all I need to do?11:02
waiganiif so, we should add that to the contributing doc - just before the instructions to rebase11:02
dimiternwaigani, check that blog post "Rerere your boat"11:02
waiganidimitern: okay, I'll keep reading11:03
c7zma, that's a terrible blog title pun11:06
waiganimattyw: thanks for the git/D3 link, great idea11:11
mattywwaigani, I use 2 resources for understanding git. that's the first one - the other one is tasdomas11:12
waiganimattyw: hehe, I'll have to bookmark that second one11:12
waiganinight all.11:14
wwitzel3natefinch: you said CI might be unblocked?11:35
sinzuisorry natefinch  https://bugs.launchpad.net/juju-core/+bug/1350983 is still open. While we got a pass, it failed most of the time we tried11:46
c7zsinzui: well, I have another plan on that one too11:54
c7zsinzui: can you confirm it's just bug 1350983 and bug 1347715 left blocking?11:55
sinzuic7z, yes, just them11:55
sinzuic7z, there is another critical regression reported by voidspace, We may need to add "ci" to it to make it block checkout bug 135344311:56
c7zI'm pretty certain the azure issue is slow disk, but not sure best how to mitigate, or the direct cause (as it's not that)11:56
c7zsinzui: devs seem blocked on the manual provider issue from not being able to reproduce it... but ci does hit it completely reliably11:58
sinzuic7z, yep. I since 1.20 always passed, and it passed before the problem revisions, I cannot fault the test.11:59
=== ChanServ changed the topic of #juju-dev to: https://juju.ubuntu.com | On-call reviewer: see calendar | Open critical bugs 1350983, 1347715, 1353443
katcomgz: standup?12:06
voidspaceTheMue: which do you prefer, the first or the second?12:09
voidspaceTheMue: http://pastebin.ubuntu.com/7979232/12:09
perrito666sinzui: hey, can you give me a hand with azure?12:09
voidspaceTheMue: I like the first as you then have a type for the function parameter12:09
voidspaceI have to reboot12:10
voidspacescrewy driver kills the mouse from time to time - so I currently have no mouse pointer12:10
sinzuiperrito666, I have a few minutes between meetings12:11
perrito666ok Ill be fast12:11
perrito666I copied one of the setups for azure from cloud city12:11
perrito666I set up the env variables from azuretoolssrc12:11
c7zkatco: sorry, sec12:12
c7ztimezone is wrong here somehow12:12
perrito666and I tried a few combinations and I always end with:12:12
perrito6662014-08-06 22:12:26 ERROR juju.provider.common bootstrap.go:120 bootstrap failed: waited for 10m0s without being able to connect: Permission denied (publickey).12:12
voidspaceaaand back. But I have to help the wife for a few minutes. Back for realsies shortly.12:22
wwitzel3realsies .. heh12:23
voidspacewwitzel3: morning12:23
wwitzel3voidspace: hola, how are you?12:23
voidspacewwitzel3: I'm good, but I have to go12:23
voidspaceback in a few12:23
TheMuevoidspace: back from lunch, just seen your question. I prefer the first approach too. but is it do and don’t or can and cannot?12:24
jam2voidspace: I'm pretty sure that while you have a type for the parameter, you can still pass "myFunc(true)"12:32
jam2voidspace: http://play.golang.org/p/q4ttwlIC4_12:35
=== jam2 is now known as jam1
natefinchjam2: yeah, that's right.  "true" and "false" are constants and thus get converted to whatever special type you want that is derived from bool12:35
perrito666sinzui: let me know if you can shed some light on my issue12:35
jam1natefinch: which is also true for enumerate strings, and lets you create an "instance" of your enumerated type that has any arbitrary string12:35
natefinchjam1: yep, it's kind of a problem at times... it's not too bad for enums based on ints, because randomly passing 8 or whatever into a function looks weird and should get caught by a review12:36
natefinchjam1, voidspace:  I'm actually -1 on making constants that just mean true or false.  If you want to make the code clearer, don't use a boolean parameter at all12:37
natefinchjam1, voidspace: just make two functions:  Foo() and FooNoConfig()    and12:38
natefinchanother option is to use an integer instead of true/false  just because it only has two values doesn't mean it has to be a boolean... no one's going to pass 0 or 1 into your function and make it past a code review (hopefully)12:39
natefinchbut really, the best answer is just two functions that in their implementation pass true/false to a single implementation function12:40
natefinchok gotta go.  Bringing my older daughter to a doctor's appointment12:40
TheMuenatefinch: sounds good, and you later simply can stuff like „iDontCare“ or „whoKnows“ ;)12:40
TheMuenatefinch: but I would always start with 1, so that a passed uninitialized variabe with value would may fail internally, e.g. inside a switch.12:41
jam1TheMue: voidspace: I'd also *highly* recommend affirmative statements, rather than something like "if !DontFoo"12:48
TheMuejam1: yep, if statements should be „if isWanted“ or „isPossible“ while the arguments are „want“, „dontWant“, „possible“ or „impossible“12:52
TheMuejam1: but only for boolean variables, fields, arguments12:53
TheMuejam1: otherwise the prefixes „is“,“can“ etc don’t match12:53
* TheMue is reminded of the always good readability of smalltalk sources12:54
dimiternvoidspace, hey13:01
dimiternvoidspace, sorry for the long delay13:01
dimiternvoidspace, i had a ~4h fight with my hardware maas, eventually giving up and testing your branch on my kvm-based virtual maas13:02
dimiternvoidspace, network setup looks fine, containers on all nodes, bootstrap included, are addressable on the same subnet (the bridge works)|13:13
katcowwitzel3: did a dumb script: #!/bin/bash\nPASSWORD=$(sudo cat $HOME/.juju/local/agents/machine-0/agent.conf |grep oldpassword |awk '{ print $2 }')\nmongo --ssl -u admin -p $PASSWORD localhost:37017/admin13:14
voidspacedimitern: awesome, that's great news13:16
wwitzel3katco: nice13:16
dimiternvoidspace, i've just reviewed your PR, LGTM13:16
voidspacedimitern: cool, thanks13:16
katcowwitzel3: seems like everyone asks how to do that13:16
voidspaceTheMue: nate's advice doesn't really make sense in this context13:17
voidspaceTheMue: we'd actually need two networkers and two configstate types to do as he suggests13:17
dimiternkatco, ha, awk fan, huh :)13:17
voidspaceTheMue: sometimes a boolean actually makes sense, I disagree with him on that point I think13:17
katcodimitern: lol i am by no means an awk master. i just struggle through it when it makes sense13:18
voidspaceTheMue: if go had named parameters it would be easier...13:18
TheMuevoidspace: now, not two networkers. two constructors to avoid the parameter13:18
TheMuevoidspace: yeah, named parameters are nice13:18
voidspaceTheMue: but the networker has to pass this paramter down into another function13:18
voidspaceTheMue: so two constructors isn't enough...13:18
voidspacewe still need to store the boolean13:18
dimiternvoidspace, it has a struct-literal syntax for kinda the same thing13:18
voidspacedimitern: heh, right - could use that I guess13:19
voidspacedimitern: that's even further down the rabbit hole13:19
dimiternvoidspace, we use this quite a lot with 3-4+ args functions13:19
voidspacedimitern: right, and there it makes sense13:19
TheMuevoidspace: the default constructor would set a field to the one value, while the second one calls the first one but then changes the field. ;)13:19
voidspacedimitern: becuase you can add or remove paramters at will too13:19
voidspaceTheMue: I don't think that's any clearer than just a boolean parameter with named constants13:20
voidspacethat's plenty readable13:20
dimiternvoidspace, I'd just define a couple of int bit flags and | them when calling NewNetworker - everyone's happy13:20
voidspacedimitern: bit flags!13:21
dimitern:) why not? networker.DontWriteConfig13:22
dimiternvoidspace, ah, sorry - i've just noticed that's the only bool arg13:22
dimiternif there were 2, i'd use bit|flags13:23
wwitzel3where is this code in question?13:23
voidspacedimitern: you mean bit flags instead of multiple bools13:23
voidspaceyeah, that would work...13:24
jam1katco: wwitzel3: fwiw, I think the "admin" user is going to be going away as direct DB access eventually, you should be trying to connect as machine-0 user, IIRC13:24
wwitzel3jam1: will the machine-0 user have the rights?13:26
dimiternvoidspace, In this case, the easiest thing really is another ctor NewSafeNetworker? { newNetworker(..,false) }, and NewNetworker also calls the implementation, but with true13:26
katcojam1: thanks for the heads up.13:26
voidspaceI give in13:26
jam1wwitzel3: a machine agent that has JobManageEnviron will have admin access on the DB13:27
jam1in HA mode, eventually machine-0 might not, but the chances of that are quite low13:27
lifelessjam1: so hi13:27
jam1I'll bring it up if we *actually* want to kill the admin user, but already the "oldpassword" stuff means it isn't quite what it used to be13:27
jam1hey lifeless13:27
* dimitern wonders how many *degrees* of bikeshedding are there :D13:27
wwitzel3jam1: ok, just wondering, I know that the issue i just fixed I wouldn't of been able to without access to the admin database.13:27
dimiternvoidspace, interestingly, i came across some problems with deploying in lxc containers - slow startup (not using btrfs?), apt-get install failing the same way (can't get the dpkg lock) - resolved --retry fixes it, mysql start hook fails with random mysql startup errors13:38
voidspacedimitern: that all sounds horrible13:38
voidspaceit never used to be so bad13:39
dimiternso maybe the lxc package had some regressive changes lately13:39
voidspaceright, maybe13:39
dimiternvoidspace, tell me about it :) combined with a couple of hours trying to get the master nuc on my maas to work (at one point no usb ports worked, i.e. no kbd, no wifi or ethernet)13:40
TheMuelifeless: in https://docs.google.com/a/canonical.com/document/d/1wfdGL_vyemT2-ncAB7KIySkKI9HbT8efKeC3Sd8ID0I/edit# the current test setup and status are described13:42
jam1TheMue: so that doc seems to say that they are both just on "lxcbr0" which is only the local host bridge13:43
TheMuejam1: yes13:43
jam1TheMue: you need a bridge that is on eth0 if you want containers on VM1 to be able to see anything on VM213:43
dimiternTheMue, that's right, and it needs to be an IPv6 bridge I think13:45
TheMuejam1: I’ve tried that too, as I found a a doc. but after shredding my net this way in the first approach even the second one didn’t work13:45
wwitzel3perrito666, ericsnow: you guys want to wait push standup back and wait for nate? Or do i at 10?13:46
TheMuejam1: so how to add an eth0 bridge for ipv6?13:46
jam1TheMue: my initial understanding is that installing "bridge-utils" creates a br0 bridge on eth013:46
jam1TheMue: though possibly you need: http://xmodulo.com/2013/04/how-to-configure-linux-bridge-interface.html13:47
perrito666wwitzel3: I guess we can wait13:47
TheMuejam1: thanks for the link, will take a look13:47
dimiternTheMue, this one is specifically for lxc+ipv6 - might help: http://blog.toxa.de/archives/60613:50
TheMuejam1: so am I right that eth0 and the virtual interfaces of the containers have to be added to the bridge13:50
TheMuedimitern: see my link list13:50
TheMuedimitern: in the document13:50
dimiternTheMue, :)13:50
dimiternhaven't checked all of them13:50
TheMuedimitern: done it exectly this way made me restoring my VM to my snapshot, can’t reach it anymore *lol*13:51
* TheMue will see how the bridge utils work now13:52
lifelessTheMue: have requested access14:00
TheMuelifeless: granted14:02
lifelessTheMue: comments left14:35
lifelessTheMue: ip -6 neigh show14:35
lifelessTheMue: is another useful command14:35
c7zwhat horseplay is that...14:35
lifelessTheMue: in short, I think your subnetting is broken - you're putting /64 prefixed addresses on a virtual bridge with a /96 route, we don't expect neighbour discovery outside that /96 to work14:36
TheMuelifeless: ah, ok, will take a look there, thanks14:37
TheMuecurrently my vms are rejecting any networking after enslaving eth0 to br0 :(14:38
perrito666whyyyyy cant I break this/14:44
sinzuiperrito666, azure is not seeing the io timeout?14:53
perrito666sinzui: nope, I am bootstrapping succesfully14:53
perrito666sinzui: I did see the io error in aws a couple of days ago14:54
perrito666that intrigued me14:54
sinzuiperrito666, we are testing 1.20 for a ppc fix now.15:01
sinzuiperrito666, jam speculates that this error is older, really just a replicate set issue and that the error we now see is just a mutation15:02
perrito666you mean the io timeout?15:02
sinzuiperrito666, yes15:02
perrito666do you have any context on that speculation/15:03
jcw4rick_h__: ping15:17
jam1bac: https://gist.github.com/jameinel/d0763eb6d8d38cfd64e115:21
=== psivaa is now known as psivaa-afk-bbl
perrito666sinzui: ok, out of 6 i only got one failure and it was dns related :|15:38
sinzuiperrito666, I am going to force a rebuild of master when 1.20 finishes the test15:43
sinzuiperrito666, did you publish your own streams? did you use --upload-tools?15:54
perrito666I used upload tools I have my streams published if you want me to try with it15:55
sinzuiperrito666, no need. I am deploying too while I wait for CI to come free15:56
perrito666too late, it was at the tip of my fingers15:57
perrito666sinzui: with my own stream also works16:14
sinzuiperrito666, That last built binary worked for me too?16:15
perrito666sinzui: that is a very hard question for me to answer man :p16:15
sinzuiperrito666, let's just stop testing for now. master will rebuild in about an hour16:15
sinzuiperrito666, s/?/./ It did work for me16:16
rick_h__jcw4: pong?19:29
jcw4Hi rick_h__ ; sorry - I don't think you saw my note in IRC a couple days ago?19:33
jcw4about actions api docs19:33
rick_h__jcw4: sorry, saw something go by but been at a sprint out of the country and not peeked at it19:34
jcw4rick_h__: dropped the ball on that, but here is a WIP pr for documentation... https://github.com/juju/juju/pull/46819:34
rick_h__jcw4: ty much19:34
jcw4rick_h__: figured you really wanted that last week, not this week :-/19:34
jcw4rick_h__: hope you're having a great sprint though :)19:34
rick_h__wheeeee! :)19:34
voidspaceevening all19:36
jcw4voidspace: 'ello :)19:36
wwitzel3natefinch: ping20:09
natefinchwwitzel3: pong20:11
wwitzel3natefinch: so I wanted further pick your brain about using other syslog library? .. is that something I should be looking at for dealing with this all-machines.log issue? Or do we want to worry about that as a seperate concern?20:12
wwitzel3natefinch: I guess that only solve the aggregation on windows, it doesn't solve the rotation problem.20:14
natefinchwwitzel3: yeah20:14
wwitzel3natefinch: should I just use logrotate (simplest path) for now?20:15
natefinchwwitzel3: yep20:15
wwitzel3natefinch: well that makes it easy :)20:16
wwitzel3well .. I don't have to make any choices .. it probably will be a pain in the ass because it is rsyslog and logrotate20:16
natefinch yeah sorry20:16
wwitzel3but at least this part was easy :)20:16
wwitzel3also how in the world should I test this?20:17
wwitzel3... wait 1 day and make sure the log rotates .. our tests already take long enough :P20:17
natefinchthat wouldn't extend them all that much ;)20:19
wwitzel3natefinch: also do I need to request logrotate get added to some ppa/apt repo? cloud something or other?20:20
natefinchfor actual unit tests.... honestly, screw it, this is an external application.  test that we set the config right20:20
natefinchwwitzel3: it may already be installed20:21
natefinchwwitzel3: and if not, I expect it'll be in whatever thing is already available20:21
natefinchi.e. sudo apt-get install should work20:22
wwitzel3natefinch: sounds good, thanks. starting on it now, I'll add a card for it20:22
wwitzel3is there a lp bug for it too?20:22
natefinchyes....... somewhere20:22
wwitzel3I searched but lp never gives me any results when I search20:22
natefinchwwitzel3: https://bugs.launchpad.net/juju-core/+bug/107821320:24
wwitzel3thank you .. need to start a new site .. lngtfy.com (let nate google that for you)20:24
natefinchhaha... search didn't find it for me, so I opened up the high bugs.... and it happened to be at the top20:24
natefinchheh reported 21 months ago20:25
jcw4Fix for a bug I saw cropping up on the CI builds: https://github.com/juju/juju/pull/48023:58
jcw4tests assuming ordered results23:58
perrito666sinzui: any news?23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!