| wallyworld | vino_: a few small things, let me know if you have questions | 00:02 |
|---|---|---|
| wallyworld | veebers: not sure tbh, charm push may no know to expand the "~" | 00:03 |
| babbageclunk | wallyworld: ok, think I've found the problem, just checking now. | 00:06 |
| wallyworld | yay | 00:07 |
| vino_ | wallyworld: sure | 00:07 |
| babbageclunk | wallyworld: that seemed to work. Trying it again with a 3-machine controller | 00:12 |
| veebers | wallyworld: /tmp/blah didn't work, ~/tmp/blah did work (both are actual files). It's ok, not blocking me (suspect it's a snap/containment thing) | 00:12 |
| wallyworld | veebers: ah, i had it the wrong way around in my mind. that makes sense for a snap yeah | 00:16 |
| === `Calvin is now known as Calvin` | ||
| babbageclunk | wallyworld: yay, that worked too, checking one more scenario (2 machines in the controller, so one non-voter) | 00:28 |
| wallyworld | babbageclunk: i'll be interested to see what fix was | 00:29 |
| babbageclunk | It was in the upgrade step - I didn't defer-close the log store, so the raft worker just hung on startup. In the past that had always been caused by the transport not getting an address, so I was looking in the wrong place, but the logging I added showed it wasn't the problem. | 00:31 |
| babbageclunk | wallyworld: ^ | 00:31 |
| wallyworld | ah, cool | 00:32 |
| wallyworld | kelvin: i left a couple of small comments | 00:32 |
| kelvin | wallyworld, thanks for reviewing, i just had a push to resolve ur comments. | 00:34 |
| wallyworld | ok | 00:34 |
| wallyworld | kelvin: looks good to land! on to next step | 00:35 |
| kelvin | wallyworld, thanks, merging now | 00:36 |
| babbageclunk | wallyworld: actually, my idea of testing a 2-machine controller relies on stuff in 2.4, so it's not necessary. Tidying up and pushing a PR now and then I'll do the unit tests. | 00:38 |
| wallyworld | sgtm | 00:39 |
| babbageclunk | (Oh, and I'll check bootstrapping 2.4 directly is still fine.) | 00:39 |
| vino_ | wallyworld: i am using IsolationSuite from github.com/juju/testing instead of BaseSuite in juju/juju/testing. | 01:03 |
| wallyworld | that sounds fine | 01:03 |
| vino_ | it introduced few issues in reading the EchoArgs. Trying to fix it. | 01:03 |
| vino_ | rest all comments make sense. | 01:03 |
| wallyworld | you could just leave the base suite out | 01:03 |
| wallyworld | i don't think any of the other charm tests use it | 01:04 |
| vino_ | yes correct. And that will fix the CI issue i was facing. | 01:04 |
| wallyworld | so let's drop it then | 01:04 |
| vino_ | Yes. charmDir_test uses IsolationSuite | 01:04 |
| vino_ | its req for the EnvPatching | 01:04 |
| vino_ | which is a good substitute for BaseSuite | 01:04 |
| vino_ | Just hold on - using IsolationSuite has introduced a path error. Which i need to identify. | 01:06 |
| vino_ | swapping to BaseSuite works well. SO there is something diff between them. I will fix that and push the commit. | 01:07 |
| babbageclunk | wallyworld: https://github.com/juju/juju/pull/8850 | 01:40 |
| wallyworld | ok | 01:40 |
| babbageclunk | Sorry, laptop died at a super-inopportune moment, but all better now | 01:40 |
| wallyworld | babbageclunk: looks ok i think. for 2.5 i think we could consider moving the common raft business logic out of the worker and into a core/raft package that the upgrader also uses | 01:48 |
| babbageclunk | wallyworld: yeah, sounds good | 01:49 |
| wallyworld | vino_: using IsolationSuite should be all you need as that provides the PatchEnvironment functionality. Or you could just use CleanupSuite | 01:52 |
| babbageclunk | wallyworld: do you think the ReplicaSetMembers function in upgrades.go needs a test? | 01:54 |
| vino_ | wallyworld: yes. IsoaltionSuite gave me trouble. I found out this CleanupSuite which is good. | 01:54 |
| vino_ | no Pathherror. | 01:54 |
| wallyworld | babbageclunk: not for this PR | 01:55 |
| vino_ | iam pushing the commit now. | 01:55 |
| babbageclunk | cool | 01:55 |
| vino_ | wallyworld: i haven't addressed one review comment - testing.ReadArgs() | 02:25 |
| vino_ | happy to discuss. | 02:25 |
| wallyworld | ok, i'll wait for that | 02:25 |
| wallyworld | did you ned help with it? | 02:26 |
| vino_ | not really for that. | 02:31 |
| vino_ | I am seeing a weird CI issue. for this NOVCS case. | 02:31 |
| vino_ | i dont see this ssue locally. | 02:31 |
| vino_ | It shd be nil returned from function. | 02:32 |
| vino_ | i am trying to look into the CI issue here. | 02:32 |
| wallyworld | vino_: it's jus a typo | 02:33 |
| wallyworld | c.Assert(err, gc.NotNil) | 02:33 |
| wallyworld | should be c.Assert(err, jc, ErroIsNil) | 02:34 |
| wallyworld | vino_: actually, all the other gc.IsNil for error checking added to the pr should also be c.Assert(err, jc.ErrorIsNil) | 02:35 |
| vino_ | ok. thank u. I was confused that i cudntnt see that issue Locally in my machine. | 02:37 |
| vino_ | thanks wallyworld. | 02:37 |
| wallyworld | i'm sureprised by that - it should have failed locally | 02:37 |
| vino_ | i can show u. | 02:37 |
| wallyworld | the logic error was checking for a not nil error when it would have been nil | 02:37 |
| vino_ | i do see here all 185 OK | 02:38 |
| wallyworld | right but the issue is that the error should not have beenj returned as not nil | 02:38 |
| wallyworld | the new function should just return nil if no vcs dir exists | 02:39 |
| vino_ | Yes. I am checking that if not nil then assert | 02:39 |
| vino_ | i am explicitly returning NIL if NOVCS | 02:39 |
| vino_ | The new function does exactly the same as u r telling. | 02:39 |
| wallyworld | right but the PR was checkinf that the error was not nil | 02:39 |
| wallyworld | line 300 was c.Assert(err, gc.NotNil) | 02:40 |
| wallyworld | this says that we expect a non-nil error | 02:40 |
| wallyworld | but we want the error to be nil | 02:40 |
| wallyworld | if that code passes when you run it locally i'm confused as to how | 02:41 |
| vino_ | the new function does the correct thing. But i am check here is wrong. | 02:41 |
| vino_ | is that what u mean - correct ? | 02:41 |
| wallyworld | yes, if the PR says c.Assert(err, gc.NotNil) then that's wrong | 02:41 |
| wallyworld | and hence the CI failure | 02:42 |
| vino_ | I get what u r saying. | 02:44 |
| vino_ | but i you telling all has to be changed. How come it is passing for me locally here.! | 02:44 |
| wallyworld | that's a different issue | 02:46 |
| wallyworld | what needs to be changed is the syntax for error checking. gc.IsNil is not how we do it as it sometimes can give wrong results if there's a pointer | 02:46 |
| wallyworld | c.Assert(err, jc.ErrorIsNil) is what we use | 02:47 |
| wallyworld | i didn't notice the use of c.Assert(err, gc.IsNil) before | 02:47 |
| wallyworld | so there's 2 issues: 1. fix the incorrect assetion to make test pass, 2. replace gc.IsNil to make code more correct | 02:48 |
| vino_ | ok sure wallyworld. | 03:00 |
| vino_ | thumper: 1:1 ? | 03:01 |
| veebers | wallyworld: It seems (in the code) that one can provide a 'revision' as a resource arg, but I don't see mention of that in the help etc. Does that ring a bell at all? | 03:14 |
| wallyworld | vino_: tim is away sick today | 03:21 |
| wallyworld | veebers: resources have revisions - each time one is updated the revsion increments | 03:21 |
| wallyworld | there's a published tuple of charn rev,resource rev | 03:22 |
| wallyworld | that's what is used by default | 03:22 |
| veebers | wallyworld: ack, understand that; it's the "juju deploy --resource <rev>" that I was confirming, i.e. use x revision of the resource. | 03:22 |
| wallyworld | right | 03:22 |
| veebers | there doesn't seem to be any docs around that (at least not in deploy help) | 03:22 |
| wallyworld | if you wanted to use a rev that's not published | 03:22 |
| wallyworld | could be the case, haven't read the docs | 03:23 |
| veebers | ack, no worries. Work needed on the helps docs there :-) | 03:23 |
| vino_ | ok. Thanks wallyworld | 03:26 |
| wallyworld | babbageclunk: how goes the unit test writing? | 03:56 |
| babbageclunk | almost done - just need to fix logging | 03:56 |
| veebers | wallyworld: with the upcoming docker resource type, if the user supplies the resource as an arg to deploy, will we (somehow) allow them to supply secrets details too? | 05:25 |
| wallyworld | no, these will come from the charm store | 05:26 |
| wallyworld | the charm storage manages all that stuff, as it does with macaroons for private charms etc | 05:27 |
| veebers | sweet, ah right you are | 05:27 |
| vino | wallyworld: can u take a look at the Pr | 05:38 |
| vino | when u get time | 05:38 |
| wallyworld | ok | 05:38 |
| adham | Hi everyone | 05:39 |
| wallyworld | vino: have you pushed chmages? | 05:40 |
| vino | yes. | 05:40 |
| adham | Because of this issue https://stackoverflow.com/questions/50970133/installed-kubernetes-on-ubuntu-and-i-see-a-lot-of-nodes-are-getting-created-in-m | 05:40 |
| adham | I deleted the VMs manually | 05:40 |
| adham | I didn't know that this would affect kubernetes deployment | 05:41 |
| vino | arent u seeing the changes yet wallyworld | 05:41 |
| vino | ? | 05:41 |
| adham | Is there a way that I can remove kubernetes deployments completely manually since conjure-down or up won't work? | 05:41 |
| wallyworld | i can see some but not all? there's still var version and I can't see ReadEchoArgs | 05:41 |
| vino | u wanted to discuss on the ReadArgs() call | 05:41 |
| adham | When I run juju list-models --debug, I see that it still connects to a vm that has been already deleted | 05:42 |
| vino | yes.what it reads after that is the empty string. | 05:42 |
| vino | we can discuss on that. | 05:42 |
| vino | once it is read it resets the ptr. | 05:43 |
| vino | thats why we cant use the ReadArgs again. | 05:43 |
| wallyworld | adham: deleting the vms directly in k8s will leave orphaned entries in the juju model. you could try remove-machinbe --force | 05:45 |
| wallyworld | vino: ok. so intead of the expectedVersion slice in the PR, it's cleaner to do what lines 167-170 of EachAsArgs() does | 05:47 |
| wallyworld | to build up the string to compare withthe version file contesnts | 05:48 |
| adham | how can I clean up at this point? @wallyworld: I had to do this if you saw my question in StackOverFlow, I have observed a huge flood of VMs once I deployed Kubernetes | 05:48 |
| adham | and that was a clean installation | 05:48 |
| wallyworld | what do you want to clean? the ensire controller? just the model? | 05:48 |
| adham | So I have MAAS installed on the same server | 05:49 |
| adham | I do not want to affect MAAS | 05:49 |
| adham | I just want to cleanup the kubernetes and then reinstall it again (but after I know why those 70+ vms got generated) | 05:49 |
| wallyworld | i've not used conjure-up to deploy k8s so i can't answer that question - the conjire up guys will be online in a few hours | 05:50 |
| adham | Ok, I can wait for the conjure team, but in the meanwhile, have you ever seen a list of VMs like this in the stack over flow? | 05:51 |
| wallyworld | if you want to keep the existing juju controller running, i think you will need to remove-machine --force all the machines for which you have manually deleted the k8s nodes | 05:51 |
| adham | I am referring to the link to my question there... | 05:51 |
| vino | wallyworld: yes. that code looks a bit professional. | 05:51 |
| adham | how do you mean Wallyworld? | 05:52 |
| adham | can you send me a sample command? | 05:52 |
| wallyworld | after you have deleted from juju the orphaned machines, you should hopwfuly be able to then delete the model | 05:52 |
| wallyworld | juju remove-machine --force <machineid> | 05:52 |
| adham | how can I get the machine id? | 05:52 |
| wallyworld | juju status | 05:52 |
| wallyworld | will show all the k8s nodes | 05:53 |
| wallyworld | from there you can see which ones correspond to the ones manually deleted | 05:53 |
| wallyworld | it's all a bit abstract without being totally familair with your setup | 05:53 |
| wallyworld | is the juju controller running on a maas node? | 05:54 |
| adham | thx wallyworld, i'm running juju status, it's taking too long | 05:54 |
| wallyworld | what version of juju? | 05:54 |
| wallyworld | if you wanted to blow everything away and start again, you could kill that one maas node and allocate it back to the maas pool, and manually elete the k8s cluster | 05:55 |
| wallyworld | you'd also need to reclaim any worker nodes | 05:56 |
| wallyworld | but i've not done what you're doing myself so can't give specific advice | 05:56 |
| wallyworld | adham: maybe you can get better help by asking in #conjure-up if it is a conjure-up issue | 05:58 |
| adham | Sorry wallyworld, just received the messages now, the version is 2.3.8-xenial-amd64 | 06:00 |
| vino | wallyworld push the commit. plz let me know. there was another minor in DEbug log comment. | 06:00 |
| adham | this server actually has the master maas controller | 06:01 |
| adham | also, the command for juju status seems to be still no response till now | 06:01 |
| adham | I will cancel it and re-run it with debug | 06:02 |
| adham | ahh, ok, it's connecting to API addresses (which has been deleted I'm assuming) | 06:02 |
| wallyworld | adham: juju status connects to the juju controller. i had thought you had only deleted k8s worker nodes? | 06:09 |
| adham | I deleted all of the extra VMs | 06:10 |
| adham | maybe I deleted that as well | 06:10 |
| adham | poof | 06:10 |
| wallyworld | vino: looks good! | 06:10 |
| adham | Wallyworld | 06:10 |
| adham | is htere a way that I can reverse everything about this deployment | 06:11 |
| adham | for the kubernetes? | 06:11 |
| vino | ok. i just pushed a last comment u gave :) | 06:11 |
| vino | i will land it now. | 06:11 |
| vino | wallyworld: in JUJU side i will update the dependencies.tsv first and then push the commit | 06:12 |
| wallyworld | adham: so if the juju controller is realy gone, then you probably just need to blow away the k8s cluster itself. i assume the worker nodes for that were all on maas? if so, and the juju controller node has already been deleted, you could just decommission those nodes in maas? | 06:12 |
| wallyworld | vino: you update the juju deps after landing the charm.v6 change | 06:13 |
| wallyworld | vino: and we need to make a small juju change also - we should trim the version string to say 100 before writing to the charm doc | 06:14 |
| wallyworld | just to be defensive | 06:14 |
| adham | how can I blow away the k8s cluster itself then? | 06:15 |
| wallyworld | sounds like you have already started to do that? if the k8s nodes are all in maas, and juju controller node has been decommissioned, then you could also decommission the k8s nodes as well | 06:16 |
| wallyworld | and return all those nodes to the maas pool | 06:16 |
| adham | In maas, I no longer see any traces for the k8s | 06:17 |
| adham | but not sure if i'm missing any area to look into | 06:17 |
| wallyworld | i *think* you may have managed to clean it all up then :=) | 06:18 |
| adham | then why juju status keeps being frozen, what do I do about this? | 06:19 |
| wallyworld | if juju controller is gone and all k8s nodes are gone then you should be ok | 06:19 |
| wallyworld | you need to remove the controller reference | 06:19 |
| wallyworld | from your local client | 06:19 |
| wallyworld | juju controllers | 06:19 |
| wallyworld | and then juju unregister <controllername> | 06:19 |
| adham | how can I be sure that this controller won't be a reference to MAAS itself? | 06:20 |
| adham | Because when I deployed k8s, i made it to use MAAS's cloud | 06:20 |
| wallyworld | maas is treated in juju as a cloud. you can see it via "juju clouds". the juju unregister command removes a controller entry from a local yaml file | 06:21 |
| adham | ahh, I see conjure-canonical-kubern-a4d | 06:21 |
| wallyworld | juju unregister conjure-canonical-kubern-a4d | 06:21 |
| wallyworld | should work | 06:21 |
| wallyworld | it just removes a yaml file entry | 06:21 |
| wallyworld | maas will still be running | 06:21 |
| wallyworld | and can be used again with conjureup | 06:21 |
| adham | $ juju unregister conjure-canonical-kubern-a4d | 06:22 |
| adham | ERROR controller conjure-canonical-kubern-a4d not found | 06:22 |
| adham | conjure-up-server-01-f34* | 06:22 |
| adham | is * part of the name? | 06:22 |
| wallyworld | the * indicates that's the current controller juju is using | 06:22 |
| wallyworld | but the names above differ | 06:23 |
| adham | the last one I mentioned with * is under Controller, the one before it is under model | 06:23 |
| wallyworld | what does juju controllers say? that's what you pass to unregister | 06:23 |
| adham | Doing so will prevent you from accessing this controller until you register it again. | 06:23 |
| adham | I am about to deregister it now | 06:24 |
| wallyworld | right, but here the controller machine has been shut down | 06:24 |
| wallyworld | you have removed the controller machine manually right? | 06:24 |
| adham | yes | 06:24 |
| adham | I guess so | 06:24 |
| adham | how can I ensure that? | 06:24 |
| wallyworld | check that there's no machine in maas with the listed ip address | 06:25 |
| wallyworld | but since status hangs and times out, it's a good bet it's gone | 06:25 |
| adham | no, there is no machine in maas | 06:25 |
| adham | If that's the case, I really want to redeploy K8s again | 06:26 |
| adham | but that flood of VMs doesn't make sense, and I don't know why they were created | 06:26 |
| adham | Do you have any idea about those machines and why possibly that they would created like that in https://stackoverflow.com/questions/50970133/installed-kubernetes-on-ubuntu-and-i-see-a-lot-of-nodes-are-getting-created-in-m ? I would really appreciate any knowledge around htis | 06:26 |
| wallyworld | so unregister will remove that orphaned entry | 06:26 |
| wallyworld | from juju client | 06:26 |
| wallyworld | adham: that's a question best asked in #conjure-up | 06:28 |
| wallyworld | i don't have any insight off the top of my head | 06:28 |
| wallyworld | lots of folks use conjure-up so if there is a bug it would be goos to get it fixed | 06:29 |
| adham | no, that's alright, I really appreciate your help here, it just made me progress, I was stuck for the last 4 days | 06:29 |
| adham | thx Wallyworld | 06:30 |
| wallyworld | adham: sorry i couldn't help more with the conjure-up side. i'm more a juju person | 06:30 |
| wallyworld | adham: you should be able to get the help you need in that other channel. if not come back here and we can chase them up :-) | 06:31 |
| adham | thx, will do! I'm in conjure-up channel | 06:32 |
| wallyworld | adham: they folkd there are mainly USA based so you may not catch them for a few hours yet | 06:34 |
| vino | wallyworld: we can chat here regarding the version string for charm. | 06:35 |
| adham | that's why | 06:35 |
| adham | that's fine* | 06:35 |
| adham | so where are you based? I'm in Australia | 06:35 |
| vino | so u want to keep 100 buffer before writing to the charm manifest | 06:35 |
| wallyworld | adham: brisbane | 06:37 |
| wallyworld | vino: not manifest. as we read in the charm from the zip, before we write to charmDoc | 06:38 |
| adham | lol | 06:38 |
| wallyworld | you too? | 06:38 |
| adham | neighbours :D | 06:38 |
| wallyworld | vino: since we don't always control what goes into charm zip | 06:39 |
| wallyworld | the best we can do is be careful about what we accept | 06:39 |
| vino | wallyworld: ok before we repackage and write to charmDoc | 06:44 |
| vino | So what we wriet is charmDoc needs to be sensible. | 06:45 |
| * vino make coffee and be back | 06:46 | |
| adham | Wallyworld, I know that this is off the topic and beyond your knowledge | 06:47 |
| adham | But I just wanted to double check with you if I have any luck | 06:47 |
| adham | when you bring up k8s environment, do you actually see funny names vms? | 06:47 |
| adham | Cuz I am conjuring up the k8s again and now I see casual-guinea, famous-jackal, etc.. Do you have similar VMs in your environment? | 06:48 |
| wallyworld | adham: those names are autogenerated. there's a "pet names" library that is sued. i can't recall the exact name | 06:49 |
| wallyworld | they take a random adjective and put with a random animal name | 06:50 |
| wallyworld | vino: correct | 06:50 |
| adham | Hmm, and do you have similar names in your environment? | 06:50 |
| adham | Is there away to have a proper naming convention? | 06:50 |
| adham | I think this is 100% pure conjure-up since I'm using conjure-up | 06:51 |
| wallyworld | adham: that's actually a maas naming conversion i think. ie the hostnames. juju doesn't care about or use hostnames as such | 06:52 |
| wallyworld | juju mainly uses machine numbers 1, 2, 3 etc that it generates | 06:52 |
| wallyworld | there might be a maas option to control hostname generation, not sure off hand | 06:53 |
| adham | ok that explains | 06:54 |
| adham | and would deploying k8s would deploy 70+ vms? | 06:54 |
| adham | I just conjured up with minimal installation no addons, and it's still "deploying", and yet I see 10 vms in maas? | 06:55 |
| adham | I mean 10 new vms | 06:55 |
| adham | 20 machines now | 06:56 |
| wallyworld | adham: there's kubernetes-core and CDK. which bundle did you choose? | 06:57 |
| adham | CDK | 06:58 |
| adham | Also, shall report the naming problem in Juju? I really recommend at least a friendly | 06:58 |
| wallyworld | i would expect to see several VMs for that as it is full HA so several redundant nodes for easyrsa etc | 06:58 |
| adham | name | 06:58 |
| wallyworld | which naming problem? the machine name generation in maas? | 06:59 |
| adham | I used to think that this is a virus or something | 06:59 |
| wallyworld | the maas node namig is a deliberate design decison to at least give the nodes unique english names | 06:59 |
| adham | Yes, I understood that part, I meant to put a RFE in juju asking for when having juju to create a machine, is to provide an english machine name at the beginning | 07:01 |
| adham | i.e controller_1 | 07:01 |
| adham | lb_1 | 07:01 |
| adham | etc...? | 07:01 |
| wallyworld | adham: juju let's ypu specify whatever controller name you want - conjure up is what's generatig the names you see | 07:02 |
| adham | ahh, ok, then this RFE would go to conjure-up | 07:02 |
| adham | quiet tricky when you have multple projects and many teams | 07:03 |
| wallyworld | for the controller name and model names | 07:03 |
| wallyworld | there may be a way to tell conjure-up, not sure | 07:03 |
| adham | thx Wallyworld, i've redirected this to conjure-up team, I am waiting for them to be available | 07:17 |
| wallyworld | np, good luvk | 07:18 |
| manadart | stickupkid: Took a look at your patch. Looks good. I would make 2 changes. | 08:03 |
| manadart | 1) Put the mock that is currently in environs/mocks into environs/testing. | 08:03 |
| manadart | 2) Instead of making a GetServerEnvironmentCert, make the cert a property. We already call GetServer on instantiation to get data from Environment - just populate it there. | 08:05 |
| stickupkid | manadart: thanks for the comment | 08:38 |
| manadart | stickupkid: NP. | 08:42 |
| manadart | Need a review: https://github.com/juju/juju/pull/8855 | 15:05 |
| rick_h_ | hml: thanks for that! Can you please file a bug around the OS_CACERT needing to be in OS config, part of add-cloud, etc. and I'd be curious your thoughts on implementation time for that. | 16:00 |
| hml | rick_h_: there’s a bug already along these lines: https://bugs.launchpad.net/juju/+bug/1777897 | 16:01 |
| mup | Bug #1777897: add-cloud fails when adding an Openstack with a self-signed certificate <juju:Triaged> <https://launchpad.net/bugs/1777897> | 16:01 |
| rick_h_ | hml: ah right thanks | 16:01 |
| hml | rick_h_: will ponder time question | 16:01 |
| knobby | wallyworld: Since conjure-up is just doing a juju deploy under the hood, I would think that the runaway VM creation would fall into juju in some way. I've never seen this with conjure-up or juju before though. I was hoping that adham had some old version of juju or something. I assume the juju debug-log would be useful if it happens again. | 18:05 |
| rick_h_ | knobby: definitely debug-log what's going on if there's some issue there | 18:18 |
| veebers | Morning o/ | 20:35 |
| wallyworld | veebers: can you jump into release call | 21:30 |
| veebers | wallyworld: omw | 21:31 |
| wallyworld | babbageclunk: now that release is out, you can haz review we mentioned last week? :-D pretty please with sugar on top https://github.com/juju/juju/pull/8839 | 21:55 |
| babbageclunk | oh yes! looking now | 21:55 |
| babbageclunk | Thanks for sending the email btw | 21:55 |
| wallyworld | babbageclunk: no worries, i also had to fix the lp upload thing etc - can't wait for that bug to be fixed | 21:56 |
| veebers | wallyworld: have we pushed on that bug yet? (I haven't >_>) | 21:57 |
| wallyworld | veebers: i *think* tim has, was going to check again this week | 21:59 |
| veebers | wallyworld: ack, cheers | 22:02 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!