| bac | hi gmb | 11:15 |
|---|---|---|
| gmb | Hi bac. | 11:23 |
| bac | gmb: hi there. have you any experience getting an 8 instance ec2 up and running? i've configured per gary's email. do you then just do the normal thing, deploying a master and slave or do you have to deploy multiple slaves? | 11:24 |
| gmb | bac: I'm just deploying the one slave. I haven't actually tried to make it build anything yet, though. | 11:26 |
| gmb | But the one slave, plus testr, plus multiple cores, should be all we need. | 11:26 |
| bac | ok, cool. i'm trying to test new steps for the slave i've added to the lpbuildbot master.cfg | 11:26 |
| bac | originally i had a slave start error...so it didn't get to the part i was interested in | 11:27 |
| gary_poster | hey gmb. it would have been fine if you had started the zope.testing thing. Better than fine, great. ;-) Did you? I'll arrange the cards if so | 11:59 |
| gary_poster | I'm assuming you completed the automation script? | 11:59 |
| gary_poster | restarting post upgrade... | 12:00 |
| bac | what is the trick for resetting the apt cache? from ec2 i'm getting hash errors. | 12:08 |
| bac | benji: do you know ^^ ? | 12:09 |
| benji | let me check my notes | 12:09 |
| bac | thanks | 12:10 |
| benji | bac: apt-get clean | 12:11 |
| gary_poster | hullo | 12:11 |
| gary_poster | through a series of misadventures I'm on the mac side | 12:12 |
| gary_poster | https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde | 12:12 |
| gary_poster | I'm waiting here should anyone care to join me ^^^ | 12:12 |
| gary_poster | bac benji gmb | 12:12 |
| gmb | gary_poster: Not yet, but I will (on both counts). I've spent my morning trying to get a working desktop environment, but I've given up and switched to OSX+SSH for the afternoon. | 12:12 |
| gary_poster | ok gmb | 12:13 |
| gary_poster | benji, I'm collecting, not filing yet, but I'll pastebin what I find here as I go | 12:38 |
| gary_poster | http://pastebin.ubuntu.com/912936/ is first | 12:38 |
| gary_poster | http://pastebin.ubuntu.com/912940/ is second, looks essentially the same (first line is mistake, should not have been copied) | 12:40 |
| * benji looks. | 12:41 | |
| gary_poster | aaaand that looks familiar... http://pastebin.ubuntu.com/912942/ | 12:41 |
| gary_poster | same http://pastebin.ubuntu.com/912944/ | 12:42 |
| benji | hmm, it looks like our previous fix for the locale whining of perl might have been good to keep | 12:42 |
| gary_poster | yeah maybe so | 12:42 |
| gary_poster | same http://pastebin.ubuntu.com/912948/ | 12:43 |
| benji | my first tack will be to treat it as a test bug, if the test doesn't care about locales then we should ignore this warning when comparing output | 12:43 |
| gary_poster | same: http://pastebin.ubuntu.com/912950/ | 12:44 |
| * gmb -> late lunch | 12:44 | |
| gary_poster | that would be nice, if possible | 12:44 |
| gary_poster | whee, same http://pastebin.ubuntu.com/912951/ | 12:45 |
| gary_poster | I see several of these sorts of errors but they are not connected clearly with a test (they break subunit formatting). http://pastebin.ubuntu.com/912955/ wgrant talked about these memcache things on the list recently. I thought the resolution was that it was caused by a newer version of some package and we were going to try and roll back | 12:47 |
| gary_poster | bzr locale again http://pastebin.ubuntu.com/912957/ | 12:48 |
| gary_poster | again http://pastebin.ubuntu.com/912959/ | 12:49 |
| gary_poster | OK, I will announce if this is *not* the same bzr locale bug. If I don't say anything, assume it is bzr locale | 12:49 |
| gary_poster | http://pastebin.ubuntu.com/912961/ | 12:49 |
| benji | I wonder why this is the first time we're seeing this error. | 12:50 |
| gary_poster | http://pastebin.ubuntu.com/912962/ | 12:50 |
| gary_poster | me too | 12:50 |
| gary_poster | this is first time on Precise beta 2. could be cause | 12:50 |
| gary_poster | http://pastebin.ubuntu.com/912965/ | 12:51 |
| gary_poster | http://pastebin.ubuntu.com/912967/ | 12:52 |
| gary_poster | http://pastebin.ubuntu.com/912968/ | 12:52 |
| gary_poster | That's it | 12:52 |
| gary_poster | So I only saw three sorts of errors | 12:52 |
| gary_poster | bzr locale | 12:52 |
| gary_poster | the Twisted reactor thing that benji fixed yesterday | 12:53 |
| gary_poster | and the connection errors that wgrant said are happening on the main buildbot | 12:53 |
| benji | cool | 12:53 |
| benji | gary_poster: do you want me to file the bug about the locale thing? | 12:54 |
| gary_poster | benji, sure thank you | 13:28 |
| benji | k | 13:28 |
| gary_poster | benji, if you give me the bug number I'm happy to add all the affected tests we found, or at least the pastebins. Or something | 14:02 |
| benji | gary_poster: 972456 | 14:02 |
| gary_poster | ack thanks benji | 14:02 |
| gary_poster | bug 972456 | 14:03 |
| _mup_ | Bug #972456: Tests can fail when bzr emits an unexpected "unsupported locale setting" warning <Launchpad itself:In Progress by benji> < https://launchpad.net/bugs/972456 > | 14:03 |
| gary_poster | gmb, fwiw, I was able to get the zope.testing tests to run by doing it in a lucid lxc (with python 2.6.5). Maybe that's the trick. I stashed the current failures at this pastebin: http://pastebin.ubuntu.com/913071/ . I saw four, though I thought frankban said we were down to three. I'm pretty tempted to dig into the subunit ones, since subunit support is so important to us, and the source of the regression. | 14:33 |
| gary_poster | The other three (testrunner-edge-cases.txt, testrunner-debugging.txt, testrunner-debugging-layer-setup.test) I'm planning on continuing to ignore | 14:34 |
| gmb | gary_poster: Noted, thanks. I'm not yet in a position to pick up the zope.testing work, so if you want to forge ahead with it and then (if you're not done with it by your EOD, which you may well be) send me a handoff email, I'll be happy to carry the baton tomorrow AM. | 14:35 |
| gary_poster | sounds perfect gmb, thx | 14:35 |
| gmb | (Or Precise might stop being a pain before then. Who knows?) | 14:35 |
| gary_poster | :-) | 14:35 |
| bac | gary_poster: juju expose for the master did not work for me. can you give me details of how you fixed it via the AWS console? | 15:14 |
| gary_poster | bac, sure (and how weird!) | 15:14 |
| gary_poster | bac, I'll do it at same time | 15:14 |
| gary_poster | go to http://aws.amazon.com/ | 15:14 |
| bac | there | 15:14 |
| gary_poster | My Account/Console -> console | 15:15 |
| gary_poster | sign in | 15:15 |
| gary_poster | click on ec2 | 15:15 |
| gary_poster | click on [N] Security Groups on right side | 15:15 |
| bac | (make sure you aren't in region 'singapore'! ) | 15:15 |
| gary_poster | :-) | 15:15 |
| gary_poster | look for group representing machine | 15:16 |
| bac | ok, now i have a ton of groups | 15:16 |
| gary_poster | juju-[name of the environment you used]-[number of the machine from juju status] | 15:16 |
| gary_poster | So for instance my environment was big-ec2 | 15:16 |
| gary_poster | and my machine was 2 | 15:17 |
| gary_poster | so I clicked on juju-big-ec2-2 | 15:17 |
| bac | 2 for the bb master, no? | 15:17 |
| gary_poster | On bottom click on "Inbound" tab | 15:17 |
| gary_poster | bac, depends on the order that you started, but yes, that's the way it is for me too | 15:17 |
| gary_poster | juju status is authoritative | 15:17 |
| bac | ok | 15:18 |
| gary_poster | In Port range on "Inbound" tab type 8010 | 15:18 |
| bac | now i already have (to the right) 8010 0.0.0.0/0 | 15:18 |
| bac | but no one is answering | 15:18 |
| gary_poster | ok, so this is not the problem | 15:18 |
| gary_poster | bac, next possibility is that the master is not actually up | 15:19 |
| bac | juju status shows the master as started | 15:19 |
| gary_poster | juju ssh 2 | 15:19 |
| gary_poster | go to /var/lib/buildbot/masters/master and look at twistd.log | 15:19 |
| gary_poster | As of now, bac, I have never encountered the situation you describe, fwiw | 15:20 |
| bac | uh oh | 15:20 |
| bac | i have no /var/lib/buildbot | 15:20 |
| gary_poster | heh | 15:20 |
| gary_poster | yeah, that doesn't sound so good | 15:20 |
| bac | i ran the hooks/install and start manually and they seemed happy | 15:20 |
| gary_poster | you are sure--you are looking as root? | 15:20 |
| bac | yes, not /var/lib/buildbopt | 15:21 |
| bac | yes, not /var/lib/buildbot | 15:21 |
| bac | yes, no /var/lib/buildbot | 15:21 |
| gary_poster | um | 15:22 |
| gary_poster | bac, I'm afraid I have no idea whatsoever. How did you deploy the master? | 15:28 |
| bac | juju deploy --config=/home/bac/juju/oneiric/buildbot-master/examples/lpbuildbot.yaml --repository=~/juju local:buildbot-master | 15:28 |
| bac | that failed due to the non-existent apt repo | 15:28 |
| bac | i fixed /etc/apt/sources.list | 15:28 |
| bac | and then did an 'apt-get update' / 'apt-get upgrade' | 15:29 |
| bac | then i ran hooks/install and hooks/start | 15:29 |
| gary_poster | oh, manually? | 15:29 |
| bac | yes | 15:29 |
| bac | how should i have done it? | 15:29 |
| gary_poster | bac, I think the right thing to have done would have been to do "juju resolved --retry buildbot-master/0" | 15:30 |
| gary_poster | there are environmental variables that are not around when you run them by hand | 15:31 |
| gary_poster | I described doing this in my second "Starting..." email yesterday | 15:31 |
| bac | gary_poster: i'm a bit confused | 15:31 |
| bac | i thought the "resolved" command was done before using 'juju debug-hooks', which i did | 15:31 |
| bac | so are you saying: | 15:32 |
| bac | juju deploy | 15:32 |
| bac | see it fail | 15:32 |
| bac | juju ssh and then fix /etc/apt | 15:32 |
| bac | then juju resolved -- and all should carry on happily? | 15:32 |
| gary_poster | *if* you do a retry | 15:33 |
| gary_poster | "resolved" alone means "I handled it; don't retry" | 15:33 |
| gary_poster | unless you add --retry | 15:33 |
| gary_poster | that means "I resolved the problem you encountered; please retry" | 15:33 |
| bac | i always used --retry, even if i was going to use debug-hooks | 15:33 |
| bac | ok, i'll shoot this environment and try again after lunch | 15:34 |
| * bac argh | 15:34 | |
| gary_poster | so...when you said "i fixed /etc/apt/sources.list" that means you did it within debug-hooks? | 15:34 |
| bac | yes | 15:34 |
| gary_poster | oh | 15:34 |
| bac | bad, huh? | 15:34 |
| gary_poster | no | 15:35 |
| gary_poster | that sounds fine on the face of it, just not what I did | 15:35 |
| gary_poster | I thought you meant that you had done it with juju ssh | 15:35 |
| bac | well, something i did caused it to remain unhappy | 15:35 |
| gary_poster | right | 15:35 |
| gary_poster | mm | 15:35 |
| gary_poster | you could try deploying another master? | 15:35 |
| bac | i don't see the benefit. i'd rather clean house and try again from fresh | 15:36 |
| gary_poster | you could try killing this master? I don't remember if they said there was a "die die die" for a machine, or if the only option is to redo an existing machine | 15:36 |
| gary_poster | bac, the only benefit is that you've already paid the price for the slave setup | 15:37 |
| gary_poster | and it is unrelated to the master | 15:37 |
| gary_poster | because you hadn't gotten that far yet | 15:37 |
| gary_poster | so if you got a working master | 15:37 |
| gary_poster | then you'd be able to connect your existing slave | 15:37 |
| bac | yeah, but i did the same lame-o dance with the slave, so i probably screwed it up | 15:37 |
| gary_poster | but I'm just brainstorming | 15:37 |
| gary_poster | I don't think it was necessarily all that lame-o | 15:38 |
| gary_poster | did you run install during the install step and start during the start step? | 15:38 |
| gary_poster | If so, AFAIK, you did everything the way it was supposed to be done, for some story | 15:39 |
| bac | ok, on the slave machine i do have /var/lib/buildbot/slaves/slave -- so it seems to be happier | 15:39 |
| bac | yes, that's what i did | 15:39 |
| gary_poster | yeah | 15:39 |
| gary_poster | It's not clear to me that you did anything wrong :-/ | 15:40 |
| bac | ok, i destroyed the master | 15:40 |
| gary_poster | ok | 15:40 |
| bac | i'll now try to redeploy | 15:40 |
| gary_poster | try again, yeah. master is fast | 15:41 |
| bac | ok, he's up and 'pending' | 15:42 |
| gary_poster | ok | 15:42 |
| bac | now 'installed' | 15:42 |
| gary_poster | huh | 15:42 |
| bac | oh, it is the same machine, so the apt problem is pre-fixed | 15:42 |
| gary_poster | right | 15:42 |
| bac | and 'started' | 15:42 |
| bac | whee | 15:42 |
| gary_poster | so is there a buildbot? | 15:43 |
| bac | yes | 15:43 |
| gary_poster | uh, great, I'm so glad we figured out this problem! :-D | 15:43 |
| bac | i will now add-relation | 15:43 |
| gary_poster | cool | 15:44 |
| bac | and it is available on the web. glad i didn't blow it all away! | 15:45 |
| gary_poster | great | 15:45 |
| bac | ok, so do i have to manually kick off a build? | 15:45 |
| gary_poster | yeah bac | 15:45 |
| bac | ok, so it tried to run my script 'init_testr.sh' but it was not there | 15:48 |
| bac | but it is there...in /var/lib/buildbot/slaves/slave/lucid-devel/build | 15:48 |
| gary_poster | bac, right mode? | 15:49 |
| bac | -r-xr--r-- 1 buildbot buildbot 395 Apr 3 15:46 init_testr.sh* | 15:49 |
| gary_poster | looks reasonable | 15:49 |
| bac | so perhaps it thinks it should be somewhere else? | 15:49 |
| gary_poster | I don't think so...lemme look at working example. Could you also give me url of web interface to master? | 15:50 |
| bac | gary_poster: yeah, the ./ made it work | 15:58 |
| gary_poster | cool bac. | 15:58 |
| bac | so i need to let the build finish and then restart another to ensure i have all of the data in the .testrepository? | 15:58 |
| gary_poster | yes bac | 16:08 |
| gary_poster | bac, it should take no time at all | 16:08 |
| gary_poster | it should be done already, in fact | 16:08 |
| bac | root@ip-10-82-27-185:/var/lib/buildbot/slaves/slave/lucid-devel/.testrepository# ls | 16:08 |
| bac | 0 1 failing format next-stream times.dbm | 16:08 |
| bac | i think we have a wiener | 16:08 |
| gary_poster | looks perfect bac :-) | 16:08 |
| bac | with that i lunch and bike | 16:08 |
| gary_poster | cool! talk to you in a bot | 16:09 |
| gary_poster | bit | 16:09 |
| benji | gary_poster: is the kanban update bot running again? and do we have to do anything to get it to update a card? | 16:22 |
| gary_poster | benji, I think it is running (80% sure), though it gets confused by bugs that do not have LP tasks (and maybe other situations) | 16:23 |
| benji | gary_poster: cool, thanks | 16:23 |
| gary_poster | we don't need to do anything to get it to update except wait and have an LP task for the bug (and maybe it must be the only one?) | 16:23 |
| gary_poster | welcome | 16:23 |
| benji | gary_poster (or bac): any thoughts on a next task? I think the main lanes are full so helping with one of those cards would seem best. | 17:35 |
| gary_poster | benji, looking | 17:35 |
| gary_poster | bac's is almost done; he just needs to wrap it up and send it off | 17:35 |
| gary_poster | I'm doing frightening, embarrassing things. Consider my most recent check-in message for our zope.testing fork: | 17:36 |
| gary_poster | bzr commit -m 'Consider this check-in suspect: I reviewed the test failures in the file and they seemed innocuous, rather than correct.' | 17:36 |
| gary_poster | I would have you join in with me but my internet connection is supposed to die for a bit soon | 17:37 |
| gary_poster | maybe we should do that anyway. You can make progress while I'm disconnected. You up for a call benji? | 17:38 |
| benji | gary_poster: I think so. | 17:38 |
| gary_poster | k | 17:39 |
| gary_poster | benji https://talkgadget.google.com/hangouts/_/extras/canonical.com/goldenhorde | 17:40 |
| gary_poster | benji left in a huff | 17:51 |
| gary_poster | or a cough | 17:52 |
| gary_poster | or a sneeze | 17:52 |
| benji | gary_poster: my machine crashed | 17:53 |
| gary_poster | benji, I figured as much. Come on by when things are normal again | 17:53 |
| * bac wraps things up | 17:57 | |
| bac | gary_poster, benji: could one of you gentlemens please review https://code.launchpad.net/~bac/lpbuildbot/remember-the-testr/+merge/100664 ? | 18:11 |
| gary_poster | bac, I'll look in a sec, sure | 18:12 |
| gary_poster | approved bac. thank you | 18:22 |
| bac | thanks | 18:23 |
| * bac looks for a card | 18:35 | |
| bac | gary_poster: did you have your francis call? | 18:36 |
| gary_poster | bac, 4PM | 18:36 |
| gary_poster | "lunch" | 18:54 |
| bac | benji: are you about to grab a spot in the coding lane? | 18:56 |
| bac | if so, anything i can help with? | 18:57 |
| benji | bac: nope, I'm helping Gary with his card | 18:57 |
| bac | okey doke. i'll throw a dart at my monitor then | 18:57 |
| gary_poster | benji, I suspect you are almost done? | 19:41 |
| benji | gary_poster: totally done: lp:~benji/zope.testing/3.9.4-fork/ | 19:42 |
| benji | I don't know if we're MP-ready or not. | 19:42 |
| gary_poster | awesome benji. Are you making an MP? Shockingly, I'd be happy to review...oh ok. What's up? | 19:42 |
| benji | gary_poster: I'll do an MP. I just wasnt' sure if there was more you wanted to do. | 19:43 |
| gary_poster | no, cool. We fixed the bug, and did some other good stuff on the way. | 19:43 |
| gary_poster | You'll be happy to know that my MIL's low-end FIOS Internet connection is close to double my maxed-out U-verse connection | 19:44 |
| gary_poster | I need to get her an AppleTV or Roku or something | 19:44 |
| gary_poster | Because she should be able to take advantage of this easily | 19:45 |
| gary_poster | Maybe I'll just run CAT5 down the interstate from NJ to NC | 19:46 |
| gary_poster | I'm sure that will work out perfectly | 19:46 |
| bac | gary_poster: hey, could i tap off it? | 19:46 |
| gary_poster | heh, sure bac | 19:47 |
| benji | gary_poster: https://code.launchpad.net/~benji/zope.testing/3.9.4-fork/+merge/100679 | 19:47 |
| gary_poster | ack benji, on it | 19:47 |
| bac | fwiw, i used to have a roku but it looks ugly and ham fisted next to the appletv | 19:47 |
| gary_poster | ok | 19:48 |
| gary_poster | yeah I like my appletv | 19:48 |
| gary_poster | benji, wrong merge target | 19:48 |
| gary_poster | so conflicts and other bad stuff | 19:48 |
| benji | darn | 19:48 |
| gary_poster | make merge target of lp:~launchpad/zope.testing/3.9.4-fork | 19:48 |
| gary_poster | benji, ^^ | 19:49 |
| benji | already done | 19:49 |
| benji | gary_poster: https://code.launchpad.net/~benji/zope.testing/3.9.4-fork/+merge/100681 | 19:50 |
| gary_poster | cool benji, much better thanks :-) | 19:50 |
| gary_poster | benji, line 8 of diff is my fault. I regarded it as a hack and I didn't mean to check it in. I suggest removal, but if you like it for some reason that's fine | 19:51 |
| benji | looking | 19:52 |
| benji | no that's evil, I'll remove it | 19:52 |
| benji | pushed | 19:53 |
| gary_poster | thanks | 19:53 |
| gary_poster | benji, funny that there was already a test of the odd behavior of list tests | 19:54 |
| gary_poster | benji, approved | 19:54 |
| benji | gary_poster: cool, I'll merge it into ~launchpad/zope.testing/trunk, then | 19:56 |
| benji | er, lp:~launchpad/zope.testing/3.9.4-fork, rather | 19:56 |
| gary_poster | benji, you mean https://code.launchpad.net/~launchpad/zope.testing/3.9.4-fork ? | 19:56 |
| gary_poster | yeah ok | 19:56 |
| benji | :) | 19:56 |
| gary_poster | :-) | 19:56 |
| benji | gary_poster: I just posted my March EC2 expenses | 19:58 |
| gary_poster | how bad were they benji? | 19:58 |
| benji | it wasn't nearly as bad as yours: $162.12 | 19:59 |
| gary_poster | Still, a new high? | 19:59 |
| benji | gary_poster: by a (base 2) order of magnitude | 20:02 |
| benji | ok, branch merged and pushed | 20:02 |
| benji | I guess saying "twice as much" would be easier. | 20:02 |
| bac | gary_poster: is the /etc/apt/source.list problem on ec2 going to be something we have to deal with long term? do we want a card to fix it? | 20:03 |
| gary_poster | bac, I don't know. Maybe we should. :-/ | 20:04 |
| bac | i'll add it. deleting later is cheap | 20:04 |
| gary_poster | benji, btw, please make a lp branch for getting the new egg in the tree | 20:14 |
| benji | gary_poster: ah, will do | 20:15 |
| gary_poster | thx | 20:15 |
| gary_poster | benji + card ;-) | 20:15 |
| benji | k | 20:15 |
| bac | gary_poster: any suggestions on a task to pick? | 20:27 |
| gary_poster | bac on call | 20:28 |
| benji | darn, my setuplxc-created dependencies directory has HTTP checkouts | 20:40 |
| gary_poster | bac, the most helpful task would be to investigate the memcache ConnectionError failures that wgrant described on the list ("[Launchpad-dev] memcache errors in ec2 and buildbot -- newer python-memcached to blame?"). They are affecting the main buildbot and us as well, so it's a generic "green buildbot" thing. | 21:21 |
| gary_poster | alternatively... | 21:21 |
| gary_poster | "teach buildbot to understand subunit in test results to properly report failure numbers in waterfall" card is interesting | 21:22 |
| gary_poster | alternatively "Fix /etc/apt/sources.list on ec2" from you. It's not that the debian repo isn't there, it's that it has the wrong hash. Not sure how that would happen | 21:24 |
| benji | gary_poster: I give up. I have the new p6 egg in the dependencies repo but I can't for the life of me get a LP build to work to make sure it's used and does the right thing. I think my cold has infected my brain. I'll try again tomorrow if someone else hasn't done it. | 21:26 |
| gary_poster | benji, ok. how weird! so you've pushed the change? | 21:27 |
| gary_poster | to the download-cache I mean? | 21:27 |
| gary_poster | the addition of the p6 egg | 21:27 |
| benji | gary_poster: yep | 21:31 |
| gary_poster | cool benji. feel better. I will be out tomorrow it looks like. | 21:32 |
| gary_poster | I'll send a note to yellow list | 21:32 |
| benji | ok; enjoy yourself | 21:32 |
| gary_poster | with suggested tasks | 21:32 |
| gary_poster | thanks | 21:32 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!