/srv/irclogs.ubuntu.com/2010/03/25/#launchpad-dev.txt

lifelessthumper: :)00:25
thumpercan anyone remember how to get local mail delivery working?00:52
thumperfor the dev environment00:52
mwhudsonapt-get install postfix isn't it?00:53
mwhudsoni don't think there's anything special to do00:53
wgrantI just use postfix with its stock local config.00:54
wgrantThen redirect root's mail to my user.00:54
wgrantAnd point Evo at my mbox.00:55
thumperwgrant: do you use a .forward file?00:57
* thumper can't recall00:57
thumperbest not to install new software while a distro upgrade is in progress01:00
lifelessyou know01:00
lifelessparallel dpkg would be fun01:00
thumperhmm postfix is installed01:01
lifelessas would fine grained locks01:01
wgrantthumper: I just use /etc/aliases01:01
=== jelmer_ is now known as jelmer
thumperhmm, my local postfix install didn't generate a main.cf01:03
wgrantdpkg-reconfigure postfix01:03
spmpostconf -n >> main.cf ;-)01:04
* thumper thinks of something that will generate an email to test01:09
lifelessmail -s foo01:10
lifelessalso01:10
lifelessexim!01:10
spmtelnet localhost 2501:10
thumperspm: ooh hardcore01:11
spmheh, more like being on hardened firewalls... :-)01:11
thumperspm: do you know about postfix setup?01:19
spmthumper: that'd be a yes. getting rusty; but yes.01:20
spmin 2001, spm updated ~ 25+ firewalls and rather horrible mail funnel from sendmail to postfix. only 1 (minor) mess in the entire process. whee!01:21
thumperspm: I'm trying to get my local postfix to route email through my servers config using the submission port01:21
thumpermy email client sends email through penhey.net:58701:22
thumperhmm...01:22
thumperperhaps I'll try something else01:22
wgrantthumper: Why do you want it sent externally, to pollute your real mailbox?01:24
thumperI was just thinking about that01:24
thumperperhaps just relay into local user mail01:24
wgrantIt's what I do.01:24
lifelessthumper: exim can do it trivially01:25
lifeless:P01:25
* lifeless flogs the horse some more01:25
thumperwgrant: what do you read your local email with?01:25
wgrantthumper: Evolution.01:26
wgrantI use it for my normal mail too, though.01:26
* mwhudson reads launchpad mail with mail(1)01:26
thumperspm: how to I remove a deferred email from the postfix queue?01:30
spmthumper: sorry - mid debugging a U1 problem. one sec.01:31
spmthumper: postsuper -d <MSGID>01:32
thumperwhere the hell is the default mailbox?01:33
thumper/var/mail ?01:33
* thumper looks01:33
wgrantthumper: /var/spool/mail/root01:33
wgrantAlso /var/mail.01:33
thumpergah, kontact still won't see it01:36
thumperI can see the mail in the file01:36
* thumper shakes his head01:38
thumperhaha01:39
thumperspam caught it01:39
spmrofl01:40
spmPEBKA.... spam filter01:40
mwhudsongrar, another stable->db-devel conflict already01:44
thumperhmm... I wonder if I should boot into gnome rather than kde when lucid starts here...01:49
=== ]reed[ is now known as [reed]
thumperhmm02:30
thumperlucid decided to uninstall postgre 8.3 and python 2.502:30
wgrantYeah, they're gone.02:31
mwhudsonreenable the launchpad ppa and install launchpad-developer-dependencies02:32
wgrantReactivate the ~launchpad PPA, and install lp-dev-deps again.02:32
wgrantDamn.02:32
mwhudson:)02:32
wgrantIs ReviewerSchedule out of date?02:34
wgrantI don't think jml is in APAC any more.02:34
thumperwell...02:34
thumperthe default font for my irc client is much smaller :)02:34
wgrantAnd al-maisan is no longer around.02:36
wgrantAnd there have been no European reviewers since Monday night, although that might be special for this week.02:36
mwhudsonthumper: interesting discovery02:50
mwhudsonthumper: the kernel import has imported about 140 megs of packs so far, which sounds about right02:50
mwhudsonthumper: the git.db file though is a hair under 6 GIGABYTES02:50
mwhudsonwhich suggests something is going a bit wrong02:51
wgrantIs that why it is so damn slow?02:51
mwhudsonwho knows02:51
mwhudsonit doesn't sound very helpful though02:51
wgrant'taking 26 hours' <- niiiiice02:51
StevenKHm. Isn't the topic out of date, and it's in fact week 3 of 10.03?02:52
wgrantEven pear looks like it's going to take about 14 hours.02:52
wgrantStevenK: I think so.02:52
=== mwhudson changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.03 | PQM is in open | https://dev.launchpad.net/ | Get the code: https://dev.launchpad.net/Getting | On-call review in irc://irc.freenode.net/#launchpad-reviews | Use http://paste.ubuntu.com/ for pastes |
StevenKmwhudson: :-)02:52
mwhudsonStevenK: don't know what you're talking about!!02:52
thumpermwhudson: OMG02:54
mwhudsonthumper: indeed02:54
thumpermwhudson: where is jelmer_ when you need to smack him around a bit02:55
thumperwell gnome wasn't pleasent02:55
mwhudsonthumper: he knows, at least02:55
thumperit killed x02:55
thumpergwibber wouldn't start02:55
thumperand made gdm take over instead of kdm02:55
* thumper has punted the little blighter back into his corner02:56
thumperback into kde now02:56
StevenKHm. What do I do if I see a typo in a branch which I didn't notice until it landed? :-(03:01
StevenKThe typo is in a debug print, but I suspect it may throw an exception when it's run.03:05
mwhudsonStevenK: submit a branch to fix it03:06
* thumper is reinstalling launchpad-developer-dependancies03:10
thumpersomehow it got lost03:10
StevenKthumper: Perhaps it got lost when the upgrade was removing obselete packages?03:11
thumperStevenK: probably03:11
* thumper shurgs03:11
StevenKHm. How does one shurg?03:11
thumperit is like shrugging, but different :)03:11
StevenK:-)03:12
* thumper afk03:14
StevenKmwhudson: Would you mind reviewing the change, it's a one-line patch03:15
mwhudsonStevenK: sure03:15
StevenKmwhudson: https://code.edge.launchpad.net/~stevenk/launchpad/fix-typo-for-bug-503258/+merge/22098 ; but you'll need to wait for the diff03:15
StevenKWhich has turned up03:21
mwhudsonStevenK: is this covered by tests?03:21
StevenKmwhudson: I could have sworn it was -- which is why I'm a little concerned how it got landed03:21
mwhudsonhmm03:21
StevenKmwhudson: The original branch (fixes-bug-503258) changes the test to make sure that custom files are not private03:23
mwhudsonStevenK: ah03:24
mwhudsonStevenK: want to add a test then?03:24
StevenKIs debug printing on when running under a test?03:25
mwhudsonStevenK: it doesn't matter does it?03:25
mwhudsondebug(1, foo.bar) will blow up if foo doesn't have a bar attribute03:25
mwhudsonno matter what debug does03:25
StevenKmwhudson: It's a string subst, though03:26
mwhudsonStevenK: even more so then03:27
StevenKDear bin/test, plz output debug output03:34
StevenKmwhudson: Ah ha! It just prints the wrong filename, but doesn't error03:38
wgrant(Un)lucky.03:38
mwhudsonStevenK: heh heh03:39
StevenKAttributeError: 'PackageUploadCustom' object has no attribute 'filename'03:39
* StevenK grumbles03:39
StevenKSo even my fixed branch is broken03:39
StevenKWhich does fail the tests, excellent03:40
* wgrant applies an axe to buildd-slavescanner.txt.03:44
StevenKmwhudson: Do you need to re-rubber-stamp the branch now that I've fixed it so that tests actually pass, or shall I land it?03:46
mwhudsonStevenK: i'll have another look quickly03:47
StevenKmwhudson: It's pushed, so you should be able to see it03:47
mwhudsonStevenK: +103:49
mwhudsonland it03:49
StevenKAye, doing so03:49
=== flacoste is now known as flacoste_afk
lifelesswheeee vm installs fly05:46
thumperstub: ping06:53
stubthumper: pong06:53
thumperstub: my lucid upgrade installed postgres8.4 and removed 8.306:53
thumperstub: I have 8.3 back from lp dev deps06:53
thumperstub: but I've lost all my setup06:53
thumperstub: should I remove 8.4?06:54
stubIt did? Mine didn't, but I had both 8.3 and 8.4 installed.06:54
thumperstub: also, I can't find the database setup code06:54
stubYou can leave it. You can even use it - the tree passes and I would have loved to switch this cycle but not enough time. I just haven't announced it.06:54
thumperah, ok06:55
thumperstub: where are the setup instructions now?06:55
thumperI looked in rocketfuel-setup but didn't see them06:55
stubDunno :-) Looking in the wiki now06:55
stubhttps://dev.launchpad.net/Running says there is a script in utilities, although I've always done things manually (per the linked DatabaseSetup page)06:56
stub./utilities/launchpad-database-setup06:56
thumperstub: will that work with 8.3 and 8.4 installed?07:02
wgrantIIRC it will prefer 8.4 if it finds it.07:03
stubI have no idea. I've never run it myself. Should be fine though - unless someone tried hard to break things it will use whichever PG is running on port 5432.07:03
wgrantI believe it finds the latest version and hopes that it's running on 5432.07:04
thumperhmm, I have 8.3 on 5432 and 8.4 on 543307:05
wgrantThe script looks like it should die in that case.07:06
thumperwell that was a big fail07:09
wgrantWhat died?07:10
thumperthe setup script07:11
thumperfailed in applying hunks07:12
thumperfailed to create the user07:12
thumperfailed on other bits07:12
thumperthen said...07:12
thumperLooks like everything went ok.07:12
thumperbullshit07:12
wgrantHahhaa.07:15
thumperstub: it is all fucked!07:17
stubthumper: https://dev.launchpad.net/DatabaseSetup still seems relevant and should work for 8.3. It should also work for 8.4 if you don't add 'max_fsm_relations = 2000' to postgresql.conf07:20
thumperstub: well, 8.4 won't start at all says it has invalid directories et al07:22
thumperstub: so I decided to go with 8.3 for now07:23
thumperhowever07:23
thumperhttp://pastebin.ubuntu.com/401000/07:23
thumperwhy can't it get the working directory?07:24
wgrantBecause you're in the dropped cluster.07:24
stubBecause that script is destroying the directory you are running it from?07:24
thumperoh arse07:25
stubI'll update that script for 8.4 soon anyway. I think it is how the ec2 instances get setup.07:26
thumperhttp://pastebin.ubuntu.com/401003/07:29
thumperstub: ^^07:29
thumperpostgres8.3 says it is running07:30
stubHow does it say that?07:30
thumper:(07:31
stubIt might be running, but not on port 543207:31
thumper * Restarting PostgreSQL 8.3 database server                                                                                                                                 [ OK ]07:31
thumperhowever07:31
thumper$ /etc/init.d/postgresql-8.3 status07:31
thumperError: Invalid data directory07:31
thumperWTF?07:31
wgrantWhy not purge both, install 8.4, and run the script?07:31
thumperwill that work?07:31
wgrantIt has for me.07:32
stubDid you run step 4?07:32
wgrantExcept I installed 8.3, because LP didn't support 8.4 yet.07:32
stubdropcluster,createcluster?07:32
stubjtv updated the script to work with 8.4 IIRC07:32
thumperstub: yeah07:32
stubI'd go with wgrant's suggestion as it will probably work.07:33
thumperok07:33
wgrantAnd even if it doesn't work, it won't get you into any worse a situation than you're in now.07:33
thumperahh07:36
thumper8.4 can't be removed because it fails to stop07:36
thumperbecause it has an invalid data directory07:36
thumperhow can I force it harder?07:37
wgrantkill it and see if it still tries to stop.07:37
thumperit isn't running07:37
* thumper wonders07:37
mwhudsonstub: ec2 test uses the script yeah07:37
wgrantMake the data directory uninvalid, or hack the script to not whinge.07:37
thumperhow do I make the data directory uninvalid07:39
thumperI don't know where it is looking07:39
lifeless'valid'07:39
lifelessthe word is 'valid'07:39
wgrantlifeless: Well, I didn't want to take it that far.07:39
wgrantIt probably doesn't need to be a valid postgres data dir. It just needs to not look completely invalid.07:39
lifelesswgrant: so, appear valid :P07:40
wgrantPfft.07:40
thumperstub: any idea how I can get 8.4 unfucked?07:43
thumperI'm losing patience rapidly07:43
stubkillall postgres ?07:43
thumperdone07:44
thumperbut the --remove triggers a stop, which fails07:44
thumperwhich stops the --remove07:44
stub--remove?07:44
thumpersorry07:45
thumpersudo aptitude remove --purge postgresql-8.407:45
thumperthat is what fails07:45
thumperRemoving postgresql-8.4 ...07:46
thumper * Stopping PostgreSQL 8.4 database server                                                                                                                                           * Error: Invalid data directory07:46
stubKill the processes, nuke /etc/postgresql, nuke /var/lib/postgresql07:46
stubAnd file a bug if you can be arsed07:46
thumperstub: do you think the uninstall will work then?07:46
stubShould. There are no config files to know what to shutdown.07:47
* thumper thrashes the HDD07:52
thumperstub: aptitude tells me that postgresql-8.3 is installed, but there is no /etc/postgresql07:54
thumperI nuked it before07:54
stubSounds like aptitude didn't do some setup step. Or maybe its the 8.3 packages on lucid - I thought you where going to use 8.4?07:55
wgrantHmm. If you purged (not just removed) postgresql-8.3, postgresql-8.4 (and maybe postgresql-common), that directory should have been recreated on installation.07:56
thumperif I try to remove postgresql-8.3 it removes all the launchpad developer deps07:57
stubWith synaptic, it would remove the launchpad-developer-deps packages sure. But not the rest of the deps.07:58
thumperaptitude did, apt-get won't07:59
wgrantstub: Regarding that EmailAddress issue, USSO was creating them until recently. Bug #523346. But I thought they fixed the data too.07:59
thumperstub: so if I install the 8.4 packages for slony, contrib, and plpython I should be good?08:01
thumperstub: 8.4 didn't add a /etc/postgresql directory either :(08:02
thumperstub: before when you said nuke, I just moved them to /junk08:03
thumperstub: could that matter?08:03
stubYou can run pg_createcluster yourself per the wiki docs, but something is screwed08:03
stubI don't see how that could matter08:03
thumperit is creating the 8.4 cluster now08:04
thumperthe launchpad database setup script needs to be run as root?08:04
stubI expect so, unless it uses sudo internally08:05
thumperit uses sudo internally08:05
thumperjust checked08:05
thumperI think I'm getting somewhere08:06
thumpermaking schema08:06
thumperon 8.408:06
thumperok08:07
thumperwork done for me08:07
thumperI think I'm up to a working launchpad environment on lucid08:07
thumperstub: thanks for your help08:07
thumperand wgrant08:07
wgrantHopefully there's still only that one test failure. I might run the whole suite again tonight.08:08
StevenKwgrant: Which test failure?08:11
wgrantStevenK: test_generateConfig in Soyuz.08:11
wgrantapt-ftparchive outputs extra hashes.08:11
wgrants/Soyuz/archivepublisher/08:11
StevenKAh. I just had test_mark_duplicate_form_overlay fail for me, thought it was related08:12
wgrantThere was a commit related to that this morning.08:13
* wgrant WTFs at builder.txt.08:17
wgrantAh.08:53
wgrantI was wondering why the start of builder.txt was so shocking.08:53
wgrantannotate shows that it hasn't been changed significantly since around r6000.08:53
wgrantEr.08:53
wgrantr600.08:53
bigjools:D08:54
* wgrant throws a quick rewrite into another branch that touches that file.08:55
wgrantbigjools: buxy himself visited earlier and asked that we enable 3.0 (quilt) in Karmic. I discussed this with some distro people a few weeks ago and they tended to agree.09:09
wgrantThe worst that will happen is that builds will fail if they use one of the couple of things that Karmic's dpkg doesn't support.09:09
bigjoolswgrant: sounds fine then09:09
* bigjools considers firewalling all IPs from China, Russia and South America09:11
wgrantbigjools: Should I poke a LOSA with instructions, or ...?09:11
wgrantbigjools: Good idea.09:11
bigjoolsbastards DoSing me trying to crack my ssh09:12
wgrantdenyhosts FTW.09:12
bigjoolsyarp09:12
wgrant235 jobs (30 hours)09:13
wgrantWOW.09:13
wgrantI am glad I haven't tried to use a PPA lately.09:13
bigjoolsI auto-drop after 3 failed attempts but it still screws my connection09:13
bigjoolsbuilders removed, it'll come back09:13
wgrantUm, 30 hours is really quite ridiculous and calls for drastic action.09:14
bigjoolsit's not ideal09:14
wgrantWaiting more than a day for a build is well beyond not ideal.09:15
bigjoolswgrant: umm anyway yes if you prepare some SQL I'm sure we can get a losa to poke it in09:18
wgrantbigjools: OK. Thanks.09:18
* wgrant works it out.09:18
bigjoolsdid anyone see https://bugs.launchpad.net/bugs/546017 ?09:26
mupBug #546017: can't add linux-meta-ti-omap to kernel/lucid package set <Soyuz:New> <https://launchpad.net/bugs/546017>09:26
wgrantLOSAs: Can you please run this query on production to enable 3.0 (quilt) and 3.0 (native) in Karmic?09:29
wgrantINSERT INTO sourcepackageformatselection (distroseries, format) SELECT id, 1 FROM distroseries WHERE name='karmic' UNION SELECT id, 2 FROM distroseries WHERE name='karmic';09:29
wgrantbigjools: Yeah, that's an odd one.09:29
wgrantAlthough I don't see that source.09:30
bigjoolswgrant: can you pastebin that query, I need to get permssion from da management09:30
mthaddonwgrant: I'd need approval from someone (BjornT?) to run that on production09:30
wgrantAh, of course.09:30
* wgrant pastes.09:30
wgranthttp://pastebin.ubuntu.com/401051/09:31
bigjoolsta09:31
bigjoolsI ran it on dogfood anyway09:31
BjornTbigjools, wgrant: why is that query needed?09:34
bigjoolsBjornT: I just updated LPS09:34
wgrantBjornT: See scrollback starting from 25 minutes ago09:34
BjornTwgrant: i prefer short summaries :)09:36
wgrantWell, the relevant scrollback is two lines.09:36
wgrant20:09:24 < wgrant> bigjools: buxy himself visited earlier and asked that we enable 3.0 (quilt) in Karmic. I discussed this with some distro people a few weeks ago and they tended to agree.09:36
wgrant20:09:40 < wgrant> The worst that will happen is that builds will fail if they use one of the couple of things that Karmic's dpkg doesn't support.09:36
bigjoolsBjornT: there's a short summary in LPS ...09:37
BjornTwgrant: well, finding two lines in 25 minutes of scrollback isn't that easy, if you don't know which lines to look for09:39
BjornTthanks bigjools09:40
wgrantTrue.09:40
BjornTapproved09:42
wgrantThanks.09:42
bigjoolsthanks BjornT09:49
mrevellowdy09:49
deryckMorning, all.10:07
wgrantbuildd-slavescanner.txt creates builder1-11, sticks them in buildergroup.builders, and then does nothing further with buildergroup.10:08
wgrantBefore my branch it does touch buildergroup, but only to call rescueBuilderIfLost, which doesn't touch that .builders at all.10:09
wgrantSo there's a whole lot of dead tests code there that looks potentially important.10:09
noodles775wgrant: is it possible that the actual tests have been gradually moved out over time, leaving just the test setup there?10:10
wgrantnoodles775: I hope so.10:10
wgrantIt looks like it might now create other equivalent builders as it goes.10:11
wgrantThe functionality does seem to be tested elsewhere, so I guess the creations and assignments can go.10:12
noodles775Great.10:13
wgrant(I'm removing buildergroup, so needed to do something with them)10:13
=== dpm is now known as dpm-bbl
=== magcius_ is now known as magcius
=== magcius is now known as magcius_
=== magcius_ is now known as magcius
gmbstub: Still around?10:59
stubyo10:59
gmbstub: Ohai. Can you cast your weather eye over this query and tell us if it's completely mad?: http://pastebin.ubuntu.com/401091/11:01
gmbstub: We want to use it to schedule the next_check of a bug watch based on the number of errors it has had over the past 5 attempts.11:01
stubThe coalesce(array( stuff seems needlessly confusing over 'count(*)'11:03
allenapstub: I tried with a count but it didn't work after a couple of minutes and the array stuff did, so I didn't spend any more time on it.11:04
stubor sum11:05
stubBut fine. It actually works?11:06
gmbstub: Yes. In my head, I imbued that question with a tone of surprise.11:06
stubSticking a subquery in the FROM clause of an UPDATE that references the table being updated - can't recall having tried that11:07
stubnm,.. not doing that11:07
stubYour missing 'BugWatch.id = bug_watch.id' in the WHERE clause11:08
stubsorry... BugWatch.id = counts.id11:08
gmbAh yes, thanks.11:08
stubYou don't *have* to do it in a single query though ;)11:09
gmbstub: It made allenap happy so I was loathe to stop him.11:10
allenapstub: Another version of the array stuff with count(*) instead: http://pastebin.ubuntu.com/401099/11:12
* stub wonders why next_check is ever null11:19
stubWouldn't it just default to NOW and be updated after every check, successful or not?11:19
allenapstub: I don't completely understand. Do you mean that we should calculated at the same time the watch is checked?11:21
gmbstub: next_check is set to NULL by checkwatches and then updated by a garbo process. We wanted to keep the scheduling logic out of checkwatches since it's complex enough already.11:22
stubok11:22
stubNot sure if that is a win, but I'll believe you :)11:22
gmbFrom our point of view, anything that makes checkwatches do less is a win :)11:23
stubSo how does that query work in the context of a DBLoopTuner loop? It is updating all of them at once.11:23
stubThat could be tens of thousands if things don't run for a few days.11:25
=== jelmer_ is now known as jelmer
gmbstub: We can run that query against a batch of BugWatch IDs.11:26
gmbThat would fit nicely with the LoopTuner way of doing things.11:27
stubok11:27
stubIf you do that though, you are still materializing the recent activity list for all bugwatches each time though, even though you are only updating a batch.11:28
stubMaybe PG can optimize that, but I doubt it.11:29
stubOh... you put the list of ids in the subquery11:29
stubthat would work11:29
allenapstub: I was just about to say that ;)11:30
bigjoolswgrant: karmic now accepts quilt11:30
wgrantbigjools: Thanks.11:30
wgrant$LOSA: Also thanks.11:30
bigjoolswgrant: I just thought, we've never pushed a source package branch through dogfood11:32
wgrantbigjools: True.11:32
bigjoolscan you think of anything that would prevent that now (apart from lack of a UI)11:32
wgrantIt should all work now.11:32
wgrantAs long as the buildd is OK.11:32
wgrantAll relevant work is long-merged.11:32
bigjoolscool - have you got a script I can steal to poke something in?11:32
wgrantI have somewhere..11:33
* wgrant hunts.11:33
wgrantbigjools: I used http://pastebin.ubuntu.com/401111/11:34
wgrantIt probably still works.11:34
* wgrant checks the DF configs for the right branch prefix.11:35
bigjoolsok11:35
wgrantHm, so plain lp:whatever should work fine.11:37
bigjoolsok I'll poke it through now11:41
wgrantI suppose you can just use any old production package branch.11:41
wgrantAnd hope that dogfood doesn't explode...11:42
* bigjools starts ripping chunks of hair out11:43
bigjoolsAttributeError: 'NoneType' object has no attribute 'utf_8_decode'11:43
bigjoolsARGH11:43
wgrantThat reappears every couple of months.11:43
wgrantYay.11:43
=== noodles785 is now known as noodles775
wgrantI always forget how to fix it by the next occurrence.11:43
bigjoolsand every f***ing day on dogfood11:44
bigjoolsmake clean11:44
wgrantOh.11:44
bigjoolswell it's only doing it on iharness  atm11:45
bigjoolsI guess I'll have to put up with plain harness11:45
=== matsubara-afk is now known as matsubara
wgrantIs it reasonably up to date, so the UI won't burn up once the build gets dispatched?11:48
bigjoolswgrant: http://pastebin.ubuntu.com/401120/11:50
bigjoolsseen that?11:50
wgrantbigjools: The branch you referred to probably doesn't exist on DF.11:50
wgrantWhat was your recipe?11:50
bigjoolsok that makes sense, I blindly copied yours :)11:50
wgrantlp:ubuntu/hello should work11:51
bigjoolsyeah11:51
wgrantbigjools: How old is the DF archive?11:53
bigjoolsold11:53
wgrantIf it lacks a reasonably recent bzr-builder, I guess the target archive will need external dependency on production primary.11:54
bigjoolswgrant: ran the script, b-m not dispatching :/12:00
wgrantbigjools: The builders are all disabled...12:03
bigjoolscobblers, why didn't I check that12:03
wgrantYou probably also need to twiddle scores, but we'll see.12:05
bigjoolswhich I really should have done before watching gcc get dispatched.12:06
bigjoolscock12:06
wgrantOops.12:06
wgrantYou can't abort it?12:09
bigjoolsnot easily12:09
bigjoolsanyway, there's no sourcepackagerecipebuild12:10
bigjoolsso something is borked12:10
wgranttransaction.commit()?12:10
* bigjools weeps12:11
bigjoolsthat really shouldn't be required unless I started a txn >:(12:11
ricotzbigjools, hello, are there plans to add more builders cause there is a huge package queue waiting to be built?12:14
bigjoolsricotz: the missing ones will return at some point, I don't know when12:14
wgrant33 hours and counting...12:14
ricotzbigjools, ok, so this is out your hands then12:15
wgrantbigjools: Is concordia the one that took 2 hours to do a 2 minute build?12:15
bigjoolsyes, the machines that are missing are not exclusively PPA builders, they get loaned when not in use for other stuff12:16
bigjoolswgrant: dunno - but I started rubidium up and got a traceback when b-m tried to dispatch the sprb to it12:16
ricotzok, thanks12:16
bigjoolsricotz: we just have to be patient I'm afraid12:17
wgrantbigjools: What was the TB?12:17
bigjoolsDB permissions12:17
bigjools2010-03-25 12:16:42+0000 [-] psycopg2.ProgrammingError: permission denied for re12:17
bigjoolslation seriessourcepackagebranch12:17
wgrantAh. Easy fix.12:17
bigjoolsyep12:17
wgrantI thought I fixed that.12:18
wgrantunless you don't regularly run security.py on upgrades.12:18
wgrantHm, maybe not.12:18
bigjoolsI do run it every time12:19
wgrantYeah, i forgot about official source package branches when adding perms.12:19
wgrantOops.12:19
=== salgado_ is now known as salgado
bigjoolsso it's building now - I guess we need to fix: https://dogfood.launchpad.net/builders/rubidium12:27
bigjools:)12:27
wgrantI thought that was meant to work now.12:27
wgrantWhat's the exception?12:27
wgrantNo canonical URL for the build?12:27
bigjoolsForbiddenAttribute: ('title', <lp.code.model.sourcepackagerecipebuild.SourcePackageRecipeBuild object at 0xec71dac>)<br />12:27
wgrantArgh.12:28
wgrantI wish we had these exposed on the API.12:28
wgrantstill building... slow...12:31
bigjoolshow long should that take?12:31
wgrantHeh. It depends, mainly on disk cache.12:31
wgrantOn my laptop the time-consuming part is extracting and upgrading the chroot, and installing the build-deps.12:31
wgrantBut that's because the disk is slow and RAM is eaten by the rest of LP.12:32
wgrantIt normally takes a couple of minutes here.12:32
bigjoolsk12:32
wgrantWhat's it say it's doing now?12:35
bigjoolsError accessing Librarian: HTTP Error 404: Not Found12:35
bigjoolswak wak oops12:35
wgrantFail.12:35
wgrantNo chroot?12:35
bigjoolsprob the source expiry script12:35
wgrantWhy would it be getting sources?12:36
bigjoolsnfi12:36
wgrantthe input to a recipe build is a chroot and recipe text.12:36
bigjoolsyou'd think12:36
wgrantHm, that LFA is still OK.12:37
wgrantNo idea which file it is?12:39
bigjoolsnup12:40
bigjoolstrying to get the log out12:40
wgrantThe only thing the behavior tries to cache is the chroot.12:40
wgrantAnd the URL I get through the API still works.12:40
bigjoolsit got as far as "Processing triggers for python-support"12:41
bigjoolsno more log available12:41
wgrantwut.12:41
wgrantIs it still going?12:41
wgrantIt shouldn't be failing with a librarian error at that point...12:42
bigjoolsseems so12:42
bigjoolsthe log must be OOD12:42
wgrantOOD?12:43
bigjoolsout of date12:43
wgrantYeah.12:43
bigjoolsoh wait12:43
bigjoolsmy bad - I didn't clear the failnotes from last time12:43
wgrantI was about to ask that.12:43
bigjoolsbuilderok is True12:43
wgrantSo it's still going?12:43
wgrantGreat12:43
bigjoolsbut not doing anything12:43
wgrantCan you SSH into the builder?12:44
bigjoolsI think I'll poke another one in shortly on a non-virtual ppa so I can ssh into it12:44
bigjoolsnot into virtual ones. no12:44
wgrantOr just flip the virtualized flag on the BQ.12:44
bigjoolsyar12:44
bigjoolsneed to un-dispatch it though12:44
wgrantMark the builder not-OK.12:44
bigjoolsgrrrr12:49
bigjoolsI really really hate the million damn predicates for dispatching12:49
wgrantIt's somehow excluded?12:50
bigjoolsI'm making a new build, much much easier12:50
wgrantI don't think we build SPRBs non-virt even for non-virt PPAs.12:51
bigjoolsit'll get queued and then I can flip the bit before it's dispatched12:51
wgrantTrue.12:52
bigjoolsffs12:52
bigjoolsIntegrityError: duplicate key value violates unique constraint "sourcepackagerecipe__owner__distroseries__sourcepackagename__na"12:52
wgrantJust change any character in the SPR name.12:53
* wgrant loves constraint name length limits.12:53
bigjools2010-03-25 12:55:36+0000 [-] exceptions.AttributeError: 'SourcePackageRecipeBuild' object has no attribute 'is_virtualized'12:55
bigjoolsawesome12:55
wgrantWha? How did it work last time?12:56
bigjoolsI'm dispatching to a non-virt builder now12:56
bigjoolsthat's the only difference12:56
wgrantHeh.12:56
wgrantMy cursor was already over the right line. How odd.12:56
wgrant        assert not (not self._builder.virtualized and build.is_virtualized), (12:56
bigjoolsthat's the badger12:56
bigjoolsgrrr12:57
wgrantYes.12:57
wgrantCouldn't you have just turned one of the non-virt builders into a virtual one?12:58
bigjoolsferraz is building it now13:02
wgrantExcellent.13:03
bigjoolsbuildrecipe is running13:04
wgrantAh, much better.13:06
wgrantTargetting my PPA?13:07
bigjoolsyes13:07
bigjoolsit's still running and not doing anything :/13:07
bigjoolsidle box13:07
wgrantEw.13:07
wgrantWhat's the latest in the log?13:07
bigjoolsand I can't get to the log >:(13:07
bigjoolsrw only for the buildd user13:08
wgrantMaybe it's checking out the branch?13:08
bigjools0.0%wa13:08
bigjoolslogtail shows bzr running with the usual locale warning13:09
bigjoolsand that's it13:09
wgrantOK, so it's probably checking out.13:09
wgrantOr being blocked by a firewall.13:09
bigjoolsit's running bzr dailydeb13:09
wgrantCan you telnet to bazaar.launchpad.net:80?13:09
wgrantAha.13:09
wgrantHmm.13:09
wgrantit's not that big a package. But let's give it a while.13:10
bigjoolsyes I can telnet there13:10
wgrant:(13:10
bigjoolsummm the command line has:13:10
bigjools/usr/bin/python /usr/bin/bzr dailydeb --no-build /home/buildd/work/recipe /home/buildd/work/tree --manifest /home/buildd/work/tree/manifest13:10
bigjoolsbut /home/buildd/work doesn't exist13:10
wgrantI think that's fine.13:12
wgrantbuildrecipe constructs the path but never mkdirs it, and it has worked locally.13:12
bigjoolsk13:12
bigjoolsI need to wait for lamont then I can see what's going on13:13
wgrantCan you check the buildd-manager log and grab the recipe text?13:13
* bigjools needs food in the meantime13:13
bigjools<bigjools> and I can't get to the log >:(13:13
bigjools<bigjools> rw only for the buildd user13:13
wgranton the master.13:13
bigjoolsoh that13:13
bigjoolssorry13:13
bigjools# bzr-builder format 0.2 deb-version 2.4-3+${rev13:14
bigjoolsno}\nlp:ubuntu/hello\n13:14
wgrantThat's the text that was sent to the slave?13:14
wgrantIf so, try connecting to xmlrpc.launchpad.net on 443.13:14
bigjoolsha13:14
bigjoolsha ha ha13:14
wgrantI was assuming it would change it to an http URL.13:15
wgrantBut I guess not.13:15
bigjools:/13:15
wgrantThe connection is hanging?13:15
bigjoolsyip13:15
=== flacoste_afk is now known as flacoste
wgrantExcellent.13:15
wgrantSo it's not actually broken.13:15
wgrantI'm glad we weren't trying to do this on Friday afternoon.13:16
bigjools2010-03-25 13:16:21+0000 [-] Scanning failed with: 'NoneType' object has no attribute 'utf_8_decode'13:17
bigjools*weep*13:17
wgrantHahah.13:18
marswgrant, "I'm glad we weren't trying to do this on Friday afternoon" - you are a wise man :)13:19
bigjoolswgrant: are you doing a security.cfg fix?13:20
wgrantbigjools: Since we've identified no other issues there, I might.13:21
wgrantAnd I think that's the only other relevant table.13:21
bigjoolsand the other UI issues need sorting13:21
wgrantI forget -- can that go to devel?13:21
bigjoolsno, db-devel13:21
bigjoolsstrictly it can, but the regex will kill the landing IIRC13:22
wgrantGrmph.13:22
wgrantOK.13:22
* wgrant looks at the UI issue.13:24
bigjoolsI thought jtv had landed something to fix that actually13:25
wgrantHe fixed it for translations builds.13:25
bigjoolsah right13:26
wgrantSPRBs are broken for other reasons.13:26
wgrantSELECT on seriessourcepackagebranch for fiera is all that you added?13:26
bigjoolswe need an interface13:26
bigjoolsyep13:26
jtvdid I miss something13:26
jtv?13:26
wgrantWe have one. The view code just doesn't respect it.13:27
bigjoolsjtv: nope - we were just testing recipe builds on dogfood and the ui blew up13:27
* jtv looks for quick exit13:27
jtvsomething I did?13:27
bigjoolsjtv: nope :)13:27
jtvphew13:27
wgrantbigjools: How quickly can you get a sysadmin to apply a crowbar to the firewall?13:28
* bigjools looks which direction the wind is blowing in13:28
wgrantbuildmailman.py REALLY doesn't like being run concurrently.13:36
=== dpm-bbl is now known as dpm
wgrantnoodles775: Do we ave a title scheme for source package recipe builds yet?13:57
noodles775wgrant: I'll send another email summarising, but the conversation the other day lead to (1) the build url will sit under the recipe's url, and (2) we need to list the builds in multiple places (ie. obviously under the recipe, but also for the PPA or Builder contexts).13:59
noodles775Is (1) what you were after?13:59
wgrantNot quite, but I might need that in a minute.14:00
wgrantI'm wanting a string to use in the builder UI.14:00
wgrantTo identify the recipe build.14:00
wgrantBut I guess the recipe schema in the designs is different, so I'll just make something up for now.14:01
sinzuijml: Can you give me your thoughts on my suggestion for ttps://bugs.edge.launchpad.net/bugs/53802414:01
mupBug #538024: No way of saying "This project is not packaged" <Launchpad Registry:Triaged> <https://launchpad.net/bugs/538024>14:01
mpt"You are here because the Launchpad Edge environment uses the Launchpad Login Service service."14:02
noodles775wgrant: ok... so it looks like there is no title yet on the SPRBuild model. Just add one... something that mentions the recipe name and the target archive and the.... hmm, base-branch short name? Not sure.14:03
wgrantnoodles775: I won't mention the target archive at this point, since that's handled by other parts of the UI in somewhat revolting ways.14:04
noodles775wgrant: huh? isn't it a relevant part of what the SPRBuild represents?14:04
wgrantnoodles775: Yes, but Builds don't do it, and the builder-related pages include the archive regardless.14:05
noodles775wgrant: I think the only reason builds don't do it is because they are always in their context (PPA or a distroseries), it is on the build detail page of course (https://edge.launchpad.net/ubuntu/+source/bash/4.1-2ubuntu1/+build/1580941)14:06
noodles775s/don't do it/don't include it in the title/14:07
wgrantnoodles775: Those pages don't exist for SPRBs yet. If you look at https://launchpad.net/builders, you'll see that the archive name is included outside the link text (which is the title).14:07
wgrantThe same thing will occur for any BuildBase.14:07
wgrantThis is all going to have to be revised once the schema is finalised; this is just to make it stop OOPSing.14:08
noodles775True.14:08
=== matsubara is now known as matsubara-lunch
Ursinhabigjools: hello14:28
Ursinhabigjools: why did you remove the qa-needstesting tag from bug 541807?14:28
mupBug #541807: Archive should have a status column  <Soyuz:Fix Committed by julian-edwards> <https://launchpad.net/bugs/541807>14:29
UrsinhaI wonder if I have a weird internet problem that makes me visible only to the rosetta people14:31
Ursinhadanilos: hi14:31
danilosUrsinha, I don't see you, sorry14:31
* wgrant neither.14:31
Ursinhadanilos: so you don't have internet problems, you're blind14:31
=== pgquiles__ is now known as pgquiles
wgrantbigjools: Merging lp:~wgrant/launchpad/fix-sprb-ui should unbreak the UI, making logtails far less invisible.14:49
bigjoolswgrant: https://dogfood.launchpad.net/builders/ferraz15:04
wgrantbigjools: *that's* a bit more encouraging.15:04
bigjoolswe'll see!15:05
wgrantbzr is taking its time...15:07
wgrantI wonder what its problem is now.15:09
wgrantARGH15:11
wgrant    DEFAULT_INSTANCE = 'edge'15:11
wgrantlp: uses xmlrpc.edge.launchpad.net by default.15:11
wgrantNot xmlrpc.launchpad.net.15:11
wgrantbigjools: ^^ :(15:12
bigjoolswgrant: fuck sake15:14
bigjoolswhyyyyyyy15:14
wgrant    # We use edge as the default because:15:14
wgrant    # Beta users get redirected to it15:14
wgrant    # All users can use it15:14
wgrant    # There is a bug in the launchpad side where redirection causes an OOPS.15:14
wgrant"Beta users get redirected to it" makes no sense, though, since bzr won't possess the cookie.15:15
bigjoolsindeed15:25
bigjoolscheck this out: https://edge.launchpad.net/~ghostonline/+archive/personal/+packages15:25
bigjoolspackages stuck at pending :/15:25
wgrantYay. What's the error?15:26
bigjoolscan't find a mention of that ppa in the logs :/15:26
wgrant...15:26
bigjoolsoooo15:27
bigjoolsits publish flag is "f"15:27
bigjoolswtf15:27
wgrantHaha.15:27
bigjoolshow!15:27
bigjoolsnoodles775: do you know if anyone was disabling publishing on any archvies and might have got the wrong one?15:28
wgrantI didn't think there was any code that actually set that.15:28
noodles775bigjools: in production? Nope.15:28
wgrantIt was a couple of weeks ago.15:28
bigjoolswe've got 115 archives where publish=false15:29
bigjools110 are ppas15:29
wgrantWhat are the rest? Debian primary and a copy archive or four?15:30
bigjoolscopy15:30
=== matsubara-lunch is now known as matsubara
sinzuibarry: ping15:31
Ursinhabigjools: hey15:35
bigjoolsUrsinha: oi15:37
bigjoolstudo bem?15:37
Ursinhabigjools: tudo bem, e você?15:37
bigjoolsUrsinha: wet15:38
Ursinhabigjools: danilos could join this conversation, he's even able to read brazilian newspapers15:38
bigjoolsotherwise great :)15:38
Ursinha:P15:38
Ursinhabigjools: it's raining here as well15:38
danilosUrsinha, bigjools: cuzao15:38
bigjoolsUrsinha: let me guess, they're all complaining about your president?15:38
bigjoolsdanilos: rofl15:39
Ursinhalol15:39
Ursinhaman15:40
Ursinhahauhau15:40
Ursinhaso, bigjools, why are you removing qa-needstesting tags from your bugs?15:40
bigjoolswgrant: *cough* publish is set in the +edit page.... *cough*15:40
Ursinhadon't you like them15:40
bigjoolsUrsinha: are you stalking me?15:40
bigjoolsthey're untestable bugs15:41
Ursinhabigjools: you can mark them as qa-untestable then :)15:41
bigjoolsrefactoring and suchlike15:41
bigjoolsDETAILS!15:41
bigjools:)15:41
Ursinhalol15:41
wgrantbigjools: W...why?15:41
Ursinhabigjools: I can do that for you15:41
bigjoolswgrant: the question is, why not15:42
* bigjools hugs Ursinha15:42
Ursinhabigjools: done15:42
Ursinha:)15:42
bigjoolsUrsinha: how's your sprint?15:43
Ursinhabigjools: sorry about the stalking, I just needed to know if you're removing because of a weird use case I haven't thought about15:43
bigjoolsyou guys need a dogfood lesson15:43
Ursinhabigjools: it's great15:43
bigjoolsah I was only joking about the stalking :)15:43
Ursinhabigjools: :)15:43
bigjoolsis it hot there?15:43
bachi abentley15:48
abentleybac, hi.15:48
abentleybac, sure.15:49
bacok15:49
Ursinhabigjools: it's always hot in here :)15:50
* wgrant sleeps.15:50
bigjoolsnight wgrant15:50
bacabentley: i've got a branch A based on devel.  i've got another branch B that depends on A.15:50
bacabentley: both A and B have merged from devel and have all recent changes15:50
abentleyYou should not merge B from devel directly.  You should merge devel into A, and A into B.15:51
abentleyThis is one of the things bzr-pipeline gives you.15:51
bacabentley: if i run 'bzr diff -r ancestor:../A' while in B i see diffs that don't exist.  it shows a diff in the Makefile even though the Makefile is the same in A, B, and devel15:51
abentleybac, you have a criss-cross merge, which means that the revision selected by -r ancestor: will be relatively old.15:53
bacabentley: ok15:53
abentleybac, it should go away if you merge A into B.  This will warn you that it is a criss-cross merge.  I suggest using merge --lca, because it is a criss-cross merge.15:54
bacabentley: ok, i just merged again from A and now the phantom Makefile diff is gone15:54
bacabentley: thanks15:54
* bac vows to start using pipelines15:54
abentleybac, no problem.15:54
=== salgado is now known as salgado-lunch
marsmatsubara, ping, do you have IE installed for some simple QA work?  I would like to do QA on staging *before* landing my changes.15:55
abentleybac, of course, you can just manually make sure not to merge the same branch into A and a branch derived from A.15:56
bacabentley: right.  i just had a temporary lapse.15:57
james_wIf class A has implements(ISomeInterface) then should ISomeInterface.providedBy(A) be True?15:57
matsubaramars, nope. there's the windows EC2 machine though. I can use that. I can help you after the meeting.15:57
abentleybac, ah, okay.15:57
james_w.providedBy(instance of A) I mean15:57
sinzuijames_w: yes15:57
leonardranyone have any experience with the python memcached client?16:45
=== Ursinha is now known as Ursinha-lunch
jmlsinzui, sure.17:01
jmlsinzui, thanks for poking me17:01
=== deryck is now known as deryck[lunch]
=== salgado-lunch is now known as salgado
=== beuno is now known as beuno-lunch
marsmatsubara, instructions are on the wiki for testing with the EC2 instance?17:28
matsubaramars, I think so. let me find them17:30
matsubaramars, https://wiki.canonical.com/Launchpad/LPAWSAccounts for the win AMI and password. you basically start it up using elasticfox or amazon console17:33
marsmatsubara, great, thanks17:34
matsubaramagcius, then $ ec2-authorize default -p 3389 -s your_ip_address/3217:34
matsubaramars, and then $ rdesktop -u administrator <ec2-instance-public-dns>:339817:34
matsubaramagcius, sorry, above message was for mars, not you17:34
marsI wonder if the ec2 toolkit survived the Lucid upgrade?17:34
matsubaranow is a great time to find out :-)17:35
marsmatsubara, should the rdesktop port be 3389 or 3398?17:59
matsubaramars, 338918:00
marsmatsubara, thanks.  3389 works, guess the second command had the typo18:00
matsubaramars, yep, sorry about that18:01
=== deryck[lunch] is now known as deryck
=== beuno-lunch is now known as beuno
jml*described*18:18
=== Ursinha-lunch is now known as Ursinha
=== Ursinha is now known as Ursinha-sprintin
=== Ursinha-sprintin is now known as Ursinha-busy
=== dpm is now known as dpm-afk
thumpermorning20:02
bigjoolshey thumper how's the ui work going for recipe builds?20:24
thumperbigjools: you should ask rockstar and abentley, but they are pretty happy20:24
rockstarbigjools, I've spent today wrangling branches to make sure we have all our db patches in this cycle.20:26
bigjoolsI tried putting a recipe build through dogfood today and found quite a few bugs20:27
rockstarbigjools, eep.  Well, we should surface all of those in the next few weeks.20:27
bigjoolsyeah Michael is going to work more on the soyuz ui side20:28
bigjoolsit seems as though the builders will need access to xmlrpc.edge.l.n as well20:28
james_wsinzui: hi, thanks for the offer, my branch is at https://code.edge.launchpad.net/~james-w/launchpad/export-code-import/+merge/2217520:46
* sinzui looks20:46
sinzuijames_w: what kind of test can we write to verify this kind of mistake does not happen again. I agree you fixed these issues, but none of the other webservice tests changed. I think the problem is that the webservice/xx-distroseries.txt test reports that driver_link is correct, but we know that is not true21:02
james_wit is true21:03
james_wI am working on a fix for lazr.restful that will stop this from ever happening again21:03
james_wbut that's not going to be ready in the next 18 hours21:03
sinzuiunderstood.21:04
james_wso it depends if you want to land the fixes with the tests that will show the issue21:04
sinzuijames_w: I understand  your point. and agree it is a separate issue. Lets land this now. I glad to know there is a comprehensive fix for this issue.21:07
james_wthanks21:07
james_wit's not pretty, but it throws a big error in your face if you make this mistake, meaning the current tests we do for the webservice will be sufficient and we won't start having to write tests that assert that the wadl contains the right thing for everything we export as well21:08
sinzuifab21:09
james_wmaybe someone that knows zope better can come up with a more elegant solution21:09
sinzuithat would be gary_poster, flacoste, and leonardr.21:10
* gary_poster waves21:10
sinzuijames_w: do you need me to land this for you?21:11
james_wand I can't work out how to write the test in lazr.restful as things don't seem to behave the same way there21:11
james_wsinzui: yes please21:11
james_wgary_poster: hi, I'm confused by all the adapters and indirection that is going in the two codebases.21:12
sinzuitests in lazr are different since we do not assume launchpad exists. We build a lot of hypothetical objects and explicitly wire them. launchpad hides that a lot21:12
james_wif you look at the merge proposal above you can see the problem is triggered by using a zope.schema.Choice where a lazr.restful.fields.ReferenceChoice is intended21:12
james_wso I'm trying to do the same thing in lazr.restful21:13
james_wto use a Choice I need to provide a vocabulary, the only one currently defined in lazr.restful is an EnumeratedType, but that seems to give different behaviour, as the test uses getMultiAdapter, and picks a different Marshaller to what we get when the bug occurs in LP21:15
wgrantbigjools: Did the test build end up working?21:15
james_wso I'm trying to understand how the getMultiAdapter call decides to pick an ObjectLookupFieldMarshaller in LP.21:15
bigjoolswgrant: nope, I gave up waiting for someone to fix the firewall21:16
wgrantYay.21:16
wgrantBut it seems to mostly work.21:16
bigjoolsyeah I think it will be fine21:16
bigjoolsI could change the recipe I suppose so it uses http21:16
wgrantIf you just hack getRecipeData to use getPublicURL instead, yeah.21:17
wgrantYou can't change the recipe data itself, since the branches are stored as references and URLs are regenerated.21:17
bigjoolsoh :/21:17
wgrantbigjools: In lib/lp/code/models/sourcepackagerecipedata.py, s/bzr_identity/composePublicURL()/21:19
james_waha! lib/canonical/launchpad/zcml/webservice.zcml defines an adapter for SQLObjectVocabularyBase21:20
=== salgado is now known as salgado-afk
wgrantHmmm. Are we have silent test failures again?21:23
wgrantI have three successful ec2test emails, with around 3 failures each.21:23
wgrantBut they get ignored.21:23
wgrantAh, no, just: lib/canonical/launchpad/pagetests/standalone/xx-form-layout.txt, lib/canonical/launchpad/pagetests/standalone/xx-opstats.txt21:24
wgrantCan somebody please re-land https://code.edge.launchpad.net/~wgrant/launchpad/rescueiflost-and-updatestatus-to-model/+merge/22016? It got hit by a spurious failure a few hours ago.21:27
=== dpm-afk is now known as dpm
bigjoolsdid pqm reject it?21:28
wgrantNo, it died from the buildd-slave.txt ProtocolError that sometimes appears.21:29
bigjoolsoh in ec221:29
wgrantYeah.21:30
wgrantI've just confirmed that it still passes locally.21:31
=== fjlacoste is now known as flacoste
wgrantbigjools: Also, those two DF fixes from last night landed.21:55
bigjoolswgrant: coolio22:01
wgrantbigjools: Since the only unique failure appears to be spurious, can you please land rescueiflost-and-updatestatus-to-model?22:03
wgrantThe non-unique failures are more concerning :/22:04
bigjoolswgrant: it's best if you grab someone else, I am about to go to bed and if it fails again my name will be mud :)22:05
bigjoolsmaybe a friendly antipodean can help22:05
wgrantbigjools: OK, thanks.22:05
bigjoolsmwhudson, while not an antipodean, is very helpful :)22:05
mwhudsonmm what's this?  shoving things into ec2 land?22:06
bigjoolsyarp22:06
wgrantmwhudson: Well, I have a branch that had two real failures and one spurious failure.22:06
bigjoolscan you land something for wgrant please22:06
wgrantThe two real failures are in trunk, so they don't matter.22:06
wgrant(for that branch, at least -- in wider matters they are rather concerning indeed)22:07
mwhudsonhttps://login.ubuntu.com/+openid -> http 50022:07
mwhudsonyay!!22:07
mwhudsonand now it works22:07
mwhudsonwgrant: are you trying to convince me to just pqm-submit it?22:08
wgrantmwhudson: I might as well try.22:08
mwhudsonwgrant: i would rather not, tbh22:08
wgrantOK.22:08
wgrantCan you EC2 it, then?22:08
mwhudsonsure22:09
wgrantThanks.22:09
bigjoolsright, glass of single malt and then bedtime, g'night folks22:09
mwhudsonwgrant: https://code.edge.launchpad.net/~wgrant/launchpad/rescueiflost-and-updatestatus-to-model22:09
wgrantNight bigjools.22:09
mwhudsonbigjools: g'night22:09
wgrantmwhudson: That's the one.22:09
wgrantmwhudson: Have a look at http://paste.ubuntu.com/401411/. It's a successful test from last night, but with failures.22:10
wgrantAll the other runs had the same issues, but also ignored them.22:10
mwhudsonwgrant: that's rather odd22:11
wgrantYes. It's not a feature I really look for in my test infrastructure, personally.22:11
mwhudsondoes it happen locally?22:12
wgrantTrying.22:12
mwhudsoni wonder if there's a way to stop thunderbird 3 tying up a cpu indexing all my mail22:13
mwhudsonah, yes22:14
mwhudsonwgrant: the test fails locally, but also fails the test run22:15
mwhudsonwhich is nice22:15
wgrantI recall a similar issue a few months ago.22:15
wgrantBut with a specific type of test case.22:16
mwhudsonthere was a problem with twisted tests22:16
wgrantThat was it, yeah.22:16
mwhudsoni think that was somewhat different22:16
wgrant:(22:16
mwhudsonyes22:18
mwhudsonoh22:19
mwhudsonhm22:19
mwhudsonerrr22:19
* mwhudson swivels to point at zope.testing22:19
mwhudsonwgrant: i am extremely suspicious about the Exception in thread Thread-9:/NameError: global name 'suberr' is not defined traceback22:20
wgrantmwhudson: Ah, yes, true.22:22
mwhudsonthe rage is building22:23
mwhudsonwe last changed our zope.testing version in 2009-08-12 though22:25
mwhudsonoh right, i think it's because the subprocess is dying in a particular way22:25
mwhudsonzope.testing-3.8.1-py2.5.egg/zope/testing/testrunner/runner.py contains a hilarious number of potential name errors :(22:26
wgrantHeh.22:27
wgrantSo jml's new zope.testing hasn't landed yet?22:27
mwhudsonno22:28
mwhudsonthough i'm not sure what the blocker is now22:28
mwhudsonhardy packages of subunit maybe?22:28
mwhudson                if e.errno == errno.EINTR:22:29
mwhudson                    # If the subprocess dies before we finish reading its22:29
mwhudson                    # output, a SIGCHLD signal can interrupt the reading.22:29
mwhudson                    # The correct thing to to in that case is to retry.22:29
mwhudson                    continue22:29
wgrantProbably buildbot.22:29
mwhudsonYES BUT ONLY IF YOU'VE IMPORTED THE errno MODULE22:29
wgrantHahah.22:29
mwhudsonotherwise this won't work so well22:30
mwhudsonwgrant: the good news is that the newer zope.testing has these bugs fixed22:33
wgrantAha.22:33
wgrantInteresting timing.22:33
mwhudsonit's another thing that will make it harder to land jml's branch22:34
wgrantIndeed.22:36
mwhudsonrage rage rage22:36
mwhudsonwgrant: ./bin/test -vvct test_dir_construction_and_trivial_running goes bang rather stunningly22:39
mwhudsonand rather slowly too: it takes about a minute22:41
* wgrant is waiting...22:42
wgrantThe missing DB issue?22:42
mwhudsonyes22:43
mwhudsonof course the last change to the file containing that test was the addition of the AGPL copyright line22:43
wgrantYAY22:43
mwhudsontrying to run it in production-devel gets the22:46
mwhudsonexceptions.ImportError: No module named psycopg222:46
mwhudsonthing that plagued buildbot for a while22:46
wgrantHmm, I regret using allcaps before, since I can't make it an any bigger 'yay' now.22:46
jpdswgrant: Use the figlet.22:47
* mwhudson does binary chop on buildbot logs22:47
mwhudsonactually it's rather easy to find22:48
wgrantIndeed.22:48
mwhudsonbecause it starts where the number of tests reported as being run drops by 200022:48
mwhudsoner22:49
mwhudson700022:49
mwhudsoni wonder how many of these 7000 tests we're no longer running fail now!22:49
wgrantOh, it's that many!?22:49
wgrantOuch.22:49
=== Peng___ is now known as Peng_
mwhudsonbuild 716 -> test 26193 tests 26193 passed22:50
mwhudsonbuild 718 -> test 19516 tests 19516 passed22:50
mwhudson(buildbot vomited on 717)22:50
wgrant.... ouch.22:50
wgrantThat wasn't long ago, was it?22:51
mwhudsonso between 10574 and 1058022:51
mwhudsonno, not very long ago22:51
wgrantPhew.22:51
mwhudsonyeah22:51
mwhudsoni'm suprised noone's noticed the test suite going faster22:52
mwhudsonhttp://pastebin.ubuntu.com/401425/22:52
wgrantYeah, I just looked. Nothing there looks suspicious, except for maybe the launchpadlib thing.22:53
mwhudsonit's hard to see which of these commits could have done it22:53
mwhudsonso next thought: check the ami versions22:53
wgrantIndeed. Didn't jml push his new image a day or two ago?22:53
mwhudson716: ami-296f834022:54
mwhudson718: ami-296f834022:54
mwhudsonnot that then22:54
* wgrant reverts 10575.22:54
* mwhudson reverts back to 1057422:55
lifeless7K missing tests?22:55
wgrant"optimisation"22:56
wgrantReverting 10575 fixes the missing DB error.22:56
wgrantWTF22:56
mwhudsonthere's some quote in the company quotes page from jamesh about making the test suite faster by deleting all the tests22:56
mwhudsonthis is just the sneakier version22:57
mwhudsonwgrant: !!22:57
* wgrant goes hunk by hunk.22:57
wgrantYeah, it's the least unobvious change.22:58
wgrantThe Launchpad.login call.22:58
mwhudsonwgrant: the test is faster when it passes than when it fails22:58
wgrantMuch, yes.22:58
mwhudsonwgrant: uhhhhhhhhh22:59
wgrant(and no, lifeless, the word is not 'obvious' :P)22:59
wgrantmwhudson: Uh?22:59
mwhudsonwgrant: just thinking about what might be going on here22:59
wgrantLEONARD!!!!23:00
wgrantIt is an odd one.23:00
wgrantAlthough I guess that probably makes requests for WADL and stuff.23:00
mwhudsonyeah i guess so23:00
wgrantAnd is going to be horrifyingly slow.23:00
mwhudsonif that's being done for each page test...23:01
wgrantI really hope not.23:01
wgrantIt takes a while to even import.23:01
wgrantAh, not too bad with a hot cache.23:02
mwhudsoni guess it's the recursive call to suite.run() in test_dir_construction_and_trivial_running that's special about that test23:02
wgrantSo basically we now have to look at all of launchpadlib to work out what the problem is.23:02
mwhudsoni dunno23:02
wgrantAnd lazr.restfulclient and wadllib.23:02
mwhudsonthe page tests that test_dir_construction_and_trivial_running runs are exceedingly trivial: empty files23:03
mwhudsonit's possible that if they made any requests at all, we'd have this effect already23:03
wgrantDoes that not set up a DB, maybe?23:03
wgrantThe opstats failures make perfect sense, but this one does not.23:03
mwhudsoni don't really know23:05
wgrantOh.23:05
wgrantPageTestLayer doesn't inherit DatabaseLayer at all.23:06
mwhudsonhm?23:07
mwhudsonrunning PageTestLayer tests certainly sets up the database23:07
wgrantOh, yes it does.23:07
wgrantConfusing inheritance hierarchy is too complex.23:07
mwhudsonwgrant: hah yes23:09
mwhudsonwgrant: if i change the page tests that test_dir_construction_and_trivial_running runs to actually make a request23:09
mwhudsoneverything goes bang23:09
wgrantGiven how close we are to freeze, do we want to just revert the two problematic hunks (the Launchpad.login, and the single test that uses launchpadlib)?23:09
wgrantAha.23:09
wgrantSo somehow the the DB is getting deconstructed. Yay.23:10
mwhudsonum23:11
mwhudsoni'm not sure what the best course is23:11
mwhudsonhaving a launchpad object in the test globs that's authenticated as a sample data admin is bonkers anyway23:12
mwhudsonfor so many reasons23:12
mwhudsonwgrant: http://pastebin.ubuntu.com/401435/ ?23:15
wgrantmwhudson: Yeah, that.23:16
mwhudsonwgrant: do you know what the failures we've seen in the log actually are?23:16
mwhudsonthe regular looking page test failures23:16
wgrantmwhudson: The opstats one is because of the queries that Launchpad.login makes. I haven't looked at the other one.23:17
* wgrant runs with Launchpad.login removed.23:19
wgrantStill fails. Probably legit.23:19
wgrantAh, field description change.23:20
mwhudsonwell that was nice23:21
mwhudsonmy keyboard stopped working!23:21
mwhudsonoh23:21
mwhudsonit was probably seahorse asking for my passphrase :(23:22
wgrantmwhudson: The pagetest failure is a team description change yesterday.23:23
* wgrant fixes.23:23
mwhudsonwgrant: ah, cool23:23
mwhudsoni wonder how many other test failures this fix will shake out of the woodwork23:24
wgrantIt's only been a couple of days.23:25
* mwhudson is tempted to pqm-submit it so everyone gets to suffer23:25
mwhudsonyeah true23:25
mwhudsonwgrant: want to push/pastebin your fix and i'll merge it?23:25
mwhudsonthumper: want to review a branch?23:26
wgrantmwhudson: Just rerunning the test now.23:26
mwhudsoncool23:27
wgranthttp://pastebin.ubuntu.com/401444/23:28
wgrantSo that's all known failures fixed.23:29
mwhudsonyeah23:30
* mwhudson shoves it into ec2, starts writing email23:32
wgrantGreat.23:35
mwhudsonah balls, forgot --headless on the ec2 test23:42
wgrantOops.23:43
mwhudsonwell that wasn't what i said i would be doing in the standup23:46
wgrantHeh.23:48
wgrantThank goodness for spurious failure, or that would have gone unnoticed for a while longer.23:48
mwhudsonyeah23:52
thumpermwhudson: sure23:54
wgrantmwhudson: Regarding point 3), it basically calls DatabaseLayer.testTearDown twice in succession.23:55
wgrantNo idea why.23:55
wgrantHmm.23:55
wgrantI guess it tears down in the sub-suite run, then at the end of the test itself.23:55
mwhudsonwgrant: but why doesn't that always fall over in a heap?23:55
wgrantDoesn't it do some optimisation about not tearing down the DB if it's not dirty?23:56
mwhudsonmaaaybe23:56
mwhudsonhttps://code.edge.launchpad.net/~mwhudson/launchpad/aiee-everything-is-broken/+merge/2218523:58
mwhudsonthumper: ^23:58
thumpermwhudson: is your branch one that reverts the launchpad change?23:58
mwhudsonthumper: it's a little more delicate than that23:58
thumpermwhudson: I just read your email thread23:58
wgrant        if (ConnectionWrapper.committed and ConnectionWrapper.dirty):23:58
wgrant            PgTestSetup._reset_db = True23:58
wgrant        ConnectionWrapper.committed = False23:58
wgrant        ConnectionWrapper.dirty = False23:58
wgrant        if PgTestSetup._reset_db:23:58
wgrant            self.dropDb()23:59
wgrant            PgTestSetup._reset_db = True23:59
wgrantLooks like if you never commit, it probably won't drop the DB.23:59
mwhudsonthough i think my diff still says "XXX outraged XXX goes here"23:59
thumper# XXX Outraged XXX goes here.23:59
thumperyeah23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!