/srv/irclogs.ubuntu.com/2010/11/19/#launchpad-dev.txt

maxbIs someone visual on the need to initialize natty-cat-lpbuildd?00:33
wgrantIt probably won't ever happen.00:35
wgrantSince we're moving to run bzr-builder outside the chroot.00:36
wgrantBut we had to roll production back from that.00:36
wgrantProbably because bzr likes to eat RAM.00:36
maxbHrm. Someone really should just copy maverick-cat-lpbuildd for now, so that natty recipe builds aren't unilaterally doomed until then00:41
wgrantI suggested that.00:42
maxbmwhudson: Are you on the case with the importd breakage, or should I send a heads-up email to launchpad-dev@ to increase visibility of the problem?00:43
mwhudsonmaxb: a heads up email is probably a good idea01:10
mwhudsoni'm completely clueless and there's no losa around now01:10
mwhudsonhm, seems like there's a problem on staging too: https://code.staging.launchpad.net/~vcs-imports/pydoctor/trunk01:11
lifelessola01:43
wgrantAfternoon.01:45
lifelessanyone else seeing ': lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave.test_dispatchBuildToSlave ' fail in devel ?01:51
lifelessugh yes, we're in testfix01:52
wgrantYeah, broken here.01:53
lifelesscare to do a binary chop to find the rev?01:55
lifelessIts after 1193601:55
wgrantI presume 1193801:55
wgrantYeah.01:55
wgrantIt can't have been tested.01:55
wgrantIs that the only failure?01:56
lifelesslets not jump to conclusions01:56
lifelessno01:56
lifelessforwarding you the mail01:56
wgrantlifeless: Those are some interesting failures.02:00
wgrant(the webservice ones)02:00
lifelessyes02:01
wgrantBut http://pastebin.ubuntu.com/534111/ fixes the real failure.02:01
wgrant11938 changed the log level and added a new message.02:02
lifelessthumper: https://bugs.launchpad.net/launchpad-code/+bug/67729002:03
_mup_Bug #677290: test_dispatchBuildToSlave (lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/677290>02:03
wgrantlifeless: Ahh, the XML-RPC change. Of course.02:06
wgrantlifeless: Also, bug #677270 raises a few issues:02:06
_mup_Bug #677270: restricted librarian broken, content decoding error <Launchpad itself:New> <https://launchpad.net/bugs/677270>02:06
wgrant1) Some restricted files 404.02:07
lifelesswgrant: my WAG about xmlrpc firewalling appears wrong.02:07
wgrant2) gzip encoding is set on the 404 page, when it's not gzipped02:07
wgrant3) It's not obvious to people that those URLs are meant to be private.02:07
lifelesswgrant: well its up to them if they want to share it02:08
wgrantlifeless: Everything else in LP requires a cookie.02:08
lifelesswgrant: it was a design feature; we could make it one-time-use if we want02:08
wgrantIt's not obvious that the librarian URL is sufficient to grant access.02:08
wgrantI don't think making them one-time tokens is a fantastic idea, but we should probably think about something.02:09
lifelesswgrant: you have a valid issue, I don't know what to do right now, and a) importds and b) testfix and c) plane flights02:09
wgrantlifeless: Of course.02:09
wgrant(also, 11938 can't have been tested -- it was in the queue too soon after the review)02:09
lifelesswgrant: 404's - no idea why, file bugs.02:15
lifelesswgrant: gzip? zomg. file a patch.02:15
wgrantlifeless: I suspect they're related. I can't see why it 404s in that case, and it only gzips in that case.02:16
lifelessthumper: ping02:16
lifelessbac: ping02:17
lifelessabentley: perhaps you're still here?02:17
mwhudsonhmm, bzr upgraded to 2.2.1 on the -11 in devel02:17
lifeless[work wise]02:17
mwhudsonwhat are the chances that got rolled out on the 15th?02:17
lifelessmwhudson: have a look between 11887 and 1192602:18
abentleylifeless: FSVO "here".02:18
=== lifeless changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: importd system failing to import | https:/​/​dev.launchpad.net/​ | Get the code: https:/​/​dev.launchpad.net/​Getting
lifelessabentley: we've got a firefight on our hands, and I'm about to be offline for an extended period02:18
lifelessabentley: I'm looking for a volunteer to get the incident report rolling, and handoff to someone in asia as soon as such a person can be located02:19
mwhudsonlifeless: if it turns out to be the bzr upgrade, i'll try to get a downgrade organized02:19
mwhudsonlifeless: i can do that02:19
lifelessmwhudson: thanks02:19
lifelessmwhudson: I haven't tweeted or anything02:20
abentleymwhudson: thanks.02:20
mwhudsoni can do that too02:20
lifelessmwhudson: you'll need to cowboy02:20
mwhudsonlifeless: if it's the bzr upgrade, that's easy02:20
lifelessmwhudson: because https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html is out of date02:21
lifelessmwhudson: we can't tell whats up qa wise.02:21
mwhudsonlifeless: fantastic02:22
lifelessI think we need to do a manual check and nodowntime deploy of the revision that merged db-stablew02:22
lifelessthen it will work again.02:22
lifelessmwhudson: also we're in test fix02:22
lifelessmwhudson: per https://bugs.launchpad.net/launchpad-code/+bug/67729002:22
_mup_Bug #677290: test_dispatchBuildToSlave (lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/677290>02:22
lifelessmwhudson: so, I don't see any sane option except cowboy02:23
mwhudsonyeah02:23
mwhudsonat least the code import systems are isolated02:23
mwhudsonin that touching them isn't going to **** up anything else02:23
lifeless https://devpad.canonical.com/~lpqateam/oops-summaries/code-import-2010-11-18.html claims very few oopses02:23
mwhudsonwell, very few imports are running02:24
mwhudsonbecause the average import time has increased by like 30002:24
lifelessmwhudson: it only had 1602:24
lifelessif every import is failing02:24
mwhudsonthat is low02:24
lifelessI'd still expect more than 16 attempts a day02:24
mwhudsonyeah, should be 4*2*24 or so02:25
lifelesshttps://lp-oops.canonical.com/oops.py/?oopsid=1783CIW1102:25
lifelesshas room to improve02:25
lifelessmwhudson: so, definitely a bug on why didn't see it02:25
lifeless=== Top 10 Time Out Counts by Page ID ===02:26
lifeless    Hard / Soft  Page ID02:26
lifeless     282 /   56  Person:+commentedbugs02:26
lifeless     101 / 5112  Archive:+index02:26
lifeless      83 /  256  BugTask:+index02:26
lifeless      31 /  317  Distribution:+bugs02:26
lifeless      28 /    2  LoginToken:+accountmerge02:26
lifeless      24 /  241  POFile:+translate02:26
lifeless      21 /   28  Person:+bugs02:26
lifeless      16 /   14  DistroSeries:+queue02:27
lifeless      11 /    0  BinaryPackageBuild:+retry02:27
lifeless      10 /  403  Distribution:+bugtarget-portlet-bugfilters-stats02:27
wgrant13:27:02 < lifeless>       11 /    0  BinaryPackageBuild:+retry02:29
wgrantThat is disturbing.02:29
lifelessI don't have the oops code sorry02:37
lifelesswgrant: so are you seeing folk copying and pasting the restricted urls into public channels or some such ?02:39
wgrantlifeless: Just the one bug so far.02:40
lifelessexperienced user? new user?02:41
wgrantNot sure. Bug #67727002:42
_mup_Bug #677270: restricted librarian broken, content decoding error <Launchpad itself:New> <https://launchpad.net/bugs/677270>02:42
wgrantOEM.02:42
mwhudsonwell i can reproduce locally02:43
wgrantThe importd thing?02:44
mwhudsonyesd02:44
mwhudsonyes, it's the 2.2.1 upgrade02:46
mwhudsonwth!03:02
mwhudsonit's hanging somewhere inside of SFTPTransport.__del__03:03
bachi lifeless, mwhudson: i just got the ping for this issue.  i can help later if you need to hand off03:03
mwhudsonbac: thanks, i will need to fairly shortly03:04
bacmwhudson: fwiw, the links in the bug report download ok in webkit-browsers03:04
mwhudsonbac: that's not the issue i'm working on :-)03:04
bacmwhudson: i'm in a meeting right now but will be around later03:04
mwhudsonhm, seems present in 2.3b1 too...03:04
bacmwhudson: do you know who is working on the issue lifeless raised?  wgrant?03:10
mwhudsonbac: no03:10
lifelesswhich issue?03:11
lifelessbac: I've raised enough I don't know to which you refer03:11
baclifeless: the one you pinged me about.  the one requiring an incident report and a handoff to someone in asia.  sorry if i didn't comprehend the full scrollback.  just trying to get up to speed.03:12
lifelessbac: mwhudson is currently handling it03:12
lifelessbac: he will want to handoff to you within the hour03:13
mwhudsonoh sorry03:13
bacok03:13
wgrantAlso, someone needs to land that testfix.03:13
lifelessbac: perhaps you could to that ^ too ?03:13
lifelessif you have time03:13
baclifeless: i can in a bit when i get out of this meeting03:17
bacwgrant: where is the testfix branch?03:36
maxbWell that was bizarre. LP just temporarily decided I didn't have permission to view https://code.launchpad.net/+code-imports/+machines03:36
mwhudsoni bet a code import branch that was private could do that03:41
mwhudsonpretty sure we don't filter for visibility anywhere03:41
wgrantbac: I don't have a branch. But there's a patch on https://bugs.launchpad.net/launchpad-code/+bug/677290 which I could turn into a branch if you don't want to.03:41
_mup_Bug #677290: test_dispatchBuildToSlave (lp.code.model.tests.test_recipebuilder.TestDispatchBuildToSlave failing <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/677290>03:41
mwhudsonbac: i'm running away03:43
pooliewell done mwhudson !03:54
LPCIBotProject devel build (229): STILL FAILING in 3 hr 41 min: https://hudson.wedontsleep.org/job/devel/229/04:02
LPCIBotLaunchpad Patch Queue Manager: [r=danilo][ui=none][bug=676657] Memory-limit recipe builds.04:02
lifelesssinzui: https://bugs.launchpad.net/malone/+bug/244998 - looks like a candidate for bridging the gap to me04:02
_mup_Bug #244998: "Also affects project" is inconsistent and obscure when in package context <confusing-ui> <motu> <Launchpad Bugs:Triaged> <https://launchpad.net/bugs/244998>04:02
lifelesswgrant: is https://bugs.launchpad.net/soyuz/+bug/251685 happening still?04:06
_mup_Bug #251685: PPA upload hangs with 1K to go <poppy> <soyuz-upload> <Soyuz:Triaged> <https://launchpad.net/bugs/251685>04:06
wgrantlifeless: I think so.04:06
wgrantlifeless: We've certainly not knowingly fixed it (although it doesn't affect the SFTP version of poppy).04:07
poolieis there a new styleguide-type requirement introduced by the 'no more globs' launchpad changes?04:11
pooliejust trying to work out how to resolve conflicts in import statements in the right way04:11
lifelesswhatever works, try not to re-export things where possible.04:11
wgrantpoolie: utilities/format-imports forces imports to match our style. But that's not changed directly by the anti-glob policy.04:12
lifelessnot that its evil in principle, but the export * from foo idiom is pretty unmaintainable.,04:12
pooliesure, i just saw it now imports many things from different places04:14
pooliei'll just tweak it if they fail04:14
pooliehas anyone seen04:20
poolieKeyError: 'STORM_CEXTENSIONS'04:20
lifelessrun 'make04:23
lifeless'04:23
pooliethanks, that was my guess04:23
lifelessnew storm eggs seem to be missing the C extensions or some such04:23
lifelessit happens to me every storm upgrade we do04:24
bacthanks wgrant.  i have to relocate and will look at that patch shortly04:26
=== Ursinha-afk is now known as Ursinha
wgrantI think we need to do something about the footer.04:31
poolieabout its appearance?04:32
wgrantThe fact that it tells people to file bugs against Launchpad.04:32
lifelesswe're getting about a bug a day for Ubuntu04:32
wgrantSometimes several.04:32
lifelessbut OTOH we were getting that before anyway04:32
lifelessits a little higher frequency but really not a Big Deal04:32
wgrantThe frequency increased immensely once the footer made it onto production.04:32
lifelesswgrant: mmmm, I'd like stats for that actually04:33
poolie"Please report any problems with Launchpad." has a highly ambiguous "with" :-)04:33
lifelessyes04:33
lifelessimprovements appreciated04:33
lifelessits in lp-production-configs04:33
poolie"I have a persistent self of ennui"04:33
poolieseems like that could merge with "contact support" too04:34
wgrantAh hm.04:34
wgrantThat link bypasses all the warnings.04:34
pooliealthough for that matter if someone lands on an ubuntu bug page, "contact support" is an obvious way to ask about problems with ubuntu04:34
pooliewhich one?04:34
poolieoh wow, so much fail04:35
pooliePlease describe the bug in a few words, for example, "weather applet crashes on logout":04:35
poolieit's _really_ inviting people to file ubuntu bugs, isn't it?04:35
wgranthttps://launchpad.net/launchpad-project has a bit of a warning, but it's not terribly obvious.04:35
wgrantThere is also a warning at the bottom of the page once you've entered the summary and description.04:35
wgrantBut that's it;.04:36
pooliei thought there was a way to put text onto the filebug page?04:36
poolieconveniently below the fold on my browser04:36
wgrantThere is.04:36
wgrantYeah.04:36
wgrantFor this case it probably needs to be in a yellow alert above the initial dupesearch.04:36
poolieand anyhow, once people have entered their text, they're fairly invested in continuing04:36
wgrantAlthough mpt will probably throttle me for suggesting that.04:36
wgrantYeah.04:36
wgrantAnd you have to read a lot of text to see the warning.04:37
wgrantIf only we had a designer.04:37
pooliea little bird told me 'soon'04:38
wgrantlifeless: The problem with AJAX file uploads is that they're impossible.04:39
wgrantYou have no choice but to post a form in an iframe.04:39
lifelessis that what gmail does?04:40
wgrantI don't know... it's been quite a few years since I used it.04:40
wgrantBut probably.04:40
pooliei think they prefer to use a flash component04:40
* wgrant vomits.04:40
poolieotherwise i think it's blocking but imbw04:41
wgrantI wish bug listings could be convinced to show a set of tags.04:44
lifelessciao04:44
pooliesayonara04:44
pooliejml, dkim tests are running, we'll see what happens05:08
pooliewgrant: shall we file a bug for the footer problems?05:38
pooliethere might be an mpt bug already05:38
wgrantpoolie: It was restricted to edge until its demise.05:38
wgrantSo there's probably not an existing bug.05:38
pooliereally? the whole footer?05:38
poolieor just the invitation to file bugs05:38
wgrantJust the invitation.05:38
pooliei'll file one then05:39
wgrantThanks.05:39
poolieok bug 677342 and bug 37733906:02
wallyworld_poolie: you still there?06:20
pooliehi, i am06:47
pooliewallyworld_: i am06:47
wallyworld_poolie: i had to go out for a few hours - belinda's car broke down in the middle of a car park :-(, had to get tow truck etc, but i think i saw that the lp bzr 2.2.1 upgrade has broken something?06:48
pooliethat sucks about the car06:49
wallyworld_yeah :-( will be expensive06:49
pooliewas that the old one or the new one?06:49
wallyworld_the new(er) one06:49
poolieanyhow, apparently gc of an sftp socket causes a hang06:49
pooliei tihnk it's now resolvede06:50
wallyworld_ok. and that behaviour came about due to the upgrade?06:50
wallyworld_they rolled back to 2.2.0?06:50
wgrantRight, the importds are back on 2.2.0.06:50
wgrantAnd it's probably already fixed in the 2.2 branch.06:50
wallyworld_just the importds?06:50
poolieso it should stay fixed when they go to 2.2.206:50
wallyworld_ok. so 2.2.2 should be released around nowish?06:51
wallyworld_so i should upgrade lp to that?06:51
wallyworld_are there any tests i should have run apart from the test_bzr* ones when i did the lp upgrade to 2.2.1?06:52
poolieindeed, it should be pretty soon06:53
poolienot that i know of06:53
pooliecould be worth finding the particular bug06:53
wallyworld_yep. i'm not entirely across the importds stuff. i just want to not see the same thing happen next time06:54
poolieiwbn if we could upgrade just one importd first and see how it behaves06:54
wallyworld_in the code review for the 2.1.1 upgrade, i listed the bzr tests i ran06:54
pooliebut, perhaps it's not so bad, it's not immediately user critical if they have trouble06:55
pooliedon't all the tests get run?06:55
wgrantLP doesn't run the bzr tests, but presumably the bzr tests ran before 2.2.1 was released.06:55
wallyworld_yes, when it lands via ec2 which it did. but i also specifically ran a few by hand first06:55
wallyworld_wgrant: yes, i ran the ones i found, something like test -vvt test_bzr*06:56
wallyworld_wgrant: there were the ones i mentioned in the mp comments. i was hopeing that if there were any others i would be told to run them06:57
wallyworld_s/there/these/06:57
=== almaisan-away is now known as al-maisan
pooliejtv: hi?07:22
jtvhi07:22
pooliehi, i'm just looking at your parallel build stuff07:23
poolieit's great it's done07:23
pooliesome of the use of phony targets is a bit strange though07:23
pooliei think we can do even better07:23
pooliein particular i think we shouldn't depend on the phony target buildout_bin, but rather on $(BUILDOUT_BIN), the actual files07:24
poolieor perhaps on specific files, like $(PY)07:24
pooliewere the $(PY) dependencies not catching everything we needed?07:24
jtvHeh… what you're saying is pretty much the way it was.  :)07:25
poolieok, and it didn't work?07:25
jtvThe old way broke with parallel builds.07:25
jtvThere were two problems:07:25
poolieyes, i know07:25
pooliei was trying to fix some of them myself07:26
jtvIt'd be fine AFAIC to depend on $(PY) instead of on buildout_bin where appropriate.07:26
jtv(I really just got annoyed with the different ${PY}/$(PY) spellings)07:26
poolie:)07:27
jtvSo if you want to make that change, go ahead.  I don't think it'll change anything in the effects, since depending on $(PY) will still require the buildout_bin target to be built.07:27
jtv(Which is the way it needs to be in order to avoid breakage)07:28
jtvYes, definitely a lot of room for improvement still and nice to hear you're interested.07:28
pooliethe problem with depending on a phony target is07:28
poolieeverything that depends on it must always be rebuilt07:28
jtvTrue.07:28
pooliesince it does not exist on disk, make can't know it's up to date07:29
jtvI missed that.07:29
pooliesecondly, when you put that on the left hand side, and you do "rm $@", make tries to delete buildout_bin07:29
poolieand doesn't delete the files it previously was deleting07:29
jtvWhoopsie!07:29
poolie:) teamwork :)07:29
jtvActually I took out the $(RM) at one point.07:30
pooliei just worked out why that's there (i think)07:30
pooliewhich is that buildout tries to be smart and not touch the file if the contents would be the same07:30
poolieunfortunately then make always rebuilds it07:30
jtvGah.07:31
jtvWouldn't a "touch" have sufficed then!?07:31
pooliei think so07:31
pooliei'm kind of inferring it07:31
jtvI have a feeling that this would solve another problem.07:32
jtvI did run into an unnecessary rebuild at one point when I tried "make -j3 default schema" (with the don't-build-everything-for-schema patch).  Made one of the targets fail.07:33
poolieit might07:33
jtvThat makefile needs some love.07:34
pooliegetting more precise makefile rules is generally good07:34
wgrantbuildout!!!07:34
jtvpoolie: …and the stupid thing bounced again.  Did I miss anything?08:01
poolienot sure08:01
pooliehttps://code.launchpad.net/~mbp/launchpad/mbp-trivial/+merge/4126808:01
poolieare my suggestions08:01
poolielet me know what you think?08:02
wgrant"It would be great if we could also avoid the fairly slow js build08:02
wgranthappening every time..."08:02
wgrantWhy should the JS build take more than a second?08:02
pooliei don't know08:04
pooliefor me it does, doesn't it for you?08:04
wgrantIt does, yeah.08:04
poolieit's well over 10s on my laptop; i really notice it08:04
wgrantI just think we should make it less insane rather than reducing the frequency at which we call it.08:04
wgrantThere's no reason for it to take a non-trivial amount of time!08:04
poolieis it just catting the files, or is it doing complicated optimization?08:05
wgrantIt's some lazr.js voodoo.08:05
wgrantI don't know what lazr.js is meant to do.08:05
pooliethus my shallow inclination to call it less08:06
wgrantOr just delete it like U1 did.08:06
pooliecould be good08:12
poolieok, good night08:12
wgrantNight.08:13
=== al-maisan is now known as almaisan-away
adeuringgood morning08:29
henningeMoin adeuring ;)08:30
adeuringhi henninge!08:30
bachi adeuring08:32
adeuringhi bac08:32
=== adeuring1 is now known as adeuring
mrevellGuten morgen09:03
=== Guest77235 is now known as jelmer
LPCIBotYippie, build fixed!10:27
LPCIBotProject devel build (230): FIXED in 4 hr 2 min: https://hudson.wedontsleep.org/job/devel/230/10:27
=== almaisan-away is now known as al-maisan
bigjools$ time make -j211:09
bigjoolsreal    1m38.038s11:09
bigjools\o/  thanks jtv11:09
jtvbigjools: my pleasure... poolie got in on the game too, so expect more improvements soon11:11
bigjoolsjtv: I can't work out why -j4 takes longer11:17
jtvbigjools: otp, will explain in a bit11:18
bigjoolsno prob11:18
=== Guest54849 is now known as jelmer
bigjoolsI'm trying to make the webservice pick up my interface in the new brave world of post-jml interfaces apocalypse, and failing miserably.11:18
=== matsubara-afk is now known as matsubara
bigjoolsdoes this look ok? http://pastebin.ubuntu.com/534239/11:21
jtvbigjools: back.  About -j4: of course 2 CPUs don't necessarily give you a 2× speedup, because the real bottleneck is often memory.  That much is obvious.11:21
bigjoolsjtv: right, I suspect it's disk-bound11:21
bigjoolswill try on my SSD in a sec11:21
* jtv won't be any help to bigjools about that pastebin, and wait for someone else11:21
jtvbigjools: I don't think it's I/O—then -j4 would help you cover the latency and so be faster, assuming you have plenty of memory.11:22
jtvBut because of that, running multiple threads/processes at the same time will slow down each of them individually.11:22
bigjoolsjtv: if it's thrashing the disk head it'll be slower11:23
jtvOf course.11:23
jtvBut I don't think the footprint will be large enough to do that.11:23
bigjoolsI've seen the same problems when running parallel test suites by virtue of VMs11:24
jtvI haven't kept up with I/O schedulers; sure there may be a bit of extra seeking.11:24
bigjoolsmake -j2 on a fresh branch bails out with bin/py missing11:24
jtvDependencies still aren't entirely complete, I guess.  Martin found a missing one earlier today.11:25
jtvIn any case, as per my email, there's currently not much theoretical gain left beyond -j2 because the wadl script is the bottleneck.11:26
jtvThis is one of those nasty things in parallelization: if you distribute n+1 tasks across n CPUs, what's your ideal speedup?11:27
jtvThe answer is n/2.11:27
jtvWhich is very disappointing.11:27
bigjools:(11:28
jtvIn the theoretical ideal case, n jobs finish n× faster than with 1 CPU, and then n-1 CPUs sit around twiddling thumbs while the lucky one does the one remaining job, still at the same speed.11:28
bigjoolssince we write three wadls, won't -j3 help?  or is something else at play11:28
jtvWe'd have to break up the script.  One nasty thing there is we'd pay 3× the ZCML overhead.11:29
bigjoolsurgh11:29
jtvIn parallel, ideally, but that's no good to whoever isn't joining the SMP party.11:29
jtvI think ZCML is really costing us quite a lot.11:29
bigjoolsboth my boxes have 4 cores.  It makes me weep to see them idle :)11:29
jtvYou'll be interested to know that, because of the slowdown effect I mentioned earlier, AMD is actually dropping SMT.11:30
jtvBecause multiple threads will still slow down a single thread, and by and large people simply have enough cores now.11:30
bigjoolsreading *all* the zcml is costing us a lot11:30
jtvDo we ever do less?11:31
=== al-maisan is now known as almaisan-away
bigjoolswe read it all in every startup.  It's crazy.  We talked about moving it to registering via code on the ML some time ago11:31
jtvI haven't looked much further but I get the impression that, with my next makefile tweak, "make schema" is basically 45 seconds of "spend several seconds processing zcml… do something very quickly, done.  Start new script: spend several seconds processing zcml... work flash done.  Start new script:" etc.11:33
jtvI got the last question at James Clark's XML presentation last month.  I asked him to swear that neither he, nor any of the other team members to his knowledge were taking money from CPU or memory vendors. :)11:35
jtvbigjools: I'm still not getting any problems with "make -j2" on a fresh branch.  Are you sure you were using the updated Makefile and that you'd run the link-sourcecode script first?11:36
bigjoolsjtv: I * thought* I was... but it seems ok now.  Ho hum.11:37
jtvWell that's the problem with debugging parallel builds.  Everything goes Heisenberg.11:37
bigjools:)11:38
bigjoolsnow I am surprised.  My i5 with SSD is slower than the AMD Phenom11:38
jtvAnd the second time you run it?11:39
bigjoolsdoing so now11:39
jtvI mean, again on a clean branch11:39
bigjoolsit failed11:39
jtvbut with the good stuff in fscache11:39
jtvfailed!?11:39
bigjoolserror about import gpgme ... !11:39
bigjoolsran again and it worked11:39
bigjoolsyes, it's Heisenberg11:40
jtvThat's a missing dependency.11:40
jtvI get the impression that some of the targets' dependencies were written from a mental model of "build this, then this, then that."11:41
jtvAnd some of them are definitely missing.11:41
bigjoolsquicker 2nd time, still 3 seconds slower than the old Phenom11:41
jtvIs it a dual-core?11:42
bigjools4 core11:42
bigjoolswell, the i5 is a fake 4 core11:43
jtvPart of the fun of optimizing CPUs for this sort of thing is that the applicability of your benchmarks matters hugely.11:43
bigjoolsit's got the old hyperthreading guff11:43
jtvVendors like AMD and Intel have their internal benchmarks.11:43
bigjoolsrandom gpgme import fail again11:43
jtvFrom the wadl script?11:43
jtvDid you run a "make clean" between builds?11:44
bigjoolsyes11:44
* jtv tries frantically to reproduce the problem11:44
bigjoolsit's failing quite often11:44
bigjoolsif I run again without a clean it works and continue11:44
bigjoolss11:44
jtvThough I went through that "make clean ; make -j2" routine so often that I would have seen it by now.11:44
bigjoolsI'll paste in a bit11:45
jtvI've seen a failure like that, but with "make -j3 default schema" and the lightweight "make schema."11:45
jtvI guess it's just too system-specific or something.11:45
jtvbigjools: just for my piece of mind, could you grep ^buildout_bin Makefile ?11:46
bigjoolsjtv: it's on my other machine so not easy to paste, were you expecting output?11:47
jtvYes.11:47
bigjoolsit's a fresh devel pulled down 20 mins ago11:47
jtvThen it should have the updated makefile, definitely.11:47
jtvIt may just be the problem Martin spotted, where the virtual buildout_bin target gets re-built unnecessarily.11:48
bigjoolsjtv: http://pastebin.ubuntu.com/534245/11:49
jtvbigjools: where did you get that verbose output?11:53
bigjoolsjtv: the command I used is at the top of the paste11:53
jtvI don't see that subvertpy build output though.11:54
deryckMorning, all.12:02
deryckadeuring, hey, did you see lifeless' email on the librarian?  Do you have time today to get that landed for him?12:11
jmlbigjools: looks right to me12:11
bigjoolsjml: that's what I thought too - but the stuff is not in the wadl :/12:11
jmlbigjools: hmm.12:11
jmlbigjools: debugging this stuff is a pain too.12:11
bigjoolsyes12:12
=== almaisan-away is now known as al-maisan
bigjoolsI've put deliberate typos in the zcml and the interface to see if they get read, and they do12:12
bigjoolswebservice.py I mean12:13
jmlbigjools: do you actually have the webservice interface decorators in builder.py?12:14
bigjoolsjml: yes, my branch worked fine until your changes landed12:14
jmlbigjools: yay inventory :)12:14
jmlbigjools: umm... hmm... thinking.12:14
bigjoolsI've moved off to something else until leonard shows up :)12:15
jmlbigjools: not a bad idea.  I'll background it while I go off and pay for the drinks I forgot to pay for last night.12:15
bigjoolslol12:15
bigjoolsleft the CC behind the bar?12:16
jmlnot even that. they had table service and when we were done we just waltzed out not even thinking that we had to pay.12:16
jmlanyway, off. back soon enough.12:16
bigjoolsand, dine and dash12:17
bigjoolsjml: ahhh make -j3 is fastest yet on my i512:35
bigjoolsjtv, sorry12:36
jtvbigjools: that's unexpected… I have no idea how CPU governors will play into this btw.12:37
jtvSay you have 4 cores and 2 build processes, and maybe postgres and some other daemon that both wake up to do work from time to time.12:37
jtvSay each tends to stick to one core.12:37
bigjools1m42s fwiw12:38
jtvNot bad.12:38
jtvThough slower than -j2, no?12:38
bigjoolsbetter than the 5m it used to take!12:38
jtvI get something like 1:32 on -j212:38
jtvYes, that's what I did it for.  :-)12:38
bigjoolshmmm I was getting 1m53s on -j212:38
jtvPoor boy.12:38
jtvThen I guess this is good, yes.  :)12:39
* bigjools will try and beat 1:3212:39
jtvBut it also means that the performance characteristics are very different in your setups.12:39
jtvMine are pretty consistent, and that may explain how a missing dependency could be "hidden" on my machine fairly consistently.12:39
jtvSurprising how you don't seem to bottleneck on the wadl build.12:40
jtvAre you sure the wadl build is completing?  Maybe benji's optimization landed already?12:40
adeuringderyck: yes, I've seen his mail. but I haven't found yet much time to look at the test failures12:41
wgrantbigjools: Can you access it if you ignore the WADL?12:41
bigjoolswgrant: que?12:41
wgrantbigjools: What if you try to retrieve a builder through the API manually, not using the WADL at all. Does that work?12:42
bigjoolswgrant: I don't know how to do that12:42
wgrantbigjools: Browse to https://launchpad.dev/api/devel/builders/bob12:42
bigjoolswell, I probably do but I can't remember12:42
bigjoolsjtv: -j2 is 2:12 !12:43
jtvbigjools: why not the 1:53 you got before?12:43
bigjoolswho knows12:43
jtvbigjools: what does your load graph look like with -j2 or -j3?  Is there a lot of time where you've got only 1 CPU busy?12:45
bigjoolsI'll check later - busy running tests now12:46
deryckadeuring, ok.  Do you think you'll have some time today to look at it?12:46
bigjoolsah windmil, I hate you12:46
adeuringderyck: well, hard to say. Right now, I'm looking a bit more, but if anybody wants a review... ;)12:47
deryckadeuring, right.  I understand.  If you have time, great.  If not, np.12:47
bigjoolswgrant: yeah it works without the wadl13:22
wgrantbigjools: Have you tried deleting the WADL to force its regeneration?13:22
wgrantThat's sometimes needed.13:22
bigjoolsyes13:22
wgrant:(13:22
bigjoolsmake clean until my fingers bleed13:22
wgrantI'm not sure that's enough.13:23
bigjoolsit is13:23
bigjoolsit removes the whole apidoc dir13:23
wgrantHm, should be, yeah.13:23
bigjoolsand I see it getting re-generated on the next make13:23
wgrant:(13:23
wgrantI've run into this before. But I cannot remember how I resolved it.13:23
bigjoolsISTR there needs to be a magic string in the interface somewhere13:25
bigjoolsotherwise the wadl parser ignores it13:25
james_wbigjools, what are you doing?13:26
james_wtrying to export a new interface?13:26
bigjoolsjames_w: I had something that worked until jml got rid of canonical.launchpad.interfaces, now I've added some zcml + a webservice.py in lp.buildmaster but it dunt werk13:27
bigjoolsbut yeah it's a new export of IBuilder13:28
wgrantYou're sure that ZCML file is actually included?13:28
bigjoolsyes13:28
wgrantbuildmaster is a little incomplete.13:28
james_wyou added a webservice:register?13:28
bigjoolsyep13:28
james_wshould work then :-)13:28
bigjoolsyep :)13:28
bigjoolshttps://code.launchpad.net/~julian-edwards/launchpad/api-expose-builders/+merge/3937913:29
james_wbigjools, what's the collection you export?13:31
bigjools /builders13:31
james_wbigjools, and no builder stuff ends up in the wadl?13:32
bigjoolsnada13:32
james_wI can't see anything obvious, sorry13:33
=== mrevell is now known as mrevell-lunch
bigjoolsthanks for looking13:36
bigjoolsI'm going to eat and hopefully Leonard will be around when I get back13:36
deryckreboot.  brb.13:44
=== mrevell-lunch is now known as mrevell
=== jaycee is now known as jcsackett
LPCIBotProject devel build (231): FAILURE in 3 hr 40 min: https://hudson.wedontsleep.org/job/devel/231/14:07
LPCIBotLaunchpad Patch Queue Manager: [r=adeuring][ui=none][bug=677020] The advanced subscription features14:07
LPCIBotare now flagged correctly on the StructuralSubscriptionView.14:07
abentleyjelmer, bigjools: ping14:19
jelmerabentley: hey14:20
abentleyjelmer: can we chat about async uploads and source pagackge recipe builds?14:20
jmlbigjools: it occurs to me that leonardr won't be around today.14:21
=== matsubara is now known as matsubara-lunch
jelmerabentley: yes, sure. mumble?14:26
abentleyjelmer: sure.14:26
abentleyjelmer: https://bugs.launchpad.net/launchpad-code/+bug/67677614:28
_mup_Bug #676776: Recipe build stuck with "Uploading build" status [UI] <confusing-ui> <recipe> <Launchpad Bazaar Integration:Triaged> <https://launchpad.net/bugs/676776>14:28
abentleyjelmer: http://pastebin.ubuntu.com/534287/14:36
=== al-maisan is now known as almaisan-away
=== almaisan-away is now known as al-maisan
=== Ursinha is now known as Ursinha-lunch
bigjoolsjml: bugger14:58
bigjoolsin that case, hey benji!14:58
benjihey bigjools, what's up?15:00
bigjoolsbenji: I've got an API problem, do you have a moment to help please?15:00
benjisure15:00
bigjoolsgreat, thanks.  Can you look at this branch https://code.launchpad.net/~julian-edwards/launchpad/api-expose-builders/+merge/3937915:00
bigjoolsit exposes /builders on the API.  It used to work before the canonical.launchpad.interfaces removal15:01
bigjoolsI added the extra zcml declaration and webservice.py in lp.buildmaster15:01
bigjoolsbut it still doesn't generate the wadl15:01
bigjoolsI'm at a bit of a loss15:01
benjilooking15:03
benjibigjools: I couldn't find a problem by inspecting the diff so I built your branch and it seems the WADL and HTML docs were built correctly; there's an entry for "builder" that includes the description "Builder instance represents a single builder slave machine..."15:22
bigjoolsbenji: what about /builders ?15:22
bigjoolsthat's the first line in the test and it fails15:23
bigjoolsmore to the point does xx-builders.txt work there :)15:23
benjiI'll run it...15:24
benjiin the mean time, try rebuilding the WADL like so: LPCONFIG=development bin/py ./utilities/create-lp-wadl-and-apidoc.py --force "lib/canonical/launchpad/apidoc/wadl-development-%(version)s.xml"15:24
=== matsubara-lunch is now known as matsubara
bigjoolsbenji: trying that now - what are all those "Unknown entry URL:" things about?15:30
bigjoolsI vaguely remember us having to do something about those in the past but then everyone stopped bothering15:31
bigjoolsbenji: and re-generating the wadl didn't help :(15:31
benjiI don't know the full details, but it's XSLT not being told to be quiet about a particular element or value.15:31
benjidarn15:32
bigjoolsI have a feeling something was needed in lazr.restful15:32
bigjoolsbenji: another thing comes to mind, I have another vague recollection of having to put some magic string in the interface somewhere to make the wadl generator see a top-level collection15:32
benjiI'm still trying to run the xx-builders.txt test. (make schema, librarian kill error message)15:32
benjibigjools: I got this error "AttributeError: 'Launchpad' object has no attribute 'builders'"15:34
bigjoolsbenji: that's it15:34
bigjools"builders" is not in the wadl15:34
benjiok, cool; digging deeper15:34
bigjoolsthanks15:34
=== Ursinha-lunch is now known as Ursinha
bigjoolsabentley: loads of codehosting tests always fail locally for me (mercurial module missing for example), do we have missing dependencies in sourcecode or the dep packages?15:48
abentleybigjools: I haven't heard of that problem before.15:49
bigjoolsproblems seem to follow me15:50
abentleybigjools: mercurial is provided as an egg.15:51
bigjoolsI see it in eggs15:52
abentleybigjools: It's not bzr-hg?  That's provided in sourcecode.15:52
abentleybigjools: Try "bin/py -c 'import mercurial'"15:54
bigjoolsabentley: it's failing when doing "from mercurial import error as hg_errors" with "No module named mercurial"15:54
abentleybigjools: And when you try it from bin/py, does it puke?15:55
bigjoolsyes15:55
abentleybigjools: Something is wrong with your mercurial egg, then.15:56
bigjoolsit's the bzr-hg stuff in sourcecode that's importing it15:56
bigjoolsseems so15:56
abentleybigjools: You could torch your eggs directory and run "make".15:57
abentleyjelmer: It appears that upload failures cause two emails now: A "build-style" "failed to upload" message and an upload-style "rejected" message.15:59
bigjoolsabentley: yeah :/15:59
jelmerabentley: this is sprb failure emails specifically?16:00
abentleyjelmer: Yep.  Why would anyone care about binary builds :-D16:00
jelmerabentley: with our analysis from earlier this afternoon in mind, that would make sense.16:01
benjibigjools: here's the fix: http://pastebin.ubuntu.com/534311/16:01
bigjoolsbenji: argh!16:01
abentleyjelmer: perhaps processChangesFile is the wrong place for notification?16:02
bigjoolsthanks... I feel stupid now16:02
jelmerabentley: Hmm16:03
abentleyjelmer: It could raise exceptions, and then the caller could handle them appropriately according to the upload type, perhaps.16:03
bigjoolsabentley: yeah trashing the eggs worked16:03
bigjoolsthanks16:03
jelmerabentley: The special casing in PackageUpload.notify() is what's biting us there, too.16:03
deryckrockstar, ping a ring a ling16:03
abentleybigjools: I'm glad.16:04
rockstarderyck, what's going gangster?16:05
deryckrockstar, you know, livin' large, kickin' js ass and takin' names.16:05
rockstarderyck, straight.16:06
deryckrockstar, so question about your work on this branch initially--16:06
rockstarderyck, shoot16:06
deryckrockstar, did you link all yui modules in base-layout-marcro, or make some determination about which we actually used?16:06
rockstarderyck, I only added the ones that were complaining on the pages that I looked at.16:06
deryckah, crap.  I worried about that.16:07
deryckrockstar, so this could be the smallest number we can get away with?16:07
rockstarderyck, yeah, although it might be bringing in things it may not care about anymore as well.16:07
rockstarderyck, that's a long shot though.16:07
deryckrockstar, since we don't do dynamic loading, how can it bring in something?  Requirements should be a straight chain, no?16:08
bigjoolsbenji: I think I'm going to file a bug about that.  Silently failing is wrong.16:08
benjiit's probably a good idea to file a bug, but I suspect the silence is by design; you can enable different parts of a web service using ZCML, so how can it know when you intentionally don't want to expose a service and when you do?16:10
jmlalso, export in __all__ so it's obvious that it is being imported to be re-exported16:11
jmlrather than for some kind of side effect or by accident16:11
rockstarderyck, well, I meant "maybe we're manually bringing in a file for a dependency that doesn't exist anymore in YUI 3.2"16:12
deryckah, right16:12
deryckok, that makes sense.  I'll just poke at it to see.16:12
bigjoolsbenji: right - at least I think it should warn about things declarated but not exported.16:14
bigjoolsdeclarated?  I mean declared.16:14
benjithat might work16:15
rockstarderyck, so we're basically concluding that it's the size of the js file that's causing all these headaches?16:19
deryckrockstar, it is indeed the size of the file.  I'm 100% certain of that.16:20
rockstarderyck, that's unfortunate.16:33
deryckrockstar, yeah.  and yui without any lp js code is 1.1 of that 1.3 mb.  and getting it smaller is hard.  Just cut 250 files (around datatype) and got it to 900Mb.16:35
rockstarderyck, that's odd.  U1 has yui AND the U1 code down to ~870K.16:37
rockstarderyck, are you including lazr-js in the "yui" you speak of?16:37
deryckrockstar, perhaps we're including too much then.16:37
deryckrockstar, I thought lazr-js was compiled separately to lazr.js16:37
rockstarderyck, no, not in launchpad it's not.16:37
rockstarderyck, at least, I'm pretty sure it's not.16:38
deryckah, I see now.  It is but included in launchpad. js.  if I'm reading this right.  let me play some more....16:38
deryckrockstar, yeah, so with our yui deps, I get 1020k.  That's no lazr.js and no code from the lp tree.  Using the files you linked in base-layout-macros.pt16:43
deryckrockstar, perhaps we can trim some, but that's still a pretty ginormous file.16:43
=== benji is now known as benji-lunch
marsrockstar, do you remember what the exact source of the 512K JS file bug was?16:53
marssocket code?  Gzip hard-coded limit?16:53
=== al-maisan is now known as almaisan-away
rockstarmars, it was spinning on a socket call, yes.17:15
rockstarmars, but we found that it was indeed a 512K limit, and I put code in there to halt when it's larger.17:16
rockstarUnfortunately, I disabled that in the branch that is now deryck's because I thought "Oh, at this point, either windmill will have fixed the bug, or we'll have to disable windmill."17:17
marsrockstar, do you have any pointers to where the discussion may have been?  #tarmac, a LP bug, #launchpad-dev?17:18
rockstarmars, I believe the discussion was on the mailing list.17:18
marsrockstar, ok, thanks, that is somewhere to start17:19
abentleyjelmer: in that bug, the user said he received an email, not two.  I wonder if the failure happened sending the second email?17:24
jelmerabentley: that would certainly explain a lot.17:28
jelmerabentley: since it's the second email that was attempting to access the 'builder' table.17:28
=== benji-lunch is now known as benji
=== Ursinha is now known as Ursinha-afk
abentleyLaunchpad Development Channel: Pulling (mirrors, imports) now fixed. | Week 4 of 10.11 | PQM open for 10.12 | firefighting: importd system failing to import | https:/​/​dev.launchpad.net/​ | Get the code: https:/​/​dev.launchpad.net/​Getting19:17
abentleyLaunchpad Development Channel | Week 4 of 10.11 | PQM open for 10.12 | firefighting: pulling (mirrors, imports) now fixed. | https:/​/​dev.launchpad.net/​ | Get the code: https:/​/​dev.launchpad.net/​Getting19:18
=== timrc_ is now known as timrc
=== Ursinha-afk is now known as Ursinha
sinzuideryck, ping19:47
deryckhi sinzui19:47
sinzuideryck, I am looking at a css issue with this page: https://bugs.launchpad.net/launchpad19:47
sinzuiThe rounded border around the form is only used on this page and only when a search was not performed19:48
sinzuideryck, It is messing with font color. I need to fix it to put a branch in review, but I do not think this style should exist. Why does this form need an exceptional style?19:49
* sinzui should see if .portlet looks right19:49
mwhudsonhas anyone tried to see if maxb's bzr fix helps the code import situation yet?19:50
derycksinzui, it doesn't need it's own style.  Was just a design element that was preserved.  We can drop the border if it gets you going.19:51
sinzuideryck, using the portlet classes, the page might look like this: http://people.canonical.com/~curtis/upstream-bugs.png19:54
derycksinzui, +1.  Looks better to me actually.19:54
sinzuideryck, my branch is near done, but I am dissatisfied with how I am leaving it19:56
derycksinzui, how so?19:56
sinzuideryck, I expect the link the evolution in ubuntu to retry my search in ubuntu. It does not because or a common bug with this form. The search param are not added to the Advanced|Simple search links19:57
sinzuiI see there is a method that will encode the params for a URL. Deryck, to you know of a reason why we do not want to ensure the params are appended to these alternate search links19:58
derycksinzui, I see no reason not to append.  I assume it's just oversight that it doesn't currently.19:59
sinzuiI will start a second branch to see if I can fix this. this will also mean that changing the sort order will preserve the query20:00
sinzuiah20:00
sinzuiI see now. We need to fix the refine this search/search again problem because we do not know if the user is starting a new search20:01
deryckright, that makes sense20:01
sinzuiIf this works, I will fix 3 or more bugs20:01
jmlg'night all.20:09
deryckthanks for taking this on, sinzui.20:12
sinzuinp. I am sure I will benefit from the fix if I can land it20:13
=== Ursinha is now known as Ursinha-afk
=== Ursinha-afk is now known as Ursinha
=== matsubara is now known as matsubara-afk
=== Ursinha is now known as Ursinha-afk
LPCIBotProject devel build (232): STILL FAILING in 3 hr 43 min: https://hudson.wedontsleep.org/job/devel/232/21:55
LPCIBot* Launchpad Patch Queue Manager: [r=leonardr][ui=none][bug=666838] Export /builders and IBuilder on21:55
LPCIBotthe webservice.21:55
LPCIBot* Launchpad Patch Queue Manager: [r=adeuring][ui=none][no-qa] Fix spurious test failure in21:55
LPCIBottest_message_sharing_migration.21:55
LPCIBot* Launchpad Patch Queue Manager: [r=jcsackett,21:55
LPCIBotsinzui][ui=sinzui][bug=667900] Add form to DistributionSourcePackage21:55
LPCIBot+index page to set upstream project link more easily.21:55
LPCIBot* Launchpad Patch Queue Manager: [r=jcsackett,21:55
LPCIBotsinzui][ui=none][bug=677077] Find merge proposal by revno21:55
LPCIBot* Launchpad Patch Queue Manager: [r=allenap][ui=salgado,21:55
LPCIBotsinzui][bug=670431] Suggest branch owner as recipe owner21:55
LPCIBot* Launchpad Patch Queue Manager: [r=julian-edwards][ui=none][bug=673590] Change the log level of21:55
LPCIBotQueueInconsistentStateError during upload processing so that21:55
LPCIBotthe logger handler for cron scripts doesn't generate OOPSes.21:55
LPCIBot* Launchpad Patch Queue Manager: [r=allenap][ui=none][no-qa] Factor out gathering of POFiles for21:55
LPCIBottranslations export to bzr branches.21:55
=== Ursinha-afk is now known as Ursinha
=== Ursinha is now known as Ursinha-afk
lifelessmorning22:25
lifelessflacoste: hi22:25
jcsacketti'm not sure how i've gotten this far without already encountering this issue, but if how does one set up a store on an object such that Store.of returns something other than None?23:00
lifelessany object returned from a store23:01
lifelessnew objects need to be added to a store first23:01
jcsackettlifeless: okay, so i must just have a logic error. i thought classes needed some sort of setup.23:12
jcsackettlifeless: thanks!23:12
lifelessStore.of(Class) ? IIRC that will return the default store23:13
lifelesswhich can be a problem when something gets forced rather than defaulting23:13
wgrantlifeless, jcsackett: Store.of(object) will only work if object has been retrieved is a model object from a store.23:31
wgrantThe IStore, IMasterStore, ISlaveStore objects are Launchpad-specific and work on classes.23:31
wgrants/objects/adapters/23:31
* lifeless wonders if his branches landed.23:43
wgrantCan someone please harrass ISD about ixokai's issue (in #launchpad)?23:56

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!