lifelesswgrant: the PPA quotas are precisely as arbitrary as the recipe volume quota00:30
lifelessthe PPA quota is less of a surrogate00:30
wgrantlifeless: Well, arbitrary is perhaps not the best word.00:32
wgrantBut the PPA quota does what it's designed to do.00:32
wgrantPlaces a hard, known limit.00:32
wgrantThe recipe one does not.00:32
lifelessI'm having some trouble following your detailed description of whats wrong; perhaps you could give me a summary instead?00:33
wgrantThe PPA disk quota prevents users from using more disk than they should.00:33
wgrantThe recipe build quota is presumably to stop users from using more buildfarm time than they should.00:34
wgrantBut it limits how many times they can use it.00:34
wgrantNot how much of it they use.00:34
lifelessas I said, its a surrogate.00:34
lifelessI'd welcome a series of bugs that would allow measuring and allocating the actual resource.00:34
pooliehi there00:38
pooliejml wrote some code to suck lp bugs into a desktopcouch db00:38
pooliei was thinking in the shower this could possibly even be offered as a standard facility00:38
pooliewhich would be pretty cool: cheap async (at least readonly) access00:39
pooliethis is very blue sky of course00:39
lifelessI'd want couch to have an OOM less sysadmin time first :)00:40
lifeless(and sadly, I'm not joking)00:40
* poolie reinterprets that as "order of magnitude" not "out of memory"00:40
poolieperhaps i'll look at his client code someday00:41
=== al-maisan is now known as almaisan-away
poolielifeless: your shoes are now available to be filled <http://webapps.ubuntu.com/employment/canonical_BSE/> - help me find someone good?00:45
lifelesspoolie: certainly.00:46
lifelessHave you tweeted yet ?00:46
pooliei can do that00:46
spmwhy the GA tracking codes? (utm_* are google analytics tracking codes. fwiw)01:00
lifelessspm: it was in my browser bar01:01
lifelesscopy-paste, you think I read these things?01:01
spmfair enough :-)01:01
lifelessdoes LoginToken:+validategpg talk to the gpg servers?03:08
* lifeless is guessing it does03:08
lifelessI wonder, should oauthnonces go in the sessiondb03:12
cr3hi folks, I created a couple projects recently without anything in them yet. would it be simpler to ask to rename them, or create another couple projects and ask to remove them?03:13
james_w`poolie: lp in desktopcouch> standard facility where?03:13
lifelesscr3: rename should be easy enough03:13
lifelesscr3: unless you have mailing lists or ppas03:13
pooliejames_w`: well, if i ever get around to it, i would look about writing an apis-couch daemon03:13
poolieeventually, and this is utter pie-in-the-sky, it would be cool to just have something like bugs.launchpad.net/bzr/+couch03:14
poolieand to through that means get a whole copy of them03:14
james_w`apis-couch? what would that do?03:14
lifelessjames_w`: map lp into couch?03:15
cr3lifeless: nothing yet, so can I ask in the channel or is there a preferable avenue?03:15
lifelessCHR should be able to help in #launchpad03:15
lifelessfailing that follow the instructions in the topic there.03:15
james_w`lifeless: well, I have a project already that doesn't need new daemons etc.03:16
cr3"CHR"? is that a nick or an acronym I'm not familiar with?03:16
james_w`I'm interested in whether poolie has other ideas03:17
cr3lifeless: by the way, might you have a moment to chat about the progress of the results tracker?03:17
pooliejames_w`: i have very few ideas ;) other than that having it in a local db that can sync smartly with the server would be nice03:17
pooliewhat's your project?03:17
=== nigelb_ is now known as nigelb
james_w`poolie: it's almost transparent to the app whether it is talking to LP or a couchdb copy of the data. The non-transparent part is that if talking to couch it has extra capabilities to push stuff back to lp03:18
pooliereally, wow03:19
pooliejames_w`: i'll check it out03:23
james_w`I'd love some help getting it up to standard03:24
james_w`it's not got full coverage of the capabilities of the API yet03:24
james_w`rolling it in to my attempt to write a twisted API library wasn't the best thing03:25
lifelessjames_w`: I'd love to see that split out03:27
lifelessjames_w`: there is a use case in LP for a txlaunchpadlib03:27
lifelessjames_w`: but we wouldn't want the couch stuff mixed in03:27
james_w`lifeless: the code is independent at that level03:28
james_w`lifeless: but the couch feature relies on the tx code as I have written it, so people can't play with that until they use twisted03:28
james_w`(and couch is only one possible implementation of the required interface, it just so happens that a JSON-based document store is kind of handy for this)03:29
cr3lifeless: sleep time, I'll catch up with you another time about the results tracker. cheerio03:30
lifelesscr3: ciao03:30
pooliethe latency is so high doing it on twisted would be good03:30
pooliejames_w`: so your code can use couch as a cache?03:32
james_w`poolie: "cache", yes03:32
james_w`poolie: you talk to LP and it sticks the documents it gets back in to couch before returning them to you03:33
james_w`poolie: at any time you like you can reconfigure the client to talk directly to couch, and you will get those documents back again.03:33
james_w`that's the read-only part03:33
james_w`then you can make changes, and it will store the modifications, and give you the updated information if asked for it again03:34
james_w`then you can ask it to iterate the modifications and send them back to LP, and the collision detection will naturally act to prevent problems there03:35
poolieso this just all works over the existing restful protocol?03:35
james_w`there are still a bunch of things that need work, and I'm not sure whether the approach taken will ever get us to 100%, but it does have an elegance03:35
james_w`poolie: yep03:36
nigelbisn't this what someone demo'd at last UDS?03:36
james_w`poolie: with a way to replace that restful protocol with queries in to couch03:37
james_w`nigelb: yeah, me, very shoddily03:37
nigelbjames_w`: lol, laptop not working et al ;)03:37
* nigelb hugs james_w` :)03:38
mwhudsonjames_w`: now do this for notmuch pls03:44
james_w`mwhudson: one day03:46
james_w`though I think I should try and add more moving parts next time03:47
jtvwgrant: something I don't get… I'm to keep a bfj fk in my new TranslationTemplatesBuild table—but where does it come from?  I see build_farm_job being set all over the place, but that all refers to BuildFarmJobOld stuff.03:57
* poolie tries txrestfulclient04:22
poolieand whacks in to bug 46135604:22
_mup_Bug #461356: desktopcouch-service crashed with ImportError in <module>() <apport-crash> <i386> <ubuntu-unr> <desktopcouch (Ubuntu):Incomplete by cmiller> <https://launchpad.net/bugs/461356>04:22
lifelessspm: suppose we could zap the first 8 months of successful-updates.txt?04:35
spmsure. one sec04:35
lifelessspm: of this year!04:35
lifeless[I think it goes back forever, so keep a copy04:36
lifelessbut its getting a tad large]04:36
lifelessspm: also, whats it up to?04:36
spmsuccessful-updates-2008.txt (already existed) also now have successful-updates-2009.txt04:38
lifelessis there a system load average python module already, I wonder04:39
spmimport system.load.averages ?04:40
spmwhich is spm doing the equivalent of import icanfly04:40
mwhudsonlifeless: os.loadavg()04:44
mwhudsoner no04:44
lifelessbug https://bugs.edge.launchpad.net/oops-tools/+bug/243554, freshly updated04:46
_mup_Bug #243554: oops report should record information about the running environment <infrastructure> <oops-tools> <Launchpad Foundations:Triaged> <OOPS Tools:Triaged> <https://launchpad.net/bugs/243554>04:46
lifelessI wonder if time.clock() is pid wide or pid wide :P04:48
mwhudsonlifeless: one of those was supposed to be thread?04:52
lifelessmwhudson: being droll about linux clarity in this area04:53
lifelesswe can use clock_gettime(CLOCK_THREAD_CPUTIME_ID) though04:53
jtvlifeless: would you mind if I just made that Librarian change now?04:54
lifelessjtv: I don't mind when you do it :)04:55
stublifeless: I've just pulled in the information from /proc before04:55
jtvlifeless: :)04:55
lifelessjtv: if you mean ...'and sneak it in the release', that would be risky, wouldn't it?04:55
jtvlifeless: that would be, and it's not what I had in mind.  Thinking more of avoiding being the subject of future "what flaming idiot made this horrible change!?" inquiries04:56
lifelessjtv: I'd hope noone in the team would take that attitude following something up04:57
lifelessjtv: and the only risk I know of, is the one I mentioned: the librarian db access is currently very tightly encapsulated; it needs to stay that way.04:57
jtvWell, so to speak.  The thing is, I'm not 100% happy about making an API where you pass "either an object or its id."04:57
lifelessjtv: so don't do that.04:57
lifelessjtv: make a separate pass-the-object API (perhaps on the object :P)04:58
lifelessand have the current id based function delegate04:58
jtvNow, what is the reason for the tight encapsulation?04:58
lifelessbecause its in twisted04:58
lifelessso its called via deferToThread04:58
* jtv likes reasons—easier to remember than rules :)04:58
lifelessit can do DB access in the thread04:58
lifelessit cannot outside of it, or all other requests in progress will block.04:59
jtvI thought it ran as a separate process?04:59
lifelessjtv: if you aren't touching code used in the librarian /server/ this won't matter - but I don't know exactly what you're touching (and be sure to check for imports :))04:59
lifelessjtv: the librarian is a twistd process, yes.04:59
jtvlifeless: I'm touching stuff in canonical.launchpad.librarian and canonical.librarian.client, but nothing in server.05:00
lifelessin the process it has a mainloop, and worker threads; the worker threads do DB access, the mainloop does all the business logic (except DB access)05:00
lifelessjtv: the server is in canonical.launchpad.librarian05:00
jtvSo client.py is sort of exceptional in there?05:01
jtvI mean, FileDownloadClient does run client-side, right?05:01
lifelessit might be nice to have the twisted code more visually distinct (e.g. in a submodule, or move the client to lp.services.librarian.client, or something.05:02
lifelessI don't want to make the scope bigger on you :)05:02
jtvI'm not going to do anything along those lines, no.  :)05:02
lifelessso, to answer your question, I presume so, but I'd need to check.05:04
lifelessAll I'd advising is a little caution and investigation in this area, as we have two dramatically different programming models in play here, and they mix poorly.05:04
lifelessspeaking of which.05:04
lifelessspm: are were there yet ? its been an hour.05:05
lifelessjtv: the speaking of which was a joke.05:05
lifelessjtv: the line before wasn't.05:06
jtvThat much was clear.  The joke I still don't get.05:06
lifelessoh, it wasn't a very good one05:06
lifelessit was leading into the spm: line05:06
jtvlifeless: still, better to have the other shoe dropped :)05:07
lifelesstaking a breather; I'll be back to heckle spm later05:13
jtvlifeless: I think there's a better solution for the librarian problem: it's a bad internal distribution of responsibilities.  In _getPathForAlias, the LFA is loaded _only_ to determine that it's visible.  The actual work doesn't involve the object at all.05:13
jtvnm; take your breather05:14
EdwinGrubbspoolie: ping05:15
pooliejames_w`: a teeny patch for you05:16
poolieEdwinGrubbs: hi there05:16
EdwinGrubbspoolie: I have some questions about the preferred way to use the apport format. The oops currently groups the request variables together, but it seems cleaner to use email.message.Message than to use another ProblemReport to make a hierarchy, so that I don't end up with multiple Date and ProblemType fields.05:24
EdwinGrubbspoolie: I also wondered if I should use the Stacktrace field for python stacktraces, or if it would be better to only use that for stacktraces created by gdb.05:25
poolieEdwinGrubbs: you can look at bzrlib.crash to see what we do05:26
pooliewe use Traceback for the python traceback05:26
pooliewhich i think is consistent with what other python programs use05:27
pooliei would probably have one thing RequestVariables containing all the variables05:27
poolieeither just as they are in the url, or decoded05:27
EdwinGrubbspoolie: oh, request variables in the oops actually is all the cgi variables like HTTP_REFERER, REQUEST_METHOD, etc. and not just the query string. I saw in the apport file format pdf that some of the hierarchical variables are stored as "name=value", but it seems more consistent to me to use "name: value". I'm trying to decide whether to use email.message.Message.as_string(), or ProblemReport().write(StringIO()) to create a05:32
EdwinGrubbs hierarchy.05:32
poolieso you agreed with gary to do it in apport now anyhow?05:33
pooliei'd probably just pprint a python dict05:33
spmlifeless: looks like it came back about 5 mins ago05:43
EdwinGrubbspoolie: well, I spent today determining how easy it would be to do now. I just saw Gary's email that he preferred to do it in the long term. However, my original solution, was almost identical to ProblemReport.write_mime() except that I don't base64 encode things and that it handled the special case of the request variables hierarchy. I really think we should use apport now. oops-tools will still be able to process the old oops05:44
EdwinGrubbs format, so the differences in the apport format don't cause any implementation problems.05:44
poolieEdwinGrubbs: so they're not really a ahierarchy, are they?05:49
pooliei mean it's just a dict of strings05:49
EdwinGrubbspoolie: right, I just meant that the whole problem report is a hierarchy, since the request variables contains multiple values.06:01
pooliethe simplest thing that could work is to pprint an array or put them in urlencoded06:05
poolieistm that using email formatting would be complicated, might break, and wouldn't help06:06
poolieditto nested appotr06:06
jtvwgrant: seems I needed to create the BuildFarmJob from the factory for my custom job type.  Passing tests again.06:10
lifelessspm: ok cool06:22
lifelessspm: so, can we enable profiling, and make the load be good ?06:22
SpamapSlifeless: hey, what ever happened with SSL improvements? Was just reviewing past threads..06:24
lifelessSpamapS: theres an RT ticket open to increase the cache length06:28
lifeless(for idle keys)06:28
lifelessand theres another open to get me access to the DC apache front end over a VPN + HTTP; I can then test a FE SSL here06:29
SpamapSah cool. I have used distcache for mod_ssl in the past to great effect before btw. ;)06:29
lifelessI'm not sure if we have dual apache or not06:30
lifelessI suspect not06:30
lifelessjtv: uhm, doesn't the name from the the LFA too ?06:31
SpamapSwow distcache's last release was in 2004 .. man its been so long since I setup an actual SSL server .. got BigIP's to do it a while back and have just been soft on SSL ever since. ;)06:31
jtvlifeless: yes, as usual I saw my mistake right after I said it—but no reason to keep you at the time.  ;)06:31
jtvlifeless: can there be thread/process boundaries in this call chain that I would not see at all?06:32
pooliewell, the txrestfulclient hello world passes again06:33
pooliethat's something06:33
pooliebut also probably enough for now06:34
lifelessjtv: deferToThread in the librarian is the call boundary06:36
lifelessjtv: when it returns from the callable supplied to that function, it comes back across the thread.06:36
jtvlifeless: I don't see that happening anywhere in the call chain from the first fetch of the LFA to the redundant second fetch—I guess that means that it's safe to re-use the same LFA object.06:38
spmlifeless: is back in profling mode06:43
spmpro-fling. hrm. maybe not quite. profiling tho....06:43
lifelessspm: and hows the load ?06:45
spmdropping. 1 3 4 atm06:45
lifelessspm: is it running with/without the patch ?06:46
spmgood question...06:46
lifelessspm: and are background jobs still disabled ?06:46
spmyup; just nowish.06:46
lifelessok thats cool06:46
lifelessspm: have they been killed :P06:46
spmoh yes. legit excuse to kill cronjobs off? opportunities like this are rare and to be enjoyed post haste!06:47
lifelessok, profiling things now06:47
lifelessfirst off, bugtask:+index06:48
lifelessand now ubuntu/++assignments06:49
lifelessok, 7 seconds rather than 1606:50
lifelessspm: the patch I put up the other day06:50
lifelessspm: can we put that on again?06:51
spmdo you have a paste handy?06:51
spmsaid file has been overwritten since06:51
lifelessone sec , finding/making06:51
lifelessspm: ^06:56
spmsorry - was horribly distracted doing the push code for the release; and made the shocking discovery that praseodymium does NOT have sl installed.06:57
spmnaturally this is a critical/urgent problem and moves to rectify needed to be made.06:57
lifelesssysadmin porn comes first, of course.06:58
lifelessdarn, I typed that :(06:58
* SpamapS cannot disagree with that06:59
* SpamapS prepares a petition to have sl added to the server cd seed07:00
spmthere. nicely taken wildly out of context.07:03
* spm bows at the appreciation07:04
spmthe fine art of context free quoting - choosing the title07:04
SpamapSdamn, seems somebody beat me to it. ;)07:05
spmlifeless: restarting with the patch; give it a few07:05
SpamapSspm: your title was better than mine. :)07:05
SpamapSwell done07:05
spmblink. something crash nicely on the restart07:06
spmwow. something is really not right here...07:09
spmoh ffs. it's doing a staging rollout AGAIN! aARGH07:09
* spm grumps off and puts in the lock file on sourcherry.07:10
spmi've killed the crontab entry as a savage "don't do that" for now. I'll see if I can manually get the app server on asuka back to right'n'goodness07:12
wgrantjtv: Sorry, I'm not completely down with the latest implementation details.07:15
jtvwgrant: I think I've done all I know I should do… question now is: what next?07:15
wgrantjtv: You have BuildFarmJob rows now?07:16
jtvI had to create them myself, which from what I see elsewhere seems to be the way.07:16
wgrantI suggest talking to noodles.07:16
wgrantHow much do you still depend on BranchJob?07:17
jtvI haven't gone through the details, but I thought it depended mainly on how much the dispatch machinery still depends on TranslationTemplatesBuildJob.07:18
jtvI mean, there's not much in there other than methods the build farm needs.07:19
wgrantOK, so you've probably done your bit for now, but talk to noodles.07:19
jtvIf the build farm can start dispatching TranslationTemplatesBuilds instead of TranslationTemplatesBuildJobs, I'm either there or very close.07:19
wgrantThat's the next step.07:19
jtvYes, I will definitely talk to him once he appears—he also promised me a review this morning.  :)07:19
wgrantAh, excellent.07:20
jtvI'm sort of eager to start cleaning out the old stuff, and sort of not looking forward to it at the same time.  :-)07:21
lifelessspm: and the conclufsion is?07:32
spmam working on getting that to argh <== atm07:33
lifelesscan you kick the profile rsync in the interim?07:33
spmso kicked07:34
spmtry to start the patched and profiling "new" version...07:36
spmit's still "starting"....07:48
spmlifeless: wooo. it's started. have at it.07:50
lifelessspm: load is still low ?07:55
lifelessspm: and does it have both patches, or only the query changing one?07:55
lifelessspm: please kick the profile rsync - thanks07:58
spmkicked and very low, 0.22 0.35 0.7107:59
lifelessspm: which patch(es) did it have?07:59
spm lib/lp/blueprints/model/specification.py and the profiling on08:00
lifelessuhm, both patches change that file :P08:00
lifelesshave  alook08:01
lifelessdoes it change the column definitions08:01
spmone sec. just trying to stop a db from faceplanting08:01
lifelessor the query08:01
spmlifeless: appears to be this one at a cursory glance at the first few lines: http://paste.ubuntu.com/489589/08:07
lifelessok, could you appyly the other as well ?08:08
spmheh sure, you have a paste handy?08:11
lifelesshang on while I check the backlog08:13
spmjust doing about 17 bazzilions things at once atm.08:14
StevenKspm: Like notmal08:14
StevenKEr, normal08:14
lifelessspm: ^08:14
spmgimme 5-10; just need to disable.notify a bunch of things in prep for the release in 45.08:14
adeuringgood morning08:22
=== spm changed the topic of #launchpad-dev to: Launchpad down/read-only from 0800-1100 UTC for a code update | Launchpad Development Channel | Week 3 of 10.09 | PQM is CLOSED | firefighting: - | https:/​/​dev.launchpad.net/​ | Get the code: https:/​/​dev.launchpad.net/​Getting | On-call review in irc:/​/​irc.freenode.net/​#launchpad-reviews
lifelessspm: I'll check back in regularly till you say its done... I can see you're busay08:41
* lifeless starts coming up with ideas to make rollouts even shorter08:46
adeuringlifeless: did you talk with deryrck about my theory of the cause of the still remaining problems of the apprt retracer? that the librarian is simply queueing request from the app server for too long?08:49
spivadeuring: it was discussed; IIRC it was diagnosed as missing firewall rules for some app servers08:52
adeuringspiv: intersting.... do you have any details?08:53
spivadeuring: see the #is and/or #launchpad-code logs from about 7 hours ago08:54
adeuringspiv: thanks!08:54
lifelessadeuring: hi08:58
adeuringhi lifeless08:59
lifelessadeuring: tcp connect timeout default is 30 seconds IIRC (you need to wait for the MSS, again IIRC)08:59
lifelessif the librarian was doing that with 5 concurrent requests we'd be sunk, also its careful to do lots of incremental bits of work08:59
lifelessso I'd expect a-diskio * 4 peak slowness, - a second or two tops - not 3009:00
lifelessadeuring: which is why I looked elsewhere09:00
lifelessadeuring: now, 4 is the number of threads our appservers have09:00
lifelessso 5 spilling you into a new appserver was a reasonable assumption :)09:01
elmoI'm changing our firewall rules so that we REJECT rather than DROP cross-site requests to make diagnosis of this kind of thing easier, FWIW09:01
lifelesselmo: thank you!09:02
adeuringlifeless: just tried my scrpit with 5 concurrent requests -- looks a bit better09:02
lifelessadeuring: a bit?09:02
adeuringno errors09:02
lifelessthats a lot better then :P09:02
adeuringlifeless:i think we should do more logging of what happens on the librarian server,09:03
poolielifeless: i've been reading a book ian gave me 'prefactoring'09:03
lifelessadeuring: We should strike a balance between non and too much... note that the librarian does OOPS as of this rollout (or was it last one)09:04
poolieit's a bit basic but it has some nice suggestions along the lines of your guide there09:04
lifelessI think QA haven't added it to the daily reportcard yet.09:04
lifelesspoolie: intereseting09:04
lifelesspoolie: can I borrow it @ UDS ?09:04
poolieif you remind me several times closer to the date :)09:04
lifelesspoolie: is this close enough?09:05
lifelesspoolie: how about now?09:05
spmlifeless: applied that 2nd patch as well; restarting now09:05
pooliegood night :)09:05
lifelesspoolie: :P09:05
lifelesspoolie: I'll remind you just before we go09:05
adeuringlifeless: well, I think the issue is not necessarily a bug in code or anything -- just that the librarian can't handle requsts fast enough09:05
lifelessadeuring: I'm not aware of issues like that09:05
lifelessadeuring: or data suggesting we have them; certainly I agree that we *need to be able to diagnose such things*09:06
bigjoolsmorning all09:06
lifelessand if the logs are insufficient, we should increase them till they are.09:06
adeuringlifeless: right09:06
lifelessadeuring: we're now logging librarian times in the appserver for downloads; we can add uploads easily09:06
lifelessfor uploads we should also add the size, perhaps in the closing bit of downloads too09:07
adeuringlifeless: ok, that would help. but i suspect that these connection timeouts are caused by the librarian, not the app server09:07
lifelessadeuring: which timeouts?09:07
lifelessadeuring: if you mean the ones apport has been having, that show up as 500 errors from the API with timeout to mizuho in them...09:08
lifelessadeuring: they were a firewall09:08
spivadeuring: the evidence I've seen suggests the librarian server is coping just fine09:08
spivadeuring: why do you think otherwise?09:08
adeuringlifeless: well, my little script causes them just fine even now09:08
lifelessadeuring: it does?09:08
adeuringyes, if it starts 8 concurrent requests09:09
lifelesscan you pastebin the appserver trace?09:09
adeuring5 seems to be better09:09
lifelessor does it seem to be identical?09:09
spiv"just now", you mean during a rollout? ;)09:09
lifelessadeuring: oh hang on. lollolllollololl09:09
lifelessadeuring: launchpad is down. Readonly mode. Zer iz no upload possible because the librarian is switched off09:10
lifelessor meant to be; if you're successfully uploading there is something really wrong.09:10
adeuringlifeless: yeah, ok, that's another possible cause09:10
adeuringbut... when exactly was the rollout started?09:10
spivAbout 10 minutes ago.09:10
lifeless11 minutes ago09:10
adeuringok.. hard to be sure then when exactly i ran the script again....09:11
adeuringok, let's try again once the rollout is done09:11
lifelessif you can provoke a connection timeout error, its a definite bug.09:11
lifelessMy first reaction is to look elsewhere than the librarian09:11
spivadeuring: so, connection timeouts are really unlikely to be a problem in the librarian server in my opinion09:12
lifelessfor a bunch of reasons.09:12
lifelessbut we can't exclude it; lets just keep the net broad.09:12
adeuringspiv: well, we _could_ see them09:12
lifelesse.g. today we found a concrete problem with the firewall config09:12
lifelessadeuring: no, you saw a firewall.09:12
lifelessadeuring: we know for *some things* it was definitely *a firewall*09:13
adeuringlifeless: what firewall issue was it?09:13
lifelessadeuring: 2 appservers are in a different datacentre.09:13
spivadeuring: it's a Twisted server that ought that ought to always be accepting connections rapidly; if connections are not being processed by that daemon in a timely fashion the only likely cause is that the librarian's host is totally swamped for disk IO09:13
adeuringahhh, so they could not access the librarian?09:13
lifelessadeuring: the firewall rules for them did not include the restricted upload port, which is what the appservers connect to to upload restricted files.09:13
lifelessadeuring: the firewall rules dropped the packets as hostile, and so at the network layer it looks like the librarian /machine/ is missing.09:14
adeuringlifeless: ah, ok, that looks like a real problem....09:14
spivadeuring: so a failure to connect() from another machine strongly suggests problems in something other than the librarian daemon.09:14
lifelessadeuring: but there may be other problems.09:15
lifelessadeuring: however, we *know* there was *a* problem, that would cause the symptoms seen before09:15
spivThat doesn't make it impossible, of course, but it does mean that assuming the daemon is the most likely source of the problem is likely to be the wrong approach.09:15
adeuringyes, i understand09:16
bigjoolslifeless: are you familiar with the distroeries +queue page?09:24
bigjoolsdistroseries, even09:24
lifelessI seem to recall it showing up on slow-pages reports09:24
bigjoolsit's the bane of my life09:25
lifelessanyhow, I'm not intimately familiar with it09:25
lifelessbut lets pretend I am09:25
bigjoolswell it's mainly intended to let archive admins move uploaded packages into the accepted state09:25
bigjoolsif they got held for some reason09:25
lifelessrighto, its the aa review queue09:25
bigjoolsthis is normally done in zopeless scripts if it's auto-accepted09:26
bigjoolswhen accepting packages we also close lots of bugs and email people09:26
bigjoolsand this is where the trouble arises with that page if any of the objects are private09:27
bigjools(not to mention the query load)09:27
lifelessand email load09:27
lifelessthat really needs to be out of appserver anyway09:27
lifelessI'm so excited09:27
bigjoolsanyway, I have a bug where it's OOPSing occasionally for someone when it tries to access private email addresses09:27
lifelessshould be able to point really clearly at email perf tomorrow09:27
lifelesswe'll have failed convertToQuestions, I'm sure.09:28
bigjoolsI am wondering if it's acceptable to remove the security proxy in carefully defined situations09:28
lifelessso, why does it try to access their email address?09:28
lifeless[clearly its ok to do that in carefully defined istuations09:28
lifelesscode exists to serve us, not the other way around; but if we can avoid it its somewhat nicer.09:28
bigjoolsit's trying to email potentially private addresses as part of a) upload notification, b) bug notification09:29
bigjoolsall done under the permission of the webapp user09:29
lifelessnow, to avoid disclosure that has to be part of the BCC right ?09:29
lifelessor a direct mail09:29
bigjoolslong term we need to jobify it of course09:29
bigjoolsdid I just make up a word? :)09:29
wgrantI don't think that's the private email address problem.09:30
wgrantIIRC it dies (possibly correctly) when trying to include it in Changed-By or Signed-By in the announcement email.09:30
lifelessI'd suggest having some method that you pass to the Person asking it to do the bit thats private09:30
wgrantBut I said that in the bug... let's see..09:30
lifelessyeah, I'd say thats correct.09:30
bigjoolsthat would be a) as I said above09:30
wgrantbigjools: There's no problem emailing to them, though.09:31
wgrantIt's including them in the email that's the problem.09:31
wgrantOh, no, other way around.09:31
wgrantThis is confusing.09:31
wgrantSo it must already rSP in places.09:31
wgrantI said in the bug:09:31
wgrantI believe it only fails if it would send a notification to the private09:31
wgrantemail address; using the private email address in the email (eg. if the09:31
wgrantperson is deactivated) seems to work fine.09:31
bigjoolsI would have thought any access of a private email would blow up09:32
wgrantYeah, it does.09:32
wgrantSo it's rSPing already.09:32
lifelessbroad brush strokes09:32
lifelesshere is how I would tackle it.09:32
lifelessI would ensure that Person is responsible for deciding when to/not to bypass the proxy09:32
bigjoolsremoveSecurityProxy does not appear in the queue.py file09:33
bigjoolsso if it's used it's elsewhere09:33
bigjoolslifeless: I'm not sure it knows enough to do that  does it?09:33
lifelessbigjools: add methods ;)09:33
bigjoolseugh :(09:33
bigjoolsPerson is already bloated09:34
lifelessmultiple places may want to be able to send an email09:34
lifeless*to* someone, with only one recipient09:34
lifelessthats reasonable to bypass the proxy -in that case-09:34
lifelessgrabbing a private email to put into a template for announcements isn't ok though.09:35
bigjoolsthat was the point I was going to make09:35
lifelessas long as folk choosing to use the method won't be confused or guided into making the wrong choice, I think its ok to do it anywhich way..09:35
bigjoolsmaybe we should refuse uploads on people who have private email addresses?09:36
lifelesswhy are we looking up their email (vs using the one in the changes file itself) ?09:36
bigjoolsit uses the preferred email09:37
lifelessso if I upload as noddy@example.com09:37
lifelessbut my preferred mai is fred@demo.com09:37
lifelessthe .changes file is regenerated with fred@demo.cpom ?09:37
bigjoolsnothing's regenerated09:38
lifelessok, so why are we looking up their email?09:38
bigjoolswe put email addresses on the email template09:38
lifelesswhats the template file09:38
lifelessit will be faster than 20 questions :)09:38
bigjoolsto, y'know, send it :)09:38
bigjoolsman it's been 3 years since I looked at this code, hang on09:39
lifelessbigjools: no, I don't understand why we need their email09:39
bigjoolsit uses changed-by, maintainer and signer09:39
lifelessthe approver did the approving; we need their Name; the uploaded uploaded it, we need their Name. Neither sent the mail, so we don't need their emails, and we send separate copies per recipient, so thats fine too.09:39
bigjoolslifeless: the current format was arrived at after extensive discussion with the ubuntu guys09:40
lifelessI think we need to involve them then.09:40
bigjoolswe need to re-create the problem first09:41
lifelessI think its incompatible to both have 'you can have your email address private in launchpad' and to be putting it in mails sent to other people.09:41
lifelesscertainly a test case will help09:41
bigjoolsyes, exactly09:41
bigjoolsI also don't understand why a preferred email address would be private09:42
lifelessits their only email ?09:42
bigjoolsactually it's all or none isn't it?09:42
lifelessI think so09:42
bigjoolsso trying to hide your email address while doing public works seems...odd :)09:43
lifelessfolk are very worried about spam09:43
lifelesschanges files go to a public list.09:43
bigjoolswe could put <private email> on the template09:43
lifelessfor instance, yes.09:44
lifelessOr an LP account url, or the SSO persistent url.09:44
bigjoolsbut the To: can't be hidden09:44
lifelessbigjools: the To: shouldn't be them anyway ?09:44
bigjoolserr From:, sorry09:44
bigjoolsactually I can't remember09:44
bigjoolsI think the uploader is CCed from memory09:44
lifelesswhen I read that template, it doesn't look like it would make sense to be 'from them', because its 'thanking them'09:44
lifelessbigjools: I have a suggestion: register a UDS spec for this.09:45
lifelessbigjools: it needs it.09:45
bigjoolswell, there's quite a few people involved in a package09:45
lifelesswell, maybe it doesn't, but I think the driving forces are complex enough it will benefit from that scale of analysis and discussion.09:46
bigjoolsI'm not comfortable with the analysis yet09:46
bigjoolsonce we get a failing test case (and when I can see the oops that might help) then we can decide where to go09:46
lifelesslets get a autoated test09:46
lifelessif I can help further I'd be delighted to do so,09:47
bigjoolsgreat, thank you09:47
lifelessbut I suspect that its going to run into a definitional problem very early on rather than a code problem; and for that the stakeholders... have to hold their stakes.09:47
bigjoolsdipped in silver nitrate?09:48
lifelessspm: hah just saw you put th epatch on... trying10:23
lifelessits gone already...10:23
lifelesswill try tomorrow10:23
lifelessbigjools: I bet that  https://launchpad.net/ubuntu/+search (Distribution:+search) will be your top timeout this cycle.10:38
lifelessanyhow, I'm going to be looking at when I get up :)10:41
lifelessadeuring: try now10:42
wgrantNow, let's see how badly Soyuz breaks on Lucid...10:47
bigjools /o\10:47
bigjoolsI love your optimism10:47
wgrantOh, code upgrade's done already too? Nice.10:47
lifelesswhee things feel sluggish10:47
=== elmo changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 3 of 10.09 | PQM is CLOSED | firefighting: - | https:/​/​dev.launchpad.net/​ | Get the code: https:/​/​dev.launchpad.net/​Getting | On-call review in irc:/​/​irc.freenode.net/​#launchpad-reviews
wgrantbigjools: Is optimism ever a good idea when Soyuz's fragility is involved? :)10:48
bigjoolshey it's a lot better than it used to nbe10:49
wgrantIt is.10:50
wgrantBut buildd-manager will still break if you think about it the wrong way.10:50
wgrantAlthough I guess that's not technically Soyuz any more.10:50
bigjoolsI need to set up a new project for it10:50
bigjoolslaunchpad-buildmaster has a nice ring to it10:50
lifelessanother project?10:52
wgrantbigjools: launchpad-buildfarm, surely?10:52
wgrantbigjools: This reminds me...10:53
bigjoolsno, buildmaster, it's specifically for the master sude10:53
wgrantHave you ever seen production or DF b-m start failing to dispatch builds, complaining that the XML-RPC build command is being given too many arguments?10:53
wgrantI've seen it locally, and had similar reports from a couple of others' local setups.10:54
wgrantAnd I partly tracked down why it happens.10:54
wgrantBut I don't know what triggered it... and I'd never heard about it happening anywhere prod-ish.10:54
spivSweet, codebrowse OOPSes appear to work.10:55
wgrant(the problem is that BuilderSlave is broken now, but we only use RecordingSlave... except for in some circumstances that appear sometimes locally, which are not entirely clear)10:55
lifelessspiv: you got one?10:56
spivBe one of the first to generate your very own codebrowse OOPS: http://bazaar.launchpad.net/~bzr-pqm/bzr/bzr.dev/annotate/head%3A/README210:56
lifelessspiv: but can you see it on lp-oops?10:56
wgrantThat's a little more friendly than the old page10:56
spivlifeless: I haven't tried yet10:56
lifelessspiv: acid test man ;)10:56
adeuringlifeless: one test run with 8 parallel uplods: one failed with a "connection timed out"10:56
spivlifeless: but even that page beats "Internal server error"10:56
lifelessadeuring: whats the oops?10:57
adeuringwait a second...10:57
spivlifeless: so far the OOPS isn't on lp-oops10:57
lifelessspiv: it may need some follow up10:58
lifelessspiv: with QA10:58
spivWhat's the typical delay for syncing?10:58
lifeless - needs to be added to the lpnet summaries.10:58
lifeless - needs to be added to the list of dirs to scan for the oops db scanner10:58
adeuringlifeless: problem is that we don't get an OOPS10:58
lifelessspiv: 3m I think10:58
lifelessadeuring: what do we get ?10:58
lifelessadeuring: if its apis check the X-Launchpad-OOPS header10:59
adeuringjust the error message "connection time out", my script doesn't print it10:59
lifelessadeuring: (I think that is where the id goes)10:59
adeuringok, I'll try to find it10:59
lifelessadeuring: to debug this we need:10:59
lifeless the backtrace10:59
lifeless the appserver it happened on10:59
adeuringi know10:59
lifelessit may be a further firewall issue on the other appservers.11:00
lifelessor something.11:00
adeuringso, how can I figure out which app server is used?11:00
lifelessanyhow... late here. If you can get the appserver + error, ask the GSAs if they can confirm that appserver has access to the restricted upload port11:01
lifelessadeuring: its in the OOPS :)11:01
adeuringah, ok11:01
lifelessI'm quite sure we generate one, just goes into a header from what gary was saying th eother week11:01
lifelessadeuring: you might like to file a new private bug11:02
lifelessadeuring: unsubscribe everyone but you11:02
lifelessand then test on it.11:02
adeuringlifeless: yeah...11:02
lifelessif its not a firewall issue I will debug it with your script (if you supply it) and the data you gather overnight; getting the OOPS is critical path to solving it.11:03
* lifeless waves gnight11:03
wgrantjelmer: Hey, how's the branch? Were you able to reproduce the issues I reported?11:04
bigjoolsnn lifeless11:04
wgrantNight lifeless.11:04
noodles775Enjoy the rest of your evening lifeless11:04
jelmer'night lifeless11:04
jelmerwgrant, yeah, fixing + qa'ing at the moment11:05
wgrantjelmer: Great.11:05
bigjoolsso, my failure-detecting b-m hasn't failed anything yet.  Is it wrong to want to see that happen? :)11:05
wgrantSorry for throwing them at you so late... I wasn't aware until yesterday that the branch was targetted at 10.09.11:06
wgrantbigjools: Does it manage to distinguish between build and builder failures?11:07
bigjoolsthat's the plan, yes11:07
jelmerwgrant: Thanks for bringing it up in the first place. You saved quite a few people the stress and extra time that would've come with a broken rollout.11:08
wgrantjelmer: I have a few other issues with the branch from a more thorough review today, but I'm sure I've caused you enough trouble for now.11:09
wgrantNone are particularly major, I don't think.11:09
=== gmb changed the topic of #launchpad-dev to: Launchpad Development Channel | Week 4 of 10.09 | PQM is open for business | firefighting: - | https:/​/​dev.launchpad.net/​ | Get the code: https:/​/​dev.launchpad.net/​Getting | On-call review in irc:/​/​irc.freenode.net/​#launchpad-reviews
deryckMorning, all.12:04
=== mrevell is now known as mrevell-lunch
=== Ursinha-afk is now known as Ursinha
=== matsubara-afk is now known as matsubara
=== mrevell-lunch is now known as mrevell
=== almaisan-away is now known as al-maisan
wgrantWas cocoplum upgraded to Lucid?14:22
wgrantgrep-dctrl now chokes on maverick's Sources files for reasons which I cannot entirely determine.14:22
wgrantOoh dear.14:23
wgrantI think cocoplum's a-f cache might be pretty fucked.14:24
wgrantbigjools: ^^14:25
wgrantmaverick's main Sources is now ~3.4MB. From an out-of-date mirror, it's ~4MB.14:25
wgrantSome sections are truncated.14:25
bigjoolswgrant: known bug with lucid a-f14:25
bigjoolsbug 63396714:26
_mup_Bug #633967: apt-ftparchive generates corrupt Sources stanzas for .dsc files without Checksums-* fields <apt (Ubuntu):In Progress> <apt (Ubuntu Lucid):In Progress> <apt (Ubuntu Maverick):In Progress> <https://launchpad.net/bugs/633967>14:26
wgrantAh, great.14:26
bigjoolslife is never too easy is it14:26
wgrant(now, what were you saying about optimism a few hours ago? :P)14:27
wgrantbigjools: I bet it's the case-sensitivity stuff in changes file parsing.14:30
wgrantThe code looks for Launchpad-bugs-fixed, but it's normally spelt Launchpad-Bugs-Fixed.14:31
bigjoolswell remembered14:31
wgrantThe uploads referenced are fine.14:31
wgrantA ha ha.14:32
wgrantA test was changed to use the former capitalisation.14:33
bigjools(that's one way of making the tests pass)14:33
bigjoolswgrant: which test was changed?14:43
wgrantbigjools: See the last hunk of the rev...14:44
wgrantLet me find it.14:44
bigjoolswhich revno?14:44
wgrantAh, no.14:44
wgrantThe tests were already broken.14:44
wgrantsync-source was changed to use the bad capitalisation.14:44
bigjoolsit's because of the changed parser we're using now14:45
wgrantdb-devel r974114:45
wgrantIt is, yes.14:45
* bigjools wonders why that didn't break the test14:45
wgrantThe test packages probably use the bogus capitalisation.14:46
bigjoolsoh, I see14:46
* wgrant greps.14:46
bigjoolsyes it does14:46
* bigjools files a bug14:46
=== james_w` is now known as james_w
bigjoolswgrant: btw, isn't it way past your bedtime? :)15:09
wgrantbigjools: It's barely midnight.15:12
tyarussoUm, how would I set up an environment of db-stable instead of devel with rocketfuel-setup?  I tried changing all of the lines I could find in the script and still got a "devel" directory under lp-branches.15:33
tyarusso...or I could try reading the help output.  *headdesk*15:35
tyarussoWhere is the "etc/zope.conf" for disabling developer mode supposed to go, and how would one do that?16:08
tyarussoAlso, I can log in with the test account, but how do I register new accounts?  I don't see any links for that yet...16:10
leonardrasking a possibly stupid question rather than wasting more time. i've got a totally up-to-date system and 'make' is failing with a bzrlib import error, "cannot import name SAFE_INSTRUCTIONS". any help?16:27
leonardrgary, lifeless -^16:27
gary_posterdon't know leonardr but will try to dupe.  maybe try a code team or bzr team member after that.16:29
leonardrmaybe rockstar can help?16:29
gary_postertyarusso: register new accounts: not exposed on dev system.  our openid server is responsible for that in the production/staging.16:30
rockstarleonardr, update download-cache?16:30
tyarussogary_poster: Oh.  Well how is someone supposed to use it then?16:30
gary_postertyarusso: etc/zope.conf: see configs/development/launchpad.conf16:30
leonardrrockstar, up to date16:31
gary_postertyarusso: sorry, don't understand question.  dev build is for developers, which approximates production just enough to do dev style testing.  We're not in charge of making new users, so we don't expose it16:31
rockstarleonardr, oh, and sourcecode also needs to be updated.16:32
rockstarSAFE_INSTRUCTIONS should come from bzr-builder, which can't be eggified.16:32
tyarussogary_poster: Okay, my goal here is to set up a Launchpad instance that we could actually use for our company.  What other pieces would I need to get to accomplish that?16:32
leonardrrockstar: sourcecode/bzr-builder is up to date at revision 6316:33
leonardrhowever, revision 63 is from january. could it have moved?16:33
rockstarleonardr, hm, it should have.  abentley?16:34
gary_postertyarusso: oh, eek.  I'm sorry, that's a huge deal that we don't support.  We open source the code to improve our free service.  ...um, you could try asking the dev list and see if anyone there has advice?16:34
abentleyrockstar, I have just landed my permit-commands branch.16:35
leonardrrockstar: i've got the pqm-managed branch at bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/bzr-builder/trunk/16:35
abentleyrockstar, in that branch, the revno is 66.16:35
rockstarleonardr, ^^16:35
abentleyleonardr, the revno in that branch is 66, so you aren't up to date.16:36
tyarussogary_poster: Hmm, okay.  Meanwhile let's try a different direction:  1) Is the stuff you use for your openid server all open source too?  2)  Do you offer hosted services that could use our own domain names & branding?  If so, what's the fee schedule like for that?16:36
leonardrabentley: ok, i've got the new stuff16:36
tyarussogary_poster: Oh, I should note - we'd be interested in hosting both open-source and proprietary projects.16:38
gary_posterbac, are you around to address tyarusso's question # 2 above?16:38
leonardrok, i think i've got a bunch of out-of-date stuff16:39
bacmrevell: can you help tyarusso?16:39
tyarussohaha, support hot potato!16:40
bacah, tyarusso, i read through all of your question.  to answer #2, no, we don't offer branded hosting16:40
tyarussobac: Hmm, okay.16:40
tyarussoThe normal stuff might be better than what we have now, but wouldn't let us put everything all in one place, which would be ideal.16:41
gary_postertyarusso: for 1) the openid server is closed source.  ...I keep revising my answer to cut to the chase sufficiently...in sum, you'd have to branch the code to make it work, and it would be hairy; maybe you could get some community people interested, dunno.16:42
gary_postertyarusso: everything in one place: this isn't my part of story, and I have to run now, but (A) you could have a group that collects your projects and (B) I'm almost 100% sure it can contain both proprietary and open-source bits.16:43
gary_posterbac, mrevell, you can fix my reply if necessary :-)16:44
=== beuno is now known as beuno-lunch
m4n1shis there any way in the Launchpad API for search a bug?16:55
m4n1shbugs has only createBug and in bug there is nothing which matches searchBug or something like that16:55
deryckrockstar, did you ever get that widget moving with help from dav?17:01
rockstarderyck, nope.  Was hoping to get with you tomorrow about it.17:01
rockstarderyck, also, I just landed yui 3.2 into lazr-js, and that's got some more debugging happiness in it, so I thought I'd merge.17:01
deryckok, let's plan on it.  I'll try to poke at your code in my evening for home work. :-)17:01
gmbm4n1sh, Find the project you want to search on and then use project.searchTasks()17:10
m4n1shgmb: so this means global bug search is not possible17:10
gmbm4n1sh, Yes, but it's not possible in the Launchpad UI, either.17:11
m4n1shgmb: I think you can https://bugs.launchpad.net/17:12
m4n1shyou can choose "All projects"17:12
gmbm4n1sh, You're right; my bad.17:12
gmbI thought we used Google for that.17:13
m4n1shme too :)17:13
gmb(Launchpad Bugs developer doesn't know about Launchpad Bugs)17:13
gmbm4n1sh, But no, that's not available in the API at the moment, I'm afraid.17:13
gmbOf that I'm sure.17:13
m4n1shI could not find it in the API17:13
m4n1shactually I am presenting a talk on PyCon India 2010 on launchpadlib17:14
m4n1shso needed to know abuot this17:14
m4n1shgmb: your help is appreciated :)17:14
gmbm4n1sh, I think that's because the API is basically an export of our underlying object model.17:14
gmbAnd the UI sits on top of that object model, so it can do things the API can't.17:14
gmbm4n1sh, There's probably a bug for it (I'm on the phone now, otherwise I'd check for you) but feel free to file one if there isn't.17:15
m4n1shgmb: sure :) thanks17:15
=== matsubara is now known as matsubara-lunch
=== benji is now known as benji-lunch
leonardrgary, rockstar: (rockstar, i know this isn't your problem, but maybe you can help anyhow)17:40
leonardractually, let me check the wiki really quick, since the problem is probably that my dev process has gotten out of sync with launchpad's17:41
=== Ursinha is now known as Ursinha-lunch
leonardrgary, rockstar: no, i'm fairly certain i've done everything right, and i'm getting an import error from shipit17:54
leonardrfrom canonical.cachedproperty import cachedproperty -> "No module named cachedproperty"17:54
rockstarleonardr, oh, shipit? Uh, I have no knowledge in that area.  :(17:54
leonardrrockstar: yeah, i said it wasn't your problem17:54
rockstarleonardr, I thought we moved all the shipit import stuff to somewhere under lp though.17:55
gary_posterI saw that leonardr...I think I ran utilities/update-sourcecode.  You've done that too?17:55
tyarussoAll right, flailing e-mail sent to the mailing list - maybe someone out there has a clue :S17:55
leonardrgary: does rocketfuel-get not run update-sourcecode? i didn't run it specifically17:56
gary_poster:-) Good luck.  It won't be a core LP dev, I strongly suspect, and I'll be surprised if someone knows what to do, but who knows.  I'm sorry that the available options don't work for you. :-/17:56
gary_posterleonardr: yeah I think so17:56
gary_posterI mean, I think it does run it17:57
gary_posterit certainly should17:57
gary_posterI'll look...17:57
gary_posterit does17:58
leonardrwhen i run it manually i get a bzr repository conversion error!17:58
gary_posterin dulwich I bet17:58
leonardrno, in pygettextpo17:58
leonardrmaybe i should just remove sourcecode and get everything again? i don't think i've run this properly for a _long_ time17:59
gary_posteryeah ,maybe so leonardr.  I did surgery instead, myself17:59
gary_posterI found the broken ones and did a fresh branch in launchpad/lp-sourcedeps/sourcecode18:00
gary_posterof the broken ones reported by update-sourcecode18:00
gary_posteryour call18:00
=== al-maisan is now known as almaisan-away
cody-somervilleWho takes care of the svn import stuff? Can someone take a look at http://launchpadlibrarian.net/54208924/vcs-imports-django-trunk.log ?18:10
=== beuno-lunch is now known as beuno
=== deryck is now known as deryck[lunch]
rockstarcody-somerville, are there branches from this branch?18:16
cody-somervillerockstar, it looks like it but most are hundreds of weeks old18:17
rockstarcody-somerville, I suggest creating a new import, and maybe deleting or Abandoning that branch.18:18
rockstarIt's still using cscvs, which we're basically not maintaining anymore.18:18
cody-somervillerockstar, https://code.edge.launchpad.net/django <-- do you see how there are two series of the same name associated with lp:django? Is that intentionally possible?18:21
rockstarcody-somerville, whoa, that's weird.  I don't know if that's intentionally possible.  Maybe sinzui knows?18:22
beunocody-somerville, one seems to be off the project group, and the other off of the prohect18:22
cody-somervillebeuno, both are just normal projects18:22
rockstarbeuno, yeah, djangoproject should probably be deleted.18:22
rockstarEr, disabled.18:22
rockstarcody-somerville, can you file a bug about that, and I'll discuss it with the team this afternoon?18:23
beunothat is something interesting and new then!18:23
cody-somervilleIt seems possible to associate any branch in launchpad with a series18:23
=== benji-lunch is now known as benji
cody-somervilleOr no... if I try to edit the series in djangoproject I get an error18:24
cody-somervilleso I wonder how the value it has now got set... maybe via web services API if it has different error checking?18:24
=== matsubara-lunch is now known as matsubara
cody-somervillerockstar, bug #63428018:34
_mup_Bug #634280: Branch of another project associated with series <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/634280>18:34
rockstarcody-somerville, thanks18:34
cody-somervillerockstar, Is it possible to edit the URL for svn imports yet or does that still require a LOSA?18:35
rockstarcody-somerville, are you trying to edit the cscvs one?18:35
cody-somervilleno, new one18:35
rockstarcody-somerville, link?18:35
cody-somervillerockstar, https://code.edge.launchpad.net/~vcs-imports/django/trunk (I renamed the old import to old-trunk)18:35
cody-somervillerockstar, It looks like maybe the trailing slash is problematic on the URL18:36
rockstarcody-somerville, looks like the old branch needs to be deleted...18:36
cody-somervillerockstar, why?18:36
rockstarcody-somerville, the urls need to be unique.18:37
cody-somervilleYou can't have two imports with the same URL?18:37
rockstarcody-somerville, nope.18:37
rockstarI just pointed the old-trunk to django/trunkDISABLED18:37
cr3in launchpad, an account has a identity url and a person has a an account and a name, does that mean that multiple people can share the same account?18:38
cr3lifeless: so why doesn't he person contain the identity url instead?18:39
cody-somervillerockstar, it looks like it  failed again... :/18:42
lifelesscr3: account is there for shipit18:42
lifelessfrom a long time ago, nothing to do with openid18:42
lifelessit has grown and changed since then, but we're trying to delete the account table18:43
rockstarjelmer, see https://code.edge.launchpad.net/~vcs-imports/django/trunk - Does that make any sense?18:43
lifelessderyck[lunch]: when you return18:43
lifelesshave a look at this:18:43
cr3lifeless: awesome, makes sense. I was wondering when logging in with openid how it might demultiplex the various person objects in the event there was a one to many relationship18:44
lifeless'do not permit one' :P18:45
leonardrgary, how many packages are in your sourcecode/?18:49
gary_posterleonardr: 2018:49
leonardrgary: i've got 17, but i used to have 56. the difference between you and me probalby reflects the continuing trend of moving things out of sourcecode18:50
leonardri ask because i was still getting an error, but i think 'make' might fix it18:51
lifelessgary_poster: hi, I'd also like you to eyeball https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1713L71818:51
gary_posteryes, yay.  on call leonardr, but http://pastebin.ubuntu.com/491137/ fwiw18:51
lifelessgary_poster: don't interrupt your call; just look when you have a chance.18:51
gary_posterack thanks lifeless18:51
lifelesswe are making -huge- numbers of memcache calls - and they appear to be capable of blocking for 20ms18:52
jelmerrockstar: it works locally; it's hard to say much about it without the error from bzr-svn on lp19:13
rockstarjelmer, do we need better error logging?  Is that what you're saying?19:13
jelmerrockstar: it would be nice to have the bzr log output as well19:14
rockstarjelmer, okay.19:15
jelmerrockstar: Actually19:15
jelmerthis looks like a launchpad-code bug19:15
jelmerthe exception is being raised from the part of the code that fetches the existing import branch19:15
rockstarjelmer, I suspected it might be.19:15
jelmermy initial thought was that it was failing to open the svn branch using bzr-svn19:16
=== Ursinha-lunch is now known as Ursinha
=== deryck[lunch] is now known as deryck
deryckhi lifeless.  Looking at the OOPS report now....19:22
lifelessderyck: so basically we're spending - FWICT - about 2 seconds in memcache on that page19:23
lifelessderyck: that combined with the low hit rate we're seeing makes me strongly suspect that turning off memcache in the template/tales will take 2 seconds off that page.19:23
lifelesswe need to do some improvements to the oops aggregation facilities now that we're putting more data in them19:24
lifeless(I had to stare rather hard at the page to get a good sense of whats going on)19:25
deryckyeah, taking me a bit to process it too...19:26
lifelessso interestingly19:26
lifelessis in the memcache key19:26
deryckyeah, I was noticing that.  Seems like stupid caching in the first place.19:27
lifelessthats an issue : it means the short and long versions of the messages won't share cache keys.19:27
lifelessbut perhaps on some pages that matters. Filing a bug on -foundations now vis-a-vis that19:27
deryckyeah, I'm trying to look at the template.  We didn't add this caching, so I need to remind myself what it's doing.19:28
lifelessderyck: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/63432619:29
_mup_Bug #634326: memcache cache keys interact poorly with query parameters <Launchpad Foundations:New> <https://launchpad.net/bugs/634326>19:29
lifelessderyck: Please understand, when I talk about turning memcache off for things, its not because I don't like it : its because I want the best performance for any given page.19:30
derycklifeless, I do understand that.  I'm not against turning off bits of it or poor uses *at all* :-)  I was just against ripping it our wholesale.19:30
deryckthough it's more accurate to say I'm *for* learning to use it correctly and having the ability to use it correctly.19:31
=== almaisan-away is now known as al-maisan
lifelessI have the sense though, that we haven't finished putting our house in order in the underlying layers yet :)19:31
deryckI completely agree19:32
deryckor have the same sense rather19:32
deryckwhich is okay if we rapidly iterate on it.19:32
deryckbut as is, it has some issues.19:32
derycklifeless, so I don't know this spelling, I guess:  "cache:authenticated,comment/rendered_cache_time,comment/index"  is that part of your logging work?19:33
lifelesshmm, for clarity by underlying layers I meant - db use; storm; tal rendering; python scheduling19:33
lifelessmy logging work puts this in19:33
deryckah, ok.  agree on that, too, though I thought you meant something different19:33
lifelessdetails=$url where $url is the memcached url we're requesting19:33
lifelessso pt:lpnet:lp/bugs/templates/bugcomment-box.pt,9760:l:1,0:MjE=,https_//bugs.launchpad.net/ubuntu/+bug/1/+index?comments=all4N2mYcTmyKyEC6n0gBJBFG19:34
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <19:34
lifelessis the url in memcache19:34
lifelessthe cache: stuff I didn't touch, thats the existing control-memcache-in-templates19:34
deryckhmmm, so I thought it was just "cache:type,howlong" so I need to refer to updated docs.19:35
lifelessif we're caching per url in the webapp thats going to go a long way to explaining the terrible hit rate.19:36
lifelessmany objects appear in many urls.19:36
lifelesse.g person, branding, bug comments (per distrotask) etc19:36
lifelesscreate-question appears to send email via a different codepath or something. darn.19:38
lifelessno, thats not it. I wonder19:38
lifelessis email spooling perhaps deferred to the end of the request?19:39
lifelessgary_poster: ^19:39
derycklifeless, so about this memcache question, are you asking me about ripping this out and seeing if we save 2 seconds?19:39
lifelessderyck: I'm speculating; I have a lot of data gathering to do now.19:39
lifelessderyck: but bugtask is one of your pain-pages19:40
lifelessand turning that off would be a very easy thing to do19:40
derycklifeless, yeah, that was going to be my concern is that we don't know the savings (if any) vs the 2 sec. cost.19:40
deryckbut I'm open to turn it off and see what oops appear.19:40
lifelessI think it would be worth trying19:40
lifelesseven with the overhead of doing a CP to get it on prod, it will be a pretty cheap experiment19:41
derycksure, I'm open to that.  With a close eye on it.  I do however wish we could feature flag it, if that works now.  So if it was a bust, we could turn it on/off easy.19:41
deryckI didn't put this in, so I don't know the problems it was trying to prevent.19:41
deryckor the knowledge that underpinned the choice.19:42
lifelessfeature flags do work, I think you'd need to repeat the contained section though19:42
lifelesswhich is a bit ugly19:43
lifelessif we had a pageid /scope/ we could disable memcache for a pageid via a feature flag19:43
lifelessthat should be reasonably easy to do, for someone familiar with the page id logic.19:43
lifelessI'll file a bug requesting that, and note we'd like to CP it to prod19:43
derycksure, sounds good.19:43
deryckleave it New please, so I'll triage it and add it to the board.19:44
deryckforces me to add a card when I have to toggle it to triaged.19:44
deryckunless this is a foundations bug ;)19:44
lifelessits a foundations arena bug, but you might find it easy enough to JDI - sec while I file it19:45
dobeyhmm, is there a known issue with the code scanner in lp right now?19:47
lifelessderyck: https://bugs.edge.launchpad.net/launchpad-foundations/+bug/63434219:47
_mup_Bug #634342: need a features 'scope' for page ids <Launchpad Foundations:New> <https://launchpad.net/bugs/634342>19:47
dobeylike it seems to not be scanning only some branches, and sending proposal e-mails without diffs for them19:48
derycklifeless, got it.  I'll decide if I can JFDI that later today or not.  which goes a bit against JFDI, I know.19:50
lifelessderyck: :P19:50
lifelessderyck: how did abel go last night20:19
derycklifeless, he was worn out from all this, so worked on an easy bug. :-)  But glad that we had it fixed for retracers.20:20
lifelessderyck: I'm glad too20:21
lifelessderyck: also on performance - https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1713S8120:25
lifelessI grabbed that for bug 1 on staging yesterday, with profile.20:25
_mup_Bug #1: Microsoft has a majority market share <iso-testing> <ubuntu> <Clubdistro:Confirmed> <Computer Science Ubuntu:Invalid by compscibuntu-bugs> <EasyPeasy Overview:Invalid by ramvi> <GNOME Screensaver:Won't Fix> <Ichthux:Invalid by raphink> <JAK LINUX:Invalid> <The Linux OS Project:In Progress> <OpenOffice:In Progress by lh-maviya> <Tabuntu:Invalid by tinarussell> <Tivion:Invalid by shakaran> <Tv-Player:New> <Ubuntu:In Progress by sabdfl> <20:25
lifelessI'm going to put it in the20:25
gary_posterlifeless, today is call day, but wading through backlog.  yes, email spooling is deferred to the end of the request.20:25
lifelessgary_poster: ahha20:26
lifelessgary_poster: can you point me at the thing that does that, so I can instrument it ?20:26
gary_posterend of transaction to be precise20:26
lifelessgary_poster: really? after request finalisation ?20:26
lifelesscan we change that?20:26
gary_posterbad idea IMO.  the idea is that if a transaction retries, we don't want to send multiple emails.20:27
lifelessto be on-part-of-something like that20:27
lifelessuhm, let me describe some symptoms20:27
lifeless32 second request20:27
gary_posterneed to go in a few; and then will be back on calls, btw20:28
lifelessit sends 115 emails20:28
lifeless2pc would solve this but be a bit nasty.20:28
lifelessdoing the email spooling from a worker thread would solve it and not distort the request time20:28
gary_posterwe're suposed to have the infrastructure to be doing this already. standard zope bit...20:29
lifelessso, anyhow, 260ms per email is one useful thing to know20:29
lifelessthat seems slow.20:29
gary_posterit's supposed to put in worker thread (in* the 2pc20:29
lifelesscould I perhaps move the bug to foundations ?20:30
gary_posterI guess...it's either something for foundations or something that foundations would be happy to advise on20:31
lifelessblue sky, does pgsql let us ask 'will this commit succeed' ?20:31
lifelessits bug 438116, I'll update the details now we have some instrumentation.20:31
_mup_Bug #438116: Timeout when converting bug into question (BugTask:+create-question) <timeout> <Launchpad Foundations:Triaged> <https://launchpad.net/bugs/438116>20:31
gary_posterlifeless, I don't yet see 260ms per email on that OOPS.  give me a hint as to where I should be looking?20:32
lifelessgary_poster: look for sending emails20:32
lifelessor 'send' perhaps20:33
lifelessanyhow, 115 rows (by subtraction)20:33
gary_postere.g. 352.36170mssendmail[Question #124730]: No vdpau-va-driver for amd64 in Maverick20:33
_mup_Bug #124730: Feisty - Sound stops after a few seconds and various apps fail to work properly <Ubuntu:Invalid> <https://launchpad.net/bugs/124730>20:33
lifelessand at the end20:33
lifelessthe request does its session update @ 32 seconds in20:33
lifelessand it finished its sql work, which for this code is ~= to finishing completely, ~6 seconds in20:33
gary_posterah ok20:34
gary_postersure. :-) from users perspective, the request should have been sent back by 6 seconds, but agreed anyway that sending email ought to be done elsewhere.  as I said, the expected pattern is that you would spool in the commit phase.  well, since that's not happening, then maybe user didn't get the reply till 6 seconds in either. :-/20:36
gary_posterdidn't get the reply 6 seconds in I meant20:36
gary_posterI need to go get boys from school20:36
gary_posterthen I have calls20:36
gary_posterI will look at how we are sending emails20:37
gary_posterif you happen to know the code path that would be nice20:37
lifelessthanks, no panic - this like other timeouts is just kanban backlog20:37
gary_poster(that is, where in LP the email is sent)20:37
gary_posterright, cool20:38
lifelessI will dig up some of that for you20:38
gary_posterthank you20:38
dobeyi guess this is the more active channel now?21:45
lifelessthis is the development channel21:55
EdwinGrubbsmatsubara: ping21:56
matsubaraEdwinGrubbs, on the phone21:56
matsubarawill ping you once I'm done21:56
gary_posterlifeless: need to go, bug re bug 633664, when you talk about double-buffering, you mean at the load balancer and in the app server, correct?22:59
_mup_Bug #633664: API file attachments are done via the appservers <Launchpad Foundations:Invalid> <https://launchpad.net/bugs/633664>22:59
gary_posters/bug re bug/but re bug/22:59
lifelessgary_poster: the load balancer isn't buffering22:59
lifelessgary_poster: the double buffering is in the appserver22:59
gary_posterbuffers in asyncore, then?23:00
lifelessgary_poster: from what you're telling me, yes.23:00
gary_posterright but, I mean, where else?23:00
lifelessgary_poster: there are some explicit things I know, and some things I don't.23:00
lifelessgary_poster: we're calling addFile in the appserver, and we have a facility to retry the request if the db conflicts -> that implies a buffer23:01
gary_poster(this isn't to ignore your other concerns, but I understand them)23:01
gary_posterbut why a second buffer?23:01
lifelessI think I've added confusion23:01
lifelesswhat I mean is 'we're not sending it directly to where it belongs, but the design is intended to support big blobs by doing that'23:02
gary_postergot it23:02
gary_posterfair enough, understood23:02
gary_postermust run :-)23:02
thumperwallyworld_: morning23:02
lifelesshappy to talk another day23:02
gary_postercool, but I understand now and agree that what we have is a problem23:03
wallyworld_thumper: happy birthday23:03
gary_posterthumper: happy birthday :-)23:03
lifelessgary_poster: cool, thanks.23:03
=== matsubara is now known as matsubara-afk
=== Ursinha is now known as Ursinha-afk
=== al-maisan is now known as almaisan-away
jelmerthumper: hi23:53
jelmerthumper: did you see the wiki page maxb added with failing bzr-svn imports summary?23:53
thumperjelmer: hi23:53
thumperI'm busy chasing a critical branch scanner fubar23:53
thumperdue to bug heat23:53
jelmerI won't bother you then :-)23:53
thumperjelmer: good that chicken finally imported :)23:53
thumperI'm sure peter is happy23:54
mwhudsonjelmer: i saw a lot of code imports marked failed overnight, do you know what that's all about?23:54
jelmermwhudson: I retried all of the ones I thought *might* be fixed and some still failed, but a lot are now actually working23:55
mwhudsonjelmer: oh ok23:55
mwhudsonthat's good23:55
jelmermwhudson, we're down to less than 100 failures in the bzr-svn/bzr-git imports, 59 of which are caused by lack of support for nested trees23:56
mwhudsonjelmer: \o/23:57
wgrantAre nested trees actually happening at some point?23:59
wgrantWell, more than they were two years ago? :P23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!