[01:33] wallyworld: BugTaskSearchParams.setTarget maybe of interest. [01:34] wgrant: context? [01:34] + all_branches = getUtility(IAllBranches).visibleByUser(user) [01:34] + wanted_branches = all_branches.visibleByUser(user).withIds( [01:34] + *branch_ids) [01:34] Also, that's got a duplicate visibleByUser [01:34] wallyworld: The preloading branch [01:34] i copied that code from elsewhere [01:34] You currently branch based on pillar interface and call setProduct/setDistribution [01:34] Rather than setTarget [01:35] ok. i just cut and pasted it. will fix along with the other typo. thanks [01:36] Cargo-culting Launchpad code is a little less likely to be optimal than in other projects, sadly. [01:36] Our codebase is in a bit of a sorry state. [01:40] it's getting better though [02:14] Wow [02:14] https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-b0f528b0f920a20462267635b6697b7a [02:15] SQL time: 4032 ms [02:15] Non-sql time: 32597 ms [02:16] Ah, it's the old "Product.has_release_files actually retrieves every detail about everything relating to every release file" trick. [02:46] wallyworld: Thanks :) [02:47] np. i hate typos [04:34] Oh. [04:53] lifeless: I still fundamentally reject the way bugsummary journalling is done, but you weren't wrong about there being a lot of fat... [04:55] There we go, 20 times faster already [04:56] * StevenK stabs stacking [04:56] Yes. [04:56] That is the correct response to stacking in Launchpad :( [04:57] Tag addition to an 80 task bug in 20ms instead of 650ms. [04:58] And there's probably another 5-10ms or so still to trim. [04:58] I can't figure out where a branch is created with the stacked_on set [04:58] I guess there really was no thought put to optimisation during the mad scramble to unbreak prod [04:58] StevenK: That would never happen on production. [04:58] Only in a test. [04:58] wgrant: I need a DB trigger then? === Guest8786 is now known as Ursinha [04:59] StevenK: No. The things that update the stacked_on field will be easily findable in the application code. [04:59] You can put guards there. [04:59] Code's code isn't insane. [04:59] Bugs' code is, which is why I used triggers for the early stuff. [05:00] Branches are also far far simpler. The only vague complications they have are retargeting and restacking. [05:00] Let me paste a diff and I can explain the problem. [05:00] The existing privacy model is sufficiently crap that we don't have to migrate it or work around it. [05:00] http://pastebin.ubuntu.com/990144/ is my diff [05:01] lp.code.model.tests.test_branch.TestBranchPrivacy.test_public_stacked_on_private_is_private fails since it's not private [05:02] You need to define rules around what happens when stacked_on is changed, and implement them. [05:04] Nasty lifeless writing n+1 query code in DB triggers :) [05:04] Although I'm not quite sure why it's so slow. [05:04] Since it's all static so it should be preplanned. [05:04] Perhaps bulk inserts optimise heap insertion somehow. [05:09] wgrant: So .createBranch() is called, and then branch.branchChanged() is called? [05:09] StevenK: That's how it normally happens, yes. [05:09] branchChanged should hopefully be the only place stacked_on is changed on production. [05:09] Looks like it [05:10] Also looks like I'll need to change makeBranch() to do the same. [05:20] wgrant: thats great news [05:20] wgrant: what did you change to make it faster ? [05:27] lifeless: First I removed the fixed_upstream thing, which I turned off display of by feature flag a couple of months back. That saved about 50% of the time. Then I fixed summarise/unsummarise/flush to do bulk inserts instead of a FOR loop. [05:27] That saved the rest of the time. [05:30] interesting [05:30] Yes. [05:30] thanks for suspending your disbelief long enough to investigate [05:30] Heh [05:30] well, a little tease is in order ;) [05:30] Thanks for poking me in this direction; it will indeed be a bit quicker to implement than the redesign. [05:31] naturally :) [05:31] anytime [05:31] It's not entirely implausible that eg. FK constraints are optimised in a batch INSERT, I guess. [05:31] Since most of the FKs will be duplicates. [07:51] good mornin === frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: frankban* (rvba) | Firefighting: - | Critical bugs: 3.47*10^2 === almaisan-away is now known as al-maisan [08:35] frankban: Oooh, is this your first shift? [08:35] yes StevenK_ [08:36] Hm, where did that _ come from? [08:36] frankban: However, congrats for standing up as a reviewer. === StevenK_ is now known as StevenK [08:37] StevenK: thanks [09:51] Ooh, a victim! :-) [09:52] frankban: I've got https://code.launchpad.net/~cjwatson/launchpad/packageset-score/+merge/105915 for you, if you fancy it [09:52] (But not if you have more urgent stuff) [09:52] cjwatson: on it in a minute [09:54] heh, a victim. [09:54] gmb: Any news on that review? https://code.launchpad.net/~dooferlad/launchpad/upcomingwork_show_incomplete_bp/+merge/105846 [10:02] jamestunnicliffe, I haven't had a chance to look at it yet; it's on my list for this morning. [10:03] gmb: Cool, thanks. === al-maisan is now known as almaisan-away [12:20] mrevell, just to confirm, we have our call today? [12:21] danilos, If you and James can make it, that would be very helpful for us. [12:21] mrevell, I can for sure, let's invite James as well [12:22] mrevell, oh, and hi btw :) [12:22] Hello! :) [12:50] Good grief the whole Archive/ArchivePermission edifice isn't half wordy. [13:20] rick_h_, hey, man. [13:20] deryck: howdy [13:20] deryck: nvm email I sent last night, problems solved I think [13:20] will bring it up in call [13:21] rick_h_, ah, ok. was about to say I couldn't find email to respond and then respond here. :) But we can chat on call then. [13:26] cjwatson: Hm, why do you have a setScore method? [13:31] wgrant: score should be visible read-only on the API and have a restricted setter; happy to take suggestions on a better way to do it, I was probably cargo-culting from nearby code [13:32] * czajkowski peers at wgrant [13:33] wgrant: it's not a competition who can stay up the latest you know :) [13:33] cjwatson: Attributes have separate read and write permissions. [13:34] cjwatson: eg. look at Builder [13:34] interface="lp.buildmaster.interfaces.builder.IBuilder"/> [13:34] interface="lp.soyuz.interfaces.buildrecords.IHasBuildRecords"/> [13:34] permission="launchpad.Edit" [13:34] set_schema="lp.buildmaster.interfaces.builder.IBuilder"/> [13:36] So maybe I can avoid the grotty IPackagesetRestricted altogether? [13:36] interface= is for getting attributes on the interface (including methods), set_schema= for setting attributes, attributes= for reading specific non-interface attributes, set_attributes= for writing them. [13:36] cjwatson: You probably need a separate permission. [13:37] Right, but I'm using launchpad.Moderate for that. [13:37] Unless anyone who can edit a packageset should be able to edit its score.; [13:37] No. Build scores control access to a site-wide resource. [13:37] There's a test for that :) [13:37] Right, so it needs a separate permission. But you could indeed do away with the interface, and just use set_attributes [13:37] But perhaps I can just do set_attributes="score". Right. [13:38] On large interfaces we usually use sub-interfaces for read permissions. [13:38] But write permissions often end up as set_attributes because they need to be split differently. [13:38] Like this. [13:39] Does the strictest apply, or is there a problem with ? [13:39] That ZCML will fail to execute. [13:40] Conflicts like that are fatal. [13:41] Ugh, so I have to enumerate all the non-Moderate attributes? A separate interface suddenly looks nicer after all. [13:41] Indeed. And a separate interface works here, since everything is world-readable. [13:41] Things get messy when you have some stuff read-restricted, and subsets of that write-restricted, so you'd end up needing like 10 interfaces. [13:42] But I can still make it be set_schema= rather than having a setScore method, I guess. [13:42] Yep. [13:42] czajkowski: It's not yet midnight :) [13:43] czajkowski: Some of us weren't up at 4am... [13:45] yeah wont be doing that again [13:45] I am glad :) [13:45] wgrant: Not entirely sure how to write the setter, though. I can't use @score.setter here, can I? Doesn't look like storm implements that. [13:45] cjwatson: No need for a setter. [13:46] It's not a property, is it? [13:46] Duh. Good point. [13:46] Right. [13:46] You should have just to change the permission and set readonly=False on the interface. [13:46] And it will all magically work. [13:46] But also tests. [13:46] But then the restricted interface has nothing in it. [13:46] You move score to that interface. [13:47] The [13:47] *Then [13:47] s/move/copy/ maybe? I need it in the read-only one too. [13:47] No [13:47] You have to have it in only one. [13:47] Oh I se [13:47] e [13:47] So you read and write [13:47] For that interface. [13:48] I think the existing tests I have should cover this [13:48] test_score_packageset_forbids_non_buildd_admin and test_score_packageset_allows_buildd_admin [13:56] deryck: Has all the translations opening been completes as far as you know? [13:57] stub, hi. As far as I know, it has *not* be complete. I filed an RT and never heard back. And I need to hand that off to jam and the blue squad now actually. [13:57] Um [13:57] wgrant: Something like http://bazaar.launchpad.net/~cjwatson/launchpad/packageset-score/revision/15260 then? [13:58] Handing an extremely fragile process that almost always breaks off to a squad that's new to Launchpad doesn't sound like the wisest of plans. [13:58] cjwatson: Much nicer, thanks. [13:59] deryck: Ta. There are some droppings in the DB I was wondering about from one of the scripts. [14:00] stub, yeah, I gave webops a script to clean that up. [14:00] deryck: ok. Is that in your RT? [14:00] stub, yes [14:00] Have the number handy? [14:00] wgrant: The only downside is that it shows up as writeable in the apidoc. [14:00] But I guess it is, sort of. [14:01] wgrant, maybe not the wisest of plans. but there is literally no one that understands the process, except jtv. so if he isn't doing it, anyone of us is a good as another. [14:02] cjwatson: Yeah, permissions have always been an issue there. [14:02] deryck: Perhaps. [14:03] I'm not sure Blue is likely to agree :P [14:03] poor smurfs [14:10] "The smurfs"; that sound much friendlier than some of the other squad nicknames. :-P [14:11] jcsackett: lol [14:59] rick_h_, hey. give me 5 minutes to settle in again and we can chat [14:59] deryck: sounds good [15:00] danilos, Are we Mumbling? [15:00] danilos, Does James have access to Mumble? [15:00] well, our Mumble server [15:00] mrevell, flacoste, jamestunnicliffe: let's meet up in https://plus.google.com/hangouts/_/extras/linaro.org/lp-meetup (James doesn't have access to our mumble server) [15:02] danilos: welcome to the 21st century :-) [15:04] flacoste, heh, thanks :) [15:04] though, 21st century has suddenly left me with no sound output [15:04] danilos: pulseaudio -k solves everything :-) [15:05] jamestunnicliffe, I was just restarting it system wide [15:08] rick_h_, I suspect the combo-loader is broke the translation import queue. Can you take a few minutes to give your thoughts: https://bugs.launchpad.net/launchpad/+bug/1000282 [15:08] <_mup_> Bug #1000282: Uncaught TypeError on translations series +imports < https://launchpad.net/bugs/1000282 > [15:09] sinzui: looking, have a call in a moment, but will check it out [15:10] sinzui: off the bat I'd say it was a requires: issue. A module isn't requiring something it needs to, or a lint issue, if a module fails to parse due to invalid JS? I'll look deeper in a second [15:10] but those are my initial thoughts [15:10] mrevell, flacoste, jamestunnicliffe: hangout hanged for me again, I'll reboot to get to a clean state [15:10] danilos: hang out [15:10] we are switching to the conf system [15:11] ok [15:11] danilos: conf code: 116 862 4336 [15:12] flacoste, ack, in and waiting for the leader [15:13] danilos: are you in? [15:13] mrevell, I am [15:14] mrevell, that's right [15:14] 12.04 has started popping up a bunch of crash reports [15:14] mrevell, there is a lot, I'll reboot [15:21] jamestunnicliffe, flacoste, mrevell: https://launchpad.net/~linaro-infrastructure/+upcomingwork [15:21] flacoste, mrevell: https://docs.google.com/spreadsheet/ccc?key=0AvOsYPy8e7yUdGkyRmx2WGFwT3NnSjdHVW04Q1pvSmc [15:25] mrevell: danilos' email is "Per-milestone metadata in blueprints" dated from 12-03-21 [15:40] Hi people, we're just trying to use staging.lp for some QA, and wanted to check whether it's just a quick outage, or is it likely to be a while before it's up again? [15:42] noodles775: let me find out [15:43] Thanks czajkowski [15:46] noodles775: just checking [15:49] flacoste, mrevell: https://blueprints.launchpad.net/linaro-infrastructure-misc/+spec/lp-workitems-qa-tracking [15:49] jamestunnicliffe, ^ [15:55] noodles775: back up there now [16:00] czajkowski: thanks! [16:08] how do I view an oops from the web again? forgot the subdomain [16:10] timrc: https://lp-oops.canonical.com/ [16:12] czajkowski, thanks [16:13] anyone else getting timeouts? I tried, https://launchpad.net/ubuntu/+search?text=live-build, and it resulted in this oops: https://lp-oops.canonical.com/?oopsid=OOPS-e439dc4c04ec939b5798cebb1d5c4177 [16:23] abentley: https://code.launchpad.net/~adeuring/launchpad/put-retry-jobs-into-celery-queue/+merge/106012 [16:27] sinzui: do you have a sample url for that JS error? have a branch to fix it, but want to make sure I qa the right place [16:35] sinzui: nvm, got it [16:46] rick_h_, sorry, I was picking my son up from school: qastaging doesn't have the combo loader on at the moment so this works https://translations.qastaging.launchpad.net/+imports where as production does not [16:46] sinzui: yea, got it, MP is up now [16:46] waiting for the diff to update [16:47] fab [16:47] https://code.launchpad.net/~rharding/launchpad/meta_1000282/+merge/106013 if you're feeling like moving along quiclly [16:47] my pleasure [16:47] sinzui: I tried to note in the bug and the MP why it failed and how to debug that in the future hopefully [16:47] let me know if it doesn't make sense [16:48] ah [16:48] but thanks for catching this, exactly the type of things hard to find automatically [16:48] rick_h_, what are the odds that the comment is irrelevant now [16:49] sinzui: not sure, I didn't look at how they were using things tbh [16:49] rick_h_, understood. We have a problem when we do not document the version that is broken. I fixed a few bugs recently by removing our hacks because YUI does work [16:50] sinzui: heh, yea. In a quick glance through the code I don't see what's 'strange' but like you say, no reference to when this was added. I guess I could annotate/match YUI versions/etc [16:50] would be great if they had a link to a bug/etc [16:51] rick_h_, you have my approval to land the fix. I will see if I can track down the version in annotate and report a bug if needed [16:51] sinzui: yea, I'll remove the comment [16:51] it doesn't say anything other than why oop/event are added to req [16:51] and honestly, those are on every page anyway [16:51] and if someone comes along, they're not going to remove those/get passing tests if it really is required [16:52] excellent point [16:52] ok, so removing comment, will get to land shortly, try to get fix out asap. Sorry to break the page [16:52] guess I should apologize to czajkowski on that one [16:53] I seem to recall hearing before that using launchpadlib_for in tests is awfully slow, and it does seem to be true here. Is there an example somewhere of a reasonably fast and current preferred style for webservice unit tests? [16:54] rick_h_: eh ? [16:54] czajkowski: the combo loader stuff broke the translations import queue page which is in the maint. watch stuff [16:54] assuming you're doing that these days as I know we used to before you started [16:54] rick_h_: ahhhh [16:55] (I don't actually need to test launchpadlib as such, only what's exposed on the webservice.) [16:55] cjwatson, there's webservice_for I think [16:55] direct requests, rather than via lplib [16:56] webservice_for_person? [16:57] cjwatson, that's the one [16:57] Thanks, I'll give that a try [16:57] from lp.testing.pages import webservice_for_person === frankban changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2 [17:22] hello [17:22] daker: aloha there [17:23] daker: coming to submit patches :-) [17:23] hhhh ツ [17:24] adeuring: I believe the recommended way of creating a naive datetime from a UTC datetime is datetime.reaplace(tzinfo=None) [17:24] daker: folks will help if you get stuck [17:24] i hope ツ [17:24] daker: feel free to ask here or on the mailing list, and they're very friendly [17:25] abentley: maybe. but then we simply assume that the time delta is correct... [17:26] adeuring: How do you mean? === matsubara is now known as matsubara-lunch [17:26] abentley: lets me try your suggestion first... [17:27] s/reaplace/replace/ [17:28] adeuring: Catching AttributeError is too broad, because db_class.makeInstance might raise an AttributeError. [17:28] abentley: http://paste.ubuntu.com/991017/ [17:28] adeuring: I recommend getattr(db_class, 'makeInstance', None) [17:29] abentley: right, that is better. but check the paste above: replace(tzinfo=None) is not usable. [17:31] adeuring: I thought if Celery was using naive datetimes, they would represent UTC. They represent local time? [17:32] abentley: yes [17:32] abentley: perhaps that can be configured [17:32] adeuring: How about datetime.astimezone then? [17:34] now_utc.astimezone(tz=None) [17:34] -> astimezone() argument 1 must be datetime.tzinfo, not None [17:34] ... a TypeError [17:36] adeuring: astimezone can be used to convert from UTC to the local timezone, and then you can strip off the tzinfo with replace. [17:37] abentley: well, ok. But is that really much better than to use the delta? [17:39] adeuring: I think it is much better than using datetime.now() twice, since the current date is not relevant to the conversion you're performing. [17:39] well, ok... [17:42] abentley: do you know how figure out the local timezone? I find the example in the Python docs a bit cumbersome... [17:42] (class LocalTimezone) [17:42] adeuring: No, I don't know how to do that. [17:43] abentley: so.... with the delta, we don't have to deal with this ;) [17:44] adeuring: Yes, but we'd be doing a lossy conversion. [17:44] adeuring: And unnecessary system calls. [17:45] abentley: well, maybe. But anyway, there is a different details that concerns me a bit there: It might make sense to add another 10 seconds or so to the delat, to allow for time skew between the DB time and the time seen by celeryd [17:46] s/delat/delta/ [17:47] adeuring: I agree that's a reason for concern. [17:48] adeuring: I wonder if it makes sense to use the delta from the current DB time? [17:49] abentley: maybe. But I would prefer a system call to another DB request... [17:50] adeuring: Using a system call doesn't address the concern of skew between the db and the celeryd system time. [17:51] abentley: right, I'll try that. [17:51] adeuring: So AFAICT, the DB time isn't relevant. [17:52] adeuring: acquireLease doesn't use DB time. [17:52] abentley: ah, right. [17:53] adeuring: I believe there's no possibility of skew. [17:53] abentley: unless we have workers on more than one machine [17:54] adeuring: Right. We don't have that, and we're not expecting to have that in the near future, at least for a single queue. [17:55] adeuring: Still, I guess a fudge factor doesn't hurt here. [17:56] abentley: right, one second should be enough [17:57] adeuring: sure. [18:07] abentley: the example from the Python docs (class Localtime in the description of the datetime module) make _four_ calls to time.mktime() if the class is used in astimezone.astimezone(Localtime()) ;) === matsubara-lunch is now known as matsubara [18:50] abentley: any progress with the review? [18:50] adeuring: No, I was waiting on your changes. [18:51] abentley: so... what should we do woth the time delta? That's my main question. I think it does not make sense to use the example from the python docs to get local timezone. [18:51] but UI leave this decision to you... [18:54] adeuring: I agree that the example from the python docs doesn't make sense, but I find it hard to accept the proposition that calling datetime.now multiple times is the cleanest way to get a naive datetime. [18:55] abentley: agreed, this not elegant but straightforward and easy to understand... [18:56] and I don't think that this is an all-important aspect of what we want to achieve. [18:57] abentley: but I am open to better suggestions [18:57] adeuring: No, respecting the lease eta was something we weren't trying to achieve at all. [18:57] abentley: well, the other option would be to drop the lease completely. but I am not sure if it is reasonable to retry jobs vers fast [18:58] ...very fast [19:01] adeuring: It's not reasonable, but the lease isn't the correct value to use here. If we're actually implementing retry delays, we should be doing exponential backoff so that outages of up to a day don't prevent the job from completing. [19:02] abentley: ok, makes sense. But this is nothing I am going to implement before friday [19:06] abentley: anyway, hereis the current diff: http://paste.ubuntu.com/991164/ [19:06] adeuring: So I suggest making it a hardcoded 10-minute delay. That will easily allow leases to expire, and give a bit of time for problems to clear. [19:07] abentley: ok [19:19] abentley: current diff: http://paste.ubuntu.com/991191/ [19:20] adeuring: Since 'generator' has a specific meaning in Python, how about 'factory'? [19:20] abentley: sure [19:21] adeuring: It looks like you're using a retry delay of 1 second, rather than 10 minutes. [19:21] abentley: gahh, fixed: retry_delay = timedelta(minutes=10) [19:25] adeuring: Are the Celery docs wrong? They say "eta must be a datetime object, specifying an exact date and time (including millisecond precision, and timezone information)" [19:25] abentley: yes. try it for yourself [19:26] i tried datetime objects with timezone myself... [19:27] We should file a bug on the documentation, then. [19:28] yes [19:31] adeuring: Okay, r=me with your latest changes. [19:31] abentley: thanks [19:34] adeuring: we should always supply an ETA, not just when there is a lease. [19:35] abentley: why? [19:36] adeuring: If we're retrying, then we must have acquired a lease, because we always acquire a lease before running, and we can't retry until we run. [19:37] adeuring: But if there was some scenario where we could retry without having acquired a lease, we would still want to wait because it doesn't make sense to retry immediately, as you've pointed out. [19:37] abentley: yes, that's the logic there: If a lease exists, this is an attemt to retry a job, so we need an ETA: but we don't need that when the job is run fpot the first time [19:37] abentley: ok, i see your point, but for now i give. simply too late here... [19:39] adeuring: I'm not retracting the r=me, but I think we should fix that. [19:39] abentley: ok, that's right [20:32] czajkowski: the CPU is getting hot :) [21:36] daker-cloud: do ask questions if you get stuck ok [21:36] lotta lp dev folks around [21:37] ok now i am running make schema === spm` is now known as spm [21:59] czajkowski: https://plus.google.com/u/0/101694416703170881163/posts/Ad3hvn385Po :) [22:04] jcsackett, mumble? [22:06] daker-cloud: cool, what are you looking to fix [22:10] not sure czajkowski, do i need a mentor ? [22:10] this thing is huge [22:12] so lp doesnt have mentors, they do have code reviewers, and you could post to the launchpad-dev mailing list with abug you are working on or want to work on to get some feedback [22:12] daker-cloud: or ask people in here like sinzui wgrant jcsackett and others who are around in your timezone [22:14] ok [22:28] czajkowski: I'm American now!? [22:28] wgrant: no, you just dont sleep === matsubara is now known as matsubara-afk [22:32] True. [23:07] wallyworld_: I've got a deployment edit all ready, let me know when your QA is done. [23:08] StevenK: done