=== bdmurray_ is now known as bdmurray [00:51] wgrant: i don't think there's a bug search api that can be used to find all bugs a person is directly subscribed to is there? you can search and find bugs for structural subscriptions on a pillar but I want to find bugs subscribed to directly [01:07] wallyworld: https://launchpad.net/~wgrant/+subscribedbugs [01:07] wallyworld: But you don't want to remove all subscriptions. [01:09] wgrant: if a person has had their access to a pillar revoked, then we want to unsubscribe them from all bugs for that pillar, no? [01:09] lifeless, wgrant: do you know if storm makes any guarantees about the default ordering of results if no order_by is specified? [01:09] bac: It does not. That would be evil. [01:09] wallyworld: Their direct access to the pillar has been revoked. [01:10] wallyworld: But they may still have access through a team. [01:10] wgrant: yeah, well we have unit tests that rely on order by id...and they fail when run out of order [01:10] bac: You'll need to order explicitly. Do not add implicit order to the model, as it is likely to break queries. [01:11] Sorting is expensive. [01:11] you don't say. :) [01:12] bac: Some old SQLObject-based classes have a default order defined, but it causes no end of performance trouble when you write what seems to be an innocent query but it actually sorts behind your back. [01:12] wgrant: the argument about team access could be applied to the case where their access is revoked to a single artifact via the sharingdetails page and we unsubscribe there [01:13] wallyworld: For example, once the UI is turned on we'll want to remove about 130 of the Launchpad grants. [01:13] thanks wgrant [01:13] wallyworld: But we'll want to keep lots of the subscriptions. [01:13] wallyworld: Because engineers have subscribed to things that they're interested in. We're just revoking their explicit access because they should get it through ~launchpad instead. [01:14] sure, so that's what i'm doing in this job - removing direct subscriptions for an individual when access is revoked [01:14] You need to remove direct *grants*, and subscriptions to artifacts that are no longer accessible. [01:15] But not all subscriptions to private artifacts. [01:15] grants are removed already, but i was told i also need to remove subscriptions also [01:16] it's in a bug on the board [01:16] You need to remove some of them, yes. [01:17] so there's 3 cases 1. revoke access to individual artifact, 2. revoke access to entire pillar, 3. revoke access to artifcats of an info type [01:18] so far in the job i've done case #1 - simply unsubscribe user from given artifact [01:18] In all cases, if there is a subscription and no remaining access (eg. through a team), the subscription must be removed. [01:18] However [01:18] This gets complicated, because eg. team membership changes affect which subscriptions are allowed. [01:19] if the grant is revoked, they won't have access even if there is a team subscription [01:19] They will so. [01:19] Well [01:19] Not a team subscription. [01:19] But a team grant. [01:20] right. but the sharing details page only shows direct access [01:20] Sure [01:20] But revoking the direct access does not necessarily revoke all access. [01:20] and when the delete button is hit, the grant is revoked and the invisual sub to the aertifact is removed [01:20] That is incorrect behaviour. [01:21] There may still be access, through a policy grant or a grant to a team. [01:21] but if there's a policy grant, you don't see that artifact on the sharing details page [01:22] so you can't revoke it individually [01:22] Ah, through that UI that is true indeed (although it might be visible on the bug page) [01:22] But it's still true for team grants. [01:23] extending that metaphore of case #1 above, if i revoke a user's access to userdata things, that user needs to be unsubscribed from those things [01:24] No. [01:24] Because you might be doing it to clean up. [01:24] Not to revoke access. [01:24] https://launchpad.net/launchpad/+sharing <- most of those grants are invalid, but the subscriptions are fine. [01:25] wgrant: see bug 933839, the second bit of the description indicates we do need to remove subscriptions [01:25] <_mup_> Bug #933839: Share nothing with a user or team < https://launchpad.net/bugs/933839 > [01:26] Bug descriptions from 3 months ago don't override logic :) [01:27] but i discussed this work over the past 2 standups and nothing was said to the contrary [01:27] It doesn't explicitly state either way. [01:27] yes it does, it says "Unsharing also removes subscriptions to bugs and branches" [01:27] One behaviour clearly produces undesirable results, the other potentially contradicts one reading of a vague statement in an old bug report. [01:29] it wasn't just in this bug report, the same issue has come up in conversation over the past little while [01:30] I'd always heard it phrased more along the lines of "subscriptions to which there is no longer access must be removed" [01:30] Not "subscriptions to artifacts from which a grant is being revoked must be removed" [01:32] I don't think the latter version makes much sense. [01:32] As it is clearly inconsistent with intended behaviour elsewhere. [01:32] I can subscribe to a bug I have access to, even if it's through a team. [01:33] * StevenK stabs BugChange [01:33] sure. the intent of the ui then becomes an issue [01:33] The UI will make it clear that people may still have access through teams. [01:33] if a user clicks the delete icon on the sharing page, they think they are revoking access for that user [01:33] I can clearly see two BugInformationTypeChange's being created, but printing a traceback when they're created is less than helpful. [01:34] wallyworld: Removing subscriptions doesn't make that true or false. [01:34] wallyworld: It's already false. [01:34] wallyworld: Because access may remain through teams. [01:36] so does access policy grant flat take account of team membership, i can't recall [01:36] Nop. [01:36] No. [01:36] Remember the concerns around private team expansion etc. which led us to defer the audit view. [01:37] but if the requirement is to unsubscribe users if direct access is revoked and they have no further access via teams, i could join apgf to teamparticiaption and check that way [01:38] That would be one way to do things, inded. [01:39] so i think the current mp which unsubscribes when revoke is done from the sharing details page is correct [01:40] wgrant: Can you see any difference in http://pastebin.ubuntu.com/961462/ ? I'm printing the arguments BugInformationTypeChange is getting created with and then forcing a stack trace [01:41] wallyworld: mmm [01:42] wallyworld: It's not incorrect, because it doesn't specify how to calculate the bug IDs. [01:42] However, I question the design. eg. this probably makes it impossible to revoke apport's access to Ubuntu bugs. [01:43] As the request will have to grab like 300000 bugs out of the DBs, calculate their IDs, put them in a list, serialise them to JSON, stuff them into a multi-megabyte text column in the DB. [01:43] wgrant: you misunderstand - the bug ids list is only for individual revokes, not info type or pillar based ones [01:44] wallyworld: Ah, that makes me a lot less worried. [01:44] i'm not *that* stupid [01:44] Most people don't quite realise the magnitude of some of the horrific exceptional cases we have. [01:45] Like, no reasonable person could think that one principal would have 300000 bug grants. [01:45] But they do :) [01:45] seriously? 300000 [01:45] Maybe only 200000 [01:45] But a few [01:45] That order of magnitude, anyway :) [01:46] * wallyworld needs food [01:50] StevenK: https://code.launchpad.net/~wgrant/launchpad/no-job-oops-prefix/+merge/104315 [01:51] * StevenK scratches his head over this double BugInformationTypeChange [01:51] * wgrant looks [01:52] I have a diff between them [01:52] Those are tracebacks for the two calls? [01:52] Yeah [01:52] wgrant: http://pastebin.ubuntu.com/961476/ is the diff [01:52] The first one is -, the second is + [01:53] StevenK: lazr.restful calls notify() itself [01:53] Presumably it diffs the object too [01:53] lazr.lifecycle woo [01:54] Using lazr.lifecycle, yeah [01:54] The one from lazr.restful is wrong, though :-( [01:54] for some reason the fact that that isn't part of launchpad any more has never really settled into my brain [01:55] wgrant: Your MP looks excellent [01:55] StevenK: How's it wrong? [01:55] -BugInformationTypeChange(None, , information_type, Unembargoed Security, Public) [01:55] +BugInformationTypeChange(None, , information_type, Unembargoed Security, ) [01:55] StevenK: Thanks [01:56] Well [01:56] You could make it not wrong. [01:56] zope.security woo [01:57] wgrant: This explains why it's going wrong, though. Bug:+secrecy is calling transitionToInformationType directly which is firing off one notify. The JS is calling it via the API and lazr.restful is doing the notification and then transitionToInformationType is doing it. [01:58] That is clear, yes. [01:58] It's only clear to me as of 60 seconds ago. [01:59] So transistionToInformationType shouldn't notify if from_api is set, or is there a better pattern? [01:59] It should perhaps not notify at all. Check what corresponding things to, eg. status/assignee changes [02:01] transitionTo{Assignee,Target} at least don't notify [02:02] * StevenK comments out the notify() and see what happens with Bug:+secrecy and the JS fire. [02:04] wgrant: Looks like Bug:+secrecy requires the notify() at least. [02:05] So I can fix Bug:+secrecy to make an API call, which I don't like, or change transitionToInformationType to not notify() when called from the API, which I don't like either. [02:05] Or change Bug:+secrecy to notify like other edit forms. [02:05] That is the correct solution. [02:06] Ah ha [02:19] bac: if they are ordered on id, how are they failing - are the objects having the wrong ids, or was the sort actually undefined ? [02:44] wallyworld, wgrant: https://code.launchpad.net/~stevenk/launchpad/double-bugdelta-js/+merge/104316 [02:45] wallyworld: I think that MP might fix your double e-mail issues, too [02:45] cool [02:47] StevenK: should you do a check to ensure info type really has changed? [02:47] like is done for private etc [02:49] wallyworld: I just didn't want to unilattery notify on both fields there. [02:49] For information_type I don't think it matters [02:50] StevenK: Did you try making it a LaunchpadEditForm? [02:51] LaunchpadEditFormView, sorry [02:51] That should fire the events automatically. [02:52] wgrant: Will that deal with private+security_related/information_type? [02:53] StevenK: It diffs the objects like lazr.restful does. [02:55] StevenK: if you do use LEFV you will likely need to override updateContextFromData() [02:56] I didn't know about LaunchpadEditFormView [02:57] My concern about diffing the object is private, security_related and information_type all change. [02:57] StevenK: Doesn't the subscriber handle filtering that? [02:58] I'm not certain, I'd left most of the notification stuff alone due to HBD [02:59] I'd prefer to switch to LEFV when I drop the legacy behaviour. [02:59] the ff complicates it so if it works as is i'm ok with that [03:00] get_bug_changes filters based on the FF [03:00] So notifying with the entire diff should be fine. [03:00] StevenK: there were no failing tests, right? so do we need a test to check that notifications for info type are as expected? [03:01] wallyworld: No, the tests all exist [03:01] Clearly not. [03:01] buildbot would have failed. [03:01] but there we no failures [03:01] I perhaps need a notification test when changing via the API [03:02] and/or a test that changing info type via the ui produces the correct notifications [03:02] wallyworld: Given that involves JS, I'm not sure how to do that. [03:03] We know it fires the API call, if the API call only generates one notification we're good [03:03] js is irrelevant here - you just need to write a normal view test [03:03] initialise a view with form={} with the right things in the dict [03:03] wallyworld: Right, hold on [03:04] wallyworld: Bug:+secrecy has always done one notification, this branch doesn't change that. [03:04] i got 2 the other day [03:04] wallyworld: The error this branch fixes is when you change the information_type via the JS on Bug:+index, since that uses the API. [03:04] The solution with the minimal amount of code, so minimal bugs, is to use LEFV [03:04] So use that [03:05] not necessarily given the ff complication [03:05] LEFV? [03:06] LaunchpadEditFormView [03:06] LaunchpadEditFormView [03:06] thanks [03:06] LaunchpadEditFormView [03:06] Peer pressure [03:06] what he said [03:07] StevenK: afaik, the js used to call the BugSecrecyEditView - there's even code in the submit action to handle it.perhaps it's changed now [03:07] wallyworld: I'll switch it to LEFV, there is already notification tests for Bug:+secrecy, I'll write a webservice test. [03:07] wallyworld: The legacy JS calls BugSecrecyEditView, yes. [03:07] ah, ok. but the new js for the info type choice popup doesn't [03:08] Yes. [03:08] i didn't really that, sorry [03:08] realise [03:08] wallyworld: I thought I explained it well enough in the MP's description. [03:09] i makes more sense if i re-read it now. i didn;t connect legacy and js [03:09] wgrant: Which is why I'm probably a little retiscent to change it to LEFV. The legacy JS calls the form. [03:09] The legacy *untested* JS [03:10] WCPGW [03:10] It's already taken me four hours nailing down this issue in the first place. [03:11] The next time I want to touch this code is when I remove the legacy bits. [03:12] +1 [03:13] StevenK: i'm still i little curious why there were no failing tests, that's my remaining concern [03:13] ideally you'd address that before landing [03:13] wallyworld: Right, so I'm writing a webservice test now [03:14] ok, r=me otherwise [03:14] Which will probably get interrupted by lunch, but meh. [04:09] Oh, blah. the bugchanges tests assume that transitionToInformationType is the one generating the notifications. [04:34] wallyworld: http://pastebin.ubuntu.com/961639/ [04:35] wallyworld: That brings the information_type and visibility tests into line with things like IBugTask.status which make the change and then notify [04:36] looking good so far, need to finish reading the diff [04:37] wallyworld: I've confirmed that if I revert the changes in the MP you reviewed that the API test fails due to getting two notifications. [04:38] that's helpful [04:38] StevenK: the change_activity 'person': private_bug.owner (was self.user); what's the reason for that? [04:39] wallyworld: I'm notify()'ing with user=private_bug.owner, I can change that to self.user and revert those bits if you wish. [04:40] StevenK: i'd just do it the same as the other similar tests for consistency [04:40] whatever way that is [04:40] wallyworld: Fixed [04:41] thanks. send it to ec2 [04:41] Gladly. [04:41] did you know [04:41] good to see this one fixed [04:41] that editing shell scripts while they are running seems to be a bad idea [04:41] who would have thought :-) [04:43] well it works for python scripts :) [04:43] wallyworld: Let me XXX BugEditSecrecyView to note about LEFV too [04:43] sounds like a plan [04:43] can't wait to get rid of more legacy [04:43] Tell me about it [04:54] Launchpad encountered an internal error during the following operation: generating an incremental diff for a merge proposal. It was logged with id OOPS-b36d837108d341fbaba9b6ef591ac4f0. [04:54] * StevenK stabs the job system over and over and over. [04:55] wgrant: did you try using an aggregate + comparise rather than AND's ? [04:55] or an aggregate with set ops ? [04:56] lifeless: You mean the abomination that's used for structural subscription sorting and is slow as hell? [04:56] NULL_COUNT and such? [04:56] not sure [04:56] well [04:56] "comparise" isn't a word [04:56] So I was trying to work out what you meant [04:57] Adding more aggregates is unlikely to help performance. [04:57] bah, typo [04:57] aggregate and compare [04:57] In this case the estimates are important for correct performance. [04:58] and aggregates are throwing it off ? [04:58] What sort of structure were you thinking of? [04:58] I've found that aggregate conditions tend to be hard to plan through. [04:59] mapping the tags into an array [04:59] what performance do you get with the AND's ? [04:59] Right, that's what's done for structural subscription filtering. [05:02] That's always entirely thwarted estimation when I've tried it in the past, but I'll recheck just in case. [05:11] lifeless: I'm not sure it's feasible, given the complex constraints that are possible. [05:11] We could do it nicely with the complex query stuff in intarray [05:12] what performance do you get with the AND's ? [05:13] https://pastebin.canonical.com/65322/ [05:15] The FTI is pretty slow, but hopefully GIN will fix that. [05:15] 540 for a moderate search is fine [05:16] FSVO [05:16] We've got that + a similar count right now [05:17] So it's more like 1.3s before we get any other time. [05:17] It actually ends up planning really well like this. [05:17] was thinking about counts [05:18] we could make that async [05:18] issue two requests [05:18] Because it has accurate stats for fti and tags [05:18] Yeah, I've considered that fairly strongly. [05:18] Also remembering counts. [05:18] And guessing from bugsummary sometimes [05:18] Most counts can be satisfied straight from bugsummary [05:18] And those that can't are mostly complex things with FTI that nobody will notice if they're cached for a while. [05:19] mmmm [05:19] caches tend to get you in trouble [05:19] Although the estimates are much better with bugtaskflat, so rough_length could possibly be a bit more workable. [05:19] But I still don't like it much for this sort of thing. [05:20] simple ones people complain about, complex ones are complex [05:27] lifeless: Also, I have a branch landing in 30 minutes that rips out the oops_prefix per-section customisability, and was thinking of pushing forward the one-config-per-user thing next week so I can eliminate the rest. Any objections? Should basically just mean adding a few new configs, getting them deployed, throwing a crontab diff at LOSAs and getting a few init scripts fixed up. [05:27] Then we can delete like 1000 lines of prod config :) [05:27] And a similar amount of config schema. [05:28] (we currently have lots of overrides in eg. the production config because different scripts run as different users, so need different error_dirs) [05:32] wallyworld: Do you want to review https://code.launchpad.net/~stevenk/launchpad/bugs-factory-information_type/+merge/104326 ? [05:32] wgrant: sounds lovely [05:33] ok [05:40] StevenK: r=me [05:40] wallyworld: Sorry for the massiveness [05:40] that's what i say to the girls [05:41] wallyworld: http://picardfacepalm.com/ [05:41] well, you started it :-) === mthaddon` is now known as mthaddon [08:01] good morning === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [08:54] wallyworld: You didn't run your new table through ec2? [08:54] Looks like it's missing person merge permissions. [08:55] rollback time [08:55] Dinner time, actually. === StevenK changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 3.47*10^2 === al-maisan is now known as almaisan-away [10:11] heavens [10:11] I need to make my branch listing page more readable. [10:17] rick_h_: did you ever get any test results from that ec2 test run? [10:18] jml: that landed [10:18] and was merged I thought [10:18] rick_h_: there was another one [10:19] oh, I didn't complete the test run, I thuoght we were just debugging why it wouldn't run [10:19] rick_h_: it's not a huge deal. I'll probably be able to run it myself today. [10:19] rick_h_: ah, right :) [10:19] my apologies, I thuoght once you had it figured out why it was ok and set [10:19] rick_h_: np. === danhg_ is now known as danhg === dpm is now known as dpm-afk === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away === abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: 3.47*10^2 [13:44] abentley, you meant that you and adeuring would stay on the call, right? not me? [13:44] deryck: yes. [13:44] abentley, great, thanks [13:51] https://code.launchpad.net/~jml/lp-dev-utils/public-ip/+merge/104273 needs review [13:52] hah, jml finding someone else today :P sorry abentley [13:52] :) [13:53] got this error on 'ec2 demo': http://pastebin.ubuntu.com/962408/ [13:54] and then this error after terminating with Ctrl+D: http://pastebin.ubuntu.com/962410/ [13:55] jml: why is it a guess? [13:55] abentley: because it might be wrong. [13:55] jml: why might it be wrong? [13:56] In general, if your HTTP routing is different from your SSH routing. [13:56] abentley: I'm being pessimistic. checkip.amazonaws.com was wrong sometimes, I don't see what makes akamai guaranteed to not be wrong. All I know for sure is that it's less wrong than checkip.amazonaws. [13:57] cjwatson: fair point. I suppose we could determine our IP via SSH? [13:57] For example, if you have a public IP address, but you connect through an HTTP proxy on another public IP address (say, for performance, or because it's mandated in some way by your network layer), then akamai would return the address of the proxy; but you might be connecting directly to the cloud instance over SSH. [13:58] Amazon had the opposite problem; in proxy cases, it would always return the address of the client system as seen by the proxy, private or not. [13:58] hmm. [13:58] I would argue that it's more common for Amazon's case to break, in the modern internet; but the other kind of breakage is *possible*. [13:58] So I support jml having added an option for this. [13:59] then IIUC the correct thing is to always use the akamai address for 80 & 443 ports and only apply the provided IP to the SSH port? [13:59] I don't know of any way to reliably determine our IP address via SSH, in the absence of some new service. [13:59] jml: The behaviour in your MP is the best I can think of. [14:00] cjwatson: thanks. === danhg_ is now known as danhg [14:00] cjwatson: And yet, the who command knows your IP/domain, and when you log in, it will tell you your last ip/domain. [14:01] jml: Anyhow, fair point that this is only a guess. [14:03] jml: although if you SSH and HTTP routings are different, then security_group.authorize('tcp', 22, 22, '%s/32' % public_ip); security_group.authorize('tcp', 80, 80, '%s/32' % public_ip) is also wrong. [14:04] abentley: I think that's what cjwatson meant by "some new service". You'd need to have an SSH service up that could be got into by everyone who needs to run 'ec2 test'. [14:05] abentley: well, that's sort of what I thought, but cjwatson responded otherwise when I raised the question with him. [14:06] abentley: But you only get to ask it that after you've told Amazon to authorise you to connect from a given IP address. [14:06] abentley: Chicken and egg. [14:06] cjwatson: I see. [14:06] jml: Oh, sorry, I misunderstood that part. Um, maybe. [14:08] jml: I'm not sure whether "transparent proxy" or "intercepting proxy" is the right terminology in this case. Do you know? [14:08] abentley: no, I don't. [14:09] jml: Okay, let's leave it as is. Rob can tell us different if he wants. [14:09] jml: r=me [14:09] abentley: thanks. [14:25] abentley: Proper proxies have a problem too, anyway. === dpm-afk is now known as dpm [15:25] hi everyone, I would like to know how to read a user's translation activity using launchpadlib [15:26] this is for ubuntu accomplishments project started by jono [15:26] any help would be appreciated [15:43] nik90: hmm. [15:43] nik90: you know about the API, right? [15:43] jml: yes I did go through the API..but couldn't find a suitable one for translation...i am new to this.. [15:43] nik90, API does not provide that useful information. Some of this information is implied in karma activity which idoes not have the detail and not over the API [15:44] sinzui: oh so using the API, I cannot access the translation activity of a user but must rely on the user's karma? [15:45] No, because karma is bogus.We only export a number which is itself ogus [15:46] nik90, https://launchpad.net/+apidoc/devel.html does not provide any means to answer you questions [15:46] sinzui, that was the API list that I went through [15:47] sinzui, the closest I came to was the translation_import_queue_entries...but that could not check the translations still [15:48] nik90, right, but entries in the queue is just uploading [15:49] sinzui, would I be able to use translation_template by checking with the user's name [15:49] would that check the translations? [15:50] nik90, https://launchpad.net/~sinzui/+karma shows a bogus score for types of activity, and a list of recent activity [15:51] sinzui: hmm, yes it does..I guess that should do it temporarily [15:51] sinzui, the thing was, I wanted to confirm that the translation made by the user were approved and useful before giving the accomplishment [15:51] nik90, I argue that we care about seeing the frequency of activity and batched listing that allows me to see contributions over a year, month, and week [15:51] but through this I wont be able to...but atleast I can check if they made a translation in the first place [15:52] sinzui: thanks for your help [15:52] nik90, ah, that is a challenge. I really don't think that can be answered at all [15:53] sinzui: going through the api to distunguish between different karma [15:54] nik90, we only provide a number over the API. karma is broken so we choose not to encourage people to use it over the API [15:55] sinzui: :( alrite that's ok === salgado is now known as salgado-lunch === deryck is now known as deryck[lunch] === salgado-lunch is now known as salgado === matsubara is now known as matsubara-lunch === deryck[lunch] is now known as deryck === benji is now known as Guest45135 === benji___ is now known as benji === matsubara-lunch is now known as matsubara [18:47] morning [18:55] deryck: / abentley: Are you guys planning on blogging (blog.l.n.) about the bundles-to-mp code being removed? [18:56] lifeless: no. [18:57] abentley: is there more to it than that ? [18:57] lifeless, hey. I didn't think we needed much public notice for it.... [18:57] lifeless: It's unlikely that anyone will notice, since no one's used the feature at all this year. [18:57] lifeless, it hasn't been used since June of last year. [18:58] calling attention to it seems like asking people who don't use it to complain about it being removed. ;) [19:00] I can understand that angle [19:00] OTOH we're been very forthright about other removals and redefinitions with a generally positive response. [19:01] I think it is better to be clear than to have folk surprised later (even though that is unlikely). [19:01] We're doing a good thing here [19:01] focusing the system on what users need [19:01] making it possible for us to develop and iterate faster [19:02] *I* think you should be proud of the removal [19:02] (btw, I presume you have it on your todo list, but https://help.launchpad.net/Code/Review still references the feature) [19:03] lifeless: Yes, I didn't want to remove it until the change had landed. [19:04] cool cool [20:24] 816831