=== matsubara-dinner is now known as matsubara === Ursinha-afk is now known as Ursinha [07:10] stub: hi, i'm getting this error from a query containing an IN expr with strings: "ProgrammingError: operator does not exist: text = bytea". i can copy and paste the sql from the traceback into a sql editor and it works fine. any clues? [07:12] the sql statement contains a call to trim if it matters [07:12] SELECT trim(leading 'pending-' from CommercialSubscription.sales_system_id) FROM CommercialSubscription WHERE (trim(leading 'pending-' from CommercialSubscription.sales_system_id)) IN ('foo', 'bar') [07:12] wallyworld: I think you are sending in a raw string instead of unicode, and it is being cast to a binary blob (BYTEA) [07:12] i'm using storm [07:12] wallyworld: And your logging doesn't notice, because to it a string is a string [07:13] Store.of(self).using(CommercialSubscription).find(SQL(voucher_expr), SQL(voucher_expr).is_in(voucher_ids)) [07:13] voucher_ids is a list of strings [07:13] Right, that's a unicode vs str issue [07:13] Are you sure about that? [07:13] i tried casting to unicode [07:13] let me try again [07:13] ok. [07:13] sales_system_id needs a unicode [07:14] surprised storm didn't pick it up - it is normally very picky [07:14] i could have sworn i did that, i'll recheck [07:14] stub: Not when you use SQL() [07:14] Oh right, foot gun [07:14] Yeah [07:14] I'm not sure that using pending- as a prefix here is sensible [07:14] Rather than adding a new column, for example [07:15] wgrant: so using SQL() breaks it? [07:15] wallyworld: No [07:15] It depends how you're building the string that you give to it [07:15] ah, hang on, it worked that time [07:15] SQL() is a way of bypassing in built Storm checks [07:15] i used SQL() because i wanted an expr with trim() [07:16] Hm [07:16] Sure, it has uses. Just have to be more careful since you are saying 'this is valid SQL, trust me'. [07:16] It's more likely that voucher_ids is a sequence of strs, actually [07:16] As the rest of the types should be inferred. [07:16] i must have messed up the unicode cast first time [07:17] voucher_ids needs to be a sequence of unicodes, since the DB column is Unicode. [07:17] i cast the voucher_ids to unicode [07:17] i must have had fat fingers the first time [07:40] good morning === jtv1 is now known as jtv === almaisan-away is now known as al-maisan === al-maisan is now known as almaisan-away [12:22] adeuring: ping, got a sec? [12:22] rick_h_: sure [12:22] adeuring: I'm looking at https://bugs.launchpad.net/launchpad/+bug/1063272 [12:22] <_mup_> Bug #1063272: Cannot view +related-projects because of private-project <403> < https://launchpad.net/bugs/1063272 > [12:23] and from the bug report, it appears the fix could just be a simple check for a PUBLIC information type [12:23] is there any reason to deal with things like checking your access via access policy and such to this you think? [12:24] rick_h_: yes: i think the user himself should be able to see a products he can access [12:24] I've not really used these pages before so my thought is that it's really only useful to other people and only showing PUBLIC types shold work [12:24] ok [12:24] after all, that's an easy way to navigate to the a product [12:25] rick_h_: but for a quick fix, returning only public prodcts is foine, I think [12:25] ok, :P don't let me cheat [12:25] adeuring: well it's just done in raw sql atm with a series of unioned joins so will be a bit messier [12:26] yeah, I can imagine ;) [12:29] wgrant: The new Person.specifications has landed. === Ursinha is now known as Ursinha-afk === abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: ~260 [14:42] abel: Do you have time for some reviews? I'm the OCR, so I can't ask the OCR. [14:48] jcsackett, I lowered the priority of the gmail attachment bug and remove you from it because gmail is forcing CRLF every 76 characters when it does it's base64 encoding. I don't think this is a critical issue. [14:49] sinzui: dig. [15:19] abentley, deryck, rick_h_: my branch with the security adapter for milestones causes again test failures for a recently added test: test_listings_consider_spec_visibility . Could you give me a few hours to try to get my branch landed (I'd like to avoid "race conditions" like those we had for the adapter for products...) [15:19] adeuring: yep, thanks for the heads up [15:20] adeuring, I can wait for my stuff. still doing calls, so not able to fix my own tests yet. [15:20] adeuring: Okay. [15:21] rick_h_, abentley, deryck: Thanks! The alternative is of course to merge this brnach: lp:~adeuring/launchpad/milestone-sec-adapter (though now I'm getting "connection closed" errors when i try to push the most recnet version...) [15:23] abentley: would appreciate thoughts on lp:~cjwatson/launchpad/pcj-queue-ordering, particularly on whether you agree with my performance handwaving [15:23] cjwatson: Okay, give me a couple minutes. [15:31] cjwatson: Can you add "Not(Job.id.is_in(seen))" to the find, and so skip the loops? [15:32] abentley, deryck, rick_h_: I started another EC2 test a few minutes ago [15:32] adeuring, cool, thanks for the heads-up. [15:33] abentley: Oh, is_in doesn't have to take a Select or similar? [15:33] cjwatson: No, it takes values. [15:33] Neat. [15:34] I'll give that a try. [15:37] cjwatson: I don't fundamentally see away around changing the number of DB queries to scaling O(n) with next(). [15:37] s/away/a way [15:37] Me neither. [15:38] Unfortunately I don't have subsecond profiling data on how long those queries usually take at the moment. [15:38] At least it's in a script, so worst case it goes slow not times out ... [15:39] abentley: Not(Job.id.is_in(seen)) causes one of the tests to enter an infinite loop in the kind of way I'd expect if that test was entirely missing. [15:40] (i.e. a test which doesn't actually process the jobs, just iterates over them all.) [15:41] cjwatson: Can I see the diff? [15:41] Sure. http://paste.ubuntu.com/1285251/ - the failing test is test_iterReady_orders_by_copy_policy. [15:42] Oh, wait, PCJ.id != Job.id perhaps [15:44] cjwatson: Yes, but seen and the is_in are both in terms of Job.id. [15:45] No, seen is in terms of PCJ.id [15:45] seen.add(job.id) -> seen.add(job.job_id) fixes it [15:46] cjwatson: great. [15:46] Pushed. === Ursinha-afk is now known as Ursinha [15:55] cjwatson: grr, diff still not updated. I think this is reasonable. You're aware you're making it possible for interactive jobs to starve mass jobs, right? [15:57] Yes. [15:57] cjwatson: The And() in the find statement is superfluous-- find accepts multiple clauses and ANDs them automatically. [15:58] The volume of interactive jobs is a couple of orders of magnitude below capacity, and hardly ever goes over maybe a couple of dozen at a time. [15:58] Thanks, removing the And. [15:58] cjwatson: I would probably leave out the else after "if job is None", since everything following will be the remainder of the loop. [15:59] Fair enough [15:59] cjwatson: As for LOC, are you saying this work is part of an arc that will ultimately remove code? [15:59] Yes. [15:59] Let me double-check numbers quickly. [16:00] Cool. [16:04] cjwatson: r=me. [16:04] The arc is, er, not exactly sure at this point but I think a few hundred so far; this should be near the end of it. I have 1465 lines queued for deletion. [16:07] cjwatson: if you did care about starving mass jobs, I'd suggest splitting interactive and mass into separate lists and yielding from each in turn in a ratio. (Say 10 interactive for every 1 mass). [16:08] Right, thanks. My preliminary analysis a week or two back suggested it wouldn't be necessary; the queue is empty or very close most of the time, except when we're auto-syncing from Debian. [16:08] (Which will be in a week or two, so I want to make sure PS doesn't come and shout at me around then ...) [16:08] :-) [16:09] In fact we'll hopefully (!) be doubling our copy job load by staging everything in -proposed first, if we get the infrastructure for that working in time. [16:18] Incidentally, recent LoC progress has been pretty fantastic. http://people.canonical.com/~cjwatson/tmp/loc-cum.png [16:35] cjwatson: you're on the TB right? [16:36] cjwatson: a mail was sent to it earlier today, if it hasn't been moderated could you please moderate it . thanks [16:38] czajkowski: Yes. Done. [16:38] cjwatson: thank you [16:51] jcsackett, I think I found the strip() that lost the last line when making the attachment [17:01] jcsackett, I would like you thoughts about the comment I added to https://bugs.launchpad.net/launchpad/+bug/898227 [17:01] <_mup_> Bug #898227: Newlines are removed from the end of bug attachments submitted by email < https://launchpad.net/bugs/898227 > === matsubara is now known as matsubara-lunch [17:21] deryck: I think that disabling privacy checks should be a separate feature flag, to avoid changing the meaning of existing tests, and to reduce risks on production (where we'd never turn it on). [17:27] abentley, I agree. === deryck is now known as deryck[lunch] [18:05] rick_h_: Could you please review https://code.launchpad.net/~abentley/launchpad/flag-enables-privacy-checks/+merge/130185 ? [18:06] abentley: loading === deryck[lunch] is now known as deryck [18:17] deryck: I'm the OCR. Could you please review https://code.launchpad.net/~abentley/launchpad/product-specifications-tests/+merge/130000 https://code.launchpad.net/~abentley/launchpad/product-specifications-storm/+merge/130002 and https://code.launchpad.net/~abentley/launchpad/product-specifications-privacy/+merge/130003 ? [18:18] abentley, sure. That is 3 MPs, correct? [18:19] deryck: Yes, 1. add tests 2. stormify 3. add privacy. [18:19] abentley, ah, gotchas. ok, jumping on them now. [18:19] deryck: Thanks. [18:19] np! === matsubara-lunch is now known as matsubara [18:20] abentley: r=me [18:20] rick_h_: Thanks! [18:25] abentley, so for the first branch…. should there be some tests removed too? If this is copied, should the other place die? Or these are slightly different? [18:27] deryck: The behaviour is different. You can list the specifications of an inactive product. You just can't see them elsewhere (like off a Person). [18:31] deryck: After Stormifying, you can't list the specifications of an inactive product. I wasn't sure whether that would cause test failures, but apparently not. I don't know what the behaviour should be. [18:34] abentley, I don't follow, sorry. The tests are no good then, i.e. you don't get the failure you expect? or you mean something else? [18:36] deryck: The existing tests of Products don't care whether it's possible to list the specifications of a disabled product. I don't know whether we care. [18:37] abentley, ah. [18:37] deryck: If we do care, then we should add tests to ensure we get whatever behaviour we want. [18:37] abentley, right, I follow now. Was just thinking myself if we care. [18:40] abentley, I don't think we care. no need for new tests. [18:40] deryck: Okay. [18:47] sinzui: what file is the block you posted from? i think you're right, but i don't recall seeing that chunk in the handler. [18:47] (it also doesn't -- to me, anyway -- explain why there was no reproducing it in a test). [18:48] jcsackett, I am not convinced. It is in messages/model/messaging.py [18:48] I have a test with the real message from the librarian, but it does not fail [18:49] abentley, all these look fine. good work. (r=me)*3 [18:49] deryck: Thanks. [18:49] sinzui: yeah--that's the problem. i couldn't seem to get any sort of similar failure. [18:50] jcsackett, I found the real message that is stored in the librarian. I am feeing it into a test that created the inbox copy, then sends it through the bugs mail handler. the base64 message always has the proper new lines [18:50] oh, that's just maddening. [18:50] and yet; in a dire hope the bug was gone i emailed an attach to the bug from my gmail account, and proved it's still an issue. [18:50] I can see that block will do odd things, but I just don't see the real example step into the block [18:51] I saw [18:52] sinzui: do we know all the callsites? could it be mutated at some other point? [18:52] jcsackett, I guess the real new news is that we can be sure the message was not corrupted when we took it from the inbox and put it into the librarian. I think the bug/message/bugcomment rules mess with it [18:53] sinzui: must be. [18:53] jcsackett, message does mutate. It does forced decoding [18:53] deryck: These three branches take us over the WIP limit. [18:54] abentley, ok, thanks for the heads up. Most of this is my fault. [18:54] np [18:55] abentley, if I don't get my stuff out of the way here shortly, I'll up the WIP limit. [18:56] The real message https://pastebin.canonical.com/76759/ [18:58] ^ jcsackett note that there are two encoding which are fine, but I think Message can do something wrong [19:00] sinzui: good to know that the raw message at least is stored properly. [19:00] sinzui: how did you check that, btw? [19:00] I looked at the message ids for the attachments to the bug in staging's db... [19:01] ...then guess that message.id - 1 would show me what incoming.py got [19:02] incoming creates a random file name and and sequenced id. I typed the url in to my browser to pull it out of the librarian [19:06] aaah. clever. [19:08] jcsackett, also, before the suspected block, there is `content = part.get_payload(decode=True)` [19:09] This is correct when I step though, but I am using that latest python 2.7 [19:09] the suspected block is the bit you posted, yes? [19:09] yes [19:09] sinzui: are we on 2.6 on production? [19:09] correct [19:09] huh. [19:12] abentley, I have my fix for that DecoratedBranch stuff now. So I can submit to ec2 now. [19:15] jcsackett, \o/ http://bugs.python.org/issue7143 [19:15] sinzui: w00t! [19:16] sinzui: so, we can't reproduce it in tests because we're using a fixed version. so we need to upgrade to fix it in prod. [19:16] jcsackett, which we plan to do [19:17] sinzui: so hurrah, this will be fixed then. [19:18] The bug is not critical, but I will tag it as python-upgrade so that we know to close it when we complete the task [19:19] sinzui: cool. [19:56] abentley, deryck, rick_h_: my branch landed [19:57] adeuring, awesome! [20:29] sinzui: I'm having trouble reproducing the milestones side of bug #1063291 [20:29] <_mup_> Bug #1063291: Project groups are broken by private projects

< https://launchpad.net/bugs/1063291 > [20:31] hmm [20:31] sinzui: This passes: http://pastebin.ubuntu.com/1285813/ [20:31] abentley, did I have urls in the original bug? [20:32] sinzui: No. [20:32] * sinzui looks on qastaging again [20:35] abentley, still looking. anon users cannot see https://qastaging.launchpad.net/launchpad-project [20:36] sinzui: Yes, I've got a fix for the private product part of that. [20:36] I thought so [20:37] sinzui: There's proprietary blueprints on that-- that's probably the reason. [20:39] I am getting timeouts creating a milestone on a private project that matches a milestone on a public project, so I am being patient [20:41] abentley, non-priv user gets a 403 on qastaging for https://qastaging.launchpad.net/launchpad-project/+milestone/3.1 [20:42] she can see https://qastaging.launchpad.net/launchpad-project/+milestones, but not the milestone itself [20:43] sinzui: So I can rephrase as "If the project has a milestone with a bug or blueprint targeted, then I cannot see the the groups's own milestone page"? [20:44] yes. [20:44] sinzui: Thanks. [20:48] abentley, I think your blueprint speculation is correct. The page loads for the no-priv user and it has just a private bug targeted https://qastaging.launchpad.net/launchpad-project/+milestone/3.2 [20:49] The 3.1 test has a proprietary blueprint from a previous test [20:51] sinzui: Cool. === Ursinha is now known as Ursinha-afk [22:30] wgrant, https://bugs.launchpad.net/launchpad/+bug/1059853 [22:30] <_mup_> Bug #1059853: BugTask:+editstatus timeout blocked on bug update setting heat < https://launchpad.net/bugs/1059853 >