[07:10] <wallyworld> stub: hi, i'm getting this error from a query containing an IN expr with strings: "ProgrammingError: operator does not exist: text = bytea". i can copy and paste the sql from the traceback into a sql editor and it works fine. any clues?
[07:12] <wallyworld> the sql statement contains a call to trim if it matters
[07:12] <wallyworld> SELECT trim(leading 'pending-' from CommercialSubscription.sales_system_id) FROM CommercialSubscription WHERE (trim(leading 'pending-' from CommercialSubscription.sales_system_id)) IN ('foo', 'bar')
[07:12] <stub> wallyworld: I think you are sending in a raw string instead of unicode, and it is being cast to a binary blob (BYTEA)
[07:12] <wallyworld> i'm using storm
[07:12] <stub> wallyworld: And your logging doesn't notice, because to it a string is a string
[07:13] <wallyworld> Store.of(self).using(CommercialSubscription).find(SQL(voucher_expr), SQL(voucher_expr).is_in(voucher_ids))
[07:13] <wallyworld> voucher_ids is a list of strings
[07:13] <wgrant> Right, that's a unicode vs str issue
[07:13] <stub> Are you sure about that?
[07:13] <wallyworld> i tried casting to unicode
[07:13] <wallyworld> let me try again
[07:13] <stub> ok.
[07:13] <wgrant> sales_system_id needs a unicode
[07:14] <stub> surprised storm didn't pick it up - it is normally very picky
[07:14] <wallyworld> i could have sworn i did that, i'll recheck
[07:14] <wgrant> stub: Not when you use SQL()
[07:14] <stub> Oh right, foot gun
[07:14] <wgrant> Yeah
[07:14] <wgrant> I'm not sure that using pending- as a prefix here is sensible
[07:14] <wgrant> Rather than adding a new column, for example
[07:15] <wallyworld> wgrant: so using SQL() breaks it?
[07:15] <wgrant> wallyworld: No
[07:15] <wgrant> It depends how you're building the string that you give to it
[07:15] <wallyworld> ah, hang on, it worked that time
[07:15] <stub> SQL() is a way of bypassing in built Storm checks
[07:15] <wallyworld> i used SQL() because i wanted an expr with trim()
[07:16] <wgrant> Hm
[07:16] <stub> Sure, it has uses. Just have to be more careful since you are saying 'this is valid SQL, trust me'.
[07:16] <wgrant> It's more likely that voucher_ids is a sequence of strs, actually
[07:16] <wgrant> As the rest of the types should be inferred.
[07:16] <wallyworld> i must have messed up the unicode cast first time
[07:17] <wgrant> voucher_ids needs to be a sequence of unicodes, since the DB column is Unicode.
[07:17] <wallyworld> i cast the voucher_ids to unicode
[07:17] <wallyworld> i must have had fat fingers the first time
[07:40] <adeuring> good morning
[12:22] <rick_h_> adeuring: ping, got a sec?
[12:22] <adeuring> rick_h_: sure
[12:22] <rick_h_> adeuring: I'm looking at https://bugs.launchpad.net/launchpad/+bug/1063272
[12:22] <_mup_> Bug #1063272: Cannot view +related-projects because of private-project <403> <private-projects> <Launchpad itself:In Progress by rharding> < https://launchpad.net/bugs/1063272 >
[12:23] <rick_h_> and from the bug report, it appears the fix could just be a simple check for a PUBLIC information type
[12:23] <rick_h_> is there any reason to deal with things like checking your access via access policy and such to this you think?
[12:24] <adeuring> rick_h_: yes: i think the user himself should be able to see  a products he can access
[12:24] <rick_h_> I've not really used these pages before so my thought is that it's really only useful to other people and only showing PUBLIC types shold work
[12:24] <rick_h_> ok
[12:24] <adeuring> after all, that's an easy way to navigate to the a product
[12:25] <adeuring> rick_h_: but for a quick fix, returning only public prodcts is foine, I think
[12:25] <rick_h_> ok, :P don't let me cheat
[12:25] <rick_h_> adeuring: well it's just done in raw sql atm with a series of unioned joins so will be a bit messier
[12:26] <adeuring> yeah, I can imagine ;)
[12:29] <abentley> wgrant: The new Person.specifications has landed.
[14:42] <abentley> abel: Do you have time for some reviews?  I'm the OCR, so I can't ask the OCR.
[14:48] <sinzui> jcsackett, I lowered the priority of the gmail attachment bug and remove you from it because gmail is forcing CRLF every 76 characters when it does it's base64 encoding. I don't think this is a critical issue.
[14:49] <jcsackett> sinzui: dig.
[15:19] <adeuring> abentley, deryck, rick_h_: my branch with the security adapter for milestones causes again test failures for a recently added  test:  test_listings_consider_spec_visibility . Could you give me a few hours to try to get my branch landed (I'd like to avoid "race conditions" like those we had for the adapter for products...)
[15:19] <rick_h_> adeuring: yep, thanks for the heads up
[15:20] <deryck> adeuring, I can wait for my stuff.  still doing calls, so not able to fix my own tests yet.
[15:20] <abentley> adeuring: Okay.
[15:21] <adeuring> rick_h_, abentley, deryck: Thanks! The alternative is of course to merge this brnach: lp:~adeuring/launchpad/milestone-sec-adapter (though now I'm getting "connection closed" errors when i try to push the most recnet version...)
[15:23] <cjwatson> abentley: would appreciate thoughts on lp:~cjwatson/launchpad/pcj-queue-ordering, particularly on whether you agree with my performance handwaving
[15:23] <abentley> cjwatson: Okay, give me a couple minutes.
[15:31] <abentley> cjwatson: Can you add "Not(Job.id.is_in(seen))" to the find, and so skip the loops?
[15:32] <adeuring> abentley, deryck, rick_h_: I started another EC2 test  a few minutes ago
[15:32] <deryck> adeuring, cool, thanks for the heads-up.
[15:33] <cjwatson> abentley: Oh, is_in doesn't have to take a Select or similar?
[15:33] <abentley> cjwatson: No, it takes values.
[15:33] <cjwatson> Neat.
[15:34] <cjwatson> I'll give that a try.
[15:37] <abentley> cjwatson: I don't fundamentally see away around changing the number of DB queries to scaling O(n) with next().
[15:37] <abentley> s/away/a way
[15:37] <cjwatson> Me neither.
[15:38] <cjwatson> Unfortunately I don't have subsecond profiling data on how long those queries usually take at the moment.
[15:38] <cjwatson> At least it's in a script, so worst case it goes slow not times out ...
[15:39] <cjwatson> abentley: Not(Job.id.is_in(seen)) causes one of the tests to enter an infinite loop in the kind of way I'd expect if that test was entirely missing.
[15:40] <cjwatson> (i.e. a test which doesn't actually process the jobs, just iterates over them all.)
[15:41] <abentley> cjwatson: Can I see the diff?
[15:41] <cjwatson> Sure.  http://paste.ubuntu.com/1285251/ - the failing test is test_iterReady_orders_by_copy_policy.
[15:42] <cjwatson> Oh, wait, PCJ.id != Job.id perhaps
[15:44] <abentley> cjwatson: Yes, but seen and the is_in are both in terms of Job.id.
[15:45] <cjwatson> No, seen is in terms of PCJ.id
[15:45] <cjwatson> seen.add(job.id) -> seen.add(job.job_id) fixes it
[15:46] <abentley> cjwatson: great.
[15:46] <cjwatson> Pushed.
[15:55] <abentley> cjwatson: grr, diff still not updated.  I think this is reasonable.  You're aware you're making it possible for interactive jobs to starve mass jobs, right?
[15:57] <cjwatson> Yes.
[15:57] <abentley> cjwatson: The And() in the find statement is superfluous-- find accepts multiple clauses and ANDs them automatically.
[15:58] <cjwatson> The volume of interactive jobs is a couple of orders of magnitude below capacity, and hardly ever goes over maybe a couple of dozen at a time.
[15:58] <cjwatson> Thanks, removing the And.
[15:58] <abentley> cjwatson: I would probably leave out the else after "if job is None", since everything following will be the remainder of the loop.
[15:59] <cjwatson> Fair enough
[15:59] <abentley> cjwatson: As for LOC, are you saying this work is part of an arc that will ultimately remove code?
[15:59] <cjwatson> Yes.
[15:59] <cjwatson> Let me double-check numbers quickly.
[16:00] <abentley> Cool.
[16:04] <abentley> cjwatson: r=me.
[16:04] <cjwatson> The arc is, er, not exactly sure at this point but I think a few hundred so far; this should be near the end of it.  I have 1465 lines queued for deletion.
[16:07] <abentley> cjwatson: if you did care about starving mass jobs, I'd suggest splitting interactive and mass into separate lists and yielding from each in turn in a ratio.  (Say 10 interactive for every 1 mass).
[16:08] <cjwatson> Right, thanks.  My preliminary analysis a week or two back suggested it wouldn't be necessary; the queue is empty or very close most of the time, except when we're auto-syncing from Debian.
[16:08] <cjwatson> (Which will be in a week or two, so I want to make sure PS doesn't come and shout at me around then ...)
[16:08] <abentley> :-)
[16:09] <cjwatson> In fact we'll hopefully (!) be doubling our copy job load by staging everything in -proposed first, if we get the infrastructure for that working in time.
[16:18] <cjwatson> Incidentally, recent LoC progress has been pretty fantastic.  http://people.canonical.com/~cjwatson/tmp/loc-cum.png
[16:35] <czajkowski> cjwatson: you're on the TB right?
[16:36] <czajkowski> cjwatson: a mail was sent to it earlier today, if it hasn't been moderated could you please moderate it . thanks
[16:38] <cjwatson> czajkowski: Yes.  Done.
[16:38] <czajkowski> cjwatson: thank you
[16:51] <sinzui> jcsackett, I think I found the strip() that lost the last line when making the attachment
[17:01] <sinzui> jcsackett, I would like you thoughts about the comment I added to https://bugs.launchpad.net/launchpad/+bug/898227
[17:01] <_mup_> Bug #898227: Newlines are removed from the end of bug attachments submitted by email <Launchpad itself:Triaged> < https://launchpad.net/bugs/898227 >
[17:21] <abentley> deryck: I think that disabling privacy checks should be a separate feature flag, to avoid changing the meaning of existing tests, and to reduce risks on production (where we'd never turn it on).
[17:27] <deryck> abentley, I agree.
[18:05] <abentley> rick_h_: Could you please review https://code.launchpad.net/~abentley/launchpad/flag-enables-privacy-checks/+merge/130185 ?
[18:06] <rick_h_> abentley: loading
[18:17] <abentley> deryck: I'm the OCR.  Could you please review https://code.launchpad.net/~abentley/launchpad/product-specifications-tests/+merge/130000 https://code.launchpad.net/~abentley/launchpad/product-specifications-storm/+merge/130002 and https://code.launchpad.net/~abentley/launchpad/product-specifications-privacy/+merge/130003 ?
[18:18] <deryck> abentley, sure.  That is 3 MPs, correct?
[18:19] <abentley> deryck: Yes, 1. add tests 2. stormify 3. add privacy.
[18:19] <deryck> abentley, ah, gotchas.  ok, jumping on them now.
[18:19] <abentley> deryck: Thanks.
[18:19] <deryck> np!
[18:20] <rick_h_> abentley: r=me
[18:20] <abentley> rick_h_: Thanks!
[18:25] <deryck> abentley, so for the first branch…. should there be some tests removed too?  If this is copied, should the other place die?  Or these are slightly different?
[18:27] <abentley> deryck: The behaviour is different.  You can list the specifications of an inactive product.  You just can't see them elsewhere (like off a Person).
[18:31] <abentley> deryck: After Stormifying, you can't list the specifications of an inactive product.  I wasn't sure whether that would cause test failures, but apparently not.  I don't know what the behaviour should be.
[18:34] <deryck> abentley, I don't follow, sorry.  The tests are no good then, i.e. you don't get the failure you expect?  or you mean something else?
[18:36] <abentley> deryck: The existing tests of Products don't care whether it's possible to list the specifications of a disabled product.  I don't know whether we care.
[18:37] <deryck> abentley, ah.
[18:37] <abentley> deryck: If we do care, then we should add tests to ensure we get whatever behaviour we want.
[18:37] <deryck> abentley, right, I follow now.  Was just thinking myself if we care.
[18:40] <deryck> abentley, I don't think we care.  no need for new tests.
[18:40] <abentley> deryck: Okay.
[18:47] <jcsackett> sinzui: what file is the block you posted from? i think you're right, but i don't recall seeing that chunk in the handler.
[18:47] <jcsackett> (it also doesn't -- to me, anyway -- explain why there was no reproducing it in a test).
[18:48] <sinzui> jcsackett, I am not convinced. It is in messages/model/messaging.py
[18:48] <sinzui> I have a test with the real message from the librarian, but it does not fail
[18:49] <deryck> abentley, all these look fine.  good work.  (r=me)*3
[18:49] <abentley> deryck: Thanks.
[18:49] <jcsackett> sinzui: yeah--that's the problem. i couldn't seem to get any sort of similar failure.
[18:50] <sinzui> jcsackett, I found the real message that is stored in the librarian. I am feeing it into a test that created the inbox copy, then sends it through the bugs mail handler. the base64 message always has the proper new lines
[18:50] <jcsackett> oh, that's just maddening.
[18:50] <jcsackett> and yet; in a dire hope the bug was gone i emailed an attach to the bug from my gmail account, and proved it's still an issue.
[18:50] <sinzui> I can see that block will do odd things, but I just don't see the real example step into the block
[18:51] <sinzui> I saw
[18:52] <jcsackett> sinzui: do we know all the callsites? could it be mutated at some other point?
[18:52] <sinzui> jcsackett, I guess the real new news is that we can be sure the message was not corrupted when we took it from the inbox and put it into the librarian. I think the bug/message/bugcomment rules mess with it
[18:53] <jcsackett> sinzui: must be.
[18:53] <sinzui> jcsackett, message does mutate. It does forced decoding
[18:53] <abentley> deryck: These three branches take us over the WIP limit.
[18:54] <deryck> abentley, ok, thanks for the heads up.  Most of this is my fault.
[18:54] <abentley> np
[18:55] <deryck> abentley, if I don't get my stuff out of the way here shortly, I'll up the WIP limit.
[18:56] <sinzui> The real message https://pastebin.canonical.com/76759/
[18:58] <sinzui> ^ jcsackett note that there are two encoding which are fine, but I think Message can do something wrong
[19:00] <jcsackett> sinzui: good to know that the raw message at least is stored properly.
[19:00] <jcsackett> sinzui: how did you check that, btw?
[19:00] <sinzui> I looked at the message ids for the attachments to the bug in staging's db...
[19:01] <sinzui> ...then guess that message.id - 1 would show me what incoming.py got
[19:02] <sinzui> incoming creates a random file name and and sequenced id. I typed the url in to my browser to pull it out of the librarian
[19:06] <jcsackett> aaah. clever.
[19:08] <sinzui> jcsackett, also, before the suspected block, there is `content = part.get_payload(decode=True)`
[19:09] <sinzui> This is correct when I step though, but I am using that latest python 2.7
[19:09] <jcsackett> the suspected block is the bit you posted, yes?
[19:09] <sinzui> yes
[19:09] <jcsackett> sinzui: are we on 2.6 on production?
[19:09] <sinzui> correct
[19:09] <jcsackett> huh.
[19:12] <deryck> abentley, I have my fix for that DecoratedBranch stuff now.  So I can submit to ec2 now.
[19:15] <sinzui> jcsackett, \o/ http://bugs.python.org/issue7143
[19:15] <jcsackett> sinzui: w00t!
[19:16] <jcsackett> sinzui: so, we can't reproduce it in tests because we're using a fixed version. so we need to upgrade to fix it in prod.
[19:16] <sinzui> jcsackett, which we plan to do
[19:17] <jcsackett> sinzui: so hurrah, this will be fixed then.
[19:18] <sinzui> The bug is not critical, but I will tag it as python-upgrade so that we know to close it when we complete the task
[19:19] <jcsackett> sinzui: cool.
[19:56] <adeuring> abentley, deryck, rick_h_: my branch landed
[19:57] <deryck> adeuring, awesome!
[20:29] <abentley> sinzui: I'm having trouble reproducing the milestones side of bug #1063291
[20:29] <_mup_> Bug #1063291: Project groups are broken by private projects <milestones> <private-projects> <projectgroups> <Launchpad itself:In Progress by abentley> < https://launchpad.net/bugs/1063291 >
[20:31] <sinzui> hmm
[20:31] <abentley> sinzui: This passes: http://pastebin.ubuntu.com/1285813/
[20:31] <sinzui> abentley, did I have urls in the original bug?
[20:32] <abentley> sinzui: No.
[20:32]  * sinzui looks on qastaging again
[20:35] <sinzui> abentley, still looking. anon users cannot see https://qastaging.launchpad.net/launchpad-project
[20:36] <abentley> sinzui: Yes, I've got a fix for the private product part of that.
[20:36] <sinzui> I thought so
[20:37] <abentley> sinzui: There's proprietary blueprints on that-- that's probably the reason.
[20:39] <sinzui> I am getting timeouts creating a milestone on a private project that matches a milestone on a public project, so I am being patient
[20:41] <sinzui> abentley, non-priv user gets a 403 on qastaging for https://qastaging.launchpad.net/launchpad-project/+milestone/3.1
[20:42] <sinzui> she can see https://qastaging.launchpad.net/launchpad-project/+milestones, but not the milestone itself
[20:43] <abentley> sinzui: So I can rephrase as "If the project has a milestone with a bug or blueprint targeted, then I cannot see the  the groups's own milestone page"?
[20:44] <sinzui> yes.
[20:44] <abentley> sinzui: Thanks.
[20:48] <sinzui> abentley, I think your blueprint speculation is correct. The page loads for the no-priv user and it has just a private bug targeted https://qastaging.launchpad.net/launchpad-project/+milestone/3.2
[20:49] <sinzui> The 3.1 test has a proprietary blueprint from a previous test
[20:51] <abentley> sinzui: Cool.
[22:30] <sinzui> wgrant, https://bugs.launchpad.net/launchpad/+bug/1059853
[22:30] <_mup_> Bug #1059853: BugTask:+editstatus timeout blocked on bug update setting heat <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1059853 >