/srv/irclogs.ubuntu.com/2012/07/04/#launchpad-dev.txt

Bert_2wgrant: but loggin in as root with the root password doesn't work either00:00
Bert_2or does it only copy the password of the one user it's created with ?00:00
wgrantBert_2: If asked to bind ~ (the '-b $USER' option) it will copy the relevant lines from 'getent passwd' and 'getent shadow'. If -b is omitted, it will use ubuntu:ubuntu00:01
Bert_2I see00:01
Bert_2so I'll have to recreate the container then I guess ?00:02
wgrantYou'll probably have to leave out -b, or manually 'chroot /var/lib/lxc/$CONTAINER/rootfs passwd $USER'00:02
Bert_2wgrant: okey, thanks :D00:02
wallyworld_StevenK: yeah, i've been otp for an hour sorry. doing it now00:13
wallyworld_StevenK: is done00:28
StevenKwallyworld_: Thanks01:01
wallyworld_StevenK: sorry it took a while, i was otherwise engaged for a bit01:02
StevenKwallyworld_: Tis all good.01:02
StevenKwgrant: So I've cribbed get_bug_privacy_filter for get_branch_privacy_filter, but I'm not sure how to collapse down the policy grant filter01:13
StevenK2012-07-04 01:15:01 DEBUG2  [PopulateBranchAccessArtifactGrant] Done. 30536 items in 304 iterations, 625.594790 seconds, average size 100.449373 (48.8121224706/s)01:28
StevenKwgrant: ^ DF01:29
wgrantStevenK: Rather than saying access_policies && foo, you can say access_policy = ANY(foo)01:32
StevenKwgrant: Does Storm export an Any, or do I need to screw around with SQL() more?01:54
wgrantStevenK: I'm not sure. At worst you'll need to add a three-liner to lp.services.database.stormexpr01:58
lifelesspretty sure it does02:08
lifelessand please add ovbvious things like that to storm itself02:08
lifelesswe have reviewers in-team, no need to make the techdebt higher than it already is02:08
StevenKlifeless: I've been looking and I can't see it02:10
lifelessah, probably doesn't then. its weak on scalars02:12
lifelessstill, as wgrant says, easy to add.02:12
wgrantlifeless: You mean !scalars?02:15
lifelessmaybe :P02:16
wgrantBug #102078502:27
_mup_Bug #1020785: lp.services.apachelogparser.base.get_files_to_parse doesn't like gzipped files over 4GiB <ppa> <Launchpad itself:Triaged> < https://launchpad.net/bugs/1020785 >02:27
wgrantCan anyone see a better solution than checking if the read size is >4GiB and gzip isize <4GiB, then ignore the file if read size % 2**32 == gzip isize?02:28
wgrantlifeless: Can I have a One Patch Policy exception for http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-stable/view/head:/database/schema/patch-2209-20-1.sql and http://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-stable/view/head:/database/schema/patch-2209-25-1.sql?02:32
lifelesswell, thats also going to fail.02:32
lifelesswhy not keep a signature of the last bytes in the file, or the mtime, or something? What are the constraints.02:33
wgrantlifeless: It'll fail in roughly 1 in a couple of billion files.02:33
wgrantWhich is certainly less than ideal.02:33
wgrantBut not fatal.02:34
lifelesswgrant: +102:35
wgrantThanks.02:35
lifelesson the file thing02:35
lifelesswhat are the constraints, why do we need to check that way?02:36
lifelesswould a logstash consumer be a better answer ?02:36
wgrantOK02:36
wgrantSo02:36
wgrantWe need a solution this decade02:36
wgrant=> logstash is not an option02:36
wgrantgzip seeking is expensive.02:37
wgrantSo we can't just seek to the end and see what is there.02:37
lifelessso these are problems not constraints.02:38
lifelessthey are obvious, the constraints aren't.02:38
lifelesswhen do we need it fixed.02:38
lifelessDoes it have cpu limits? latency limits?02:39
lifelessimagine you're telling a particularly annoying dev about the problem for the first time :)02:39
wgrantWe need to be able to progressively parse logs02:39
wgrantIdeally without gunzipping 50GB of data we've already read02:40
wgrantEvery time02:40
spmwgrant: I've done ... similar, but if I understand this code I'm looking at correctly, basically pull in a buffer of gzipped data, and pass that to gzread to decode and give back.02:40
spmof course, this is C, and doing it the hard way is expected.02:41
lifelesswgrant: well, that seems unsubstantiated; really not trying to be difficult. but "why not" ?02:41
wgrantnotsureifserious02:41
lifelessam02:41
lifelessalso you can fingreprint a compressed file trivially02:41
lifelessso not sure why you aren't proposing that02:41
lifelessas a short term stop gap at least02:42
wgrantWe could store and check the CRC, but that's probably only marginally better than the filesize here, and it involves a schema change.02:42
lifelessis that a constraint?02:43
wgrantIt's a very undesirable thing, as it means I can't fix it before lunch.02:43
lifelessanyhow, short term hack - just store the gzip file length.02:44
lifelessif the files are append only02:44
wgrantHow's that better than my initial suggestion?02:44
wgrantIt corrupts the database02:45
wgrantAnd requires SQL to unbreak the history.02:45
lifelessthis seems like a fraught discussion02:46
lifelessI'd rather move it to voice, or pass; We're trying to solve the same stuff, so the contention is wholly unneeded.02:46
wgrantStoring the bad size seems strictly worse than changing the check -- I can't see any benefit.02:46
wgrantCan you explain?02:46
lifelessyou want to optimise handling of files, avoiding rereads; I'd do that by fingerprinting the primary source of data, not be depending on a documented-as-broken field.02:47
wgrantOh02:48
wgrantBy "gzip file length" you mean the length of the compressed file?02:48
lifelessyes02:48
wgrantNot the file length from gzip.02:48
lifelessright02:48
wgrantCan't do that, since we need to know where to seek to within the file.02:48
wgrantSo we know which line to start at.02:48
lifelessyes, it would then be different for existing things in the db, but depending on how you handle repeated reads that might not be an issue.02:48
wgrantSo that requires a schema change too02:48
lifelesspersonally, I'd do the schema change; yes it means another day or two to get the fix deployed, but its much less likely to bite you in the bum later.02:49
lifelessalternatively, use memcache to store out-of-file fingerprints and have fallback code for when thats missng, but - gnnngh, I don't really like that approach except perhaps as a stopgap.02:49
wgrantmemcache is only sensible when it's not hideously expensive to recalculate.02:50
wgrantIn this case we store logs for months/years02:50
wgrantWorth hundreds of gigabytes02:50
wgrantSo the loss of the data is a huge problem.02:50
lifelesswhat seems odd to me is that we have code even attempting to process old logs; theres no high water mark ?02:51
lifelessif there isn't we have quadratic overheads anyhow02:51
wgrantIt's linear over a couple of thousand files and DB rows in a single run.02:51
wgrantThere's no watermark because we don't assume anything about the filenames.02:52
lifelessif you want to do a quickiefixie, I've no objection. But I do think its worth considering how to make it deal with a couple of OOM of growth.02:52
wgrantSo ops can do as they please with logrotation etc.02:52
lifelesswgrant: you assume a single file can get altered though02:52
StevenKlifeless: Out of memory?02:52
wgrantlifeless: It can02:52
lifelessStevenK: order of magnitude02:52
StevenKOH! Orders of magnitude02:52
wgrantlifeless: It gets appended to throughout the day.02:52
StevenKYeah, just got it02:52
lifelessStevenK: cool02:53
wgrantlifeless: Also, not just that it can get appended to.02:53
wgrantlifeless: We parse 10 million lines per run02:53
wgrantlifeless: Logs have well over 10 million lines.02:53
wgrantSo we'd have to only store the size of the gzip file once we reached the end.02:54
wgrantStoring the physical location in the file isn't going to work, because of the trailer.02:54
lifelesssure, because thats how you're handling repeated reads (and optimising for finding-what-to-process-next).02:55
wgrantlifeless: How would you propose to do it?02:56
lifelessI'd suck stuff off of the logs as it happens and put it in a persistent queue, process it just in time from there.02:56
wgrantIf I had a list of things that were good ideas, queueing up thousands of writes a second on any data store we have today would probably not be on that list.02:57
lifelesswgrant: 3K per second isn't high for amqp.02:58
lifelesswgrant: we've got several OOM headroom. Particularly for small things, which this is.02:58
lifelesswell, at least two anyhow.02:58
wgrantIt also prevents us from ever disabling the process, or reading historical logs.02:58
lifelessAn we can shard the queues (and servers) if needed.02:58
lifelessit doesn't discard the logs, we can indeed do historical logs if needed, but they become a special case, not the general case.02:59
wgrantAnyway02:59
lifelessDisabling the process can be done a few ways; have a stateful sucker-upper; disable the consumer (and just rotate queues if we have too many 10's of GB of logs pending).02:59
wgrantThat rewrite is at least 10 years away02:59
wgrantHow would you do it this century?03:00
wgrantYou seem to have a strong opinion that there's a better way than storing these offsets.03:00
lifelessI think that when you're re-reading append only files, storing where you got to in it is fine.03:00
wgrant12:55:13 < lifeless> sure, because thats how you're handling repeated reads (and optimising for finding-what-to-process-next).03:01
StevenKHmm03:01
wgrantThat suggests you think it's not fine :)03:01
lifelessI think for incremental and startup performance you don't want to read the file content at all unless you have work to do, so storing a stat fingerprint along with the amount read is important.03:01
wgrantlifeless: Ah, perhaps once the number of files is about 3 orders of magnitude higher, yeah.03:02
lifelessI think that having linear growth of files to handle, which is going to jump by three in the short term, suggests that considering all files each run may have a shorter lifespan than you think.03:02
wgrants/files/files per parser instance/03:02
wgrantlifeless: It *should* have a shorter lifespan than I think.03:02
wgrantlifeless: But it won't.03:02
lifelessso, right now, I'd do a schema patch, adding a string. I'd use bzrlibs stat finger print function to fingerprint files, and I'd set that for files where we processed to the end of the file as we saw it.03:03
lifelessI'd nag lifeless about getting near-realtime log shipping in place and prepare a cunning plan for doing it better later this year.03:04
wgrantThe fingerprinting thing is completely unhelpful for the problem we've encountered here.03:05
wgrantIt's an unrelated optimisation.03:05
wgrantThe problem is that we have this file.03:06
wgrantWe haven't completely parsed it yet.03:06
wgrantBut it hasn't changed since we last tried.03:06
lifelesscan you explain why that makes it unhelpful ?03:06
wgrantWe can't skip based on the fingerprint check unless we only set the fingerprint once we hit EOF.03:07
wgrantOtherwise we'll skip files that we haven't completely parsed.03:07
StevenKMy Any addition doesn't work. :-(03:07
wgrantAnd we haven't hit EOF here; that's the entire problem.03:07
wgrantSo the fingerprint check is unrelated.03:07
wgrantStevenK: Oh?03:07
lifelesswgrant: I don't follow. Consider this logic:03:08
StevenKwgrant: Can you prod at http://pastebin.ubuntu.com/1074183/ after you and lifeless are done. Time for lunch.03:08
lifelessdoes the fingerprint match the file (no fp matches no file)03:08
lifelessonploooooooooooooopiuuuuu03:08
lifelesscat03:08
lifelessif yes, open, ungzip and seek to the last recor~6~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~03:09
lifelessBAH03:09
wgrantStevenK: +class Any(CompoundOper):03:10
wgrantStevenK: ANY is not a compound operator03:10
lifelessif yes, open, ungzip and seek to the last recorded offset and process from there. On halt, if we finished the file, store the fingerprint we obtained at the beginning of this.03:10
wgrantStevenK: It's a function.03:10
StevenKwgrant: It's a NamedFunc ?03:10
wgrantRight03:10
wgrantlifeless: That's precisely what I described, right.03:11
StevenKwgrant: That looks better, but I think the SQL is still bong: SELECT Branch.id FROM Branch WHERE Branch.id = 77 AND Branch.information_type IN (1, 2) AND COALESCE((Branch.access_grants)&&(SELECT ARRAY_AGG(TeamParticipation.team) FROM TeamParticipation WHERE TeamParticipation.person = 243654), false) AND (Branch.access_policy) = ANY((SELECT AccessPolicyGrant.policy FROM AccessPolicyGrant JOIN TeamParticipation ON TeamParticipation.team = Acce03:12
wgrantStevenK: Where'd the array_agg go from the APG bit? If you can remove that easily, as it seems, then just say access_policy.is_in(apgs). No need for ANY03:13
wgrantlifeless: Oh. I missed the "and I'd set that for files where we processed to the end of the file as we saw it" you said earlier as I was dying of optimism poisoning from the following line.03:14
wgrantFalse optimism poisoning, that is.03:15
lifelesswell03:25
lifelessyou can call it that03:25
lifelessanyhow03:25
lifelesswhats the issue with that algorithm? How will it not work ?03:25
lifelessI'd hate to be suggesting something bong03:25
wgrantWith the crucial "only set on EOF bit" that I missed you saying, it should be fine.03:26
lifelessthere are other ways to fpr files, but the bzr one is shall we say battle tested.03:26
wgrantHeh03:27
StevenKwgrant: http://pastebin.ubuntu.com/1074238/04:10
wgrantStevenK: Indeed,04:13
StevenKwgrant: Indeed?04:18
wgrantStevenK: That looks roughly correct. Does it work?04:18
StevenKIt returns []04:18
StevenKI think the public query should not be ANDed in04:18
wgrantAh04:18
wgrantYes04:19
wgrantThat should be an or04:19
wgrantpublic OR artifact grant OR policy grant04:19
wgrantI meant the complex bits of the query looked correct :P04:19
StevenKHaha04:19
StevenKwgrant: Do you think returning the branch id is okay? I'm really interested if one of those three is True for branch.id == self.id04:20
StevenKCan't COALESCE against three, I guess04:20
wgrantStevenK: Why would you want to COALESCE against three?04:21
wgrantYou basically want to SELECT 1 FROM Branch WHERE id = $BRANCH_ID AND $VISIBLE04:21
wgrantIn this case you're selecting branch.id, but that's just as good04:21
StevenKHm, still returns []04:22
StevenKOh, but it's supposed to. I think this is the pre-subscribe check04:23
wgrantWhat's it meant to return?04:23
wgrantAh04:23
StevenKwgrant: So my next question is I want something more elegant than "return Store.of(self).find((Branch.id,), conds) == [self.id]04:24
StevenKWith an ending double quote04:24
StevenKAnd putting a list around the store, but handwave04:25
wgrantStevenK: More like 'return not store.find(1, Branch.id == self.id, $VISIBLE_FILTER).is_empty()'04:26
wgrant(Storm might not like the 1 very much, but it doesn't at all matter what it is)04:29
StevenK(1,), even?04:30
StevenKAttributeError: 'int' object has no attribute '__dict__'04:30
wgrantYeah, it's probably trying to autotable it04:30
wgrantWhich won't work well04:30
StevenKSame error with (1,) :-(04:31
wgrantMight as well just say Branch, then04:31
StevenKYeah, we don't give a toss about what it returns04:32
StevenKwgrant: http://pastebin.ubuntu.com/1074270/04:40
wgrantStevenK: Does it work?04:41
StevenKwgrant: Nope04:41
StevenKReturns 0 rows before subscription which is fine, and 0 rows after which isn't.04:42
wgrantHave you checked the contents of AAG?04:42
StevenKNo AAG created :-(04:48
wgrantThat would be the issue. Missing FF?04:49
StevenKIBranch.subscribe() does not use a FF to guard04:49
StevenKHm, maybe the sharing service still thinks subscription confers visibility04:50
StevenKSort of. getUtility(IAllBranches).visibleByUser() needs porting04:53
StevenKWhich I'm trying to work out without my brain dribbling out of my ears.04:53
StevenKwgrant: http://pastebin.ubuntu.com/1074301/05:14
wgrantStevenK: You'll want to remove the helper methods that you've eliminated references to, but indeed.05:16
wgrantStevenK: It may be worth seeing how the query looks if you reimplement the Branch method in terms of BranchCollection, as as I did with Bug.userCanView05:17
StevenK% bzr damage05:21
StevenKUsing submit branch /home/steven/launchpad/lp-branches/devel05:21
StevenKHealing: 4 lines of code05:21
StevenKI approve05:21
StevenKHm, neat. permission denied on APG05:27
=== matsubara-afk is now known as matsubara
=== Ursinha is now known as Guest51695
wgrantstub: Hi, I've got a couple of DB reviews up for you when you have some time07:03
huwshimiThis is the appropriate way to run a single test right? 'testr run -- -t lib/lp/bugs/tests/bug.py'07:06
huwshimiIt should fail, but it's passing07:07
stubwgrant: k07:08
wgranthuwshimi: For Python modules you want to specify something like '-t lp.bugs.tests.bug'07:08
wgranthuwshimi: But lp.bugs.tests.bug doesn't have any tests in it. It's just a helper module.07:08
wgrantPerhaps you mean lp.bugs.tests.test_bug07:08
huwshimiwgrant: Oh, I see where I've gone horribly wrong. Thanks.07:10
huwshimiI've changed a table to a list and in lib/lp/bugs/tests/bug.py the print_bugfilters_portlet_unfilled() helper wants to grab the table and print its contents with print_table(). Any suggestions on how to print it now that it's a ul?07:16
StevenKprint_table is probably defined somewhere in that test07:17
huwshimiStevenK: It's defined in lp.testing.pages07:18
huwshimiStevenK: And there doesn't seem to be print_ul or anything07:18
StevenKhuwshimi: BeautifulSoup?07:19
huwshimiStevenK: You mean, just print out the contents?07:19
StevenKhuwshimi: You could use a doctest matcher, but no, I don't mean just print out the contents.07:21
StevenKhuwshimi: BeautifulSoup will parse the output of print_bugfilters_portlet_unfilled() and you can then inspect it with asserts.07:22
wgranthuwshimi: This is in a doctest, right?07:23
huwshimiwgrant: No, this is in the helper07:23
wgranthuwshimi: Well, yes, but it's probably eventually used by a doctest. The function prints it out, so you don't want to directly make assertions. I'd try the usual extract_text(find_tag_by_id(...)) pattern, see if that yields usable output.07:24
huwshimiI see so I just need to get it to extract the same data in the same format as the table printer was doing. I think I'm on it.07:26
wgrantRight, hopefully that's easy enough.07:27
wgrantIf you can't get it exactly the same, get something that looks sensible and change the callsites :)07:27
huwshimiThanks :)07:29
huwshimiWell, that wasn't too bad07:53
adeuringgood morning07:55
wgrant:(08:11
wgrantpostgres upsets me08:11
wgrantIt won't use an index including 'foo IS NULL' to satisfy 'foo IS NOT NULL';08:11
wgrantczajkowski: Ew08:18
czajkowskiwgrant: morning to you too08:19
wgrantNot really. But we like OpenStack, so we might be able to arrange something with a sufficient degree of evil.08:19
wgrantIndeed, morning.08:19
czajkowskiwgrant: wrong channel08:19
wgrantIMO it belongs here, but I guess it's arguable.08:20
czajkowskiit's early08:20
lifelesswgrant: whats the specific index its not using ?08:21
wgrantlifeless: bugsummary(distribution, sourcepackagename, tag IS NULL) WHERE distribution IS NOT NULL08:22
wgrantlifeless: It'll use that for a tag IS NULL query08:22
wgrantAnd if I s/tag IS NULL/tag IS NOT NULL/ on both it'll work08:22
lifelesstag is not null where distribution is null ?08:24
wgrantlifeless: Confused.08:25
lifelesswgrant: may be a stats thing not an index thing; e.g. many rows predicted08:25
wgrantlifeless: That's pretty much disproven by the fact that inverting the last element of the index causes the query to use it.08:25
lifelessoh hangon.08:25
lifelesswgrant: you say'08:26
lifeless20:22 < wgrant> lifeless: It'll use that for a tag IS NULL query08:26
lifeless20:22 < wgrant> And if I s/tag IS NULL/tag IS NOT NULL/ on both it'll work08:26
lifelessso both ways around work.08:26
lifelessOr you mistyped somewhere.08:26
wgrantYes. Both ways work.08:26
jmlhello08:26
wgrantBut only with the matching index.08:26
lifelessso08:26
wgrantBut both are satisfiable from either.08:26
lifelessbugsummary(distribution, sourcepackagename, tag IS NULL) WHERE distribution IS NOT NULL looking for 'tag IS NOT NULL' is what doesn't work ?08:26
wgrantYes.08:27
lifelesswgrant: I presume you have a distribution in the query as well ?08:27
wgrantlifeless: Naturally.08:27
wgrantOtherwise it wouldn't use that index :)08:27
lifelesswell, as it isn't using the index, I felt I should check.08:28
lifeless:>08:28
lifelessjml: elloh08:28
wgrantjml: Re. your email, that excludes test deps, I guess?08:28
lifelesswgrant: so why index tag IS NULL; looking for the specific sub summary?08:28
wgrantlifeless: We currently maintain separate indices on tag IS NULL and tag IS NOT NULL08:29
wgrantWas trying to avoid that.08:29
lifelesshmm, I forget the detail08:29
lifelesshave you tried DISTINCT FROM ?08:29
lifelesswgrant: and NOT tag IS NULL ?08:30
jmlwgrant: yes, the output. the script is just a Pythonically aware grep for 'import'08:30
jmlwgrant: I'm probably going to buff it up so it recurses directories (or maybe packages)08:31
jmland ignores internal imports08:32
jmland then maybe consults some kind of db to figure out which things are modules and which aren't08:33
jmlbut, you know08:34
wgrantlifeless: NOT (tag IS NULL) has the same behaviour as tag IS NOT NULL08:36
wgrantWhich doesn't really make much sense.08:36
stubwgrant: It never will, because an IS NULL index contains no information about rows without a null in that column.08:36
wgrantstub: Not a partial on tag IS NULL08:36
wgrantstub: But an index with the boolean "tag IS NULL" as one of its elements.08:36
lifelesscomputed indices are evil08:37
lifelesswgrant: why doesn't NOT (tag is NULL) behaving the same as tag IS NOT NULL, not make sense ?08:37
wgrantlifeless: While you're here, what do you think about soren's question in #launchpad? We'd need to set archive.signing_key on the new PPA fairly soon after creation.08:38
wgrantlifeless: Well, it was possible that it didn't consider IS NULL and IS NOT NULL to be inversions of the same operator.08:38
wgrantBut it clearly does.08:38
lifelessI was hoping it might :)08:39
lifelesswhat about DISTINCT FROM ?08:39
lifelessuhm, so I think doing what soren asks would be good; be better to automate it (at least to the extent of sysadmin only apis).08:40
lifelessor sometihng.08:40
lifelessI'm not a huge fan of manual fiddling, but this is a case where our model makes decisions more stressful for users, its largely incumbent on us to deal with the side effects.08:41
wgrantExactly.08:41
wgrantAlthough APIing this sounds beyond perilous.08:42
stubI think NOT( ) has to assume the contents of the parenthesis is true, false or null. IS NOT NULL is a single operator and it is known it returns true or false.08:43
stubwgrant: You go into detail about bugsummary journaling performance, but isn't that irrelevant for writes atm?08:44
wgrantstub: Not at all. It's the most expensive bit of big bug updates apart from the usual Storm n+1 query thing.08:45
stubwgrant: Yes, and are big bug updates a performance problem?08:46
wgrantstub: We have about 3 timeout bugs about them.08:46
wgranteg. approving Linux SRUs08:46
stubDo you know what sort of number of bugs are being done in a batch?08:46
wgrantCan create about 50 tasks in one transaction, because the kernel team are insane.08:46
wgrantWe end up with bugs with 80ish tasks, 2/3 of them created in one transaction.08:47
wgrantIf the bug has 8 tags, which is fairly common, that's a lot of journal rows.08:47
wgrantThe last task will unjournal 79*8*afew rows, journal 80*8*afew rows.08:47
stubHmm... So I'm not against performance improvements, just improving performance doesn't really solve scalability problems.08:47
wgrantIt stops bulk task addition from being quadratic.08:48
stubWe can fix it so creating 80 tasks works on our hardware in a single transaction, then some sod will try and create 81.08:48
stubAnd things like dropping foreign key constraints mean we should have DELETE triggers too (they are there already I think?)08:51
wgrantstub: Well, the FKs really shouldn't have been on these tables at all. They actually cause correctness bugs.08:51
stubWe normally don't bother with them in this situation. Can't recall why we had them.08:52
stubI'm wondering if we rely on ON DELETE CASCADE behaviour somewhere.08:52
wgrantWe don't.08:52
wgrantEven if we did, it would be wrong.08:52
wgrantSince the deletion process would also remove the references in the original tables.08:53
wgrantWhich would create journal entries.08:53
wgrantSo you'd end up with negative counts in bugsummary after journal rollup.08:53
wgrantNegative counts are bad :)08:54
wgrantThough we have some :/08:54
wgrant289 bugsummaryrows with count < 0 :/08:55
stubThat would be a logic flaw somewhere in the triggers, wouldn't it?08:56
wgrantThere's a lot of corruption in Ubuntu's kernel bugs. I've identified a couple of bugs in the initial population code that cause some of them, but it doesn't explain all of it.08:57
wgrantThe triggers themselves weren't significantly buggy AFAICT. Just the initial population query.08:57
wgrantlifeless: Given https://pastebin.canonical.com/69406/  I propose https://pastebin.canonical.com/69407/08:58
stubwgrant: I wonder if we should dump doing this in triggers entirely, but I guess that would require reliable async job processing.09:04
jammgz: did you see that there is one more failing test on python-2.7? It looks like a doctest that is (again) overly sensitive.09:05
wgrantstub: I came close to doing that when doing BugTaskFlat, but before we can sensibly do that we need both reliable async job processing and something resembling a sensible data access layer in the app.09:05
stubr=stub09:10
wgrantstub: Thanks.09:13
wgrantstub: Other disclosure stuff (that doesn't touch the existing bugs model so directly) is using celery-based eventual consistency fairly extensively.09:14
stubYeah, just wondering if we are investing too much in rabbit. Complex beast, and more complexities to ensure we don't drop tasks.09:16
stubUsing PG as the task queue for celery could suit us much better, and I think celery supports that out of the box.09:17
wgrantIndeed, although it hasn't been problematic yet, and we're not using it for absolutely critical stuff.09:18
stubIIRC We haven't chosen it for some things because of this problem, or chosen more convoluted implementations.09:19
wgrantstub: Right, the recent port of the classical job system to celery involves a regular task to reschedule jobs that remain pending but without a rabbitmq job.09:25
wgrantWe don't have pure-rabbit jobs.09:25
stubYer, that extra complexity.09:25
stubIs that generic, or do we have to do this every time?09:25
stubGuess you get it if you use the lp job system.09:26
wgrantstub: It currently only support branch scan jobs, I believe, but it's pretty simple to genericise. I imagine it will be once it's fully deployed and tested for branch jobs.09:27
lifelesswgrant: +109:30
wgrantlifeless: On the signing_key SQL?09:31
lifelessyes, caveat being to follow the normal procedure :)09:31
wgrantOf course, thanks.09:31
jamlifeless: I thought we didn't run tests for code that wasn't launchpad proper. I'm getting a test failure on python-2.7 inside the launchpadlib test suite (in the egg)10:36
jamand the plot thickens. As near as I can tell, you run the lplib tests by running "python setup.py test", but that suite doesn't load the doctests... so they are only being run by launchpad bin/test, and it is in an egg, so we need a new version of launchpadlib, a new release, and then updating the versions.cfg file10:43
jambac: I noticed you are prominent in launchpadlib development. Can I ask you some questions?10:43
czajkowskijam: dont think he's on today10:44
jamczajkowski: thanks10:44
jamI imagine neither is benji, or anyone else in the US10:44
czajkowskijam: as is most of USA10:44
wgrantjam: Some launchpadlib tests only run as part of the Launchpad test suite, since they need to test against Launchpad.10:53
Bert_2Hi, I'm about to do some changes to my newly rockfuel'ed launchpad, now I am to make some config changes to prevent launchpad from going into dev mode, I was hoping to use this page: https://dev.launchpad.net/WorkingWithProductionConfigs but while it says it describes editing configuration it actually doesn't seem to explain anything else apart from repo and branch stuff (which are not as important as getting out of devmode), so help ?10:54
jamwgrant: so if I have a fix for a launchpadlib test, I have to get it committed into launchpadlib trunk, then officially packaged and uploaded, then update versions.cfg, right?10:57
jambut I can't run that test outside of launchpad itself...10:58
jamwgrant: do you just hack the code in the egg directory directly?10:58
wgrantjam: You're correct on all counts.10:59
jam /cry11:00
wgrantBert_2: That page mostly documents how Canonical LP engineers should work with our production configs. The closest thing to a public production config example is the old configs from before the open-sourcing, in r8666. Are you aware of the trademark and copyright restrictions that apply to the Launchpad name and images?11:02
Bert_2wgrant: I'm not fully aware of that no, I thought I should only remove the launchpad logo11:03
wgranthttps://dev.launchpad.net/LaunchpadLicense11:04
Bert_2k, so basically if I lose the logo and icons and ask permission to use the name launchpad I'm fine ?11:04
jamand the hole deepens. The specific contents of the config file is set by lazr.restful.authorize_auth.OAuthAuthorizer...11:05
wgrantBert_2: You'd be best off changing the name.11:05
wgrantjam: You need to change the config file?11:05
jamwgrant: we have a doc test that asserts the exact bytes of the content11:05
jamand in py2.7 it changed 2 things (a blank entry now has a trailing space, and the order is changed)11:06
jamI'm guessing the ordering is defined by iter(dict)11:06
jamand that 2.7 just does it differently.11:06
jamIt was a very brittle test, waiting to break11:06
Bert_2wgrant: and can that be done through the configfile or am I forced to change the name by hand in every python file ?11:07
wgrantjam: Oh, lovely. Can you just change the test to be a bit more flexible?11:07
jamwgrant: I was thinking to put a "sorted()" around it, yeah.11:08
wgrantBert_2: There's no config option. The one other production instance I know of basically ran sed over all the python files and templates.11:08
jamI'm just struggling to even run it in isolation :). I think I at least have a plan of attack.11:08
wgrantjam: Heh11:08
Bert_2wgrant: alright, that can be done, and how about going from development to production mode, how's that done, cause the wikipage isn't all that clear :S11:09
wgrantBert_2: You'll see the existing public configs under configs/11:10
wgrantYou select a config to use with the LPCONFIG environment variable. It defaults to 'development'.11:10
jamwgrant: so I think I'm going to monkey patch ConfigParser.SafeConfigParser in part of a doc test. That sounds sane, right? :)11:10
cjwatsonAny objection to me updating the copyright year in LaunchpadLicense?11:10
wgrantcjwatson: I was going to do that myself, but feel free :)11:10
cjwatsonDone11:11
jam(no monkey's will be patched, I swear)11:11
wgrantBert_2: I'd copy the development config, and fix up the paths and domains. In launchpad.conf of your new config, remove the 'devmode on' line. Most of the settings are in launchpad-lazr.conf.11:11
wgrantjam: If it works...11:11
Bert_2wgrant: alright, so if I change paths, domains, the devmode line and then the usual emailaddresses and numbers for question deprecation etc. I'm fine ?11:12
=== matsubara is now known as matsubara-afk
ivorygmb: could you take a look at this? https://code.launchpad.net/~ivo-kracht/launchpad/bug-921901/+merge/11323411:14
gmbivory, Sure - thought the topic's out of date :)11:15
=== gmb changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: - | Firefighting: - | Critical bugs: 4.0*10^2
wgrantBert_2: IF you want it to send email you'll also have to reconfigure a couple of things. zcml/package-includes/mail-configure-normal.zcml redirects all email to root@localhost, and you might want to enable the immediate_mail.send_email in launchpad-lazr.conf as well. You might also want to do away with the sampledata, and instead create a clean DB using the scripts in https://code.launchpad.net/~wgrant/launchpad/bootstrap-db-from-scratch11:18
Bert_2wgrant: awesome, I thought going out of devmode would give me a clean db, so thx for saving me the frustration :D11:19
wgrantBert_2: Changing an appserver configuration file fortunately won't erase your database :)11:27
Bert_2wgrant: I thought the database was generated by a python script that read the config, but I didn't really take the time to read the sourcecode, I've barely been out of exams and this development platform is rather important for our student's body11:43
jamwf11:52
jamwgrant: do you have any experience rolling out an updated EC2 AMI?11:52
jamI found EC2Test on the wiki, I'll try following that one11:56
wgrantjam: Why do you need to?11:58
wgrantBut yes, that page is correct.11:58
jamwgrant: we want to get the test suite running w/ python-2.711:59
jamso that we can get production moved to precise afterwards11:59
wgrantjam: Not until we've scheduled it on production we don't.11:59
jamwgrant: I'm working with flacoste to do... 1) upgrade db-devel to run py2.7 along with ec2-test running py 2.711:59
jamand then leave devel running 2.611:59
jamonce we have both passing12:00
jamthen we can upgrade prod12:00
jambut we can't upgrade prod until we know it works, and have a clean test suite.12:00
jamhe wants ec2-test to fail early rather than going into testfix on db-devel, and buildbot will run on devel w/ 2.6 before it goes to qastaging12:01
wgrantI'm not sure it's a great idea to upgrade ec2 unless we know we're going to be upgrading production within a very few weeks, or we're going to be in trouble when people start using 2.7isms and breaking buildbot.12:01
jamwgrant: so we need a migration path, which I was asked to work on. Certainly each step gets reviewed, etc.12:01
wgrantSure.12:02
wgrantLet's not just actually deploy the 2.7 image to everyone until we have a schedule worked out.12:02
flacostewgrant: the plan is to upgrade quickly, really not more than a few weeks13:06
=== salgado is now known as salgado-brb
rick_h_adeuring: ivory https://plus.google.com/hangouts/_/3dd64d7a6ec2e029e3e9abfc99a72a95ff0bef6c?authuser=0&hl=en-US13:30
=== salgado-brb is now known as salgado
jammgz: did you see my python-upgrade bug? I submitted a fix for it to launchpadlib14:07
=== abentley changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: abentley | Firefighting: - | Critical bugs: 4.0*10^2
mgzI saw the mp14:08
jam it will take a bit to finish that one off, but that was the last failure after manually merging your branches.14:08
jammgz: did they all land in trunk?14:08
mgzguess I can review that? not sure what the funny lp reviews system applies to14:08
mgz^jam, yup landed now and tickets moved horizontally with lp2kanban14:09
jammgz: what is the approval process for launchpadlib code?14:54
mgzjam: no idea, my experience has been "bug someone after mp has been forgotten by everyone"14:54
mgzor was that a lazr thing...14:54
jammgz: I think it is a more-than-one-projects thing14:55
jambut everyone I've seen in launchpadlib on lp's "active in this project" is asleep or on holiday. so I might try later.14:55
jameg, tomorrow14:55
bacjam: have a look at https://dev.launchpad.net/HackingLazrLibraries14:55
jam /wave bac, not doing July-4 stuff yet?14:56
bacjam: it has good stuff, especially if you need to make a release14:56
bacbah, our founding fathers would be embarrassed by our slothful coworkers.  :)14:56
mgzwe probably don't need a release, if we can just bump the lp dep to a specific revno14:56
jammgz: it is an egg, not a sourcecode14:57
* mgz is not sure if that's a done thing though14:57
jambac: so who actually commits stuff to launchpadlib trunk?14:58
bacjam: i think the short answer is 1) make a dev branch, don't work in your copy of trunk, 2) make a MP and get it reviewed, 3)  merge to trunk and commit with a good message including [r=reviewer], 4) push.  you'll then need to decide on whether to make a release, create an egg, and bump versions.cfg in a LP branch.  probably best to just read that long wiki page, now that i think about it.14:59
jam(a member of the lazr-developers team)14:59
jambac: oh, I know all about that stuff :)14:59
jamI've done 1-314:59
jam"merge to trunk", you just do it directly?14:59
jamor it goes through pqm (the page says pqm)14:59
bacjam: yep14:59
bacno bots14:59
bacoh really?  ok, my memory is faulty then15:00
* bac defers to the wiki15:00
jamI'm not sure I'm in lazr-developers15:00
jamI'm in launchpad15:00
jam(launchpad-dev? whatever that group is)15:00
jambac: looks like I can commit straight to lp:launchpadlib15:02
bacjam, you should be in this group, indirectly: https://launchpad.net/~lazr-developers15:02
bacyep15:02
jambac: it looks like 'bin/test' doesn't run the doctests either. Only running "bin/test" in launchpad itself, but not in launchpadlib15:03
bacyes, iirc getting the test to run is done via a LP branch15:04
ivoryabentley: could you review this for me? https://code.launchpad.net/~ivo-kracht/launchpad/bug-562532/+merge/11341415:08
abentleyivory: sure.15:08
abentleyivory: How does that remove a page request?15:09
ivoryabentley: allthe other edit buttons are sprite icons, only those were <img> tags15:10
abentleyivory: I'm used to sprites looking like '<span class="add sprite">'.  Why is this one different?15:13
rick_h_abentley: it's removing the request to: https://launchpad.net/@@/edit via the src of the <img> tag. and referencing the sprite in the CSS.15:14
abentleyrick_h_: ivory already answered that.  Do you know what's up with this <button class="lazr-btn yui3-activator-act">?  I'm used to sprites looking like '<span class="add sprite">'.15:16
jambac: I cannot register with pypi, as my user doesn't have perms. I've done the upload and changes to trunk.15:16
jambac: do you have rights and can 'bin/buildout setup . register' for me?15:16
ivoryabentley: i just copied it, i cant really tell you why it lookslike that ...15:17
rick_h_abentley: ah no, I'm guessing he copied it from somewhere.15:17
rick_h_ivory: where did you grab it from?15:17
ivoryrick_h_: from the other implementaions of that icon15:17
jambac: I'm leaving for end-of-day now. If you get a chance, please do. Or poke benji to do so? I'll try to address it tomorrow if it hasn't happened yet.15:18
abentleyivory: Okay, if there's no reason not to use "sprite edit", I think we should do it that way.15:19
rick_h_abentley: adeuring heads up, afk for a sec while I pick up the family from the parade. brb15:21
adeuringrick_h_: ack15:21
bacjam: ok, i'll look at it15:35
czajkowskibac: thought you weren't around today ?15:37
bacczajkowski: i decided to work today and bank a swap day.15:43
beunohow independent of you15:46
bacbeuno: yeppers15:47
czajkowskibac: cool15:50
=== matsubara-afk is now known as matsubara
=== salgado is now known as salgado-lunch
ivoryabentley: wouldn't the icon be on the other side if "sprite edit" is used?16:31
ivoryabentley: additionally i should mention that the <button class="lazr-btn yui3-activator-act"> is already used in this template16:34
abentleyivory: I wouldn't think so.  In any case, "lazr-btn yui3-activator-act" does not mean "edit icon" to me, and I doubt it would for most LP developers.16:35
bacjam: lplib 1.10.1 uploaded to pypi.  thanks for the fix!17:13
czajkowskibac: abentley would one of you mind lookinat https://bugs.launchpad.net/launchpad/+bug/1020983  I don't know about receipes . please.17:13
_mup_Bug #1020983: Recipe default text versioning is wrong <Launchpad itself:New> < https://launchpad.net/bugs/1020983 >17:13
=== salgado-lunch is now known as salgado
bacczajkowski: he may have a point in that bug.  i'd mark it triaged/low.17:43
czajkowskibac: ack17:46
lifelessjam: wgrant: pass --develop to buildout to point it at the launchpadlib source tree and you can edit it in situ - it won't use the egg.18:03
abentleyczajkowski: I know about bzr, but not about packaging.  IMO, he's wrong-- the version number is unique, can't be confused with a release version number, and that's all that's required.18:04
lifelessabentley: there is another constraint you have missed, and that the bug filer hasn't called out either.18:07
lifelessabentley: but I think its what motivates the filing; the packages produced by the recipe should version higher than the package they are derived from but lower than the next version of that package.18:08
lifelessI'm not entirely sure his proposals are much better at that, though they will handle some cases better.18:09
abentleylifeless: I think this is a distributed numbering problem that can't be completely solved.18:10
abentleylifeless: It's quite possible that a point release "should" version lower, and I think it's possible for reasonable people to disagree on which should version lower.18:12
abentleylifeless: Depending on the nature of the branch, of course.18:13
lifelessyah18:17
lifelessI guess I'm mainly advocating that we get a clear understanding of what the default intends to achieve and document that somewhere discoverable.18:18
lifelessthen, if the default is mismatched with our intent we can change it, and bugs that want to change our intent we can actually reason about18:18
abentleylifeless: that makes sense.18:27
abentleylifeless: So if we grant that we want to sort later than the version of the packaging we're based on, that suggests we should use debversion, not debupstream.18:37
lifelessI agree.18:38
lifelessI wonder if we wrote our original intent down, perhaps in a doctest or something.18:38
abentleylifeless: It was me and james_w flailing around, IIRC.18:39
lifelesshah :)18:39
abentleylifeless: I believe we were using debversion originally, so there may be a related MP.18:40
lifelessor something in bzr log18:44
lifelessok, time to prep for my joy of joy 3 1/2 hours of meetings.18:45
abentleylifeless: bug 652109 inspired this self-reviewed proposal: lp:~thumper/launchpad/recipe-default-version into lp:launchpad and I don't think there was any discussion of whether debversion would be more appropriate as a default.18:51
_mup_Bug #652109: Default Recipe text should contain {debupstream} <lp-code> <qa-ok> <recipe> <Launchpad itself:Fix Released by thumper> < https://launchpad.net/bugs/652109 >18:51
abentleylifeless: sorry, proposal url: https://code.launchpad.net/~thumper/launchpad/recipe-default-version/+merge/4166318:52
lifelessabentley: cool archaeology :)18:52
lifelessso, we're doing a drunkards walk more or less; might be time to take another step :)18:53
lifelessanyhow, I have to run - sorry.18:53
jambac, lifeless: Unfortunately, it appears that someone a19:16
jammade a launchpadlib 0.10.0 which was not landed in launchpad itself19:17
jambecause it breaks the test suite19:17
jam(issues with doctests expecting a reply_token? to be valid and not None)19:17
jamso we'll need an 0.10.2 after I can sort it out. It looks like benji did the 0.10.0, I wonder if he meant to land it, but found out late it broke the doctest suite because it doesn't run via 'bin/test' in launchpadlib alone.19:18
jamlifeless: i also haven't tracked down how you specify --develop to bin/buildout, since it seems to want a command first.19:18
abentleylifeless: https://dev.launchpad.net/Code/RecipeBuildDebversioning19:27
lifelessjam: https://dev.launchpad.net/HackingLazrLibraries?highlight=(buildout:develop)19:38
jamlifeless: thanks19:38
lifelessapplies to LP itself too19:38
cjwatsonI noticed launchpadlib test failures somewhere a few days ago, which then went away.  Is it possible 0.10.0 landed briefly and then was reverted?19:52
cjwatson(I'm afraid I don't remember the failures.)19:52
=== matsubara is now known as matsubara-afk
lifelesscjwatson: is sbsign intended to be 64-bit only? what about arm...?21:44
=== almaisan-away is now known as al-maisan
=== al-maisan is now known as almaisan-away
cjwatsonlifeless: not this month's problem22:24
lifelesscjwatson: is that 'no its not' ?22:25
cjwatsonthe interface doesn't mandate 64-bit-only, it's just that that's all that's implemented right now because that's the urgent bit22:25
cjwatsonor rather amd64-only22:25
lifelessgotchya22:25
cjwatson(I didn't write sbsign, just did the integration - but it's all the same binary format for UEFI so I'd be fairly surprised if it weren't basically trivial to adjust)22:26
cjwatsonI'm told Steve has a patch to make it handle i386, at least, but I didn't have that to hand when doing the QA22:27
lifelessI'm just being nosy22:27
lifeless100% curiosity22:27
StevenKwgrant, wallyworld: https://code.launchpad.net/~stevenk/launchpad/drop-populate-branch-aag/+merge/11346723:31
StevenKwgrant: And it seems BranchCollection makes use ClassAlias, so access_{policy,grants} might have to be added to the model.23:57

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!