/srv/irclogs.ubuntu.com/2011/11/24/#launchpad-dev.txt

StevenKIs twisted.internet.error.CannotListenError: Couldn't listen on any:2121: [Errno 98] Address already in use.00:22
StevenKIs ^ on ec2 normal?00:22
wgrantStevenK: It started appearing a week or two ago.00:34
wgrantBut yes.00:34
wgrantHmm.00:38
poolieStevenK, see my mail a week ago00:39
wgrantWe should probably present a subtle beta notification to everyone when there's something optional on a page, and allow them to opt-in then and there.00:39
pooliewgrant, i agree with the ammendment that not every beta is going to be available to every person00:39
poolienecessarily00:39
wgrantpoolie: Right, a feature flag to turn on the option for certain teams, I guess.00:40
poolieperhaps plenty of them arethough00:40
wgrantIIRC Gmail did something like this years ago.00:41
wgrantNot sure if they still do.00:41
wgrantLabs or something, but seems Google discontinued all of that.00:41
poolieyes, google love it00:41
pooliethey do things i really envy00:41
pooliesuch as correlating this with their logs00:41
poolieto see if it makes things slower, crashier, more popular00:41
wgrantI hope we can get to a point eventually where that would be useful.00:42
wgrantRight now it's not.00:42
pooliestevenk https://bugs.launchpad.net/launchpad/+bug/89420500:43
_mup_Bug #894205: spurious test_poppy failures <poppy> <soyuz> <spurious-test-failure> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894205 >00:43
wgrantWe need an ec2test -D00:43
pooliefor what?00:43
wgrantSo we can break into the instance once it hits that failure.00:43
wgrantAnd see what's going on.00:44
poolieoh postmortem00:44
pooliecan't be that hard00:44
pooliewouldn't ec2 test --attached -t '-D' do the job?00:45
poolieexcept perhaps you don't want to do that every time just in case it fails00:46
wgrantOr perhaps if we just immediately emailed on failure.00:46
wgrantOr indeed just ran a few instances and watched ec2 list00:46
poolieyeah00:46
pooliei thought ec2 list used to give a progress percentage00:48
wgrantNo00:48
pooliebut apparently not any more?00:48
pooliewell00:48
wgrantIt never did.00:48
wgrantJust a time.00:48
pooliemaybe i'm thinkingo f bzr pqm00:48
pooliei guess once you're experienced it's enough to know it takes about 4h00:48
wgrantYeah00:48
pooliethe other thing i would like to do here is make the flags admin ui a bit better00:50
poolieoh and developer control on *staging00:50
poolieso much to do00:50
wgrantwallyworld__: What if we said something like "Albert Einstein | [Proprietary: everything (!)] [Security: 6 bugs, 4 branches (!)] [(+)]" (where (!) is the edit icons, (+) is the add icon)00:52
wgrantwallyworld__: Gets rid of the obscure "observer" and "restricted observer" terms.00:53
wgrantRemoves the third column.00:53
wgrantAnd shows more directly what's what.00:53
wallyworld__wgrant: sounds reasonable at first glance00:53
pooliewgrant, so00:53
poolieself.factory.makeBugComment()00:54
pooliefails if i don't previously log in as a user00:54
poolieis this a bug? it seems inconsistent with other factory things00:54
wgrantpoolie: A bug, but an unsurprising one.00:54
pooliethis is from https://code.launchpad.net/~mbp/launchpad/888353-microformats/+merge/82767 circa line 5900:54
=== micahg_ is now known as micahg
wallyworld__StevenK: wgrant: have you used ec2 test recently? does it work for you?01:15
StevenKec2 test and ec2 land are mostly the same01:15
wallyworld__yes, that's what i thought. but first ec2 test complained there was no submit branch01:15
wallyworld__so i added one to branch.conf01:15
wallyworld__then it gets to starting ec2 and complains there's a missing revision01:15
StevenKWhat are you trying to do?01:15
StevenKOh, you do everything in one working directory, don't you?01:15
wallyworld__just try ec2 test to confirm it still works for me01:15
wallyworld__i have a lightweight checkout01:15
wallyworld__i guess it needs submit_branch=... so that it knows where to grab the code from inside the ec2 instance01:15
wgrantwallyworld__: I used ec2 test 12 hours ago.01:15
wgrantWorked fine.01:15
wallyworld__hmmm. don't know what broken for me then :-(01:15
wgrantwallyworld__: rocketfuel-setup adds a submit_branch01:15
wgrantTo ~/.bazaar/locations.conf01:15
wallyworld__wgrant: mine has a submit_branch:policy = appendpath01:16
wgrantEr01:16
wgrantreally?01:16
wgrantpublic_branch:policy and push_location:policy should be appendpath01:17
wgrantsubmit_branch should not.01:17
wallyworld__yeah, it's been that way forever. maybe is mis-copied from somewhere01:17
wgranthttp://paste.ubuntu.com/747751/ is what I use.01:17
wallyworld__i'll try those settings. but i'm sure ec2 test used to work01:18
wgrantec2 land will work, since it gets it from the MP01:19
wgrantec2 test uses submit_branch.01:19
wallyworld__right, makes sense01:20
wallyworld__wgrant: so now the code in ec2 says public branch out of date01:21
wgrantwallyworld__: The public branch is probably out of date :)01:22
wgrantOr wrongly configured.01:22
wallyworld__yet, i'm running ec2 test from a branch i've just bzr pulled to01:22
wgrantIt has the same name?01:22
wallyworld__as what?01:23
wgrantYour local branch and LP branch need to have the same name.01:23
wgrantOr you need to configure public_location explicitly.01:23
wallyworld__ah, that may be it01:24
pooliei have been using ec2 test a lot recently01:24
wallyworld__my public_branch is bzr+ssh://bazaar.launchpad.net/~wallyworld/launchpad/devel01:25
wallyworld__my subit branch is bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel01:25
wgrantAh01:26
wgrantThere's the problem.01:26
wgrantpublic_branch needs to the be the public URL of the current branch.01:26
wgrantThat's why it uses appendpath01:26
wgrantso my observer-db-2 branch maps to bzr+ssh://bazaar.launchpad.net/~wgrant/launchpad/observer-db-201:27
wallyworld__wgrant: so, how to fix? edit ~/.bazaar/locations.conf?01:27
wgrantwallyworld__: Right.01:27
poolieif this is misconfigured i don't understand how ec2 would be working for you at all01:28
poolieor are you always giving the full url?01:28
wgrantIndeed, even ec2 land should fail here.01:28
wgrantIf public_branch is wrong.01:28
wallyworld__it's all worked so far for whatever reason01:29
wallyworld__i have a single working tree and use bzr switch01:29
wallyworld__if the working tree has the current branch i want to land, i just type ec2 land01:29
wallyworld__if the working tree is switched to a different branch, i used ec2 land fullurl01:30
wallyworld__wgrant: so my locations.conf is the same was yours, yet i get the public and submit branch mismatch01:32
wgrantwallyworld__: Right, I use multiple working trees.01:33
wgrantwallyworld__: Where are your branches?01:33
wgrantDirectly in lp-branches?01:33
wallyworld__yes01:33
wgrantHmm01:33
wallyworld__and the branch directories are empty01:33
wgrantThat should still work.01:33
wallyworld__except for .bzr01:33
wgrantwgrant@lucid-test-lp:~/launchpad/lp-branches$ bzr co --lightweight rip-out-accountpassword rip-out-accountpassword-co01:34
wgrantwgrant@lucid-test-lp:~/launchpad/lp-branches$ bzr info rip-out-accountpassword-co | grep public public branch: bzr+ssh://bazaar.launchpad.net/~wgrant/launchpad/rip-out-accountpassword01:35
wgrantWFM :/01:35
wgrantYou don't have it configured in branch.conf?01:35
wallyworld__my branch.conf has parent_location01:35
wallyworld__to point back to the parent devel chechout on disk which i pull into from lp and then branch from when i create a new branch01:36
poolie[/home/mbp/launchpad/lp-branches/work/.bzr/branches/]01:37
pooliesubmit_branch = bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/devel01:37
poolie[/home/mbp/ianb/lp-branches/] actually01:37
wallyworld__that matches my submit_branch when i type bzr info01:38
wallyworld__but ec2 test complains that something is out of date01:38
pooliewhat specifically?01:38
wallyworld__bzrlib.errors.PublicBranchOutOfDate: Public branch "bzr+ssh://bazaar.launchpad.net/~wallyworld/launchpad/devel" lacks revision "launchpad@pqm.canonical.com-20111123175547-p02iyjil3n13end4"01:39
wallyworld__which it does if i goto ~wallyworld/launchpad/devel01:39
wallyworld__but i'm running ec2 test from a branch which is fully up to date via bzr pull01:40
poolieare you wanting to run the tests on devel?01:40
wallyworld__i just wanted to get ec2 test working, because i ec2 demo is based off that and i want to see if ec2 demo works01:41
pooliethere is a special 'ec2 test --trunk' option that doesn't do a merge01:42
wallyworld__sounds like my config is hosed somehow, yet ec2 land, bzr pull, bzr push etc all work01:42
pooliewallyworld__, i think your setup has a confusion between a branch called devel owned by you, and the real devel owned by launchpad-pqm01:43
poolieit's kind of bad this state is possible01:43
pooliethe short answer is to either use --trunk or test one of your own real branches01:43
wallyworld__i tried one of my own real branches and got similar errors01:43
wallyworld__--trunk is fine for ec2 test but i really wanted to see if ec2 demo works with a branch containing some changes01:44
wallyworld__since it was stated ec2 demo is broken, and i wanted to see why/how01:45
pooliereally?01:45
StevenKwallyworld__: ec2 test should work fine01:45
StevenKSince ec2 land runs test with details from the MP01:46
poolieec2 demo also works fine for me01:46
pooliewallyworld__, give me the name of one of your branches?01:46
wallyworld__poolie: someone stated on the thread i started about interactive demos that ec2 demo was broken01:46
wallyworld__poolie: delete-all-bugtasks-88920201:47
wallyworld__StevenK: ec2 land works fine for me. but ec2 test doesn't (i haven't run it in ages, and tried a bit earlier for the first time in a while)01:47
wallyworld__perhaps my workflow of having a single local checkout of devel and branching off that and using a single working tree which i switch between isn't very common01:49
StevenKYa think?01:49
pooliei think it's common01:49
pooliei use colos which are similar enough01:49
pooliewallyworld__, can you be more specific about what's going wrong01:49
pooliedo you get a traceback or something?01:49
StevenKsteven@liquified:~/launchpad/lp-branches% ls -1 | wc -l01:50
StevenK13001:50
wallyworld__i have all my branch dires in lp-branches too01:50
wallyworld__but use devel-sandbox as my only dir with a working tree01:50
pooliethat's fine01:51
pooliethere is no good reason why that would affect ec201:51
poolielifeless, what actually is bitrotten in ec2 demo?01:52
wallyworld__poolie: i tried it again in my devel-sandbox with a previous branch and it worked this time01:52
wallyworld__poolie: but it didn't work in my local trunk working tree that i use to branch from01:53
poolieso01:53
pooliethat's probably best fixed by adding this configuration01:53
poolie[/home/ianb/launchpad/lp-branches/devel]01:53
pooliepublic_location = lp:launchpad01:53
poolieso it knows it's just a mirror01:54
wallyworld__ah, ok. that's great. i'll try it. thanks :-)01:54
wallyworld__poolie: that worked. the key though is public_branch. thanks.01:59
pooliewallyworld__, all good now?02:41
pooliei might have some lunch02:41
lifelesswallyworld__: I work as you do03:07
lifelesswallyworld__: I think its pretty common03:07
wallyworld__saves a lot of time checking out, and disk space too03:08
wallyworld__s/checking out/branching03:08
mwhudsonto be pedantic, i think you do mean checking out03:09
mwhudsonor tree building03:09
mwhudsonbranching is not the bit you wait for :-)03:09
wallyworld__it is if you branch from lp:launchpad rather than a local ondisk copy03:09
wgrantNo.03:10
wgrantWell, only if you don't have a shared repo.03:10
wgrantIn which case you are wrong.03:10
mwhudsonif you're using a shared repo, you're doomed :)03:10
mwhudson+not03:10
wgrantExactly.03:11
wallyworld__maybe i misremembered then. i could have sworn branching off lp:launchpad took a fair bit long than branching from local03:11
wgrantWell, that's orthogonal.03:12
wgrantYou can branch locally perfectly well without colo.03:12
wgrantbzr branch devel somethingelse03:12
wallyworld__right03:13
StevenKsteven@liquified:~/launchpad/lp-branches% du -sh .03:14
StevenK21G.03:14
StevenKCrumbs03:14
wgrantStevenK: How!?03:14
wgrantDo you never delete branches?03:14
StevenKNot usually03:14
wallyworld__i don't delete branches either and mine is 1.5GB03:17
wallyworld__and that includes a few non lp branches03:18
wallyworld__ls -1 | wc -l -> 25403:18
nigelbthat's crazy.03:40
nigelbI delete branches after they're deployed.03:40
wallyworld__nigelb: because i only use one working tree, the branches are just directories, and i keep them around locally for easy reference04:22
nigelbNice.04:23
nigelbI ran out of space on my hard-disk.04:23
nigelbToo much code :)04:23
StevenK/dev/mapper/sys-home   46G   39G  4.7G  90% /home04:23
StevenKBut I have another 390Gb that is unallocated. :-)04:24
nigelbI need to buy more hard-disk space.04:26
nigelbGoing to work with lots of code soonish.04:26
nigelbMorning poolie!04:26
nigelbErr, Afternoon, rather :)04:26
pooliehey there04:27
* StevenK deletes 10Gb of old branches04:29
wallyworld__StevenK: wgrant: attempting to make a multi-pillar bug private - do you agree that it should raise ValueError or do you prefer a custom exception type?04:32
wgrantA custom type.04:33
wallyworld__ockey dokey04:34
wgrantOMG04:35
wgrantPython 3.3 will have slightly less stupid Unicode.04:35
wgrantIt won't just support the BMP.04:36
StevenKYou mean like Perl 5.10 YEARS ago?04:36
* nigelb sees StevenK's subtle stab.04:40
wgrantBah, no jtv.04:45
StevenK(Pdb) p self.errors05:21
StevenK[WidgetInputError('owner', u'Maintainer', ConstraintNotSatisfied(<Person at 0xd3471d0 foobarbaz (Foobarbaz)>))]05:21
StevenK:-(05:21
wgrantHmm?05:23
StevenKProduct:+edit-people05:23
wgrantSure. But why is that saddening?05:23
StevenKBecause the error that is set is "Constraint not satisfied"05:24
StevenKIt's due to the validator failing, but I'm not sure how to deal with it properly.05:27
wallyworld__StevenK: could be trying to assign an open team as a pillar owner05:50
StevenKI am!05:50
StevenKThat's the point.05:50
StevenKJust the error message is horrible.05:51
wallyworld__that's how it bubbles up from storm i guess05:51
StevenKRight05:51
wallyworld__we should catch that and rethrow05:51
StevenKwallyworld__: Catch it where?05:51
StevenKIt's already in self.errors in the validate() of the view.05:52
wallyworld__hmmm. not sure then off hand05:52
wallyworld__in the model code somewhere05:52
wallyworld__so that by the time the view sees it, it is a nice error05:52
StevenKCan you think of an example?05:52
wallyworld__doesn't validate() take the form values prior to executing the form action?05:53
wallyworld__so a check could be done there that the owne isn't open05:53
wallyworld__and the field error set accordingly05:53
wallyworld__but the vocab shouldn't allow such a team to be picked anyway05:54
StevenKwallyworld__: The first thing validate() does is 'if len(self.errors) >= 1: return'05:54
wallyworld__so my guess as to how validate works is wrong :-(05:54
wallyworld__seems backwards05:55
wallyworld__shouldn't validate happen first, then the action05:55
StevenKIt does happen that way05:55
StevenKwallyworld__: I can explain this easier over mumble, since it seems I've confused you.05:56
wallyworld__StevenK: let me look quickly at the code05:56
StevenKwallyworld__: The view in question is ProductEditPeopleView05:57
wallyworld__StevenK: so looks like custom_widget('owner', .....) needs to have the correct vocab passed in05:59
wallyworld__so that only closed teams or people can be picked05:59
StevenKwallyworld__: But I didn't use the picker to generate that error05:59
wallyworld__you typed into the field directly?06:00
StevenKAnd hit the button, yes06:00
wallyworld__ok. so the picker vocab is still wrong :-)06:00
StevenKRight, so I can fix that at the same time, but my question still stands06:00
wallyworld__but i don't understand how validate() has errors already set06:00
wallyworld__at the very start of the method06:01
wallyworld__when no validation has been done yet06:01
StevenKLaunchpadFormView has done it? Or storm's\?06:01
wallyworld__perhaps. it seems apparent something is having a go at validation before the view nominated validate() is called06:02
StevenKAgreed.06:03
wallyworld__based on the form schema06:03
wallyworld__and the declared fields06:03
wallyworld__but that's the limit of my knowledge06:03
StevenKIt seems ugly to just reach into self.errors and pull out ConstraintNotSatisfied and replace it06:03
wallyworld__yup. let me check something06:03
wallyworld__StevenK: so the storm validator for closed teams raises a OpenTeamLinkageError06:05
wallyworld__not sure how this is munged into a constraint error06:05
StevenKBy Zope, I guess06:05
StevenKI have no idea how validators work06:05
wallyworld__i do seem to recall a doc test checking for OpenTeamLinkageError06:05
wallyworld__rather than ConstraintError06:06
wallyworld__sadly, i have in the past set breakpoints inside LaunchpadFormView to track the flow when a form submit is done06:07
wallyworld__you may need to do the same here06:07
wallyworld__bbiab06:07
=== almaisan-away is now known as al-maisan
huwshimiwallyworld__: I've just sent a somewhat lengthy reply to the list about prototyping. Once you've had a chance to read it I'm happy to have a bit more of a chat here if it helps (not saying you should read it now).06:13
wgrantjtv: Evening.06:32
jtvHi wgrant06:41
jtvSorry I had to email; internet connection where I was was just unusable.06:42
=== al-maisan is now known as almaisan-away
wgrantjtv: Did you end up testing recipe builds on DF?06:43
jtvYes.06:43
jtvAhem.06:43
jtvNot on DF.06:43
jtvOn staging.06:43
wgrantAh, I just tested on DF anyway.06:43
wgrantLooks good.06:43
jtvOn staging as well.06:43
wgrantjtv: there is a lot of commit/DatabaseTransactionPolicy/dosomething/commit which seems like it could be done in a single context manager. You also unexplainedly abort in a few places.06:47
wgrantIt's all a bit of a mess, but I guess that can't really be helped...06:47
wgrantWithout the rewrite that can't happen.06:48
jtvRight. Ideally, I'd like to abort everywhere.06:48
jtvBecause it's all supposed to be read-only transactions.06:48
jtvBut that might upset tests.06:48
wgrantYeah.06:48
jtvOTOH aborts may possibly be slower, depending on implementation.06:48
wgrantIf it wasn't for tests I would have required you to also have the context manager assert that it was already a read-only txn.06:48
jtvIt's a risk that's hard to manage.  :/06:49
jtvThat's also why I have all these minimal read-write regions: can't afford to have anything in there yield.06:49
wgrantIt would be nice if we could add a hook to the reactor to confirm that the transaction is read-only before and after each callback.06:50
jtvYeah.  I asked around, but Twisted doesn't seem to have anything like that.06:50
jtvUltimately Robert is right: the twisted-based daemon and the ORM-backed logic shouldn't even be in the same process.06:51
jtvAlthough adding more moving parts is no happier a prospect than twistification probably was.06:52
wgrantWell.06:52
wgrantA Twisted bridge between two XML-RPC sides is probably not that bad.06:52
wgrantAnd the current architecture *could* work well, I suspect, except that the code architecture is still from the non-Twisted world.06:53
jtvWe don't have just 2 sides though: there's the slaves, the builder logic, the build manager, and Librarian.06:53
wgrantAnyway, I'd really like to see you use 'with write_transaction:' or something like that, rather than the commit/DatabaseTransactionPolicy/commit06:53
wgrantLike with dbuser.06:53
wgrantMm.06:54
wgrantSort of, I guess.06:54
jtvHere's where I regret that it took so long.  I forget so many details.  What I remember is that write_transaction had both general problems and specific problems.06:55
wgrantI don't mean the existing write_transaction.06:55
jtvOne of the general ones was that it doesn't enforce a fresh transaction.06:55
wgrantWhich is a function decorator that also retries.06:55
wgrantJust something to encapsulate what you repeat.06:55
jtvSo you can get more of these weird “A and C have been done, but B aborted somehow” effects.06:55
jtvwgrant: Ah yes, another problem with read_transaction is that it allows you to write to the database.  I didn't want that in this case, because of the high probability that there would be accidental, untested database writes in the read-only sections.07:02
wgrantjtv: Sure, the existing decorators are not useful here. I should have used a different name.07:03
jtvNaming is hard.  Any specific suggestions?07:04
wgrantwith a_promise_that_i_am_not_evil_twisted_code07:06
jtvwith hand_on_heart:07:11
jtvDo we provide download statistics for user PPAs anywhere?07:57
StevenKIn the DB07:58
wgrantjtv: And the API08:00
jtvOnly there?08:00
wgrantYes.08:01
wgrantI only added an initial API, then got busy and never designed a UI.08:01
jtvwgrant: Oh well, I'll tell them it's your fault then.  Thanks.08:03
jtvwgrant: meanwhile, any further thoughts on my MP?  Anything I need to change?  Do you think it's landable?08:05
wgrantjtv: I don't like the duplication. It's too easy to forget a commit. Apart from that it looks pretty sane, but I need to go over it again.08:09
jtvThe duplication of what?08:09
wgrantcommit/DatabaseTransactionPolicy/commit08:09
* stub reads up on celery08:30
rick_h_j08:56
lifelessstub: how much of the last fdt was slony / fti/trusted / the actual patch ?09:04
lifelessstub: e.g. how much identifiable fat do we have ?09:04
stublifeless: I don't think we have done a rollout with the new Slony yet, so I don't know the real, current answer to that.09:04
stublifeless: I don't think we can tell, because to minimize overhead we collapse everything update.py does into a single big .sql script rather than running EXECUTE SCRIPT several times. We can tell how long it takes to run security.py vs. update.py, but beyond that?09:05
* stub wonders if the timestamps in LaunchpadDatabaseRevision have the information, or if transaction_timestamp vs. statement_timestamp dance that has to be done means they lie.09:09
stuboh... staging has been new slony for ages. duh.09:09
wgrantstub: We did an fdt with the new slony on Monday09:12
wgrantAnd probably Friday too09:13
stublifeless: http://paste.ubuntu.com/747987/ is the relevant section from staging.  51s to apply no patches (but this still resets security and applies trusted.sql)09:13
wgrantYeah, 18th and 21st there were fastdowntimes.09:13
* stub looks for the production logs09:14
wgrantlifeless: :( postgres on carob is hungry.09:14
stubhttp://paste.ubuntu.com/747996/ is relevant stuff from a production outage, 3:12 outage09:22
stubThe fat seems to be in waiting for sync and things to propagate. About 10 seconds everytime we pause and ask if everything is ok and caught up.09:24
stub13s to lock all the tables and run trusted.sql and the single db patch. The db patch thinks it took 2.3 seconds.09:26
stub47s for that update to propagate and be run on the slaves and success to be reported back to the master.09:26
stubWe run security.py in series, but that is only a few seconds as we bypass slony entirely for that.09:27
wgrantrvba: Can you QA bug #867941, please?09:27
_mup_Bug #867941: person:+activereviews times out <affects-ubuntu> <code-review> <qa-bad> <timeout> <Launchpad itself:Fix Committed by rvb> < https://launchpad.net/bugs/867941 >09:27
wgrantYour rollback is on qastaging.09:27
rvbawgrant: oh, great, I was just waiting for it to be deployed to qastaging.09:28
wgrantrvba: buildbot finally passed :)09:28
rvbaFinally! :)09:28
wgrantstub: The previous fastdowntime might be better to analyse.09:28
stubThe biggest fat I can identify is adding new stuff we have created to replication. If we create new tables, we need to call EXECUTE SCRIPT twice.09:29
wgrantstub: What are our slony intervals nowadays?09:29
wgrantWe'll hopefully do a fastdowntime tomorrow.09:29
wgrantI might do a double one.09:29
wgrantThe second being a no-op.09:29
wgrantSo we can see how that goes.09:30
wgrantAnd we can fit both in 5 minutes, so it's fine.09:30
stubdefault settings if you mean the slond config timings09:30
=== almaisan-away is now known as al-maisan
stubso 2 seconds for sync-check-interval. We could lower that and see if it speeds up all the handshaking09:31
wgrantHmm.09:32
wgrantWhat's the other one? 10s?09:32
wgrantIIRC that's the deaful09:32
wgrantdefault09:32
stubwgrant: That is a heartbeat - shouldn't affect handshaking09:33
wgrantYou'd think not.09:33
wgrantBut in my testing locally it did, IIRC.09:33
wgrantIt's been a few months, though.09:33
stubhmm09:34
stubwell, easy enough to tweak it and see what happens. We are not going to overload things changing these settings09:34
stubBut only one at a time of course.09:34
stubwgrant: Any opinion on changing sync-check-interval or sync-interval-timeout first?09:35
mrevellHallo09:36
wgrantstub: I am no longer sufficiently informed to have such an opinion.09:40
lifelesswgrant: yeah, it is09:41
rvbalifeless: Hi Rob, any way for me to access an oops report… like manually or something? The web application seems to be still jammed ;).09:45
rvbaI suppose the old oops cleaning up stuff is taking more time than anticipated.09:46
stubwgrant: I'll knock the sync-check-interval (how often a daemon polls to see if a sync should be sent) to half, 1s. Lets see if it affects that 10s time.09:47
stubwgrant: huh. 'If the node is not an origin.... wasteful for this value to be much less than sync_interval_timeout'. So yeah, twiddle them both.09:53
stubIdeally we would change this just before the rollout...09:54
lifelessrvba: grep for the id in the relevant dir on carob09:56
lifelessrvba: the relevant dir for staging/qasstaging is /srv/oops-amqp/<instance>/<day>09:57
lifelessrvba: for production its /srv/launchpad.net/logs/production/host/<day>09:57
rvbalifeless: Ok, thanks… for the tip.09:57
lifelessrvba: it should be unjammed now09:58
rvbalifeless: I can access the tools not… for my qastaging oops from 10 minutes ago is not there.09:58
rvbas/not/now/09:58
rvbas/for/but/09:59
lifelessrvba: if its not there yet, its probably in the rabbit queue waiting for the consumer to catch up09:59
rvbaRight…09:59
wgrantDo we have a graph of that, or just a nagios check?09:59
lifelessfor that there isn't much to do but wait ><09:59
lifelesswe want a graph, I think liam was doing/has done, but I can't see it on tuolumne10:00
rvbaI think gnuoy is working on a graph for that.10:00
lifelesswwaiting for him to say 'done' before asking where it is ;)10:00
gnuoywgrant, we have graphs of oops queue lengths in staging and qastaging10:01
wgrantOoh10:01
* wgrant looks10:01
wgrantrvba: qastaging's amqp2disk is doing lots of stuff.10:01
gnuoyI've just landed a fix to stop there being missing data10:01
wgrantOr at least spamming a lot of syscalls.10:01
rvba+activereviews for ~ubuntu-branches issues 169 (as opposed to 1400+ previously) but I'd like to see the forced oops to make sure all the repeated statements are gone.10:01
rvba169 queries that is.10:02
wgrantYeah, it's writing out OOPSes.10:02
wgrantSo it should be there eventually.10:02
wgrantgnuoy: Ah, nice.10:02
rvba"QAStaging Rabbit Oops Queue" … cool10:03
wgrant300 unacked messages is a bit odd.10:03
wgrantI guess we'll see what happens once the queue reduces.10:03
gnuoyThere are also graphs for rabbits rss and the mnesia db size10:03
lifelesswgrant: pipelining perhaps10:03
rvbaVery nice, thanks gnuoy!10:03
gnuoyrvba, no problem10:04
wgrantlifeless: Except that it's been that way for 11 hours10:04
wgrantconstant10:04
wgrantMay just be amqp2disk hanging during your terrorisation of postgres, though.10:04
wgrantIt's certainly not being fast at the moment, but it is working.10:06
wgrantI wonder how it will cope with fastdowntime.10:06
wgrantrvba: What's the OOPS ID you're looking for?10:06
rvbawgrant: OOPS-9daabd974426cdd97b512b23d6906f5510:07
rvbaIt's not in /srv/oops-amqp/qastaging/2011-11-24 yet10:07
wgrantThat may be because it is under a different filename.10:08
wgrantHowever, not in the DB yet either.10:08
wgrantI guess it's near the end of the queue.10:08
rvbaThe number of queries plus the fact that it's not (always) timing out tells me it's qa ok but I'd like to be sure there is no repeated stmt left…10:09
wgrantYep10:10
stubWhat are we using rabbit for atm? Just OOPS or is txlongpoll in there as well?10:10
wgrantrvba: https://lp-oops.canonical.com/?oopsid=OOPS-9daabd974426cdd97b512b23d6906f5510:11
wgrantAnd indeed, amqp2disk has gone silent.10:11
wgrantSo it was near the end of the queue.10:11
wgrantAnd we are now caught up.10:11
wgrant123 repetitions10:11
wgrantHowever, only one significantly repeated query.10:12
wgrantSo much better.10:12
rvbaRarg, fetching branches info…10:13
lifelesswgrant: that would fit10:15
rvbaI'm already fetching data for target/sources/prerequisite branches…10:15
lifelesswgrant: x number pipelined, and the consumer wedged10:16
wgrantlifeless: Possibly. We'll see soon.10:16
wgrantWhen the graph updates.10:16
lifelesswgrant: no, when the queue clears10:16
wgrantrvba: Fortunately we have tracebacks :)10:16
lifelesswgrant: if its pipelined it will be constant(ish)10:16
wgrantlifeless: The queue s clear.10:16
rvbawgrant: indeed! tracebacks ftw!10:16
lifelessah cool10:16
wgrant    if can_access and self.stacked_on is not None:10:17
wgrantYou're not preloading stacked_on, I guess :)10:17
lifelessthats spelt wrong10:17
lifelessself.stacked_onID is not None10:17
wgrantBut damn, 169 queries with 123 able to be trivially eliminated... not  bad.10:17
lifelessunless you're actually accessing the stacked_on branch10:17
wgrantlifeless: Unless it accesses the attribute in the next line.10:17
wgrantWhich is probable.10:17
lifelessyeah, so needs eager loading10:17
lifelessrvba: nice work10:18
wgrant            if self.stacked_on not in checked_branches:10:18
wgrant                can_access = self.stacked_on.visibleByUser(10:18
wgrant                    user, checked_branches)10:18
rvbathanks lifeless.10:18
wgrantis the next bit10:18
rvbaYep, you're right wgrant, stacked_on branches.10:18
wgrantrvba: I guess that will only really be a significant problem for ~ubuntu-branches, because everyone else will work with only a few trunks.10:20
rvbawgrant: right, but fixing up ~ubuntu-branches is my goal!10:20
wgrantYep.10:21
wgrantrvba: So, qa-ok, then?10:21
wgrantWe should deploy.10:21
rvbawgrant: yep10:21
wgrantrvba: Can you tag the bug and request a deployment once the QA report is updated?10:22
rvbawgrant: sure10:23
rvbabug tagged10:23
rvbaI've got another branch coming up to reuse the same eager loading method to fix product:+activereviews.  Right now it can be as bad as issuing 1800+ queries.10:24
wgrantYeah.10:24
wgrantFortunately they're very quick queries.10:24
wgrantBut still really bad.10:24
rvbaRight, but still.10:24
* jml waits for diffs10:30
jtvallenap: done with your review.10:33
allenapjtv: Thanks!10:35
=== allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugtasks: 292
wgrantAha, green deployment report.10:38
rvbawgrant: ok10:38
jml9 minutes and counting10:39
wgrantjml: Is that including scan time?10:39
jmlwgrant: Don't think so, LP has known there are pending changes for that whole time.10:40
wgrantjml: Nowadays it shows that also when a scan is pending.10:40
jmlwgrant: interesting.10:40
jmlso, yeah, probably including scan time.10:41
wgrantDoes the branch page also have the pending change warning?10:41
jmlwgrant: it's finished now10:42
jmlwgrant: so no :)10:42
wgrantHeh10:42
wgrantExcellent timing.10:43
wgrantrvba: Thanks.10:45
rvbawgrant: Glad to help out with that, I know you've been doing that a lot lately.10:46
rvbawgrant: funny, the check performed on the stacked_on branch is recursive! The only was to load all the branches in one query is to use a recursive CTE.10:52
wgrantrvba: Ah, how lovely.10:52
wgrantNearly not worth it.10:53
rvbaYeah, that's what I was thinking, I'll just preload on level of stacked_on branches.10:53
rvbas/on/one/10:53
wgrantGood plan.10:55
lifelessrvba: don't need one query10:56
lifelessrvba: need 2 - one for the collection, and then the stacked set for the batch (in the pre iter hook) - and I would definitely use a CTE at this point10:57
lifelesswgrant: did you say didrocks was an edge user?10:58
rvbalifeless: ok, you think it's worth it? Also all these branches will have to be checked by branch._checkBranchVisibleByUser10:58
wgrantlifeless: I fixed that yesterday.10:58
lifelesswgrant: thanks10:58
wgrantlifeless: That's why the edge graphs are back to their old levels.10:58
wgrantbdmurray also migrated his stuff away10:59
wgrantWill attack more tomorrow.10:59
lifelessrvba: yes, they need to have visiblity checked, but the eager loading query should do that10:59
lifelessrvba: as in, your eager loading query should filter non-visible stacked-on branches10:59
lifelessrvba: I think its worth it yes, we have some silly deep stacking chains10:59
rvbalifeless: all right, let's do this all the way then :)11:00
rvbalifeless: not sure I understand "your eager loading query should filter non-visible stacked-on branches", my plan is to preload all the stacked_on branches, then let the python sort the visibility without having to hit the database.11:01
lifelessrvba: as a matter of principle, you shouldn't pull stuff from the DB the user cannot see.11:02
lifelessrvba: as a matter of pragmatism, filtering in the query is faster - becaues the python visibilty rules trigger late evaluation11:03
rvbalifeless: Okay, so this mean I'll have code the visibility rules in sql right?11:03
rvbameans even11:03
rvbas/code/to code/11:03
lifelessrvba: they are already coded, in BranchCollection11:03
wgrantIt probably already is in the main query.11:03
lifelessrvba: you should be able to use a (new) BranchCollection to make the query for stacked on, with the CTE as an additional filter11:04
wgrantYeah.11:04
wgrantThe main query for that page already does the privacy check.11:04
rvbalifeless: Okay.11:04
wgrantIt might just not prepopulate the security adapter cache.11:04
wgrantBah, although it's not recursive.11:05
lifelessright, and if it doesn't, it a) should and b) thats even more reason not to retrieve things that the user cannot see :)11:05
rvbaallenap: can you please have a look at https://code.launchpad.net/~rvb/launchpad/activereviews-bug-893702/+merge/83172 ?11:08
allenaprvba: Certainly.11:08
rvbathx11:08
=== matsubara-afk is now known as matsubara
rvbalifeless: hum… to check the visibility, BranchCollection materializes the list of visible branches (to use branch.id in (...)) … and that's precisely what's making the main query really slow.  I think I'll have to be very careful and maybe I won't be able to use BranchCollection here because the remedy may be worse than the disease If I do …11:26
lifelessrvba: or you need to teach branchcollection a couple of ways to check...11:36
lifelessrvba: e.g. BranchCollection()...EagerLoadById() -> checks per branch rather than all-visible etc11:36
rvbalifeless: yeah, I need to think about it. I'm sure there is proper way, but I'll need to be careful when doing the qa to make sure I'm not issuing one more giant query :)11:37
rvbalifeless: If you don't mind, I'll ping you again when I have something ready so you can have a look.11:41
lifelesssure11:47
rvbaThanks.11:52
=== al-maisan is now known as almaisan-away
jmlhey12:34
andy_jsHow do I get launchpad to use openid.net instead of testopenid.dev?12:35
jmldid you get rid of ReadyService?12:35
wgrantjml: launchpad-buildd no longer uses it, but other things do.12:37
jmlwgrant: what does it use instead?12:37
wgrantandy_js: grep for testopenid.dev in the configs and change it to openid.net, I guess.12:37
wgrantjml: It tries to connect to the port.12:37
jmland keeps retrying until syn/ack?12:38
wgrantI believe so.12:38
jmlthanks.12:38
andy_jswgrant: It's not that simple I'm afraid.  Changing 'enable_test_openid_provider' in schema-lazr.conf to False seems like the right solution, but after I do that the login page says 'OpenID Provider Is Unavailable at This Time'12:42
wgrantandy_js: That just disables the serving of testopenid.12:42
wgrantIt doesn't affect its use.12:42
andy_jsBut once you disable 'enable_test_openid_provider' launchpad no longer tries to redirect to 'testopenid.dev' (which is not available on my non-local instance)12:45
wgrantNo.12:45
wgrantBecause it can't perform discovery on it.12:45
wgrantIt'll complain that it's unavailable instead.12:45
wgrantI don't know if openid.net supports the discovery functionality that we require.12:46
andy_jsSo does launchpad support OpenID or not?12:47
wgrantLaunchpad supports OpenID, but currently only in SSO mode, to a predefined provider.12:48
wgrantI only know of one instance other than launchpad.net, and it uses CommunityID as its OpenID provider.12:48
wgrantlaunchpad.net uses Ubuntu SSO, which is powered by canonical-identity-provider.12:49
wgrantHowever, any provider that supports Yadis or XRDS discovery should work.12:51
andy_jsAll I'm trying to do is set up an instance of launchpad which is remotely accessible.12:53
andy_jsThe only blocker for me is getting login to work.12:53
wgrantYou are aware of the branding licensing issues?12:53
andy_jsI thought it was free for non-commercial use.  Is there more to it than that?12:54
wgranthttps://dev.launchpad.net/LaunchpadLicense12:54
wgrantThe branding must be changed for a production instance.12:54
wgrantCommercially or non-commercially.12:55
andy_jsAh that's no problem.  I was going to rebrand it for my project anyway.12:55
wgrantThe code itself is AGPLv3, so you can use it for whatever you wish. The images and name need to be changed, but that's not an insurmountable obstacle.12:56
wgrantWhy are you not using launchpad.net, out of interest?12:56
andy_js1) I want complete control12:56
andy_js2) I don't think my architecture is supported (solaris-i386, solaris-amd64)12:57
wgrantAh, for PPAs, I see.12:57
wgrantThat's the same reason the only other instance I know of exists -- to build PPAs for Debian.12:58
andy_jsYep.  The possibility to have ppa with tight integration with bug tracking and scm is really enticing12:58
wgrantGetting a productionish build farm up and running is a bit of effort, but doable.12:59
andy_jsIs it possible to run ppa's without setting up a build-farm?13:00
wgrantNot if you want binaries. There has to be somewhere to build them :)13:00
andy_jsSo I can't just upload using dput?13:00
andy_jsBecause I don't think our architecture is quite mature enough for doing the build-farm route.13:01
wgrantThere is an old option to the upload processor that should let you upload source and binaries, but it hasn't been used for a few years.13:02
wgrantWhy do you say that?13:02
andy_jsI haven't been able to create a working ISO in a few monthes so I have nothing current to install on the build servers (that, and I don't actually have any build-server :P)13:03
andy_jsAnway, if I can upload stuff using dput then it should be able to do what I want.13:04
andy_jsIf it's an old crusty open that doesn't work quite right I guess I'll have to fix it13:04
wgrantLP was designed from the start to have sources uploaded to it, and build binaries itself. If you just want to publish an archive with pre-built binaries, why not use something a little simpler and less terrifying, like reprepro?13:05
andy_jsreprepro is what I'm using now and it does the job fairly well13:06
andy_jsI want the other things that launchpad offers (bug tracking, mailing lists, scm management, the ability to let people create their projects)13:07
andy_jsAnd in the future I do intend to make use of the build-farm feature13:07
wgrantI suspect that you underestimate the resources and effort required to maintain a complete Launchpad instance.13:08
andy_jsIt seemed like it would be easier than writing my own software which is what I started to do.13:10
andy_jsIs it really that of a pain in the arse to maintain?13:10
wgrantProbably, but for everything except non-Ubuntu PPAs you can just use launchpad.net...13:10
andy_jsWell I'm actually getting my sources from Debian so I guess that wouldn't be acceptable.13:12
wgrantHmm?13:12
andy_jsHmm what? :)13:13
=== almaisan-away is now known as al-maisan
wgrantI'm not sure of the relevance of you getting your sources from Debian.13:13
andy_jsI thought you just said PPA's were only for Ubuntu?13:15
andy_js(on launchpad.net, anyway)13:15
wgrantlaunchpad.net will only build for releases and architectures that Ubuntu supports.13:15
wgrantMy suggestion would be that you track your bugs and projects and everything except packages on launchpad.net.13:16
wgrantSince you're not going to be using build farm functionality yet anyway, PPAs probably provide little to no benefit to you.13:16
wgrantIf you do really want to use them, you could use a local instance for PPAs, I suppose. But maintaining your own codehosting and mailinglist setups is no particularly small effort, and I don't see much benefit over using launchpad.net.13:17
andy_jsI want to be able to give people the ability to create their own PPA and upload package via dput.13:21
wgrantAh, including uploading binaries?13:22
andy_jsYes13:22
wgrantThat would require some hackery of the upload policies of a local instance. The secure upload paths do not permit binaries. But that is easily changed.13:23
andy_jsHackery is fine as long as what I want to do can be acheived without having to rewrite major chunks.13:25
andy_jswgrant: Thank you very much for your help so far btw :)13:26
wgrantIt was done regularly until a couple of years ago. Security updates were built outside LP, and source+binaries uploaded at once.13:26
wgrantWe still have tests for that, so it shouldn't have bitrotten.13:26
wgrantAnyway, I should sleep.13:28
=== matsubara is now known as matsubara-lunch
abentleyallenap: could you please review https://code.launchpad.net/~abentley/launchpad/history-model/+merge/83310?15:30
allenapabentley: I'm not going to be able to finish that before I have to go out. I will be working this evening, from 2000 UTC to 2200 UTC. Would it be okay if I did it then?15:34
abentleyallenap: sure, no big rush.15:34
allenapabentley: Cool.15:37
=== allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugtasks: 292
=== al-maisan is now known as almaisan-away
=== salgado is now known as salgado-lunch
=== matsubara-lunch is now known as matsubara
=== salgado-lunch is now known as salgado
flacosteabentley: shouldn't the Order by buttons reflects the list of fields displayed?17:46
abentleyflacoste: they should, but we haven't done that yet.17:47
flacosteah ok17:47
flacostegmb: you actually reported a known bug that deryck is fixing, it should have been mentioned as a known issue in the blog post17:51
flacostemaybe we should do a duplicate guess contest17:51
flacostei bet we ar going to get 5 duplicates of that bug17:52
gmbflacoste: Yeah, I noticed it got duped. My bug title was obviously far enough away from the original bug for the match to fail.17:52
lifelessrighto, vanadlism undone20:13
lifelessall hail deleting of bugtasks too20:13
=== matsubara is now known as matsubara-afk
allenapabentley: Do you have a few minutes to talk about PackagingJob?22:03
lifelessallenap: just the lad22:04
andy_jsHow do I get login to work in a production environment?22:04
lifelessallenap: I want to talk to you aboot script activity ;)22:04
abentleyallenap: sure.22:04
lifelessallenap: I'm happy to queue behind you and abentley22:04
allenaplifeless: Yeah, I know :) I really wanted to reply on the wiki but I have hardly had a moment :-/22:04
allenapCool, abentley first, he's closer to EOD.22:05
allenapabentley: In trying to open translations for Precise we first have to erase all existing data that has been created for it.22:06
allenapabentley: We have a script to do that: http://paste.ubuntu.com/748706/22:06
allenapHowever, there are also references to some of that data from PackagingJob records. Nearly 80k of them iirc.22:07
allenapabentley: Jeroen reckoned that we could probably delete them, but that we'd need to recreate them.22:07
allenapabentley: Your name and Danilo's is all over PackagingJob, hence why I'm asking you.22:07
abentleyallenap: My work on packaging has been SourcePackageRecipeBuilds.  Those did use a PackagingJob, IIRC.22:09
abentleyOh, and I guess the translation stuff counts, too.22:10
abentleyallenap: So, I don't think you need to recreate those records.22:11
allenap\o/22:11
allenapWe can just delete and be done?22:11
abentleyallenap: Records that are not active are strictly for historical record.22:12
abentleyallenap: If you're deleting the Precise translations, then you wouldn't need info about the jobs related to those Precise translations.22:12
abentleyallenap: So yes, I think you can just delete and be done.22:13
allenapabentley: Great. So, as long as the job records are in terminal states we can trash them. If they're not in terminal states we ought to get them to terminal states before trashing them.22:13
abentleyallenap: I don't think you even need to get them to a terminal state, since you'll be deleting whatever they could act on.22:14
abentleyallenap: but of course, you want to avoid doing this for non-Precise jobs.22:15
allenapabentley: Okay, tip top. Thank you, I think that's all I need to know.22:15
abentleyallenap: cool.22:15
allenapabentley: Yes, accidentally overstepping has been my main fear when crafting this script.22:16
lifelessallenap: voice?22:17
allenaplifeless: Okay, let me re-read your last comment first though :)22:17
lifelesssure22:17
allenaplifeless: Okay, ready.22:20
huwshimilifeless: Just before I do it, I want to bump bug 894449 to critical. It creates a completely broken UI and would oops if there were such things for the UI. Is there any reason why it shouldn't be critical (in case I missed something)?22:33
_mup_Bug #894449: It's possible to create an entirely useless bug listing <bug-columns> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/894449 >22:33
wgrantThat's only High.22:33
lifelesshuwshimi: It doesn't prevent users doing what they want, and it only affects users explictly opted into the beta program.22:34
wgrantUsers can make the UI in a beta feature show nothing if they explicitly tell it to.22:34
lifelesshuwshimi: So I don't think it meets our guidelines for critical.22:35
lifelesshuwshimi: *however* there is a bug that the new UI breaks some searches, so I think we have to turn it off anyhow.22:35
lifelesshuwshimi: because *that* bug is critical.22:35
huwshimilifeless: OK, no problems22:35
=== wallyworld_ changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld | Critical bugtasks: 292

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!