/srv/irclogs.ubuntu.com/2009/10/14/#launchpad-dev.txt

* mwhudson shoves branching-ubuntu into ec2 land (at last!) and goes to lunch01:36
james_w\o/01:37
james_wthanks mwhudson01:37
mwhudsonjames_w: you can see the diff at https://code.edge.launchpad.net/~mwhudson/launchpad/branching-ubuntu/+merge/13269 if you want01:38
mwhudsonjames_w: shouldn't you be asleep?01:38
james_wI should01:38
james_wso I will look tomorrow01:38
mwhudsoncool01:40
mwhudsonthumper: didn't i worry at you about which zcml was executing in scripts vs the webapp?02:22
mwhudsonthumper: looking at the errorreportingutility lookup failures02:23
thumpermwhudson: yes you did02:23
mwhudsonoh well, use globalErrorReportingUtility and don't worry over much i guess :/02:23
thumpermwhudson: my branch adds the registration of the utility for scripts02:24
thumpermwhudson: after talking with flacoste_afk02:24
mwhudsonthumper: ah cool, that's a better solution02:24
thumpermwhudson: the code should be moved from c.l.webapp to lp.services.error02:24
thumperbut hey02:24
mwhudsonthumper: world hunger also needs fixing02:25
thumpermwhudson: :)02:26
thumpermy desire to file bugs about ubuntu drops significantly as soon as things work again, even if intermittantly02:26
mwhudsonhmm02:37
mwhudsonthe commit message area in https://code.edge.launchpad.net/~mwhudson/launchpad/branching-ubuntu/+merge/13269 looks seriously ugly02:37
thumpermwhudson: yes it does02:39
* thumper has a cunning plan02:39
thumpermwhudson: https://pastebin.canonical.com/23283/03:54
thumperhttps://code.edge.launchpad.net/~andrea-bs/launchpadlib/fix-set_data_store/+merge/61903:54
thumperhas a null date review requested03:54
thumpercausing the oops03:54
thumpernow how the hell did that happen?03:54
mwhudsonthumper: that's really olkd03:54
mwhudsonold03:54
thumper619? yeah03:55
thumperI wonder if it slipped through early before we knew what we were doing03:55
thumper...03:55
thumperrejected the proposal and now we have a working active reviews again03:58
thumpershould look into that at some stage03:58
thumpermaybe add a DB constraint03:59
thumpermwhudson: http://pastebin.ubuntu.com/292816/ ideas?04:05
mwhudsonthumper: apt-get install python-boto ??04:05
thumperwhat is boto?04:05
mwhudsonlike launchpadlib is to launchpad, boto is to amazon web services04:06
mwhudsonthumper: have you run ec2 on this laptop yet?04:06
thumperjust about to04:06
thumpermwhudson: does land have --dry-run?04:07
mwhudsonthumper: yes04:07
thumperheh, it tells me the mp is not approved :)04:07
thumperif I approve it does that make it r=me or you?04:07
mwhudsonthumper: it looks at the votes04:08
mwhudsonthumper: finish autoapprove already :-)04:08
thumpermwhudson: help me come up with the db schema and we can land it this cycle04:08
* thumper needs to copy across aws credentials04:20
mwhudsonsigh05:50
mwhudsonso it turns out that writing a simulator for a confusingly concurrent situation is, in and of itself, confusingly concurrent05:51
stubCan mocker help?05:52
mwhudsonstub: several processes involved so maybe not05:53
mwhudsonhmmm straage, two successive calls to acquireBranchToPull are returning the same branch06:10
mwhudsonwell, that's a bug to fix i guess06:32
mwhudsonargh06:54
mwhudsonthere's no way to tell the difference between a branch which failed last time that the puller is currently working on and one that failed last time :(06:55
mwhudsoni can think of various stupid ways that you could encode this, but maybe we should just use jobs06:56
mwhudsonright *now* however, i'm going to the pub06:59
adeuringgood morning08:21
mrevellmorning09:15
wgrantbigjools: I think I am attempting the impossible (establishing the matching publication for a deb's ddeb).09:27
bigjoolswgrant: it should be possible09:28
bigjoolsyou need to join via the BPR09:28
wgrantbigjools: I know, but there is no way to identify which of the potentially several BPPHs matches (think overrides).09:29
bigjoolswgrant: the ddeb should not be overridden differently09:29
noodles775wgrant: let me know if you've got any improvements for bug 45012409:29
mupBug #450124: Improvements to IBuilder.findBuildCandidate() <soyuz-build> <Soyuz:In Progress by michael.nelson> <https://launchpad.net/bugs/450124>09:29
ubot3`Malone bug 450124 in soyuz "Improvements to IBuilder.findBuildCandidate()" [High,In progress] https://launchpad.net/bugs/45012409:29
mupBug #450124: Improvements to IBuilder.findBuildCandidate() <soyuz-build> <Soyuz:In Progress by michael.nelson> <https://launchpad.net/bugs/450124>09:29
mupBug #450124: Improvements to IBuilder.findBuildCandidate() <soyuz-build> <Soyuz:In Progress by michael.nelson> <https://launchpad.net/bugs/450124>09:29
bigjoolsGAH09:29
wgrantbigjools: Matching based on overrides sounds horribly fragile.09:30
bigjoolswho keeps putting bots in here09:30
wgrantAnd it's not been donebefore.09:30
* wgrant hunts out jussi01.09:30
bigjoolsit's not an override, it's a matching publication in the same series, component and pocket09:30
wgrantBut there's nothing unique about that.09:31
bigjoolsapart from the archive09:31
wgrantI mean, I can easily have duplicate (bpr, archive, distroseries, pocket, component, section)09:32
jussi01wgrant: just get one of your ops to quiet it for now is probably the best course of action09:33
jussi01wgrant: I dont have access to ubot3, thats nalioth, but if you just quiet it, it sits here and isnt a problem09:33
wgrantjussi01: ubot3 says you own it :(09:33
jussi01wgrant: unfortunately thats because it syncs the factoids from ubottu09:34
wgrantjussi01: Ahh.09:34
bigjoolswgrant: well yes any model code can screw up the DB, but if it's encapsulated properly and the rules are self-contained then I don't see a problem09:34
wgrantbigjools: I have a deb and ddeb, in universe. I promote them to main, then demote them again.09:35
wgrantNow there are three publications for each, but two are duplicates.09:35
wgrantI now remove the deb, and it tries to drag along the ddeb.09:35
wgrantBut how does it know whihc publication is the right one?09:35
bigjoolswgrant: the one that's PUBLISHED09:36
wgrantbigjools: But I did all three override changes semi-accidentally in a single publication cycle.09:36
bigjoolsor the one with the most recent ID09:36
wgrant(it has happened)09:36
bigjoolsthen you need to supersede old publications if you find inconsistencies09:37
wgrantInconsistencies?09:37
bigjoolsgimme 10 I need to have a call09:37
wgrantSure.09:37
jmlhello09:49
maxbHi, has anyone else seen failures on    lib/canonical/launchpad/pagetests/webservice/xx-wadl.txt ?09:50
jmlhmm09:52
wgrantThat name is familiar. What sort of failure?09:52
maxbwadl_schema.validate(wadl) returning False09:52
noodles775maxb: passes here locally (r9691 on karmic)09:52
maxbnoodles775: How "interesting" :-)09:53
BjornTmaxb: the wadl file is generated automatically, so my guess is that something went wrong generating it09:56
jmlmaxb, I get that failure when running the test on the stable branch.09:56
jmlupdating download-cache & running 'make build'09:57
jmlstill get the error09:59
bigjoolswgrant: ok back10:00
bigjoolsso, you just need to take the publication with the most recent ID10:00
bigjoolsthe publisher should supersede the others10:00
wgrantbigjools: Even if the latest one is no longer active? That sounds convenient.10:01
wgrantAlthough still a bit ew.10:01
bigjoolsyeah10:01
bigjoolswhich makes me think, you might want to check that the publisher will work with debug archives :)10:02
wgrantActually, this isn't safe.10:02
wgrantdeb_bpph.supersede() should also supersede the corresponding ddeb.10:02
bigjoolsyes10:03
wgrantThis is to handle the case that a package ceases to produce a ddeb.10:03
wgrantBut what if it doesn't cease to produce it?10:03
wgrantHmm.10:03
wgrantI guess it should be safe as long as we only look within the build.10:03
wgrantYeah, I guess that will work.10:04
bigjoolsI don't quite follow this case10:04
wgrantI wasn't thinking properly. There is no problem.10:05
bigjoolsok :)10:05
wgrantIt is guessing, but very safely.10:05
bigjoolsif we assume the publisher will deal with old publications, taking the most recent one is always safe10:05
wgrantRight. I had forgotten about other restrictions on the search.10:06
wgrantMaybe it's not safe. How I hate guessing.10:07
wgrantI have a deb and ddeb, both with Published publications. I promote them, giving them a new Pending publication each.10:08
wgrantThe publisher runs.10:08
wgrantIt sees that the deb is superseded, and calls deb_bpph.supersede()10:08
wgrantThe Pending ddeb publication screams and dies.10:08
wgrantThis is not correct behaviour.10:08
wgrants/deb_bpph/deb_old_bpph/10:09
jmlbigjools, are there docs anywhere describing how the buildmaster works?10:10
wgrantHa ha ha.10:10
wgrantjml, this is Soyuz.10:10
bigjoolsROFL10:10
jmlfair point.10:11
noodles775buildd-dispatching.txt ?10:11
jmlwgrant, do _you_ have docs describing how the buildmaster works?10:11
bigjoolsthge buildmaster was oringally hacked up by the early LP devs (daniel etc) and more recently re-written by Celso in a hurry10:11
bigjoolsyes, the doctest is the closest we have10:11
wgrantjml: Although I know it internally fairly well now, I only have docs on how to run it.10:11
jmlyeah, I remember him being in a hurry. :)10:11
bigjoolswhen wasn't he? :)10:12
bigjoolsjml we have a plan on how to generalise the buildmaster side but it was written before the re-write of the scanner10:13
wgrantbigjools: Any ideas on the above example?10:13
* bigjools missed that, just a sec10:13
jmlbigjools, is the plan written down?10:13
wgranthttps://dev.launchpad.net/Soyuz/Specs/BuilddGeneralisation10:13
jmlI might have a poke around the code once I do today's CHR duties & finish off this damn imports branch and review al-maisan's branch.10:13
jmlI guess that means tomorrow.10:13
jmlwgrant, thanks.10:14
bigjoolswgrant: why would it scream and die?10:14
maxb    <wadl:method name="PUT" id="HostedFile-put"/>10:14
maxb    <wadl:method name="DELETE" id="HostedFile-put"/>10:14
maxb^ Well, oops :-)10:14
wgrantbigjools: Because it has the latest ID, so gets superseded.10:15
bigjoolsjml: the code is pretty bad, most of it was written before we had real coding standards.  Feel free to hassle me to explain stuff.10:15
* al-maisan takes a peek at lp:~jml/launchpad/upload-permission-joy10:15
bigjoolswgrant: ok I see10:16
bigjoolswgrant: if the last publication is pending then it should be clever enough to go to the previous one10:18
wgrantbigjools: This is getting more and more into inaccurate guess territory, and sounds like it will go horribly wrong if there are multiple pending publications at once.10:19
jmlbigjools, I remember someone asking me for a generic sftp server based on the one we have in codehosting...10:19
maxbI'd like one of those too :-)10:19
jmlbigjools, I guess that's for a different thing10:19
bigjoolsjml: yeah we could do with replacing poppy10:20
jmlmaxb, remind me to actually put license text in txsshserver one of these days.10:20
bigjoolswgrant: there is already code like this in the publisher10:21
maxbjml: Your verification of the xx-wadl.txt failure was on jaunty/karmic? py2.4?10:21
bigjoolswgrant: I suggest we try and map out all of the operations that can happen like this, and work through them carefully10:21
jmlmaxb, karmic.10:22
jmlmaxb, ./bin/test -1cvvt xx-wadl.txt in stable branch10:22
* maxb wonders if this is a mysterious jaunty vs. karmic oddity10:23
jmlmaxb, I'm deleting the unversioned files in lib/canonical/launchpad/apidoc and running the test again.10:23
maxbjml: the wadl tested in the test is generated on the fly10:23
jmlmaxb, ... as my experiment just demonstrated :)10:24
maxbI'll have to dig out a jaunty system and retest there10:25
* bigjools laughs a lot at https://answers.launchpad.net/soyuz/+question/8577310:27
bigjoolsjml --^10:29
maxbHow does the wadl generation actually work? There is stuff in the comments in the wadl which I don't find by grepping anywhere in the source?!10:33
jmlbigjools, heh heh.10:36
jmlbigjools, what's the blueprint URL?10:36
wgrantmaxb: Possibly stuff in lazr.restful(client)?10:36
bigjoolsjml: I haven't done one yet, I will sort it out soon10:36
jmlbigjools, no worries.10:38
bigjoolsjml: I cleaned up the buildd manager one yesterday as that's more important to me right now :)10:38
maxbHmm... looks like the bugged wadl is copied verbatim from wadl-root.py in lazr.restful10:39
jmlbigjools, ahh sure.10:39
jmlbigjools, I can just register the blueprint and link it to the already-written spec10:39
bigjoolsjml: I can do it, it's on my list of things to do anyway10:40
jmlbigjools, ok.10:41
* jml would really love to scrub the launchpad-code blueprints at some point10:41
jmuxHi - is there a good way to debug traversal errors? I'm getting a backtrace from base-layout-macros.pt for "json" and I didn't really find a way to find the original error.10:41
jmlhmm10:43
jmljmux, nothing leaps to mind.10:43
jmlhttps://dev.launchpad.net/Debugging might help.10:43
maxbeww. lazr.restful has no tags :-(10:44
james_wdoes logger.exception in LP log sys.exc_info() or similar?10:44
jmljames_w, it really depends.10:45
james_w258+ except:10:46
james_w259+ ok = False10:46
james_w260+ self.logger.exception(10:46
james_w261+ "Unexpected error checking %s!", db_branch)10:46
jmljames_w, that behaviour comes standard in Python :)10:46
james_wI'd like to know that this logs the exception, otherwise we are going to be scratching our heads when something goes wrong10:46
jmljames_w, http://www.python.org/doc/2.4.4/lib/node341.html10:46
james_wah, so it does10:47
wgranteg. try to catch an exception and call logging.exception("blah")10:47
jmljames_w, I'll need more context10:47
james_wthanks, will be useful in future10:47
jmljames_w, because it's the log _handlers_ that really make the difference10:47
james_wI'm looking at https://code.edge.launchpad.net/~mwhudson/launchpad/branching-ubuntu/+merge/1326910:47
james_wdef checkNewBranches(self):10:47
james_wso in this case logger comes from LaunchpadScript10:48
jmljames_w, which in turn uses canonical.launchpad.scripts.logger.logger10:49
jmljames_w, which uses a default Python StreamHandler10:49
jmlwhich is fine.10:50
james_wcool, thanks for checking10:51
jmlnp10:52
jmlthanks for asking :)10:52
=== dpm_ is now known as dpm
wgrantbigjools: http://williamgrant.id.au/f/1/2009/ddeb-supersedure-cases.html describes most supersedure cases, I think.11:12
bigjoolswgrant: ok, thanks, I'll look through it in a bit11:15
jmlbigjools, the buildd-generalizing spec is for running arbitrary jobs, right12:37
bigjoolsjml: yes12:37
jmlbigjools, is there another spec for generalizing across machine type? (physical machine, local vm, ec2 instance etc)12:38
bigjoolsthere's a bug12:38
bigjoolswell12:38
bigjoolsthere's a bug for building any type of arch/virtual on the existing builders12:39
bigjoolsthere's no spec for going as far as what you say there12:39
jmlok, thanks.12:39
bigjoolswhich I think is slightly crazy BTW :)12:40
jmlperhaps it is.12:40
jmlwhat I'm actually worried about is build capacity12:41
bigjoolsyes, me too12:41
jmls/worried/thinking/12:41
bigjoolswe'll need a lot more builders12:41
jmlyeah.12:41
bigjoolswe'll also need some more optimisation done to the buildd-manager12:41
bigjoolsbecause finished builds are uploaded synchronously12:41
jmloh right, I remember talking to cprov about that.12:42
bigjoolsit's a very hard problem to solve12:42
elmoerr, why?12:42
bigjoolserror conditions mainly12:42
bigjoolsI forgot the exact details12:43
elmommk12:43
elmoI may be overlooking something, but Debian has done uploads asynchronously for over a decade12:43
jmlbigjools, ISTR it as more being a case of getting round to it.12:43
bigjoolswell don't let me stop you fixing the buildd-manager then ;)12:44
elmoit just means the extra section of the state machine where a package can be either 'uploaded' or 'installed' (i.e. the upload completed successfully) and you just have to make sure nothing stays in 'uploaded' for overly long12:44
elmoalso, why are we worried about capacity?12:44
bigjoolsah I remember more about this now12:44
jmldammit. etsu lunch again.12:44
bigjoolsthe upload processor is single-threaded12:44
wgrantBut it's run in a subprocess.12:44
bigjoolswith a lock12:45
wgrantOh. Right.12:45
bigjoolsand no, it's not a subprocess when uploading binaries12:45
wgrantThen why does the builddmaster have a config option for the command to run to upload the binaries?12:45
bigjoolsNFI12:45
wgrantIt spawns process-upload.py.12:46
wgrantThis is why it takes /forever/.12:46
bigjoolshmmm last time I looked it didn't do that12:46
bigjoolsbut anyway, it's still has a bunch of issues that prevent it running more than one copy at once12:47
jmlI am full of unhappiness and empty of food.12:47
jmlbrb.12:47
bigjoolsand there are other issues with the buildd-manager just dumping the binary in an upload queue - for example how to deal with upload failures12:47
wgrantbigjools: lp.buildmaster.buildergroup.BuilderGroup.buildStatus_OK certainly seems to spawn a subprocess.12:47
jmlbigjools, how do you deal with them now?12:54
bigjoolsit marks the build as failed12:54
jmlbigjools, I see.12:56
bigjoolswhich the upload processor can't do, since it doesn't know about the build12:56
wgrantIt does, actually.12:57
jmlbigjools, you could write the builder asynchronously12:57
wgrantThe upload processor sets the build to Successfully Built, and attaches the binaries.12:57
wgrantIf it completes successfully, the buildmaster attaches the build log, sets the finish time, unassigns the builder, deletes the buildqueue.12:57
wgrantIf it fails, it attaches the upload log and marks the build as Failed to Upload.12:58
wgrantBut the upload processor does (and has to) know about the build.12:58
=== matsubara-afk is now known as matsubara
bigjoolswgrant: of course, you're right.  I think I need more coffee.13:11
=== salgado-afk is now known as salgado
=== mrevell is now known as mrevell-lunch
sinzuiEdwinGrubbs: barry: bac: standup in two minutes14:33
EdwinGrubbssinzui: standup?14:39
barryreviewers, interested developers, and lurkers: ameu reviewers meeting -> #launchpad-meeting in 5m14:55
barryameu reviewers meeting -> #launchpad-meeting15:00
barryabentley: -> #launchpad-meeting.  i'm about to talk about pipelines :)15:04
=== mrevell-lunch is now known as mrevell
Fox_1_hi all15:48
Fox_1_people do you know the place where I can get documentation about putting Launchpad on Debian15:49
Fox_1_?15:49
sinzuiFox_1_: you are in unexplored territory. I am not aware is has been done, let alone documented16:00
sinzuibac: ready for our call?16:01
bacyes16:01
=== flacoste_afk is now known as flacoste
jmlleonardr, were you making a trusted client for the API?16:46
leonardrjml: yes, it is on my list of things to do, but pretty far down the list16:46
leonardri can point you to the bug if you want to take over16:46
jmlleonardr, what if I don't want to take over but just want to leer speculatively, one day dreaming of having enough time to want to take over?16:47
leonardri think pointing you to the bug would still be appropriate16:48
leonardrit might not be as difficult as you think, i did write the client, i just didn't write the tests for it16:48
jmlheh.16:48
jmlleonardr, in that case, please point me to the bug report.16:49
jmlI'm actually planning on working with launchpadlib on Canonical time during the Bazaar sprint in November.16:49
jmlso I might do trusted client work on the plane or something. maybe.16:49
leonardrjml: https://bugs.launchpad.net/ubuntu/+source/ubuntu-dev-tools/+bug/38729716:52
mupBug #387297: manage-credentials should not ask for Launchpad password directly <ubuntu-dev-tools (Ubuntu):Triaged> <https://launchpad.net/bugs/387297>16:52
ubot3`Malone bug 387297 in ubuntu-dev-tools "manage-credentials should not ask for Launchpad password directly" [Medium,Triaged]16:52
leonardrhttps://code.launchpad.net/~leonardr/launchpadlib/trusted-client16:52
mupBug #387297: manage-credentials should not ask for Launchpad password directly <ubuntu-dev-tools (Ubuntu):Triaged> <https://launchpad.net/bugs/387297>16:52
jmlI thought we asked ubot3 to leave. :(16:53
jmlleonardr, thanks.16:53
intellectronicabigjools: halp. i don't know anything about upload permissions. can you give me some pointers? do you know how the driver permission for a distroseries is calculated?17:06
bigjoolsintellectronica: I don't know that, I only know about upload permissions17:06
jmlintellectronica, for a distroseries?17:08
jmlhm17:08
jmlwait. I'm focusing.17:08
intellectronicajml: nm, i think i just found out17:08
jmlintellectronica, glad to hear it.17:08
=== matsubara is now known as matsubara-lunch
=== beuno is now known as beuno-lunch
=== salgado is now known as salgado-lunch
=== beuno-lunch is now known as beuno
mrevellnight all18:04
=== matsubara-lunch is now known as matsubara
=== salgado-lunch is now known as salgado
barrydanilos: ping19:21
Ursinhabarry: danilos is ill today, and it's pretty late for him, I don't think he'll answer in a reasonable time :)19:27
barryUrsinha: okay, thanks.  danilos hope you feel better soon!19:28
=== ubot3` is now known as ubot3
=== EdwinGrubbs is now known as Edwin-lunch
* jgay attempts to install launchpad. 21:05
jgayDoes anybody know if I will likely get a successful build and install on intrepid?21:10
mwhudsonjgay: unlikely21:11
mwhudsonjgay: nothing fundamental, but some of the packages needed haven't been built for intrepid i think21:11
jgaymwhudson: ok, thanks. I'll see how far I get before things start to breakdown.21:12
rockstarmwhudson, abentley, who's hosting today?21:16
mwhudsonrockstar: good question!21:16
rockstarmwhudson, would you like me to?21:16
abentleyrockstar: Go for it.21:16
mwhudsonrockstar: sure21:16
rockstarThe windmill tests don't run concurrently as threads, do they?21:30
barryrockstar, mwhudson thumper's off line today.  do you still want to meet today?21:41
rockstarbarry, yeah, abentley said today was pretty exciting.21:41
barryrockstar: okay cool.  i'll update the minutes and we can meet in 19m21:41
rockstarbarry, ack21:42
mwhudsonbarry: ok21:42
=== Edwin-lunch is now known as EdwinGrubbs
barryrockstar, mwhudson, wgrant  -> #launchpad-meeting22:01
=== salgado is now known as salgado-afk
* rockstar goes on a short walk22:12
wgrantintellectronica: jml refactored the upload permission stuff recently, so it should be pretty easy to remove the duplicated logic from the bug nomination permissions.22:50
=== ursula is now known as Ursinha
=== matsubara is now known as matsubara-afk

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!