/srv/irclogs.ubuntu.com/2015/06/29/#launchpad-dev.txt

wgrantblr: Landed the getAdministratedTeams timeout fix, so hopefully qastaging will be a bit happier soon.01:14
blrwgrant: ah great01:42
maozhouI build package of glibc on my launchpad server,then I have published it ,but the state of glibc is still Uploading ,why?03:00
wgrantmaozhou: What do you mean by "published it"?03:01
wgrantIt can't be published if it hasn't been uploaded yet.03:02
maozhouI run "echo "123123" |sudo -S scripts/process-upload.py -vvv --builds -C buildd /var/tmp/builddmaster"03:04
maozhouLP_DISABLE_SSL_CERTIFICATE_VALIDATION=1 /home/kylin/ubuntu-archive-tools/queue -l dev -s utopic accept03:04
maozhouscripts/process-accepted.py -vv ubuntu03:04
maozhouscripts/publish-distro.py -vv03:04
maozhouscripts/publish-distro.py -C03:04
wgrantmaozhou: What's the 'echo "123123"'?03:05
wgrantprocess-upload.py does not read from stdin.03:05
maozhouthe password of the user who run the script03:06
wgrantThat's certainly unusual.03:07
wgrantYou'll need to check the logs from process-upload.py to see why your build was not uploaded.03:07
maozhouon another launchpad server ,  I do it by same way ,the state of the pachage is successful03:08
maozhouwhere is the logs from process-upload.py?03:09
wgrantstderr03:09
maozhouwgrant: the log is http://imgur.com/mVLK4PE    http://imgur.com/25WeYTn    http://imgur.com/B8RcZg2   , but I can't find any message about "glibc"03:35
wgrantmaozhou: You seem to have a lot of warnings and errors there. I'd be working those out before trying to understand why another upload hasn't appeared.03:39
wgrantBut considering searching /var/tmp/builddmaster for a directory name ending in PACKAGEBUILD-$BUILDID, where $BUILDID is the ID of the build that is stuck Uploading.03:40
maozhouAnd I find a pachage which bigger than 256M, can't  be uploaded succeed03:42
wgrantWhat goes wrong when you try?03:42
maozhouIf a package builded succeed on my launchpad, if the binary package of it  is bigger than 256M, when run process-upload.py , It's error.03:45
wgrantError?03:45
wgrantWe regularly deal with packages larger than a gigabyte without a problem.03:47
wgrantIt's conceivable that there may be a problem with doing it on a 32-bit system, but I've not seen anything like that.03:48
maozhouyes, the OS is 12.04 32bit03:50
wgrantAh, that's inadvisable for a production deployment.03:51
wgrantI'd expect it to work, but it's not thoroughly tested at scale.03:51
maozhouthe  launchpad server  should be deployed on 64bit OS, right?03:54
wgrantmaozhou: All of our production systems are running on amd64, yes.03:55
wgrantmaozhou: The test suite works fine on i386, but it doesn't test large datasets.03:55
maozhouok, I konw03:58
maozhouI know03:59
cjwatsonwgrant: My tentative plan for dose-builddebcheck was to see if it could be run on the builder side07:23
wgrantcjwatson: What benefit does that give us?07:30
wgrantcjwatson: AFAICS it just makes it less likely that we can autoretry depwaits.07:31
cjwatsonActual accurate build-dep analysis rather than guesswork07:31
cjwatsons/build-dep/dep-wait/07:31
cjwatsonThe case at hand in this bug could dep-wait rather than fail.07:31
wgrantIn the current environment we need to be quite conservative with depwaits.07:31
wgrantThough it's already imperfect because of virtual packages, I guess.07:32
cjwatsonOne thing I was thinking of was that we could try sbuild, and if it fails to install build-deps we could then run dose-builddebcheck to find out why.07:32
wgrantThat is, we should avoid creating depwaits that can't be automatically retried when the dep becomes available.07:32
wgrantReturning a deeper dep makes that more likely, as one of the deps in between could change such that the dep still doesn't exist, but the package's build-deps are now satisfiable.07:33
cjwatsonIn which case you'll get a new answer from d-b07:33
wgrantOnly if we run it again.07:33
wgrantOn what condition do we trigger it to rerun?07:34
wgrantThere are a few depwait builds in PPAs and such.07:34
cjwatsonOn a new build ...07:34
wgrantRight, that means we lose depwait autoretry.07:34
wgrantThen what value does depwait provide?07:34
cjwatsonUm?  No we don't07:34
wgrantfoo depends on bar07:35
wgrantbar depends on baz, which doesn't exist.07:35
wgrantbaz isn't meant to exist; it's a bug in bar.07:35
wgrantNew bar is uploaded, not depending on baz any more.07:35
cjwatsonI'm proposing having launchpad-buildd run it if sbuild reports a missing dep07:35
wgrantfoo's build is depwait on baz, and will never be autoretried.07:35
cjwatsonAs a replacement for the regexes07:35
cjwatsonAh, right, yes, that's a problem.07:36
wgrantCurrently in that case we fail.07:36
wgrantAnd everyone knows a failure is sinister.07:36
cjwatsonThough, the way that d-b works means that we would have control over how deep a dep-wait we choose to accept.07:36
wgrant(a depwait may be sinister in the virtual case, but non-existent virtual packages are conveniently rare)07:36
lifelesswgrant: is it ever dexter?07:39
wgrantlifeless: Most of the time :)07:40
cjwatsonHm, but d-b doesn't tell us the full chain, I clearly misremembered that07:45
cjwatsonSo not as workable as I thought :(07:45
wgrantd-b is useful, I think, if you can run it beforehand and then run it regularly until it succeeds.07:45
cjwatsonRight, as you say that's difficult with lots of archives and bootstrap etc. though.07:46
cjwatsonAlthough the bootstrap archive could be considered to be under manual control only, I suppose.07:47
cjwatsonThere are two cases there: port bringup and bootstrapping cycles in normal operation.  In the latter you normally have a specific sequence of packages in mind and don't want other things to automatically pop up.07:49
cjwatsonNot sure about the former.07:49
wgrantYeah.07:57
cjwatsonIf we can ignore that case then it becomes "just" a performance problem of grabbing all the Packages/Sources files to alphecca for any current dep-wait build and analysing the lot.08:00
cjwatsonAnd stopping trying to figure out the dep-wait target from sbuild output.08:00
wgrantThe quotes are the reason I haven't tried it.08:01
cjwatsonI should see how long it would actually take.08:01
wgrantUnfortunately we can't exactly optimise it like we should publish-distro.08:05
wgrantThat is, skip archives that haven't changed.08:05
cjwatsonIndeed.08:05
wgrantBecause people occasionally do upload to the primary archive, and a few things depend on it.08:05
maozhouI dputed a package to my launchpad server, and accepted it , but the queue is empty, why?09:12
wgrantmaozhou: Why would it be in the queue after you'd accepted it from the queue?09:22
maozhouthe build queue is empty, there are no build job after accepted the package.09:24
wgrantmaozhou: What did process-upload say about it?09:25
maozhouit say "Finished checking upload" and "Setting it to ACCEPTED"09:31
wgrantDid it say it created any builds?09:32
wgrantShould it have created any builds?09:32
wgrantHave builds previously worked on that series on that instance?09:32
maozhouNO , it haven't creanted any builds.09:35
wgrantDoes it have appropriately DistroArchSerieses and chroots configured?09:35
maozhouIt's the first package on this test launchpad server.09:36
maozhouthe DistroArchSerieses is default , ubuntu/utopic , and I have dput the chroot to it.09:38
wgrantOne cannot dput a chroot.09:38
wgrantAnd ubuntu/utopic does not exist by default.09:38
maozhouI can  access the url of "launchpad.dev/ubuntu/utopic"09:40
wgrantHow did that get there?09:40
maozhouand I run "LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1   ./manage-chroot -l dev -s utopic -a i386 -f chroot-ubuntu-utopic-i386.tar.bz2 set" to dput the i386 chroot file .09:41
wgrantAh, so not dput.09:41
wgrantHow did you create utopic?09:42
wgrantDoes https://launchpad.dev/api/devel/ubuntu/utopic have a link to the chroot?09:42
maozhouyes, It's exist09:42
wgrantWhat is the full output of process-upload?09:42
cjwatsonE: Build-Depends dependency for sbuild-build-depends-r-bioc-biovizbase-dummy cannot be satisfied because candidate version of package r-bioc-variantannotation can't satisfy version requirements09:43
cjwatsonhm, that's less helpful than I might have wished for, apt09:43
wgrantcjwatson: Hum,.09:44
cjwatson(trying apt-get build-dep after realising that of course the way sbuild constructs the dummy Sources means that --[no-]resolve-alternatives should be fine)09:44
maozhouhow to verify  https://launchpad.dev/api/devel/ubuntu/utopic have a link to the chroot?09:45
wgranthttps://launchpad.dev/api/devel/ubuntu/utopic/i386, rather09:46
maozhouoh, the chroot_url is empty.09:49
wgrantThe manage-chroot, then, did not succeed.09:49
maozhoubut I have ran "LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1   ./manage-chroot -l dev -s utopic -a i386 -f chroot-ubuntu-utopic-i386.tar.bz2 set" , there is no error.09:50
wgrantConsider that your hostname resolution my have issues.09:51
wgrantSince you're using launchpad.dev everywhere, you must have some serious /etc/hosts entries.09:51
maozhouin the file of "/etc/hosts"  ,  the ip is the ip of launchpad server09:52
wgrantDoes api.launchpad.dev resolve to the right IP address?09:52
maozhouhow to  verify  api.launchpad.dev resolve to the right IP address?09:55
wgrantping or host, for example.09:56
maozhouyes, when ping launchpad.dev, the ip is right09:57
wgrantNot launchpad.dev.09:58
wgrantapi.launchpad.dev.09:58
maozhouoh, ping apt.launchpad.dev , the ip is 127.0.0.88, t's incorrect ip ?10:01
wgrantI can't tell you that.10:02
wgrantI know not the layout of your configuration.10:02
maozhouoh, I'm sorry , I  looking in the wrong , on the machine who ran  "LP_DISABLE_SSL_CERTIFICATE_VALIDATION=1   ./manage-chroot -l dev -s utopic -a i386 -f chroot-ubuntu-utopic-i386.tar.bz2 set" , ping api.launchpad.dev, the ip is  the launchpad server10:06
wgrantCan you see the relevant request in the Launchpad webapp access log?10:07
maozhouwhere is the Launchpad webapp access log?10:09
wgrantIn the development configuration it is logs/launchpad-access.log10:10
cjwatsonwgrant: Trying to understand http://bazaar.launchpad.net/~wgrant/launchpad/product-aps-set/revision/17584 - in what way does ProductSet.getProductPrivacyFilter require a policy or artifact grant?  AFAICS it requires one of (product is public, product has no information type, policy grant)10:14
wgrantcjwatson: AccessPolicyGrantFlat is a combination of AccessPolicyGrant, AccessArtifactGrant, and AccessPolicyArtifact.10:16
wgrantThat is, one can use it to query whether someone has a grant for a policy, or a grant for any artifact under that policy.10:16
cjwatsonah I see10:16
wgrantSomeone with a subscription to a bug in a private project must be able to see the project, so the AAG's row in APGF is what allows them that access.10:17
cjwatsonSo with this we'll basically stop using APGF for the project privacy filter10:17
wgrantCurrently implemented in getProductPrivacyFilter (buggily) and SharingService.checkPillarArtifactAccess.10:17
wgrantCorrect.10:17
wgrantAPGF was originally devised for the listing on +sharing, but it proved useful for access checks too.10:18
wgrantWait, no, not correct.10:19
wgrantAPGF will still be used.10:19
wgrantWhat we're removing is the join through AccessPolicy.10:19
* cjwatson looks at some more branches at once. I think I get it now.10:25
wgrantWe still use APGF to get the set of valid APs.10:25
wgrantBut we can now do that once, and compare it against the precached list on each Product.10:25
wgrantLike we do on Bug/Branch/etc.10:25
wgrantThe only difference from the artifact case is that for Product we also allow artifact grants to count.10:26
cjwatsonWould it be worth using AccessPolicySource.find instead of findByPillar so that we can query for just the types we care about?10:29
cjwatson(in _cacheAccessPolicies)10:29
cjwatsonAlso, GitRepository etc. calls it access_grants instead10:29
wgrantAh, yes, find is usable now.10:30
wgrantaccess_grants is a list of Person.ids who have AAGs for the artifact.10:30
wgrantEach artifact has access_grants (a list of Person.ids), and access_policies (a list of AccessPolicy.ids)10:31
cjwatsonI think they mostly just have a single access_policy10:35
wgrantAh, yes, all except bugs have just one, that's true.10:36
wgrantBugs may have multiple, as they may affect multiple pillars.10:36
cjwatsonwgrant: We actually can make useful use of dose-builddebcheck, I've worked out (after swearing a lot at apt-get build-dep's failure to provide useful failure information).  Algorithm as follows.  First, try sbuild, which I think should be modified to use apt-get build-dep anyway but it doesn't really matter here.  If that fails and the output matches a build-dep failure of some kind, run dose-builddebcheck --explain --failures, ...16:33
cjwatson... passing all candidate Packages files and a Sources containing just the source we're building.  If it reports no reason, PACKAGEFAIL.  If it reports a reason with no depchain, then that's a broken direct build-dependency and can safely be turned into DEPWAIT.  If it reports a reason with a depchain, then that's an indirect broken dependency, and for the reasons you gave above that needs to be PACKAGEFAIL.16:34
cjwatsonNot as potentially elegant as using dose-builddebcheck pre-build somehow, but we know that's hard in our setup, and this deals with the problems with getting an accurate dep-wait out of apt-get build-dep.16:35
cjwatsonTakes 8s on my laptop to analyse something universe-sized, 1s to analyse something main-sized.16:37
cjwatsonWe could optimise a bit by only doing this in cases where it's hard to get sensible information out of apt-get build-dep.16:38
cjwatsonwgrant: Any progress on webhooks, BTW?18:45
wgrantcjwatson: Hm, that's extraordinarily slow.19:57
cjwatson*shrug* for something at the end of an sbuild run I'm not very bothered and haven't tried to optimise it20:09
cjwatsonI've probably missed something20:09
wgrantYeah, just means it's completely impossible to run as something like retry-depwait eventually :/20:09
cjwatsonUnless I missed something blatantly obvious, which is probable :)20:10
cjwatsonAlso it depends how that scales; if it's spending most of its time just parsing the big Packages file then that's less of a problem than if it's lots of time analysing per-package.20:12
blrmorning20:26
blrcjwatson: ah, thanks for catching the mode edge case20:48
blrthis bit of code did worry me in the past... glad we have a few more tests in place.20:49
wgrantcjwatson: Did you look over product-aps-use?21:41
cjwatsonwgrant: I got started on it, but only just22:07
wgrantcjwatson: k22:07
cjwatsonquite a bitty day, I had to take B out to go suit shopping22:07
wgrantI won't land the first bits until you've confirmed it's all good.22:08
wgrantThough I guess the schema patch is safe.22:08
wgrantcjwatson: The refactored diff processing is much more readable, thanks.22:11
cjwatsonwgrant: What do you think of asking webops for https://pastebin.canonical.com/134245/ ?  It's obviously PII, but in discussion a while back we seemed to collectively think it would be OK ...23:19
wgrantcjwatson: Sounds reasonable to me.23:23
blrcjwatson: did you see my comment re: comment diff email?23:33
blrcurrently we're not preserving comments on dirty headers at all (I can't recall if we agreed on that, or that's an oversight)23:34
blrbzrlib parses a modified file with properties changed as a 'dirty header'23:40
cjwatsonblr: Replied to your comment (indeed, you're correct)23:44
cjwatsonblr: I don't think it should be possible for comments to go missing from any position, so that would be a good thing to fix.23:45
cjwatsonMaybe just treat it much the same way you do comments on patch headers23:46
wgrantcjwatson: The devel buildbot green is false. It looks like a query count test broke.23:53
wgrantThough it may be spurious, as I don't see how you could have broken it.23:53
cjwatsonthanks.23:53
wgrantUnless it was test ordering, ew.23:53
cjwatsonIt looks like it could benefit from record_two_runs23:57
cjwatsonIt does fail on a single run here23:58

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!