=== skaet_ is now known as skaet [02:16] wgrant: I thought bug supervisor turned up on IProduct:+edit ? [02:17] Ah [02:17] Configure bug tracker [02:17] IE, UI trainwreck [03:11] NCommander: but the trend in Launchpad itself is to start trying to split things out into microservices where possible, rather than embedding them all in a single giant webapp [03:12] (and for those to be affiliated to Launchpad in the necessary ways) [03:52] cjwatson: er, wasn't one of the motivations of writing launchpad in the first place to prevent having a million microapps (i.e., bugzilla+dak) [03:53] I don't personally care one way or another, but having centralized one-stop place for packages/images/builds/archives makes a lot of sense to me [03:55] what did I miss ? [03:57] 13:10:00 -!- lifeless [~robertc@125.7.18.100] has quit [Ping timeout: 252 seconds] [03:57] 13:11:24 < cjwatson> NCommander: but the trend in Launchpad itself is to start trying to split things out into microservices where possible, rather than embedding them all in a single giant webapp [03:57] 13:12:18 < cjwatson> (and for those to be affiliated to Launchpad in the necessary ways) [03:57] 13:23:44 -!- lifeless [~robertc@125.7.18.100] has joined #launchpad-dev [04:02] wgrant: thanks [04:03] NCommander: separate codebase != separate UI || siloed behaviour [04:03] NCommander: LP set out to create an integrated set of behaviours [04:03] Which it succeeded at [04:03] NCommander: separate services provide natural scaling and HA boundaries, assuming the services are well defined [04:03] NCommander: bugzilla vs dak is not the same as microservices making up the whole product. [04:04] Fair enough [04:04] NCommander: for an example of another microservice approach, look at s3 + ec2 + cloudscaling etc [04:04] THat being said, I stilla rgue that ISO mastering would fall under Launchpad; image builds are already dependent on the LP service and this provides ways for flavors to control their own respins/etc. [04:04] consistent feel, integrated management consoles and so on, but very focused facilities that are orthogonal [04:05] I think iso mastering should move onto the regular buildds. [04:05] No. [04:05] The live filesystem component should move onto the regular buildds. [04:05] It doesn't make sense for the whole of ISO mastering to do so. [04:06] cjwatson: whats the difference there, and why? [04:06] The line I'm drawing is the part that can be run architecture-independently, and that benefits from being on a machine with a full local mirror etc. [04:07] The live filesystem component is the part that must be run on a system of the target architecture. [04:07] And that certainly belongs on the buildds. [04:07] cjwatson: you need to run the arch indep part one per arch though, right ? [04:08] Yes, but there are useful economies of scale. [04:08] cjwatson: from bandwidth access to the mirror ? [04:08] For example. [04:09] More like same filesystem, since some ISOs use hardlink trees. [04:09] cjwatson: ah, interesting. I haven't climbed into that code. [04:09] cjwatson: thanks. [04:09] Maybe the importance of this will change as more things move to live CDs, but I don't want that to entirely dominate the design. [04:10] cjwatson: do you reuse the hardlink trees for things on different days ? [04:10] I'm not sure I understand the question. [04:10] IIUC you build N > 1 ISOs using hardlink trees for common content ? [04:11] We reuse the mirror. But hardlink trees are cheap to build so we just do that afresh for each image build. [04:11] When you build the same targets the next day, do you build fresh hardlink trees, or evolve the prior one ? [04:11] The former, then. [04:12] anyhow. I'm actually arguing for horizontal scalability and automated from-scratch provisioning of the build system, rather than 'use the buildds' per se. [04:12] closer to stateless etc etc etc [04:13] In terms of scaling, building live filesystems massively dominates; we're a long way off the point where the central system has any trouble keeping up with the extra bits and pieces of ISO mastering it does at the rate that separate builders can deliver live filesystems. [04:14] Obviously that changes at some point horizontally but it's quite some distance away. [04:14] kk [04:16] I was more considering the possibility of removing squashfs builds are a seperate step and merely master an entire ISO in a single go on a buildd [04:16] */2 cents* [04:16] I know you were, but I don't think that's desirable at this point. [04:17] cjwatson: what downside do you see? [04:17] See above. [04:18] In any case I don't think this should even need to be considered for livefs-in-LP work. [04:18] As far as that's concerned, the build job should be calling BuildLiveCD or some equivalent. [04:18] cjwatson: above you discussed efficiencies, but unless you hit a bottleneck that doesn't translate to a specific or even necessary downside. [04:18] What it outputs is the business of the job and the consumer, not of LP. [04:18] lifeless: The downside is that we should be moving to livefs-in-LP without having to refactor the entire cdimage build system along the way. [04:19] And it's actually really quite a considerable amount of code to move around. [04:19] cjwatson: so, two separate [potential] migrations, rather than one that moves + rearranges? If so I concur. [04:20] * NCommander agrees with that, as moving the squashfs building is considerably less diserpative ATM [04:20] and as I said in the beginning, long-term [potential] goal [04:21] cjwatson: also, wth are you awake? [04:22] *You're* asking me that? :-) [04:22] * NCommander is dealing with a rather mysterious issue that the buildd complains UNKNOWNSUM [04:22] WOrks with a precise chroot sha1sum, breaks with a lucid one [04:22] (Random insomnia) [04:23] grep "No URL" -r * [04:23] er [04:24] oh [04:24] the only reason my trigger code worked was the precise chroot was cached in the filecache for the buildd [04:28] cjwatson: do we have any images that still require livecd.sh? (I have a handler that calls that, live-build, and ubuntu-defaults-image) === jtv1 is now known as jtv [04:50] lifeless: I see your iwl is as stable as ever. [04:53] StevenK: I'm on ethernet [04:53] StevenK: Hotel ethernet. [04:53] That's as stable as iwl [04:54] wgrant: What sort of cruft is sinzui talking about it bug 1036189? [05:02] StevenK: eg. if I set my project to only use Proprietary bugs and branches, and I change all the Private and Private Security artifacts to Proprietary, there's no need to keep Private and Private Security around [05:02] The APs and APGs can be deleted. [05:04] That sounds more like a job Job rather than a garbo [05:05] StevenK: We'd have to trigger it on every transitionToInformationType [05:05] So no [05:06] Right [05:06] wgrant: So if it's a garbo what do I search on? [05:06] StevenK: Hm? [05:08] wgrant: So if I want to pick products that need AP{,G}s deleted, how do I do it? [05:09] StevenK: Find APs that are forbidden, and then find the subset that are unused [05:10] wgrant: Right, so find APs that do not make sense given what the product is set up for. [05:11] StevenK: Right [05:25] wgrant: StevenK: and you won't let folk file bugs that noone-can-see? [05:26] lifeless: We will, but the project owner would have to explicitly revoke the permission [05:26] The default is always that the owner can see each type [05:26] note too the race conditions at the time of revoking [05:26] (and the owner gets a warning on +sharing if they do that) [05:26] Oh? [05:27] new bug of the type the permission is being revoked for [05:27] owner may think there are none, but \/ races [05:27] We don't prevent filing a bug that nobody can see [05:27] sure [05:27] If the project owner wants to do stupid things, we let them do stupid things, but they can easily recover [05:27] saying that doesn't mean humans won't assume ... [05:28] same race exists for setting the project to only proprietary, if you do stuff inline, of course. [05:29] Changing the allowed types (eg. setting the sharing policy to Proprietary only) doesn't affect existing artifacts. [05:29] But it does stop you from making new ones of the illegal types [05:29] cool [05:29] * StevenK tries to cook a query [05:40] wgrant: So I'm looking for projects where bug_sharing_policy or branch_sharing_policy isn't PUBLIC and they have APs for PRIVATESECURITY and USERDATA. I'm guessing it's a little more complicated, but it seems those are the only policies that are excluded. [05:52] wgrant: Right? [05:53] StevenK: If you look near the top of lib/lp/code/model/branchnamespace.py you'll need the dict of allowed types [05:53] s/need/see/ [05:54] Does that also follow through for bugs? [05:54] There's a similar one for bugs in a branch that I haven't landed yet. It's basically identical. [05:56] wallyworld_: In the sharing_policy = PUBLIC garbo branch, would it be better to check that the project is set to public already (ie. not private_bugs and not exists branchvisibilitypolicy) rather than checking for the lack of a commercial subscription? If we're landing this now, it'll also mean that commercial project setup changes now. [05:56] I'd suggest we defer landing it until new projects are being set up using sharing_policy [05:57] wgrant: i'm not sure why commercial project setup would change? the job only updates non commercial projects [05:58] we discussed the need to do this in the call - we wanted to ensure that the many projects were migrated for people before beta [05:58] wallyworld_: If a project happens to accidentally be created non-commercial and only made commercial later, the old commercial setup procedure will appear to work, but do nothing [05:59] private_bugs will be set, but bugs will be public [05:59] sure, but if a project is made commercial, it would need to have it's sharing policy set at the same time [06:00] Right, so we'd need to advise the people who do commercial setup (probably just czajkowski and us nowadays?) to unset sharing policies [06:00] That may be acceptable, but we need to say that first :) [06:00] yes, curtis is all over it [06:01] Sounds good, then. But we do need to check that the project is actually configured to be public, not just that it has no commercial subscription. [06:01] Or we may be in for a nasty surprise if anything's misconfigured. [06:01] Since commercial admins can set private_bugs and BVPs without a commercial sub [06:01] So who knows what madness lies out there today... [06:01] by "public" you mean check the licence? [06:01] wgrant: morning [06:02] heya czajkowski [06:02] wallyworld_: Needs to have private_bugs = false, and there must be no BranchVisibilityPolicies at all [06:02] czajkowski: Hi [06:02] gentles all; is it still possible for people and/or teams to have name aliases? I can't seem to find any such UI modification for same any more? possibly it was only for projects?? [06:02] wgrant: so theoretically only commercial projects can have private_bugs = true right? [06:02] spm: People/teams never could [06:02] spm: Projects can [06:02] I believe myself and mrevell will be talking to people who have commercial projects next weeek and going over the new setup with them once myself and mrevell know more and know fully about the set up [06:03] wgrant: right. ta. I misrecall. thanks mon [06:03] wallyworld_: Sort of [06:03] wallyworld_: "theoretically" in that "in an ideal world, there's no reason for it to be any other way" [06:03] wallyworld_: But it's by no means enforced or even suggested anywhere in LP that that should be the case [06:03] As a commercial admin, I can go over to Random Open Source Project and set private_bugs and make all branches private only to canonical. [06:04] In the old model [06:04] and that would be a mistae, no? [06:04] By policy, yes. [06:04] But it will work absolutely fine. [06:05] And LP won't warn you that you've done something that doesn't make sense. [06:05] And isn't meant to be supported. [06:05] so if it's not supported then the garbo job can do what makes sense, or? [06:06] wallyworld_: It's not supported, but it may well exist and if it does then the current garbo algorithm will cause private stuff to become public. [06:06] "unsupported" private stuff [06:06] s/it may well exist/there is no reason it wouldn't exist/ [06:06] Sure [06:07] But someone making a silent misconfiguration when touching privacy is not a valid reason to make all their stuff public quietly :) [06:07] ie if people don't have a commercial subscription, can they reasonably expect their stuff to be private? [06:08] wallyworld_: If they set private_bugs to true and created private BVPs and LP didn't complain and it worked for 5 years, then yes. [06:08] how would they set private bugs to true? i thought only an admin could do that? [06:08] wallyworld_: We have admins all over the company [06:08] (commercial admins, not just full admins) [06:09] OEM, HWE, and even non-Canonical groups like Linaro have commercial admins [06:09] LP doesn't make the supposed rules clear, so assuming that they've been followed is... not exactly reliable. [06:10] hmmm, ok. i initially saw the issue as "freeloaders" no longer getting something they hadn't paid for [06:10] Well, there's that side of the issue. [06:10] But that's relatively minor [06:11] compared to the risk of making confidential information public without telling anyone [06:11] i can do a followup branch then to tighten the rules for what projects are migrated. custis seemed happy with the current approach when we discussed it during the call [06:14] launchpad_dogfood=# SELECT COUNT(*) FROM product WHERE private_bugs AND NOT EXISTS (SELECT 1 FROM commercialsubscription WHERE commercialsubscription.product = product.id); [06:14] count [06:14] ------- [06:14] 8 [06:14] (1 row) [06:14] That branch is dangerous :) [06:14] fsvo [06:15] They're all canonical/linaro projects [06:15] So by definition not freeloaders [06:15] do i really need to check bvp or is private_bugs sufficient? [06:17] wgrant: I've been staring at this query for like ten minutes and still doesn't make sense. I know what I'm trying to say, but not how to say it. :-( [06:18] welcome to my world StevenK :) [06:19] StevenK, wallyworld_: gimme a sec [06:20] wallyworld_: There are 30 projects with private, private only, or forbidden BVPs and no commercial subscription [06:20] czajkowski: Haha [06:20] wgrant: ok, i'll add in bvp checks also [06:20] bvp? [06:20] wallyworld_: Thanks. [06:21] wallyworld_: I didn't think there'd be this many, tbh :/ [06:21] branch visibility policy [06:21] ahh yes *headdesk* [06:21] czajkowski: they're going away soon so don't even bother to try and understand them :-) [06:36] wgrant: i'd be ok to ignore bvp = public? [06:37] wallyworld_: Yeah, projects that have become public afterwards can't have their BVPs removed, so they're just all set to public [06:37] wallyworld_: A project behaves as if it has no BVPs if it only has public BVPs [06:37] ok, so bvp > 1 in my query [06:37] A single Private, Private Only or Forbidden rule means it's not public [06:37] yep [06:58] have folks come acorss liciencing choices/discussions before on LP https://bugs.launchpad.net/launchpad/+bug/1037685 [07:03] wgrant: So no help for me? :-( [07:04] StevenK: Sorry, what do you have so far? [07:05] wgrant: Roughly 'store.find(Product).find(Product, ' because I've trying to figure out how the next bit should look. :-( [07:06] czajkowski: So project licensing we don't really tend to care about -- we really only care is it an open source license or not? [07:07] StevenK: So, one thing you could do is batch through the products, grab their APs, compare them to the set [07:07] StevenK: cant really reply with I don't care now can I :0 [07:07] :) [07:07] StevenK: It's only going to be garbo-daily, so it doesn't need to be a stunningly fast single query [07:08] wgrant: Right, I figured that bit [07:13] StevenK: Then you compare the information types from the access policies with the ones allowed by *_sharing_policy. Any that don't match the sharing_policy, you query accesspolicyartifact to see if there are any artifacts using it. If not, you delete the APG and AP [07:13] s/APG/APGs/ [07:15] wgrant: Yes, I understand that bit, what I've been stuck on is how to tackle the query that finds if there is work to do. [07:16] I'd just iterate through all the products [07:16] There's not very many in the scheme of things. [07:16] And we only need to do this dailyish [07:48] NCommander: no, livecd.sh is historical interest only. [07:48] cjwatson: well, we still use it for old images, and generally, I think we want the capability to respin those until they leave support, no? [07:56] good morning [08:13] NCommander: so the way it works is that we always build live images in a chroot of the relevant series, with the relevant series' livecd-rootfs installed [08:13] NCommander: the only piece that's series-independent is BuildLiveCD [08:13] NCommander: if at all possible I'd strongly recommend that you simply call BuildLiveCD [08:13] I'm aware. I'm re-implementing BuildLiveCD into the buildd slave. [08:14] Is there a reason to do that rather than calling it? [08:14] It has quite a bit of distro-specific logic. [08:14] BuildLIveCD can't be series-independent because the chroots are always presisene unles syou want to pull it in from a PPA or other location [08:15] BuildLiveCD isn't stored in the chroot. [08:15] In addition, there isn't a lot of distro-specific logic, and by having the interface one level up, its easier to pass new options to build images or other engineering fun [08:15] cjwatson: so then we want to install it on every buildd by hand? (or make it part of launchpad-buildd? [08:15] It's installed on every buildd by hand at the moment. [08:16] I guess I can see the logic in reimplementing it. If you do, you need to keep all the things it does, including, yes, livecd.sh. [08:18] Damn, now that bizarre failure I was seeing in EC2 has infected buildbot. [08:22] cjwatson: Which? [08:22] I don't see one [08:23] lp.soyuz.browser.tests.test_archive_webservice.TestArchiveWebservice.test_getAllPermissions_constant_query_count [08:23] on db-devel === almaisan-away is now known as al-maisan [08:23] cjwatson: It's odd that it's doing both cookie-based sessions and OAuth [08:24] But what's the extra query? [08:24] Maybe; but the test is that there are the same number of queries for one perm vs. two. [08:24] Trying to reproduce locally at the moment. [08:24] I tend to just decrease the limit in the test to get it to give me a dump [08:25] Yeah. [08:25] Then compare to the buildbot failure, and fail to work out what has changed :) [08:25] I'm mostly waiting for my beard to grow while make runs, at the moment. [08:25] Heh [08:25] make compile is pretty fast [08:26] and usually sufficient for non-launchpadlib tests [08:26] This is a launchpadlib test. [08:26] launchpadlib, or the test suite's webservice client thingy? [08:27] LaunchpadWebServiceCaller plus launchpadlib_for, apparently. I was slotting into the existing TestArchiveWebservice. [08:27] Ah [08:27] But I specifically wanted to ensure there was no late evaluation hidden in lazr.restful. [08:27] WebServiceCaller doesn't need WADL, launchpadlib_for does [08:27] Right [08:29] It must be at least slightly random, because my second attempt to get uefi-ppa-no-unapproved through EC2 succeeded. [08:31] cjwatson: Right, this sort of thing often depends on whether you're the first webservice test in the process or not [08:31] Oh [08:32] This is probably complaining there's one too few queries. [08:32] * wgrant counts.. [08:32] No, one too many :( [08:32] There goes the session secret theory [08:34] Mm, it's expected != other. [08:38] Oh. If I put another list(self.main_archive.getAllPermissions()) at the start of the test, that triggers the failure. So that suggests it's something a bit like what you're suggesting. [08:39] Also, is lp.buildmaster.tests.test_builder.TestSlave.test_status_after_build a known-flaky test? [08:46] The extra query in the second run is "134-2193@SQL-main-master SELECT Component.id, Component.name FROM Component WHERE Component.id = 1 LIMIT 1". Oddly, it's listed at the start. I wonder if it's a leftover from the first run somehow [08:50] Also odd because there's already a "134-138@SQL-main-master SELECT Component.id, Component.name FROM Component WHERE Component.id = 1 LIMIT 1" in the first run. [09:04] * cjwatson discovers record_two_runs, whose purpose seems to be to deal with this. === mrevell_ is now known as mrevell [09:10] we'd like to get https://code.launchpad.net/~james-w/udd/binary-scanning-series/+merge/119248 landed, but I've forgotten how to do it for udd (tarmac? pqm? merge & hope?) [09:45] Plain old bzr [09:46] wgrant: https://code.launchpad.net/~cjwatson/launchpad/testfix-getallpermissions/+merge/120094 [09:46] (or any other reviewer really, but I was talking to wgrant about this above) [09:46] cjwatson: Looking [09:47] Well, once the diff is there [09:48] Ah, it won't be there for a while [09:48] fdt [09:48] * wgrant looks at loggerhead [09:50] cjwatson: r=me. I didn't know about that nice helper. [09:50] nor did I before grep happened to turn it up :) [12:26] hi adeuring are you reviewing today? === bac changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: bac | Firefighting: - | Critical bugs: 4.0*10^2 [12:26] bac:sorry, forgot to change the toipic.. === adeuring changed the topic of #launchpad-dev to: http://dev.launchpad.net/ | On call reviewer: bac, adeuring | Firefighting: - | Critical bugs: 4.0*10^2 [12:27] adeuring, quite a few branches on +activereviews. hope we can knock those out today. [12:29] bac: ok, i'll start with "fix-.pocket-queue-admin-series" === al-maisan is now known as almaisan-away === almaisan-away is now known as al-maisan [14:28] * deryck switches work locations, back online shortly [15:16] adeuring: Thanks for your review of fix-pocket-queue-admin-series. While I don't object to making the change you mentioned for clarity's sake, you said that it was in LP's style guide, and I can't find it anywhere in https://dev.launchpad.net/PythonStyleGuide or in at least the first few linked pages I looked through. Could you point me to what I missed? [15:17] cjwatson: let me see (actually, I claimed that this rule exists, but I haven't looked at our style guide for longer time...) [15:20] cjwatson: https://dev.launchpad.net/CodeReviewChecklist?highlight=%28elif%29 [15:20] aha, thanks [15:20] though that might be cruft... [15:21] cjwatson: the idea idea of the rule is that it should be clear that all possible cases of the if/elif/elif... are covered -- just prevent "casual" errors [15:22] aye [15:22] I've put it back now [15:34] jcsackett: do you have time to discuss the IBranchEditableAttributes interface. Project maintainers don't have permission to use change the attrs. This is a problem since the branch is shared with the project, and they cannot respond to cases where data is disclosed, or the branch owner is no longer associated with the project. [15:35] jcsackett: I want you to read the interface before we talk because I was surprised by some of the attrs in it === salgado is now known as salgado-lunch [15:41] sinzui: i'll put up the file now, and ping you in a few. [15:44] sinzui: short read. g+? [15:45] yes [15:45] sending invite momentarily. === al-maisan is now known as almaisan-away === salgado-lunch is now known as salgado === matsubara is now known as matsubara-lunch === deryck is now known as deryck[lunch] === matsubara-lunch is now known as matsubara [18:57] bac: do you have time to review https://code.launchpad.net/~sinzui/launchpad/project-branch-permissions/+merge/120219 === deryck[lunch] is now known as deryck [20:19] Is there some way to get my testfix from devel earlier merged over to db-devel so that its buildbot can start working again? [20:21] Actually, isn't buildbot-poll supposed to do that? [20:37] cjwatson: buildbot-poll will do it if the db-devel builder isn't in proper testfix mode [21:23] benji: added [21:32] lifeless: thanks [22:36] wgrant: I fear it may be in testfix mode, though [22:36] So does it need somebody to merge manually? [22:41] cjwatson: Not sure if codehosting's still up, but indeed. [22:42] You can either merge the testfix branch, or just say the whole stable merge is a testfix. [22:45] wgrant: with bzr lp-land or something more arcane? [22:47] cjwatson: I'd just pqm-submit stable directly [22:48] bzr pqm-submit -m '[testfix][r=foo] Merge db-stable rWHATEVER' --public-location=bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable --submit-branch=bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel [22:48] or so [22:48] ok, will have a look in a bit, modulo whatever bits of LP are still alive [23:00] wgrant: not being a reviewer, can I use r=wgrant? [23:00] cjwatson: Be my guest [23:09] done [23:10] Thanks. [23:10] * cjwatson slightly freaked out by glimpsing "bzr: ERROR: no such option: -0" in PQM output [23:10] I thought I'd removed that check. [23:10] Maybe not [23:11] It's pointless and doesn't work any more, anyway [23:14] cjwatson: ok, now that I slept on it. WHy do we want to keep BuildLiveCD? The API I have for Launchpad lets the client choosing the build choose the chroot and the series its building (by default, whatever series you choose to build will be built in said chroot aka; lucid build in lucid using livecd.sh, precise build in precise using live-buid (or ubuntu-default-image) [23:14] Is there another piece of functionality I'm overlooking? [23:14] It may not be too outrageous to replace BuildLiveCD with something equivalent, indeed. [23:15] Now that I've thought about it, our process for modifying that involves RT. Changing that to involve landing an LP branch might actually be an improvement. [23:15] Would be easier to see what's currently running.