[01:12] <lifeless> wallyworld_: WHATS UP?
[01:12] <lifeless> bah, caps lock
[01:13] <wallyworld_> lifeless: it's ok, i already sorted it. the deployment report had a rev that should have been tagged rollback=xxxx and i was wondering how to fix it
[01:13] <wallyworld_> and no-one else was shown as logged into irc :-)
[01:14] <StevenK> Missing rollback tags are annoying
[01:14] <StevenK> Since you can't add them after the fact
[01:17] <wallyworld_> yeah, so i found out
[01:27] <lifeless> qatagger is fixable :)
[01:33] <wgrant> lifeless: Yay for fixing bug watches.
[01:40]  * StevenK doesn't get it.
[01:40] <StevenK> Replicating the conditions of bug 892025, and I don't see the link. But I do on qas/prod
[01:40] <_mup_> Bug #892025: Forbidden following link to configure code hosting <403> <qa-ok> <trivial> <ui> <Launchpad itself:In Progress by stevenk> < https://launchpad.net/bugs/892025 >
[01:41] <wgrant> StevenK: You probably have permission to set it, but can't see the dev focus branch.
[01:41] <wgrant> Unauthorized: (<Branch u'~hexr-dev/hexr/trunk' (406169)>, 'unique_name', 'launchpad.View')<br />
[01:41] <StevenK> Oh
[01:42] <StevenK> So configure_codehosting should learn that too?
[01:42] <wgrant> No.
[01:42] <wgrant> Well, maybe.
[01:42] <wgrant> It's difficult.
[01:42] <wgrant> Leave it for now.
[01:42] <wgrant> We'll revisit it if it's still a problem once we've fixed privacy.
[01:43] <StevenK> Bah, Gavin hinted at the issue in the description too
[01:43] <wgrant> I just looked at the traceback :)
[01:44] <StevenK> I saw the error page and assumed it was "You can't see this page, go away"
[02:22]  * StevenK stares at BugContextMenu.nominate()
[02:22] <StevenK> You can Target if you're a driver, nominate if you're a bug supervisor and everyone else can bugger off?
[02:23] <lifeless> wgrant: context? my email I presume?
[02:25] <wgrant> StevenK: Yes
[02:25] <wgrant> lifeless: Yes
[02:32] <StevenK> wgrant: How is that related to bug 297528, then?
[02:32] <_mup_> Bug #297528: Permissions not checked properly when deciding whether to present "Target" or "Nominate" link <lp-bugs> <trivial> <ui> <Launchpad itself:Triaged> < https://launchpad.net/bugs/297528 >
[02:34] <wgrant> StevenK: Well, drivers or uploaders can target.
[02:35] <wgrant> But whatever decides to show Target/Nominate doesn't take into account uploaders.
[02:35] <wgrant> So for uploaders it says it will nominate, when it will in fact target.
[02:36] <StevenK> wgrant: Right, so BugContextMenu.nominate() needs a elsif I can upload, which says target.
[02:36] <wgrant> Yes, but that needs to be faster than it probably currently would be.
[02:38] <wgrant> StevenK: There's more MixedVisibilityErrors, if you want them.
[02:38] <StevenK> Oh?
[02:38] <wgrant> ProductSet:+all
[02:39] <StevenK> Can I have an OOPS too?
[02:39] <wgrant> Should be pretty obvious :)
[02:39] <wgrant> Given what the page does
[02:39] <wgrant> But OOPS-01faea627b2e0a135da5d8e3640e22ef
[02:40] <StevenK> Oh, it returns every signle product?
[02:51] <StevenK> wgrant: So I guess we want to hide the product if the user can't view the registering team?
[02:52] <wgrant> Possibly.
[02:52] <wgrant> I didn't think private teams were allowed to own projects :/
[02:53] <wgrant> We may be best to leave this.
[02:53] <wgrant> As the new rules should mean that the team is to be disclosed.
[02:53] <StevenK> Chewie
[02:53] <StevenK> Chewie exports system settings through a menu service such as dbusmenu and/or gmenu
[02:53] <StevenK> Registered by <hidden> on 2011-12-02
[02:53] <wgrant> That's not the one that's the main problem here.
[02:54] <wgrant> But it's a similar case.
[02:54] <wgrant> dpkg-coverity is the one here, though I can see the team due to some kind of subscription
[02:54] <StevenK> Actually, disclosing teams in a public role should fix this, you're right
[02:54] <wgrant> Right.
[02:54] <wgrant> But that requires discussion in Budapest.
[02:54] <StevenK> And beating lifeless with a stick
[02:54] <StevenK> I think
[02:55] <StevenK> So I have 1.5 days left. And Curtis has forbidden me to have any branches in progress over the break.
[02:56] <wgrant> Heh
[02:59] <lifeless> StevenK: hmm ?
[03:00] <wgrant> We have things to discuss in Budapest.
[03:00] <StevenK> wgrant: So I guess I should find a critical
[03:14] <StevenK> Hm, I think bug 739070 could be fixed with two new columns, but I doubt it's completly doable in the time I have left.
[03:14] <_mup_> Bug #739070: Archive:+repository-size timeout retrieving many hundreds of package sizes <timeout> <Launchpad itself:Triaged> < https://launchpad.net/bugs/739070 >
[03:15] <StevenK> And wgrant may shoot me for wanting to add two new columns to Archive.
[03:17] <wgrant> Correct.
[03:18] <StevenK> :-(
[03:27] <StevenK> lifeless: Due to your work with OOPSes, I guess bug 788518 can be closed?
[03:27] <_mup_> Bug #788518: staging services share OOPS prefix <critical-analysis> <Launchpad itself:Triaged> < https://launchpad.net/bugs/788518 >
[03:28] <StevenK> And bug 805546, I guess
[03:28] <_mup_> Bug #805546: persontransferjob does not have a unique oops prefix <Launchpad itself:Triaged> < https://launchpad.net/bugs/805546 >
[03:32] <wgrant> StevenK: Indeed.
[03:33] <wgrant> StevenK: Are you on launchpad-error-reports?
[03:33] <wgrant> StevenK: There's an issue with that OOPS work of lifeless' which somebody need to look at.
[03:34] <StevenK> wgrant: I am not
[03:34] <wgrant> 2011-12-18 09:52:03 ERROR   Unhandled exception -> http://launchpadlibrarian.net/87826942/iDGKk0EWB2rD9ZfBovt3JrR3nLJ.txt (unsupported operand type(s) for +: 'NoneType' and 'str')
[03:34] <wgrant> From checkwatches on loganberry.
[03:37] <StevenK> I guess either config[self._default_config_section].oops_prefix is None?
[03:37] <wgrant> It looks that way, but I can't see how.
[03:38] <StevenK> _default_config_section is error_reports
[03:38] <wgrant> Indeed.
[03:39] <StevenK> You were fiddling with the configs last week ... :-P
[03:39] <wgrant> Yes, but lifeless fiddled with this bit :)
[03:40] <StevenK> huwshimi: AH ha! You have QA to do.
[03:40] <huwshimi> StevenK: I do!
[03:40] <huwshimi> StevenK: Telstra have been out trying to fix my internet all morning
[03:41] <StevenK> They may have succedded?
[03:42] <huwshimi> StevenK: It appears that way
[03:42] <huwshimi> StevenK: At least it is working, if  a little slow
[03:42] <StevenK> wgrant: I don't get it either.
[03:47] <StevenK> wgrant: Right, production error_reports has no oops prefix, and the code is expecting one if reporter is None.
[03:48] <huwshimi> hmm.. I thought I had 4 branches to qa
[03:49] <wgrant> StevenK: Lies.
[03:49] <wgrant> StevenK: production/launchpad-lazr.conf's oops_prefix is PRODUCTION.
[03:49] <StevenK> wgrant: Oh?
[03:50] <StevenK> steven@liquified:~/lp-production-configs% grep -c oops_prefix production/launchpad-lazr.conf
[03:50] <StevenK> 0
[03:50] <wgrant> bzr pull
[03:50] <StevenK> That's with rev 247
[03:51] <wgrant> srsly?
[03:51] <wgrant> r567 is the latest.
[03:52] <StevenK> bzr pull gives "No revisions to pull."
[03:52] <wgrant> You be pulling from the wrong place.
[03:52] <StevenK> bzr+ssh://bazaar.launchpad.net/%2Bbranch/lp-production-configs/ is the wrong place?
[03:53] <wgrant> That's the right place.
[03:53] <wgrant> Ah
[03:53] <StevenK> steven@liquified:~/lp-production-configs% bzr revno bzr+ssh://bazaar.launchpad.net/%2Bbranch/lp-production-configs/
[03:53] <StevenK> 247
[03:53] <wgrant> That was the crontabs revno I was looking at.
[03:54] <wgrant> 247 is indeed configs
[03:54] <StevenK> Now, grep yourself and take back your lies comment :-)
[03:54] <wgrant> lpnet-lazr.conf is where PRODUCTION is specified.
[03:54] <wgrant> And production/launchpad-lazr.conf inherits that.
[03:54] <wgrant> $ LPCONFIG=production bin/py
[03:54] <wgrant> Python 2.6.5 (r265:79063, Apr 16 2010, 13:09:56)
[03:54] <wgrant> [GCC 4.4.3] on linux2
[03:54] <wgrant> Type "help", "copyright", "credits" or "license" for more information.
[03:54] <wgrant> >>> from canonical.config import config
[03:54] <wgrant> >>> config.error_reports.oops_prefix
[03:54] <wgrant> 'PRODUCTION'
[03:55] <StevenK> Then what crack is checkwatches on?
[03:55] <wgrant> That's why I asked you to investigate :)
[03:56] <StevenK> Oh, hah.
[03:56] <StevenK> class CheckWatchesErrorUtility(ErrorReportingUtility):
[03:56] <StevenK> _default_config_section = 'checkwatches'
[03:56] <wgrant> That would do it.
[03:56] <wgrant> murder
[03:56] <StevenK> Kill it with a large amount of fire?
[03:56] <wgrant> Ideally.
[03:57] <wgrant> lib/lp/services/mailman/monkeypatches/xmlrpcrunner.py:    _default_config_section = 'mailman'
[03:57] <wgrant> :D
[03:57]  * StevenK feels ill.
[04:25] <huwshimi> hmmm... why might I be getting this pqm failure? http://paste.ubuntu.com/774941/
[04:27] <wgrant> huwshimi: Check the attachments.
[04:27] <wgrant> We're in testfix.
[04:28] <StevenK> db-devel went bang with 4 failures and 31 errors.
[04:28] <wgrant> They are real, too.
[04:28] <StevenK> ProgrammingError: permission denied for relation milestonetag looks like the root cause
[04:29] <wgrant> lifeless is more than adequately opinionated on this matter.
[04:29] <StevenK> lifeless said bad idea and it landed anyway?
[04:29] <StevenK> wgrant: https://code.launchpad.net/~stevenk/launchpad/force-checkwatches-production/+merge/86187
[04:30] <wgrant> StevenK: disapprove
[04:30] <StevenK> Awww
[04:30] <StevenK> Why?
[04:31] <wgrant> That will leave the reporter as just PRODUCTION.
[04:31] <wgrant> Rather than PRODUCTION-checkwatches
[04:31] <wgrant> I suspect.
[04:31] <wgrant> You probably want to call .configure(section_name='checkwatches')
[04:32] <StevenK> Yeah, I was just reading that.
[04:33] <huwshimi> wgrant: The pqm failure complained about the commit message being in the wrong format. I assume that's because the branch is attached to a bug that already has a branch that has landed. Do I need to do something to get it to play nice for the second branch?
[04:33] <StevenK> huwshimi: So, someone needs to fix db-devel.
[04:33] <StevenK> I suspect the easiest thing to do is to rollback the milestonetag rev
[04:34] <huwshimi> StevenK: I think that was intended for wgran
[04:34] <huwshimi> *wgrant
[04:35] <StevenK> huwshimi: The PQM failure complained about the commit message not containing [testfix] since db-devel failed on buildbot.
[04:35] <StevenK> That's what "We're in testfix" means.
[04:35] <StevenK> I'll fix it after lunch.
[04:37] <huwshimi> StevenK: Oh right
[04:37] <huwshimi> StevenK: But it also complained on Friday when other branches of mine landed
[04:39] <huwshimi> StevenK: I thought it might have been because the commit message was missing [bug=894442]
[04:43] <StevenK> huwshimi: PQM doesn't care about the content of the tag, it would happily deal with [bug=999999]
[04:43] <StevenK> wgrant: Diff updated, if you're happy with it, I'll self-review and toss it at ec2.
[04:44] <huwshimi> StevenK: I think it does. It's failed because of the commit message: http://paste.ubuntu.com/774951/
[04:45] <StevenK> Note the lack of [bug=
[04:45] <wgrant> huwshimi: You need either [bug=foo] or [no-qa]. That is why that one failed.
[04:45] <StevenK> Or [no-qa]
[04:45] <wgrant> But the current failure is because of testfix.
[04:45] <huwshimi> StevenK, wgrant: Yes, that's what I was saying :)
[04:46] <StevenK> wgrant: No fair approving. But thanks. :-)
[04:46] <huwshimi> I was also wondering why it might not have appended that, and thought it might have had something to do with the bug already having a landed branch
[04:56] <lifeless> wgrant: StevenK: It landed before I commented, which is fine; its not something I'm going to veto anyhow, because we do need to solve the users problems; I'm hopeful we can solve the deeper cause of friction.
[05:20] <StevenK> lifeless: I was about to revert it anyway
[05:26] <StevenK> lifeless: Did you see the two bugs I linked?
[05:31] <StevenK> wgrant: Did we disable the beta bug listing on qas/prod?
[05:33] <lifeless> StevenK: I'm on leave; not following much of anything. Do I need to see them ?
[05:33] <wgrant> StevenK: It's disabled on prod.
[05:33] <wgrant> And qas is a separate team.
[05:34] <StevenK> lifeless: I'd like to close them, I was wondering if you agree.
[05:34] <StevenK> wgrant: Which team on qas, I'd like to QA rick_h__'s change.
[05:35] <wgrant> custom-buglisting-demo
[05:38] <StevenK> wgrant: Thanks, qa-ok
[05:55] <lifeless> StevenK: links ?
[05:56] <StevenK> lifeless: bug 788518 and bug 805546
[05:56] <_mup_> Bug #788518: staging services share OOPS prefix <critical-analysis> <Launchpad itself:Triaged> < https://launchpad.net/bugs/788518 >
[05:56] <_mup_> Bug #805546: persontransferjob does not have a unique oops prefix <Launchpad itself:Triaged> < https://launchpad.net/bugs/805546 >
[05:57] <lifeless> so, uhm
[05:58] <lifeless> what report will ptj be using now ?
[05:58] <lifeless> will someone be able to tell its ptj that an error occurs in
[05:59] <lifeless> for the staging one, definitely downgrade as we are no longer losing oopses
[05:59] <StevenK> Downgrade, or close?
[05:59] <lifeless> if we have reasonable reporters for ~everything on staging, then its closable. I'm not sure if we do yet - because not everything uses configure(section=)
[05:59] <StevenK> lifeless: I just fixed checkwatches
[05:59] <wgrant> We should just use script_name and be done with it.
[05:59] <lifeless> things that don't reconfigure with a specific section will be getting logged as 'STAGING' whether or not they are an appserver.
[06:00] <lifeless> indeed, doing a systematic fix across the system is a great idea. Its a little more than script-Name though
[06:00] <wgrant> Slightly.
[06:00] <StevenK> lifeless: Happy to slam it to High?
[06:01] <lifeless> StevenK: high makes sense to me, along with a comment about the status, of course
[06:01] <lifeless> wgrant: jobs need consideration
[06:01] <StevenK> lifeless: Which you made in October
[06:01] <wgrant> Yes.
[06:01] <wgrant> That's the only real issue I can see.
[06:02] <lifeless> StevenK: other code changes have happened since then, so its worth noting that they *haven't* corrected everything.
[06:02] <lifeless> StevenK: its almost always worth a short comment when changing things, helps keep everyone on the same page
[06:02] <lifeless> wgrant: I'd like to move all lp to a single reporter, and use topic to split out scripts
[06:03] <lifeless> wgrant: be a good idea to do at the same time
[06:03] <wgrant> Once topics exist, sure :)
[06:03] <lifeless> wgrant: they already exist
[06:03] <lifeless> wgrant: its where pageids are reported
[06:03] <wgrant> Ah, true.
[06:04] <StevenK> huwshimi: You can reland your branch
[06:04] <huwshimi> StevenK: :D
[06:04] <huwshimi> thanks
[06:04] <StevenK> huwshimi: Sorry, I could have told you 15 minutes ago, but I got distracted.
[06:05] <huwshimi> StevenK: I can't believe you would do that to me
[06:06] <StevenK> Hah
[07:14] <lifeless> is https://bugs.launchpad.net/launchpad/+bug/39630 a dup of the other domination bugs ?
[07:14] <_mup_> Bug #39630: Older packages are removed when newer versions FTBFS <boobytrap> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/39630 >
[07:16] <lifeless> also, I'm amazed https://bugs.launchpad.net/launchpad/+bug/40096 hasn't had a look-in
[07:16] <_mup_> Bug #40096: do not show architecture 'all' builds as i386 <confusing-ui> <lp-soyuz> <oem-services> <trivial> <Launchpad itself:Triaged> < https://launchpad.net/bugs/40096 >
[07:16] <nigelb> Bug listing is disabled for beta testers temporarily or something?
[07:17] <wgrant> lifeless: Howso?
[07:17] <wgrant> nigelb: Yes.
[07:17] <nigelb> wgrant: Ah! Thanks.
[07:17] <lifeless> wgrant: howso to which ?
[07:17] <wgrant> lifeless: The latter.
[07:17] <lifeless> wgrant: because it looks like a shallow UI tweak
[07:17] <wgrant> lifeless: ROFLOL
[07:18] <wgrant> How do you propose to determine that? :)
[07:18] <wgrant> We'd need to set a flag on the build at upload-time.
[07:18] <lifeless> wgrant: that its only arch-any ?
[07:19] <wgrant> arch-all, but yes.
[07:19] <lifeless> wgrant: bah, ENOPACKAGINGENOUGH
[07:19] <wgrant> And then what if we now confuse people because normal i386 builds still build arch-all, so you have to wait for them anyway?
[07:20] <lifeless> you mean 'we dispatch arch-all builds to the i386 queue' ?
[07:20] <wgrant> In the case of an arch-any package, the i386 build builds the arch-all bits
[07:20] <wgrant> So it should perhaps be presented as "i386 + all"
[07:21] <wgrant> The UI is not trivial.
[07:21] <wgrant> And it requires model changes too.
[07:21] <wgrant> And it matters to nobody after their first 5 minutes with LP.
[07:21] <lifeless> don't all the builders build -all, and we discard them on other builders? or do we actually call the specific targets..
[07:21] <wgrant> So it's unsurprising it hasn't been tackled :)
[07:21] <wgrant> No.
[07:21] <wgrant> Only nominatedarchindep builds arch-indep.
[07:21] <lifeless> k
[07:22] <lifeless> do you happen to remember the bug about only i386 being allowed to build -all ?
[07:25] <wgrant> Bug #158004
[07:25] <_mup_> Bug #158004: Arch independent packages are only built on i386 <lp-soyuz> <ppa> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/158004 >
[07:25] <wgrant> lifeless: Bug #217427 is also slightly relevant
[07:25] <_mup_> Bug #217427: Please support arbitrary arch/buildd affinity for arch:all builds <feature> <lp-soyuz> <motu> <soyuz-build> <Launchpad itself:Triaged> < https://launchpad.net/bugs/217427 >
[07:28] <wgrant> lifeless: ENOTTRIVIAL
[07:29] <wgrant> Ah
[07:29] <wgrant> You've already fixed it.
[07:29] <wgrant> Thanks.
[07:29] <lifeless> https://bugs.launchpad.net/launchpad/+bug/40096/comments/3
[07:29] <_mup_> Bug #40096: architecture 'all' builds are shown as i386 <lp-soyuz> <oem-services> <Launchpad itself:Triaged> < https://launchpad.net/bugs/40096 >
[07:29] <lifeless> also overhauled the summary
[07:44] <lifeless> wgrant: so what do you think of bug https://bugs.launchpad.net/launchpad/+bug/39630
[07:44] <_mup_> Bug #39630: Older packages are removed when newer versions FTBFS <boobytrap> <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/39630 >
[07:44] <wgrant> lifeless: Half fixed.
[07:44] <lifeless> wgrant: do me a favour and update its metadata?
[08:01] <stub> lifeless: So since you are obviously now taking your holiday that seriously, I worked out an edge case with moving the 'add new tables to replication' code outside of the downtime window that would seize up replication.
[08:03] <stub> A db patch that adds a new table, and adds a constraint to an existing table that references the new table. If data gets added to the existing table referencing the new table before the new table is replicated, we are screwed.
[08:04] <stub> We could say we are fine though, provided we promise that no attempts to access the new table are made before the deploy has fully completed (so no code 'if this table exists, fill in this data')
[08:05] <lifeless> stub: cannot add references outside downtime anyway
[08:06] <lifeless> stub: because they require exclusive locks to update the table mgmt triggers
[08:07] <stub> lifeless: We create the new table and the reference during downtime, we exit the outage, we add the new table to replication (proposed workflow)
[08:07] <lifeless> stub: oh right, different case.
[08:07] <lifeless> stub: uhm, yes, so we know that no code can possibly be using the new column or table
[08:07] <stub> yer, I can't think of how this could bite launchpad.
[08:08] <stub> Unless triggers are involved :-)
[08:08] <lifeless> stub: a default value that causes trouble could be made, but that is something we can pick up in review
[08:08] <stub> A default value will not be referencing a non-existing row in the new table
[08:09] <stub> We could only shoot outselves in the foot with a trigger. Like BugSummary.
[08:10] <stub> So if we go ahead with the change, we would need to ensure that creating tables and any code using the tables (standard or triggers) must land separately and be deployed separately
[08:12] <lifeless> we could define an empty trigger so that the definition is there
[08:13] <lifeless> can update the procedure in NDT, but not the existence/absence of a trigger
[08:15] <stub> So it is an obscure case to catch during review, and very rare. So it is unlikely to happen, but we are likely to miss it if it does happen.
[08:16] <stub> Recovery would not be amusing on the live system.
[08:16] <lifeless> we can shoot ourselves in the foot many different ways
[08:17] <lifeless> uhm
[08:17] <lifeless> I'm ok with the risk, if we document it well and prominently
[08:17] <lifeless> are you?
[08:19] <stub> I'll sleep on it :)
[08:19] <lifeless> I can think of other ways we can mess ourselves up and not have qa catch it :)
[08:19] <lifeless> (like e.g. bad indices or queries)
[08:21] <wgrant> stub: We're in huge trouble if we're touching a not-yet-replicated table anyway, aren't we?
[08:21] <stub> lifeless: That doesn't involve taking the system down, manually copying data to slaves and massaging replication events
[08:22] <wgrant> Because the new data won't get replicated...
[08:22] <stub> wgrant: Only if it is referenced by a replicated table, which is this case
[08:22] <wgrant> (unless you manually clone it across first, which seems silly given it never needs to happen)
[08:22] <wgrant> stub: Does adding the table to replication also replicate the existing data?
[08:22] <wgrant> I assumed it would not.
[08:22] <stub> It will
[08:23] <wgrant> Ah
[08:23] <wgrant> That seems excessive.
[08:23] <wgrant> But perhaps.
[08:23] <stub> The trick is that adding a table to replication won't happen until pending events have been processed, and in this case pending events won't be processed because the slave is attempting to insert data its constraints won't let it.
[08:23] <wgrant> Right.
[08:24] <lifeless> stub: I'm not sure the constraints are checked on the slave
[08:24] <lifeless> stub: because slony disables all the triggers on the slaves, and constraints are triggers
[08:24] <stub> Hmm... easiest recovery would actually be to disable the foreign key constraints on the slave, add tables and wait for catch up, and reenable foreign key constraints.
[08:24] <lifeless> stub: I suggest trying it locally and seeing if it actually borks
[08:25] <wgrant> I was thinking I'd seen constraint violations on a slave locally.
[08:25] <stub> foreign key constraints are not disabled IIRC.
[08:25] <wgrant> But that's only from a unique index.
[08:25] <wgrant> So it's possible they're disabled.
[08:25] <lifeless> unique indices will stay live, thats index structure
[08:25] <wgrant> Exactly.
[08:25] <wgrant> I was just thinking I had evidence that they were left enabled, then realised that it wasnt.
[08:28] <stub> Are continual WARNINGs that we need to fix HIGH or CRITICAL?
[08:28] <stub> Bug #906193
[08:28] <_mup_> Bug #906193: BugHeatUpdater never completes <Launchpad itself:New> < https://launchpad.net/bugs/906193 >
[08:29] <lifeless> arguably critical - we have no way to handle/detect 'garbo job X is not getting to run' and thats an operational issue
[08:29] <lifeless> being able to detect that probably involves fixing the 'blah was interrupted' warnings
[08:30] <lifeless> poolie: http://pad.lv/OOPS-78b4ff3d44483f940b04d6a2ef5c321e 404's
[08:30] <stub> Only emailing WARNING and above, we could easily enough tie that into alerts
[08:31] <lifeless> sorry, not enough context to grok that
[08:31] <lifeless> wgrant: just got an ajax oops link to work \o/
[08:31] <lifeless> wgrant: OTOH https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-78b4ff3d44483f940b04d6a2ef5c321e
[08:32] <stub> We (should be) running scripts logging everything to file, but only emitting WARNINGS and above to stdout.  We could tie that into nagios.
[08:32] <stub> Or just grep the log files for WARNING|ERROR|CRITICAL
[08:33] <poolie> hi lifeless, ok
[08:33] <lifeless> we recently overhauled all the cronscripts in prod to log everything to a file
[08:33] <stub> If todays log and yesterdays log contain warnings or above, flag the service as failed.
[08:33] <lifeless> high logging levels generate OOPSes already
[08:35] <lifeless> stub: so being interrupted isn't a problem per se; its being starved for days that matters, we need something with a longer view of things, or something.
[08:36] <stub> Yes, so a day or two running with warnings should be enough to prompt someone to take a look and either file a bug or get the alert disabled for a few days until transient issues have been cleared.
[08:37] <stub> We could also happily enough log each garbo job to ScriptActivity
[08:37] <stub> each task I mean
[08:37] <lifeless> stub: indeed, though after the rewrite please :)
[08:38] <lifeless> stub: does a WARNING log item cause an OOPS or is it ERROR and above ?
[08:38] <stub> Sure. Log file grepping -> nagios could be done right now if we want
[08:38] <stub> It should be WARNING and above.
[08:38] <stub> WARNING should generate an informative OOPS, ERROR and above a standard OOPS.
[08:39] <lifeless> ok, so we're getting warnings nows. I agree with your analysis FWIW - just leave the reporting as is and fix the issues that show up.
[08:39] <lifeless> stub: 'informative' doesn't exist anymore as a flag, got pruned in the redo.
[08:40] <lifeless> wgrant: could you do me a favour and expand on the rather brief description in https://bugs.launchpad.net/launchpad/+bug/78466 ?
[08:40] <_mup_> Bug #78466: FTPArchiveHandler doesn't support allowed_suites <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/78466 >
[08:44] <wgrant> lifeless: It's not clear?
[08:44] <lifeless> wgrant: its opaque as a frog sitting in ink
[08:47] <lifeless> cjwatson: are you perhaps around ? could use your input on bug 87012
[08:47] <_mup_> Bug #87012: Cannot start developing next ubuntu release before the prior one is released <lp-soyuz> <soyuz-publish> <Launchpad itself:Triaged> < https://launchpad.net/bugs/87012 >
[08:47] <poolie> lifeless, fixed, i think, except i'm now having network trouble
[08:47] <lifeless> poolie: thanks!
[08:49] <wgrant> lifeless: I think it's probably just about obsolete.
[08:49] <wgrant> Not much uses allowed_suites directly any more.
[08:49] <wgrant> They just check dirtiness, which a-f already does.
[08:53] <lifeless> wgrant: so, invalid?
[08:54] <wgrant> I think so.
[08:54] <wgrant> In a disaster-recovery situation we're thoroughly screwed anyway, so it doesn't really matter.
[08:54] <lifeless> wgrant: doit :)
[08:55] <wgrant> Is done.
[08:55] <lifeless> stub: is https://bugs.launchpad.net/launchpad/+bug/90809 in progress ?
[08:55] <_mup_> Bug #90809: Launchpad should run with standard_conforming_strings=on in postgresql.conf <lp-foundations> <Launchpad itself:Triaged by stub> < https://launchpad.net/bugs/90809 >
[08:56] <lifeless> stub: actually, what I mean is
[08:56] <lifeless> stub: did we do this, or are we in deep dodo with respect to it ?
[08:57] <adeuring> good morning
[09:01] <wgrant> lifeless: "This is a broken-by-design issue: its not safe to run two different
[09:01] <wgrant> LPCONFIG's from the same source instance, ever."
[09:01] <wgrant> We do that all the time.
[09:01] <wgrant> Mostly on loganberry/ackee
[09:01] <lifeless> then we're freaking lucky that its not breaking all over the shop
[09:01] <wgrant> But also cocoplum/germanium/wildcherry/sourcherry
[09:01] <wgrant> How?
[09:01] <lifeless> and we have to stop doing that
[09:02] <wgrant> What state are they going to write out?
[09:02] <lifeless> the overrides zcml
[09:02] <StevenK> bigjools: O hai!
[09:02] <lifeless> wgrant: its presumably identical for us now, but wasn't in the past; if its not written atomically, we must be freakishly lucky
[09:04] <StevenK> bigjools: So, we have two issues this weekend and today, both of them due to work your squad has done. buildd-manager broke badly on the weekend due to bugs 905853, 905855 and 906079; and germanium was reverted since the poppy-sftp breakage caused germanium to hit a load of 1700.
[09:04] <wgrant> lifeless: Hm, that is remarkably, remarkably evil.
[09:04] <wgrant> And has very little justification.
[09:04] <lifeless> wgrant: I went blind when I found it ~ a year ago
[09:04] <lifeless> wgrant: working on test parallelisation w/out lxc
[09:04] <wgrant> StevenK, bigjools: Note that poppy-sftp survived until just a few hours ago.
[09:05] <poolie> lifeless, confirmed that oops url does work
[09:05] <wgrant> lifeless: I've actually managed never to run into that before.
[09:05] <poolie> i started turning ec2test into a juju charm
[09:05] <poolie> ...
[09:05] <poolie> kind of ok
[09:05] <lifeless> poolie: charm or charms ?
[09:05] <wgrant> poolie: I considered that a couple of weeks back.
[09:06] <wgrant> But decided that juju scared me.
[09:06] <poolie> lifeless, one charm at first
[09:06] <poolie> so it's kinda monolithic and a bit of a degenerate case
[09:13] <StevenK> frankban: Hi. I reverted your milestonetag revision on db-devel since it broke buildbot.
[09:15] <lifeless> bigjools: hey, so bug https://bugs.launchpad.net/launchpad/+bug/117557 - doesn't that cause an OOPS ?
[09:15] <_mup_> Bug #117557: Nascent Upload code doesn't check properly for bad distroseries <lp-soyuz> <soyuz-upload> <tech-debt> <Launchpad itself:Triaged> < https://launchpad.net/bugs/117557 >
[09:15] <frankban> StevenK: ok, thanks
[09:15] <lifeless> bigjools: you say it won't, but I don't understand how it wouldn't.
[09:17] <bigjools> lifeless: the exception is caught and translated to an error returned to the user
[09:18] <stub> lifeless: We are not in deep doodoo - I'll be dealing with standard conforming strings settings with the PG upgrade. IIRC the things that were quoting oldskool were psycopg2 and maybe a few items in comments.sql. Doesn't really matter as the old format is still supported.
[09:19] <lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/118870 - you comment that its a strange view, but it looks ok to me.
[09:19] <_mup_> Bug #118870: $sourcepackage/+changelog only shows one entry per suite (distroseries-pocket) <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/118870 >
[09:20] <poolie> wgrant, so as an initial wrapper around starting an instance it seems ok
[09:20] <poolie> using something like this to start ppa builders might be more interesting
[09:21] <wgrant> lifeless: The strange mangled view I speak of was later filed as bug #144620
[09:21] <_mup_> Bug #144620: Some displayed sourcepackagerelease changes files don't have attribution <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/144620 >
[09:21] <lifeless> stub: what about https://bugs.launchpad.net/launchpad/+bug/120404 ? - do we still need to do this audit ?
[09:21] <_mup_> Bug #120404: Database permission audit <lp-foundations> <Launchpad itself:Triaged> < https://launchpad.net/bugs/120404 >
[09:22] <wgrant> lifeless: It's not relevant to the bug beyond confirming it wasn't a regression, anyway.
[09:23] <stub> lifeless: I'd like to, yes
[09:23] <stub> lifeless: Unless we go ahead and drop permissions entirely ;)
[09:25] <lifeless> wgrant: I'm not sure that the bug is actionable then, because the changelog looks ok to me
[09:25] <lifeless> stub: indeed, and thats a curly discussion :)
[09:26] <wgrant> lifeless: It's indeed no longer as it was described in the bug.
[09:27] <wgrant> lifeless: It's not ideal, but it's not quite so bad.
[09:27] <wgrant> It seems to show the changelog_entry from every publication in the series.
[09:27] <wgrant> Rather than just one per pocket.
[09:27] <lifeless> wgrant: can you update the bug?
[09:28] <wgrant> Blah, OK.
[09:30] <frankban> StevenK: can you confirm the problem is here? https://lpbuildbot.canonical.com/builders/lucid_db_lp/builds/1596/steps/shell_6/logs/stdio
[09:32] <StevenK> frankban: The summary is much better for that sort of thing
[09:32] <StevenK> frankban: But the first error is "ProgrammingError: permission denied for relation milestonetag" which puts the smoking gun firmly in your hand.
[09:34] <frankban> StevenK: ok, seen
[09:34] <wgrant> lifeless: Hah
[09:35] <lifeless> ?
[09:39] <wgrant> I think archiveremovalredesign fixed it.
[09:39] <wgrant> But annotate keeps misleading me.
[09:42] <wgrant> Ah, no, was accidentally fixed in r5613
[09:42] <wgrant> [r=barry] Fixing bug #179028 (+files does not work for removed
[09:42] <wgrant> sources). Now SourcePackage class is able to traverse to source
[09:42] <wgrant> package releases removed from disk.
[09:42] <_mup_> Bug #179028: +files doesn't work for removed SPRs <lp-soyuz> <Launchpad itself:Fix Released by cprov> < https://launchpad.net/bugs/179028 >
[09:46] <lifeless> wgrant: https://bugs.launchpad.net/launchpad/+bug/144620 is stale too
[09:46] <_mup_> Bug #144620: Some displayed sourcepackagerelease changes files don't have attribution <lp-soyuz> <Launchpad itself:Triaged> < https://launchpad.net/bugs/144620 >
[09:47] <wgrant> lifeless: Fixed
[09:55] <bigjools> wgrant: only poppy on germanium went bong?
[09:59] <wgrant> bigjools: Yes.
[09:59] <wgrant> Same as last time.
[09:59] <wgrant> I assume it's load-based.
[09:59] <wgrant> We should watch cocoplum's carefully, but I don't want to revert it unless we have to, because of the publisher optimisations.
[10:00] <bigjools> yeah
[10:00] <bigjools> gah it's that weird logging thing again
[10:00] <wgrant> Really?
[10:00] <wgrant> The top of the latest log has that.
[10:00] <wgrant> But only because rotation isn't happening.
[10:01] <bigjools> bWTF
[10:01] <bigjools> why
[10:01] <wgrant> Because logrotate isn't set up, I assume :)
[10:01] <bigjools> oh it has 2 log files, one is old style one is new
[10:01] <bigjools> nothing to do with that
[10:02] <bigjools> there's some recursive logging going on
[10:02] <bigjools> lifeless: need your halp
[10:02] <bigjools> https://bugs.launchpad.net/launchpad/+bug/906211
[10:02] <_mup_> Bug #906211: poppy died in a pulp of its own swappiness <Launchpad itself:Triaged> < https://launchpad.net/bugs/906211 >
[10:02] <bigjools> the recent changes made no difference
[10:03] <wgrant> bigjools: Huh, didn't notice that recent bit.
[10:03] <wgrant> Oh.
[10:03] <wgrant> bigjools: That's *after* the revert.
[10:03] <wgrant> Of course your reverted changes didn't help :)
[10:04] <bigjools> ah
[10:04] <bigjools> what time was it reverted?
[10:05] <wgrant> An hour after the >1000 load was reported.
[10:05] <wgrant> For that hour it was running the new code, but restarted.
[10:05] <wgrant> 07:42Z the rollback completed
[10:05] <wgrant> This should be on LPS, hopefully.
[10:06] <bigjools> geez, log looks perfectly normal
[10:06] <wgrant> Look towards the end.
[10:06] <wgrant> During the deathspiral.
[10:06] <bigjools> hmmm
[10:07] <bigjools> what bit?
[10:07] <wgrant> There's very frequent DTPFactory logs or whatever it is.
[10:07]  * wgrant relooks.
[10:07] <wgrant> Oh, blah, has it rotated the logs away again.
[10:07] <bigjools> it's there
[10:07] <bigjools> poppy-sftp.log.1006
[10:08] <wgrant> There's also one from around the outage in ~wgrant/poppy-sftp.log
[10:08] <wgrant> Oops
[10:08] <wgrant> just ran ls in lp_upload
[10:08] <bigjools> bad idea
[10:08] <lifeless> bigjools: hi
[10:09] <bigjools> lifeless: hi - sorry was looking at wrong log, but still could do with your help!
[10:09] <wgrant> bigjools: Look at the last couple of seconds before the world ended.
[10:09] <wgrant> eg. 06:18:48
[10:09] <bigjools> yeah
[10:09] <wgrant> Dozens of lines of DTPFactory crap.
[10:10] <bigjools> hmmm
[10:10] <bigjools> I see a lot of entries from hyperair
[10:11] <wgrant> That's true, but from a different IP.
[10:11] <wgrant> The initial symptom was that SSH auth failed.
[10:11] <wgrant> But that took a day or so to appear.
[10:11] <bigjools> that has been happening on branches too
[10:11] <wgrant> Hmmmmm, that's true.
[10:11] <wgrant> Very occasionally.
[10:11] <wgrant> But entirely unexplained.
[10:12] <bigjools> well some guy filed a question about it, was consistentely failing for him
[10:12]  * bigjools has cold fingers and can't type today
[10:12] <wgrant> You mean the question on like Friday about it?
[10:12] <wgrant> I thought that was poppy.
[10:13] <lifeless> bigjools: do you think https://bugs.launchpad.net/launchpad/+bug/39630 is high still?
[10:13] <bigjools> ah yes it was poppy
[10:14] <bigjools> lifeless: yes
[10:14] <wgrant> However, I checked when I saw that and it was working at that point.
[10:14] <wgrant> So it broke then fixed itself :)
[10:15] <bigjools> this looks like a DoS to me
[10:15] <wgrant> It does.
[10:15] <wgrant> But I'm more likely to blame poppy.
[10:15] <bigjools> right - we need a connection limit
[10:16] <bigjools> in the meantime, let's have a word with mr hyperair
[10:16] <lifeless> doing the RT for haproxy on the uploaders will get you that for free
[10:16] <lifeless> (just saying)
[10:16] <wgrant> Is that 1962 a port number?
[10:16] <wgrant> I'm not sure how that logging works.
[10:16] <lifeless> bigjools: so, you're set, you don't need me ?
[10:16] <bigjools> lifeless: yes, thanks and sorry to bother
[10:16] <lifeless> hey no worries
[10:16] <lifeless> I chose to be on IRC :P
[10:16] <lifeless> (and to re-triage 75 high bugs...)
[10:16] <bigjools> wgrant: so I think we can revert the revert
[10:17] <bigjools> given that the old one has known issues
[10:17] <wgrant> bigjools: There's still the SFTP issue.
[10:17] <bigjools> was that happening before the new code rollout?
[10:17] <wgrant> No.
[10:17] <bigjools> the sftp code has not been touched
[10:17] <bigjools> hard to see how it was broken
[10:17] <wgrant> It's clearly broken it somehow.
[10:18] <wgrant> NFI how, though :/
[10:18] <lifeless> bigjools: SSHServer was touched wasn't it ?
[10:18] <wgrant> It's happened twice now.
[10:18] <lifeless> bigjools: which is the backing for sftp
[10:18] <bigjools> lifeless: yes - but only logging
[10:18] <lifeless> agreed; just being pedantic
[10:18] <bigjools> lifeless: can you see how that might cause auth fails?
[10:18] <bigjools> pedanticism is good here
[10:19] <lifeless> bigjools: only thing I can think of is something logging triggering a barf which then oops, or something.
[10:19] <wgrant> Remember that auth goes via XMLRPC
[10:19] <lifeless> did we see oopses for the auth errors?
[10:19] <wgrant> So it's async.
[10:19] <bigjools> there's things like "unauthorized login: unable to get avatar id"
[10:19] <bigjools> wgrant: ?!
[10:19] <bigjools> lifeless: AHA
[10:20] <bigjools> there's a timeout failure
[10:20] <bigjools> right at the same time as the auth starts
[10:20] <wgrant> timestamp?
[10:20]  * bigjools pastes
[10:20] <bigjools> https://pastebin.canonical.com/57356/
[10:21] <wgrant> Ah, yes.
[10:21] <wgrant> So there is.
[10:21] <bigjools> seems to happen on all attempts
[10:21] <wgrant> I was looking at the wrong hour.
[10:21] <wgrant> Yeah
[10:21] <wgrant> So it sits there for 30s.
[10:21] <wgrant> And then returns a TimeoutError
[10:21] <bigjools> and as usual Twisted is unhelpful with no traceback
[10:21] <lifeless> firewall issue?
[10:22] <wgrant> No.
[10:22] <wgrant> Since a restart fixes it.
[10:22] <bigjools> I think someone has got frustrated and set up a loop to keep retrying
[10:22] <wgrant> Interesting.
[10:22] <wgrant> bigjools: 2011-12-16 10:19:40
[10:22] <lifeless> nn
[10:22] <wgrant> 4 seconds before the first occurrence of the TimeoutError
[10:22] <wgrant> There's lots of DTPFactory again.
[10:22] <bigjools> nn lifeless
[10:23] <wgrant> From that same IP.
[10:23] <wgrant> Hmm
[10:23] <wgrant> Interesting.
[10:24] <wgrant> My tests were 3 minutes before that.
[10:25] <wgrant> However, I did those same tests on cocoplum and it survived.
[10:31] <bigjools> wgrant: I can see others connecting to SFTP and validating ok
[10:31] <wgrant> Where?
[10:32] <bigjools> 02:22:36+0000
[10:33] <bigjools> INFO:poppy-sftp.Hooks:Post-processing finished
[10:33] <bigjools> oh it's ftp
[10:33] <bigjools> ignore me
[10:33] <wgrant> heh
[10:37] <cjwatson> lifeless: 87012> commented
[10:37] <bigjools> wgrant: so I uploaded to cocoplum over SFTP ok
[10:38] <wgrant> bigjools: Yes, cocoplum continues to work.
[10:38] <wgrant> It's not had to be restarted once.
[10:38] <bigjools> germanium is not happy
[10:38] <wgrant> Isn't it fine now it's reverted?
[10:38] <bigjools> and coupled with the other issue where it needs restarting every 6 days ...
[10:38] <bigjools> ignoring the revert
[10:38] <wgrant> The new code needs restarting every 6 *hours*.
[10:38] <wgrant> On germanium.
[10:39] <StevenK> Perhaps germanium is just getting hit so much harder than cocobanana.
[10:39] <bigjools> nice,p nearly an hour to do a soyuz logsync
[10:39] <wgrant> That's been my assumption for a long time.
[10:39] <wgrant> bigjools: It's because germanium is in its logdeathspiral.
[10:40] <wgrant> So it's redownloading 30GB of logs every time.
[10:40] <StevenK> germanium is hellishly overloaded. News at 11.
[10:40] <bigjools> wgrant: so, right, the first failure was just after your tests
[10:40] <wgrant> It's not overloaded.
[10:40] <wgrant> Our code is just shit.
[10:40] <wgrant> :)
[10:40] <wgrant> bigjools: Yep
[10:40] <wgrant> 3 minutes later or so.
[10:40] <StevenK> Sadly, that isn't news.
[10:40] <wgrant> bigjools: Very odd.
[10:40] <wgrant> bigjools: But I did very similar things to cocoplum several times over the weekend, AFAICR.
[10:40] <bigjools> I'd prefer not to judge until we know everything
[10:40] <wgrant> And it continues to live...
[10:58]  * gmb wishes pushing a branch up to a local codehosting instance didn't take so flaming long.
[10:58]  * gmb finds something else to do in the meantime
[11:13] <wgrant> gmb: What sort of branch?
[11:13] <wgrant> It's normally lightning fast for me, unless you get hit with big 2a fetch slowness.
[11:21] <gmb> wgrant, A bigun. I'm working on bug 808930.
[11:21] <gmb> My complaint should have read "_this_ branch"
[11:22] <wgrant> gmb: If you want to cheat, you can just copy it to the right place under /var/tmp/bazaar.launchpad.dev/mirrors/
[11:22] <gmb> wgrant, Wouldn't I then need to manually create the job(s) for the branch scanner?
[11:23] <gmb> Or is there a cheat for that, too?
[11:23] <wgrant> gmb: You can then do a no-op push to create the job.
[11:24] <gmb> Ah!
[11:24] <gmb> Yes, of course.
[11:24] <gmb> wgrant, You wise, wise monkey you. Thanks.
[11:24] <wgrant> A no-op push may not work -- but if not, just push -r-2 or so to ensure the head changes.
[11:24] <gmb> Understood.
[11:26] <wgrant> gmb: You may want to talk to jelmer about this.
[11:26] <wgrant> Since he has a branch to rework bits of the scanner.
[11:26] <wgrant> Relevant bits, I assume -- it made it several times slower.
[11:26] <gmb> wgrant, We talked last week about it. This bug looks very much like death by SQL.
[11:26] <wgrant> Ah
[11:26] <wgrant> Didn't know that.
[11:27] <gmb> Lots of SELECTS and INSERTS in RevisionSet.new()
[11:27] <wgrant> Ah
[11:27] <wgrant> In that case an initial scan might not trigger it.
[11:27] <wgrant> The initial scan is a special-case.
[11:27] <wgrant> Which is much more efficient in the general case.
[11:27] <gmb> Ah.
[11:28] <wgrant> But it depends on the branch.
[11:29] <gmb> Hmm.
[11:30] <gmb> Well, I'll get this one into codehosting .dev and see what happens. Having to re-do a bunch of stuff since my Oneiric VM decided to go wonky this morning; I had to restore from a snapshot.
[11:31] <wgrant> gmb: You may be able to use the example in comment #5 to reproduce the partial scan slowness.
[11:31] <wgrant> (it's even one of yours)
[11:31] <wgrant> It's still broken on prod.
[11:32] <gmb> Ah yes. I hadn't scanned that far down the bug :). I was working with info from benji, since I've taken this over from him.
[11:37] <wgrant> There may well be two reasonably separate bugs here.
[11:40] <gmb> wgrant, You mean death by SQL + the bug for which jelmer has been making optimisations?
[11:40] <gmb> Or something else
[11:40] <gmb> ?
[11:41] <wgrant> The initial and partial scans may be separate code bugs, but both are covered in that single report.
[11:41] <gmb> Ah, ISWYM.
[11:41] <gmb> Good point.
[11:42] <wgrant> Because we know that scanning a Launchpad branch from scratch works.
[11:42] <wgrant> But bringing up to date a Launchpad branch that is a year old appears not to.
[11:43] <gmb> Indeed.
[11:43] <wgrant> Then there's another separate issue, where inserting a whole lot of new Revisions (not BranchRevisions) takes too long.
[11:43] <wgrant> Mostly occuring with new massive svn/git imports.
[11:43] <wgrant> So there's probably at least three bugs here, conflated in that one report :)
[11:43] <gmb> Oh, joy.
[11:44] <wgrant> I aim to please.
[11:44] <gmb> :)
[11:44] <gmb> I'll try and explore the first two, at least. Not sure I'll have time to tackle the third problem.
[11:45] <wgrant> You may have little choice.
[11:45] <gmb> Sssh
[11:45] <wgrant> Scanning a massive branch into a dev instance may well fail with issue #3, because all the Revisions are new.
[11:45] <gmb> Don't want to think about that yet...
[11:45] <gmb> Actually, that would fit well with what Benji's observed so far, AFAICT.
[11:46] <gmb> (I've got lots of how-to-follow-his-tracks info but no actual data to analyse)
[11:46] <wgrant> Yeah
[11:57]  * gmb lunches
[12:54] <stub> We should fix that. No need to create all the revisions in one transaction.
[14:27] <rick_h__> benji: have a sec to review? https://code.launchpad.net/~rharding/launchpad/batch_nav_900900/+merge/85935
[14:27] <rick_h__> or sorry, I'll bug jcsackett since he just arrived and originally peeked at it
[14:27] <rick_h__> so benji nvm and jcsackett can you look at https://code.launchpad.net/~rharding/launchpad/batch_nav_900900/+merge/85935 again when you get settled?
[14:28] <jcsackett> rick_h__: happy to look again. :-)
[14:28] <rick_h__> jcsackett: ty much
[14:28] <jcsackett> same branch i looked at the other day?
[14:28] <benji> k
[14:28] <rick_h__> jcsackett: yep, just modified the test for the lower batch
[14:29] <rick_h__> jcsackett: missed the test over in lp/app when the change was in canonical/launchpad so thanks for catching that and saving me a test run
[14:29] <jcsackett> rick_h__: your welcome. with the change mod, this is r=me. :-)
[14:30] <jcsackett> s/change mod/test change/
[14:30] <rick_h__> jcsackett: awesome, thanks
[14:30]  * jcsackett goes to drink his coffee so he typoes less.
[14:52] <rick_h__> has anyone run into this ec2 land error lately? https://pastebin.canonical.com/57375/
[14:52] <rick_h__> it worked over the weekend, so wondering if my package updates from last night have caused me pain now
[15:13] <jml> is it possible to get the binary package urls from a BinaryPackagePublishingHistory object?
[15:14] <jml> looking in publishing.py, the answer is no.
[15:14] <bigjools> jml: urls to what exactly?
[15:15] <jml> bigjools: the debs
[15:15] <bigjools> jml: ah, no, but it's trivial to contruct a path to the archive
[15:15] <jml> bigjools: what SourcePackagePublishingHistory.binaryFileUrls gets you the URLs for, except from one BPPH rather than the BPPH's associated with an SPPH
[15:16] <bigjools> that's returning librarian URLs
[15:16] <bigjools> I think it was done for the security guys
[15:17] <jml> so it's not actually recommended to download files from the URLs that Launchpad provides?
[15:19] <bigjools> either is fine
[15:19] <bigjools> but I tell people to use the archive url, because it's less likely to be unavailable
[15:20] <jml> bigjools: and the way that's trivially constructed is to inspect the distro, component and binary package name to make a URL to the deb in the pool area, right?
[15:21] <bigjools> that sort of thing
[15:21] <bigjools> but if you want to get it from the API, file a bug :)
[15:21] <jml> well, I might submit a patch.
[15:21] <bigjools> even betterer
[15:23] <cjwatson> benji: could you have a look at https://code.launchpad.net/~cjwatson/launchpad/soyuz-test-mail-ordering/+merge/85979 ?  Just a test fix
[15:23] <benji> cjwatson: sure
[15:24] <bigjools> jml: actually you need to return a proxied librarian file like the other bits of code, in case it's a private archive
[15:24] <jml> bigjools: yeah. I was planning on factoring out what's in SPPH, and have SPPH delegate to BPPH to figure out binary urls.
[15:25] <bigjools> jml: ok, careful it doesn't end up potato programming
[15:25] <jml> bigjools: will do.
[15:28] <jcsackett> sinzui: do you know if wallyworld is actually working on bug 885524?
[15:28]  * sinzui looks
[15:29] <sinzui> jcsackett, not directly
[15:29] <jcsackett> sinzui: yeah, it looks like he's working on a related bug. so i'll leave that alone to avoid stomping on toes.
[15:30] <sinzui> jcsackett, wallyworld__is working on bug 904295 which is a pre-req and may actually fix the issue
[15:30] <sinzui> jcsackett, lets mumble about the remaining cards. I have to pick one too
[15:31] <jcsackett> sinzui: dig. be on mumble in just a moment.
[15:34] <benji> cjwatson: the branch looks good
[15:39] <rick_h__> adeuring: deryck  https://pastebin.canonical.com/57375/
[15:40] <deryck> rick_h__, we lost you.
[15:43] <adeuring> deryck: https://code.launchpad.net/~adeuring/launchpad/bug-903359/+merge/86244
[15:44] <cjwatson> benji: lovely, thanks.  I have no PQM access; do you think you could send it to EC2 for me?
[15:44] <benji> cjwatson: sure
[15:44] <cjwatson> ta
[15:54] <benji> cjwatson: EC2 has been engaged: http://ec2-184-72-167-164.compute-1.amazonaws.com/
[15:55] <deryck> rick_h__, did your bazaar.conf change by any chance?  i.e. does it still exist in $HOME/.bazaar ?
[15:56] <rick_h__> deryck: no, it's still there and seems normal
[15:56] <rick_h__> deryck: I did have to make some changes to my locations.conf to get a manual land in pqm over the weekend
[15:56] <rick_h__> but that was just adding some email settings. Removing them again doesn't help me
[15:57] <deryck> rick_h__, ok, it seems to be choking over email and signing the submission some how.
[15:57] <rick_h__> deryck: right
[15:58] <deryck> gary_poster, are you around?
[15:58] <gary_poster> deryck, hi
[15:58] <deryck> gary_poster, hi.  Trying to help rick_h__ fix this error: https://pastebin.canonical.com/57375/  and I'm really not sure what's up. Have you seen that before?
[16:00] <gary_poster> deryck, no.  that's a curious one.  The bzr-2.5 makes me suspicious though
[16:00] <gary_poster> I wonder if 2.4 will be better
[16:00] <deryck> ah yeah, missed that.
[16:00] <gary_poster> though I guess that means we are all using 2.5 though?
[16:00] <deryck> and it's using the egg version not his local bzr.
[16:00] <gary_poster> right
[16:00] <deryck> I didn't realize we did that.
[16:01] <gary_poster> me either deryck :-P  or at least didn't remember, I dunno :-)
[16:01] <deryck> heh
[16:01] <rick_h__> my local bzr is  2.5.0~bzr6383.6136~ppa4002~oneiric1
[16:01] <deryck> seems kind of wrong to me, but I'm sure someone had a good reason to.
[16:02] <deryck> but I guess if we're all on dailies we'd be on that.
[16:02] <gary_poster> rick_h__, I see this on my oneiric: $ bzr --version
[16:02] <gary_poster> Bazaar (bzr) 2.4.2
[16:02] <rick_h__> gary_poster: yea, I did an upgrade this morning and that seems to have blown me up
[16:02] <gary_poster> ah ha
[16:02] <rick_h__> gary_poster: I saw bzrlib and launchpadlib updates
[16:02] <rick_h__> gary_poster: yea, just call me the canary in the coal mine I suppose
[16:03] <gary_poster> rick_h__, heh :-)  so we have two options I guess
[16:03] <deryck> yeah, I'm still on 2.4.2 as well, though I thought I ran the daily.  Guess I killed it at some point.
[16:03] <gary_poster> 1) try going back to 2.4.2
[16:03] <gary_poster> 2) dig in and fix it.
[16:03] <gary_poster> :-P
[16:04] <gary_poster> I vote for option 1 but somebody has to pay the price
[16:04] <gary_poster> it would be nice if that person had some bzr experience though
[16:05] <gary_poster> rick_h__, can you try option 1?
[16:05] <rick_h__> gary_poster: yea looking into it. Set a pin and upgrade again?
[16:06] <gary_poster> rick_h__, yeah.  (I'd do it in aptitude but that's because that's what I'm comfortable with)
[16:06] <rick_h__> gary_poster: ok, looking. Been years since I've pinned a package
[16:07] <gary_poster> The thing that will need to be fixed will be either pqm or bzr (or both conceivably)
[16:07] <gary_poster> rick_h__, do it in aptitude then. It's easy :-)
[16:07] <gary_poster> Maybe synaptic too
[16:07] <rick_h__> gary_poster: installed
[16:07] <gary_poster> cool
[16:09] <rick_h__> gary_poster: so the 'F' command?
[16:10] <gary_poster> rick_h__, this is what I do, for a temporary test change:
[16:10] <gary_poster> find the package in aptitude
[16:10] <gary_poster> Press enter
[16:10] <gary_poster> scroll to the bottom
[16:10] <gary_poster> you should see the different options to install
[16:10] <gary_poster> with an i on 2.5 presumably
[16:10] <rick_h__> gary_poster: yea, I see the p    2.4.2
[16:11] <gary_poster> right
[16:11] <gary_poster> press "+" on that
[16:11] <gary_poster> that will be a temporary install I think.  If not,
[16:11] <gary_poster> yeah, no that should do it.  If you want to keep it, use "=" on the package in the future
[16:12] <rick_h__> ugh, looking, wants to remove 134 package
[16:12] <gary_poster> (or remove the bzr daily ppa :-) )
[16:12] <gary_poster> :-(
[16:12] <deryck> rick_h__, also, if you're on the daily, you can remove that from sources and go back to the 2.4.2 in O.
[16:12] <gary_poster> just do downgrade?
[16:12] <gary_poster> to
[16:12] <deryck> sorry gary_poster just said what I said :)
[16:13] <rick_h__> gary_poster: well I hit + on it and it says "Suggest 134 removals"
[16:13] <rick_h__> gary_poster: not sure if it's going to carry out that suggestion though I suppose
[16:13] <gary_poster> deryck, I don't think that alone will downgrade (because of doing something similar recently), you have to explicitly downgrade or else the packages hang around
[16:13] <rick_h__> yea, if I remove the ppa I'd still have to remove/reinstall or downgrade
[16:14] <gary_poster> rick_h__, ok, if it is not working easily then this is a blindman leading...you take a list at the suggestions and see what you think, or we can find someone else to talk to :-)
[16:14] <gary_poster> I mean, I am a blind man leading :-)
[16:14] <rick_h__> gary_poster: going to see if I can remove the ppa and see what it wants to do to remove bzr
[16:14] <rick_h__> then reinstall
[16:14] <gary_poster> ok
[16:15] <gary_poster> hopefully this is actually the problem :-P
[16:15] <gary_poster> it seems like it though
[16:15] <gary_poster> I need to head out soon
[16:15] <rick_h__> gary_poster: ok thanks
[16:15] <gary_poster> my wife and baby are calling me to an early lunch :-)
[16:15] <deryck> adeuring, r=me on that change.  looks great!  Thanks.
[16:15] <adeuring> deryck: thanks!
[16:16] <deryck> rick_h__, gary_poster -- yeah, I'm sorry, I meant drop the daily from sources and uninstall/reinstall.  That's how I usually do it. Maybe that's the clunky way.
[16:17]  * deryck has weak apt foo
[16:18] <rick_h__> deryck: yea, that's worked for removing/adding
[16:19] <rick_h__> deryck: working on updating devel/make clean/etc to test out
[16:22] <gary_poster> rick_h__, I suggest shooting a note to launchpad-dev about what you encountered so that others know the problem and can investigate
[16:22] <gary_poster> assuming this fixes it
[16:22] <rick_h__> gary_poster: will do
[16:23] <gary_poster> thx
[16:25] <rick_h__> deryck: gary_poster landing is processing thanks for the notice of the 2.5 and help with that
[16:33] <bigjools> deryck: apt-get package=<release name>
[16:35] <deryck> bigjools, ah, thank you.
[16:35] <bigjools> de nada
[16:37] <deryck> adeuring, I reproduced the double count locally, saw it doing: https://bugs.launchpad.dev/ubuntu/++profile++show/+bugs
[16:37] <deryck> adeuring, and this fixes it:  http://pastebin.ubuntu.com/775460/
[16:37]  * adeuring is looking
[16:37] <deryck> I had to build up my local data to get a couple batches as we discussed this morning.
[16:37] <deryck> without the extra batch we don't even do a single count because we don't need lastBatch.
[16:40] <adeuring> deryck: very nice fix!
[16:40] <deryck> adeuring, thanks!  seems simple enough too.
[16:40] <adeuring> yeah
[16:40] <deryck> I'll fix my branch to use this and update the MP.
[16:41] <deryck> adeuring, I guess we can drop all the stuff to avoid lastBatch now.  I did the pastebin patch in devel, where the older use of lastBatch still exists.
[16:41] <adeuring> deryck: right, the old patch should be obsolete
[16:42] <deryck> adeuring, ok, cool, that was my thinking too.
[17:39] <deryck> adeuring, do you care to weigh in or review the updated https://code.launchpad.net/~deryck/launchpad/avoid-extra-buglist-count-901124/+merge/85940 ?
[17:40] <adeuring> deryck: sure
[17:40] <adeuring> i'll look
[17:40] <deryck> adeuring, thanks
[17:43] <jml> is there an API for determining a source package name given a binary package name?
[17:43] <adeuring> deryck: r=me
[17:43] <deryck> adeuring, thanks much!
[18:45] <sinzui> jml: no. guessPublishedSourcePackageName() is closed to what you want, but I was thinking of removing it once we update all the callsites to use the new DSP vocab
[18:45]  * sinzui ponders if the vocab is public in a way to used it
[18:53] <lifeless> probably not
[18:56] <deryck> lifeless, hi.  I believe I have finally *really* fixed the double count issue now.
[18:56] <deryck> lifeless, I'd appreciate you looking again at the MP if you don't mind.
[18:56] <lifeless> deryck: I have
[18:56] <lifeless> deryck: :)
[18:56] <deryck> ah ok :)
[18:57] <lifeless> deryck: it looks good to me, the new-batch object is a common cause of this
[18:57] <deryck> thanks!
[18:57] <lifeless> deryck: I didn't look at the larger context; a lot of other views use a cached property rather than an explicit instance variable
[18:57] <lifeless> but 6/1 1/2 dozen of the other
[18:57] <deryck> yeah, I'm not picky either if you prefer the other way.
[18:57]  * lifeless shrugs
[19:25] <lifeless> deryck: I *had* replied, but lp hasn't seen it.
[19:25] <lifeless> deryck: I suspect incoming mail processing is naffed
[19:26] <deryck> ah, I wondered if that's what you meant.
[19:28] <lifeless> and now I can't find it in my outbox.
[19:28] <lifeless> ENOIDEA.
[19:30] <lifeless> something odd is happening though, I'm trying to add a comment on https://code.launchpad.net/~deryck/launchpad/avoid-extra-buglist-count-901124/+merge/85940 and its stuck on 'Saving' for way more than the 30s ha proxy timeout :(
[19:30] <deryck> hmmm, weird.
[19:30] <deryck> let me try....
[19:31] <deryck> lifeless, I just added a comment no problem.
[19:31] <deryck> less than a second XHR call.
[19:32] <lifeless> weird
[19:33] <lifeless> oh
[19:33] <lifeless> merge proposal
[19:33] <lifeless> -> long poll
[19:34] <lifeless> I wonder if I'm hitting my browser host connection limit
[19:34] <lifeless> yes, I was
[19:34] <lifeless> JULIAN!
[19:34] <lifeless> :)
[19:37] <deryck> ah ha! :)
[19:37] <lifeless> bug 906482
[19:38] <lifeless> bug 906482 ?
[19:38] <lifeless> bah https://bugs.launchpad.net/launchpad/+bug/906482
[19:39] <lifeless> deryck: this explains why I hadn't commented previously, it was stuck in a browser tab
[19:39] <lifeless> deryck: for an hour or so :)
[19:39] <deryck> heh, ouch
[19:40] <deryck> lifeless, but I've seen your browser.  You really do have too many tabs open. ;)
[19:41] <lifeless> see, thats what 16GB of ram is *for*
[19:44] <deryck> heh.
[19:44] <deryck> I stand corrected then. :)
[19:45] <lifeless> that and skyrim
[19:45] <deryck> no, that's what a PS3 is for. ;)
[19:45] <lifeless> heh, Lynne plays it on the PS3
[19:45] <deryck> that and dcuo.
[19:45] <lifeless> it is -much- shinier on a PC
[19:45] <deryck> yeah, most games are actually.
[19:46] <deryck> but it's so comfortable gaming from my bean bag on the floor in front of my 47 inch TV.
[19:46] <lifeless> there was a period where the consoles were better, but then moores law kicked in :)
[19:46] <deryck> heh, yeah
[19:46] <lifeless> bean bags++
[19:46] <lifeless> anyhow, as I'm on leave, I think I'm going to go exercise some of that there gaming stuff ;)
[19:49] <deryck> heh. 3 more days for me.
[20:25] <lifeless> sinzui: nice curly bug there
[20:25] <sinzui> ?
[20:25] <lifeless> 611617
[20:26] <sinzui> lifeless, yes. I really am not sure what to do to close it
[20:26] <lifeless> I don't think the person-linking aspect matters, because the uploader will always be a person (they have to gpg sign the package, which they cannot do as the team)
[20:27] <lifeless> the maintainer field can be a team - its free form text; there are two issues:
[20:27] <lifeless>  a) only allow specifying a private team as maintainer when the uploader can see that team/ is a member of the team
[20:27] <lifeless>  b) disclosure of the team because of this
[20:28] <lifeless> -> I think
[20:29] <sinzui> I do not think the team should be disclosed in the case of a private team's archive. Copying to a public archive makes the relationship public
[20:29] <lifeless> right, I'm saying that there is a probing attack
[20:29] <sinzui> ah
[20:29] <lifeless> i could upload to my public ppa with the maintainer set to 'private-canonical-XXX'@lists.l.n
[20:30] <lifeless> and if it links, voila, I can see the team.
[20:34] <sinzui> lifeless, I was thinking of migrating the logintoken code to lp.services tonight. logintoken is not a good name. I was considering naming the module authentication, then I consider the value it offers is a verification workflow. Maybe lp.services.verification
[20:36] <lifeless> thats the email verification stuff ?
[20:37] <lifeless> there is a bug about that using bounces@c.c., and a suggestion to use a new envelope-sender address. Might want to fix that in passing
[20:37] <lifeless> it should really be a separate service I think though, let anyone at the company trigger a 'is this a real email' probe.
[20:39] <lifeless> sinzui: I've replied on stakeholders, I hope I make sense
[20:39] <sinzui> lifeless, agreed. Thanks for the stakeholder reply too
[20:59] <poolie> hi all
[20:59] <poolie> rick_h__, gary_poster, hi, what is the bzr problem you're talking about?
[21:01] <gary_poster> poolie, hi.  jelmer replied on the list about it
[21:02] <gary_poster> poolie, see launchpad-dev thread "upgrade today got me bzr 2.5 and broken ec2 land"
[21:06] <rick_h__> poolie: yep, did an upgrade this morning and got a bzr 2.5 with the bunch
[21:06] <rick_h__> poolie: which pqm plugin wasn't ready for but it's coming it appears
[21:09] <poolie> ok
[21:10] <poolie> a bit out of sync
[21:17] <poolie> jelmer, we might need some glue to let you check flags from inside the ssh server?
[21:17] <poolie> which would be worth having anyhow
[21:18] <jelmer> poolie: that would be nice
[21:18] <jelmer> poolie: I was actually wondering about the forking lp service
[21:18] <jelmer> poolie: it seems we have enough data to let people switch to the forking server by specifying a different bzr_remote_path
[21:21] <jelmer> so that we could have the forking lp server available without immediately forcing it on all codehosting users
[21:31] <poolie> that would probably work
[21:39] <poolie> we could possibly do something like this in a client controlled way
[21:50] <lifeless> we need, if we don't have it already, and xmlrpc flag checking api
[22:34] <sinzui> wallyworld__, see https://bugs.launchpad.net/launchpad/+bug/855670
[22:35] <mwhudson> lifeless: i add a xmlrpc flag checking api
[22:35] <mwhudson> i keep meaning to revisit anonymous ssh connections using it
[22:43] <wgrant> lifeless: "and if the package is
[22:43] <wgrant> copied to a new context, inform the copier it will lead to disclosure"
[22:43] <wgrant> Uh, how exactly?
[22:43] <wgrant> We can't change *every single API in Launchpad* to prompt if there's a private team involved.
[22:43] <wgrant> That's just stupid.
[22:48] <sinzui> StevenK, this is the bug I was thinking of bug 567088
[22:50] <StevenK> https://bugs.launchpad.net/launchpad/+bug/906151
[22:53] <poolie> lifeless, http://www.stathat.com/
[22:54] <jml> sinzui: that means there's no functional way to get binary package urls from bpph objects over the API. that's ok. will fix.
[22:54] <poolie> o/ jml
[22:54] <jml> poolie: hi
[22:54] <poolie> hi there
[22:57] <wgrant> jml: There's not always a corresponding file in the archive.
[22:57] <wgrant> jml: You may be better off exposing the webapp URL of the file, on BPPH or BPB.
[22:57] <jml> wgrant: and yet spph has binaryFileUrls
[22:58] <wgrant> Right, those are webapp URLs.
[22:58] <wgrant> Not archive ones.
[22:58] <jml> wgrant: I'd be happy w/ webapp urls. and if I were to expose that'd be the natural path.
[22:58] <jml> would just refactor binaryFileUrls to delegate properly
[22:58] <wgrant> The source name is not relevant to those, so you can probably even calculate them now.
[23:02] <jml> wgrant: I haven't even looked to see what they are
[23:03] <jml> black box ftw.
[23:03] <wgrant> Heh
[23:03] <wgrant> /ubuntu/+archive/primary/+files/FOO_VERSION_ARCH.deb
[23:38] <huwshimi> If you add a second branch to a bug that is already fix-released will that branch get deployed to qa staging?
[23:39] <huwshimi> (this bug: https://bugs.launchpad.net/launchpad/+bug/894442)
[23:39] <StevenK> huwshimi: Yes
[23:39] <huwshimi> hmmm...
[23:40] <huwshimi> StevenK: Any reason why this one hasn't then?
[23:40] <StevenK> huwshimi: Is the branch linked to the bug?
[23:40] <huwshimi> StevenK: Yeah
[23:40] <huwshimi> StevenK: Wait, it doesn't look like it merged
[23:40] <StevenK> I was about to say :-P
[23:42] <huwshimi> StevenK: shush
[23:44] <lifeless> poolie: yeah, basically statsd
[23:45] <lifeless> poolie: we're outside the free zone anyhow, ignoring disclosure considerations
[23:51] <poolie> lifeless, yeah, basically statsd+better displays, i think
[23:51] <poolie> free zone?
[23:52] <poolie> i'm not saying we should actually use it