[07:26] <micahg> any LOSAs around?
[07:26] <spm> micahg: yup; whats up?
[07:26] <micahg> I think a couple builders are stuck
[07:27] <micahg> https://edge.launchpad.net/builders/lansones
[07:27] <micahg> https://edge.launchpad.net/builders/radon
[07:28] <spm> I'm guessing  23 minutes for a build of those packages is considered long?
[07:28] <micahg> just got a failure for one
[07:29] <micahg> it did seem long though
[07:29] <wgrant> I wonder if buildd-manager was asleep for a while.
[07:29] <micahg> and the other just finished
[07:29] <micahg> maybe that was it
[07:29] <wgrant> Nothing had been dispatched in > 20 minutes.
[07:32] <spm> I've been getting bunches of alerts all week from the buildd side - very transient tho. it looks like "something" is running MUCH slower vs actually breaking tho.
[07:33] <wgrant> The logging should be much less unclear now.
[07:33] <wgrant> eg. it will actually tell you what it's doing for most operations.
[07:34] <StevenK> spm: Looks like you forgot to feed the hamsters again
[07:34] <spm> StevenK: we stopped feeding them ages back; they'd just poop on the servers more; so we've setup some spare server fans we had lying around and attached tiny whips to 'em.
[07:35] <spm> alongh the principle of "the beatings will continue until morale improves"
[07:35] <spm> it seems to work more effectively
[07:35] <StevenK> spm: Just wait, the hamsters will rise up and form a union, and then you'll be doomed!
[07:37] <spm> they wish. that's when we start substituting the hamsters on the pentagrams where previously we were using the Black Cockrel's in our secret damned sysadmin secret biznis.
[07:38] <noodles775> Are these hamsters related to bug 557391?
[07:38] <wgrant> Ah, that could be it.
[07:38] <spm> noodles775: we have SO got to have another word about your use of facts in serious discussions.... ;-)
[07:39] <noodles775> ;)
[07:39] <wgrant> I don't see how a lock file could block buildd-manager, though -- does it really take out and release a lock every cycle?
[07:39]  * wgrant looks.
[07:39] <noodles775> I couldn't even see a shared lockfile (it uses the name of the script)... I didn't get a chance to look deeper.
[07:40] <noodles775> StevenK: did you get a chance to look at that bug?
[07:40] <mwhudson> could it block inside postgres </wild spectulation>
[07:40] <mwhudson> ?
[07:40] <wgrant> mwhudson: I considered that... possibly.
[07:41] <noodles775> Yeah, a transaction lock was ...
[07:41] <wgrant> I forget how queuebuilder handles transactions.
[07:41] <spm> wouldn't be the first time we've had funky locks in the DB....
[07:41] <wgrant> Even though I've touched it a lot in the past couple of weeks.
[07:41] <spm> wgrant: so... it's your fault????
[07:41] <wgrant> spm: Well, the first bits of the refactoring landed a couple of hours ago, so hopefully not!
[07:42] <spm> heh. and I was just getting the sacrificial knives out and specially blunted. darn it.
[07:43] <mwhudson> what'
[07:43] <mwhudson> s that think from the discworld?  an unhangover or something
[07:43] <mwhudson> oh, hangunder
[07:43] <wgrant> Oh, no, I did touch the relevant method in 10.03. But the transaction lifetimes should be identical.
[07:44] <spm> mwhudson: ha; no. for a change; I did originally type sharpening; but figured that blunt ones would hurt more so... >:)
[07:44] <mwhudson> spm: i meant that wgrant's changes might me so incredible they're affecting things before they're deployed
[07:45] <wgrant> How is queuebuilder being invoked?
[07:50] <spm> mwhudson: that really wouldn't surprise me....
[07:50] <spm> wgrant: cron. every 20 mins.
[07:50] <spm> oh wait. that's the rescore.
[07:51] <wgrant> Do we know which one is being contentious?
[07:51] <wgrant> The --score-only, or the not --score-only?
[07:51] <spm> --score-only
[07:52] <spm> noodles775: there's a question. how IS the queuebuilder invoked? via cesium/buildd manager?
[07:53] <wgrant> b-m doesn't know about it at all.
[07:53] <wgrant> It needn't even be on cesium, but it probably is.
[07:53] <noodles775> spm: I was just hunting through the crontabs on dogfood to try to find it.
[07:55] <spm> :-)
[08:04] <noodles775> spm: I can't see it anywhere in the crontabs I have access to (and I don't have access to /var/spool/cron/crontabs)
[08:05] <spm> noodles775: we DID used to run a queuebuilder on cesium every hour. but that was disabled for an 'indefinite' period in late 2008. so....
[08:05] <noodles775> spm: so how is it being determined that it's causing the lock? Can you see the process?
[08:06] <noodles775> Or maybe lamont was just guessing? I'll ask him later if he can fill in the bug with further details if he's got them.
[08:06] <spm> noodles775: it seems to be a combo of things.
[08:06] <spm> 2010-04-08 07:46:00+0100 [-] startBuild(http://americium.ppa:8221/, transmission, 1.92+r10460-0ubuntu0~lucid, Release)
[08:06] <spm> 2010-04-08 08:04:25+0100 [-] americium (http://americium.ppa:8221/):
[08:07] <wgrant> Oh.
[08:07] <spm> by way of example. is a rather long gap - that's triggering an alert
[08:07] <wgrant> That's probably just a builder being stupid.
[08:07] <wgrant> Although the timeouts should catch most things, if a PPA builder reset trigger hangs then buildd-manager will.
[08:08] <wgrant> And that looks suspiciously like that.
[08:09] <spm> possibly - all I know is I'm getting a lot of those this week :-(
[08:13] <mwhudson> wgrant: are you working on making buildd-manager less bonkers?
[08:13] <wgrant> mwhudson: I've removed most of it.
[08:13] <mdke> anyone around who is an "LOSA"?
[08:13] <mwhudson> wgrant: impressive
[08:13] <spm> mdke: what's up?
[08:13] <wgrant> But this particular issue is more related to the rapid partial asynchronisation of it a couple of years ago, which left us with lovely creatures like RecordingSlave.
[08:14] <mwhudson> wgrant: that was the bonkersness i was thinking of
[08:14] <mdke> spm: I wonder if you could take a look at this question for me. I'm sorry to push for it, I'm sure you have plenty in the queue, but it's preventing me from uploading a branch - https://answers.edge.launchpad.net/launchpad-code/+question/106509
[08:14] <noodles775> wgrant: so the recording slave is receiving the msg to start a build, but then hangs? (why were the recording slaves introduced? wasn't it because the scan cycle was taking too long?)
[08:15] <noodles775> wgrant: also, can you update the above bug with anything you find? Thanks!
[08:16] <spm> mdke: done
[08:16] <wgrant> noodles775: The RecordingSlave lets all the synchronous model code be asynchronous. It takes XML-RPC calls and returns a fake response, and buildd-manager later replays the calls asynchronously.
[08:16] <mdke> spm: fabulous, thanks a bunch
[08:17] <wgrant> So while it prints startBuild there, the slave call may actually happen much later.
[08:18]  * noodles775 tries to see why that would mean that the buildd-manager doesn't log anything after the startBuild (like "resetting builder...")
[08:19] <wgrant> noodles775: That's true.
[08:19] <wgrant> I haven't had a good look just yet.
[08:19]  * wgrant gets a 10.03-like branch.
[08:19] <spm> mdke: np
[08:20] <wgrant> spm: So those two americium lines are immediately adjacent?
[08:20] <spm> wgrant: yup. straight from the raw log
[08:21] <spm> we have an alert for the log not benig updated in 5-10 minutes; hence why I've had a few of these.
[08:23] <wgrant> spm: if there are no private builds involved, can you pastebin a few of the subsequent log lines?
[08:25] <spm> wgrant: um. no. for exactly that reason. :-/
[08:25] <wgrant> Bah.
[08:27] <spm> typically - all the others I can easily find are also private. wonder if cuase/effect or just sheer coincidence.
[08:28] <mdke> spm: mwhudson: not sure if you're available to help, but after spm has done his thing, the import is failing (https://code.edge.launchpad.net/~vcs-imports/gnome-user-docs/master)
[08:28] <wgrant> The builder varies?
[08:28] <spm> wgrant: I'm guessing you're after the text between the '***** RESULT ***** ;;;;******************' bits
[08:28] <mwhudson> mdke: :(
[08:28] <wgrant> spm: I don't care about that at all.
[08:28] <spm> yeah. looking at one for nannyberry atm
[08:28] <mdke> mwhudson: yeah
[08:29] <mwhudson> jelmer: ^^ can you look at mdke's failure?
[08:30] <wgrant> spm: So the hang is between the startBuild() and the ************MOREASTERISKS****** RESULT *****EVENMORE******?
[08:30] <mwhudson> mdke: i guess the conversion could be done with bzr-git by hand
[08:31] <spm> wgrant: like this: http://paste.ubuntu.com/410900/
[08:31] <mdke> mwhudson: I don't know
[08:32] <wgrant> spm: Thanks.
[08:32] <mwhudson> mdke: if i unhook the branch from being the dev focus, at least you'll be able to push your branch
[08:32] <wgrant> So the problem isn't in the asynchronous part. This is good.
[08:33] <mwhudson> mdke: or you could do that i guess
[08:33] <mdke> mwhudson: that's correct I think, the problem is the repository format between Launchpad and my local repo is not compatible
[08:34] <mdke> mwhudson: yes, I'll do that now
[08:35]  * mwhudson really needs to get away from the computer
[08:36] <mdke> mwhudson: ok, thanks for your help. Have a nice evening
[08:42] <wgrant> All I can think of is that BQ.markAsBuilding might be conflicting with queuebuilder's scoreCandidates. While I moved that method in 10.03, the transaction use there hasn't changed.
[08:42] <wgrant> The other few dozen lines between the log messages look perfectly safe.
[09:46] <jelmer> mwhudson, mdke: looking
[09:55] <rdb> is there any documentation available on cscvs?
[09:55] <wgrant> From what I've seen, 'run away'.
[09:57] <jelmer> mwhudson, mdke: no idea, works here locally :-(
[10:19] <BUGabundo> morning
[10:19] <BUGabundo> should click images attached to LP open a new tab?
[10:19] <BUGabundo> or use the same?
[10:23] <BUGabundo> using chromium daily, fyi
[10:30] <rdb> wgrant, 'run away'? :-? It looks quite complicated
[10:44] <bialix> interesting is it possible to bookmark a projects on LP? there is related projects to some user. what if I just want to track the progress of some project, but I'm not part of it?
[10:45] <bialix> something to quick access projects on LP from LP user home page
[11:00] <sebi`> when writing an announcement, is there a way to use text-markup?
[11:20] <czajkowski> aloha :)
[11:21] <czajkowski> just wondering is it possible to create a bug and mark it just for one council to be able to view - I want to create Ubuntu Loco council bugs to track issues? and only the people on this team to access the bug ?
[11:28] <bialix> czajkowski: there is security issue checkbox
[11:29] <BUGabundo> deryck: should click images attached to LP open a new tab? or use the same?
[11:30] <deryck> czajkowski, yeah, what bialix said.  That's the only way to file a private bug and once the bug is reported, you can uncheck security but leave private enabled.
[11:30] <deryck> BUGabundo, click images?  What do you mean?
[11:30] <BUGabundo> deryck: an attached screenshot for example
[11:30] <deryck> BUGabundo, ah.  Should open in the same window.  We don't do anything to make them open new.
[11:31] <BUGabundo> :\\
[11:31] <idnar> I think you can have a project set to create private bugs by default
[11:31] <BUGabundo> I'll open a wishbug
[11:31] <idnar> I'm not sure if that's done for projects that don't pay for a commercial license, though
[11:31] <BUGabundo> don't like to lose previous page
[11:31] <idnar> BUGabundo: can't you control-click or middle-click them?
[11:31] <BUGabundo> yep
[11:32] <idnar> that seems like a better solution than forcing a new window / tab on everyone
[11:32] <BUGabundo> but the behaviour I expect for clicking attached files is new tab with focus
[11:32] <deryck> idnar, private by default is only for commercial projects
[11:32] <czajkowski> bialix: deryck cheers
[11:32] <BUGabundo> other then see it replace the bug page
[11:33] <czajkowski> just when I create community bugs, there seems to be half a dozen folks already assigned to bugs and I want to avoid that
[11:33] <deryck> BUGabundo, yeah, feel free to file a wishlist bug, but I don't think we would ever do that by default.
[11:33] <BUGabundo> shame :\
[11:33] <deryck> Sorry, but I think more users would be upset by that than those that would like that. :-)
[11:40] <BUGabundo> really?
[11:41] <deryck> BUGabundo, that's my assumption based on the fact that users *generally* don't want new windows and controlling whether it's a new window or new tab is a browser setting.
[11:41] <czajkowski> deryck: is there any reason why some folks are automatically assinged to community bugs? that aren't council members??
[11:42] <deryck> czajkowski, if someone has subscribed to all bugs for your project they will automatically be subscribed to new bugs.
[11:42] <deryck> BUGabundo, but really, there's no way to know for sure without asking users somehow.
[11:43] <czajkowski> deryck: oh how does one do that say for community bugs?
[11:47] <deryck> czajkowski, go to the bugs section for a project, look down the right side of the page where the X open bugs links are.  Just below that there is a "Subscribe to bug mail" link.
[11:49] <czajkowski> deryck: cheers
[11:51] <czajkowski> deryck: cheers, that answer my question then, if I want to create LoCo council bugs and have only the 6 of us get them that's not possible
[11:52] <deryck> czajkowski, not by default, that's right.  Launchpad assumes use is for open projects if you don't have a commercial subscription.
[11:53] <czajkowski> deryck: what I'm trying to do is log issues that are mailed to the 6 of us to keep track of and some are delicate
[12:01] <deryck> czajkowski, I understand.  Unfortunately, I don't think there's a way to support this with launchpad currently.  You can mark security as mentioned above when filing the bug, but that's it really.
[12:17] <czajkowski> deryck: ok thanks
[13:21] <mdz> should Launchpad be emailing my own comments back to me when I reply to a merge proposal? it doesn't seem helpful
[13:27] <deryck> hi mdz.  Yeah, it normally does that.  and yeah, it's not really helpful.
[13:28] <mdz> deryck, should I just add a filtering rule to discard the messages, like I do for bugs, or is this likely to change?
[13:28] <lifeless> it also emails you when you edit a bug/merge proposal in the web ui
[13:28] <lifeless> also not useful :)
[13:29] <deryck> so this will change from bugs very soon... mdz, not sure about the MPs.  Looking at bugs now...
[13:29] <mdz> deryck, oh, that's good news
[13:30] <deryck> mdz, lifeless -- we have a focus on better bug notifications coming next for us.  See:  https://dev.launchpad.net/LEP/BetterBugSubscriptionsAndNotifications
[13:30] <deryck> jml, do you know if the email noise from yourself for MPs is likely to change?
[13:31] <jml> deryck: what noise?
[13:31] <jml> (I guess not)
[13:31] <deryck> jml, when you reply to an MP, you get a mail back for your own comment.  see mdz's comment above.  ^^
[13:32] <jml> oh right.
[13:32] <jml> deryck: we ought to change it, I don't know of any plans to do so currently
[13:33] <deryck> mdz, so mail filter looks the best route then for MPs.
[13:33] <jml> deryck: further, it's crappy that we have things set up in such a way so that it's easy for you to fix this problem for bugs but not for merge proposals
[13:34] <deryck> jml, *agreed*
[13:34] <deryck> this is some of the worst pain of launchpad dev to me.
[13:35] <jml> deryck: I'll ask thumper about it when I speak with him later today
[13:36] <deryck> cool
[13:37] <jml> as for the more general problem of the respective notification systems being divergent, no concrete next actions spring to mind
[13:38] <deryck> jml, I wonder if as part of the better-notifications work we shouldn't spend some time unifying notifications across LP?  users don't see the distinction, I'm sure.
[13:38] <deryck> that's huge, I know.
[13:39] <jml> deryck: it's tempting.
[13:39] <jml> deryck: want to think about it some more and then we can talk about it in our call tomorrow?
[13:39]  * jml actually needs to think it through
[13:40] <deryck> jml, definitely we can talk about it then.  I think it takes it from a 1-2 month story currently to a 3-5 month story, though.  depending on how many engineers are on it.
[13:40] <deryck> maybe I'm a pessimist about how quickly things can be done in lp though :-)
[13:41] <deryck> jml, did you see my cycle time email I sent?
[13:42]  * jml looks
[13:43] <deryck> it's very long, sorry.  week old now.
[13:43] <jml> deryck: yeah, it's tagged with @READ :)
[13:44] <jml> deryck: so I will read it...
[13:44] <deryck> jml, yeah, understood. :-)  Basically, we do dev and review in 3-4 then spend average of 8 days beyond that in queuing and landing.
[13:44] <deryck> for a 12 day cycle time on a bug or task.
[13:44] <deryck> toooooooo long.
[13:45] <jml> deryck: yeah. In some projects, queueing and landing is a sub-minute operation :)
[13:45] <deryck> jml, exactly!
[13:51] <hyperair> does anyone know what happened to the BGO bug status trackers?
[14:13] <deryck> hyperair, hi.  Sorry but I don't know what you mean by BGO.  Is that a project on LP?
[14:15] <persia> deryck: Just from backscroll: one nice feature about LP sending you your own notes from bugs is that the local mailspool contains a complete record of the bug.  This may be less useful for merge proposals or similar.
[14:16] <deryck> persia, interesting.  Assuming you replied on LP itself, right?  If you reply via mail, that should already be there, correct?
[14:16] <persia> Depends on your client configuration: it may not be in the same threading view.
[14:17] <persia> (some clients auto-merge send and receipt, some don't)
[14:17] <persia> Mind you, the answer might be "go use a different mail client", or "set up your filters to do something useful" :)
[14:22] <jml> deryck: BGO often means "bugzilla.gnome.org"
[14:23] <deryck> ah
[14:23] <deryck> persia, that's a nice perspective though, thanks.
[14:24] <deryck> hyperair, are you asking about updates to bugs linked to the gnome bugzilla?  if so, it's being worked on.  Very close but still not there yet.
[14:29] <hyperair> deryck: ah cool. thanks for the information.
[15:18] <mok0> w
[16:50] <qense> I've got a problem with the sprints section of Launchpad. At <https://edge.launchpad.net/sprints/uds-m> I -- Sense Hofstede -- am listed as attending, but I can still press the 'Register Yourself' button and http://summit.ubuntu.com/uds-m still thinks I'm not attending. I've marked myself as attending yesterday evening, so it can't be that I just have to wait another few minutes and refresh.
[16:55] <deryck[lunch]> qense, that seems an issue with the summit.ubuntu.com app, since you're listed in the launchpad sprints page.
[16:55] <deryck[lunch]> qense, I'm not sure who to ping about that, but I can check after I eat.
[16:56] <qense> deryck[lunch]: the strange thing is that the Launchpad page for the summit also still allows me to (+) Register my attentdance.
[16:56] <qense> deryck[lunch]: but have your lunch first :)
[16:59] <persia> qense: OOh, cool.  I can replicate.  I'm not sure it's an LP issue though, as LP showed me as attending both before and after I played with summit and reregistered.
[17:01] <qense> persia: But shouldn't the (+) button go away on Launchpad when you've registered yourself as attending?
[17:01] <persia> I don't think so: I think it's an artifact of being able to edit one's attendance.  Yes, it's a bug, but just a minor UI bug (should be pencil rather than +)
[17:04] <qense> then it's just an Summit issue
[17:04] <persia> I think so.
[17:04] <persia> Plus the minor UI bug.
[17:06] <qense> ok
[17:07] <qense> persia: There is going to land a new version of Summit real soon, iirc, so I'll wait for that to arrive.
[17:07] <persia> No idea.
[17:07] <persia> jcastro: Do you know who knows about summit, or in which channel we should ask questions, or how to file bugs?
[17:08] <qense> iirc there is a 'summit' project on LP for that
[17:08] <jcastro> persia: lp.net/summit
[17:08] <jcastro> persia: Daviey is your man
[17:09] <persia> jcastro: Cool, thanks :)
[17:10] <jcastro> persia: I've assigned him some bugs already so get them in quick before scheduling starts!
[17:10] <persia> qense: Are you handling this bug, or shall I file it?  If you are, please subscribe me.
[17:11] <qense> persia: I'll file it and subscribe you.
[17:12] <persia> qense: Thanks :)
[17:13] <persia> I'll confirm it in a bit.
[17:13] <qense> persia: thanks for that!
[17:14] <qense> persia: You're subscribed!
[17:15] <persia> Confirmed and me-too'd.
[17:15] <qense> great!
[19:03] <cbx33> hey guys got a question
[19:03] <cbx33> I see a packge ina  ppa, but when I try to apt-get install it, it's not there
[19:04] <cbx33> and it's not in the packages.gz file either
[19:04] <cbx33> any ideas?
[19:06] <james_w> cbx33: did it fail to build?
[19:06] <cbx33> I don't htink so
[19:07] <cbx33> it is the zfs-fuse package from brcha ppa
[19:08] <james_w> which series?
[19:08] <james_w> it's not available for karmic or lucid
[19:08] <cbx33> hmmm
[19:08] <cbx33> that's odd
[19:08] <cbx33> a page i saw had it add it for karmic
[19:09] <cbx33> the title of the post was zfs-fuse for 9.10
[19:38] <deryck> sinzui, does registry get sprint app bugs?
[19:39] <sinzui> no, They exist for the sake of blueprints
[19:41] <sinzui> deryck, I retargeted a sprint bug to blueprints a few hours ago
[19:42] <deryck> sinzui, ah, ok.  Thanks.
[19:46] <micahg> is  that right, 16k jobs for the i386 PPAs?
[19:52] <persia> micahg: Maybe an archive-rebuild-test?
[19:52] <micahg> persia: maybe
[19:52] <persia> Those are low-priority (below PPA) typically.
[19:57] <sladen> https://bugs.launchpad.net/malone/+bug/558642  <-- celebrity bug popularity fail
[20:01] <kfogel> sinzui: Do you know if there's any way for me to set an arbitrary user as an Answer Contact for Launchpad?
[20:01] <kfogel> sinzui: I click "Set answer contact" to get to https://answers.edge.launchpad.net/launchpad/+answer-contact, but my choices there are: myself, or one of a set list of teams.  I can't just type in a user's name.
[20:05] <sladen> kfogel: isn't on the basis of a user choosing to subscribe themself?
[20:05] <sladen> themselve
[20:06] <kfogel> sladen: yeah, I'm going to ask the user if he can set himself (but suspect he can't).
[20:06] <sinzui> kfogel, you cannot make someone do a task
[20:06] <sinzui> kfogel, though it would be nice if I could subscribe a user to a question that concerns him
[20:07] <kfogel> sinzui: ok
[20:07] <sinzui> kfogel, a user can choose the be an answer contact for *any* project.
[20:08] <kfogel> sinzui: oh, great, I didn't realize that
[20:09] <sinzui> kfogel, Launchpad recognises that the support and translation communities are often autonomous from the core project. we let communities contribute by choice rather than by authority
[20:09]  * sinzui may have the formation of his next blog post
[20:10] <kfogel> sinzui: :-)
[20:11] <kfogel> sinzui: this is all around that user who asked matsubara if he (the user) could be an answer contact for Launchpad.  After all this digging, it turns out he could have just set himself as a contact from the get-go.
[20:11] <kfogel> It sounds like, anyway.
[20:13] <sebi`> does launchpad allow any sort of markup for, let's say, announcements, and kinds thereof?
[20:17] <matsubara> kfogel, you're right. I just tested on staging with my test account and by setting myself as the answer contact I was able to edit a question's assignee
[20:17] <matsubara> kfogel, so just telling the guy to set himself as an answer contact will solve the trick
[20:18] <matsubara> s/the trick/the problem/
[20:29] <cbx33> sladen, hey long time dude
[21:03] <mdke> jelmer, mwhudson - what should I do to try and resolve this? File a bug? On what project?
[21:04] <jelmer> mdke: I'm not sure where the problem is exactly; I can't reproduce the failure locally
[21:04] <jelmer> So perhaps this can be resolved by updating the version of bzr-git on Launchpad
[21:05] <mdke> jelmer: so should I file a question / bug for that?
[21:06] <jelmer> mdke: Good question
[21:06] <jelmer> mwhudson: there?
[21:07] <jelmer> mwhudson: I'll push a fix for the -0000 / +0000 patch in a bit, perhaps we could try that patch on staging?
[21:11] <mwhudson> jelmer: yeah ok
[21:14] <bodhi_zazen> Is there a limit to how many admins can be on a project or team ?
[21:14] <bodhi_zazen> And can only the project owner add an admin, or can another admin add one ?
[21:15] <persia> bodhi_zazen: No, and yes
[21:15] <bodhi_zazen> TY