| nigelb | bryceh: just fyi, your blog has some trouble with openid ;) | 01:29 |
|---|---|---|
| nigelb | anyway, thats a neat new feature :) | 01:29 |
| bryceh | nigelb, it does? howso? | 02:07 |
| bryceh | I didn't even know it could do openid | 02:07 |
| bryceh | nigelb, although I did notice nobody wrote comments :-/ | 02:07 |
| nigelb | bryceh: heh, I tried to login with LP | 02:11 |
| nigelb | failed | 02:11 |
| nigelb | you want me to grab the error? | 02:11 |
| bryceh | yeah | 02:13 |
| bryceh | I upgraded my web server to lucid, and drupal got upgraded, however it seems there is some bug that causes upgraded drupals to break in lucid | 02:13 |
| bryceh | (which is why I haven't been blogging for a long time) | 02:13 |
| bryceh | maybe that bug is why user registration doesn't work either | 02:14 |
| nigelb | yeah | 02:17 |
| nigelb | hold on, doing it again | 02:17 |
| ajmitch | is the drupal issue that it's using deprecated functions? | 02:19 |
| nigelb | it probbaly is | 02:19 |
| nigelb | lucid only has 5.3 | 02:19 |
| ajmitch | </3 php | 02:19 |
| nigelb | http://pastebin.com/r3fTKwmQ | 02:20 |
| bryceh | ajmitch, no it looks like just a bug in the upgrade script, doesn't set fields to autoincrement | 02:20 |
| ajmitch | lucid does have a drupal6 package, fwiw, but it's even OT for here :) | 02:20 |
| nigelb | yeah, autoincrement thing :) | 02:21 |
| bryceh | nigelb, ok yeah I've reproduced that error | 02:21 |
| nigelb | that explains the comments :) | 02:21 |
| nigelb | why don't you move to something like wordpress for ease of use? | 02:22 |
| bryceh | nigelb, why? | 02:29 |
| nigelb | drupal is nice, but just "complicated" ;) | 02:30 |
| bryceh | nigelb, I develop on launchpad and x.org, I shouldn't be put off by "complicated" ;-) | 02:31 |
| bryceh | nigelb, is wordpress open source? | 02:31 |
| nigelb | bryceh: hahahaaha | 02:31 |
| nigelb | and yes, its open source | 02:32 |
| nigelb | also complicated for visitors is what I meant :) | 02:32 |
| bryceh | well, if I wanted something simple for visitors I'd just post to facebook | 02:36 |
| nigelb | hahah | 02:37 |
| nigelb | but that still needs signup | 02:37 |
| nigelb | btw, you're still in Prague? | 02:38 |
| bryceh | nope | 02:38 |
| bryceh | got home last night | 02:38 |
| nigelb | :) | 02:41 |
| nigelb | someday I gotta start hacking on LP. | 02:41 |
| wgrant | nigelb: JOIN US! | 02:44 |
| nigelb | wgrant: I'm just worried | 02:44 |
| nigelb | I use this system for work | 02:44 |
| bryceh | nigelb, it might be interesting for the LP folks to know what is holding you back, so they can work to eliminate that as an issue | 02:44 |
| nigelb | so, can't have my apache dying on me | 02:44 |
| wgrant | nigelb: LP doesn't usually eat systems alive, and if you're really worried you can use a VM. | 02:45 |
| nigelb | the instructions say that the settings used for working on LP means usual web development cannot happen | 02:45 |
| bryceh | nigelb, do you not have another system you could use for development on? | 02:45 |
| wgrant | rocketfuel-setup will try to remove PHP (because it installs apache2-mpm-worker), but that can be worked around. | 02:45 |
| wgrant | launchpad-database-setup obliterates all PostgreSQL databases, but that can also be worked around if required. | 02:46 |
| nigelb | bryceh: not yet. My own system is in the shop awaiting motherboard replacement | 02:46 |
| wgrant | But apart from that, it coexists fine. | 02:46 |
| wgrant | Just adds some vhosts to Apache. | 02:46 |
| * nigelb developes on PHP :( | 02:46 | |
| nigelb | VM seems like a good idea. | 02:47 |
| wgrant | nigelb: that just means you have to tweak rocketfuel-setup to not install apache2-mpm-worker. | 02:47 |
| wgrant | It's not actually required. | 02:47 |
| bryceh | nigelb, well, like wgrant says, running in a VM works too. that's how brian does it | 02:47 |
| wgrant | But yes, a VM works too. | 02:47 |
| nigelb | I'll let a VM run rocketfuel overnight today :) | 02:47 |
| * nigelb gets sucked in LP | 02:48 | |
| bryceh | wgrant, maybe the newbie intro docs should just be written from the start to nudge the user towards setting up launchpad in a VM? it'd sidestep so many issues and worries | 02:51 |
| wgrant | bryceh: Possibly. I'm not sure who manages those docs these days, though, since Karl disappeared. | 02:52 |
| nigelb | disappeared? | 02:53 |
| ajmitch | there was someone who'd stepped up to that role I think | 02:53 |
| lifeless | there is a volunteer, yes. | 03:20 |
| bryceh | who? | 03:29 |
| lifeless | it was announced to the list | 03:31 |
| bryceh | heh this is a bad sign if no one knows | 03:31 |
| lifeless | public I think. | 03:31 |
| lifeless | I know, it just not paged in. | 03:32 |
| wgrant | lajjr | 03:32 |
| lifeless | also, I present, another victem of the heat. | 03:32 |
| bryceh | nigelb, anyway I think you may be right I should just use wordpress. I had visions of using drupal for some more sophisticated website stuff, but these days I hardly have time to mess around with updating my website anyway | 03:35 |
| nigelb | bryceh: heh :) | 03:35 |
| nigelb | just be sure to update regularly | 03:35 |
| bryceh | unfortunately that means losing all my old blog posts | 03:36 |
| bryceh | nigelb, does something bad happen if you don't update regularly? | 03:36 |
| lifeless | grrr jmkuhn - thanks mwhudson. | 03:37 |
| nigelb | bryceh: vulnerabilities get reported often and get security updates | 03:37 |
| nigelb | if you're lazy, you get owned :D | 03:37 |
| bryceh | that sounds bad | 03:38 |
| lifeless | bryceh: wordpress.com ++ | 03:38 |
| bryceh | nigelb, is updating as simple as apt-get upgrade wordpress or is more to it? | 03:38 |
| nigelb | there would be a zip file i guess which you would use | 03:38 |
| bryceh | ick | 03:38 |
| nigelb | I've never done it. I just use wordpress.com ;) | 03:38 |
| lifeless | there is also a bzr branch you can just pull | 03:38 |
| nigelb | yay, bzr++ | 03:39 |
| nigelb | lifeless: that should be a snap then :) | 03:39 |
| mtaylor | lifeless: I just coped the planet.ubuntu.com idea of launchpad team + bzr branch of config for adding to blog syndication | 03:40 |
| mtaylor | lifeless: total win - should be a launchpad feature... | 03:40 |
| lifeless | cool | 03:40 |
| lifeless | hmm | 03:40 |
| lifeless | interesting folk are here | 03:40 |
| lifeless | here is a crazy idea I had. | 03:40 |
| lifeless | what if every object was [crudely] available as a 1-item-long rss feed (using accept-encoding negotiation) | 03:41 |
| mtaylor | interesting | 03:41 |
| wgrant | To what end? | 03:41 |
| lifeless | pubsubhubbub | 03:41 |
| * mtaylor hates it that lifeless answer word is real and that I know what it is | 03:42 | |
| lifeless | we would pingthesemanticweb on a change to any object [excluding private in the first instance] | 03:42 |
| lifeless | that is monitored by at least one hub | 03:42 |
| wgrant | Hmm. | 03:42 |
| lifeless | and you can get callbacks when the object has changed. | 03:42 |
| mtaylor | would make it easier to monitor things like lists-of-available-merge-requests | 03:43 |
| lifeless | want to know when a build has finished on arm + i386? subscribe to the build | 03:43 |
| mtaylor | any thing that gets me callbacks will make me happy as far as those things go | 03:43 |
| nigelb | bryceh: I stand corrected. There is a press-button upgrade feature. | 03:44 |
| bryceh | nigelb, mm | 03:45 |
| wgrant | mwhudson: Hmmm, so you're actually going to fix LP and modularise it (and the model classes)? | 04:16 |
| mwhudson | wgrant: well | 04:57 |
| mwhudson | wgrant: to some extent or other | 04:57 |
| spm | rewrite from scratch. in php. | 05:06 |
| mtaylor | ewwww | 05:07 |
| * wgrant stabs spm repeatedly. | 05:07 | |
| * mtaylor pokes spm until he stops | 05:07 | |
| ajmitch | spm: that's the official word from what the LOSAs want then? | 05:07 |
| spm | ajmitch: um.... | 05:07 |
| ajmitch | next he'll be saying that PHP 5 is too new, let's go for 4 | 05:08 |
| spm | tho I will concede my troll has been matched, and the trolling stacks raised higher than I'm prepared to go. | 05:08 |
| spm | Oooo! didn't think if that! ta! yes! php4 ftw. | 05:08 |
| * ajmitch shouldn't continue down this path | 05:10 | |
| mtaylor | fuck that... cobol forms are much more powerful | 05:10 |
| mtaylor | spm: I just got a message that said that ~swift-bugs is subscribed to openstack - but I cannot for the life of me see _why_ that would be the case | 05:11 |
| spm | mtaylor: hrm. looking | 05:12 |
| mtaylor | spm: also, I think we're clear to delete all of those -private things | 05:12 |
| spm | mtaylor: ahh. sweet. I'll do that first. it's possible one is causing the other. | 05:12 |
| mtaylor | mmm. good point | 05:12 |
| spm | openstack-private gone | 05:13 |
| mtaylor | woot | 05:14 |
| spm | swift-bugs-private swift-core-private gone | 05:14 |
| spm | gah. swift-private is being recalcitrant, per that answers request you have. been looking at that this morning to no joy | 05:15 |
| mtaylor | glad to know I haven't stopped causing you grief :) | 05:15 |
| spm | actually.. thinking about that - I suspect it may have been self inflicted, by myself.one I need to chase. | 05:16 |
| spm | mtaylor: I assume the ozone ones can go too? | 05:17 |
| mtaylor | spm: yes. all ozone can die | 05:18 |
| spm | kk | 05:18 |
| wgrant | Is there some DB load issue at the moment? | 05:31 |
| wgrant | For the past 24-36 hours, my script that heavily uses getBuildRecords has timed out every time. | 05:31 |
| wgrant | It previously only timed out one every few hours. | 05:31 |
| spm | that's not cool... | 05:31 |
| spm | 1 min load is higher than usual, but 5 and 15 is normal. light even. | 05:32 |
| spm | wgrant: you wouldn't happen to know what table that references? | 05:34 |
| wgrant | spm: BinaryPackageBuild, PackageBuild and BuildFarmJob, mainly. | 05:34 |
| spm | is that talking to edge? I'd suggest not but... ? | 05:36 |
| wgrant | It may be... let's see. | 05:37 |
| wgrant | It moves around depending on where the API is least broken. | 05:37 |
| spm | heh | 05:37 |
| wgrant | It's currently using edge. | 05:37 |
| spm | I'm seeing a few timeouts for queries with BinaryPackageBuild in them, all via edge | 05:38 |
| wgrant | These should be 12-15 minutes past the hour. | 05:38 |
| spm | ta! that helps hugely. | 05:39 |
| spm | hrm. these are all almost uniformly on the hour; so possibly not you. | 05:40 |
| wgrant | Hmm. | 05:41 |
| spm | wine, firefox, kdm,? | 05:42 |
| spm | ubiquity is another | 05:42 |
| wgrant | My script requests everything with a build failure | 05:42 |
| bryceh | nigelb, http://www2.bryceharrington.org:8080/wordpress/?p=3 | 05:42 |
| spm | wgrant: are you getting an oops from these? the commonish query that somewhat matches is taking around 10secs based on a quick random grab. cop enough of those... | 05:45 |
| wgrant | spm: I'm getting OOPSes, but this old launchpadlib doesn't display them. | 05:46 |
| wgrant | Maybe I should upgrade that host to Lucid at some point. | 05:46 |
| spm | bleh | 05:46 |
| wgrant | Hm, wow, nearly a year since the open sourcing. | 05:48 |
| === mtaylor is now known as mtaylor|beach | ||
| wgrant | Argh. | 06:06 |
| wgrant | https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/177 is redispatching constantly. | 06:06 |
| wgrant | Presumably because there is no longer a warty/i386 chroot. | 06:06 |
| wgrant | Ah, and there are hoary and breezy builds too. | 06:06 |
| nigelb | bryceh: w00t w00t! | 06:08 |
| lifeless | wgrant: timeouts ? | 08:15 |
| wgrant | lifeless: Yes. spm reminded me that you reduced the timeout last week. | 08:16 |
| lifeless | there is a bug script hammering one bug on prod with trouble | 08:17 |
| lifeless | the hourly edge graph is clean though | 08:17 |
| lifeless | what issue are you having? | 08:17 |
| lifeless | wgrant: ^ | 08:18 |
| wgrant | lifeless: distroseries.getBuildRecords is frequently timing out for me. | 08:18 |
| lifeless | got an oops id ? | 08:18 |
| wgrant | This old launchpadlib doesn't display them, so no :/ | 08:19 |
| lifeless | can you get one, please ? | 08:19 |
| wgrant | OK. You can't easily find OOPSes for a particular method? | 08:19 |
| * wgrant turns httplib2 debugging on. | 08:20 | |
| spm | i wonder... if oopses record the remote IP, lifeless, you should be able to fgrep -R on devpad and hunt thru them all? brute force, but... | 08:21 |
| wgrant | It's hopefully reproducible. | 08:22 |
| lifeless | wgrant: no, not yet. | 08:22 |
| wgrant | It normally happens within the first five minutes of the script, and it's running with debug logging on now... | 08:22 |
| lifeless | I'll file a bug on finding oops by method | 08:22 |
| lifeless | spm: seeing ip addressess is a bug | 08:23 |
| lifeless | we need to fix that - and seeing private urls : both should be limited to losa and gsa | 08:23 |
| lifeless | https://bugs.edge.launchpad.net/oops-tools/+bug/607087 | 08:24 |
| _mup_ | Bug #607087: enable 'search by method' <OOPS Tools:New> <https://launchpad.net/bugs/607087> | 08:24 |
| lifeless | wgrant: any sign ? | 08:30 |
| wgrant | lifeless: It seems to be an excellent Heisenbug. | 08:31 |
| wgrant | arrrgh. | 08:31 |
| wgrant | Just died now. | 08:31 |
| lifeless | cool | 08:31 |
| wgrant | But there's no X-Lazr-Oops header... | 08:31 |
| lifeless | its not in the oops report either FWIW | 08:31 |
| lifeless | wgrant: :( | 08:33 |
| lifeless | what is there ? | 08:33 |
| lifeless | pastebin ? | 08:33 |
| wgrant | Hm. Maybe timeouts don't create an X-Lazr-Oops header :( | 08:34 |
| lifeless | wgrant: this is what edge was last night - | 08:34 |
| lifeless | === Top 10 Time Out Counts by Page ID === | 08:34 |
| lifeless | Hard / Soft Page ID | 08:34 |
| lifeless | 63 / 618 DistroSeries:EntryResource | 08:34 |
| lifeless | 12 / 71 Person:+participation | 08:34 |
| lifeless | 9 / 13 ScopedCollection:CollectionResource | 08:34 |
| lifeless | 8 / 29 BugTask:+index | 08:34 |
| lifeless | 5 / 50 Sprint:+temp-meeting-export | 08:34 |
| lifeless | 5 / 6 Archive:+copy-packages | 08:34 |
| lifeless | 4 / 2 DistroSeries:+index | 08:34 |
| lifeless | 3 / 3 BugTask:+editstatus-page | 08:34 |
| lifeless | 2 / 3 Milestone:+index | 08:34 |
| lifeless | 2 / 1 Person:+map | 08:34 |
| lifeless | wgrant: we do get timeout oops on edge apis | 08:34 |
| wgrant | It's probably the first one. | 08:34 |
| lifeless | OOPS-1660EB2408 | 08:34 |
| wgrant | It would come under the first one, that is. | 08:34 |
| wgrant | The request URL should have ws.op=getBuildRecords | 08:35 |
| StevenK | !pastebin | 08:35 |
| lifeless | QUERY_STRING: build_state=Failed+to+build&ws.op=getBuildRecords&ws.start=200&ws.size=50 | 08:35 |
| lifeless | StevenK: !sanity | 08:35 |
| wgrant | That's the right kind of request. | 08:35 |
| wgrant | GET /1.0/ubuntu/maverick?build_state=Failed+to+build&ws.op=getBuildRecords&ws.start=1750&ws.size=50 | 08:35 |
| wgrant | That's the one that failed just now. | 08:35 |
| StevenK | lifeless: You should know better. :-P | 08:36 |
| wgrant | I'd only expect 24 hard OOPSes a day from this, though. | 08:36 |
| lifeless | StevenK: 12 lines in a totally dedicated channel is not a problem | 08:36 |
| lifeless | StevenK: when there are no other discussions taking place. | 08:36 |
| lifeless | StevenK: we are, after all, here to *communicate* | 08:36 |
| wgrant | lifeless: I can has that OOPS? | 08:37 |
| wgrant | lifeless: Ah, thanks for the review. | 08:37 |
| lifeless | SQL time: 12978 ms | 08:38 |
| lifeless | Non-sql time: 1499 ms | 08:38 |
| wgrant | Aha. | 08:38 |
| wgrant | What're the big queries? | 08:38 |
| wgrant | Or are there lots? | 08:38 |
| lifeless | http://pastebin.com/DsDmLqLk | 08:39 |
| lifeless | one repeated query is the culprit I think | 08:39 |
| wgrant | Forgive me -- I've no idea what those columns mean. | 08:39 |
| wgrant | I'd really like to know where that query is coming from. | 08:41 |
| wgrant | Hmm. | 08:41 |
| wgrant | Maybe current_source_publication, I guess. | 08:41 |
| wgrant | Although that query should be pretty quick. | 08:41 |
| wgrant | lifeless: Do you have sufficient powers yet to land that branch? | 08:43 |
| lifeless | wgrant: probably but EBUSY | 08:47 |
| lifeless | wgrant: please grab someone else | 08:47 |
| wgrant | lifeless: OK. | 08:47 |
| lifeless | wgrant: oh, the 37 is repetitions | 08:47 |
| lifeless | row reps time avg time-avg db-id query | 08:48 |
| lifeless | there are also two very slow queries | 08:48 |
| lifeless | 12 seconds total | 08:49 |
| lifeless | http://pastebin.com/VsQpzqUq | 08:49 |
| lifeless | wgrant: ^ | 08:49 |
| lifeless | wgrant: thats actually the issue, probably. | 08:49 |
| wgrant | WTF is it doing a count for. | 08:49 |
| lifeless | the rest is inefficient but broadly noise. | 08:49 |
| wgrant | Thanks. | 08:50 |
| wgrant | Those are indeed the problems. | 08:50 |
| lifeless | wgrant: I'd like to lower the timeout again today; is that ok with you, given you're about to propose a fix ? :) | 08:50 |
| wgrant | lifeless: My other scripts are reasonably happy, so go ahead. | 08:50 |
| wgrant | I fully support any measures to make LP less slow. | 08:52 |
| === almaisan-away is now known as al-maisan | ||
| mrevell | Hey up | 09:01 |
| wgrant | Can I get an EXPLAIN ANALYZE executed, or is that a stub thing? | 09:01 |
| spm | wgrant: any of the losas can do it for you. fire away. | 09:03 |
| wgrant | spm: Can you EXPLAIN ANALYZE the second query on http://pastebin.com/VsQpzqUq, please? | 09:04 |
| spm | wgrant: http://paste.ubuntu.com/465800/ | 09:09 |
| wgrant | spm: Which DB was that? main master? | 09:10 |
| spm | one of the slaves | 09:11 |
| wgrant | Ah. Thanks. | 09:11 |
| lifeless | spm: it was running on main | 09:41 |
| lifeless | spm: you might want to check its the same? | 09:42 |
| lifeless | actually nevermind | 09:42 |
| lifeless | 7.7secs is 7.7secs | 09:42 |
| wgrant | It's about a second slower, so it's pretty simialr. | 09:42 |
| spm | heh, i'd be impressed if that made a difference, and no, looks near as identical. | 09:43 |
| spm | 7.9 vs 7.7; within the bounds of load/server abuse | 09:43 |
| wgrant | There are some recipe builds redispatching constantly, since they are targetted to series without chroots. | 09:58 |
| wgrant | https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx | 09:59 |
| bigjools | rockstar: can you fix that? | 10:04 |
| * bigjools sighs at people making builds for breezy, hoary and warty | 10:05 | |
| bigjools | the UI should have blocked that | 10:05 |
| wgrant | It does. | 10:05 |
| wgrant | I bet the API doesn't, though. | 10:05 |
| rockstar | wgrant, why are they targetted to series without chroots? | 10:06 |
| wgrant | But we should fail builds if the chroot's missing, I think. | 10:06 |
| bigjools | rockstar: see the series I mentioned | 10:06 |
| wgrant | rockstar: Because Ted thought it was fun, I guess. | 10:06 |
| rockstar | bigjools, the UI does block that, so I guess it was an API thing? | 10:06 |
| bigjools | validation in browser code is largely useless now we have the API :/ | 10:06 |
| bigjools | yay for fat models | 10:07 |
| bigjools | rockstar: yeah it must be | 10:07 |
| wgrant | bigjools: Also, we have a whole lot of impossible builds being created in the primary archive when security updates with failed builds are copied. | 10:07 |
| rockstar | bigjools, truthfully, that should probably take place in the model anyway. | 10:07 |
| rockstar | wgrant, can you file a bug? | 10:07 |
| wgrant | They get marked as failed immediately, but they clutter up build listings and probably shouldn't exist at all. | 10:07 |
| * rockstar does not have a shortage of recipe bugs to fix | 10:08 | |
| bigjools | heh | 10:08 |
| wgrant | rockstar: One for preventing creation, and another for dealing with broken ones? | 10:08 |
| wgrant | (since they can become broken after creation) | 10:08 |
| rockstar | wgrant, I think the broken ones is probably a buildmaster issue, right? | 10:08 |
| wgrant | rockstar: No. You pick the chroot in lp.code somewhere, and that'd be what's crashing. | 10:09 |
| wgrant | Also, what's going on with porting SPRBs and TTBJs to the new model? | 10:09 |
| bigjools | it needs to fail the build | 10:09 |
| bigjools | in progress | 10:10 |
| wgrant | Excellent. | 10:10 |
| bigjools | the latter is pronounced as "Titty Bee Jay" BTW | 10:10 |
| wgrant | Heh. | 10:11 |
| bigjools | rockstar: so those builds that won't ever complete, you should poke some SQL at the LOSAs to remove them | 10:11 |
| bigjools | and add a "cancel" button to the UI | 10:11 |
| poolie | raising the tone | 10:11 |
| wgrant | There's a cancel branch in progress, right? | 10:11 |
| rockstar | bigjools, there is a cancel button already for admins. | 10:11 |
| wgrant | I thought I saw it floating around last week. | 10:11 |
| bigjools | ummm | 10:11 |
| rockstar | losa ping | 10:11 |
| bigjools | rockstar: why only admins? | 10:12 |
| mthaddon | rockstar: hi | 10:12 |
| bigjools | should be the user, and buildd-admins | 10:12 |
| rockstar | bigjools, because that's what I wrote the code to do. | 10:12 |
| bigjools | :) | 10:12 |
| wgrant | What does it do when the build is in progress or finished? | 10:12 |
| rockstar | wgrant, where are these builds? | 10:12 |
| poolie | wgrant, i thought <https://code.edge.launchpad.net/~leonardr/launchpad/true-anonymous-access/+merge/30088> might please you | 10:12 |
| wgrant | rockstar: the three pending builds on https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx | 10:12 |
| wgrant | poolie: Indeed, I was pleased to see that. | 10:13 |
| rockstar | mthaddon, I have some builds that probably need to be cancelled. | 10:13 |
| bigjools | rockstar: what does the cancel button do BTW? mark them with a new status or delete them? | 10:13 |
| rockstar | mthaddon, https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/175 https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/176 and https://code.edge.launchpad.net/~ted/+recipe/indicator-applet-dx/+build/177 | 10:14 |
| * rockstar makes note to head punch ted | 10:14 | |
| wgrant | I'm somewhat concerned that Code is doing things that are impossible. | 10:14 |
| rockstar | bigjools, I originally was deleting the build, but abentley thumped me, so I'm just changing the status now and deleting the buildqueue record if there is one. | 10:14 |
| wgrant | Like deleting branches related to recipes related to builds. | 10:14 |
| mthaddon | rockstar: what is "wrong" with what ted has done here? | 10:14 |
| wgrant | And cancelling builds that are in progress. | 10:14 |
| rockstar | mthaddon, requesting builds against unsupported distros. | 10:15 |
| bigjools | rockstar: great, sounds fine, with the caveat that wgrant is raising | 10:15 |
| mthaddon | rockstar: should he not be allowed to do that? if so is that a bug that we let him do that? | 10:15 |
| wgrant | mthaddon: Yes, bug #607125 | 10:16 |
| _mup_ | Bug #607125: Forbid creation of recipe builds for obsolete series through the API <Launchpad Bazaar Integration:New> <https://launchpad.net/bugs/607125> | 10:16 |
| rockstar | I see your caveat, and raise a "well, it's either that, or assume the buildmaster will fix it, which has been a bad assumption so far" | 10:16 |
| mthaddon | thx wgrant | 10:16 |
| * mthaddon adds to losa watched bugs | 10:16 | |
| wgrant | rockstar: Well, cancelling an in progress build will at the moment either not work, or not work and knock the builder out until somebody notices that it has been idle for weeks. | 10:16 |
| rockstar | wgrant, there should be a cancel button on there now though. | 10:16 |
| rockstar | wgrant, this is why we left it up to the admins. The reason it's here is because builds keep getting re-tried even though they are always failing. | 10:17 |
| bigjools | we have some work to do to cancel in-progress builds | 10:17 |
| mthaddon | rockstar: those have been cancelled | 10:17 |
| bigjools | rockstar: can you change that permission to buildd-admins please | 10:17 |
| rockstar | mthaddon, great, thanks. | 10:17 |
| rockstar | bigjools, you can... :) | 10:17 |
| bigjools | rockstar: I would but we have all these buildd-manager bugs you keep exposing ;) | 10:18 |
| * rockstar refrains from "it's open source" comment. | 10:18 | |
| * bigjools might do the same with the next buildd-manager bug | 10:18 | |
| rockstar | bigjools, this is a hell of a pissing match... "My code is more broken than yours!" "Nuh uh!" | 10:18 |
| bigjools | lol | 10:19 |
| wgrant | Ah, it already raises a CannotBuild when the chroot is missing. | 10:19 |
| wgrant | But that just causes the dispatch to do nothing. | 10:19 |
| wgrant | So it's going to be retried on the next run. | 10:19 |
| rockstar | wgrant, ...unless we've cancelled the build. | 10:20 |
| wgrant | rockstar: Right. | 10:20 |
| rockstar | wgrant, "Cancel build" is there so that we don't have to make the losas run SQL. | 10:20 |
| wgrant | I speak of the general case. | 10:20 |
| rockstar | It shouldn't have to exist at all. | 10:20 |
| bigjools | one of my b-m bugs is to detect failures and work out whether to fail the job or the builder | 10:20 |
| rockstar | Luckily, spr is only available on edge. | 10:20 |
| * bigjools cringes at the 2nd meaning of SPR now :( | 10:21 | |
| rockstar | bigjools, maybe just see if the builder failed the last 5 builds before the builder commits seppuku. | 10:21 |
| bigjools | rockstar: that's the exact plan | 10:21 |
| rockstar | bigjools, my use of SPR is more important now. | 10:21 |
| bigjools | it can see if it's the same builder or the same job | 10:21 |
| * rockstar only works on important things | 10:21 | |
| wgrant | bigjools: We also need to fire the reset trigger if a builder is being stupid. | 10:21 |
| bigjools | haha | 10:21 |
| wgrant | bigjools: eg. during an abort. | 10:21 |
| wgrant | Or when we get a <Fault 8002: error> | 10:21 |
| bigjools | I have a branch that does that, but I backed it out | 10:22 |
| wgrant | :( | 10:22 |
| rockstar | wgrant, also, recipe builds are about to be able to be re-scored as well. | 10:22 |
| bigjools | wgrant: it disabled builders that were not resetting quick enough :/ | 10:22 |
| wgrant | bigjools: Oh, right. | 10:22 |
| wgrant | bigjools: Once we no longer need to preserve ddebs on the buildd filesystem, is there any need for the virt/non-virt distinction? I think the architecture restriction magic can handle it all. | 10:24 |
| bigjools | well yes | 10:25 |
| wgrant | It would also mean we wouldn't have to lie about ivy and kaylaberry. | 10:25 |
| bigjools | we still need a split between non-virt and virtual i386/and64 | 10:26 |
| bigjools | amd* | 10:26 |
| wgrant | Why? | 10:26 |
| wgrant | Once our queue discipline doesn't suck, there's no reason to. | 10:26 |
| rockstar | bigjools, it'd be nice if we could return the spr builds to be "anything but arm" instead of i386 only. | 10:26 |
| bigjools | archive restriction won't work there | 10:26 |
| wgrant | bigjools: Why not? | 10:26 |
| bigjools | rockstar: agreed | 10:26 |
| bigjools | wgrant: because we don't want PPA builds ending up on distro builders | 10:27 |
| wgrant | bigjools: But the distro builders don't need to be special. | 10:27 |
| bigjools | yes, they do | 10:27 |
| wgrant | Whyso? | 10:27 |
| bigjools | they're not PPA builders | 10:27 |
| bigjools | distro has its own hardware | 10:27 |
| wgrant | Ah, so it is political. Yay. | 10:28 |
| bigjools | besides, not resetting the builder shaves 30 seconds off the build time | 10:28 |
| wgrant | Anyway, regarding the "anything but arm" thing: we could just leave them on i386 and let amd64 builders build i386 and lpia as well, as has been planned for years. | 10:30 |
| bigjools | yeah there's a bug about that | 10:31 |
| bigjools | it would need a bit of work server-side :) | 10:32 |
| wgrant | Not really. | 10:32 |
| wgrant | The slave side is done, but not merged. | 10:32 |
| wgrant | And the master is simple enough. | 10:32 |
| wgrant | Although it might make /builders a little odd. | 10:37 |
| lifeless | mthaddon: hi | 10:45 |
| mthaddon | lifeless: hi | 10:45 |
| lifeless | I has a new config CP on the wiki page | 10:45 |
| mthaddon | lifeless: you've added it to ones already done rather than requested ones | 10:46 |
| lifeless | oh darn | 10:46 |
| lifeless | fixing | 10:46 |
| mthaddon | lifeless: and since this is an edge one, it's not really a cherry pick - just a request for an edge rollout | 10:46 |
| lifeless | mthaddon: ok | 10:47 |
| lifeless | mthaddon: can you please rollout edge then? Sorry about not getting the process right :( | 10:47 |
| bigjools | wgrant: the hard part is the build dispatch ETA | 10:48 |
| wgrant | bigjools: Yeah. But arch-agnostic builds completely break that too. I think we need a new way to do things. | 10:51 |
| mthaddon | lifeless: ugh, we don't have a new commit to "stable" since the last edge rollout, so it'd make a rollout currently a very manual process | 10:51 |
| bigjools | wgrant: they don't break it if they end up in the same arch-indep builders each time | 10:51 |
| bigjools | it's when the arch is indeterminate, mixed with determinate builds that you have an issue | 10:52 |
| bigjools | queueing theory is a bitch | 10:52 |
| wgrant | Yes :( | 10:52 |
| lifeless | mthaddon: here's what I'd like to do: I want to drop the edge timeout to 12 seconds. The ppr suggests 10 might be safe, but I don't totallly trust it. | 10:53 |
| lifeless | mthaddon: I'd like to find out during our work day if that change is an issue so we can rollback in a controlled fashion. | 10:54 |
| mthaddon | lifeless: the next auto rollout would be tomorrow morning (assuming there's a new landing to stable between now and then), so how about that? | 10:55 |
| lifeless | mthaddon: it would make me sad. But if thats the best we can do, its the best we can do. | 10:58 |
| lifeless | mthaddon: also, its a bit risky. | 10:58 |
| lifeless | mthaddon: because if the auto rollout happens, and its bad, we're in the poo, and I explicitly want to avoid that. | 10:58 |
| mthaddon | lifeless: there are LOSAs around during the auto-rollout, or if someone lands a change to stable between now and then, kicking off a rollout is fairly trivial | 10:59 |
| lifeless | mthaddon: could you just do an empty commit then, from praseodymium ? | 10:59 |
| lifeless | checkout lp:launchpad/stable; bzr commit --allow-unchanged ? | 11:00 |
| mthaddon | lifeless: I *could* but this is more "working around our process rather than fixing our process" which makes me sad | 11:00 |
| lifeless | mthaddon: it makes me sad too | 11:00 |
| lifeless | mthaddon: I don't understand why our process has this limitation. | 11:00 |
| lifeless | Can you tell me about the limitation ? | 11:00 |
| mthaddon | lifeless: the limitation is that we want to be able to push and build code without taking down the current app server - we do that by pushing to a new directory | 11:01 |
| mthaddon | lifeless: the name of the new directory is determined by the revno of the top level bzr dir in the code tree | 11:01 |
| mthaddon | lifeless: so unless we have a commit to that, we'd be pushing over the top of the existing directory | 11:01 |
| mthaddon | lifeless: which our deployment scripts would stop from allowing | 11:02 |
| lifeless | right | 11:03 |
| mthaddon | s/would stop from allowing/wouldn't allow/ | 11:03 |
| lifeless | that would be bad. | 11:03 |
| lifeless | so, a few things occur | 11:03 |
| lifeless | we could make the top level number be a composite - lprevno,configrevno | 11:03 |
| lifeless | we could use a datestamp - lprevno,$time | 11:03 |
| lifeless | or even $time | 11:03 |
| lifeless | do you have any other ideas? | 11:04 |
| mthaddon | lifeless: that would be problematic in terms of how the deployment scripts handle other projects (we don't just use it for LP), but the lprevno,$time would work | 11:04 |
| lifeless | ok, where is the script, so that I can put a patch together | 11:04 |
| mthaddon | I mean the lprevno,configrevno wouldn't work, but lprevno,$time would work | 11:04 |
| mthaddon | lifeless: lp:~canonical-losas/deployment-manager/trunk | 11:04 |
| lifeless | mthaddon: RepositoryFormatKnitPack1 - you could benefit by upgrading that :) | 11:06 |
| wgrant | bigjools: Is there a reason that external_dependencies are between the archive itself and the other archive dependencies in sources.list? Would the world explode if I moved it to the end (it would make some refactoring simpler)? | 11:07 |
| bigjools | I think so, let me remember | 11:07 |
| wgrant | It's possible that people are relying on it. But that's a bit crazy :/ | 11:08 |
| * mthaddon does it through the UI | 11:08 | |
| bigjools | I think they are - it was for OEM's migration | 11:08 |
| wgrant | I guess I could just insert them after the first line. | 11:08 |
| bigjools | mthaddon: is that a bumper sticker? :) | 11:08 |
| bigjools | wgrant: what are you changing? | 11:08 |
| mthaddon | bigjools: no, that's "LOSAs do it through the UI" :) | 11:09 |
| wgrant | bigjools: https://code.edge.launchpad.net/~wgrant/launchpad/bug-598345-restrict-dep-contexts/+merge/30203 | 11:09 |
| bigjools | mthaddon: :D | 11:09 |
| bigjools | wgrant: yay! | 11:10 |
| bigjools | btw I have a new buildd-manager that I finished (sorta)} last week | 11:10 |
| lifeless | wgrant: so, please tell me you're going to fix the soyuz api :) | 11:10 |
| wgrant | Yeah, I saw that. Looks good. | 11:10 |
| wgrant | lifeless: Hm? The timeouts? | 11:11 |
| bigjools | wgrant: are you stalking me? I didn't even make an MP yet! | 11:11 |
| lifeless | yes, in particular | 11:11 |
| wgrant | bigjools: I guess I'll just change it to insert the external deps after the first real dep entry. | 11:11 |
| wgrant | bigjools: I stalked lots of LP branches over the weekend :P | 11:11 |
| bigjools | lifeless: we need a way of jobbing copy-packages before we get rid of the major source of timeouts ;) | 11:12 |
| lifeless | bigjools: you could use the jobs system | 11:12 |
| lifeless | no ? | 11:12 |
| bigjools | yes - but it would be a lot of work | 11:12 |
| lifeless | how is it done now ? | 11:12 |
| bigjools | all synchronously in the appserver | 11:12 |
| lifeless | aiieeee | 11:12 |
| bigjools | it issues a shitload of queries | 11:13 |
| bigjools | the copy checking is very complicated | 11:13 |
| wgrant | lifeless: This is Launchpad. Launchpad's motto is "Let's do everything in the appserver until it times out, then ignore it for a while after it starts to." | 11:13 |
| bigjools | but it is well-factored so it's just a case of UI changes | 11:13 |
| wgrant | bigjools: But isn't a lot of the time taken in verifying that the copy can actually happen successfully? | 11:14 |
| bigjools | wgrant: yes, that's exactly what I am talking about | 11:14 |
| lifeless | wgrant: that was last month. | 11:14 |
| bigjools | the copy itself is trivial | 11:14 |
| bigjools | the checking - not so much | 11:14 |
| lifeless | rabbit is now available. | 11:14 |
| wgrant | Hmm. Doing that asynchronously sounds a bit awkward. | 11:14 |
| lifeless | the jobs system is pretty good even without rabbit. | 11:14 |
| bigjools | lifeless: I hope you saw my comment on that bug about more daemons | 11:15 |
| wgrant | lifeless: There isn't really a jobs "system"... | 11:15 |
| lifeless | bigjools: I did, but I don't have an answer other than 'yes, we should fix it so we don't monkey with apache, pgsql, etc/hosts, etc etc etc' | 11:15 |
| bigjools | lifeless: the problem is endemic | 11:15 |
| lifeless | agreed | 11:16 |
| lifeless | this is why I develop lp in a VM | 11:16 |
| bigjools | that's not a cure, it's pain relief | 11:16 |
| lifeless | If you have suggestions for how to develop software using rabbit without having rabbit present, I'm all ears. | 11:16 |
| bigjools | rabbit ears? | 11:16 |
| StevenK | Bwahaha | 11:16 |
| bigjools | we just need a way of selectively starting this stuff when needed | 11:16 |
| bigjools | during make run, or during the tests | 11:17 |
| lifeless | in this case, perhaps file a bug on rabbitmq in debian | 11:17 |
| wgrant | bigjools: Is buildd-manager really so slow at the moment because of process-upload.py, or is the candidate selection query also taking a while? | 11:17 |
| lifeless | the package being how it is isn't really my fault :) | 11:17 |
| bigjools | I am really fed up of more daemons running in the background :( | 11:17 |
| lifeless | bigjools: we shouldn't/can't use the system rabbit for tests anyhow. | 11:17 |
| bigjools | lifeless: it's not unreasonable that it starts when you install it | 11:17 |
| lifeless | bigjools: please don't think I'm advocating having it running all the time: I'm not. | 11:18 |
| bigjools | but the lp packages should turn it off | 11:18 |
| bigjools | wgrant: most of it is upload | 11:18 |
| lifeless | bigjools: that would worry me; too much chance of turning it off on prod. | 11:18 |
| bigjools | the next slowest part is dispatching (it takes 10 seconds to send all the files to the builders) | 11:18 |
| bigjools | candidate selection is very fast | 11:18 |
| wgrant | bigjools: Hmm. But there are only 59 builders... and it gets out to 15 minutes sometimes. | 11:19 |
| bigjools | lifeless: make start should start it | 11:19 |
| lifeless | no | 11:19 |
| bigjools | ....... | 11:19 |
| lifeless | we should not use the system instance at all | 11:19 |
| lifeless | make start should run a private instance using the system binary | 11:19 |
| bigjools | wgrant: yeah that's upload time | 11:19 |
| wgrant | Hm, I guess that is Hm, I guess that only means 15s of upload time, yeah. | 11:19 |
| lifeless | which may be what you mean | 11:19 |
| wgrant | Crazy. | 11:19 |
| bigjools | lifeless: yep | 11:19 |
| lifeless | althought, again, make start on prod - we have to be very careful not to mess up there. | 11:20 |
| wgrant | lifeless: But we do the same sort of thing with the librarian, so there's precedent. | 11:20 |
| lifeless | wgrant: the way we use the librarian causes lots of headaches, testing included. | 11:20 |
| wgrant | lifeless: True. | 11:21 |
| lifeless | mthaddon: I am admiring of your unit tests :P | 11:21 |
| mthaddon | lifeless: er.... yeah | 11:22 |
| bigjools | I'd really like "make start" to start everything it should do, given the LPCONFIG | 11:23 |
| jml | I'd like to not start Launchpad using make targets. | 11:30 |
| lifeless | jml: what would you like to start using make targets? | 11:31 |
| jml | lifeless, are you trolling? | 11:31 |
| lifeless | only a little | 11:32 |
| jml | lifeless, well, I don't understand the question. But to elaborate on what I meant, I'd like to not have make targets for running Launchpad or its bits, and instead use actual scripts. | 11:32 |
| jml | maybe even the same scripts mthaddon uses! | 11:32 |
| lifeless | we should talk about this later | 11:33 |
| jml | lifeless, yes. | 11:34 |
| jml | lifeless, after you review my branch, for example :P | 11:34 |
| bigjools | my point was that I don't want to think about what to start | 11:34 |
| lifeless | I was thinking after LP is fast | 11:34 |
| jml | lifeless, please review my branch before then. | 11:34 |
| lifeless | haha | 11:34 |
| lifeless | seriously though, if someone else reviews it its all good too. | 11:35 |
| lifeless | uhm, those logouts shouldn't be needed now though, should they ? | 11:35 |
| bigjools | lifeless: are you lowering hard or soft timeouts on edge? or both? | 11:36 |
| bigjools | lifeless: also have you notified launchpad-users? | 11:37 |
| bryceh | lifeless, LP is fast??? | 11:37 |
| james_w | what they did for couchdb was make a couchdb-bin package with all the binaries, and a couchdb package with the init script. | 11:47 |
| james_w | doing a similar thing would be l-dev-deps could depend on rabbitmq-bin | 11:47 |
| bigjools | +1 | 11:48 |
| bigjools | james_w: while you're there! can you check out the LEP for derived distros please? | 11:48 |
| bigjools | in particular please make sure we've captured all your requirements | 11:49 |
| === al-maisan is now known as almaisan-away | ||
| wgrant | Can someone please land lp:~wgrant/launchpad/remove-obsolete-buildd-junk? | 11:58 |
| jml | wgrant, sure, if it's approved. | 11:59 |
| wgrant | jml: It is indeed approved. | 11:59 |
| lifeless | bigjools: prod can't go lower until we deploy edge to prod | 11:59 |
| lifeless | bigjools: because its got many more timeouts as a baseline | 12:00 |
| jml | wgrant, running ec2 land now. | 12:00 |
| wgrant | jml: Thanks. | 12:00 |
| jml | np | 12:00 |
| bigjools | lifeless: that's not what I asked! :) you know soft vs hard? | 12:00 |
| lifeless | yes | 12:00 |
| lifeless | sorry, got ELOCAL for a sec | 12:01 |
| lifeless | anyhow, edge - dropping both | 12:01 |
| lifeless | bryceh: that is the goal. | 12:01 |
| lifeless | deryck: hai | 12:05 |
| deryck | lifeless, hi | 12:05 |
| lifeless | deryck: If you guys have any slack, or the ability to deal with oops and near-oops, I have a few things for you :) | 12:06 |
| deryck | lifeless, yes, we can deal with oops and near-oops. | 12:06 |
| lifeless | deryck: so, I sent you a mail listing 4 pages that are near to timing out on prod | 12:07 |
| lifeless | but also | 12:07 |
| wgrant | Is stub around this week? | 12:07 |
| lifeless | on friday night there was a big boom | 12:07 |
| lifeless | wgrant: once he gets home, yes. | 12:08 |
| wgrant | lifeless: Thanks. | 12:08 |
| lifeless | deryck: on friday night, one API call went 8boom8 | 12:08 |
| deryck | lifeless, re: mail. I marked it to look at today, and will respond with comments. But generally, making those a priority is welcome and not a problem for me. | 12:08 |
| lifeless | 4000 times | 12:08 |
| deryck | lifeless, I saw that. The message stuff, right? Same API stuff bryceh was asking about at the epic, right? | 12:08 |
| lifeless | poolie and jtv have done some analysis | 12:08 |
| lifeless | I guess, yes. | 12:09 |
| deryck | lifeless, yeah. gmb is testing a fix as we type. :-) | 12:09 |
| lifeless | awesome | 12:09 |
| bryceh | lifeless, haha | 12:09 |
| lifeless | bryceh: everyone wants it to be fast. | 12:10 |
| lifeless | so we shall be fast. | 12:10 |
| lifeless | deryck: https://lp-oops.canonical.com/oops.py/?oopsid=1659B1002 for instance | 12:10 |
| lifeless | deryck: poolie reckons its just the calculation of the canonical url per item | 12:10 |
| deryck | hey, there's bryceh :-) go to bed. ;) | 12:10 |
| * deryck is reading the OOPS | 12:11 | |
| bryceh | deryck, yeah, 4am == well past bedtime | 12:11 |
| === salgado is now known as salgado-lunch | ||
| deryck | lifeless, so having processed all the info now, I think gmb and I can chase up a fix today for this. And I'll respond in depth to the timeout email before EOD for me. | 12:17 |
| lifeless | awesome | 12:18 |
| lifeless | we're getting up some momentum on performance | 12:18 |
| lifeless | if we can just get a bit of a cadence going we'll be at 5 seconds in no time | 12:18 |
| gmb | deryck, lifeless: Have fix will travel: http://pastebin.ubuntu.com/465875/ | 12:18 |
| wgrant | lifeless: Crank the timeout down to 1s. | 12:18 |
| wgrant | Then see how quickly things get fixed :P | 12:18 |
| lifeless | wgrant: that would just fuck everyone over | 12:19 |
| wgrant | :( | 12:19 |
| gmb | There's already tonnes of test coverage. | 12:19 |
| lifeless | gmb: isn't messages.first() or something even cheaper? | 12:19 |
| gmb | lifeless, It might be; I can never remember the Storm runes. Let me try it. | 12:19 |
| lifeless | its not first, I don't remember them either | 12:20 |
| lifeless | food time | 12:20 |
| wgrant | bigjools: Am I OK to go through with https://code.edge.launchpad.net/~wgrant/launchpad/refactor-_dominateBinary/+merge/29667? It seems to be a good cleanup, and is somewhat necessary for ddeb stuff. | 12:20 |
| bigjools | my eyes | 12:21 |
| bigjools | what have you changed? | 12:22 |
| wgrant | Oh, right, I haven't added a description yet. Oops. | 12:22 |
| wgrant | You know the atomic arch-indep domination stuff? | 12:22 |
| bigjools | vaguely | 12:22 |
| bigjools | I've never done anything in that code | 12:23 |
| gmb | lifeless, FTR, first() appears to shave ~2s off test run time, though that could be the weather or the proximity of chickens to the developer for all I know. | 12:23 |
| bigjools | it's headfuck territory | 12:23 |
| wgrant | bigjools: Yes. Exactly. I'm trying to unheadfuck it. | 12:23 |
| wgrant | At the moment, publications have a supersede() method. | 12:23 |
| wgrant | It basically just sets datesuperseded and the status. | 12:24 |
| bigjools | wgrant: in tandem with my b-m changes, we might start unheadfucking soyuz | 12:24 |
| wgrant | The Dominator handles setting supersededby, and it also handles atomic domination rules. | 12:24 |
| wgrant | (atomic domination means that if an arch-indep publication is superseded, other publications of the same binary immediately get superseded too) | 12:24 |
| wgrant | This makes for exceedingly ugly code, and even uglier tests. | 12:25 |
| wgrant | So I've moved all of that into the model. | 12:25 |
| deryck | gmb, so like lifeless I think use .first, but otherwise this looks good to me. | 12:25 |
| wgrant | So Dominator just calls publication.supersede(dominant), and the model works out what else needs to die, sets supersededby, etc. Dominator just finds the candidates and tells them to do their stuff. | 12:25 |
| gmb | deryck, Cool, preparing a branch now. | 12:26 |
| wgrant | A subsequent branch replaces the tests with some that suck less. | 12:26 |
| gmb | s/branch/mp | 12:26 |
| wgrant | *Then* i can extend it to dominate ddebs properly, which is the whole aim of this. | 12:26 |
| deryck | gmb, I wonder if we should look at bug 606911 while we're here, and then cowboy to staging and hit with an api script to see if we're better? | 12:26 |
| _mup_ | Bug #606911: Bug.indexed_messages unused? <Launchpad Bugs:New> <https://launchpad.net/bugs/606911> | 12:26 |
| deryck | gmb, especially if that is just dropping unused code. | 12:26 |
| bigjools | wgrant: ok seems like a rational thing to do | 12:26 |
| bigjools | wgrant: my concerns are: performance and performance | 12:27 |
| wgrant | bigjools: No performance changes here. | 12:27 |
| wgrant | It just shuffles methods into the model and wraps them up. | 12:27 |
| bigjools | ok - I'd still like to do testing pre and post-branch on DF to make sure | 12:28 |
| gmb | deryck, Yeah, I'm thinking that too. | 12:28 |
| * bigjools doesn't apologise for being paranoid | 12:28 | |
| wgrant | Heh, OK. | 12:29 |
| bigjools | I hate finding my own code when searching on google for stuff | 12:30 |
| wgrant | Haha. | 12:30 |
| gmb | deryck, Ahaa. Except that indexed_messages is exported as 'messages'. | 12:30 |
| gmb | So that would be a big bag of fail. | 12:30 |
| wgrant | I hate finding all this Soyuz code that hasn't been touched by anyone who's still around, and is revolting. | 12:30 |
| gmb | wgrant, That's *why* they're not still around. | 12:31 |
| wgrant | gmb: Heh. | 12:31 |
| gmb | deryck, I suggest we just try cowboying the initial_message fix for now. | 12:31 |
| deryck | gmb, that's fine with me. | 12:32 |
| deryck | gmb, I can investigate the other bug today after I settle in. Then, you can go back to subscriptions hacking after this fix. | 12:33 |
| gmb | deryck, Okay. I'll work up an mp for the performance fix anyway. | 12:35 |
| deryck | gmb, yes, do. I wasn't suggesting dropping this fix. Just that you don't have to worry about the other bug for now. | 12:36 |
| gmb | deryck, Right. I managed to put the word "anyway" in there without meaning to. What I meant was "now" | 12:37 |
| deryck | heh, fair enough | 12:37 |
| lifeless | gmb: is it first () ? | 12:54 |
| gmb | lifeless, Yes | 12:54 |
| lifeless | hah, who knew :) | 12:54 |
| gmb | lifeless, Well intuited :) | 12:54 |
| lifeless | gmb: you win some, you win some more | 13:02 |
| jml | wgrant, oops. forgot to update download-cache. trying again. | 13:03 |
| jml | wait. no. | 13:03 |
| jml | ec2 land has been broken by recent changes to Launchpad. | 13:05 |
| lifeless | gmb: https://code.edge.launchpad.net/~gmb/launchpad/bug-606914/+merge/30259 | 13:05 |
| === mrevell is now known as mrevell-lunch | ||
| wgrant | jml: Awesomeness. | 13:06 |
| gmb | lifeless, Thanks. Will remove the whitespace; bad habit. | 13:07 |
| jml | wgrant, I'm going to fix those before landing your branch. | 13:07 |
| lifeless | gmb: my overly general rule of thumb is 'if it needs vws in a function, it probably needs two functions' | 13:07 |
| lifeless | jml: what changes broke ec2land? | 13:08 |
| gmb | lifeless, Good rule :) | 13:08 |
| jml | lifeless, a change to the webservice. | 13:08 |
| lifeless | jml: .next() | 13:08 |
| jml | lifeless, AttributeError: 'Entry' object has no attribute 'queue_status' | 13:08 |
| lifeless | grah | 13:08 |
| lifeless | thanks | 13:08 |
| jml | lifeless, I think jkakar has a patch for better testing support with launchpadlib | 13:09 |
| lifeless | jml: a bit, yes. | 13:09 |
| lifeless | he's trying to get it landed | 13:10 |
| lifeless | but launchpadlib is a bit single-writer-gated atm | 13:10 |
| jml | hmm. | 13:11 |
| poolie | jam OOPS-1661ED1057 | 13:11 |
| poolie | hi there deryck | 13:11 |
| poolie | thanks for looking at the attachment bug | 13:11 |
| jml | queue_status is still being exported :\ | 13:11 |
| jml | benji, good morning. | 13:12 |
| benji | good morning, jml; how goes it? | 13:12 |
| lifeless | gmb: when you finish with that, I have a question on your reset watches stuff | 13:12 |
| jml | benji, alright. | 13:12 |
| jml | fixing bugs. not taking prisoners. | 13:13 |
| deryck | hi poolie. np. glad the fix was easy. | 13:13 |
| deryck | gmb did the heavy lifting :-) | 13:13 |
| jml | benji, there are a few pending patches from Launchpadders to zope.testing. would you be interested in reviewing & landing them? | 13:15 |
| jml | (if not, I'll try to find sidnei) | 13:16 |
| === almaisan-away is now known as al-maisan | ||
| gmb | lifeless, Sure, ask. | 13:16 |
| benji | glad to take a look | 13:16 |
| jml | benji, https://code.edge.launchpad.net/zope.testing/+activereviews | 13:16 |
| lifeless | who will be looking for bad attempts to reset bug watches | 13:16 |
| jml | lifeless, wgrant: false alarm. it's my bug. | 13:17 |
| gmb | lifeless, I'm not sure what you mean. Do you mean "who will be looking to see which bug trackers need their watches resetting?" or "who will be looking to make sure no-one maliciously resets watches?" | 13:17 |
| gmb | Or something else? | 13:18 |
| lifeless | the second | 13:18 |
| poolie | deryck, i'd be interested to review the result when it's up for review, for my own education | 13:19 |
| gmb | lifeless, Any UserCannotResetWatchesErrors should appear in the OOPS report, shouldn't they? So we should spot them fairly quickly. | 13:19 |
| deryck | poolie, ok, it's up. let me find the link. | 13:19 |
| gmb | lifeless, Only LP Devs and Admins are able to do the resetting or for that matter see the reset button at all. | 13:19 |
| deryck | poolie, https://code.edge.launchpad.net/~gmb/launchpad/bug-606914/+merge/30259 | 13:20 |
| poolie | got it thanks | 13:20 |
| lifeless | gmb: who is going to be *looking* for that though | 13:20 |
| lifeless | its going to show up in 'exceptions' | 13:20 |
| lifeless | gmb: and how is it different to any attempt to circumvent permissions - e.g. creating a new private branch, viewing a private bug etc. | 13:21 |
| gmb | lifeless, I'm not sure that it is different... Ah, but, instead of the custom Exception it would make more sense to raise the same kind of exception as the other attempts to circumvent permissions do. | 13:21 |
| gmb | I don't know if anyone actively looks for those errors, but I think it's a safe bet that someone does because they get reported fairly quickly. | 13:22 |
| gmb | Let me do some digging. | 13:22 |
| poolie | gmb then i suspect bug 424671 won't be fixed by this | 13:23 |
| _mup_ | Bug #424671: attachments: accessing attachment's message is very slow <api> <Launchpad Foundations:Triaged by leonardr> <https://launchpad.net/bugs/424671> | 13:23 |
| poolie | but it's similar | 13:23 |
| poolie | as you probably already know | 13:23 |
| lifeless | gmb: I think the normal permission denied behaviour is really all that is needed | 13:24 |
| lifeless | gmb: unless you have some particularly special horror story we're expecting | 13:24 |
| gmb | lifeless, No, I just wrote that stuff about half an hour before getting on a plane and didn't think about it very hard :) | 13:25 |
| lifeless | :) | 13:25 |
| gmb | I'll change it appropriately. | 13:25 |
| gmb | poolie, I agree that they're similar but I think they should be fixed separately (if only because I knew exactly how to fix the first bug but wasn't sure I grokked a solution for the second). | 13:27 |
| lifeless | if it fixes the timeout, I'm happy | 13:27 |
| lifeless | there are 4 other near-timeouts that can queue between 'better' and 'perfect' for this API, IMHO :) | 13:27 |
| jml | vocabularies.txt is 1428 lines long. | 13:28 |
| wgrant | jml: Don't look at archive.txt... | 13:29 |
| jml | I probably will eventually. | 13:29 |
| jml | I'm on a quest to remove all the imports in canonical.launchpad.interfaces.__init__ | 13:30 |
| wgrant | ahahah. | 13:30 |
| jml | my current plan is to do it one import at a time | 13:31 |
| jml | but to fix all of the imports in every file I touch | 13:31 |
| jml | so that it gets easier as I go along | 13:31 |
| wgrant | Right. | 13:31 |
| lifeless | wgrant: your VWS in supersede doesn't really gel for me - please remove or comment | 13:36 |
| * gmb -> lunch | 13:37 | |
| wgrant | lifeless: What's the issue with it? | 13:39 |
| lifeless | its not clear why these lines need special attention drawn to them | 13:39 |
| wgrant | Should I remove all blank lines from both methods? | 13:40 |
| lifeless | I generally follow pep8 here | 13:41 |
| lifeless | by which I mean 'yes, unless you really want special attention' | 13:41 |
| wgrant | Hm, OK. | 13:41 |
| lifeless | http://www.python.org/dev/peps/pep-0008/ | 13:41 |
| lifeless | " Use blank lines in functions, sparingly, to indicate logical sections. | 13:42 |
| lifeless | " | 13:42 |
| wgrant | They are, in my opinion, indicating logical sections. | 13:42 |
| wgrant | It's unfortunate that one of those sections is a super() call, and that it has another section on top of it. | 13:42 |
| lifeless | thus my request to add a comment :) | 13:43 |
| wgrant | Like what? "# Set date and status."? | 13:44 |
| lifeless | wgrant: if its self evident, I don't understand why its really a different section; if its not self evident, surely it should have a comment? | 13:49 |
| lifeless | wgrant: however, its up to you, this isn't a blocking issue. | 13:49 |
| === mrevell-lunch is now known as mrevell | ||
| wgrant | lifeless: I don't think it deserves its own section. But the segments of code above and below it do. | 13:53 |
| wgrant | lifeless: Thanks for the reviews. | 13:54 |
| lifeless | de nada | 13:57 |
| === Ursinha-afk is now known as Ursinha | ||
| === salgado-lunch is now known as salgado | ||
| wgrant | lifeless: Are you landing those, or must I coerce others? | 14:07 |
| lifeless | coerce | 14:07 |
| wgrant | k | 14:07 |
| === vednis is now known as mars | ||
| === beuno_ is now known as beuno | ||
| wgrant | Can someone please ec2 land lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts, lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/really-publish-ppa-ddebs? | 14:27 |
| lifeless | deryck: I like your team strat | 14:49 |
| deryck | lifeless, for testing? | 14:49 |
| lifeless | and cycle time | 14:49 |
| deryck | ah, right. Cool. | 14:50 |
| lifeless | jml: busy? | 14:51 |
| lifeless | jpds: if you need a hand on https://code.edge.launchpad.net/~jpds/launchpad/fix_116279/+merge/17940 gimme a shout | 14:54 |
| mars | james_w, ping | 15:08 |
| jml | lifeless, hi | 15:13 |
| lifeless | jml: where could I find you? | 15:13 |
| jml | lifeless, community room | 15:13 |
| jml | gary_poster, mars: hello. there's a doctest issue? | 15:14 |
| mars | jml, looking | 15:14 |
| gary_poster | jml: hey. Did you see my discussion about the number of doctests in #launchpad-code | 15:14 |
| gary_poster | With mars | 15:14 |
| jml | gary_poster, it's gone down, right? | 15:14 |
| gary_poster | That would be the fastest way to catch up | 15:14 |
| gary_poster | right | 15:14 |
| jml | ok. | 15:14 |
| gary_poster | and I think I know why, and that it is ok | 15:14 |
| jml | oh good. :) | 15:14 |
| gary_poster | but would love confirmation | 15:14 |
| mars | looking now | 15:14 |
| gary_poster | yeah, cool | 15:15 |
| mars | simple summary compare | 15:15 |
| jml | gary_poster, can you tell me why you think why? | 15:15 |
| jml | rockstar, I'll catch up w/ you re private branch lookup after this. | 15:15 |
| gary_poster | jml: "buildbot went from reporting 26777 tests to 8490 tests when jml landed the branch to use stdlib doctest. It looks like it might be OK: the amount of time didn't change, so I think this is because Zope's doctest counts each block as a test, while the normal doctest code counts the whole thing as one. " | 15:15 |
| jml | yeah, that sounds right. | 15:16 |
| jml | gary_poster, mars, I don't really know how I'd confirm that, other than comparing the fulltext buildbot output. | 15:17 |
| mars | yep, doing so | 15:18 |
| mars | test lines before: 8941 | 15:18 |
| mars | test lines after: 8976 | 15:18 |
| mars | so yes, a) zope counts doctests multiple times, and b) our test suite runtime sucks three times more than I thought it did | 15:19 |
| benji | it's not quite that zope's doctest counts tests multiple times, its that it counted tests slightly differently than the stdlib doctest | 15:20 |
| mars | yep | 15:21 |
| mars | didn't notice that before | 15:21 |
| mars | by the way, I found that a good sized part of our testrunner time is being spent parsing doctests | 15:27 |
| mars | which is a pity because it happens in the test discovery phase, and after the doctest is discovered, then the new layer's subprocess means that the work has to be done again | 15:29 |
| benji | that surprises me; doctests should be very cheap to parse | 15:29 |
| mars | I tried profiling the layer setup, and doctest parsing was a big part of it | 15:30 |
| mars | it was volume more than anything | 15:30 |
| lifeless | \o/ | 15:31 |
| lifeless | benji: 'parse' + 'cheap' does not exist in python. | 15:31 |
| jml | rockstar, I've commented on the bug for private branches. I figured that was the best way. I'm available for face-to-face convo if you'd like one. | 15:32 |
| mars | lifeless, that is part of that mysterious uncounted timesink when you first fire up the runner. I couldn't isolate .pyc cleanup due to disk caching issues. | 15:34 |
| benji | btw, mars: remember last week when we were wondering if the lower-memory EC2 instances would work for ec2 test? I watched one of the instances when running the tests and it ended up using 2.6G of RAM, of which 1.3G was cache/buffers | 15:34 |
| lifeless | mars: nice | 15:34 |
| mars | I also suspect the importfascist may be sinking some time, again due to volume of use | 15:35 |
| mars | benji, what does 50% memory use as cache/buffer percent mean? | 15:36 |
| benji | that's primarily disk cache | 15:36 |
| StevenK | That the kernel is using it as a disk buffer | 15:36 |
| poolie | salgado, ping for review of https://code.launchpad.net/~leonardr/launchpad/true-anonymous-access/+merge/30088 | 15:36 |
| leonardr | poolie, salgado says it's ok | 15:37 |
| poolie | sweet | 15:37 |
| leonardr | i just need to change a comment and i'll land it | 15:37 |
| poolie | fire one! | 15:37 |
| poolie | thanks for doing it | 15:37 |
| salgado | poolie, yeah, did that over IRC some time ago. just updated the MP, though | 15:38 |
| lifeless | jtv1: hi | 16:18 |
| jtv1 | hi lifeless | 16:18 |
| lifeless | jtv1: https://code.edge.launchpad.net/~ursinha/launchpad/add-ec2land-rules-orphaned-branches/+merge/29255 needs a follow up review | 16:18 |
| === jtv1 is now known as jtv | ||
| lifeless | per your request to urshina | 16:18 |
| jtv | lifeless: cool, thanks for letting me know | 16:18 |
| lifeless | bah | 16:18 |
| Ursinha | lifeless, I haven't finished that yet | 16:18 |
| Ursinha | doing that today | 16:18 |
| lifeless | irsinha | 16:18 |
| jtv | oh | 16:18 |
| lifeless | Ursinha: oh, ok. | 16:18 |
| lifeless | I'll put it back to wip for now hten | 16:18 |
| lifeless | \o/ | 16:23 |
| lifeless | no outstanding reviews | 16:23 |
| lifeless | -> chemist | 16:23 |
| Ursinha | lifeless, ok then, I'm finishing details to resubmit, what I couldn't finish last week :) | 16:26 |
| Ursinha | and I have a hilight for urshina here, the mispelling happens quite often I consider myself that too | 16:28 |
| lifeless | Ursinha: I'm sorry ;) | 16:29 |
| Ursinha | no problem :) | 16:29 |
| === Ursinha is now known as Ursinha-nom | ||
| lifeless | bigjools: hi | 16:51 |
| bigjools | hello | 16:51 |
| lifeless | bigjools: if you feel particularly strongly either way about admins & PPA creation via the API, please do what you think is right. | 16:52 |
| lifeless | bigjools: I'm trying to express a logic chain in the review not to mandate that it go either way. | 16:52 |
| bigjools | lifeless: well I don't feel strongly enough to cry if it's not done, as I said on the review | 16:52 |
| lifeless | bigjools: the underlying reason why I am arguing against a special case here is that the admins have said they want *less* interrupts. | 16:52 |
| lifeless | and given that any user can make a PPA via the web ui already, I don't understand why we'd need an admin override facility here. | 16:53 |
| lifeless | bigjools: sure; I'm being explicit that its your choice. | 16:53 |
| lifeless | I *feel* that we should default one way; however its always changable, and its in your part of the product :) | 16:54 |
| bigjools | lifeless: I understand your concern. Mine is that one day we'll need it and the facility is not there. I don't expect LOSAs to routinely get asked to make PPAs! | 16:54 |
| lifeless | I'm also still feeling moderately crap and I was communicating badly in the review which is why I've upgraded to real-time ;) | 16:55 |
| * bigjools hears ya | 16:55 | |
| lifeless | bigjools: thinking through it more I have another reason/condition around admins making arbitrary PPA's | 16:55 |
| lifeless | we require a CoC signature from PPA users. | 16:55 |
| lifeless | would that be done on the admin, or on the user the admin is making the PPA on [in the current code]? If on the admin, then that would be a bug ? | 16:56 |
| bigjools | interesting point | 16:59 |
| maxb | lifeless: When you're done with this discussion, I have a question for you - In https://answers.launchpad.net/launchpad-code/+question/117193 you asked me to file a bug for the "class of problem" of requiring LOSA intervention to make code imports work. I don't understand why. Shouldn't bugs be about concrete issues that we can fix? | 16:59 |
| lifeless | gary_poster: hi | 17:07 |
| gary_poster | hey lifeless. | 17:07 |
| lifeless | gary_poster: I want to talk cronscripts with you in a bit | 17:07 |
| lifeless | I've just realised I should answer maxb, and get dinner first. | 17:07 |
| lifeless | gary_poster: the theme will be 'making them safer to upgrade' | 17:07 |
| maxb | I will be around later, I would hate to keep someone from their dinner :-) | 17:08 |
| lifeless | maxb: the class of problem is 'needing to run ssh <details> and accept the key' | 17:08 |
| gary_poster | lifeless: BTW, I intend to write up the PQM/tarmac requirements; I realized this weekend that there are some bits that I still need to clarify | 17:08 |
| lifeless | gary_poster: awesome | 17:08 |
| gary_poster | lifeless: cronscripts: cool | 17:08 |
| lifeless | gary_poster: I saw the split to a subspec | 17:08 |
| maxb | lifeless: oh, ok I see what you meant now | 17:08 |
| lifeless | maxb: that class of issue is actionable and automatable. | 17:08 |
| lifeless | maxb: :) | 17:08 |
| lifeless | ok, awol for a little bit | 17:09 |
| jml | by "actionable", presumably you don't mean that we can sue. | 17:09 |
| lifeless | jml: who knows.... the phantom knows | 17:09 |
| jml | shadow. | 17:09 |
| lifeless | jml: I'm grabbing a club sandwich if you want to join me | 17:09 |
| jml | phantom doesn't know anything. | 17:09 |
| jml | lifeless, sounds good | 17:09 |
| jml | lifeless, once this ec2 land detaches :) | 17:10 |
| === al-maisan is now known as almaisan-away | ||
| lifeless | jml: I has a table | 17:17 |
| jml | lifeless, k. still detaching. | 17:17 |
| jml | done | 17:17 |
| === flacoste is now known as flacoste_lunch | ||
| === danilos is now known as daniloff | ||
| === beuno is now known as beuno-lunch | ||
| === Ursinha-nom is now known as Ursinha-afk | ||
| === Ursinha-afk is now known as Ursinha | ||
| === flacoste_lunch is now known as flacoste | ||
| === beuno-lunch is now known as beuno | ||
| === mtaylor is now known as mtaylor|lunch | ||
| lifeless | gary_poster: hi, so if you're still @ work | 19:33 |
| gary_poster | Hey lifeless. Yeah | 19:33 |
| lifeless | to change to the 'release features when they are ready' workflow | 19:34 |
| lifeless | we need to drastically simplify and reduce the downtime windows involved in rollouts | 19:34 |
| lifeless | james troup and I did a mental audit | 19:34 |
| gary_poster | lifeless: we release edge every night | 19:34 |
| lifeless | and I'm going to file a bunch of bugs related to this | 19:34 |
| gary_poster | isn't that the model we want to pursue? | 19:35 |
| lifeless | gary_poster: we do, but - and this is one of the problems with the current setup - that is only a small fraction of the lp instances around. | 19:35 |
| gary_poster | ah, ok | 19:35 |
| lifeless | gary_poster: so we need to do what we've done for edge and solve the issues | 19:35 |
| lifeless | for instance | 19:35 |
| gary_poster | ok, fair enough | 19:35 |
| lifeless | we have a lot of cronscripts | 19:35 |
| gary_poster | (BTW: https://dev.launchpad.net/Foundations/Proposals/SimplifyMergeMachinery) | 19:35 |
| lifeless | great, queued it up to read | 19:35 |
| gary_poster | cool | 19:36 |
| lifeless | so in particular, we have a lot of cronscripts | 19:36 |
| lifeless | which run 'prod' | 19:36 |
| lifeless | so we will be updating their code more frequently | 19:36 |
| gary_poster | run 'prod': that means, there is no "edge" equivalent | 19:36 |
| lifeless | I'm wondering if there is something we can do to make it safer to do | 19:36 |
| gary_poster | ? | 19:36 |
| lifeless | right | 19:36 |
| lifeless | so at the moment their 'deploy frequency' is 1/month | 19:37 |
| gary_poster | Eh, yeah, I hate the cronscript story. Touching them is terrifying. | 19:37 |
| gary_poster | My long term preference is that they disappear | 19:37 |
| lifeless | agreed | 19:37 |
| lifeless | however we have to break the deadlock somewhere ;) | 19:37 |
| gary_poster | right | 19:38 |
| lifeless | 170 cronscripts! [estimated] | 19:38 |
| gary_poster | yeah | 19:38 |
| gary_poster | so, why is touching them terrifyingL | 19:38 |
| gary_poster | : | 19:38 |
| lifeless | so the bug is going to be a bit generic | 19:38 |
| gary_poster | it is terrifying because the tests for them largely suck, AFAIK | 19:38 |
| lifeless | but in general we want to aim at being able to automatically upgrade them | 19:38 |
| gary_poster | Is there a problem other than the one I perceive? | 19:39 |
| lifeless | in the short term that could be manual but smoother than today. | 19:39 |
| lifeless | gary_poster: yes, several. | 19:39 |
| gary_poster | cool, should I wait for the bug report? :-) | 19:39 |
| lifeless | - they run from cron, so either they try to run while we're upgrading, or they may still be running while we upgrade | 19:39 |
| gary_poster | ah | 19:39 |
| lifeless | - they run from the 'current' symlink, not the 'active revno dir', so they could in principle do late buggy imports (but this is rare, we can ignore for now) | 19:40 |
| lifeless | - some of them have terrible knock on effects if interrupted, and take 15-20 minutes every hour. | 19:40 |
| * gary_poster tries to look at the cron scripts so they feel bad about themselves | 19:41 | |
| gary_poster | ...not much effect... | 19:41 |
| gary_poster | I have some vague ideas on how to address this, that have to do with trying to come up with cheap machinery and/or patterns for cronscript devs to follow | 19:42 |
| gary_poster | I feel constrained both because I don't have any knowledge or even oversight of the cronscripts, and because I don't want to spend too much time on them, knowing that I want them to die. | 19:43 |
| gary_poster | Perhaps it would be reasonable to discuss the machinery that we want to have to replace them in the long term | 19:43 |
| lifeless | perhaps | 19:44 |
| gary_poster | to see if it would inform the short term stepe we might take | 19:44 |
| lifeless | Its late for me right now | 19:44 |
| gary_poster | Fair enough | 19:44 |
| * benji wonders if gary_poster is thinking of zc.async | 19:44 | |
| lifeless | I wanted to frame the discussion | 19:44 |
| lifeless | broadly, I think we need something to wrap them | 19:44 |
| lifeless | to quiesce them when we're preparing to deploy | 19:44 |
| gary_poster | benji, sure, but we won't be using that here, I don't think :-) | 19:44 |
| lifeless | to alert us when important ones are running still | 19:44 |
| lifeless | and to insulate us mid-upgrade | 19:45 |
| gary_poster | I assume some sort of LOSA gesture like "I plan to upgrade things in half an hour so let's start quieting things down" would be reasonable | 19:45 |
| lifeless | https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadRollout | 19:46 |
| lifeless | for now, absolutely | 19:46 |
| lifeless | https://wiki.canonical.com/InformationInfrastructure/OSA/LaunchpadRollout#Comment out Cronjobs and Kill DB Backups (approx 30-40 mins before rollout) | 19:46 |
| lifeless | thats terrifying | 19:46 |
| lifeless | we have a tool to comment out cronjobs! | 19:46 |
| gary_poster | sigh | 19:46 |
| gary_poster | Another aspect to this is that the cronscripts themselves have had no constraints on them to my knowledge | 19:47 |
| lifeless | indeed | 19:47 |
| gary_poster | But again, I suspect that we want to retain that freedom for now | 19:47 |
| lifeless | so I plan to conduct an audit | 19:47 |
| gary_poster | Or else we will be bogged down | 19:47 |
| lifeless | etc | 19:47 |
| lifeless | but yeah | 19:47 |
| gary_poster | yeah, cool | 19:47 |
| lifeless | my basic thought is to have a single constant wrapper around all the scripts | 19:48 |
| lifeless | so that rather than commenting them out and hoping they stop etc | 19:48 |
| lifeless | the wrapper can check a flag somewhere | 19:48 |
| gary_poster | you make some flag to make it a no-op | 19:48 |
| lifeless | and not run | 19:48 |
| gary_poster | yeah | 19:48 |
| lifeless | right | 19:48 |
| lifeless | for now, I'm going to file a bug on each area of concern for the new workflow | 19:48 |
| gary_poster | OK, sounds reasonable | 19:49 |
| lifeless | this one is the least well formed | 19:49 |
| lifeless | which is why I wanted to kick off some neuron | 19:49 |
| lifeless | s | 19:49 |
| gary_poster | Cool | 19:49 |
| gary_poster | Oh, I'll just choose one :-) | 19:49 |
| lifeless | :) | 19:49 |
| lifeless | \o/ lsprof has landed | 19:50 |
| lifeless | thanks mars | 19:50 |
| gary_poster | great news, mars | 19:50 |
| gary_poster | and thanks to you too lifeless | 19:50 |
| lifeless | de nada | 19:50 |
| mars | cool, one step closer | 19:50 |
| gary_poster | ...so, yeah, such a cron script wrapper could be fairly complex. I think the trick will be to come up with a plan for one that is as simple and failsafe as possible. I'l think about it, and I'll hope to brainstorm with you about it later. | 19:52 |
| lifeless | great | 19:52 |
| === mtaylor|lunch is now known as mtaylor | ||
| lifeless | elmo: around? | 20:10 |
| lifeless | elmo: I'm trying to remember whats on loganberry that is rollout-sensitive | 20:10 |
| benji | what's the normal turn-around for a (relatively simple) RT? | 20:13 |
| lifeless | yes | 20:13 |
| lifeless | more precisely, its either in the priority queue, or 'sometime over the horizon' - like us, they have more incoming than outgoing. | 20:14 |
| lifeless | so you should talk to francis | 20:14 |
| mtaylor | rockstar: what's the reasoning behind the need for a merge request to have a commit message? | 20:16 |
| mtaylor | lifeless: not sure it's relevant, but on my openstack dev deployment, I'm using a hudson master and scheduled jobs instead of cron to do repetitive tasks | 20:17 |
| lifeless | losa ping | 20:22 |
| lifeless | abentley: hey, can I ask you a question about the importd's | 20:22 |
| lifeless | flacoste: rt 40477 | 20:30 |
| flacoste | lifeless: before or after staging PQM deployment? | 20:35 |
| flacoste | (assessing priority) | 20:36 |
| flacoste | RT 39150 | 20:36 |
| lifeless | yeah | 20:36 |
| lifeless | staging pqm will only be useful when I get some cycles to do the thing for gary | 20:36 |
| lifeless | so before I think | 20:36 |
| lifeless | I was crook in the weekend, so didn't get anything interesting done. | 20:37 |
| abentley | lifeless, pong | 20:37 |
| lifeless | hey abentley | 20:37 |
| flacoste | lifeless: ok | 20:37 |
| flacoste | leonardr: is RT 39922 still relevant? | 20:37 |
| lifeless | I wanted to know about the interactions between upgrades and the importd service | 20:38 |
| leonardr | flacoste, humor me and tell me the summary while i look that rt up | 20:41 |
| leonardr | flacoste: got it | 20:42 |
| leonardr | flacoste: no, the performance test has been run and there is no need to do anything else right now | 20:42 |
| lifeless | flacoste: also rt 40480 | 20:44 |
| lifeless | abentley: so if I wanted to upgrade the code on the importds, does it imply downtime, and interruption to long running tasks, and if so how much downtime and how much interruption. | 20:45 |
| flacoste | leonardr: thanks, i'll make it as obsolete or invalid watherver their status is | 20:56 |
| flacoste | lifeless: while I look at that other RT, mind looking at my follow-up to your review :-) | 20:56 |
| lifeless | sure | 20:56 |
| flacoste | lifeless: bumbed above PQM staging deployment | 20:58 |
| lifeless | thanks | 20:59 |
| lifeless | gnight | 21:00 |
| abentley | lifeless, I don't know the deployment story of the importds. | 21:01 |
| lifeless | abentley: who would? [that knows the code too, so we can talk about how to make it slick] | 21:02 |
| abentley | lifeless, mwhudson. | 21:02 |
| lifeless | thanks | 21:02 |
| flacoste | gary_poster: do you remember if l-d-d is updated every buildbot run? | 21:06 |
| gary_poster | flacoste: I think so, but things may have changed. want me to look? | 21:06 |
| flacoste | gary_poster: i don't think it would have changed, i remember not being the case | 21:07 |
| flacoste | so if we fixed that at some point, i'm pretty sure it works now | 21:07 |
| m4n1sh | is the staging server getting an update? | 21:08 |
| gary_poster | looking' | 21:08 |
| m4n1sh | it breaks when using API | 21:08 |
| gary_poster | flacoste: ah, no, it does *not* update l-d-d | 21:08 |
| lifeless | Ursinha: have you or matsubara asked for the daily staging setup yet ? | 21:09 |
| lifeless | in RT I mean, if not, I will. | 21:09 |
| gary_poster | (matsubara is out of office) | 21:09 |
| gary_poster | m4n1sh: yes, getting update | 21:09 |
| m4n1sh | gary_poster, thanks. does it happen daily? | 21:09 |
| gary_poster | yes, m4n1sh | 21:09 |
| m4n1sh | gary_poster, Thanks for the info | 21:10 |
| gary_poster | np | 21:10 |
| mtaylor | rockstar: tags don't seem to be propgating through tarmac | 21:10 |
| flacoste | lifeless: can you remove your Needs fixing? | 21:11 |
| flacoste | vote | 21:11 |
| flacoste | so that ec2 land doesn't stomp me on the fingers | 21:11 |
| mwhudson | morning | 21:12 |
| m4n1sh | gary_poster, is there any attribute on launchpad API's me object which is private to all? | 21:12 |
| Ursinha | me | 21:12 |
| Ursinha | lifeless, no | 21:12 |
| m4n1sh | gary_poster, I mean, I want to use it as an auto password... that attribute should not be visible in any case on launchpad.net/~foo | 21:13 |
| lifeless | Ursinha: ok, I've filed RT 40482 | 21:13 |
| lifeless | flacoste: that seems buggy | 21:13 |
| lifeless | flacoste: like, it should trust that when I say approve, I'm not being schizotic. | 21:13 |
| lifeless | flacoste: I've fixed it but perhaps you could make a sane bug report up | 21:14 |
| lifeless | ? | 21:14 |
| flacoste | i will | 21:14 |
| lifeless | thanks! | 21:14 |
| lifeless | I'm really going to sleep now. | 21:14 |
| deryck | Later on, all.... | 21:14 |
| flacoste | the bug is that it looks at the approver to determine the r= | 21:14 |
| flacoste | and since there are no reviewers approving... | 21:15 |
| lifeless | flacoste: right, but *my* vote as a reviewer has to be treated as approve | 21:15 |
| lifeless | flacoste: given I approved the merge. | 21:15 |
| lifeless | ciao | 21:15 |
| flacoste | chao, sleep weel | 21:15 |
| flacoste | well | 21:15 |
| gary_poster | m4n1sh: mmm, I stared at your questions and I think I understand sort of. You want to hack/abuse an attribute in some way, right? | 21:17 |
| m4n1sh | gary_poster, actually i want to use launchpadlib in a django app | 21:17 |
| gary_poster | cool | 21:17 |
| m4n1sh | but django needs a password in authentication module | 21:17 |
| m4n1sh | but i want to use launchpad OAuth | 21:18 |
| m4n1sh | so when the user returns after giving the permissions | 21:18 |
| m4n1sh | i dont want him to ask the password | 21:18 |
| m4n1sh | but there should be some field which is private to the user | 21:18 |
| gary_poster | oh! so...if the person is authenticated by launchpadlib, he's ok by you? | 21:18 |
| m4n1sh | which i can use as the implicit password | 21:18 |
| m4n1sh | yes | 21:18 |
| m4n1sh | it is | 21:18 |
| m4n1sh | everything is done.. | 21:18 |
| gary_poster | OK | 21:19 |
| m4n1sh | but just to satisfy django's authentication module i need a password | 21:19 |
| gary_poster | You can't override it? In the abstract, that feels like the right thing to do, but I don't know Django's auth stuff so I'm just waving my hands. | 21:19 |
| m4n1sh | same here. I thought there should be way around | 21:20 |
| m4n1sh | django has a openid module | 21:20 |
| m4n1sh | gary_poster, have a look here | 21:21 |
| m4n1sh | http://docs.djangoproject.com/en/1.2/topics/auth/#authentication-in-web-requests | 21:21 |
| m4n1sh | esp in authenticate() method | 21:21 |
| gary_poster | Well, it sounds like you are in hack land | 21:21 |
| gary_poster | If you are in hack land... | 21:21 |
| === almaisan-away is now known as al-maisan | ||
| m4n1sh | the thing wont keep quiet without a password | 21:21 |
| m4n1sh | http_etag would suffice? | 21:21 |
| gary_poster | no | 21:22 |
| m4n1sh | if i am in a hack land then? | 21:22 |
| m4n1sh | gary_poster, i am not able to think any way to overcome this password part. Something needs to be done | 21:23 |
| gary_poster | can you just say that the password for *everyone*is an empty string, or some other hardcoded blather, but make sure that it is not actually usable? If you have a launchpadlib object for the person, then you stuff the hardcoded password in; otherwise, you make sure that the machinery doesn't work? | 21:23 |
| gary_poster | To be clear, this sounds absolutely horrible to me :-) | 21:23 |
| m4n1sh | gary_poster, same here. and i have no idea about the security aspect too | 21:23 |
| m4n1sh | if i go this way | 21:24 |
| gary_poster | If I were you I'd be banging around on Django docs or code or mailing lists or something to figure out a way to subclass the auth machinery | 21:24 |
| gary_poster | stashing the password... | 21:24 |
| gary_poster | ...is hackland. :-/ | 21:24 |
| gary_poster | I'll look in a little bit if you really want me to (I don't know of an answer offhand) | 21:25 |
| gary_poster | but I have to hope Django lets you substitute this bit | 21:25 |
| m4n1sh | gary_poster, I think so | 21:25 |
| m4n1sh | I found django-oauth too, but launchpadlib is so darn easy that I don't want more stacked up modules | 21:25 |
| gary_poster | sure | 21:25 |
| m4n1sh | probably I would go by the hack you suggested. Let's see. | 21:26 |
| m4n1sh | probably I might shoot off a mail to django mailing lists. I know lots of flames might return, but still I will learn something new | 21:26 |
| m4n1sh | gary_poster, well looks like you are busy. I don't want to disturb you anymore | 21:27 |
| m4n1sh | Thanks a lot for your help | 21:28 |
| gary_poster | m4n1sh: sorry it wasn't more, but yeah, busy. :-/ http://docs.djangoproject.com/en/dev/topics/auth/ seems to imply that looking at set_unusable_password might help; and that trying to see how LDAP support works might give you a pattern to follow | 21:30 |
| m4n1sh | gary_poster, I think you got exactly what I was looking for. | 21:31 |
| m4n1sh | I am an idiot. Looked at the whole page, but couldn't find out that very line | 21:32 |
| gary_poster | :-) | 21:32 |
| m4n1sh | it is like a fine print :) | 21:32 |
| gary_poster | takes another person sometimes | 21:32 |
| m4n1sh | gary_poster, I think the last hurdle also solved | 21:32 |
| gary_poster | awesome! | 21:32 |
| m4n1sh | now I can go ahead and start working on the app | 21:32 |
| m4n1sh | thank you very much (no this is not sarcasm, I mean it) | 21:33 |
| gary_poster | lol | 21:34 |
| gary_poster | np glad I could help | 21:34 |
| flacoste | mars, gary_poster: i got errors compiling numpy!?! | 21:41 |
| flacoste | on ec2 land | 21:41 |
| mars | flacoste, that's odd. I have not heard of that. | 21:41 |
| flacoste | mars: actually, that's not the real error | 21:42 |
| flacoste | never mind | 21:42 |
| mars | ok | 21:42 |
| flacoste | i forgot to add commit the dep in download-cache | 21:42 |
| === al-maisan is now known as almaisan-away | ||
| wgrant | Can someone please ec2 land lp:~wgrant/launchpad/bug-598345-restrict-dep-contexts, lp:~wgrant/launchpad/refactor-_dominateBinary and lp:~wgrant/launchpad/really-publish-ppa-ddebs? | 22:53 |
| maxb | urgh. Do we really need a rabbitmq server just to develop any bits of launchpad? | 23:02 |
| wgrant | You will soon. | 23:04 |
| wgrant | But it will probably be started by make run soon. | 23:04 |
| wgrant | mwhudson: Could you be convinced to land the three branches I referenced above? | 23:43 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!