| LPCIBot | Project db-devel build (105): SUCCESS in 4 hr 0 min: https://hudson.wedontsleep.org/job/db-devel/105/ | 01:55 |
|---|---|---|
| === Ursinha-bbl is now known as Ursinha | ||
| wallyworld | anyone running launchpad and storm 0.18, psycopg2 2.2 successfully? release notes for storm 0.18 implies this should work now? | 02:36 |
| wallyworld | i still get the famous ProgrammingError: operator does not exist: text = bytea | 02:36 |
| wallyworld | oh well, back to psycopg 2.1.3 | 02:37 |
| wgrant | wallyworld: It's not a Storm bug. | 02:38 |
| wgrant | It's a Launchpad one. | 02:38 |
| wgrant | Storm has been 2.2-compatible for a while now, IIRC. | 02:38 |
| wallyworld | yeah, but i thought that it could also be fixed in the storm layer as well? clearly i was wrong :-) | 02:39 |
| LPCIBot | Project devel build (161): FAILURE in 3 hr 58 min: https://hudson.wedontsleep.org/job/devel/161/ | 02:39 |
| wallyworld | i just saw in the storm 0.18 release notes that "- Make storm compatible with psycopg2 2.2 " assumed :-) | 02:40 |
| * mwhudson waves | 05:26 | |
| * thumper waves at mwhudson | 05:47 | |
| * thumper EODs | 05:47 | |
| LPCIBot | Project devel build (162): STILL FAILING in 3 hr 29 min: https://hudson.wedontsleep.org/job/devel/162/ | 06:09 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=jtv][ui=none][no-qa] Improved tests for | 06:09 |
| LPCIBot | translations.ProductSeriesView and ProductSeriesLanguagesView. | 06:09 |
| LPCIBot | LaunchpadObjectFactory does not require a language code for | 06:09 |
| LPCIBot | POFiles and such anymore. | 06:09 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=edwin-grubbs][ui=none][bug=664060] Allow bug supervisors to | 06:09 |
| LPCIBot | configure the bug tracker section of their project. | 06:09 |
| jtv | bye thumper | 06:11 |
| henninge | I wonder how to get check_permission working in a unit test (a view test). | 06:15 |
| henninge | I thought "login_person" would do the trick but it seems not. | 06:16 |
| henninge | It always returns "True" | 06:16 |
| === Ursinha is now known as Ursinha-afk | ||
| henninge | It may be a layer thing. | 06:34 |
| adeuring | good morning | 08:38 |
| mrevell | Hi | 09:27 |
| thumper | mrevell: morning | 10:16 |
| === almaisan-away is now known as al-maisan | ||
| thumper | mrevell: thanks for the bug reports, I'll get on to it :) | 10:16 |
| mrevell | thanks thumper :) I'll probably file a couple more later and also reply to poolie's comments | 10:19 |
| thumper | mrevell: I've got some ideas, and jml suggested just getting something going so you can then suggest improvements | 10:19 |
| mrevell | Cool, I'd be happy to do that, yeah | 10:20 |
| === matsubara-afk is now known as matsubara | ||
| lifeless | jml: around? | 12:31 |
| === Ursinha-afk is now known as Ursinha | ||
| jml | lifeless: sort of | 12:42 |
| lifeless | jml: could I get your stamp on the layers patch | 12:42 |
| lifeless | for ec2land | 12:42 |
| jml | lifeless: sure | 12:42 |
| lifeless | thank you | 12:42 |
| lifeless | jml: https://code.edge.launchpad.net/~lifeless/launchpad/zope.testing/+merge/39423 for easy reference | 12:48 |
| lifeless | jml: thank you | 13:04 |
| lifeless | Ursinha: we might want to run the stuff that does 'fixed by a commit' at high frequency - or is it the tagger? | 13:13 |
| matsubara | bac, around? | 13:13 |
| === mrevell is now known as mrevell-lunch | ||
| LPCIBot | Project devel build (163): STILL FAILING in 3 hr 57 min: https://hudson.wedontsleep.org/job/devel/163/ | 13:53 |
| lifeless | flacoste: ping | 13:59 |
| === mrevell-lunch is now known as mrevell | ||
| StevenK | leonardr: Hi, are you around? | 14:02 |
| leonardr | StevenK: yes | 14:03 |
| StevenK | leonardr: There seems to be a change made recently with launchpadlib that it wants to write to /root/.launchpadlib during the test suite | 14:03 |
| leonardr | StevenK: this is when you're testing launchpad, or are you using launchpadlib trunk? | 14:04 |
| StevenK | leonardr: So the slaves are connected to as root, and then I sudo to a hudon user. | 14:04 |
| StevenK | leonardr: Sorry, this is when Hudson is running the full test suite on its build slaves | 14:04 |
| StevenK | lifeless: Speaking of Hudson, you wanted a module enabled | 14:05 |
| lifeless | no, I was saying we might need to | 14:05 |
| lifeless | I need to talk with ops | 14:06 |
| StevenK | lifeless: Sorry, this was like 3 weeks ago when you said you wanted the Chuck Norris plugin :-P | 14:06 |
| lifeless | and it would be 'write a module' (either python or java) to do the devel->stable push. | 14:06 |
| leonardr | ok. /root/.launchpadlib is the default location when you run as root. we ran into this problem before and changed the launchpadlib constructor used by launchpad so that it would use a temporary directory | 14:06 |
| lifeless | StevenK: oh yes. JFDI | 14:06 |
| StevenK | lifeless: Why!? | 14:06 |
| StevenK | leonardr: Has this code recently changed, then? | 14:09 |
| leonardr | StevenK: no, but if someone wrote a test that created a Launchpad object without going through lp.testing.launchpadlib_for, you'd get the old behavior back | 14:09 |
| leonardr | can you pinpoint when the directory is created? what gets put in that directory? | 14:10 |
| flacoste | lifeless: i'm back, sorry, family start-up got stalled this morning | 14:11 |
| StevenK | leonardr: Certainly: https://hudson.wedontsleep.org/job/devel/163/consoleText | 14:11 |
| StevenK | leonardr: If you search for '/root/.launchpadlib' | 14:12 |
| Ursinha | lifeless, it's tagger | 14:16 |
| jam | it looks like the build machine iridium is hung trying to build a webkit build, can anyone confirm? | 14:17 |
| wgrant | jam: Indeed. lamont ^^ | 14:18 |
| leonardr | StevenK: so, these errors are in places that don't use lp.testing.launchpadlib_for -- the launchpadlib tests themselves, which are outside of launchpad, and a launchpad test that uses login_with | 14:18 |
| leonardr | as to why this is a problem now, i couldn't speculate--the code on my end hasn't changed | 14:19 |
| leonardr | i think the best solution would be to change all those calls to use a temporary directory, as launchpadlib_for | 14:19 |
| leonardr | does | 14:19 |
| StevenK | leonardr: I have to admit, I am curious as to why it's only occuring now | 14:27 |
| lamont | meh | 14:27 |
| LPCIBot | Project db-devel build (106): FAILURE in 4 hr 1 min: https://hudson.wedontsleep.org/job/db-devel/106/ | 14:27 |
| lamont | jam: and what I can tell you is what we all can see from the web ui in launchpad | 14:27 |
| poolie | how cute: https://bugs.edge.launchpad.net/bzr-explorer/+bug/667782 | 14:28 |
| StevenK | Rargh, it just impacted db-devel too | 14:28 |
| jam | lamont: I'm sitting here with shadeslayer (Rohan), who is trying to understand what is going on. Any chance we can just kick it? | 14:28 |
| lamont | I just stabbed it in the face | 14:29 |
| jam | wgrant: is abentley the right person to poke to see if we can understand what is failing here? | 14:29 |
| jam | It looks like it is failing during the branch update | 14:29 |
| jam | but it seems an odd place to get hung | 14:30 |
| lifeless | Ursinha: ok cool. | 14:30 |
| lifeless | Ursinha: and its running how often now ? | 14:30 |
| jam | (I wonder if it just isn't reporting the 'next' thing.) | 14:30 |
| Ursinha | lifeless, every 5 mins | 14:30 |
| lifeless | awesome | 14:30 |
| lifeless | abentley: hi; can I help you QA 638637 in some way? | 14:30 |
| lifeless | https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html | 14:30 |
| leonardr | StevenK: is it possible that /root/.launchpadlib existed before and has been removed? the error is in trying to create the directory | 14:31 |
| jam | abentley: I'm sure deployment-stable takes precedence over the webkit stuff :) | 14:31 |
| wgrant | jam: It could have been bzr hanging there, or the builder could have had a seizure and exploded. | 14:31 |
| StevenK | leonardr: Let me see if a slave is still up | 14:31 |
| abentley | lifeless: You can push a branch, and see if it gets scanned and stuff, I guess. | 14:31 |
| wgrant | lamont: You can't get a shell in a virt builder? | 14:32 |
| jam | wgrant: sure, but both are pretty unlikely | 14:32 |
| wgrant | jam: The latter isn't particularly unlikely. | 14:32 |
| Ursinha | lifeless, you are not reusing branches anymore? | 14:34 |
| StevenK | leonardr: There is no /root/.launchpadlib | 14:34 |
| leonardr | StevenK: my speculation was that there might have been one earlier (preventing this problem) and that its removal caused this problem | 14:35 |
| jam | lamont: it seems that stabbing it in the face leaves it in the "currently building" state? | 14:35 |
| lifeless | Ursinha: Its always been adhoc :) | 14:37 |
| lifeless | Ursinha: sometimes I will, sometimes I won't. | 14:37 |
| Ursinha | :) | 14:37 |
| lifeless | abentley: qa staging codehosting is not working yet; am going to get it working with losa | 14:37 |
| jam | wgrant: any way to change the status from "it is currently building and should have finished 12hours ago" into some sort of "the build has failed" state? | 14:37 |
| jam | It is a bit confusing to figure out what is going on. | 14:38 |
| abentley | lifeless: I'd normally qa it on staging, anyhow. | 14:39 |
| wgrant | jam: The builder facestab should have done that. | 14:41 |
| wgrant | I suspect it needs harder stabbing. | 14:41 |
| wgrant | Or buildd-manager is broken. | 14:42 |
| jam | lifeless: what is the issue with codehosting on staging? (I'm concerned it might be my stuff, though comments from yesterday said that it wasn't) | 14:46 |
| lifeless | jam: staging is working, qastaging isn't | 14:48 |
| jam | lifeless: k, I guess i'm not fully aware of what the differences are | 14:50 |
| noodles775 | mars: Hi! Have you had a chance to look at gmb's issue (launchpad-dev mailing list, 'Problems with FeatureFlags and test isolation')? | 14:51 |
| jam | wgrant: well, atm I suspect buildd-manager, because webkit has 2 jobs that were killed but are now listed as "currently building" | 14:52 |
| mars | noodles775, no, I have not looked at it. I think deryck said he would have a look | 14:53 |
| noodles775 | mars: ok, thanks. I'll check with deryck when he's available. | 14:54 |
| mars | noodles775, I asked gary_poster about someone in our team picking that up, but then deryck offered to help | 14:54 |
| gary_poster | y | 14:54 |
| flacoste | bigjools: are you sure you wanted to land the new async build manager? it will be rolled-out soon | 15:22 |
| lifeless | abentley: ok, so it scanned my new branch and an incremetnal push ok | 15:33 |
| lifeless | abentley: is that sufficient, do you think ? | 15:33 |
| abentley | lifeless: Well, merge detection also needed to be tested, but I've done that, and marked it qa-ok. | 15:33 |
| lifeless | awesome | 15:33 |
| lifeless | Edwin-afk: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html - whats up with 11762 | 15:42 |
| === matsubara is now known as matsubara-lunch | ||
| bigjools | flacoste: yes, it's QA-OK. | 15:53 |
| bigjools | I need to mark the bugs. | 15:54 |
| bigjools | jml: around? | 15:55 |
| flacoste | bigjools: that's great! | 15:56 |
| bigjools | flacoste: I need to land one more branch though, but it won't block rollout. The new process seems to hammer the database harder than I want, so I need to back off the polling intervals. | 15:57 |
| flacoste | ok | 15:57 |
| lifeless | sinzui: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html - whats up with 11762 | 16:07 |
| lifeless | mwhudson: http://launchpad.net/bugs/660264 is ok right, in the absence of config changes? | 16:08 |
| * mwhudson looks | 16:10 | |
| mwhudson | lifeless: it's ok in the sense of 'do no harm' indeed | 16:11 |
| lifeless | StevenK: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html rev 11778 bug 664380 | 16:19 |
| StevenK | lifeless: Er? It says edwin for me | 16:20 |
| lifeless | StevenK: yes, look at the bottom | 16:20 |
| StevenK | Abh | 16:20 |
| StevenK | s/b// | 16:20 |
| StevenK | lifeless: I will look at doing that after I've emptied my plate | 16:21 |
| lifeless | StevenK: is it lunchtime? | 16:22 |
| StevenK | lifeless: Sigh | 16:22 |
| StevenK | I'm in the middle of two things. Mental plate | 16:22 |
| lifeless | mwhudson: jam: so forking server for bzr+ssh - can you describe somewhere what needs to be changed to test it? bazaar.qastaging is now working with the 'normal' service, so its ready to reconfigure to test. | 16:22 |
| lifeless | StevenK: cool, great and thanks. | 16:22 |
| mwhudson | lifeless: i believe jam has written at least one email containing the needed steps | 16:23 |
| mwhudson | lifeless: Subject: Re: QAing the forking codehosting server | 16:23 |
| lifeless | mwhudson: not in my mail | 16:24 |
| mwhudson | bah | 16:24 |
| * mwhudson forwards | 16:24 | |
| mwhudson | lifeless: it would be good to time bzr ping bzr+ssh://bazaar.qastaging.launchpad.net/ a few times before we test it i guess | 16:25 |
| lifeless | bah, I don't have that plugged in | 16:26 |
| lifeless | but yes | 16:26 |
| mwhudson | i guess doing in from somewhere where ssh setup latency is not the dominant factor would be a good idea too... | 16:26 |
| jam | echo hello | ssh bazaar.qastaging.launchpad.net bzr serve --inet --directory=/ --allow-writes | 16:26 |
| jcsackett | leonardr: ping. | 16:27 |
| leonardr | jcsackett, hi | 16:27 |
| jam | a bit long, but always works, and doesn't need to have extra code installed :) | 16:27 |
| mwhudson | it tells me permission denied in just 0.7s! | 16:28 |
| * mwhudson fiddles | 16:28 | |
| jcsackett | leonardr: hi. i just read your review note. do you want me to add notice in the 1.7.0 block or add a new one? | 16:28 |
| jcsackett | (in NEWS.txt) | 16:29 |
| jam | mwhudson: it lets me in with a proper ssh-key | 16:29 |
| mwhudson | yeah | 16:29 |
| leonardr | jcsackett: add it to 1.7.0, it's not out yet | 16:29 |
| jam | with my password cached, it is 3.3s from a machine in France | 16:30 |
| jam | interestingly, bazaar.staging denies me | 16:30 |
| EdwinGrubbs | lifeless: why does https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html list my bug as qa-bad, when it is tagged qa-ok? I didn't think I was supposed to remove the bad-commit-N tag. | 16:30 |
| jam | though I have a different key on that machine | 16:30 |
| jam | non-staging is 3-7s for me | 16:31 |
| jam | (3, 7.1, 4.6, ...) | 16:31 |
| lifeless | EdwinGrubbs: you're not | 16:31 |
| jcsackett | leonardr: dig. and what's the landing policy for launchpadlib? looks like all the landings on trunk have been by you? or feel free to point me to docs if they're around for this. | 16:31 |
| lifeless | EdwinGrubbs: you're meant to have landed a commit with [rollback=xxxxx] where xxxxx is the bad commit | 16:31 |
| lifeless | EdwinGrubbs: until thats landed the qa tagger ignores the qa-* tags. | 16:31 |
| leonardr | jcsackett: bzr co the trunk, then merge and commit | 16:32 |
| lifeless | EdwinGrubbs: this is because we have to pair the bad commit with the fixed commit. | 16:32 |
| EdwinGrubbs | lifeless: but what if I fix the bug instead of rolling it back. Should I still tag it with [rollback=xxxx]? | 16:32 |
| jcsackett | leonardr: dig. thanks! | 16:32 |
| lifeless | EdwinGrubbs: yes, sadly. | 16:32 |
| lifeless | EdwinGrubbs: is that what happened here? | 16:32 |
| EdwinGrubbs | yes | 16:32 |
| lifeless | EdwinGrubbs: what commit fixed it ? | 16:33 |
| jam | shadeslayer: ATM, it looks like we're disabling your daily build, until we can figure out why it is crashing. | 16:34 |
| EdwinGrubbs | lifeless: if you look at revision 11763, it has the same info, since I had tried to get a losa to cancel the previous pqm run. I thought the cancelling had succeeded, which is why I landed it again with the same message. | 16:34 |
| shadeslayer | jam: feel free to :) | 16:34 |
| EdwinGrubbs | lifeless: I guess I should just land an empty branch with [rollback=xxxx] in it then. | 16:35 |
| lifeless | EdwinGrubbs: no need | 16:36 |
| lifeless | EdwinGrubbs: I'll explicitly call it out | 16:36 |
| EdwinGrubbs | ok, thansk | 16:36 |
| StevenK | lifeless: 11778 QA'd | 16:47 |
| === matsubara-lunch is now known as matsubara | ||
| jml | bigjools: hi | 16:56 |
| bigjools | jml: hi - just wanted to tell you that I tracked down why the b-m was using so much CPU | 16:57 |
| bigjools | probably best to explain in person | 16:58 |
| flacoste | jml: if you have time, i'd like your opinion on the new release schedule proposed by henninge, especially its impact on the bug jam | 16:58 |
| lifeless | flacoste: I need to talk with you about a) storm and b) my leave (which is still darn vague) | 17:02 |
| jml | flacoste: I'll try to look at it today. | 17:08 |
| lifeless | mwhudson: I forwarded that mail to losas@ | 17:11 |
| jam | lifeless: you don't ever get to leave. I thought you knew that | 17:11 |
| lifeless | jam: hah | 17:12 |
| lifeless | ;p | 17:12 |
| === benji is now known as benji-lunch | ||
| === al-maisan is now known as almaisan-away | ||
| LPCIBot | Project db-devel build (107): STILL FAILING in 3 hr 58 min: https://hudson.wedontsleep.org/job/db-devel/107/ | 19:10 |
| lifeless | flacoste: calling | 19:12 |
| === benji-lunch is now known as benji | ||
| lifeless | is it really still week 1? | 19:37 |
| LPCIBot | Project devel build (164): STILL FAILING in 3 hr 58 min: https://hudson.wedontsleep.org/job/devel/164/ | 19:39 |
| rockstar | abentley, yo. | 19:47 |
| abentley | rockstar: yo. | 19:47 |
| rockstar | abentley, I'm having a hard time writing a test that will reproduce what Chex did today. | 19:48 |
| abentley | rockstar: have you tried it out locally? | 19:48 |
| rockstar | abentley, I wonder if you might have a better idea what's going on there. | 19:48 |
| rockstar | abentley, I have. I also wrote a test that will do exactly the same thing. | 19:48 |
| rockstar | abentley, I just thought you might have a cursory "oh yeah, I think that problem might be in X" If you don't, don't worry about it. I'll sort it out. | 19:50 |
| abentley | rockstar: Nothing strikes me. I'd look at whether updateContextFromData is doing badness. | 19:52 |
| rockstar | abentley, I suspect there's something admin specific here. | 19:52 |
| jam | rockstar: you mean accidentally reassigning to himself? | 20:21 |
| jam | mwhudson explained what happened in a different (but similar) page | 20:21 |
| rockstar | jam, if I see him, I will ask. | 20:22 |
| jam | he explained it to me | 20:22 |
| jam | said that there is a dropdown which contains a list of your id + groups you are in | 20:22 |
| jam | and tries to select the current group which matches the owner | 20:22 |
| jam | but if it can't find that group, it just selects the first line | 20:23 |
| jam | which is you | 20:23 |
| jam | but for admins | 20:23 |
| jam | they aren't in the existing group | 20:23 |
| jam | so it always selects themselves | 20:23 |
| jam | ideally, admins would just have a text-entry box | 20:23 |
| jam | since they can set it to anyone | 20:23 |
| jam | otherwise you have to play a game of "find the common group", set it to that, then have the user set it back to themselves | 20:23 |
| lifeless | jam: so, I've forwarded your instructions to losas; but if you wanted to drive the testing of the forking service that wuld be best | 20:23 |
| lifeless | jam: the code is deployed | 20:24 |
| jam | lifeless: I'm happy to do the testing, but what about startup scripts, etc? | 20:24 |
| lifeless | hop into launchpad-ops | 20:24 |
| lifeless | ask losa there | 20:24 |
| rockstar | jam, huh. I have a test that would indicate that's not what's happening, but what you say makes sense. | 20:25 |
| jam | rockstar: I don't know specifically what is happening. Just that mwh said it was happening on another page, and they fixed it somehow | 20:26 |
| rockstar | jam, yeah, I just talked to sinzui about it. | 20:26 |
| jam | ls | 20:26 |
| jam | sorry, mt | 20:26 |
| jam | rockstar, is poolie near you? vila would like to chat with him in #bzr | 20:28 |
| rockstar | jam, poolie is not here. Just deryck and I. | 20:28 |
| jam | np | 20:29 |
| rockstar | jam, actually, turns out that's EXACTLY what's wrong. | 20:35 |
| rockstar | jam, also, I apparently already had a test that CLAIMED to be testing it, but it wasn't. | 20:35 |
| jam | rockstar: I'm glad I could be proxy-help for mwh | 20:36 |
| rockstar | jam, :) Thanks. | 20:37 |
| lifeless | deryck: hi | 20:42 |
| deryck | hi lifeless | 20:43 |
| lifeless | if you're in the main hall, I'm just qaing a bugs patch | 20:43 |
| lifeless | would like your input | 20:43 |
| lifeless | https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html | 20:43 |
| deryck | lifeless, sure, be right there. | 20:43 |
| thumper | morning | 20:49 |
| thumper | abentley: ping | 20:51 |
| abentley | thumper: pong | 20:51 |
| thumper | abentley: quick call? | 20:51 |
| abentley | thumper: sure. | 20:51 |
| lifeless | gary_poster: hi | 20:59 |
| gary_poster | hey lifeless | 20:59 |
| lifeless | gary_poster: https://devpad.canonical.com/~lpqateam/qa_reports/deployment-stable.html rev 11792 | 20:59 |
| gary_poster | lifeless, just changed it | 21:00 |
| gary_poster | like, seconds ago :-) | 21:00 |
| lifeless | sweet | 21:00 |
| lifeless | deryck: https://bugs.edge.launchpad.net/malone/+bug/659085 | 21:03 |
| * deryck looks | 21:03 | |
| deryck | lifeless, so I'm 85% sure that one is qa-ok. But I cannot know for sure without pinging allenap about it. | 21:08 |
| deryck | allenap, around? | 21:09 |
| allenap | deryck: Whassup? | 21:09 |
| deryck | allenap, see the bug lifeless pointed me at above... is it qa-ok? | 21:10 |
| thumper | abentley: https://edge.launchpad.net/ubuntu/+source/gstreamer0.10 | 21:10 |
| allenap | deryck: I have tried a few times to test this in staging/qastaging but it keeps timing out, so qa-untestable I'd say. | 21:11 |
| === matsubara is now known as matsubara-afk | ||
| deryck | allenap, there's nothing changed in functionality? Just moving code around? | 21:12 |
| allenap | deryck: The implementation has changed quite a bit, but the tests didn't much. | 21:13 |
| deryck | allenap, hmmmm | 21:13 |
| deryck | allenap, what should I look at to qa this on staging? | 21:13 |
| * allenap tries to remember | 21:14 | |
| abentley | thumper: https://code.edge.launchpad.net/~abentley/+archive/ppa/+builds | 21:19 |
| allenap | deryck: Okay. Calculating structural subscribers has been refactored, but the code changed little. The difference is that the bug change notification code now uses the new structural subscriber code (getStructuralSubscribers) instead of getBugNotificationsRecipients. | 21:19 |
| allenap | deryck: So we need to change some bugs and check that structural subscribers are notified. | 21:20 |
| bigjools | abentley: I have bad news for you. Backing out the buildd revision has cured our build farm ills. There's not much chance of it going back in. | 21:20 |
| allenap | deryck: I *think* that if it's wrong we will send too little bug mail. | 21:20 |
| allenap | deryck: Well, or too much. | 21:20 |
| deryck | allenap, ah, ok. I can poke at that then and see what happens. | 21:21 |
| bigjools | abentley: but I would like us to analyse the changes and how they could have caused buildds to become slow to respond to probes | 21:21 |
| abentley | bigjools: okay. Do you have any logs? | 21:22 |
| bigjools | abentley: sort of. It looks like it's eating memory which causes the machine to slow down too much (swapping) | 21:23 |
| bigjools | abentley: we have got strace logs, tcp dumps but no buildd logs since they are virtual | 21:24 |
| bigjools | abentley: what changes did that buildd revision have? | 21:24 |
| abentley | bigjools: It has an updated version of bzr-builder, it builds outside the chroot, and it uses xen-detect to ensure it doesn't build in an unvirtualized situation. | 21:28 |
| bigjools | abentley: I wonder if bzr-builder regressed somehow then | 21:29 |
| thumper | bigjools: ok, so where to from here? | 21:29 |
| bigjools | thumper: we need to re-create this problem on dogfood | 21:30 |
| thumper | bigjools: ok, do it! :) | 21:30 |
| bigjools | then we can start to diagnose it outside of production | 21:30 |
| * bigjools stares at thumper | 21:31 | |
| * thumper deflects the stare with a cunning mirror like shield | 21:31 | |
| allenap | deryck: It's still timing out all the time for me. | 21:32 |
| bigjools | thumper: are you batfink in disguise! | 21:32 |
| deryck | allenap, every bug is timing out? Or trying to do something times out? | 21:32 |
| allenap | deryck: Trying to create a structural subscription (I don't have any) is timing out. | 21:33 |
| deryck | ah | 21:33 |
| allenap | deryck: Ah. If I go to +subscribe directly it works. I can't get any project pages to work. | 21:33 |
| deryck | allenap, do you think it's related to your work? Or just qa/staging? | 21:33 |
| deryck | ah right | 21:33 |
| lifeless | allenap: what project did you try? | 21:34 |
| allenap | lifeless: malone and scriptify (one of mine with no artifacts apart from a single trunk branch) | 21:34 |
| lifeless | allenap: also we can raise the timout on qastaging instantly if needed (via a feature flag) | 21:35 |
| allenap | Ah yes. | 21:35 |
| lifeless | we should look at the timeout you got though, it may be a problem for prod too | 21:35 |
| allenap | lifeless: Okay, I'll file a bug for it. | 21:37 |
| lifeless | allenap: got an OOPS ID ? | 21:38 |
| lifeless | allenap: so, do we think this change is ok ? | 21:38 |
| allenap | lifeless: I think it's fairly safe, but I haven't been able to QA it so far. I also (or someone) needs to look in the shared catch-all mailbox. | 21:39 |
| allenap | Chex: Did you find a shared mailbox for qastaging? | 21:40 |
| lifeless | allenap: launchpad-ops, ping the losa :) | 21:40 |
| allenap | lifeless: OOPS-1762QS22 | 21:43 |
| jcsackett | bzr uncommit and bzr shelve may be the two best commands ever. | 21:43 |
| lifeless | :) | 21:43 |
| StevenK | bzr needs a 'bzr moo' | 21:43 |
| lifeless | garh | 21:44 |
| jcsackett | 'bzr garh' might actually work. :-P | 21:44 |
| jcsackett | "This version control tool doesn't have cow powers. It does however have a pained `lifeless`." | 21:44 |
| StevenK | jcsackett: It's super cow powers :-P | 21:45 |
| allenap | lifeless: I think it might be a cold cache kind of problem, because it's now snappy on staging, having timed out consistently 10-20 minutes ago. | 21:45 |
| lifeless | allenap: or load, ok. | 21:45 |
| StevenK | lifeless, jcsackett: http://paste.ubuntu.com/521648/ | 21:46 |
| jcsackett | StevenK: that's terrible. and hilarious. | 21:48 |
| wgrant | bigjools: Why can't we get logs? | 21:55 |
| wgrant | bigjools: Surely someone has access to the host... | 21:55 |
| bigjools | securitai | 21:55 |
| bigjools | it's a VM | 21:55 |
| wgrant | Yes. | 21:55 |
| wgrant | But there is a host. | 21:55 |
| wgrant | Host can poke at VM FS. | 21:55 |
| bigjools | which does not have any logs | 21:55 |
| bigjools | no, it can't | 21:55 |
| bigjools | securitai | 21:56 |
| wgrant | !? | 21:56 |
| lifeless | allenap: https://lp-oops.canonical.com/oops.py/?oopsid=1762QS22 - 11seconds of sql | 21:56 |
| wgrant | bigjools: Have you checked the bzr versions on the VMs? | 21:57 |
| bigjools | I've not checked anything | 21:57 |
| mwhudson | build-dependcies: ... package-that-adds-lamonts-ssh-key-in-postinst ... | 21:57 |
| wgrant | Unless you are running a very strangely mangled Xen, I don't see how the host can't poke around in the VM's filesystem and grab logs. | 21:58 |
| allenap | lifeless: That query has some EXISTS smells in it too. I'll file a bug for it now. | 21:58 |
| lifeless | allenap: exists isn't necessarily bad, but I agree that it would benefit from eyeballing. | 21:58 |
| lifeless | allenap: - so +1 on that | 21:58 |
| wallyworld_ | abentley: thumper: standup? | 22:03 |
| allenap | lifeless, deryck: I have to go now. There's still no word on the qastaging mailbox. I am also away tomorrow, but I can probably find an hour in the morning if necessary. | 22:05 |
| lifeless | allenap: how can I help | 22:07 |
| lifeless | allenap: e.g. if the mail is in the mailbox, its ok ? | 22:07 |
| abentley | wallyworld_: thumper is having a call with flacoste. | 22:07 |
| wallyworld_ | abentley: ok. i'llbe here when he's done | 22:08 |
| allenap | lifeless: Mail for bug changes should be sent to structural subscribers. If, say, a bug task is assigned to a milestone to which someone is subscribed, they should also get bug mail. Also distro subscribers should get source package bug mail. | 22:08 |
| lifeless | ok and in this case we shouls expect ...? | 22:09 |
| allenap | lifeless: The same bug mail as everyone else, but with a rationale that explains that they're subscribed to a milestone, or the distro. | 22:12 |
| lifeless | kk | 22:13 |
| allenap | lifeless: Just to check, r11794 is not-ok, but everything else can be pushed out? | 22:16 |
| allenap | So it's not blocking lots of stuff yet. | 22:16 |
| lifeless | allenap: its blocking 20 revs | 22:18 |
| allenap | lifeless: Ah, okay, deployment-stable.html doesn't show the subsequent revs that are prevented from rolling out. | 22:19 |
| thumper | abentley, wallyworld_: standup now? | 22:19 |
| wallyworld_ | yep | 22:19 |
| thumper | http://www.shopwithmaverick.com/wp-content/uploads/wpsc/product_images/meerkats%20wall%20back%20-4.jpg :-) | 22:21 |
| lifeless | allenap: yes | 22:25 |
| mars | allenap, fwiw, we're working on that | 22:26 |
| lifeless | Ursinha: hey | 22:30 |
| lifeless | Ursinha: what did you think of my qatagger improvement requests | 22:31 |
| flacoste | lifeless: https://code.edge.launchpad.net/~flacoste/launchpad/ppr-constant-memory/+merge/39578 if you have some time | 22:33 |
| lifeless | how much memory is it taking now ? | 22:35 |
| flacoste | lifeless: i'd say something like 800M, but that's because of the 400M cache size | 22:36 |
| flacoste | lifeless: i didn't get that locally (since 300k are using 56M of storage) | 22:36 |
| Ursinha | lifeless, I'm implementing the "codebrowse url on reports" now, which is fairly simple | 22:36 |
| lifeless | \o/ | 22:36 |
| Ursinha | the other one I'll have to take a deeper look | 22:37 |
| flacoste | lifeless: i'm it's using now 576M on sodium (still only inserting) | 22:37 |
| Ursinha | I'm fixing the bug that shows all revisions instead of all until the not-ok one | 22:37 |
| Ursinha | that flacoste reported | 22:37 |
| flacoste | with over 5M requests inserted | 22:37 |
| flacoste | i think it will think up to ~5h to generate a daily though | 22:38 |
| flacoste | since it took 2h last time | 22:38 |
| thumper | bzr missing --mine-only ../db-devel | 22:40 |
| * wallyworld_ afk briefly to do school dropoff | 22:47 | |
| rockstar | thumper, do you know much about custom widgets in forms? | 22:51 |
| rockstar | (sinzui was supposed to help me, but I think he forgot) | 22:51 |
| thumper | rockstar: some | 22:51 |
| thumper | whazzup? | 22:51 |
| rockstar | thumper, I need to use a different widget in the form in the user has launchpad.Admin on the form. | 22:52 |
| rockstar | thumper, we discovered a bug today where the select box for choosing on owner for a recipe re-assigns ownership of the recipe if an admin edits it. | 22:52 |
| thumper | rockstar: take a look at the branch edit form | 22:52 |
| thumper | rockstar: we do exactly that | 22:52 |
| thumper | rockstar: for admins editing branches | 22:52 |
| rockstar | I looked around, but couldn't see anything. | 22:53 |
| * rockstar looks | 22:53 | |
| thumper | rockstar: around line 1000 of code.browser.branch.py | 22:53 |
| rockstar | Oh crap. I think I figured it out... | 22:55 |
| rockstar | I was on the right track, just forgot to call the setUpFields on the superclass... :/ | 22:55 |
| LPCIBot | Project devel build (165): STILL FAILING in 3 hr 33 min: https://hudson.wedontsleep.org/job/devel/165/ | 23:12 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=jml][ui=none][no-qa] [r=jml]Fix layers to teardown always not just | 23:12 |
| LPCIBot | sometimes, | 23:12 |
| LPCIBot | fixing teardown in our code base and freeing us from the tyranny of | 23:12 |
| LPCIBot | atexit. | 23:12 |
| LPCIBot | * Launchpad Patch Queue Manager: [r=allenap][ui=none][bug=667687] Fix deactivateAccount to handle | 23:12 |
| LPCIBot | drivers correctly. | 23:12 |
| gary_poster | wallyworld_: back? | 23:27 |
| wallyworld_ | gary_poster: yes. thanks for the great email. i was going to respond this morning | 23:27 |
| wallyworld_ | i'll do the stuff you outlined | 23:28 |
| gary_poster | wallyworld_: great. happy to help | 23:28 |
| gary_poster | (all I was going to do was see if there was anything we ought to talk about before I ran away) | 23:28 |
| gary_poster | so I'm running away then :-) | 23:29 |
| gary_poster | bye | 23:29 |
| wallyworld_ | gary_poster: np. the email spelt it all out extremely well. bye | 23:29 |
| thumper | :) | 23:43 |
| thumper | Done. 9470605 items in 949 iterations | 23:43 |
| thumper | CodeImportEventPruner | 23:43 |
| lifeless | \o/ | 23:44 |
| lifeless | is that 9.4M items? how long did it take? | 23:44 |
| thumper | lifeless: a while, but not too long | 23:52 |
| thumper | lifeless: and yes it is 9.4M items | 23:52 |
| lifeless | thumper: how long, do you know? | 23:52 |
| thumper | 9.4 mostly useless items | 23:53 |
| thumper | lifeless: I can check | 23:53 |
| lifeless | please, I'm very curious | 23:53 |
| thumper | lifeless: it won't be as long with the next run time though | 23:53 |
| thumper | lifeless: as it'll only be deleting around 30k items | 23:53 |
| thumper | nah, 20k items | 23:53 |
| thumper | lifeless: 3778.052 seconds, | 23:54 |
| thumper | average size 9979.563215 (2506.74270824/s) | 23:54 |
| thumper | so based on the time there, daily garbo for this should take around 4s | 23:55 |
| lifeless | 1 hour?! wow. Was it multiple transactions ? | 23:55 |
| thumper | lifeless: yes, 949 of them | 23:55 |
| lifeless | whew :) | 23:55 |
| thumper | lifeless: written the garbo way | 23:55 |
| lifeless | yeah | 23:55 |
| lifeless | I trusted | 23:55 |
| lifeless | I'm just happy it panned out | 23:55 |
| thumper | it did | 23:55 |
| thumper | now to see if it makes a difference at all | 23:55 |
| thumper | hopefully with that and the fix I've done for mailing lists we should see a difference | 23:56 |
| thumper | on the private xmlrpc server | 23:56 |
| * thumper closes up for lunch | 23:58 | |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!
