[00:02] sinzui: yes, looks like it was r13573, bug 809508 [00:02] <_mup_> Bug #809508: Suggest branches with current bug number when associating a bug with a branch. < https://launchpad.net/bugs/809508 > [00:02] commit msg: Make the search box stay disabled and the spinner visible until all outstanding searches are finished [00:03] mp was self reviewed, so something slipped through there [00:05] wallyworld: Should we abort the rollout? [00:05] wallyworld: If you're unclear, just revert it [00:28] does anybody else get an error on this page: https://api.launchpad.net/1.0/~ubuntu-br/homepage_content ? [00:28] wgrant: ^^ [00:29] or lifeless [00:29] whoever is awake [00:29] cjohnston: this is the cause of our lpupdate failure I think [00:29] That doesn't look like JSON to me [00:30] hmm.. where is that coming from [00:31] it's part of the response from https://api.launchpad.net/1.0/~locoteams/members [00:31] that part of the json, brazil's homepage_content [00:32] is where our script if throwing exceptions [00:35] so something is wrong it looks like on LPs end in the Homepage Content area [00:35] possibly [00:35] I'm still not quite sure what though [00:36] I wander if its something with the localization of it [00:38] I think it's more likely that it's not specifying that it's unicode [00:51] StevenK: Your browser Accepts HTML, so it will give it HTML. [00:52] wgrant: Even wget doesn't show JSON [00:52] It does. [00:52] Well, curl does. [00:52] "Usu\u00e1rios do Ubuntu no Brasil.\n\nTime para usu\u00e1rios do Ubuntu no Brasil.\n\nComece Aqui: - http://www.ubuntu-br.org/comece\n\nP\u00e1gina Principal: - http://www.ubuntu-br.org\nWiki: - http://wiki.ubuntu-br.org\nPlaneta: - http://planeta.ubuntu-br.org\nF\u00f3rum - http://ubuntuforum-br.org/\nDocumenta\u00e7\u00e3o: - http://wiki.ubuntu-br.org/Documentacao\n\nAntes de entrar neste time, voc\u00ea deve ler, concordar ... [00:52] ... e assinar o C\u00f3digo de Conduta do Ubuntu (http://www.ubuntu-br.org/codigodeconduta)." [00:53] wget does too. [00:53] Content-Type is correctly "application/json" [00:53] No encoding is required, as it's ASCII. [00:53] mhall119: What's the error your script is throwing? [00:58] yeah, I'm getting json [00:58] wgrant: http://paste.ubuntu.com/658298/ [00:58] I can access the members collection without trouble. [00:58] I've tracked it down so simplejson trying to decode a string [00:59] Is it possible you've got a corrupt cache? [00:59] Try removing your launchpadlib cache. [00:59] I don't have access to do that on cranberry [00:59] and my local dev isn't even getting me past lp_login, not sure why [01:02] ok, I just need to get my local dev working to the point where I can watch this with a debugger [01:03] I doubt it will happen locally. [01:09] mhall119: there is a vangard on [01:10] cjohnston: I've already got them doing it [01:10] cool [01:10] now we wait until 10pm [01:10] the cron should have run again by then [05:12] Project devel build #945: STILL FAILING in 5 hr 48 min: https://lpci.wedontsleep.org/job/devel/945/ [07:02] wgrant: did you have any luck trying the new publish-ftpmaster yesterday? [07:06] jtv: It finished not long after you EODed. [07:06] jtv: Expedited security processing works fine. [07:07] You have made me so happy. [07:07] I'm pretty happy with the whole thing now, but I need to check over the hooks a bit. [07:07] I'd appreciate that — as soon as you're satisfied with them, we can make the switch. [07:12] jtv: They all look pretty sensible. [07:12] What's the worst that it could do... [07:12] *cough* [07:14] wgrant: then I shall proceed to move this towards production… [07:14] …albeit with some trepidation. [07:15] *immense [07:15] But I think we can do little else now. [07:26] good morning === almaisan-away is now known as al-maisan [08:31] Hi [08:42] ah yes, bigjools: when copying custom uploads from a previous Ubuntu series, I don't suppose the pocket matters much? [08:50] jtv: nup [08:51] Argh. But the changes file does. It'd be nice if I could just copy the LFA id, but that's not currently supported. [08:51] DistroSeries.createQueueEntry expects to have to store the changes file into the Librarian. [08:52] You are creating new PUs and PUCs? :/ [09:15] wgrant: Have to. What's the problem with that? [09:16] you don't have to; it's just easier than the rather messy and time consuming proper fix. [09:17] Perhaps understandable. [09:17] But definitely another pile in the hack pile. [09:17] another hack in the hack pile [09:17] That's the one. [09:19] It's like delayed copies, except even worse defined :( [09:30] Internet connections: you just can't trust 'em. [10:49] Is staging used to test anything in particular ? I wonder if now would be a good moment to get the branch with the newer bzr deployed there for the extended QA discussed on the list. [10:51] jelmer: DB patches (at least for the next week) and mailing lists and some build farm stuff. [10:51] Pretty much everything else is on qastaging these days. [10:53] wgrant: Ah, cool - thanks. [10:53] Time to land my branch and QA it then. [10:54] jelmer: Land it? Do you mean get it cowboyed onto staging? [10:54] Also, staging is having a bit of evil performed on it a bit the last couple of weeks, for testing fastdowntime. [10:54] Not sure if stub has much more of that planned. [10:54] ah, hmm [10:54] wgrant: waiting on pgbouncer just now, for prod. [10:55] lifeless: We're going with the 3m downtime? [10:55] wgrant: stub has it down to ~2m [10:55] wgrant: slony deletes and creates ~ 500 triggers on all nodes. [10:55] I had a 2-slave no-op update down to 45s here by skipping trusted.sql, IIRC. [10:56] The master takes around 12 seconds. [10:56] So the triggers don't take tooooo long. [10:56] we may be seeing cold cache table metadata on staging timings [10:56] I'm still looking at shaving time off. Skipping trusted.sql if unmodified is doable - just need to store a hash of the file in the db somewhere. [10:57] stub: Can't we treat trusted.sql like the rest of the schema? [10:57] Have a base + patches? [10:57] And we can deprecate updating it [10:57] lifeless: Surely the metadata should be pretty tiny. [10:58] wgrant: We already have had to with the bugsummarywork [10:58] wgrant: I'm wondering with whom to coordinate this exactly, is there a point of contact/process? [10:58] stub: Are you planning to do more testing on staging for fastdowntime? [10:58] jelmer: fastdowntime is live on staging, any more tests are just improving the process. [10:58] stub: Right, so let's do that everywhere and stop recreating all those functions each time :) [11:01] wgrant: The nice thing about trusted.sql is you can edit your code in place, get diffs etc. Current approach by punting that to the dbpatches means cut & paste to tweak a stored procedure, and locating the current implementation involves grep, and source will eventually disappear when we clean out the old db patches. Not sure if we can easily solve any of the downsides, but thoughts welcome. [11:01] stub: Agreed, and that really hurt during the bugsummary debacle. [11:02] But reapplying trusted.sql is really slow. [11:02] (cycling the baseline will need some more thought too, as it is tied up with trusted.sql, but I need to fix that already) [11:02] wgrant: Yup. It seems to be a per-statement overhead in there somewhere, which is why I already dropped comments.sql out of patch application. [11:03] I could be evil and package trusted.sql up in a stored procedure that applies the contents :-) [11:04] your timing on trusted.sql is good though - I hadn't gone to that granularity yet [11:05] I forget exactly, but I think removing trusted.sql saved around 40s. But it only started at around 100s, so I'm not sure what's up with staging. [11:06] make -C database/replication devsetup is awesome, btw. [11:09] Project devel build #946: STILL FAILING in 5 hr 56 min: https://lpci.wedontsleep.org/job/devel/946/ [11:21] stub: What intervals are staging's slons running at? [11:24] The variance gets a lot lower if I reduce the no-SYNC interval to 2s from 10s. [11:24] 39-41s without trusted.sql, rather than 45-55s. [11:25] Which I guess makes sense. [11:26] Since it's waiting for synchronization, with no replicatable write activity. [11:26] And if it's waiting for an extra sync on both sides of trusted.sql, that could explain much of the difference. [11:29] Nope, adding trusted.sql adds 45s even with 2s forced syncs. [11:49] wgrant: I landed a branch last night that drops staging down to 1 second syncs and 5 second forced syncs to see if it changes things significantly. [11:49] 1 second sync check, 5 second forced sync [11:49] stub: What was it before? [11:49] We may want to push through some update right before each sync. [11:49] 2 second sync check, 10 second forced sync (defaults) [11:49] Ah. [11:50] I don't think it will be significant - maybe shave 10 seconds off, or 15 if we drop the sync poll time to 0.5. [11:50] But worth testing in case. [11:50] Adding a third slave has added 15s to the trusted.sql time... [11:51] Which is about right. [11:51] 12-14s to apply to the master, then apparently a similar time to each slave, serially? [11:51] I'm still not entirely clear on how the DDL gets propogated. [11:52] I guess I should read more on that. [11:54] Blah, but it slows down the no-trusted.sql upgrade by nearly 4s too. [11:54] I guess that matches up fairly well, but is still disappointing. [11:54] Particularly if prod has ... 8 replicas? [11:54] Maybe 6. [11:54] I can't count. [11:58] wgrant: I'm looking into where to copy the latest custom uploads to a new distroseries. Julian suggests I might use the hook we already have for the case where a distroseries is being published for the first time. What do you think? [12:00] jtv: I suggest you do whatever is quickest and easiest to remove when we fix custom uploads properly. That may be that. [12:00] But it's possibly appropriate to put it right in IDS, if you're creating new PUCs. [12:00] Since it doesn't need to manipulate the disk directly. [12:00] But it also doesn't currently use DSP. [12:01] DSP? [12:01] DistroSeriesPArent? [12:01] Yes. Well, the code I have now doesn't actually care whether you use it or not; but the initial plan is to do this based on previous_series. [12:01] Or whatever the old parent_series got renamed to. [12:01] Right, that sounds somewhat sensible. [12:01] Still more appropriate for IDS IMO, if you're creating PUCs. [12:02] Does IDS get run just once per derived series? [12:02] Yes. [12:02] yes, but even creating PUCs is not enough, you need to force them through process-accepted [12:03] You'll need to create PUs and PUCs, and process-accepted will handle them in its own special way. [12:03] it makes me shudder [12:03] I believe I've expressed my opinions on process-accepted and this new strategy. === vila_ is now known as vila [12:04] least-worst option for now. We don't have time to redesign this [12:04] Yes. [12:04] It is terrible, but probably least-worst. [12:04] * bigjools observes the ivory towers in the distance [12:06] wgrant: An interesting thing about trusted.sql is that it doesn't need to be applied via slonik at all [12:06] stub: It shouldn't have to be, no, so we could possibly work around it. [12:06] But it may still be slow without it. [12:07] yes, but I'm looking at 2 minutes. If I can shave 40 seconds off that, it is a big win percentage wise. [12:07] It does mean it is being applied outside the db patch transaction [12:07] The no-op upgrade SQL (just the two UPDATE LDR statements) takes 15s here when executed directly through slonik with three slaves. [12:07] So we have 30s of overhead in the script. [12:07] Probably the sync on either side. [12:08] bigjools: I need to force my copied PUCs through process-accepted? Won't that run anyway? [12:08] jtv: Yes. [12:08] Just create them on new ACCEPTED PUs, and all will be good. [12:08] jtv: it's ok as long as there's a PU in ACCEPTED state [12:08] Well, bad, but it will work. [12:09] it is a horrible hack but then custom uploads are a horrible hack :( [12:09] So I have to create my PUs as ACCEPTED? [12:09] yes [12:09] Given we have already applied stuff from trusted.sql live, I guess we can live without it being applied in-transaction with the db schema updates [12:09] jtv: Maybe just create them and then pu.setAccepted. [12:09] that will also work [12:09] Er. [12:09] No, there's setDone... don't think there's setAccepted. [12:10] Just acceptFrom* [12:10] But one of those might work. [12:10] stub: Hm. [12:10] stub: trusted.sql applies in 100ms directly. [12:10] Let me force it through slonik and see what happens... [12:11] 13s. [12:11] WTF [12:11] So something is causing it to execute all the syncs serially, or something. [12:12] Where that doesn't happen with the no-op patch SQL. [12:12] * wgrant tries merging them into one file. [12:13] With the minor issue that they already end up as one file. [12:13] So where is the extra 45s coming from. [12:14] Blaaah. [12:14] The 13s overhead is on each execute_script. [12:17] wgrant: When we check for sync, we wait for the sync to be confirmed by ALL nodes. So that blocks until a node has finished doing something like apply a patch and it gets around to acking. [12:17] stub: Yeah, but the patches take <100ms each. [12:17] So either the slon goes to sleep for 15s after applying the patch, or something else is happening. [12:24] stub: Overhead vanishes if I change upgrade.py's run_sql to merge it all into one big file, with a single execute_script. [12:24] wgrant: I'm thinking on why patches sometimes get serialized, rather than applied on slaves simultaneously. I think slave2 might block on slave1 saying 'sure - go ahead' because slave1 has already started processing a dbpatch. [12:24] Well, it's down to below 2s, at least. [12:24] cool [12:24] we don't do that already? [12:24] No. [12:24] oh... one execute script [12:24] We create one slonik script. [12:25] With multiple SQL files, and multiple execute_script stanzas. [12:25] It looks like the syncs around the second one end up serialised, somehow. Might watch slon logs tomorrow. [12:25] Which is really going to hurt production, if it's 16s per slave rather than 3s. [12:25] so we want an assert in there that len(open(script).readlines()) < 1000, or we overflow a slony constant I think set at compilation time [12:25] dafuq? [12:26] Seriously? [12:26] yup [12:26] we are weird. agile is foreign to most db environments and dbas [12:26] Our sort of automation tends to scare the bejesus out of people. [12:27] stub: Are the syncs in execute_script insufficient? [12:27] We currently seem to sync on either side. [12:27] Which is an extra $while. [12:27] Overhead in upgrade.py is still 30s. [12:28] Will be more on prod. [12:28] But I wonder if you can try the upgrade.py SQL merge thing on staging? [12:28] See how far down it takes it. [12:28] Even better if you can reduce the forced sync interval. [12:29] wgrant: Sure. Stick in a MP for db-devel (we want the fastdowntime stuff on that branch still) [12:29] I have no problem lowering the sync check and forced sync times if it helps. It won't affect our load. [12:30] As you say, it should only save a few times the reduction. Might get better numbers tomorrow when I am awake. [12:31] ./src/parsestatements/scanner.h:#define MAXSTATEMENTS 1000 [12:31] Kill me now. [12:31] aaaaaaa [12:31] This is not sensible! [12:31] ./src/parsestatements/scanner.c:int STMTS[MAXSTATEMENTS]; [12:31] D: [12:32] malloc is hard, I guess. [12:32] wgrant: Shouldn't be a genuine problem given we won't want a backlog of patches to apply in one hit. [12:32] wgrant: esp if we decide to pull trusted.sql out [12:33] $ wc -l /tmp/trusted.sql [12:33] 2111 /tmp/trusted.sql [12:33] I guess there are some nice long statements in that. [12:33] Will be fun counting them. [12:33] yes - a whole stored procedure counts as '1' [12:33] oic [12:33] Yeah. [12:34] You are counting semi colons out side of strings, which you can assume are 'string' or $$string$$. Or just ignore the problem for now. [12:34] I think it will be ignored for now. [12:34] Will see how upgrade.py copes if it's violated. [12:34] I mean, it will probably cause slon to segfault or something... [12:34] No biggie. [12:35] it will barf in the transaction and roll back. [12:35] meybe picked up at slonik compilation time [12:35] Hah, all my logging changes conflict with yours. [12:35] comments.sql will make it explode :) [12:39] * stub disappears for an hour [12:51] stub: https://code.launchpad.net/~wgrant/launchpad/single-execute_script/+merge/70432 [12:59] * wgrant sleeps. [13:00] g'night wgrant [13:03] abentley, when you are around, if you could let me know which of bugs 820510, 820511 and 820516 are worthy of critical, or help me decide at least, maybe yellow squad could take one or more of those. === wgrant changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 247 - 0:[#######=]:256 === allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap | Critical bugs: 247 - 0:[#######=]:256 [13:06] gary_poster: 820511 and 820510 give the best bang for buck, but I don't think either one is truly critical, and marking them critical makes it hard for us to see what's truly critical. [13:07] abentley, adeuring: I have to switch locations now. Should be back in time for the stand-up. [13:07] henninge: cool. [13:09] abentley, ok thanks. "hard to see what's truly critical"...I don't think it's any different than the existing situation with the critical bug tag, but let's not go there. Practically, do you agree with wgrant that this is something that should be tackled now, even when our focus is supposed to be on "critical" bugs? [13:11] gary_poster: I think it makes sense to tackle soon. I don't think the focus on "critical" bugs is meant to exclude us working on anything else. [13:12] abentley, ok. I'll encourage yellow to grab at least one of those, asking you for guidance when we get there. [13:12] thanks [13:13] gary_poster: np. [13:15] allenap: O HAI! [13:15] StevenK: HAI! [13:16] Whassup? [13:16] allenap: You marked https://code.launchpad.net/~stevenk/launchpad/populate-bprc/+merge/69412 as Needs Fixing a week ago, and you have since been smacked down by wgrant and lifeless. Can you reconsider? :-) [13:17] StevenK: There were other points in the review :) [13:18] allenap: Right, so I'll look at getCandidateBPRs(). I disagree about the loop size -- if it gets killed half-way through a loop, for example? [13:18] allenap: And I don't agree the test is dependant on test data. How? === jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: allenap, jcsackett | Critical bugs: 247 - 0:[#######=]:256 [13:20] StevenK: Why do you only want to do one thing round the loop? I suppose it keeps transactions short, but it's meant to tune for than anyway isn't it? [13:21] allenap: So, each iteration reads a .deb from the librarian, parses the contents and starts tossing rows into BPRC and BPP. If that loop gets interruptted, do I end up with half of the data in the tables and half not? [13:22] StevenK: No? Why is that a concern? I don't think it interrupts in the middle of a run. And transactions. [13:22] allenap: My basis for this was the PSC work -- and that was actually unpacking source packages and dealing with a bunch of temporary files, so it was more critical it wasn't killed in the middle of a run [13:23] allenap: Conceeding the loop tuning question -- I'll bump it 20 or so [13:23] StevenK: Okay :) [13:24] allenap: So, the test is dependant on sample data? [13:25] StevenK: It looked that way, but I didn't dig much. If you say it's not then that's all I want to know. [13:26] StevenK: What about 3? That doesn't actually look like it's going to work how it is. [13:26] allenap: I didn't want to include another .deb in the tree, so I make use of one that's already there due to archiveuploader tests [13:26] StevenK: Oh, ignore me. EPARSE. [13:27] StevenK: So, r=me now, but consider point 4. [13:27] You'd prefer self.layer... ? [13:27] Wait, ignore me, I can't read. [13:33] allenap: -1 for including the IRC log in the MP :-P [13:34] StevenK: I just added a tl;dr. What's wrong with an IRC log? [13:34] allenap: Just teasing :_P [13:34] :) === al-maisan is now known as almaisan-away === Ursinha-afk is now known as Ursinha [14:20] allenap: I've fixed my MP, could you add it to your queue? https://code.launchpad.net/~rvb/launchpad/bug-820900/+merge/70434 [14:20] rvba: Sure. [14:20] allenap: Thanks. [14:22] rvba: r=me [14:22] allenap: \o/ [15:34] thanks mrevell [15:43] Hi cjohnston [15:43] no problem at all cjohnston :) thanks for your help === beuno is now known as beuno-lunch [16:13] jcsackett, can you review https://code.launchpad.net/~sinzui/launchpad/person-picker-expand-1/+merge/70459 [16:14] The MP is lying about the changes from the prerequisite branch. I attached the actual changes [16:14] sinzui: thanks, i was just looking at the MP diff and getting confused about the scope of the changes. :-) [16:23] sinzui: r=me. [16:27] jcsackett, thank you. [16:27] sinzui: you're welcome. :-) [16:42] zomg. === allenap changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: jcsackett | Critical bugs: 247 - 0:[#######=]:256 [16:42] err, wrong channel :P [17:03] Project devel build #947: STILL FAILING in 5 hr 53 min: https://lpci.wedontsleep.org/job/devel/947/ === beuno-lunch is now known as beuno [17:45] jcsackett: Could you please review https://code.launchpad.net/~abentley/launchpad/induce-latency/+merge/70471 [18:01] abentley: sorry, i missed your msg; i'm looking at the MP now. [18:01] jcsackett: cool. [18:05] abentley: looks good to me, r=me. [19:50] bug 820510 [19:50] <_mup_> Bug #820510: hard to turn on extra logging for twisted job runner < https://launchpad.net/bugs/820510 > [19:54] So, I killed rocketfuel-setup yesterday. I think it was right in the middle of the "bzr pull" step. Now, when I run "rocketfuel-get", I get "ERROR: No WorkingTree exists". [19:54] BTW, which pastebin do we use here? [19:55] any that ou want [19:55] How do I fix my bzr repo? (I'm familiar with hg and svn, but I'm a bzr n00b.) [19:56] * sixstring searches for paste.lifeless.org. ;) [19:56] http://www.pastie.org/2321492 [19:57] put set -x at the top of rocketfuel get so you can see the failing bzr command [20:13] It's failing at "bzr up ~/launchpad/lp-sourcedeps/download-cache". [20:13] In hg world, I'd just blow away that directory and re-clone it. Any bzr advice? [20:14] BTW, the bash mojo in rocketfuel is impressive. [20:21] OK, I'll try this: (1) mv ~/launchpad ~/launchpad-busted (to get it out of the way) then (2) re-run rocketfuel-setup. [20:27] That seemed to work. [20:27] I don't suppose there's a log-bot logging this stuff for posterity? === matsubara is now known as matsubara-afk === jcsackett changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: - | Critical bugs: 247 - 0:[#######=]:256 [22:54] Yippie, build fixed! [22:54] Project devel build #948: FIXED in 5 hr 50 min: https://lpci.wedontsleep.org/job/devel/948/ [23:35] wallyworld: You said you had a better bzr plugin? [23:35] wgrant: yes, i'll email it to you [23:35] Thanks. Going to submit it upstream? [23:36] ooh, for what? [23:36] poolie: PyCharm. [23:36] that'd be good to send it up to them [23:36] if it has significant new features maybe we can put it on the blog? [23:36] wallyworld: Also, do you have any hints on setting up the project? You said to create it outside the branch, but I'm not sure how to get everything into it. [23:37] The original is already on LP (https://launchpad.net/bzr4j) [23:37] wgrant: do you know why sinzui marked bug 345349 invalid ? [23:37] <_mup_> Bug #345349: Notifications about private branch linkages sent to unprivileged subscribers < https://launchpad.net/bugs/345349 > [23:38] wgrant: the email may help. my copy of the bzr plugin has changes which i've submitted to the lp project but which have not yet been merged in [23:38] lifeless: No, and the call finished like 2 minutes ago :( [23:38] wgrant: read the email first and then ping me with any questions [23:39] wallyworld: Indeed, that helps a lot, thanks! [23:39] wgrant: np. it's a very terse summry. there's potentially a lot more you may need help with. but see how you go.... [23:42] wgrant: also, i have debuging working when running lp inside an lxc container [23:42] wallyworld: oh nice [23:42] wallyworld: what did you do to make that work ? [23:43] you have to add a couple of lines to the top of bin/run [23:43] and tweak a setting inside the debug config of pycharm to tell it to connect to a "remote" instance [23:45] lifeless: you did realise we were talking about pycharm? i'm not sure about debugging using pdb etc [23:46] wallyworld: yes, I did [23:46] cool, just checking :-) === wallyworld changed the topic of #launchpad-dev to: https://dev.launchpad.net/ | On call reviewer: wallyworld* (jtv) | Critical bugs: 247 - 0:[#######=]:256 [23:56] wallyworld: Should I create a separate empty directory for the project? [23:56] I can't find a way to add external stuff. [23:57] wgrant: yes, in my case, i created a ~/PyCharmProjects directory and saved the lp project there [23:58] then you can got o settings->project structure and add a content root [23:58] the content root is your working tree [23:58] Ahh, thanks. [23:58] and from there you mark stuff as sources and exclude things as per my screenshots [23:59] Yup.