[01:14] sinzui: I have a output log for your script run on staging, not sure if it completed properly or not. Do you want to see the log? [01:45] spm: Has gina been crashing for ~ a week, or is it robust enough to withstand the un-understandable packages which have turned up in the Debian archive? [02:18] that was a weird oopsy crashy thing [02:19] Hm? [02:19] my machine more or less hung [02:19] only more or less though [02:19] i could type into bash [02:19] but the processes i started didn't get anywhere [02:20] Hmm. Disk access was hanging? [02:20] mwhudson: 'dmesg' next time [02:25] lifeless: that was one of the things that didn't do anythign [02:25] and i think it recovered actually, but i was into the reboot by then [02:40] wgrant: I'm not personally aware of any failures - but have been away for a few days... [02:49] EdwinGrubbs: thanks for getting that QA done on staging! Just to confirm, I'm guessing you also qa'd the 'distroseries milestone timeout' there (the CRB page only has the +index page entry as qa'd). Is that right? [02:50] Welcome back spm :) [02:50] noodles775: The previous fix applied to both bugs, and I erroneously thought that new fix did also. it is not timing out on staging, but I don't know why not. [02:53] noodles775: :-) [03:01] noodles775: _surely_ you should be asleep [03:02] It's only 4am! [03:02] mwhudson: normally yes, but I'm just a little keen to see that things go smoothly ;) [03:02] i see [03:03] * mwhudson reminds himself to never volunteer to be RM [03:04] mwhudson: but you had all that experience with BB. You should be poifect for the role! [03:04] spm: itym "have enough mental scar tissue" [03:05] * spm notes that trolling when the other person can't do Punch-In-The-Nose over TCP/IP is perferable [03:05] mwhudson: same same [03:06] Isn't the release ~6 hours away? [03:06] mwhudson: speaking of which, do you know where the two NEEDSTESTING remaining items on the code team testplan. [03:06] I don't see any announcements of it anywhere. [03:07] noodles775: it seems that my fix didn't [03:08] noodles775: I need to try again to work out what the problem is [03:08] wgrant: yes it is, I sent the schedule to lp-dev, I'm not sure whether mrevell has an email prepared... [03:09] mrevell won't be up before the release. [03:09] mwhudson: up for the call? [03:10] thumper: ok, if your certain, move it to BAD (?), and let me know if you plan to get it in a re-roll. [03:10] noodles775: not entirely certain [03:10] noodles775: I'm going to try to reproduce locally [03:10] noodles775: but apparently the revision has been rolled out and the page still oopses [03:10] noodles775: so in that sense it is bad [03:11] thumper: ok [03:12] noodles775: wait though [03:12] noodles775: I think it could be the staging librarian masking it [03:12] :) yay, ok. [03:12] spm: how do we get stuff from the production librarian to the staging librarian? [03:12] spm: what is copied across? [03:13] thumper: istr it's proxied - kinda sorta handwavy. [03:13] hmm... [03:13] http://staging.launchpadlibrarian.net/30419774/mp-unicode.msg [03:13] spm: gets processing failed [03:13] spm: any ideas how or why? [03:14] huh. and the real one works. looking... [03:16] thumper: one sec [03:16] Do you guys know about abentley's entry? bug 436325? [03:16] Bug #436325: Diffstat generation chokes on binaries (and others) [03:18] wgrant: mrevell will be around *for* the release, but I take from what you're saying that there's usually an announcement email sent to LP-users before the release? [03:19] * noodles775 looks through the archive [03:19] thumper: ok call? [03:19] noodles775: Normally sent to launchpad-announce, I think, so that people know that LP will be unavailable. [03:19] gr... I don't even know if I'll have access to send there. Thanks wgrant... checking now. [03:20] mwhudson: yep [03:22] noodles775: They've classically been sent 24-36 hours before. [03:23] However, I hope that lists.c.c announcement lists have a not-after-1am guard enabled. [03:23] wgrant: yeah - I'm not sure why it's not been sent this time, but I'll update the rm-wiki notes to put a check in there. Organising now. [03:24] noodles775: Thanks. [03:24] People get unhappy enough even when it is well announced. [03:24] :/ [03:25] thumper: haha. the staging librarian is down atm; staging rollout in progress. give it another 10-20 and try try again. [03:30] mwhudson: OOPS-1404EB48 [03:31] https://lp-oops.canonical.com/oops.py/?oopsid=1325EA163 [03:45] spm: is the staging librarian up? [03:45] nope. something looks nicely borked... [03:46] mwhudson: http://pastebin.ubuntu.com/309108/ [03:48] thumper: alas. it is still working - the restore is gradually building eggs. perhaps abnother 45 mins wait may be appropriate :-/ [03:49] noodles775: it may be fixed [03:49] noodles775: it certainly works locally [03:50] great! [03:50] noodles775: I clicked on the link in staging which took me to edge, not staging [03:50] and I didn't notice [03:50] :-| [03:50] thumper: do you know about abentley's bug listed there too? [03:51] noodles775: yes, it is bad but not a blocker [03:55] wgrant: ahh. in answer to your Q earlier about gina. I (you probably already know...) present bug 449408 [03:55] Bug #449408: Need scriptactivity monitoring of "gina" [03:56] spm: I've seen that, yes. [03:56] gina is a very special breed of evil. [03:56] wgrant: that means a lot; hearing you say that as in. ;-) [03:57] * spm has visions of gina owning it's own plane of the netherhells [03:57] Heh. [03:58] It'll need a bit of hacking soon :( [03:58] by the sound of; hacking with an axe [03:58] anywas. bbs. school run time. [03:58] Just with some v3 support. [04:08] Is postgresql any faster than mysql? [04:08] random question for the channel :P [04:08] Is that a valid question? [04:08] I don't have any basis for knowing [04:17] https://dev.launchpad.net/Soyuz/Specs/BuilddManagerUploadDecoupling [04:17] https://dev.launchpad.net/Soyuz/Specs/BuilddGeneralisation [04:27] wgrant and others: I've been chatting with flacoste and we've decided to reschedule for Thursday 5th 09.00 UTC due to the lack of announcement. [04:27] noodles775: Probably a good idea. [04:27] potentially 90mins of complete offline (no read access) is too scary - and we still don't have anyone who can send to lp-announce atm. [04:27] But... 5:30am. Ouch. [04:28] Ooh. [04:28] That's bad. [04:28] No read only? [04:28] wgrant: nope - librarian machines are being updated. [04:28] Ah. [04:28] That really does need an announcement. [04:28] yup. [04:29] updating identica now... [04:29] Thanks for working that out. [04:29] ... and thanks for pointing that out :) [04:49] jamesh, got time to help with a nagging problem I've been having that probably involves storm? [04:49] jtv: sure [04:49] great, thanks. [04:49] The problem is this: we have a script that's using up too much memory. [04:50] have you thought of buying more memory? [04:50] It shouldn't be pinning anything in memory across iterations of a particular loop. [04:50] yes, we considered that but I'm Dutch. [04:50] The only class from the Launchpad source tree that it seems to be allocating in great numbers is DBEnumVariable. [04:51] How hard is it to backup a launchpad server if you run everything on a single system? [04:51] jamesh: There are no other LP objects in memory that could explain having so many of those around. [04:52] MTecknology: not very, just back up the database. [04:52] jtv: and file system? [04:52] MTecknology: and... sleepwalking again? :) [04:52] well, you generally get one Variable for each field of each live instance Storm object instance [04:52] And the bzr branches, and the librarian. [04:52] jtv: hm? [04:52] and a few get created during expression compilation, but they shouldn't last long [04:53] MTecknology: oh, that'd be codehosting. I wouldn't know about that part. :/ [04:53] jamesh: this is after transaction.commit() & gc.collect() [04:53] jtv: what code in particular are we talking about? [04:54] just so I've got some idea of what's happening [04:54] jamesh: in the LP tree, lib/lp/translations/scripts/message_sharing_migration.py [04:54] jamesh: in there, look for _mergeTranslationMessages [04:55] * jtv sneezes [04:55] damn this cold weather [04:56] jamesh: that loop looks a lot more complex than it is [04:56] most of it is just ways to avoid pinning ORM-backed objects in memory across iterations [04:57] in essence it's "for potmsgset in potmsgsets: for tm in potmsgset.getAllTranslationMessages(False, prefetch=False): message.shareIfPossible()" [04:58] The rest is just a dance to make it store only objects' ids across iterations. [05:01] jtv: http://paste.ubuntu.com/309149/ <- I haven't tested this code, but it should give you some idea of what objects Storm knows about [05:03] jamesh: that may not tell us much... there's >100× more of these DBEnumVariable objects floating around than there are objects in Store._alive [05:03] (In the dump I have) [05:04] jtv: do you know what's referencing those objects? [05:04] If only I did... [05:04] This is where the storm internals start to matter, I suspect. [05:04] gc.get_referrers() can help [05:04] Oh, I knew about get_referents but not this one [05:05] I don't have any good sample code to play with for it [05:05] no worries, I can slot this right into our debug dumps. [05:06] you sometimes need to use it recursively though [05:06] the reference chain for most instance attributes is instance -> dict -> attr [05:06] The current dumping code is ready to go through 2 levels [05:06] so again no worries there [05:06] I just thought there was only get_referents and not get_referrers [05:07] get_referrers is a fair bit more expensive [05:07] since you essentially need to ask all the container objects whether they reference the one in question [05:08] Anyway, I have in memory: 92 POTMsgSets, 65 Languages. That's about it for the real Launchpad objects. Then, 19K DBEnumVariables, 334 CachedPropertys, 123 DBSchemaEnumCols, 99 PublicPersonChoices, 71 ContextTitles, and change. [05:09] The Language objects may get cached somewhere; I haven't bothered to look. I have no idea why the POTMsgSets would stick in memory unless Storm just likes to keep them in cache across transactions. The rest is a complete mystery. [05:09] how many enum columns are on each of those classes? [05:10] e.g. if POTMsgSet has 5 enum cols, then it would account for 460 DBEnumVariables [05:11] jamesh: might bools be implemented in this way? [05:11] no [05:12] jtv: DBEnumVariable has an _enum attribute pointing at the associated lazr enum [05:12] then judging by the interfaces, there's one enum in Language and one in POTMsgSet. [05:12] might help to see what those variables are for [05:15] https://pastebin.canonical.com/24271/ has counts for non-lp objects too. [05:15] might be an easier way to lay the blame than get_referrers [05:16] (scroll down to the last section for the meaningful results - should have trimmed the earlier sections) [05:16] jamesh: as per stub's clever suggestion we're dumping referrer info for some random samples [05:18] So 108423 storm.variable.IntVariables lying around, most likely buried in tuples or dictionaries along with functions (I suspect validators) [05:19] so, for a storm object you have a dual ObjectInfo that Storm manages behind the scenes [05:19] inside that object there is a tuple of Variable objects representing the primary key [05:19] (We are actually making a list of integer ids rather than some object that can be cast to an integer, aren't we?) [05:20] and a mapping from from columns to variables for the other fields [05:20] stub: good question... I'm just getting [obj.id for obj in result] [05:20] so IntVariable in a 1-tuple would be primary_vars [05:21] Variables as values in a dict are probably other columns [05:23] stub: I pushed a new version of the branch that, for the random samples, dumps referrers instead of referents. [05:23] maybe run again with that? [05:24] k [05:24] Gimme a package ;) [05:26] jtv: Did you tweak the filter? We don't seem to have enough launchpad database objects to account for everything so I suspect we need to turn the object filter off like I did last run [05:26] stub: package: eog [05:27] Gah. What is the branch again btw? Lost my shell history. [05:27] stub: can do... it's a matter of removing the "filter" argument from the dump_mem call [05:27] lp:~jtv/launchpad/debug-mem-2 [05:29] stage 1 underway (using my existing local hacks for no filter and more readable sample, not that the sample turned out to be useful) [05:29] I commented out the filter on my end as wel [05:29] l [05:30] what's the local hack for a more readable sample? [05:31] jtv: https://pastebin.canonical.com/24309/ [05:32] I don't think the exception handling on get_class is needed... hasn't barfed on me yet [05:32] has it barfed on you? [05:34] jtv: It did when I was trying to inspect contents of tuples - some bzrlib late loading stuff. Not needed now I suspect. [05:35] I was trying to categorize the tuples but it was too noisy. [05:36] * stub goes to the fly lice sho [05:36] stub: bong appetit [07:07] Morning :) [07:08] Morning mrevell -- want to announce the downtime? [07:10] wgrant, This is me back after a couple of days holiday. Roll-out downtime looks as though it's on the status page to me. [07:13] mrevell: So that has completely replaced launchpad-announce as the announcement mechanism? There has always been an email to launchpad-announce 24-36 hours before. [07:13] wgrant, I'm just checking to see what's been done and what hasn't... [07:15] wgrant, We're still waiting on definite confirmation of the roll-out time. I'll announce as soon as I have it. [07:16] mrevell: OK, great. This one is particularly important to announce well ahead of time... [07:20] wgrant: http://identi.ca/launchpadstatus [07:21] al-maisan: I know that, but most people don't. [07:21] wgrant: great. [07:22] And people rightly get unhappy when LP goes down for 90 minutes without an announcement. [07:22] wgrant: agreed. [07:26] jamesh: one thing we keep seeing in the random samples of live objects is _break_on_local_diverged, so presumably we've got a lot of those [07:26] that'd be the Reference() caches [07:27] If you have a reference Foo.bar using Foo.bar_id => Bar.id, Storm caches the value of Foo.bar and sets up an event handler to break that connection when Foo.bar_id changes [07:27] (such as happens when you commit the transaction [07:29] So what happens to those event handles when we commit? [07:30] on commit, all alive objects are invalidated [07:30] so non-primary vars are cleared [07:30] You've mentioned primary vars before.. what are they? [07:30] the columns that make up the primary key [07:31] just "id" for pretty much everything in Launchpad, IIRC [07:31] Yes [07:31] the primary vars aren't cleared because that's what is used to tie the Python object to the database row [07:37] jamesh: that begs two questions: why is the Python object still alive? And why so many more of these than live ORM-backed objects? [07:38] With the delay, is 3.1.10 absolutely frozen, or is there a possibility of getting in a trivial fix (in code only used by sync-source.py) to prevent crashes caused by v3 source packages breaking the autosyncer? [07:39] jtv: well, one possibility is that you've hit a bug. [07:39] but I don't really have enough info to say one way or the other [07:41] I also saw one dict that just mapped the string "remote" to a POTMsgSet. Would that be a Storm thing as well? [07:41] https://bugs.edge.launchpad.net/storm/+bug/435962 was reported recently about References from primary keys. I don't suppose that would apply to your situation? [07:41] Bug #435962: Reference can return a cached invalidated object if the local key is the primary key [07:41] that'd be the cache inside the Reference, yes. [07:41] jamesh, jtv: Here is the final log from the eog export: https://pastebin.canonical.com/24314/ [07:41] c/export/merge-job-thingy [07:42] stub: thanks [07:42] we really need better __repr__() on the various Storm classes [07:43] stub, jamesh: note that this dump shows referrers for the random sample, not referents [07:46] Thats a lot of EventSystem instances [07:48] about one per objectinfo, so I'd guess those are the callback hooks for key changes [07:48] about as many "function" objects... are those generated on the fly for each object? [07:50] "frame" is a stack frame, meaning a local variable? [07:51] I think so. If they were frames stored in tracebacks we would see object count for that. [07:52] * stub toys with the idea of disabling the cextensions implementation of EventSystem as a shot in the dark [07:53] jtv: Got another package for me to prepare, running the first two stages? [07:54] I'd be a bit surprised if the EventSystem impl was introducing bugs [07:54] stub: file-roller [07:54] maybe users of the API, but not the actual implementation [07:55] So would I actually (everything else would be exploding too, although this is our most extreme job in the current code base as far as object loading is concerned) [07:59] do we have an easy way to find out how many objects were loaded overall? [08:04] stub: so... are you trying that idea of disabling cextensions? Shot in the dark it may be, but it hits a lot of stuff we can't see. [08:06] I'm running stage 1&2 of fileroller in preparation for whatever we do next. That might be disabling EventSystem from cextensions, better logging of the events so we can see what is holding onto them, or whatever you come up with ;) [08:06] At the moment, I'm reading email. [08:07] stub: I for one am way out of my depth by now [08:08] And wtf would a stack frame be referencing all this stuff? [08:10] you mentioned other ways of referencing frames... could something be holding a whole stack incarnation in memory? [08:10] like maybe a cextension forgetting to unregister something even? [08:21] jamesh: is it possible for a C extension to pin a stack frame in memory, if it forgot to drop a reference or something? [08:22] jtv: I wouldn't think so. Not any more than a Python stack frame [08:22] I mean, could it pin a Python stack frame in memory? [08:23] So many of these objects that should live inside Storm-backed objects seem to be referenced from the stack. [08:24] stub said something interesting earlier: when I do [obj.id for obj in query_result], do I really get ints? Or is there any chance that I might be getting IntVariables or something? [08:25] IIRC tracebacks and exceptions can leak stack frames if not handled correctly, but I don't think we are seeing that. [08:25] obj.id should be returning an integer - it always has in the past [08:27] jtv: well, the only Python stack frames that stick around outside of the C call stack are generators [08:27] all the others are essentially tied to C function calls [08:33] * jtv is grepping for 'yield' [08:34] comprehensions are generators too I think [08:36] you can have references to local variables held in cell objects too when you work with closures [08:36] (i.e. a nested function that uses variables from its parent scope) [08:37] stub: less likely that we're keeping references to those around though [08:38] so that's what those cell objects are... not getting many of those though [08:38] I saw those cell instances but have no idea what they are. [08:39] jamesh: oh, you're saying a cell can hold actual stack frames in memory, as I was suspecting the cextensions of doing? [08:40] no [08:40] a stack frame can hold cells that can hold locals [08:40] (assuming you haven't been storing tracebacks and the like in local variables ...) [08:40] (we haven't) [08:42] the cells don't seem to be doing much in the scheme of things [08:42] So am I correct and the current thinking is we are somehow leaking ObjectInfo instances (ObjectInfo's have a self._event==EventSystem)? [08:43] good morning [08:43] hi adeuring [08:43] hi jtv! [08:44] stub: that's my impression, but the frames seem to be doing it. [08:46] looking at https://pastebin.canonical.com/24314/ random sample 3, there also seems to be an EventSystem referencing an ObjectInfo. [08:46] Through a frame, so the EventSystem seems to be holding a frame in memory. [08:46] None of our code will deal with ObjectInfo directly. [08:47] A Store has an EventSystem, as does each ObjectInfo [08:47] oh, wait, it's a list containing a frame and an EventSystem. [08:48] So we have [frame, EventSystem], where frame holds a weak reference to an ObjectInfo. [08:48] hmm... weak ref shouldn't be the problem though [08:49] but we are seeing lots of that pattern: a list containing a frame and one other reference. [08:50] Is that something that happens in the ObjectInfo? [08:53] Or is it something that python does for generators or closures? [09:04] prelim done [09:04] cool... now to figure out what to do with it [09:04] I notice that there are 5 twisted.Failure objects [09:05] wonder if those might be holding stack frames? It'd be odd for them to hold frames with Storm stuff, I guess [09:09] I have no idea how Twisted would be involved in a script like this... does storm talk to it? [09:10] bigjools: hi [09:11] hey thumper [09:11] bigjools: is it worth us having a call? [09:11] thumper: what do you want to talk about? [09:11] bigjools: have you looked at your email? [09:11] bigjools: I sent something about 4.5 hours ago [09:11] I have a shitload of email! [09:12] and kmail is a PITA, it hangs folder views when resuming >:( [09:12] one sec [09:13] thumper: there's no chance of this working at the end of 3.1.11 [09:13] nor 3.1.12 [09:14] why? [09:14] there's too much work to do, and many of the key people are sprinting in November [09:14] jamesh: we've got 5 Failure objects hanging around Twisted... any idea where those might come from and how we get rid of them? [09:14] I'm not saying we migrate soyus code away [09:15] just make it possible to run jobs in a protected environment [09:15] jtv: I imagine that is just noise from all the component architecture gumph that loads up [09:15] you need a VM for that [09:15] jamesh: or are they just unrelated failures to connect to services that this script isn't using? [09:15] bigjools: and that is how the soyuz code does it? [09:15] jtv: I'm going to drop to a debugger once the object count goes about 1.5 million [09:15] like the PPA VMs - they get totally reset after each PPA build [09:15] yep [09:16] bigjools: what is the setup and teardown time of the vms? [09:16] stub: cool [09:16] thumper: thankfully *very* quick, the IS guys did some great optimisation and it takes about 5 seconds IIRC [09:17] bigjools: do we have some to test with in staging? [09:17] no, just dogfood [09:17] we have one only [09:17] well that's going to be a problem [09:17] yeah [09:17] we have only one vm? [09:18] one builder [09:18] whether it's a VM or not is irrelevant to how we talk to it [09:18] a builder can run multiple vms? [09:18] other than resetting the machine [09:19] the one we have is virtual I think, but yeah we can run many builder VMs on a single box IIRC [09:19] thumper: do you want to jump on a call with me and danilos? [09:19] bigjools: both translations and code will need arbitrary jobs to run in protected environments [09:20] I know :) [09:20] I want to work out how we are going to make this happen [09:20] my problem is it is 22:20 and my brain is fuzzy [09:20] yeah [09:20] I have the same problem and it's only 0920 [09:21] so basically we already have an environment to do this - the problem is that it's built around soyuz [09:21] bigjools: what I want is a flag on the job class that says "run me in a protected env" and have the job runner just do it [09:21] yes [09:21] https://dev.launchpad.net/Soyuz/Specs/BuilddGeneralisation [09:21] * thumper ndos [09:21] and I have a plan --^ [09:22] but there are a number of stumbling blocks - but by the end of this week I will have a big list of smaller issues to tackle to get us to the point where we can get this working for non-soyuz jobs [09:22] bigjools: mwhudson will be looking at this over 3.1.11 [09:23] fabulous [09:23] bigjools: danilo has also offered some resources I think [09:23] so I think the best thing he can work on is to refactor IBuilder [09:23] bigjools: and I gather that foundations may also be able to provide someone [09:23] bigjools: if you can provide some break down of the work [09:23] bigjools: we'll take a look at it [09:23] I will [09:23] ta [09:23] in fact I am [09:24] fantastic even [09:27] thumper: could we have a call about this some time? i'd like to learn more about the current job system, and how you want to extend it. [09:27] BjornT: sure, but not right now :) [09:28] BjornT: abentley is going to be doing some work on it next cycle [09:28] * thumper signs off [09:28] thumper: yeah, i wasn't suggesting to do it now :) [09:28] * thumper smiles [09:28] BjornT: I suggested 2-3h after the TL call [09:30] bigjools: ok, that's probably fine for me [09:50] Hey stub, would you be able to go through and update: https://dev.launchpad.net/FoundationsTestPlan/3.1.10 ? There's quite a bit of QA listed there. [10:00] stub: any luck? [10:01] noodles775: I've moved one item. The other two of mine still need QA (one I'm not sure how to test, the other I don't think I have the resources to test without taking staging down for half a day) [10:01] jtv: So every leaked ObjectInfo I look at is similar to this: [10:01] (Pdb) random.choice([o for o in gc.get_objects() if isinstance(o, ObjectInfo)]) [10:01] {: {'remote': }, 'sequence': 1, : {'remote': }, : {'remote': }} [10:02] Thanks stub. [10:03] jtv: inf.get_obj() returns None, which seems to mean it really has been lost [10:04] stub: and yet it keeps that reference to the POTMsgSet [10:04] jamesh: were the "remote" entries meant to be cleaned up as the objects fell out of cache maybe? [10:04] Would the POTMsgSet be caching any back references I wonder? [10:05] stub: not that I've seen so far [10:05] jtv: they should get cleared out when the key for the reference gets cleared [10:05] they are meant to be strong reference [10:07] stub: and these POTMsgSets were not supposed to be in cache any more, right? [10:08] Pdb) pprint(inf) [10:08] {: {'remote': }, [10:08] : {'remote': }, [10:08] : {'remote': }, [10:08] 'sequence': 1} [10:09] It might well still be in the cache. Why are we interested in the POTMsgSet and not in whatever is holding a reference to this ObjectInfo? [10:10] Referrers are: [10:10]