=== Lo-lan-do [n=roland@mirexpress.internal.placard.fr.eu.org] has joined #bzr === Demitar [n=demitar@c-212-031-182-147.cust.broadway.se] has joined #bzr [12:14] igc: can you merge my bytes_to_gzip patch?, I'm definite its the right thing now even with memory copies - 3 seconds win with it [12:14] igc: even with my faster-sha patch [12:14] sure. I'll do that RSN === Bambi_BOFH [n=kgoetz@gnewsense/friend/kgoetz] has joined #bzr === mrevell [n=matthew@canonical/launchpad/mrevell] has joined #bzr [12:31] New bug: #141382 in bzr "Test failures when TMPDIR points at a symlink" [Low,New] https://launchpad.net/bugs/141382 [12:33] igc: thanks! [12:33] igc: I've decided path_content_summary is the go [12:34] Yes, I think it is [12:34] igc: it may be wrong eventually but for now its an improvement, IFF I add the sha lookup [12:34] so I'm going to add that today [12:34] 4% review just sent to the list btw [12:35] so lifeless, you have a lot of merges needed to pqm :-) [12:35] I can do most of them today if you wish [12:35] what order do you want them submitted if any? [12:36] make bytes would be a good start === RichardL [n=Skippy@78.32.35.169] has joined #bzr [01:05] jelmer: what bug is your bundle on ? [01:06] lifeless, which bundle? [01:06] the pqm one [01:06] I'm looking at bug 117197 [01:06] Launchpad bug 117197 in pqm "Doesn't install required Python package" [High,Triaged] https://launchpad.net/bugs/117197 [01:06] but theres no linked branch or bundle [01:07] lifeless: oh, I send the bundle to you. I'll attach it to the bug report, didn't know that was actually open [01:07] s/send/sent/ [01:07] you filed it :) [01:08] whoops, that's the second time that happens [01:09] ok, bundle is now attached === AfC [i=andrew@office.syd.operationaldynamics.com] has joined #bzr [01:27] igc just had a power failure === poolie [n=mbp@ppp112-44.static.internode.on.net] has joined #bzr === kkubasik [n=kkubasik@pool-71-178-249-90.washdc.fios.verizon.net] has joined #bzr [01:42] hey, I just found bug# 133989 [01:42] its where bzr cant branch from svn on windows [01:43] with a FleExitst error [01:43] has anyone heard anything as far as possible solutions etc? === orospakr [n=orospakr@CPE001c1019cfc4-CM0011ae034e04.cpe.net.cable.rogers.com] has joined #bzr [01:47] kkubasik, that's a bug in bzr's windows support [01:47] jelmer: are you sure? [01:47] jelmer: I thought so too, but only bzr-svn users seem to hit it [01:47] lifeless, yes, it was also reported independently against bzr [01:47] oh ok [01:47] ahhh [01:47] do we know the cause yet? is it a change in python ? [01:48] kkubasik: this is recent FWIW [01:48] https://bugs.edge.launchpad.net/bzr/+bug/139253 [01:48] Launchpad bug 139253 in bzr "bzr branch fails with "File exists" error." [Undecided,New] [01:48] bump that to critical please [01:48] and triaged [01:48] done [01:49] user 1m20.565s [01:49] commit is sneaking down still :) [01:49] lifeless: which tree? [01:50] moz [01:50] initial [01:50] nice [01:50] (I'm working on incremental, but have to run the initial each time as well to stop regressions) === jelmer has just started on shallow branches again [01:51] as everything else seems to be fast enough nowadays :-) [01:51] I remember when status took that long for one file in the moz tree :-P [01:51] \o/ [01:52] elmo: initial commit in bzr.dev is 3m20 user [01:53] elmo: and 6+minutes wallclock [01:53] what's the wall for you commit? [01:53] I get [01:53] real 1m26.859s [01:53] user 1m20.565s [01:53] sys 0m4.420s [01:53] nice [01:53] packs: ) [01:53] whenever linux is slow, write your own filesystem FTW [01:53] (and I was just being bitter - I have too many machines running bzr 0.8) [01:54] elmo, that's the dapper version? [01:55] poolie: yes [01:55] we have not done an SRU yet [01:55] good morning lifeless [01:56] hi poolie [01:56] igc has had a power failure [01:57] :) [01:57] Brisbane was famous for power failures when i was a kid [02:00] poolie: bzrlib/tests/inventory_implementations/basics.py [02:00] is named strangely, why isn't it test_basics.py ? [02:00] no good reason === igc [n=igc@ppp121-45-206-141.lns1.bne1.internode.on.net] has joined #bzr [02:07] I'm back [02:17] thanks jelmer that was very helpful [02:18] igc: my patch for rename correctness isn't merged [02:18] igc: if it was approved, could you merge that too ? [02:18] sure [02:18] just doing the 4% one now [02:22] vila: please keep news alphabetically sorted now [02:22] vila: as per the thread on the list [02:23] F comes after H [02:23] in NEWS, but shouldn't [02:23] wait, F doesn't come before H [02:23] oh nvm === jdong hangs head in shame [02:23] just... please... hit /clear and pretend that never happened :) === Bambi_BOFH is now known as kgoetz === Sigma [n=yann@pdpc/supporter/active/Sigma] has joined #bzr [02:27] poolie: pushing 2768 of repository branch [02:27] poolie: its up to date with bzr.dev some more with that [02:28] everyone's beeping at me === NamNguyen [n=namnt@203.162.163.50] has joined #bzr === jml [n=jml@ppp121-44-213-76.lns1.hba1.internode.on.net] has joined #bzr [03:22] lifeless: rename_one fix submitted to pqm now fyi [03:22] danke [03:30] elmo: btw why don't you upgrade bzr across all the systems [03:30] elmo: there are many bug fixes you should really have [03:32] hi I'm using bzr 0.18 with bzr-cvsps-import and it keeps throwing memory errors on cvs v files that are larger than 20MB. Is there any way at all around this? (aside from adding more ram...) [03:33] Vantage13: theres no existing knob that I'm aware of. Can you please file a bug on this; and if you know some python I'd be happy to help you work up a patch to fix this [03:33] lifeless: is this a bzr-cvsps-import bug or bzr itself? [03:34] Vantage13: I would say bzr-vcsps-import [03:49] lol [03:49] knits$ bzr vommit [03:49] bzr: ERROR: unknown command "vommit" [03:49] now we need to implement it :) [03:50] Spec: Bulimic Repacker: bzr vomit / bzr gag.... [03:58] spiv: ping [03:59] Hm. 'vomit' should be an alias for 'uncommit'... [03:59] lifeless: pong === poolie_ [n=mbp@ppp112-44.static.internode.on.net] has joined #bzr [04:10] spiv: hows teh fix? Can we chat about it? [04:10] Sure. [04:11] (you mean on phone or here?) [04:11] either [04:11] lifeless: review of 40% one done now [04:11] phone focuses the mind [04:13] Basically it's working, but the tests are a PITA. It's awkward to construct broken versionedfiles to run the tests on. I have an automated test for it, but the scaffolding is a bit ugly. [04:14] I think it's probably time I fired it off to the list for review despite that. [04:16] New bug: #141411 in bzr-cvsps-import "Import fails with memory error on files larger than 25MB" [Undecided,New] https://launchpad.net/bugs/141411 [04:16] spiv: how about a phone chat then, see if we can find a way to express the problem with test approach [04:17] lifeless: sounds good. === Admiral_Chicago [n=FreddyM@st074039212101.monm.edu] has joined #bzr === debboy [n=debboy@dsl-58-6-241-187.wa.westnet.com.au] has joined #bzr === debboy [n=debboy@dsl-58-6-241-187.wa.westnet.com.au] has left #bzr ["Leaving"] === jdong_ [n=root@ubuntu/member/jdong] has joined #bzr === jml_ [n=jml@dsl-210-15-197-192-static.TAS.netspace.net.au] has joined #bzr === jdong_ is now known as jdong [04:35] anyone know what alterations I'd have to make to remove a file or have bzr-cvsps-import skip a file? I tried removing the cvs v file, and removing it from the CVSROOT commitlog but it still seems to know the file is missing and error out... === jml_ is now known as jml === igc food [04:52] head_set = self._repo_graph.heads(parent_candiate_entries.keys()) [04:52] heads = [] [04:52] for inv in parent_invs: [04:52] if ie.file_id in inv: [04:52] old_rev = inv[ie.file_id] .revision [04:52] if old_rev in head_set: [04:52] heads.append(inv[ie.file_id] .revision) [04:52] head_set.remove(inv[ie.file_id] .revision) [04:55] correct_parent_versions = [parent_version for parent_version in all_parent_versions in parent_version in g.heads(all_parent_versions)] === kgoetz [n=kgoetz@gnewsense/friend/kgoetz] has joined #bzr [05:23] I haven't seen much in the way of bzr <-> p4. Is there any hope of something saving me from this predicament? :P [05:25] I've heard rumours, but not seen any code [05:26] Vantage13: hi [05:26] Vantage13: looking at the backtrace, I wonder if its a bug in our patiencediff. [05:27] Vantage13: you could breakpoint in line 608 of knit.py to get the texts that are being diffed [05:27] (_get_matching_blocks() does a diff) [05:27] Hmm, rumors are better than nothing, I suppose. tailor explodes in numerous different ways, and only deals with part of what I need. [05:28] breakpoint... .py...how do you guys debug? [05:28] breakpoints, logs, traceing, debug output [05:29] pdb is a very good tool [05:29] Hmm, wow. I've never used pdb. [05:29] put [05:29] import pdb;pdb.set_trace() [05:29] in a .py file somewhere, you'll get the hang of it easily [05:32] lifeless: I know the file that's causing the problem. It's a large data import text file. The diff between the two versions is pretty huge [05:33] Vantage13: right, I'm wondering if the memory problem is not (its a 25Mb file), but (these two texts cannot diff properly with bzrlib) [05:38] lifeless: how do you insert a breakpoint there? [05:38] import pdb;pdb.set_trace() === Admiral_Chicago [n=FreddyM@ubuntu/member/admiral-chicago] has joined #bzr === jamesh_ is now known as jamesh === jml_ [n=jml@ppp121-44-213-76.lns1.hba1.internode.on.net] has joined #bzr === Peng [n=mnordhof@fl-69-69-140-112.dyn.embarqhsd.net] has joined #bzr === jml_ is now known as jml [06:00] ok, I can pull bzr.dev now, with the fix to not propogate parent corruption [06:01] good news. I'm just merging your 40% one now btw [06:01] thanks! [06:01] but there are plenty others of yours in BB still to merge :-) [06:09] lifeless: around? [06:10] Vantage13: yes [06:11] lifeless: so I've set_trace right before the loop and it's stopped, however, i'm not sure what I should be doing at this point or what I should be looking for [06:11] well [06:11] it probably will pass some number of times [06:11] so 'c' [06:12] many times until it crashes, then you can add a condition to only break into the debugger on that case [06:12] lifeless: i've actually got it on the one that crashes (since the previous calls were correctly processed the first time) [06:12] ok cool [06:12] you should have two texts there [06:13] I have [06:13] /usr/lib/python2.4/site-packages/bzrlib/knit.py(609)_merge_annotations()-> for i, j, n in seq.get_matching_blocks(): [06:13] and the prompt [06:16] ok, one sec [06:16] k [06:16] print annotated [06:16] print parents [06:16] print len(merge_content.text()) [06:16] both are True [06:17] print len (content.text()) [06:17] 768676 [06:17] hmm, parents should be a list, not a boolean [06:17] 770413 [06:17] wow, thats quite long :) [06:17] have a look via top - how much memory is the process using at the moment [06:17] lifeless: yeah, as I said :) [06:18] 17.8% [06:18] however we do diff entire iso's using this code [06:18] 17% of what ? [06:18] 370m Virt [06:18] sorry, I just grabbed the %MEM column [06:18] 360m RES [06:18] ok, large but not insane [06:18] (given the size of thing you are dealing with) [06:19] lets save these two texts [06:20] lifeless: save how? [06:20] bzrlib.transport.get_transport('file:///tmp/').put_bytes('a', ''.join(merge_content.text())) [06:20] bzrlib.transport.get_transport('file:///tmp/').put_bytes('b', ''.join(content.text())) [06:20] that should create two files on disk - check they exist and look ok to you [06:20] yup [06:21] from a shell [06:22] do python -m bzrlib.patiencediff /tmp/a /tmp/b [06:22] theory is that this will blow up [06:24] odd. It's saying module bzrlib.patiencediff not found, even though it is... [06:24] ok [06:24] just do /path/to/bzrlib/patiencediff.py /tmp/a /tmp/b [06:26] running... [06:27] have a look in top, is it growing ? [06:28] at the moment, no. It seems fixed at 170m [06:29] how big is each file ? [06:29] 18M each [06:30] so 36M from 170 134 [06:30] 370M + 134 = 504M === lifeless [n=robertc@ppp245-86.static.internode.on.net] has joined #bzr === n2diy [n=darryl@wlk-barre-208-103-148-20.dynamic-dialup.coretel.net] has joined #bzr === synic [n=squish@pdpc/supporter/student/synic] has joined #bzr === keir [n=keir@206-248-159-109.dsl.teksavvy.com] has joined #bzr === james_w [i=jw2328@jameswestby.net] has joined #bzr === jeremyb [n=jeremy@unaffiliated/jeremyb] has joined #bzr === alla [n=alla@soy.cyber.com.au] has joined #bzr === meuh [n=meuh@pdpc/supporter/active/meuh] has joined #bzr === LarstiQ [n=larstiq@cust.7.157.adsl.cistron.nl] has joined #bzr [06:34] lifeless: this was the last message I got from you "370M + 134 = 504M". was that the last one you sent? === dous_ [n=dous@124.104.8.104] has joined #bzr [06:34] 14:30 < lifeless> which is big, how much ram do you have ? === duckx [n=Duck@tox.dyndns.org] has joined #bzr === ThomasAH [n=thomas@aktaia.intevation.org] has joined #bzr [06:35] lifeless: the machine has 2GB [06:43] ok [06:43] did it finish the diff ok ? [06:43] yup === orospakr [n=orospakr@bas4-ottawa23-1096745062.dsl.bell.ca] has joined #bzr === Admiral_Chicago [n=FreddyM@ubuntu/member/admiral-chicago] has joined #bzr [06:50] strange [06:50] so if you step through the loop [06:50] where does it crash === herzel88 [i=herzel@gateway/tor/x-9df12031e7d90e5f] has joined #bzr [06:53] len(self.a), len(self.b), matches, 10) [06:53] MemoryError: > /usr/lib/python2.4/site-packages/bzrlib/patiencediff.py(244)get_matching_blocks() [06:53] hmm [06:53] and its on that pass through [06:53] ? [06:54] then continuing through I also get [06:54] len(self.a), len(self.b), matches, 10) [06:54] have you run out of memory now ? [06:54] MemoryError: > /usr/lib/python2.4/site-packages/bzrlib/knit.py(609)_merge_annotations() [06:54] yea the exception is being raised frame by frame [06:54] what does top show now ? [06:55] 397m [06:55] still not enough to get OOM [06:55] I have to pop away for a bit; [06:56] basically we need to figure out what is causing the error [06:56] I'd try stepping (s) into the function that fails now [06:56] try to get to the point that the next 's' will cause a failure [06:57] then investigate the variable etc that are involved and see if any are obscenely large or anything like that [07:04] lifeless: this seems to be the loop [07:05] 48 -> for i in xrange(len(a)): 49 line = a[i] 50 if line in index: 51 index[line] = None 52 else: 53 index[line] = i [07:05] in patiencediff.py === ssokolow [n=ssokolow@bas1-barrie18-1242373239.dsl.bell.ca] has joined #bzr [07:07] Does anyone know of a Bazaar cheat sheet similar to the Mercurial one and the edit of the Mercurial one to describe Git made by Zack Rusin? [07:08] ssokolow, http://doc.bazaar-vcs.org/bzr.dev/en/quick-reference/quick-start-summary.svg [07:08] poolie_: Thanks. [07:09] you're welcome [07:10] Vantage13: so this should shrink memory use [07:10] I think [07:15] lifeless: ? [07:15] Vantage13: can you identify the exact point it fails ? [07:16] lifeless: without stepping through every single entry in file? Probably not... [07:16] yah [07:17] uhm [07:17] add a print i [07:17] in there [07:17] run to failure, [07:17] then you can add a if i == failure_value: import pdb;pdb.set_trace() === Admiral_Chicago_ [n=FreddyM@st074039212101.monm.edu] has joined #bzr [07:21] lifeless: that 40% one now merged into bzr.dev. Can you please check you're happy with how I merged & tweaked it? [07:22] am doing [07:22] thanks [07:24] lifeless: ok, i'm there [07:24] so in theory, 's' will cause a crash [07:25] does it ? [07:25] index[line] = i [07:25] throws the memory error [07:26] i=699050 [07:27] line = V3S 6V2,BC,Surrey,1 [07:27] is there a limit to the size of the data structure index? [07:28] ok [07:28] lets get back there [07:28] and we won't run that line :) [07:28] if your memory use is reasonable at this point I'm going to consider it a CPython bug probably [07:28] >>> print type(index) [07:28] 'dict' [07:30] how do you print the total number of entries in the dictionary? [07:31] len(index) [07:31] print repr(line) [07:31] lets be sure there's no funky business [07:32] 'V3S 6V2,BC,Surrey,1\r\n' [07:32] so I think you have found a bug in python. [07:32] whats the process size ? [07:32] 397m VIRT, 386m RES [07:32] yah [07:33] jml: what do you think? 'index[line] = i' throws a MemoryError [07:33] lifeless: "weird". [07:34] I recall spiv tracking down a bug in dict [07:34] Vantage13: print len(index) [07:34] this may be it [07:34] lifeless: did I? [07:34] lifeless: yeah, that rings bells. [07:34] 699051 [07:34] I don't remember it :) [07:34] spiv: in launchpad, some time back [07:34] or perhaps it was jamesh, but I thought it was you [07:34] hmm [07:35] Hmm, I do remember spending a lot of time digging through the dictobject.c code at some point. [07:35] I forget why though. [07:36] rotfl [07:36] Something involving the difference between how it handled string keys (which it special cases) and other things. [07:36] lifeless, spivs exomemory [07:36] spiv: oh, perhaps it was security proxies as keys [07:36] or something like tha [07:36] so its bizarre to get an Out of memory error at this point [07:37] lifeless: oh, I do remember that bug sort of now. [07:37] there is 400Mb in the process [07:37] lifeless: it wasn't a bug in dict, it just looked like it. [07:37] this line is putting line into into [07:37] There was a __del__ somewhere deleting things from a dict when I didn't realise it. [07:37] Anyway, this is a large dict. [07:37] Vantage13: can you do 'print line in index' [07:37] And inserting in a dict occasionally will trigger a resize. [07:38] spiv: power of two expansion ? [07:38] lifeless: True [07:38] Which I believe happens by allocating a whole new table, moving stuff into the new table, then removing the old table. [07:38] spiv: ^ line is already in there [07:38] Yeah, I think it's power of two. [07:38] spiv: worst case it would allocate 800M (twice current memory), for 1.2Gb, still well within comfort - this machine has 2GB [07:38] Vantage13: do you have ulimits in place ? [07:38] It has a loop doing "newsize <<= 1". [07:39] lifeless: nope. unlimited [07:40] wait [07:40] virtual memory seems to have one... [07:40] suspiciously set to 409600 [07:40] ahha! [07:40] there we go, memo to self, check ulimits earlier [07:41] lifeless: I don't see why you say the line is already in there; Vantage13 says "index[line] = i" is triggering the memory error after all. [07:41] 15:37 < lifeless> Vantage13: can you do 'print line in index' [07:41] 15:38 < Vantage13> lifeless: True [07:41] Ah, ulimits. [07:41] I've quoted the two lines that make me say the line is already in there [07:42] lifeless: ah, I missed that, thanks. [07:43] Vantage13: so, remove the debug statements and try again. I wager it will work [07:43] any idea off hand what would be setting that ulimit? I assume it's a boot or login script... [07:43] uhm /etc/defaults/ may have something [07:43] theres also bash .d stuff in /etc/somethingorother === poolie_ [n=mbp@ppp112-44.static.internode.on.net] has joined #bzr === dous_ is now known as dous [08:03] lifeless: are you good to merge now or shall I keep helping? If so, I'll merge "micro-tweaks to sha routines" next assuming you still want it in [08:03] lifeless: we seem to have moved past that file, so i'm thinking that was it [08:04] Vantage13: :) [08:04] lifeless: Thanks for all the help! [08:04] igc: please merge it, I'm good but help is help. [08:04] no problem [08:05] lifeless, is that a 'tweak' vote for the traceback change? [08:05] ie can i merge it with that change? [08:05] poolie_: sure [08:05] if my arguments are cogent [08:05] and you are convinced [08:06] sure, i just was being lazy [08:06] :) [08:06] succesfully [08:08] ulimits are set in /etc/security/limits.conf as well === pmezard [n=pmezard@nor75-4-81-56-59-92.fbx.proxad.net] has joined #bzr === hdima [n=hdima@idealer.cust.smartspb.net] has joined #bzr === metze_away is now known as metze [08:27] igc: a review of my knit random id patch would be cool too [08:27] I'm doing it now [08:27] :) [08:27] I'm just benching all my combined changes finally [08:28] I had *too many* branches to deal for a bit there [08:28] :-) [08:28] Hence my keenness today to get bzr.dev up to date [08:28] its still not quite *all* branches [08:28] but much closer [08:29] initial commit figures [08:29] real 1m28.816s [08:29] so lifeless, I'm confused re that patch ... [08:29] user 1m20.373s [08:29] sys 0m4.572s [08:29] which is good [08:29] no regression [08:30] mmmhmm ? [08:30] the changes to _add() don't make any sense [08:30] *blink* [08:30] add_version gaining random_id I get [08:30] really? [08:31] but nothing in that patch calls _add [08:31] is it already called by existing code? [08:31] incremental with 5 new files, [08:31] real 0m56.226s [08:31] user 0m49.507s [08:31] sys 0m3.744s [08:31] and incremental with specified file [08:31] real 0m23.494s [08:31] user 0m22.117s [08:31] sys 0m0.872s [08:31] big difference there [08:32] @@ -828,7 +828,7 @@ [08:32] self._check_add(version_id, lines, random_id, check_content) [08:32] self._check_versions_present(parents) [08:32] return self._add(version_id, lines[:] , parents, self.delta, [08:32] - parent_texts, left_matching_blocks, nostore_sha) [08:32] + parent_texts, left_matching_blocks, nostore_sha, random_id) [08:32] that hunk calls add [08:32] but I missed a hunk in my edits [08:32] I'll resubmit in a second [08:32] I was about to say "sure - but what calls it?" [08:33] :-) [08:33] _add is called there [08:33] or do you mean add_versions ? [08:33] cool [08:33] now, add_versions makes sense [08:33] s/now/no/ [08:35] just supersede that path [08:35] patch [08:37] sent [08:38] thanks [08:39] bbiab === dous [n=dous@124.104.0.255] has joined #bzr [08:47] that's all good now lifeless [08:48] lifeless, would suggest renaming random_id to something that doesn't look like it's a random id [08:49] like id_is_random === RichardL is now known as rml [08:52] poolie_: good suggestion; there is a idiom in play so we should do it for more than this one patch [08:52]