=== yofel_ is now known as yofel [00:22] right, I should really go to bed. [00:22] night =) [00:22] i'll be on [00:23] I'll stay logged in here, say how far you get [00:24] kk! [02:19] wgz, I finally got the sdk installed and created the vcvarsamd64.bat file, but its still giving me ValueError: [path] (which i guess means it failed to call vcvars) [02:19] i guess i'll talk to you about it when you are up =P === AuroraBorealis is now known as aurora|away === aurora|away is now known as AuroraBorealis [06:21] i seriously hate compiling things haha [10:12] AuroraBorealis: I think you need to read "Trick distutils" onwards in [10:13] i did that [10:13] still doesn't work [10:13] i might of put that line in msvc9compiler in the wrong place but i dont think i did [10:14] okay, I'll take another look at your traceback and see if I can work out what's up [10:15] http://bugs.python.org/issue7511 [10:15] "You can work around it [10:15] by finding and executing vcvars32.bat or vcvars64.bat before running [10:15] setup.py [10:15] " [10:15] yep thats the exact error i'm getting [10:16] http://paste.ubuntu.com/700959/ [10:18] did you get the 64-bit components installed correctly as well? [10:18] has some step-by-step it might be worth reviewing [10:22] it says to check an option that was checked by default [10:22] but i don't have 'cl.exe' [10:22] cause the folder wasn't created there [10:22] (that was the folder i created in the 'trick distutils' blog post [10:23] checked by default is fine, is you have the `bin\amd64\cl.exe` fine somewhere [10:23] well the folder they say that cl.exe is in wasn't created by default [10:23] i had to create the directory and the vcvarsamd64.bat file [10:23] sure, but can you find it somewhere else? [10:24] i see cl.exe in Microsoft Visual Studio 9.0/VC/bin [10:24] if so, I'd go ahead and open a new cmd window, call vcvarsamd64.bat, then run setup.py and see what happens [10:25] oh gee [10:25] i think its screwing up on the uf-8 bom [10:26] well [10:27] i ran it, the terminal text turned green [10:27] and now its compiling [10:27] heh, that's possible [10:27] it said call is not a windows batch file/command [10:27] yeah, that's an annoying way to have got stuck :) [10:28] so when i install this [10:28] will it always use this version of bzr lib? or will running the exe from the windows installer override it [10:29] it won't interfere with the exe version [10:29] k [10:29] Well. now that installed [10:29] i forget what else i need to do [10:29] lp:meliae lp:python-fastimport lp:bzr-fastimport [10:30] I recommend testing the bzr you've just installed, should be able to run it with Python27\Scripts\bzr.bat [10:32] seems to of worked! [10:32] good good! [10:36] ok [10:36] all 3 installed [10:38] now what! [10:39] well, it would be nice to test that the memory dumping worked but... [10:39] could just start the import [10:40] start it again from the beginning? [10:40] `Scripts\bzr fast-import ..\tmp_linux\linux.fe \bzr-import` or similar [10:40] hm. [10:40] cause i can't seem to continue cause of another bug [10:40] yeah [10:41] you could try repacking the failed import for a faster big-memory usage [10:41] but with 64 bit process you probably won't OOM now anyway [10:41] is there a command to repack? [10:44] `bzr pack` [10:44] ...it took me way too long to find that, I fail at name guessing/remembering [10:45] you can look the windows task manager for ball park memory usage numbers [10:47] yeah i'm watching it [10:47] 280,000 KB [10:50] wgz: have you installed bzr guess ? [10:50] the -Dmem_dump option is needed to write the stats on OOM, but that probably won't happen now so breakin and calling bzrlib.trace._dump_memory_usage directly is probably more useful anyway [10:50] lifeless: nope [10:50] wgz: give it a spin :) [10:51] $ bzr repack [10:51] Command 'repack' not found, perhaps you meant 'pack'? [y/n]: [10:51] will it dump the memory automatically using meliae or do i have to do the -Dmem_dump thing [10:51] ha, and I was just about to doubt that it'd know to remove the 're' prefix [10:51] I should have more trust in lifeless. [10:51] wgz: it uses a distance calculation :) [10:52] wgz: $ bzr pdck [10:52] Command 'pdck' not found, perhaps you meant 'pack'? [y/n]: [10:52] ^It will only auto dump if you OOM, which with a 64-bit process and 8 gigs probably isn't going to happen [10:52] (and needs the flag) [10:52] k [10:52] but with ctrl+break then a bit of typing in the debugger we can get what we want anyway [10:52] so this seems like its going to work, although its using 600,000KB atm hehe [10:53] 64 bit is a lost faster o.o [10:53] lot* [10:56] scusi, just going to pick apples, but I'll be around - try the import from the beginning if that works [11:13] kk =) [11:19] okay. how high did the repack get up to on its own? [11:37] well [11:37] its currently repacking "chk:chk node 861000/987188" [11:39] that's good, and what's the mem usage? [11:39] right now 650,000 KB [11:39] highest was around 850,000 KB [11:39] ha, so it has stabilised. [11:40] it jumped up a lot when it started doing the chk: index/nodes and whatever [11:40] so, if you want to try something [11:40] do ctrl+break in the terminal [11:41] the pause|break button? [11:41] yup. [11:41] this should then put you in the python debugger [11:41] then you can do... [11:41] pdb> from bzrlib.trace import _dump_memory_usage [11:42] pdb> import sys [11:42] pdb> _dump_memory_usage(sys.stderr) [11:42] and finally to resume, [11:42] pdb> c [11:43] well that crashed it [11:43] paste me the output? [11:43] didn't give any output [11:43] just said "python27_64.exe has stopped working" [11:43] ah, msgbox from windows? [11:43] yeah [11:44] at which point? [11:44] there are the windows dump files [11:44] _dump_memory_usage(sys.stderr) [11:44] okay... meliae apparently has issues on win64 then >_< [11:45] gahhhhh [11:46] if you could file a bug against and stick the stuff windows gave you as an attachment, that would be great [11:46] the good news is that using 64-bit process looks like it'll let the fast-import succeed [11:46] yeah [11:46] but running out of memory on 32 bit still seems like a problem [11:48] yup, and the usage of both fast-import and repacking seem to both be problematic [11:50] yes [11:50] i restarted it again, using 600,000 at stage 2 so yeah. [11:55] so file a bug against meliae? [11:56] right. [11:56] you could also run it's testsuite to see if that crashes (and can then be narrowed down to one test) [11:57] can do that with... `cd meliae` `Python27\python setup.py build_ext -i` `Python27\python -m unittest meliae.tests.test_suite` [11:57] just use a new window, no need to interrupt the repack [11:59] ...I get two failures, but they look shallow [12:00] haha [12:00] the unit tests crash. [12:00] that's good. [12:00] what debugs the debugger? [12:00] WATCHMENNNNNNN [12:00] you can now try smaller subsets till you find one test that does it [12:01] how do i run the individual tests? replace the meliae.tests.test.suite with the individual files? [12:02] start with meliae.tests.test_scanner [12:02] crash :< [12:02] and then you can narrow further with the classes in that file, then the methods [12:02] i found the names i'll run em all [12:04] is it because i'm not running cmd.exe in administrator mode? [12:04] it's likely one (or more) type errors where the extension code is assuming sizeof(int) == sizeof(*ptr) [12:05] test__intset works, test__loader works, test__scanner crashes, test_loader crashes, test_perf_counter works, test_scanner crashes [12:05] which is true for win32 and the 64-bit scheme used on nix [12:07] ah :3 [12:13] anyway, you've got enough to file a bug now [12:13] yeah [12:13] there should be a way to get debugging symbols for a proper C stack trace, but I've not done it with cython before [12:13] doing that now [12:19] i have no idea either xD [12:19] https://bugs.launchpad.net/meliae/+bug/864593 [12:19] Ubuntu bug 864593 in Meliae "Crashes on win64 when running test suite" [Undecided,New] [12:23] another note, bzr is using over 1,200,000 KB when repacking now o.o [12:28] hm. [12:28] i'll just let it run. never debugged python stuff this indepth before .. [12:30] can you run another smaller chunk of the meliae test suite? start with meliae.tests.test__scanner.TestSizeOf [12:30] if that still crashes, add .test_None [12:31] that didn't crash, 3 tests failed [12:31] okay, that's what I was hoping for [12:31] failing tests on the sizeof things implies logic errors, that could cause crashes in bigger chunks of code [12:31] stick those three failed tests output in a comment on the bug [12:32] it seems that they are always off by 4 [12:32] yeay! alignment. [12:32] 37 != 33, 102437 != 102433, 38!=34 [12:32] paste them in the bug report and I'll have a look [12:33] posted! [12:39] hm, just strings, probably just the 2.7 check above not taking alignment into account rather than something more fundamental. [12:40] aww [12:41] try each of the other classes in that file in turn [12:43] TestDumpInfo crashes python [12:44] all others pass [12:44] okay, that's good, gives us a narrowed scope. [12:45] not much, but a bit. [12:46] this pyrex ` ctypedef long size_t [12:46] is wrong, but probably unimportant, I'm never sure what the manual definitions actually matter for [12:47] can i try an individual method in a test case? [12:47] that's in _scanner.pyx and there are some similar ones in the other pyx files, could try deleting them and recompiling, I'll keep looking [12:48] ^yup, just .method_name on the end [12:48] in the TestDumpInfo case they all call the same helper which goes to dump_object_info so should all crash [12:49] yeah they all crash [12:52] ...this could be something as silly as the IO failing [12:52] oh? :o [12:53] have you tried deleting those size_t defs yet? [12:53] in __scanner.py and then recompiling? [12:54] yup, then run one of the TestDumpInfo cases again [12:54] what do i delete? these? size_of = _scanner.size_of [12:54] oh [12:54] nvm haha [12:54] nope, just the ctypedef at the top. [12:55] long is 4 bytes on win64, and size_t is 8 bytes [12:55] I think cython should ignore that decl and use the right thing, but you never know [12:56] sorry i suck at c, what do i delete? lots of size_t things [12:59] just the one line at the top of _scanner.pyx "ctypedef long size_t" [12:59] oohhh the pyx file [13:00] hm, yeah, that shouldn't matter I think. [13:01] yeah, still crashes [13:03] okay, let's eliminate the possibility of it being the io [13:10] okay, I've added a test to TestDumpInfo that does a bit less [13:10] http://paste.ubuntu.com/701020/ [13:10] then run with meliae.tests.test__scanner.TestDumpInfo.test_int_no_file and it passes here [13:11] I'm interested if that simplified version still crashes for you. [13:12] ugh, tabs, [13:13] i dont need to recompile right [13:15] and that test passed [13:15] eh he he. [13:15] it printed my two little notes as well, right? [13:15] yeah [13:15] now change recurse_depth=0 from 0 to 1 and try again. [13:16] printed a period too [13:16] that works as well [13:18] okay, change that back, and change `sio.write` to `file("nul", "wb")` [13:18] which, if the IO is indeed the problem, will crash. [13:19] yeeeup. crashes. [13:20] so, either meliae and python are linked against different runtime libraries, or there's a bug [13:21] * AuroraBorealis whimpers at the fact of runtime libraries not working :<< [13:30] okay, so, nothing's jumping out at me, and I need some lunch [13:30] haha [13:31] take your time, its 6 am here so i probably should go to bed sometime. [13:31] but there are two things I can think of that would help (bar getting a stacktrace... which may have actually been just as easy) [13:32] one is to use dumpbin.exe to look at python27.dll and _scanner.pyd to see if they're importing from the same runtime [13:33] the other is to try commenting/printing in _scanner.pyx to narrow down exactly what crashes, using our minimal test [13:33] what is dumpbin.exe? [13:33] it just lists symbols in the exe/library, either msvc or the win7 thing should have come with it [13:34] on the trial and error front, for starts [13:35] in dump_object_info comment the _dump_object_info calls so just the fflush happens, and see if there's still a crash [13:36] remember to rerun `setup.py build_ext -i` each time after changing a pyx file [13:37] if that works, can just commenting the fwrite call in _file_io_callback [13:37] which test do i run, yours? [13:39] running your test again and commenting out the _dump_object_info it still crashes [13:47] heh, that does seem just like mismatched FILE* then [13:48] how did the install get that wrong... [13:56] so, possible workaround for now [13:56] [13:59] doesn't crash, i get this: http://paste.ubuntu.com/701045/ [14:05] cool, now run the rest of those tests [14:06] (that one fails because we were just discarding the output as what we cared about was if the IO crashed) [14:07] i get all the failures i was getting before, yours and then TestMemObj and TestObjManager failing [14:08] http://paste.ubuntu.com/701050/ [14:09] okay, so reverting everything but the workaround should mean you have a functional meliae to install and use on bzr [14:10] can do `bzr shelve -destroy` to get rid of the other changes [14:10] *--destroy [14:12] i was just downloading the tarballs cause launchpad is still not accepting my ssh key for some reason =/ [14:13] ha, well, I think we didn't change anything else important so go ahead and install [14:13] yeah [14:13] just restored the tarballs copy and redid the change [14:15] you should probably post that workaround to the bug report as i have no idea what that did xD [14:16] I'll do it [14:17] okay, so as a test, start a repack, then do the ctrl+break and my instructions from earlier before we got side-tracked, you should get a .json dump of where the memory usage is [14:18] ok [14:37] so, where is it dumping the json [14:39] it should tell you, but in %TMP% basically [14:39] k. still dumping xD [14:40] whereever the tempfile module creates things [14:41] well [14:41] the json file is 4 gb [14:41] and cannot be opened by my text editor [14:41] that's not overly suprising. [14:42] do you think paste.ubuntu.com would be mad if i pasted it there? =P [14:43] ehehe [14:44] the strip_duplicates.py script may reduce it somewhat, and 7z or bzip2 should compress it quite well [14:44] yeah [14:44] trying with 7z [14:45] oh yeah. compression on text files is sexy [14:46] gives some ideas of what you can do with the dump [14:46] the summary should be pastebinable at least :) [14:51] yeah [14:53] loading~ [14:59] omgggggg [14:59] it failed x.x [15:00] 'address not present" [15:00] * AuroraBorealis tries the dump again [15:19] any luck second time? [15:20] stillll dumping [15:27] keeps saying an address is not present =/ [15:28] how frustrating. [15:28] yeah D: [15:29] no idea whats going on now... [15:30] try with a much smaller bzr process first, so at least you can then send me the dump and it won't take as long to retry? [15:30] k [15:31] eg, `bzr info` and interrupt if (if you're fast enough) [15:34] there are still some casts in the meliae source I don't like but the problem might be elsewhere [15:34] www.kramidnarg.com/stuff/bzr_memdumptn8bb_.7z [15:34] that still gives the same error if i try to laod it [15:37] okay, I get the KeyError too. [15:40] k [15:41] i'm probably going to go to bed now [15:41] when i get up i guess we can figure out why its not loading the dumps :< [15:41] looks like there are some refs that aren't in the dump, probably doesn't need to be a fatal error [15:42] sleep well! [15:42] don't dream of running out of memory. [15:42] =P === AuroraBorealis is now known as aurora|zzz === marienz_ is now known as marienz [21:20] I have a quick question about something really trivial that is disproportionately hard to figure out how to do in Bazaar. [21:20] I have a directory containing a program, and I want to import it into Bazaar. However, I don't want to make said directory a branch (easy), I want the branch to be in a shared repository (already created) and the original directory become a working tree for that branch. How might this be done?