| === yofel_ is now known as yofel | ||
| wgz | right, I should really go to bed. | 00:22 |
|---|---|---|
| AuroraBorealis | night =) | 00:22 |
| AuroraBorealis | i'll be on | 00:22 |
| wgz | I'll stay logged in here, say how far you get | 00:23 |
| AuroraBorealis | kk! | 00:24 |
| AuroraBorealis | wgz, I finally got the sdk installed and created the vcvarsamd64.bat file, but its still giving me ValueError: [path] (which i guess means it failed to call vcvars) | 02:19 |
| AuroraBorealis | i guess i'll talk to you about it when you are up =P | 02:19 |
| === AuroraBorealis is now known as aurora|away | ||
| === aurora|away is now known as AuroraBorealis | ||
| AuroraBorealis | i seriously hate compiling things haha | 06:21 |
| wgz | AuroraBorealis: I think you need to read "Trick distutils" onwards in <http://mattptr.net/2010/07/28/building-python-extensions-in-a-modern-windows-environment/> | 10:12 |
| AuroraBorealis | i did that | 10:13 |
| AuroraBorealis | still doesn't work | 10:13 |
| AuroraBorealis | i might of put that line in msvc9compiler in the wrong place but i dont think i did | 10:13 |
| wgz | okay, I'll take another look at your traceback and see if I can work out what's up | 10:14 |
| wgz | http://bugs.python.org/issue7511 | 10:15 |
| wgz | "You can work around it | 10:15 |
| wgz | by finding and executing vcvars32.bat or vcvars64.bat before running | 10:15 |
| wgz | setup.py | 10:15 |
| wgz | " | 10:15 |
| AuroraBorealis | yep thats the exact error i'm getting | 10:15 |
| AuroraBorealis | http://paste.ubuntu.com/700959/ | 10:16 |
| wgz | did you get the 64-bit components installed correctly as well? | 10:18 |
| wgz | <http://www.mathworks.com/support/solutions/en/data/1-6IJJ3L/index.html?solution=1-6IJJ3L> has some step-by-step it might be worth reviewing | 10:18 |
| AuroraBorealis | it says to check an option that was checked by default | 10:22 |
| AuroraBorealis | but i don't have 'cl.exe' | 10:22 |
| AuroraBorealis | cause the folder wasn't created there | 10:22 |
| AuroraBorealis | (that was the folder i created in the 'trick distutils' blog post | 10:22 |
| wgz | checked by default is fine, is you have the `bin\amd64\cl.exe` fine somewhere | 10:23 |
| AuroraBorealis | well the folder they say that cl.exe is in wasn't created by default | 10:23 |
| AuroraBorealis | i had to create the directory and the vcvarsamd64.bat file | 10:23 |
| wgz | sure, but can you find it somewhere else? | 10:23 |
| AuroraBorealis | i see cl.exe in Microsoft Visual Studio 9.0/VC/bin | 10:24 |
| wgz | if so, I'd go ahead and open a new cmd window, call vcvarsamd64.bat, then run setup.py and see what happens | 10:24 |
| AuroraBorealis | oh gee | 10:25 |
| AuroraBorealis | i think its screwing up on the uf-8 bom | 10:25 |
| AuroraBorealis | well | 10:26 |
| AuroraBorealis | i ran it, the terminal text turned green | 10:27 |
| AuroraBorealis | and now its compiling | 10:27 |
| wgz | heh, that's possible | 10:27 |
| AuroraBorealis | it said <randomgibberish>call is not a windows batch file/command | 10:27 |
| wgz | yeah, that's an annoying way to have got stuck :) | 10:27 |
| AuroraBorealis | so when i install this | 10:28 |
| AuroraBorealis | will it always use this version of bzr lib? or will running the exe from the windows installer override it | 10:28 |
| wgz | it won't interfere with the exe version | 10:29 |
| AuroraBorealis | k | 10:29 |
| AuroraBorealis | Well. now that installed | 10:29 |
| AuroraBorealis | i forget what else i need to do | 10:29 |
| wgz | lp:meliae lp:python-fastimport lp:bzr-fastimport | 10:29 |
| wgz | I recommend testing the bzr you've just installed, should be able to run it with Python27\Scripts\bzr.bat | 10:30 |
| AuroraBorealis | seems to of worked! | 10:32 |
| wgz | good good! | 10:32 |
| AuroraBorealis | ok | 10:36 |
| AuroraBorealis | all 3 installed | 10:36 |
| AuroraBorealis | now what! | 10:38 |
| wgz | well, it would be nice to test that the memory dumping worked but... | 10:39 |
| wgz | could just start the import | 10:39 |
| AuroraBorealis | start it again from the beginning? | 10:40 |
| wgz | `Scripts\bzr fast-import ..\tmp_linux\linux.fe \bzr-import` or similar | 10:40 |
| wgz | hm. | 10:40 |
| AuroraBorealis | cause i can't seem to continue cause of another bug | 10:40 |
| wgz | yeah | 10:40 |
| wgz | you could try repacking the failed import for a faster big-memory usage | 10:41 |
| wgz | but with 64 bit process you probably won't OOM now anyway | 10:41 |
| AuroraBorealis | is there a command to repack? | 10:41 |
| wgz | `bzr pack` | 10:44 |
| wgz | ...it took me way too long to find that, I fail at name guessing/remembering | 10:44 |
| wgz | you can look the windows task manager for ball park memory usage numbers | 10:45 |
| AuroraBorealis | yeah i'm watching it | 10:47 |
| AuroraBorealis | 280,000 KB | 10:47 |
| lifeless | wgz: have you installed bzr guess ? | 10:50 |
| wgz | the -Dmem_dump option is needed to write the stats on OOM, but that probably won't happen now so breakin and calling bzrlib.trace._dump_memory_usage directly is probably more useful anyway | 10:50 |
| wgz | lifeless: nope | 10:50 |
| lifeless | wgz: give it a spin :) | 10:50 |
| lifeless | $ bzr repack | 10:51 |
| lifeless | Command 'repack' not found, perhaps you meant 'pack'? [y/n]: | 10:51 |
| AuroraBorealis | will it dump the memory automatically using meliae or do i have to do the -Dmem_dump thing | 10:51 |
| wgz | ha, and I was just about to doubt that it'd know to remove the 're' prefix | 10:51 |
| wgz | I should have more trust in lifeless. | 10:51 |
| lifeless | wgz: it uses a distance calculation :) | 10:51 |
| lifeless | wgz: $ bzr pdck | 10:52 |
| lifeless | Command 'pdck' not found, perhaps you meant 'pack'? [y/n]: | 10:52 |
| wgz | ^It will only auto dump if you OOM, which with a 64-bit process and 8 gigs probably isn't going to happen | 10:52 |
| wgz | (and needs the flag) | 10:52 |
| AuroraBorealis | k | 10:52 |
| wgz | but with ctrl+break then a bit of typing in the debugger we can get what we want anyway | 10:52 |
| AuroraBorealis | so this seems like its going to work, although its using 600,000KB atm hehe | 10:52 |
| AuroraBorealis | 64 bit is a lost faster o.o | 10:53 |
| AuroraBorealis | lot* | 10:53 |
| wgz | scusi, just going to pick apples, but I'll be around - try the import from the beginning if that works | 10:56 |
| AuroraBorealis | kk =) | 11:13 |
| wgz | okay. how high did the repack get up to on its own? | 11:19 |
| AuroraBorealis | well | 11:37 |
| AuroraBorealis | its currently repacking "chk:chk node 861000/987188" | 11:37 |
| wgz | that's good, and what's the mem usage? | 11:39 |
| AuroraBorealis | right now 650,000 KB | 11:39 |
| AuroraBorealis | highest was around 850,000 KB | 11:39 |
| wgz | ha, so it has stabilised. | 11:39 |
| AuroraBorealis | it jumped up a lot when it started doing the chk: index/nodes and whatever | 11:40 |
| wgz | so, if you want to try something | 11:40 |
| wgz | do ctrl+break in the terminal | 11:40 |
| AuroraBorealis | the pause|break button? | 11:41 |
| wgz | yup. | 11:41 |
| wgz | this should then put you in the python debugger | 11:41 |
| wgz | then you can do... | 11:41 |
| wgz | pdb> from bzrlib.trace import _dump_memory_usage | 11:41 |
| wgz | pdb> import sys | 11:42 |
| wgz | pdb> _dump_memory_usage(sys.stderr) | 11:42 |
| wgz | and finally to resume, | 11:42 |
| wgz | pdb> c | 11:42 |
| AuroraBorealis | well that crashed it | 11:43 |
| wgz | paste me the output? | 11:43 |
| AuroraBorealis | didn't give any output | 11:43 |
| AuroraBorealis | just said "python27_64.exe has stopped working" | 11:43 |
| wgz | ah, msgbox from windows? | 11:43 |
| AuroraBorealis | yeah | 11:43 |
| wgz | at which point? | 11:44 |
| AuroraBorealis | there are the windows dump files | 11:44 |
| AuroraBorealis | _dump_memory_usage(sys.stderr) | 11:44 |
| wgz | okay... meliae apparently has issues on win64 then >_< | 11:44 |
| AuroraBorealis | gahhhhh | 11:45 |
| wgz | if you could file a bug against and stick the stuff windows gave you as an attachment, that would be great | 11:46 |
| wgz | the good news is that using 64-bit process looks like it'll let the fast-import succeed | 11:46 |
| AuroraBorealis | yeah | 11:46 |
| AuroraBorealis | but running out of memory on 32 bit still seems like a problem | 11:46 |
| wgz | yup, and the usage of both fast-import and repacking seem to both be problematic | 11:48 |
| AuroraBorealis | yes | 11:50 |
| AuroraBorealis | i restarted it again, using 600,000 at stage 2 so yeah. | 11:50 |
| AuroraBorealis | so file a bug against meliae? | 11:55 |
| wgz | right. | 11:56 |
| wgz | you could also run it's testsuite to see if that crashes (and can then be narrowed down to one test) | 11:56 |
| wgz | can do that with... `cd meliae` `Python27\python setup.py build_ext -i` `Python27\python -m unittest meliae.tests.test_suite` | 11:57 |
| wgz | just use a new window, no need to interrupt the repack | 11:57 |
| wgz | ...I get two failures, but they look shallow | 11:59 |
| AuroraBorealis | haha | 12:00 |
| AuroraBorealis | the unit tests crash. | 12:00 |
| wgz | that's good. | 12:00 |
| AuroraBorealis | what debugs the debugger? | 12:00 |
| AuroraBorealis | WATCHMENNNNNNN | 12:00 |
| wgz | you can now try smaller subsets till you find one test that does it | 12:00 |
| AuroraBorealis | how do i run the individual tests? replace the meliae.tests.test.suite with the individual files? | 12:01 |
| wgz | start with meliae.tests.test_scanner | 12:02 |
| AuroraBorealis | crash :< | 12:02 |
| wgz | and then you can narrow further with the classes in that file, then the methods | 12:02 |
| AuroraBorealis | i found the names i'll run em all | 12:02 |
| AuroraBorealis | is it because i'm not running cmd.exe in administrator mode? | 12:04 |
| wgz | it's likely one (or more) type errors where the extension code is assuming sizeof(int) == sizeof(*ptr) | 12:04 |
| AuroraBorealis | test__intset works, test__loader works, test__scanner crashes, test_loader crashes, test_perf_counter works, test_scanner crashes | 12:05 |
| wgz | which is true for win32 and the 64-bit scheme used on nix | 12:05 |
| AuroraBorealis | ah :3 | 12:07 |
| wgz | anyway, you've got enough to file a bug now | 12:13 |
| AuroraBorealis | yeah | 12:13 |
| wgz | there should be a way to get debugging symbols for a proper C stack trace, but I've not done it with cython before | 12:13 |
| AuroraBorealis | doing that now | 12:13 |
| AuroraBorealis | i have no idea either xD | 12:19 |
| AuroraBorealis | https://bugs.launchpad.net/meliae/+bug/864593 | 12:19 |
| ubot5 | Ubuntu bug 864593 in Meliae "Crashes on win64 when running test suite" [Undecided,New] | 12:19 |
| AuroraBorealis | another note, bzr is using over 1,200,000 KB when repacking now o.o | 12:23 |
| wgz | hm. | 12:28 |
| AuroraBorealis | i'll just let it run. never debugged python stuff this indepth before .. | 12:28 |
| wgz | can you run another smaller chunk of the meliae test suite? start with meliae.tests.test__scanner.TestSizeOf | 12:30 |
| wgz | if that still crashes, add .test_None | 12:30 |
| AuroraBorealis | that didn't crash, 3 tests failed | 12:31 |
| wgz | okay, that's what I was hoping for | 12:31 |
| wgz | failing tests on the sizeof things implies logic errors, that could cause crashes in bigger chunks of code | 12:31 |
| wgz | stick those three failed tests output in a comment on the bug | 12:31 |
| AuroraBorealis | it seems that they are always off by 4 | 12:32 |
| wgz | yeay! alignment. | 12:32 |
| AuroraBorealis | 37 != 33, 102437 != 102433, 38!=34 | 12:32 |
| wgz | paste them in the bug report and I'll have a look | 12:32 |
| AuroraBorealis | posted! | 12:33 |
| wgz | hm, just strings, probably just the 2.7 check above not taking alignment into account rather than something more fundamental. | 12:39 |
| AuroraBorealis | aww | 12:40 |
| wgz | try each of the other classes in that file in turn | 12:41 |
| AuroraBorealis | TestDumpInfo crashes python | 12:43 |
| AuroraBorealis | all others pass | 12:44 |
| wgz | okay, that's good, gives us a narrowed scope. | 12:44 |
| wgz | not much, but a bit. | 12:45 |
| wgz | this pyrex ` ctypedef long size_t | 12:46 |
| wgz | is wrong, but probably unimportant, I'm never sure what the manual definitions actually matter for | 12:46 |
| AuroraBorealis | can i try an individual method in a test case? | 12:47 |
| wgz | that's in _scanner.pyx and there are some similar ones in the other pyx files, could try deleting them and recompiling, I'll keep looking | 12:47 |
| wgz | ^yup, just .method_name on the end | 12:48 |
| wgz | in the TestDumpInfo case they all call the same helper which goes to dump_object_info so should all crash | 12:48 |
| AuroraBorealis | yeah they all crash | 12:49 |
| wgz | ...this could be something as silly as the IO failing | 12:52 |
| AuroraBorealis | oh? :o | 12:52 |
| wgz | have you tried deleting those size_t defs yet? | 12:53 |
| AuroraBorealis | in __scanner.py and then recompiling? | 12:53 |
| wgz | yup, then run one of the TestDumpInfo cases again | 12:54 |
| AuroraBorealis | what do i delete? these? size_of = _scanner.size_of | 12:54 |
| AuroraBorealis | oh | 12:54 |
| AuroraBorealis | nvm haha | 12:54 |
| wgz | nope, just the ctypedef at the top. | 12:54 |
| wgz | long is 4 bytes on win64, and size_t is 8 bytes | 12:55 |
| wgz | I think cython should ignore that decl and use the right thing, but you never know | 12:55 |
| AuroraBorealis | sorry i suck at c, what do i delete? lots of size_t things | 12:56 |
| wgz | just the one line at the top of _scanner.pyx "ctypedef long size_t" | 12:59 |
| AuroraBorealis | oohhh the pyx file | 12:59 |
| wgz | hm, yeah, that shouldn't matter I think. | 13:00 |
| AuroraBorealis | yeah, still crashes | 13:01 |
| wgz | okay, let's eliminate the possibility of it being the io | 13:03 |
| wgz | okay, I've added a test to TestDumpInfo that does a bit less | 13:10 |
| wgz | http://paste.ubuntu.com/701020/ | 13:10 |
| wgz | then run with meliae.tests.test__scanner.TestDumpInfo.test_int_no_file and it passes here | 13:10 |
| wgz | I'm interested if that simplified version still crashes for you. | 13:11 |
| wgz | ugh, tabs, <http://paste.ubuntu.com/701022/> | 13:12 |
| AuroraBorealis | i dont need to recompile right | 13:13 |
| AuroraBorealis | and that test passed | 13:15 |
| wgz | eh he he. | 13:15 |
| wgz | it printed my two little notes as well, right? | 13:15 |
| AuroraBorealis | yeah | 13:15 |
| wgz | now change recurse_depth=0 from 0 to 1 and try again. | 13:15 |
| AuroraBorealis | printed a period too | 13:16 |
| AuroraBorealis | that works as well | 13:16 |
| wgz | okay, change that back, and change `sio.write` to `file("nul", "wb")` | 13:18 |
| wgz | which, if the IO is indeed the problem, will crash. | 13:18 |
| AuroraBorealis | yeeeup. crashes. | 13:19 |
| wgz | so, either meliae and python are linked against different runtime libraries, or there's a bug | 13:20 |
| * AuroraBorealis whimpers at the fact of runtime libraries not working :<< | 13:21 | |
| wgz | okay, so, nothing's jumping out at me, and I need some lunch | 13:30 |
| AuroraBorealis | haha | 13:30 |
| AuroraBorealis | take your time, its 6 am here so i probably should go to bed sometime. | 13:31 |
| wgz | but there are two things I can think of that would help (bar getting a stacktrace... which may have actually been just as easy) | 13:31 |
| wgz | one is to use dumpbin.exe to look at python27.dll and _scanner.pyd to see if they're importing from the same runtime | 13:32 |
| wgz | the other is to try commenting/printing in _scanner.pyx to narrow down exactly what crashes, using our minimal test | 13:33 |
| AuroraBorealis | what is dumpbin.exe? | 13:33 |
| wgz | it just lists symbols in the exe/library, either msvc or the win7 thing should have come with it | 13:33 |
| wgz | on the trial and error front, for starts | 13:34 |
| wgz | in dump_object_info comment the _dump_object_info calls so just the fflush happens, and see if there's still a crash | 13:35 |
| wgz | remember to rerun `setup.py build_ext -i` each time after changing a pyx file | 13:36 |
| wgz | if that works, can just commenting the fwrite call in _file_io_callback | 13:37 |
| AuroraBorealis | which test do i run, yours? | 13:37 |
| AuroraBorealis | running your test again and commenting out the _dump_object_info it still crashes | 13:39 |
| wgz | heh, that does seem just like mismatched FILE* then | 13:47 |
| wgz | how did the install get that wrong... | 13:48 |
| wgz | so, possible workaround for now | 13:56 |
| wgz | <http://paste.ubuntu.com/701043/> | 13:56 |
| AuroraBorealis | doesn't crash, i get this: http://paste.ubuntu.com/701045/ | 13:59 |
| wgz | cool, now run the rest of those tests | 14:05 |
| wgz | (that one fails because we were just discarding the output as what we cared about was if the IO crashed) | 14:06 |
| AuroraBorealis | i get all the failures i was getting before, yours and then TestMemObj and TestObjManager failing | 14:07 |
| AuroraBorealis | http://paste.ubuntu.com/701050/ | 14:08 |
| wgz | okay, so reverting everything but the workaround should mean you have a functional meliae to install and use on bzr | 14:09 |
| wgz | can do `bzr shelve -destroy` to get rid of the other changes | 14:10 |
| wgz | *--destroy | 14:10 |
| AuroraBorealis | i was just downloading the tarballs cause launchpad is still not accepting my ssh key for some reason =/ | 14:12 |
| wgz | ha, well, I think we didn't change anything else important so go ahead and install | 14:13 |
| AuroraBorealis | yeah | 14:13 |
| AuroraBorealis | just restored the tarballs copy and redid the change | 14:13 |
| AuroraBorealis | you should probably post that workaround to the bug report as i have no idea what that did xD | 14:15 |
| wgz | I'll do it | 14:16 |
| wgz | okay, so as a test, start a repack, then do the ctrl+break and my instructions from earlier before we got side-tracked, you should get a .json dump of where the memory usage is | 14:17 |
| AuroraBorealis | ok | 14:18 |
| AuroraBorealis | so, where is it dumping the json | 14:37 |
| wgz | it should tell you, but in %TMP% basically | 14:39 |
| AuroraBorealis | k. still dumping xD | 14:39 |
| wgz | whereever the tempfile module creates things | 14:40 |
| AuroraBorealis | well | 14:41 |
| AuroraBorealis | the json file is 4 gb | 14:41 |
| AuroraBorealis | and cannot be opened by my text editor | 14:41 |
| wgz | that's not overly suprising. | 14:41 |
| AuroraBorealis | do you think paste.ubuntu.com would be mad if i pasted it there? =P | 14:42 |
| wgz | ehehe | 14:43 |
| wgz | the strip_duplicates.py script may reduce it somewhat, and 7z or bzip2 should compress it quite well | 14:44 |
| AuroraBorealis | yeah | 14:44 |
| AuroraBorealis | trying with 7z | 14:44 |
| AuroraBorealis | oh yeah. compression on text files is sexy | 14:45 |
| wgz | <http://jam-bazaar.blogspot.com/2010/08/step-by-step-meliae.html> gives some ideas of what you can do with the dump | 14:46 |
| wgz | the summary should be pastebinable at least :) | 14:46 |
| AuroraBorealis | yeah | 14:51 |
| AuroraBorealis | loading~ | 14:53 |
| AuroraBorealis | omgggggg | 14:59 |
| AuroraBorealis | it failed x.x | 14:59 |
| AuroraBorealis | 'address <number here> not present" | 15:00 |
| * AuroraBorealis tries the dump again | 15:00 | |
| wgz | any luck second time? | 15:19 |
| AuroraBorealis | stillll dumping | 15:20 |
| AuroraBorealis | keeps saying an address is not present =/ | 15:27 |
| wgz | how frustrating. | 15:28 |
| AuroraBorealis | yeah D: | 15:28 |
| AuroraBorealis | no idea whats going on now... | 15:29 |
| wgz | try with a much smaller bzr process first, so at least you can then send me the dump and it won't take as long to retry? | 15:30 |
| AuroraBorealis | k | 15:30 |
| wgz | eg, `bzr info` and interrupt if (if you're fast enough) | 15:31 |
| wgz | there are still some casts in the meliae source I don't like but the problem might be elsewhere | 15:34 |
| AuroraBorealis | www.kramidnarg.com/stuff/bzr_memdumptn8bb_.7z | 15:34 |
| AuroraBorealis | that still gives the same error if i try to laod it | 15:34 |
| wgz | okay, I get the KeyError too. | 15:37 |
| AuroraBorealis | k | 15:40 |
| AuroraBorealis | i'm probably going to go to bed now | 15:41 |
| AuroraBorealis | when i get up i guess we can figure out why its not loading the dumps :< | 15:41 |
| wgz | looks like there are some refs that aren't in the dump, probably doesn't need to be a fatal error | 15:41 |
| wgz | sleep well! | 15:42 |
| wgz | don't dream of running out of memory. | 15:42 |
| AuroraBorealis | =P | 15:42 |
| === AuroraBorealis is now known as aurora|zzz | ||
| === marienz_ is now known as marienz | ||
| Enma_Hinobara | I have a quick question about something really trivial that is disproportionately hard to figure out how to do in Bazaar. | 21:20 |
| Enma_Hinobara | I have a directory containing a program, and I want to import it into Bazaar. However, I don't want to make said directory a branch (easy), I want the branch to be in a shared repository (already created) and the original directory become a working tree for that branch. How might this be done? | 21:20 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!