[00:22] <wgz> right, I should really go to bed.
[00:22] <AuroraBorealis> night =)
[00:22] <AuroraBorealis> i'll be on
[00:23] <wgz> I'll stay logged in here, say how far you get
[00:24] <AuroraBorealis> kk!
[02:19] <AuroraBorealis> wgz, I finally got the sdk installed and created the vcvarsamd64.bat file, but its still giving me ValueError: [path] (which i guess means it failed to call vcvars)
[02:19] <AuroraBorealis> i guess i'll talk to you about it when you are up =P
[06:21] <AuroraBorealis> i seriously hate compiling things haha
[10:12] <wgz> AuroraBorealis: I think you need to read "Trick distutils" onwards in <http://mattptr.net/2010/07/28/building-python-extensions-in-a-modern-windows-environment/>
[10:13] <AuroraBorealis> i did that
[10:13] <AuroraBorealis> still doesn't work
[10:13] <AuroraBorealis> i might of put that line in msvc9compiler in the wrong place but i dont think i did
[10:14] <wgz> okay, I'll take another look at your traceback and see if I can work out what's up
[10:15] <wgz> http://bugs.python.org/issue7511
[10:15] <wgz> "You can work around it
[10:15] <wgz> by finding and executing vcvars32.bat or vcvars64.bat before running
[10:15] <wgz> setup.py
[10:15] <wgz> "
[10:15] <AuroraBorealis> yep thats the exact error i'm getting
[10:16] <AuroraBorealis> http://paste.ubuntu.com/700959/
[10:18] <wgz> did you get the 64-bit components installed correctly as well?
 has some step-by-step it might be worth reviewing
[10:22] <AuroraBorealis> it says to check an option that was checked by default
[10:22] <AuroraBorealis> but i don't have 'cl.exe'
[10:22] <AuroraBorealis> cause the folder wasn't created there
[10:22] <AuroraBorealis> (that was the folder i created in the 'trick distutils' blog post
[10:23] <wgz> checked by default is fine, is you have the `bin\amd64\cl.exe` fine somewhere
[10:23] <AuroraBorealis> well the folder they say that cl.exe is in wasn't created by default
[10:23] <AuroraBorealis> i had to create the directory and the vcvarsamd64.bat file
[10:23] <wgz> sure, but can you find it somewhere else?
[10:24] <AuroraBorealis> i see cl.exe in Microsoft Visual Studio 9.0/VC/bin
[10:24] <wgz> if so, I'd go ahead and open a new cmd window, call vcvarsamd64.bat, then run setup.py and see what happens
[10:25] <AuroraBorealis> oh gee
[10:25] <AuroraBorealis> i think its screwing up on the uf-8 bom
[10:26] <AuroraBorealis> well
[10:27] <AuroraBorealis> i ran it, the terminal text turned green
[10:27] <AuroraBorealis> and now its compiling
[10:27] <wgz> heh, that's possible
[10:27] <AuroraBorealis> it said <randomgibberish>call is not a windows batch file/command
[10:27] <wgz> yeah, that's an annoying way to have got stuck :)
[10:28] <AuroraBorealis> so when i install this
[10:28] <AuroraBorealis> will it always use this version of bzr lib? or will running the exe from the windows installer override it
[10:29] <wgz> it won't interfere with the exe version
[10:29] <AuroraBorealis> k
[10:29] <AuroraBorealis> Well. now that installed
[10:29] <AuroraBorealis> i forget what else i need to do
[10:29] <wgz> lp:meliae lp:python-fastimport lp:bzr-fastimport
[10:30] <wgz> I recommend testing the bzr you've just installed, should be able to run it with Python27\Scripts\bzr.bat
[10:32] <AuroraBorealis> seems to of worked!
[10:32] <wgz> good good!
[10:36] <AuroraBorealis> ok
[10:36] <AuroraBorealis> all 3 installed
[10:38] <AuroraBorealis> now what!
[10:39] <wgz> well, it would be nice to test that the memory dumping worked but...
[10:39] <wgz> could just start the import
[10:40] <AuroraBorealis> start it again from the beginning?
[10:40] <wgz> `Scripts\bzr fast-import ..\tmp_linux\linux.fe \bzr-import` or similar
[10:40] <wgz> hm.
[10:40] <AuroraBorealis> cause i can't seem to continue cause of another bug
[10:40] <wgz> yeah
[10:41] <wgz> you could try repacking the failed import for a faster big-memory usage
[10:41] <wgz> but with 64 bit process you probably won't OOM now anyway
[10:41] <AuroraBorealis> is there a command to repack?
[10:44] <wgz> `bzr pack`
[10:44] <wgz> ...it took me way too long to find that, I fail at name guessing/remembering
[10:45] <wgz> you can look the windows task manager for ball park memory usage numbers
[10:47] <AuroraBorealis> yeah i'm watching it
[10:47] <AuroraBorealis> 280,000 KB
[10:50] <lifeless> wgz: have you installed bzr guess ?
[10:50] <wgz> the -Dmem_dump option is needed to write the stats on OOM, but that probably won't happen now so breakin and calling bzrlib.trace._dump_memory_usage directly is probably more useful anyway
[10:50] <wgz> lifeless: nope
[10:50] <lifeless> wgz: give it a spin :)
[10:51] <lifeless> $ bzr repack
[10:51] <lifeless> Command 'repack' not found, perhaps you meant 'pack'? [y/n]:
[10:51] <AuroraBorealis> will it dump the memory automatically using meliae or do i have to do the -Dmem_dump thing
[10:51] <wgz> ha, and I was just about to doubt that it'd know to remove the 're' prefix
[10:51] <wgz> I should have more trust in lifeless.
[10:51] <lifeless> wgz: it uses a distance calculation :)
[10:52] <lifeless> wgz: $ bzr pdck
[10:52] <lifeless> Command 'pdck' not found, perhaps you meant 'pack'? [y/n]:
[10:52] <wgz> ^It will only auto dump if you OOM, which with a 64-bit process and 8 gigs probably isn't going to happen
[10:52] <wgz> (and needs the flag)
[10:52] <AuroraBorealis> k
[10:52] <wgz> but with ctrl+break then a bit of typing in the debugger we can get what we want anyway
[10:52] <AuroraBorealis> so this seems like its going to work, although its using 600,000KB atm hehe
[10:53] <AuroraBorealis> 64 bit is a lost faster o.o
[10:53] <AuroraBorealis> lot*
[10:56] <wgz> scusi, just going to pick apples, but I'll be around - try the import from the beginning if that works
[11:13] <AuroraBorealis> kk =)
[11:19] <wgz> okay. how high did the repack get up to on its own?
[11:37] <AuroraBorealis> well
[11:37] <AuroraBorealis> its currently repacking "chk:chk node 861000/987188"
[11:39] <wgz> that's good, and what's the mem usage?
[11:39] <AuroraBorealis> right now 650,000 KB
[11:39] <AuroraBorealis> highest was around 850,000 KB
[11:39] <wgz> ha, so it has stabilised.
[11:40] <AuroraBorealis> it jumped up a lot when it started doing the chk: index/nodes and whatever
[11:40] <wgz> so, if you want to try something
[11:40] <wgz> do ctrl+break in the terminal
[11:41] <AuroraBorealis> the pause|break button?
[11:41] <wgz> yup.
[11:41] <wgz> this should then put you in the python debugger
[11:41] <wgz> then you can do...
[11:41] <wgz> pdb> from bzrlib.trace import _dump_memory_usage
[11:42] <wgz> pdb> import sys
[11:42] <wgz> pdb> _dump_memory_usage(sys.stderr)
[11:42] <wgz> and finally to resume,
[11:42] <wgz> pdb> c
[11:43] <AuroraBorealis> well that crashed it
[11:43] <wgz> paste me the output?
[11:43] <AuroraBorealis> didn't give any output
[11:43] <AuroraBorealis> just said "python27_64.exe has stopped working"
[11:43] <wgz> ah, msgbox from windows?
[11:43] <AuroraBorealis> yeah
[11:44] <wgz> at which point?
[11:44] <AuroraBorealis> there are the windows dump files
[11:44] <AuroraBorealis> _dump_memory_usage(sys.stderr)
[11:44] <wgz> okay... meliae apparently has issues on win64 then >_<
[11:45] <AuroraBorealis> gahhhhh
[11:46] <wgz> if you could file a bug against and stick the stuff windows gave you as an attachment, that would be great
[11:46] <wgz> the good news is that using 64-bit process looks like it'll let the fast-import succeed
[11:46] <AuroraBorealis> yeah
[11:46] <AuroraBorealis> but running out of memory on 32 bit still seems like a problem
[11:48] <wgz> yup, and the usage of both fast-import and repacking seem to both be problematic
[11:50] <AuroraBorealis> yes
[11:50] <AuroraBorealis> i restarted it again, using 600,000 at stage 2 so yeah.
[11:55] <AuroraBorealis> so file a bug against meliae?
[11:56] <wgz> right.
[11:56] <wgz> you could also run it's testsuite to see if that crashes (and can then be narrowed down to one test)
[11:57] <wgz> can do that with... `cd meliae` `Python27\python setup.py build_ext -i` `Python27\python -m unittest meliae.tests.test_suite`
[11:57] <wgz> just use a new window, no need to interrupt the repack
[11:59] <wgz> ...I get two failures, but they look shallow
[12:00] <AuroraBorealis> haha
[12:00] <AuroraBorealis> the unit tests crash.
[12:00] <wgz> that's good.
[12:00] <AuroraBorealis> what debugs the debugger?
[12:00] <AuroraBorealis> WATCHMENNNNNNN
[12:00] <wgz> you can now try smaller subsets till you find one test that does it
[12:01] <AuroraBorealis> how do i run the individual tests? replace the meliae.tests.test.suite with the individual files?
[12:02] <wgz> start with meliae.tests.test_scanner
[12:02] <AuroraBorealis> crash :<
[12:02] <wgz> and then you can narrow further with the classes in that file, then the methods
[12:02] <AuroraBorealis> i found the names i'll run em all
[12:04] <AuroraBorealis> is it because i'm not running cmd.exe in administrator mode?
[12:04] <wgz> it's likely one (or more) type errors where the extension code is assuming sizeof(int) == sizeof(*ptr)
[12:05] <AuroraBorealis> test__intset works, test__loader works, test__scanner crashes, test_loader crashes, test_perf_counter works, test_scanner crashes
[12:05] <wgz> which is true for win32 and the 64-bit scheme used on nix
[12:07] <AuroraBorealis> ah :3
[12:13] <wgz> anyway, you've got enough to file a bug now
[12:13] <AuroraBorealis> yeah
[12:13] <wgz> there should be a way to get debugging symbols for a proper C stack trace, but I've not done it with cython before
[12:13] <AuroraBorealis> doing that now
[12:19] <AuroraBorealis> i have no idea either xD
[12:19] <AuroraBorealis> https://bugs.launchpad.net/meliae/+bug/864593
[12:23] <AuroraBorealis> another note, bzr is using over 1,200,000 KB when repacking now o.o
[12:28] <wgz> hm.
[12:28] <AuroraBorealis> i'll just let it run. never debugged python stuff this indepth before ..
[12:30] <wgz> can you run another smaller chunk of the meliae test suite? start with meliae.tests.test__scanner.TestSizeOf
[12:30] <wgz> if that still crashes, add .test_None
[12:31] <AuroraBorealis> that didn't crash, 3 tests failed
[12:31] <wgz> okay, that's what I was hoping for
[12:31] <wgz> failing tests on the sizeof things implies logic errors, that could cause crashes in bigger chunks of code
[12:31] <wgz> stick those three failed tests output in a comment on the bug
[12:32] <AuroraBorealis> it seems that they are always off by 4
[12:32] <wgz> yeay! alignment.
[12:32] <AuroraBorealis> 37 != 33, 102437 != 102433, 38!=34
[12:32] <wgz> paste them in the bug report and I'll have a look
[12:33] <AuroraBorealis> posted!
[12:39] <wgz> hm, just strings, probably just the 2.7 check above not taking alignment into account rather than something more fundamental.
[12:40] <AuroraBorealis> aww
[12:41] <wgz> try each of the other classes in that file in turn
[12:43] <AuroraBorealis> TestDumpInfo crashes python
[12:44] <AuroraBorealis> all others pass
[12:44] <wgz> okay, that's good, gives us a narrowed scope.
[12:45] <wgz> not much, but a bit.
[12:46] <wgz> this pyrex `    ctypedef long size_t
[12:46] <wgz> is wrong, but probably unimportant, I'm never sure what the manual definitions actually matter for
[12:47] <AuroraBorealis> can i try an individual method in a test case?
[12:47] <wgz> that's in _scanner.pyx and there are some similar ones in the other pyx files, could try deleting them and recompiling, I'll keep looking
[12:48] <wgz> ^yup, just .method_name on the end
[12:48] <wgz> in the TestDumpInfo case they all call the same helper which goes to dump_object_info so should all crash
[12:49] <AuroraBorealis> yeah they all crash
[12:52] <wgz> ...this could be something as silly as the IO failing
[12:52] <AuroraBorealis> oh? :o
[12:53] <wgz> have you tried deleting those size_t defs yet?
[12:53] <AuroraBorealis> in __scanner.py and then recompiling?
[12:54] <wgz> yup, then run one of the TestDumpInfo cases again
[12:54] <AuroraBorealis> what do i delete? these? size_of = _scanner.size_of
[12:54] <AuroraBorealis> oh
[12:54] <AuroraBorealis> nvm haha
[12:54] <wgz> nope, just the ctypedef at the top.
[12:55] <wgz> long is 4 bytes on win64, and size_t is 8 bytes
[12:55] <wgz> I think cython should ignore that decl and use the right thing, but you never know
[12:56] <AuroraBorealis> sorry i suck at c, what do i delete? lots of size_t things
[12:59] <wgz> just the one line at the top of _scanner.pyx "ctypedef long size_t"
[12:59] <AuroraBorealis> oohhh the pyx file
[13:00] <wgz> hm, yeah, that shouldn't matter I think.
[13:01] <AuroraBorealis> yeah, still crashes
[13:03] <wgz> okay, let's eliminate the possibility of it being the io
[13:10] <wgz> okay, I've added a test to TestDumpInfo that does a bit less
[13:10] <wgz> http://paste.ubuntu.com/701020/
[13:10] <wgz> then run with meliae.tests.test__scanner.TestDumpInfo.test_int_no_file and it passes here
[13:11] <wgz> I'm interested if that simplified version still crashes for you.
[13:12] <wgz> ugh, tabs, <http://paste.ubuntu.com/701022/>
[13:13] <AuroraBorealis> i dont need to recompile right
[13:15] <AuroraBorealis> and that test passed
[13:15] <wgz> eh he he.
[13:15] <wgz> it printed my two little notes as well, right?
[13:15] <AuroraBorealis> yeah
[13:15] <wgz> now change recurse_depth=0 from 0 to 1 and try again.
[13:16] <AuroraBorealis> printed a period too
[13:16] <AuroraBorealis> that works as well
[13:18] <wgz> okay, change that back, and change `sio.write` to `file("nul", "wb")`
[13:18] <wgz> which, if the IO is indeed the problem, will crash.
[13:19] <AuroraBorealis> yeeeup. crashes.
[13:20] <wgz> so, either meliae and python are linked against different runtime libraries, or there's a bug
[13:21]  * AuroraBorealis whimpers at the fact of runtime libraries not working :<<
[13:30] <wgz> okay, so, nothing's jumping out at me, and I need some lunch
[13:30] <AuroraBorealis> haha
[13:31] <AuroraBorealis> take your time, its 6 am here so i probably should go to bed sometime.
[13:31] <wgz> but there are two things I can think of that would help (bar getting a stacktrace... which may have actually been just as easy)
[13:32] <wgz> one is to use dumpbin.exe to look at python27.dll and _scanner.pyd to see if they're importing from the same runtime
[13:33] <wgz> the other is to try commenting/printing in _scanner.pyx to narrow down exactly what crashes, using our minimal test
[13:33] <AuroraBorealis> what is dumpbin.exe?
[13:33] <wgz> it just lists symbols in the exe/library, either msvc or the win7 thing should have come with it
[13:34] <wgz> on the trial and error front, for starts
[13:35] <wgz> in dump_object_info comment the _dump_object_info calls so just the fflush happens, and see if there's still a crash
[13:36] <wgz> remember to rerun `setup.py build_ext -i` each time after changing a pyx file
[13:37] <wgz> if that works, can just commenting the fwrite call in _file_io_callback
[13:37] <AuroraBorealis> which test do i run, yours?
[13:39] <AuroraBorealis> running your test again and commenting out the _dump_object_info it still crashes
[13:47] <wgz> heh, that does seem just like mismatched FILE* then
[13:48] <wgz> how did the install get that wrong...
[13:56] <wgz> so, possible workaround for now

[13:59] <AuroraBorealis> doesn't crash, i get this: http://paste.ubuntu.com/701045/
[14:05] <wgz> cool, now run the rest of those tests
[14:06] <wgz> (that one fails because we were just discarding the output as what we cared about was if the IO crashed)
[14:07] <AuroraBorealis> i get all the failures i was getting before, yours and then TestMemObj and TestObjManager failing
[14:08] <AuroraBorealis> http://paste.ubuntu.com/701050/
[14:09] <wgz> okay, so reverting everything but the workaround should mean you have a functional meliae to install and use on bzr
[14:10] <wgz> can do `bzr shelve -destroy` to get rid of the other changes
[14:10] <wgz> *--destroy
[14:12] <AuroraBorealis> i was just downloading the tarballs cause launchpad is still not accepting my ssh key for some reason =/
[14:13] <wgz> ha, well, I think we didn't change anything else important so go ahead and install
[14:13] <AuroraBorealis> yeah
[14:13] <AuroraBorealis> just restored the tarballs copy and redid the change
[14:15] <AuroraBorealis> you should probably post that workaround to the bug report as i have no idea what that did xD
[14:16] <wgz> I'll do it
[14:17] <wgz> okay, so as a test, start a repack, then do the ctrl+break and my instructions from earlier before we got side-tracked, you should get a .json dump of where the memory usage is
[14:18] <AuroraBorealis> ok
[14:37] <AuroraBorealis> so, where is it dumping the json
[14:39] <wgz> it should tell you, but in %TMP% basically
[14:39] <AuroraBorealis> k. still dumping xD
[14:40] <wgz> whereever the tempfile module creates things
[14:41] <AuroraBorealis> well
[14:41] <AuroraBorealis> the json file is 4 gb
[14:41] <AuroraBorealis> and cannot be opened by my text editor
[14:41] <wgz> that's not overly suprising.
[14:42] <AuroraBorealis> do you think paste.ubuntu.com would be mad if i pasted it there? =P
[14:43] <wgz> ehehe
[14:44] <wgz> the strip_duplicates.py script may reduce it somewhat, and 7z or bzip2 should compress it quite well
[14:44] <AuroraBorealis> yeah
[14:44] <AuroraBorealis> trying with 7z
[14:45] <AuroraBorealis> oh yeah. compression on text files is sexy
 gives some ideas of what you can do with the dump
[14:46] <wgz> the summary should be pastebinable at least :)
[14:51] <AuroraBorealis> yeah
[14:53] <AuroraBorealis> loading~
[14:59] <AuroraBorealis> omgggggg
[14:59] <AuroraBorealis> it failed x.x
[15:00] <AuroraBorealis> 'address <number here> not present"
[15:00]  * AuroraBorealis tries the dump again
[15:19] <wgz> any luck second time?
[15:20] <AuroraBorealis> stillll dumping
[15:27] <AuroraBorealis> keeps saying an address is not present =/
[15:28] <wgz> how frustrating.
[15:28] <AuroraBorealis> yeah D:
[15:29] <AuroraBorealis> no idea whats going on now...
[15:30] <wgz> try with a much smaller bzr process first, so at least you can then send me the dump and it won't take as long to retry?
[15:30] <AuroraBorealis> k
[15:31] <wgz> eg, `bzr info` and interrupt if (if you're fast enough)
[15:34] <wgz> there are still some casts in the meliae source I don't like but the problem might be elsewhere
[15:34] <AuroraBorealis> www.kramidnarg.com/stuff/bzr_memdumptn8bb_.7z
[15:34] <AuroraBorealis> that still gives the same error if i try to laod it
[15:37] <wgz> okay, I get the KeyError too.
[15:40] <AuroraBorealis> k
[15:41] <AuroraBorealis> i'm probably going to go to bed now
[15:41] <AuroraBorealis> when i get up i guess we can figure out why its not loading the dumps :<
[15:41] <wgz> looks like there are some refs that aren't in the dump, probably doesn't need to be a fatal error
[15:42] <wgz> sleep well!
[15:42] <wgz> don't dream of running out of memory.
[15:42] <AuroraBorealis> =P
[21:20] <Enma_Hinobara> I have a quick question about something really trivial that is disproportionately hard to figure out how to do in Bazaar.
[21:20] <Enma_Hinobara> I have a directory containing a program, and I want to import it into Bazaar. However, I don't want to make said directory a branch (easy), I want the branch to be in a shared repository (already created) and the original directory become a working tree for that branch. How might this be done?