[00:14] Hello, I was wondering what is the impact of using Bazaar Explorer on a Git repository. [00:14] I have all of the proper tools for bzr installed like bzr-git. So far I have just found that it could not pull down a git repository. [00:22] Is any one in here? [00:26] hi [00:26] yes [00:26] what specifically fails? [00:38] Are there any issues with python 2.7 (bzr, bzr-svn) - that should cause me to use python 2.6? [00:40] None that I know of [00:42] poolie: Sorry did not notice you responded. Well it appears that the git plugin is read only for git repos. [00:43] poolie: I was hoping to use a nice tool like Bazaar explorer to help me contribute to Telepathy [00:43] poolie: But I have not found a single complete GIT tool, so I guess I will just have to guess at it. and be CLI only. [00:44] wolfrage: i didn't think it was readonly [00:45] i think you do have to use 'dpush' rather than 'push', per http://doc.bazaar.canonical.com/migration/en/foreign/bzr-on-git-projects.html [00:45] poolie: http://doc.bazaar.canonical.com/plugins/en/git-plugin.html [00:47] ah, i think think title might just be a bit out of date [00:47] s//that title [00:47] the project homepage says "All operations except for "push" are supported" [00:48] hmmm ok, but with dpush I should be able to push the git repo to an online repository [00:48] what do you mean by online? [00:49] gitorious account [00:49] you want to push from a local git repo to a remote one? [00:49] what happens if you try just bzr push? [00:49] yes [00:51] poolie: stby, it seems to have not error'd this time, one sec. [00:52] anyone using bzr-hg can do a test to me? [00:53] poolie: bzr: ERROR: Unsupported protocol for url "git@gitorious.org:wolfrage-telepathy/otr.git" [00:53] poolie: Full out put: Run command: bzr push git@gitorious.org:wolfrage-telepathy/otr.git [00:53] bzr: ERROR: Unsupported protocol for url "git@gitorious.org:wolfrage-telepathy/otr.git" [00:56] wolfrage: bzr-git uses URLs like git://... [00:57] spiv: good catch that was for ssh [00:58] Or git+ssh:// [00:58] how would I have it do ssh on git, or is that not possible... [00:59] so this should work then? git+ssh://gitorious.org/wolfrage-telepathy/otr.git [00:59] Maybe git+ssh://git@gitorious.org/wolfrage-telepathy/otr.git [00:59] (I'm not a bzr-git expert) [01:00] spiv: k trying [01:01] spiv: Thanks that worked! [01:01] Great :) [01:05] OK one more question and then I will leave you guys alone for a little while. How can I pull and merge the revisions that have occurred in this repository: http://git.collabora.co.uk/?p=telepathy-spec.git;a=summary and merge them back into mine. More... explanation on the way. [01:06] Basically the repository I have has not been updated for nearly 3 months. I want it up to date before I being to much work on it. [01:07] Is that asking too much? [01:07] That would be a merge right? [01:08] pull if the revision history hasn't diverged, merge if they have. [01:09] i.e. pull is good for updating a mirror of another branch, merge is what you use when you have an independent branch and want to incorporate changes from another branch. [01:09] OK this one is independent and has been for almost a year, and last got a update three months ago [01:09] So I should go with a merge. [01:19] when I try to merge it fails with permission denied === Ursinha-afk is now known as Ursinha [01:32] poolie: this is aj00200 with that Unicode bug. Thanks for all your help :) [02:06] poolie: the problem is that bzr tries to load the whole file into memory. This is a real blocker for our team to consider bzr.. :( [02:10] let's say I create a branch from upstream release 1.0, then commit my patches, then merge upstream changes, then improve my patches etc. Is there any way to generate a patch for 1.0 release, that is, a diff of the changes since release 1.0 but ignoring all merges? [02:12] Not in any remotely automatic way. [02:12] I'm thinking on setting up two branches for that, one mypatch-r1.0 as main entry point for improving the patch, and other mypatch-dev from which I merge both upstream and mypatch-r1.0 changes [02:13] You could attempt it by a series of cherrypicks. You could try finding (or building) a patch composition tool and doing the whole diff, then reverses of each merge (sorta the complement of the cherrypick method). [02:13] More flippant-cum-HHOS'ly, you could convert it into darcs and then commute the patches ;) [02:14] HHOS? High Horse Of Silliness? [02:15] Well, I was thinking Ha Ha Only Serious, but that works too :p [02:18] cherrypicks? flippant-cum-HHOS? darcs? [02:20] RenatoSilva: Well, it's a bit yflurgic, but as a TRX it's less nurrific than your average KSTRN-ish arglefarx. :] [02:20] o.O, it seems easier just keeping the two branches [02:21] Very probably. [02:22] I just thought if I could do something like bzr diff -r 1.. --ignore-merges [02:23] That, no. Trying to actually do it would be decidedly Not That Simple(tm). [02:29] ok thanks fullermd [02:55] Wanted to stop back and thank you guys, now with a little bit of GIT on the command line I can run my repository with Bazaar, and do it smartly! [03:00] excellent [03:25] * spiv -> late lunch [06:15] mkanat: nice essay :) [06:15] poolie: Thanks! === oubiwann-holiday is now known as oubiwann === oubiwann is now known as oubiwann-holiday [07:37] hi all [07:38] g'morning Vincent [07:38] jelmer: heya ! [07:39] jelmer: I submitted a mp for bzr-search fixing the failing tests [07:39] \o/ [07:39] vila: I submitted a MP for most bzr-xmloutput test failures (only one left) earlier [07:39] jelmer: I digged the xmloutput a bit more... and it's even more obscure now, I ... huh ? [07:39] jelmer: meh, there was only one failure when I started... [07:40] let me see your mp... [07:41] vila: the daily build is failing with 4 [07:41] meh, I was pretty sure the custom escape_data was already fixed.... [07:42] mgz: escape_data xmloutput, can you refresh my memory ? [07:42] vila: it may not be an issue when the in-tree elementtree is used [07:42] hmm, could it be py27 related ? [07:44] jelmer: or is it that you're using 0.8.6 and I use trunk ? [07:44] jelmer: qlog says: look at revno 147 [07:46] vila: I'm using lp:bzr-xmloutput and indeed python 2.7 [07:46] jelmer: bzr-xmloutput-0.8.6+bzr148~ppa112+113~natty1 should include it which means... meh wth [07:48] jelmer: right, which should mean the bzrlib.xml_serializer logic needs one more fix for 2.7 ? [07:49] jelmer: and your remaining failure is for test_info (which is the one I'm digging) [07:50] vila: the remaining failure for me is bzrlib.plugins.xmloutput.tests.test_info_xml.TestInfoXml.test_info_locking [07:51] yup, a single test to test 8 combinations, I've got a parameterized version now with 2 failures and 3 errors [07:53] vila: also, more test failures in the natty bzr daily build :-/ [07:53] I am using Bazaar Explorer with a GIT repo, and I am trying to push a commit to my remote branch, as an update. But I keep getting this: [07:53] Run command: bzr push [07:53] but at this point I need to talk to verterok as the intent of the xmlinfo command is unclear [07:53] error: refusing to create funny ref 'None' remotely [07:53] bzr: ERROR: None failed to update [07:54] vila: I see the timestamp error you mentioned earlier, but also: "AttributeError: 'InstrumentedTestResult' object has no attribute 'report_error'" [07:54] jelmer: the later sounds like a testtools version mismatch [07:55] hi vila! [07:55] poolie: hello ! [07:55] vila: Ahm, hmm [07:55] 'evening poolie [07:55] jelmer: argh, you're using testtools trunk [07:55] jelmer: did I complain yesterday about you *not* using trunk ? [07:55] :D [07:55] doomed if you do, doomed if you don't :D [07:56] vila: :-) I'm surprised it only breaks on natty though, not on Maverick [07:56] wolfrage: hi [07:56] wolfrage: "bzr push" won't work for git branches, you can only "bzr dpush" (though arguably the error message should be clearer) [07:56] jelmer: It's a zen thing, testtools failures causes are of an infinite nature [07:56] jelmer: Hi sorry about the interupt [07:57] jelmer: that's because when you test, you want to always be prepared to accept new failures, zen thing... [07:58] jelmer: ahh, I see. I got confident as it was able to push it as a new branch. But I guess updates will not. Since I am using Explorer, if i run the command "bzr dpush" do I need any other parameters? [07:59] wolfrage: I'd be very surprised if it was able to push it as a new branch [07:59] wolfrage: bzr dpush requires the URL to the branch to push to [07:59] vila: :) [08:00] jelmer: I was surprised too, but poolie was talking to me at the time that it did it. | Ok I will give it a go with the url attached. Thank you. [08:02] jelmer: the daily builds are a nice complement to babune, but we face a common issue here: the combinations and their related failures can be daunting at times [08:03] jelmer: does the command need any dashes "bzr dpush --url" or just "bzr dpush url" [08:03] wolfrage: just "bzr dpush ", much like you would use "bzr push" [08:04] vila: The intent isn't really to provide CI but rather to allow users to more easily follow current trunk, I'd rather catch these errors before we'd actually get to the package building stage. [08:04] jelmer: hehe, but isn't that the intent of CI ? :D [08:06] jelmer: don't get me wrong, I'm very happy about the daily builds, they address combinations that I can't address so far in babune (not do I really plan to address ;) [08:06] s/not do/nor do/ [08:06] jelmer: did not work is it ok to post the error? [08:07] vila: Yeah, I guess that's also a form of QI :-) [08:07] wolfrage: sure [08:07] vila: Errhm, CI [08:07] * jelmer clearly has been watching too much QI lately [08:07] jelmer: Run command: bzr dpush git+ssh://git@gitorious.org/wolfrage-telepathy/otr.git [08:07] bzr: ERROR: and are in the same VCS, lossy push not necessary. Please use regular push. [08:07] jelmer: :) [08:07] * wolfrage thinks that is a good show. [08:07] wolfrage: ah, you've got a git repository locally that you're pushing from, not a bzr branch? [08:08] jelmer: Sorry thought I included that detail [08:08] wolfrage: I figured you were pushing from a local bzr branch to a remote git branch, I didn't realise you had a git branch locally as well. [08:08] Yes using bzr-git to push my local git to a remote git [08:09] vila: I'll chase down the testtools error. Do you have any idea about the timestamp? [08:09] wolfrage: in that case "bzr push" is indeed the right thing to use, can you file a bug about the issue you saw? [08:09] jelmer: not from the top of my head, pretty surprising [08:10] jelmer: but I vaguely remember something like that happening in the past [08:10] jelmer: sure. I was wondering should it be filled under bzr on launchpad or is there a bzr-git [08:10] wolfrage: there's a separate bzr-git project [08:11] jelmer: copy. [08:37] jelmer: Bug #690547 https://bugs.launchpad.net/bzr-git/+bug/690547 [08:37] Ubuntu bug 690547 in Bazaar Git Plugin "GIT to GIT bzr push Fails "funny ref 'None' remotely"" [Undecided,New] [08:37] Launchpad bug 690547 in Bazaar Git Plugin "GIT to GIT bzr push Fails "funny ref 'None' remotely"" [Undecided,New] https://launchpad.net/bugs/690547 [08:39] wolfrage: any chance you can add the relevant error log in ~/.bzr.log as well? [08:40] wolfrage: the fix is probably that we should push to refs/heads/master, something that we're not explicitly doing at the moment [08:41] jelmer: Working attachment. Interesting, It somehow committed a delete of the branch. It is ok, but not sure how that happened. Or maybe Gitorious just screwed up. GIT makes my head hurt [08:43] jelmer: Is it ok if I delete my email out of the logs. [08:44] wolfrage: of course [08:45] jelmer: As soon as it finishes what it is doing currently I will submit the log. [08:49] jelmer: I wanted to know I see how to add files to a git with bzr but how do I remove them? [08:52] wolfrage: it should be the same as regular bazaar [08:53] wolfrage: please note though that bzr-git's main focus so far has been on interoperating with remote git repositories, the local support is still quite experimental - in particular as we can't really map the git index to anything in bzr [08:55] jelmer: I know I am asking alot of the software, unfortunately all of the other GIT tools can not stack up to Bazaar Explorer, which is sad considering it was made for bzr not GIT. But it is helping me to contribute to the community because of it's versatility. [09:03] jelmer: I do hope that you will not mind me making more bugs though as I find them. It seems that Add has no effect and Remove fails with permission denied which makes no sense as I own all of the files and directories. [09:05] wolfrage: no, please do file more bugs [09:06] wolfrage: please note though that these don't have a high priority for me at the moment, I'm mainly concentrating on the remote git repo use case [09:07] wolfrage: patches are very much welcome though [09:07] jelmer: OK I understand priorities. I just appreciate that the Software has helped me along this far. I really was not impressed by the other GIT tools. [09:08] jelmer: LOL, After I get through with Telepathy, then I will come back over here. But I can only tackle so many bugs at once too. lol. [09:10] jelmer: Log is attached, The error is at least a few operations back. But it was attempted multiple times. [12:03] spiv: changing signatures for branch creation broke looms :-/ [13:07] I have deleted the hg directory from plugins dir but it's still showing up in bzr plugins: hg 0.2.0dev in C:\Program Files\Bazaar\plugins\hg. [13:07] How to purge it? [13:10] bzr hg-import is even running. How can that be? It's like bzr-hg is cached somehwere?! [13:18] c:\>ls "C:\Program Files\Bazaar\plugins\hg" => __init__.pyo commands.pyo info.pyo revspec.pyo [13:18] c:\>dir "C:\Program Files\Bazaar\plugins\hg" => not found o.O [13:24] RenatoSilva: 'bzr plugins -v' should tell you [13:32] vila: (11:07:02) RenatoSilva: [...] but it's still showing up in bzr plugins: hg 0.2.0dev in C:\Program Files\Bazaar\plugins\hg. [13:33] vila: where I got that path with -v [13:33] vila: seems a crazy windows 7 bug, I just rm -r'ed it [13:34] RenatoSilva: then it's probably windows specific and the plugin may be in bzrlib.zip [13:34] RenatoSilva: see if you can reinstall bzr without the plugin [13:34] vila: I removed it with rm -r [13:35] vila: bzr-hg now disappeared [13:35] RenatoSilva: *I* understand and trust you, yet, bzr plugins -v seems to disagree with you [13:36] vila: it was like the hg dir was still existing in the filesystem "partially". For explorer and dir, it didn't exist, but for bzr and ls and rm it did [13:36] RenatoSilva: try a reboot first to exclude nasty caches (doubtless) [13:36] somebody else ran into similar problems [13:36] meh, why do I say doubtless when I doubt... [13:36] yup, as jelmer says, this rings a bell [13:37] vila: I removed hg dir yesterday, and turned pc on today, and the issue was there [13:37] wowwwww [13:37] I'm not sure if it was windows 7 as well, but it seems like the GUI doesn't always display the actual contents [13:37] vila: not sure if it's clear but after rm -r, bzr plugins doesn't list bzr-hg anymore [13:38] * vila blinks [13:38] RenatoSilva: what do you mean with rm -r though? [13:39] RenatoSilva: no, it's not clear [13:39] jelmer: I thought about that, but I could create the hg dir in explorer! [13:39] jelmer: at the same time hg was already there (for some tools) [13:42] let me explain again form beginning. I removed bzr-hg yesterday by deleting C:\Program Files\Bazaar\plugins\hg. Today I turned computer on and noted that it was still there in bzr plugins. The -v options was showing that exact path. I went to explorer, and it wasn't there. I went to win32 terminal, and dir command couldn't find it. [13:44] But ls.exe could (I have MSYS)! And so rm.exe. Weirdly, I could create the hg dir again in explorer, even with it already existing for ls and bzr [13:45] So I just tried rm -r /c/Program Files/Bazaar/plugins/hg, and it worked, bzr plugins stopped showing bzr-hg in the list [13:46] * the -v option === zyga is now known as zyga-food === Ursinha is now known as Ursinha-lunch === Ursinha-lunch is now known as Ursinha [15:45] Is there a gitstats for Bazaar? If (e.g.) gitstats worked internally on git-fast-export format (don't know if it does), it would work for anything that had a fast-export frontend, no? === Ursinha is now known as Ursinha-afk === Ursinha-afk is now known as Ursinha === tremolux_ is now known as tremolux [16:10] vila/jelmer: yes, that xmloutput thing was fixed on both trunks [16:10] I'm not sure what the failures were from [16:10] but looking at I'm a little concerned [16:11] mgz: including python 2.7 ? [16:11] mgz: by the way, I nailed the last failures on osx 10.6 only to be bitten by a new one ;) [16:11] just seen that as well and will review. [16:11] mgz: I still get them on trunk: [16:12] TypeError: _escape_cdata() takes exactly 2 arguments (1 given) [16:12] mgz: patch awaiting review... too fast :) [16:12] anyway, not using a function from bzrlib.xmlserialiser would be fine [16:12] but the one you've written in does less than the etree one, which was already insufficient [16:13] mgz: the only difference with the etree one is that it doesn't take an encoding argument [16:13] right, so... passing non-ascii either unicode or bytes becomes risky. [16:14] it's a bad interface which doesn't help, but it's even easier to use incorrectly now. [16:14] mgz: we weren't passing an encoding before either [16:14] mgz: so that situation doesn't really change [16:14] no, but the etree 1.3 function defended against bad bytestrings [16:14] also, string.replace jelmer? how retro are we? :) [16:15] mgz: I just took whatever etree was using :-) I guess that can be text.replace now, indeed... [16:15] the problem in the first place was etree changed this function, which python 2.7 picked up. [16:16] I'd like to just make xmloutput less retarded about xml generation, but that's a big job. [16:17] anyway, I'll look into why you were getting failures, I thought I'd checked different version combinations [16:18] wasn't helped by most of the bzr-xmloutput test suite failing here. [16:19] mgz: FWIW I don't think we can use the stock _escape_cdata since it always encodes in python 2.7 [16:19] well, it's a question of which misuse we want to break [16:20] most of xmloutput happens to work because the data is ascii, it's not hard to break it. [16:20] mgz: there are more than a few places where the text passed into _escape_cdata is unicode, and the caller expects it to come back as unicode too [16:22] okay, yep, this is a problem. [16:24] will put review in that mp. === beuno is now known as beuno-lunch [16:36] mgz: what do you mean with xmloutput less retarded about xml generation? [16:38] it's doing the print statement and string interpolation approach [16:39] I think I've found a bug: http://pastie.org/1379843, that error pops up on a gui dialog and then qlog list commits without the messages [16:39] this can work if you're an expert about what the xml spec says, but anyone normal just gets it wrong in all kinds of edge cases. [16:39] vila: jam's reviewed your mp already, so I don't need to :) [16:39] mgz: indeed :) [16:39] jam: thanks and hi ! [16:40] mgz: I dealed with some xmloutput weaknesses some time ago, and some of them still exist [16:41] mgz: I contributed some patches for encoding fixes, but work is still needed [16:41] example of some wrongness: [16:41] except Exception, e: [16:41] xml += '%s' % \ [16:41] _escape_cdata(str(e)) [16:41] mgz: for example, bzr xmlstatus is putting cp1252 in the preamble, which is W3C invalid iirc [16:42] what happens if the exception instance stringifies to non-ascii? [16:42] well for sure, cp1252 is not a proper encoding [16:42] that depends on in iana. [16:42] some of the cp* encodings are listed, some (like cp932, which would actually be useful) aren't. [16:43] *on the iana. [16:43] http://www.iana.org/assignments/character-sets <- allows "windows-1252" [16:45] they should just add the "cp" prefix as an alias to all windows numbered encodings they list there [16:45] instead of it being pot-luck. [16:46] anyway, there's an easy fix xmloutput could use [16:46] the output-based-on-terminal-encoding is just daft. [16:47] should instead either do utf-8 with no prologue, iff the terminal is utf-8, or ascii with &#...; escaping. [16:47] but that means touching all the output code, which I don't want to do without actually making it robust. [16:48] Some things don't even like UTF-8 AFAIK. The MS libraries had a bug where they would emit ¬ characters happily enough and then drop dead trying to parse their own output. [16:49] xml needs prologue [16:50] I mean, if you have a chance to tell what encoding it is, why not say? tools reading that output will need to guess it's utf-8 if you don't declare it [16:50] nope, it's speced very clearly. [16:50] without prologue, it depends on the bom, without bom it's utf-8 [16:50] mgz: what you're saying is addressed in bug 394943 [16:50] Launchpad bug 394943 in bzr-xmloutput "Declared and actual XML encoding should match, and the encoding should be XML valid" [High,Confirmed] https://launchpad.net/bugs/394943 [16:51] thus, if you want a utf-8 xml document, ommitting the prologue is fine. [16:51] Indeed. Many things just use the platform encoding by default (esp Java which assumes that all InputStream is the JVM encoding - which is cp1252 (or whatever the equivalent is) on Windows [16:51] mgz: without bom it's utf-8? how can you guess that? you know because you make bzr-xmloutput do it, but apps doesn't have a way to know it [16:52] "without prologue, it depends on the bom, without bom it's utf-8", do you mean this is the XML spec? iirc it doesn't even allow you to remove the prologue [16:54] BOMs are a PITA anyway ; so many tools don't handle them right, so many XML output encoders don't write one (although some do). [16:55] http://www.w3.org/TR/2008/REC-xml-20081126/#dt-xmldecl [16:56] note that's not a MUST. [16:56] A SHOULD requirement is probably something you ... should .. make the effort to implement. The BOM thing is a MAY for UTF-8 ; and I'd prefer to see it left off because it will break casual compatibility with many tools if left in. [16:56] awilkins: why would xml generators choose BOM over preamble, specially when BOMs are a PITA, and W3C obligates you to write a preamble? :P [16:57] RenatoSilva, Oh, quite - I'd rather not see BOM, and I'd prefer a preamble. [16:57] does it stand must != should? I didn't know that [16:58] because in dictionary, you must do it is the same as you should do it afaik [16:58] yes, and given the way things have developed, even SHOULD is a little strong. [16:59] If you neglect a MUST you've not met the standard - missing a SHOULD will attract "tch, tch" and frowns but not get you labelled non-compliant. [16:59] And a MAY is a nice-to-have. [17:00] see 4.3.3 as well, which has some encoding specific directives. [17:00] AFAIK cp1252 is the same as ISO_8859_1 [17:00] anyway, for this plugin none of this should matter regardless of the terminal encoding [17:01] awilkins: nope, it swaps out the C1 control characters for some extra glyphs [17:01] Gah, MS at their finest. [17:02] ^shouldn't matter, because unlike rio which causes issues here, xml *does* have a proper escpaing mechanism if the encoding can't represent a character [17:02] afaik cp1252 is a superset of latin1 [17:03] mgz: see comments 15-18 in bug 394943 [17:03] Launchpad bug 394943 in bzr-xmloutput "Declared and actual XML encoding should match, and the encoding should be XML valid" [High,Confirmed] https://launchpad.net/bugs/394943 [17:05] not printing mojibake is always worth while. [17:05] it's only hard to fix because the code is a mess, particularly over what's unicode, what are bytestrings, and so on. [17:06] anyway, does the xml spec stand that if an xml misses encoding declaration, you can use boms to inform the encoding, and does it say if there's no bom you shall read it as utf-8???? this sounds weird [17:07] yes, see 4.3.3 [17:07] mgz: yeah that bug is hard to fix because the whole code needs to be checked [17:07] it's worded at little backwards, but: [17:07] "...it is a fatal error for an entity including an encoding declaration to be presented to the XML processor in an encoding other than that named in the declaration, or for an entity which begins with neither a Byte Order Mark nor an encoding declaration to use an encoding other than UTF-8." [17:08] is there something more general as well... [17:11] mgz: "In the absence of external character encoding information (such as MIME headers), parsed entities which are stored in an encoding other than UTF-8 or UTF-16 MUST begin with a text declaration" [17:11] right, "other than" [17:11] so, utf encodings don't need it. [17:13] mgz: imho this is pretty much a fallback is case there's no way to know what encoding it is than it's stimulating you to not use the xml preamble. As the doc says, you must not, but you should, use preambles. Unless you have a specific good reason for not using: [17:13] mgz: "SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course. " [17:14] right, and that was reasonable advice. [17:14] but in the years since the spec was written, no other version of xml has gained traction, and utf-8 is increasingly the default encoding everywhere. [17:15] so it's less important than it was to state the bleeding obvious. [17:16] well anyway, I'd recommend not removing the preamble, it doesn't hurt anyway. And I disagree, as I think you disagree too, with bzr-xmloutput trying to guess the output encoding to use. It should use a single static encoding, preferably utf8. [17:16] I'm not quite saying that. [17:16] There's no reason to print utf-8 to a non-utf-8 terminal, so it should just not do that. [17:17] which is trivial. [17:17] I don't get you [17:17] bzr-xmloutput is not supposed to be used in a terminal [17:18] the annotated xml spec is informative on the thinking at the time. [17:18] ? [17:18] http://www.xml.com/axml/notes/Enc1.html "Very little of the world's text is stored in Unicode encodings" [17:19] http://www.xml.com/axml/notes/ASCII.html "As the spec says, pure ASCII files are UTF-8 as they sit, and thus don't require an encoding declaration. The problem is that a lot of ASCII files are not quite pure." [17:19] ^ it's not intended to be, but these things do get used on the terminal, where's that version-info bug... [17:20] and as it's trivial to avoid printing junk to a terminal, it should. [17:21] why would one use bzr-xmloutput in terminal? examples? [17:23] Spent too much time working on web apps and can't read plain text anymore? :P [17:23] see bug 518609 [17:23] mgz: I personally don't like the idea of converting non-ascii to XML char entities [17:23] Launchpad bug 518609 in Bazaar "Unicode exception occurs by "version-info"" [Medium,Fix released] https://launchpad.net/bugs/518609 [17:24] there's no reason to use that command rather than one that works, but we got three bugs filed about it. [17:25] Renato, I think you're overestimating how hard it is to conditionally escape characters if a terminal is detected. [17:25] it's like, three lines of code in a well constructed program. [17:26] mgz: for values of 'well' known long after the program has been written :-P [17:26] mgz: not that I disagree ;-) [17:28] mgz: how about version-info, what does it have to do with xmloutput? [17:28] it's an internal-use command with escaping issues. [17:29] mgz: sorry, and how does this take you to "I must now use xmloutut"? [17:30] mgz: do you mean xmloutput has an xml version for that command and it does work? [17:30] no, I'm just pointing out that it's easier to write a command correctly than assume people won't try and break it. [17:32] anyway, I now need to go and clean the house before I get yelled at. [17:33] ok, but back to the question, why one would use bzr-xmloutput in terminal? it's not supposed to be used there. If you want to use it, then use a terminal with utf8 support. That won't be a problem for Linux users, not even for Windows users, they can chcp 650001 before command call. [17:33] mgz: nah, nobody will yell at you for cleaning xmloutput :) [17:34] So I really see no reason for xmloutput caring about terminal encoding. It should use a static utf8 encoding, that will make the actual use of xmloutut a lot easier (in bzr-java-lib, bzr-eclipse etc) [17:36] I think you think I'm suggesting something I'm not. === beuno-lunch is now known as beuno [17:36] :) [17:40] what's funny about xmloutput at the moment is that it's more crazy than before, because it mostly used to declare utf-8 and write something else, but now it gets some system default encoding and writes based on the terminal encoding, or something like that (for example win32 console is cp850, and so it is xml output, but declared as cp1252) [17:41] (continuing the example, but if I use mintty, which is not a proper win32 console, it writes latin1 bytes) [17:47] I am getting yelled at. [17:47] look, all it needs is something like: [17:47] output = sys.stdout [17:47] if getattr(output, "encoding", None) != "utf-8": [17:47] output = codecs.lookup("ascii")[3](output, "xmlescape") [17:48] and a proper method of writing xml rather than the current mess. [17:48] (which is the hard bit) [17:50] writing that is easier than saying Don't Do That whenever someone accidentally uses an xml command on a non-utf terminal, and proper parsers are perfectly capable of handling numeric entities [17:50] I don't like escaping chars to xml entities [17:51] I'd rather go with just printing the actual char [17:51] RenatoSilva: that may not be possible [17:51] jelmer: why not? [17:51] you prefer UnicodeEncodeError bug reports? [17:52] this sounds like a pretty irrational fear, and not one you'd ever actually encouter [17:52] mgz: iirc there's a way to print bytes to stdout, without encoding mess [17:52] either, as you say, no one ever prints this to a terminal, in which case you never see it [17:52] mgz: in a way that you delegate the encoding mess to the terminal or what reads stdout [17:52] or someone does try that, and you want bzr to break rather than show them something understandable. [17:53] RenatoSilva: the characters might be utf8 while the terminal is iso8859-9, so some characters would not be printable. [17:54] anyway, this is trivia, and really didn't need this much discussion. [17:54] jelmer: there's no terminal, bzr xml-output is not to be used in a terminal [17:54] what we've been talking about for half an hour has nothing to do with the actual bug that exists, or what needs to be done to fix it. [17:54] jelmer: even if so, then the user just changes terminal's encoding before running xmloutput [17:54] so I'm really going/ [17:55] RenatoSilva: So the user has to be told to change their encoding (which might be nontrivial) [17:55] mgz: I don't like your idea of entities that's why I'm discussing. I suspect it would break bzr-java-lib and bzr-eclipse more than they already are [17:56] RenatoSilva: I don't see what is problematic about entities. [17:56] RenatoSilva: Then we should fix them, don't they use standard XML parsing libraires? [17:57] jelmer: ok xml parsers should be ok with that, but not sure if bzr-java-lib and bzr-eclipse will be ok. Kind of a feeling from having writing some patches to them [17:59] jelmer: I'm not sure on the exact purpose of entities but why allow non-ascii in XML then? I mean, why not make XML standard ascii-only with using char entities for specifying non-ascii? [17:59] RenatoSilva: If they don't use standard XML parsing libraries then I suspect we have worse things to worry about. [18:00] jelmer: I wouldn't be surprised if they don't, that's what I mean :) [18:00] RenatoSilva: That's possible but can use significantly more disk space for some languages if e.g. the document is stored as UTF16. [18:02] jelmer: so it's an alternative for when there's a reason for using them [18:03] RenatoSilva: and I think this is a perfectly valid situation to use them [18:04] jelmer: for example you need to send the xml somewhere but before it gets there something reads it and tries to decode it as it was ascii-only, raising errors. So you use entities as workaround for the xml passing fine over it and getting fine in the destination [18:05] jelmer: well I don't think making things pretty to crazy users reading xml output in terminal rather than using the regular bzr commands is a good reason, it's just that what I think. [18:06] RenatoSilva: I don't think it's a very big deal, but it does mean we get to avoid a bunch of encoding-related issues. And I haven't heard any disadvantages of using entities yet. [18:11] * maxb skims scrollback and asks: Use entities for which characters, precisely? [18:13] maxb: characters that can't be represented using current terminal's encoding [18:14] That seems sane, just so long as the current encoding in the absence of a terminal being involved isn't affected by the system's terminal encoding [18:15] maxb: but the command in question is not supposed to be used in terminal, so why care so much about crazy users doing it? [18:16] RenatoSilva: because it's a hopefully simple way to do something reasonable in a terminal. Doing something reasonable rather than erroring outright is better. [18:18] maxb: as I said iirc there's a way to send raw bytes to stdout in python, so no exception would be raised. The only issue would be junk in terminal, but who cares? The user should not be doing that anyway. Besides, it's easy to fix, because Linux terminals support utf8, and in windows it's just chcp65001 [18:19] RenatoSilva: Your proposal would be acceptable behaviour, IMO, but if we can do better with little effort, why not do so? [18:19] maxb: the issues I have seen with bzr-xmloutput, bzr-java-lib and bzr-eclipse, were related to doing things wrong, not simply by the pure fact of not having chars represented as entities [18:19] OOI, what is the way to send raw bytes to stdout? [18:21] RenatoSilva: "Let's emit XML that is compatible with the terminal encoding" is not incompatible with "Let's do things right". You seem to be suggesting otherwise. [18:22] * jelmer -> foods [18:24] maxb: "Let's emit XML that is compatible with the terminal encoding" [18:26] maxb: that's valid for commands supposed to be run in terminal, not bzr-xmloutput. Besides, I see it from the opposite perspective, like the natural, and hence good way is having the actual bytes, then I start to think of the advantages of converting to entities, and I see no reason for that, in the specific case of xmloutput :) [18:29] Your goal is to get valid data out for programmatic use. Other people may also have the goal of XML that is valid if copy/pasted from a terminal. The two goals are not incompatible [18:29] maxb: convert actual bytes to entities, then converting them back to the same actual bytes. I just think this is useless overhead (ok it's useful for displaying in terminal, but still useless, because terminals are not for xmloutput) [18:31] There is no useless overhead, because if you're writing to a pipe to another program, you'd be operating in UTF-8, and the only entities you'd be using are the five basic ones to escape characters that have meaning in XML itself [18:31] I may have the goal of xmloutput making some coffee, but that doesn't mean it makes sense so that developers should work on it :P [18:36] RenatoSilva: if converting encoding and decoding special characters to entities has an overhead that's in any way noticable overhead then you're transmitting way too much data with XML anyway. [18:37] Anyone know a good document about the guts of python's stdout encoding handling I could read? [18:41] jelmer: I'm not sure if I understood correctly, but the idea is convert every non-ascii char into a xml char entity, right? So that the data is ascii-only, right? Well imagine you use bzr-eclipse to fetch the commit logs on a sufficiently big branch, it may become slow enough for being annoying. [18:44] RenatoSilva: no, only convert characters not representable in the target encoding [18:44] RenatoSilva: Other things will hit you much much worse. [18:44] jelmer: I just had an insight, why discuss this so much, xmloutput could just have an argument for that :D like --use-entities or --use-actual-bytes :) [18:44] RenatoSilva: the character conversion won't have any noticable impact [18:44] ew. NO> [18:45] maxb: what are you saying no to ? :-) [18:45] Pointless options are fail, both for understandability and maintainability [18:48] maxb: ah ok so it's using entities only for unrecognized byte sequences? ok, but converting to entity doesn't really fix the problem, it just changes from python unicode exceptions from some more high level error in the application which will be fed with such crazy data [18:49] maxb: not that I disagree with that, but just noting [18:50]