[00:19] jelmer: lp:~jelmer/bzr-pqm/fix-branch-root [00:19] jelmer: why the rejection? [00:24] lifeless: it just fixed the hack that was there, somebody else fixed it properly in the meantime by using the lp API [00:32] morning [00:55] lifeless: https://bugs.edge.launchpad.net/bzr/+bug/562666 [00:55] Launchpad bug 562666 in bzr "2a fetch is very cpu intensive" [Undecided,New] [01:38] hi igc, still here? [01:39] hi poolie [01:39] Dirk might be willing to do x64 builds (see mail) [01:39] that would be cool if it works [01:40] poolie: it certainly would [01:40] poolie: I'll reply to Dirk later today [01:40] cool, i thought you would, just wanted to make sure you saw it [01:41] poolie: yep, I did === retupmoca` is now known as retupmoca [02:25] can someone remind me how to add an existing branch to a shared repository? [02:27] You mean reconfigure --use-shared? [02:27] fullermd: thankyou [02:40] * igc out for a few hours [02:47] bzr: ERROR: exceptions.ImportError: cannot import name XMLSerializer [02:47] aroo? [02:48] I haven't seen that for a while. [02:49] IIRC it has been due to installing bzr on top of an older installation without first cleaning out the .pyc files. [02:49] If that's not the issue, then more details please :) [02:55] anyone know of some good documentation for bazaar's python API? need to write some stuff that programmatically queries a repo [03:01] spiv: GREAT. [03:01] spiv: well. that would fix the activity [03:01] krobertson: our pydoc is ok [03:02] krobertson: for specific use cases we generally advise people to look at the code in builtins.py [03:02] lifeless: thanks, I will check that out [03:31] spm, lifeless: pqm has (I assume) failed my branch twice, but has not sent me a failure email [03:32] spiv: Ahh I think I know why - and it's likely your fault. again. ;-) [03:32] It doesn't want to hurt your feelings... [03:32] spiv: smtplib.SMTPSenderRefused: (552, '5.3.4 Message size exceeds fixed limit', 'pqm@bazaar-vcs.org') <== I haven't yet had a chance to chase down tho [03:32] spm: erk [03:33] that's a *guess* mind. but I've noticed 2 of them... [03:33] spm: that would be because of the log of the failing test run, I suppose :( [03:33] spiv: 8.30 last night; and 11.05 this morning. Sound about right? [03:33] Yep. [03:34] Any chance you can send me the most recent one manually? [03:34] How big is the offending message, do you know? [03:35] I suspect they get dropped; but the pqm log for it may exist... [03:38] spiv: ~spiv/bzr/diff-relock ? [03:40] spm: right [03:47] spm: oh, can we get that limit raised ? :) [03:47] lifeless: I suspect so. :-) submit an RT? it'll be a GSA task. [03:48] spiv: https://chinstrap.canonical.com/~spm/patch.1271201969.log.gz [03:48] spm: how big is that unzipped ? [03:48] lifeless: 44Mb [03:48] woah [03:48] hmm [03:49] we may want to strip passing tests sooner rather than later. [03:49] spm: thanks! [03:49] lifeless: yeah [03:50] thats a pqm tweak [03:50] lifeless: afaict, something changed around the 2d of April (~ 3K gzip) and the 6th 22K, and later on the 6th, was 44Mb. [03:50] spm: subunit detailed streams [03:50] spm: we're getting much more data about errors now [03:50] ahhhh [03:51] wasn't sure if know change; or not. coolio. [03:51] Not to mention much more data about successes ;) [03:51] known* [03:51] heh [03:52] spm: sent to launchpad at rt.c.c [03:52] ta [03:52] spm: which I'm told makes an instant right-queue task [03:52] spm: is pjdc an appropriate naggee for this ? [03:53] lifeless: not atm unless super zomg critical [03:53] spm: we shouldn't leave errors getting blocked for long [03:53] Erik's the current VG, so our mornings. [03:54] theres no vg listed atm ? [03:56] he finishes around midday I believe [03:56] 11. close enough. DST changes. [03:57] ok [03:57] so I want to make sure this is changed before 5pm - or whever you clock off [03:57] its my fault its been left like this :( [04:03] lifeless: re bug 562079 [04:03] Launchpad bug 562079 in bzr "incorrect warning after update" [High,Confirmed] https://launchpad.net/bugs/562079 [04:03] is this actually a regression, ie something that used to work? [04:03] poolie: I'm pretty sure it is [04:03] also, please put a meaningful subject line on [04:03] "incorrect warning after update" could mean almost anything [04:03] I think the rearranged update code is picking up some noise [04:04] poolie: I do try to put meangingful subjects [04:04] poolie: I'm sorry that that one isn't meaningful enough [04:04] np [04:08] is there code for doing the moral equivalent of ls -lR on a transport somewhere already? [04:09] iter_files_recursive [04:09] but it does not directly get the stat values [04:09] or do you want a human-oriented one? [04:09] mwhudson: for http you'll need webdav [04:09] mwhudson: to do listing [04:09] lifeless: we can switch away from http [04:10] poolie: thanks, i think that'll do [04:10] poolie: on the subject of subjects etc; you seem to remind people a lot; perhaps you could just lead by example and say 'I've changed the subject to XXX which I think is clearer' [04:15] lifeless: for a dump copy, does http://pastebin.ubuntu.com/414067/ sounds vaguely like what you meant wrt ls -lR ? [04:15] mm [04:16] i guess i should sort them [04:20] mwhudson: yeah [04:20] as a first approx thats simplest and it shouldn't a) infinite loop or b) trigger often [04:21] simple is good here [04:21] lifeless: fwiw, i don't think i've said this yet, this is only for the initial mirror of the branch [04:22] mwhudson: sure, you have and I got that ;) [04:22] ok cool [04:23] maybe not explicitly but I'm passingly familiar with the problem [04:45] hey spiv... it looks like your _move_index code has higher overhead than we predicted [04:45] I'm profiling loggerhead [04:46] and a bit of code that iterates over 'get_parent_map()' in order to build up the mainline history [04:46] ends up calling _move_* 5k times for the mainline [04:46] which ends up taking 3.7s of runtime [04:50] I can reproduce it by just using "branch.revision_history()" [04:50] certainly something we usually avoid, but still, 5k get_parent_map() calls shouldn't end up costing 3.5s of wall-clock time [04:52] one possibility... [04:52] hit_names is a list [04:52] and you do "for name, index in X: if name in hit_names" [04:53] which means that if you have 20 indices [04:53] and all are hit [04:53] you end up doing 20x20 lookups, rather than 20x1 lookups [04:54] I'll poke a bit [04:58] changing it to a set didn't really help [04:58] changing it to only _move_to_front every 10th call to iter_entries did [04:59] (dropping total time from 3.5s => .35s...) [05:20] found a better fix, at least for my use case [05:20] if the given indexes are already at the front, skip trying to reorder [05:20] happens 5138 out of 5153 times [05:23] jam: ah, nice [05:23] I think this is bug #562429, btw [05:23] Launchpad bug 562429 in bzr "performance regression for 'log' in trunk?" [High,Confirmed] https://launchpad.net/bugs/562429 [05:23] but haven't actually looked at the bug close enough yet. [05:24] jam: yeah, I just guessed that :) [05:25] I'm not sure about the hit_names being a set vs a list [05:25] "if the given indexes are already at the front, skip trying to reorder" sounds good to me. [05:25] I *think* a set is better [05:25] but since the real improvement is to just skip [05:25] I'll leave that code alone for now [05:25] Yeah. [05:29] Or to rephrase: "if no accessed index was a miss, then don't reorder." [05:30] How common is it (in your use case at least) that iter_entries doesn't look at every index in self._indices? [05:31] https://code.edge.launchpad.net/~jameinel/bzr/2.2-move-to-front-if-changed-562429/+merge/23371 [05:31] spiv: so I'm iterating the mainline [05:31] I may hit a few of them on the way back [05:31] but the last few thousand are all in the same index [05:32] I'm thinking that iter_entries could note how many it looked at, and tell _move_to_front, so that _move_to_front can avoid touching the last half if only the first half needs to change. [05:32] spiv: something like that could be worthwhile, but the above fix is pretty trivial [05:32] and solves my prob completely [05:33] to give an impression, my '.rix' has 1.3MB as the largest, next is 120k, 88k, 6x40k, 4x4k, 8x1k [05:33] So orders of magnitude each, and I'm guessing 90% of my query is that first index [05:34] spiv: alternatively, your iteration loop could see once all hit_names have been found [05:34] and stop the loop [05:34] wouldn't that do the same? [05:36] jam: which loop? The one in _move_to_front_by_index? [05:38] The loop in _move_to_front_by_name would benefit from that, but I don't think that alone would benefit as much as your current fix. [05:39] I expect most of the cost is in _move_to_front_by_index, not _move_to_front_by_name. [05:40] Although there is a little bit of repeated effort when _move_to_front_by_name is invoked, because it calls _move_to_front_by_index which constructs the same list comprehension. [05:41] spiv: as an example: http://paste.ubuntu.com/414088/ [05:41] The way _move_to_front_by_index has to zip-then-unzip the self._index_names + self._indices lists doesn't help. [05:42] and yes, the cost is in _move_to_front_by_index according to lsprof [05:42] That avoids the packing and repacking [05:42] well, with a bug [05:42] http://paste.ubuntu.com/414089/ [05:42] that should be better [05:43] Oh, right, yes that approach would be a little better. [05:43] anyway, *my* fix avoids calling all the siblings entirely [05:43] lifeless: oh good idea re commenting on the bug [05:43] Right. [05:43] hi jam [05:43] hi poolie [05:43] spiv: it would be nice to be 'cheaper' in case we don't know about other cases [05:43] Hmm [05:43] I was also thinking about the 'ordering' issue [05:43] That's true. [05:44] And iter_entries() should preserve the existing order [05:44] given that is *why* we are bothering to reorder them [05:44] I don't expect http://paste.ubuntu.com/414089/ will perform worse than the current implementation in any case. [05:44] so if the first indexes are all 'active' then we won't be thrashing the first few for minor improvements [05:44] which my original proposal of "sort by count() of hits" [05:44] could have done [05:45] (though again, in my particular use case, it wouldn't matter) [05:45] Right. [05:45] And of course, over the network the relative cost of a little CPU thrashing vs. extra IO looks different to what you're looking at. [05:45] sure [05:46] poolie: econtext I think [05:46] So if we can keep both down reasonably easily, we're both happy ;) [05:46] from before our conversation, about explaining why i'm retargeting [05:46] s//resummarizing [05:47] spiv: right. Also, don't all the sibling entries have identical index ordering? [05:47] jam: yeah :/ [05:48] well, good and bad [05:48] it means that if we built a map of where to move them [05:48] we could just apply it multiple times [05:48] however [05:48] the good is that if we know this one is sorted well enough [05:48] then we don't have to worry that we aren't sorting a sibling [05:48] Right. [05:48] poolie: oh right [05:51] poolie: and for laughts, a bugfix for the bug we discussed [05:51] https://code.edge.launchpad.net/~lifeless/bzr/update/+merge/23372 [05:52] lifeless: merge: approve [05:53] though there would be potential other cases if you had >1 pending merge and only one of them ended up filtered [05:53] right [05:53] but hey, no need to be perfect about it [05:53] thats the history analysis or api changes to feed those decisions up the stack that I refer to [05:57] poolie: https://code.edge.launchpad.net/~lifeless/bzr/update/+merge/23372 - using my patched hydrazone [05:57] *zine* [05:57] spiv: some perf results [05:58] bzr.dev: 2.142s [05:58] my paste w/o shortcut: 1.491s [05:58] shortcut: 1.111s [05:59] thanks [05:59] shortcut w/ paste: 1.111s [05:59] enough variability to be within the noise with or without the paste [05:59] so... we may want to merge it, as it would have helped [06:00] but the shortcut helps more [06:00] spiv: oh, and the reason I put it in _move_... is because it is called from 2 places [06:00] _iter_entries and iter_entries_by_prefix [06:01] lifeless: what's with the "Status Approved => Queued; Queue Position => 68" message? [06:02] jam/lifeless: any thoughts on the possibly ip problems in https://answers.edge.launchpad.net/bzr/+question/98657 [06:02] poolie: other than saying "I've seen similar throughput issues from launchpad" I don't really know [06:02] jam: cool, that's about what I'd expect. [06:03] jam: I think I'm happy for you to merge the paste too, because as you say there might be other access patterns that the shortcut doesn't entirely fix. [06:04] jam: I'm glad that the results are matching up with our analysis of the problem :) [06:11] mwhudson: hey, I beg to differ: '# Looms suck.' [06:11] jam: moving towards using the queueing abstraction in lp [06:11] lifeless: don't beg, just invoke /usr/bin/diff :P [06:12] lifeless: i guess someone should test if that bit is still necessary [06:12] jam: I realise the extra mails may be a little annoying, I'm going to beg^Wask for a little patience [06:12] i have this recollection of being extremely angry getting that code to work [06:12] mwhudson: and *file a bug* if it is [06:12] (i think weaves broke it too, or something ridiculous like that) [06:23] oh [06:24] is 2.2 open ? can't seem to choose it when proposing for merge [06:25] lifeless: trunk [06:25] 2.2 is still just a merge from trunk away [06:25] it is just there to make the RM manager's life easier [06:25] well, at least how I used to RM [06:28] speaking of which.. poolie did we ever get pqm happy with your 2.2 tarball? [06:28] no, still on my plate for today [06:28] k [06:28] night everyone [06:28] so i should close my mail :) [06:28] night [06:44] jam: I was noting that you merged to 2.2 in pqm [07:02] back [07:43] poolie: sorry for the spam messages from me; still learning hotkeys etc [07:47] Howdy, anyone with a sec to help with a question regarding shared repository formats? [07:48] sure, just ask [07:48] thx1 [07:48] question is: can I find out what the repo format of lp:ipython is? [07:48] bzr info gives me: [07:49] Standalone branch (format: unnamed) [07:49] Location: [07:49] branch root: bzr+ssh://bazaar.launchpad.net/~ipython-dev/ipython/trunk/ [07:49] bzr: ERROR: Parent not accessible given base "bzr+ssh://bazaar.launchpad.net/~ipython-dev/ipython/trunk/" and relative path "../../../../%7Evcs-imports/ipython/main/" [07:49] I need to create a shared repo that I can then push from, but all my attempts end up with [07:49] try bzr info -v nosmart:lp:ipython [07:49] sorry [07:49] Using saved push location: lp:ipython [07:49] bzr: ERROR: RemoteRepository(bzr+ssh://bazaar.launchpad.net/~ipython-dev/ipython/trunk/.bzr/) [07:49] is not compatible with [07:49] KnitPackRepository('file:///home/fperez/ipython/repo/.bzr/repository/') [07:49] different rich-root support [07:49] try bzr info -v nosmart+lp:ipython [07:49] that may work better [07:49] aha! [07:49] many thanks! [07:50] Standalone branch (format: pack-0.92) [07:50] L [07:50] That's what I needed :) [07:52] fperez: I suspect though [07:52] fperez: that the vcs import is in 2a or similar, and that you'll want to upgrade trunk, and your local branches - everything - to 2a. [07:53] #launchpad can talk about branch mgmt stuff in more detail. Well we know here too, but that is the right channel :P :) [07:53] Mmh, let me try the push I was about to make and see how it goes. Back in a sec. [07:54] moin [07:55] lifeless: actually it just worked: [07:56] amirbar[lp]> bzr push [07:56] Using saved push location: lp:ipython [07:56] Pushed up to revision 1232. [07:56] thanks a lot, you saved me a ton of grief [08:00] fperez: ok cool [08:00] fperez: glad I could help [08:27] hello. i am trying to write a test case for bug #413406 [08:27] Launchpad bug 413406 in bzr "bzr export fail with directory contain non-ascii char." [Medium,Confirmed] https://launchpad.net/bugs/413406 [08:27] http://pastebin.com/sb8hMUU1 the test doesn't seem to throw the exception while doing the same from command line throws the exception [08:28] hmm something seems to have broken bzr-search :< [08:28] do i need to do something differently in the test? [08:28] parthm: intersting [08:30] lifeless: the fix is quite simple actually http://pastebin.com/SS5TKqFw i am assuming bzr uses utf8 internally. is that correct? the only thing i don't have is an automated test :( [08:32] parthm: we use unicode internally [08:32] parthm: utf8 in some code paths where we have audited it carefully [08:33] parthm: if name is the filename on disk we're meant to create, it sounds like the user in that bug has an invalid fs encoding [08:34] hmm, bug shows that assumption is wrong [08:35] parthm: I've commented on the bug [08:36] parthm: you need to use the fs encoding, not utf8, because its a pathname on disk we're setting [08:37] lifeless: makes sense [08:37] lifeless: i suppose they fixed it in py3.0 :) [08:37] parthm: possibly, but likely not. [08:39] parthm: as fo the unit teset [08:39] test [08:39] try [08:39] self.run_bzr(['export', '--format', 'tgz', u'test.tar.gz']) [08:40] lifeless: yes. that makes the test case work. thanks. [08:41] I'd add a comment there that we're tickling a posixpath.py bug and need a unicode argument to do so. [08:41] for clarity. [08:41] lifeless: for fs encoding does bzr provide an api or is os.getfilesystemencoding() the suggested way? i see osutils doing "_fs_enc = sys.getfilesystemencoding() or 'utf-8'" [08:41] lifeless: good point. will do that. [08:42] look in osutils [08:42] we do have a wrapper/cache I think [08:43] lifeless: osutils.get_user_encoding? [08:44] no, thats the terminal [08:44] _fs_enc [08:45] is what is used throughout the module [08:46] lifeless: will use that. maybe that should be _fs_enc? or exposed via a api? [08:49] spell it osutils._fs_enc [08:49] for test friendliness [08:49] it is exposed through an API :) - osutils._fs_enc [08:51] lifeless: :) thanks for your help. i will raise a merge proposal for this fix. [08:51] cool [09:02] looks like bzr 2.1.1 was not announced. is missing on the feed on homepage. https://launchpad.net/bzr/+announcements [09:06] and from the channel topic :-) [09:29] * igc dinner [09:49] poolie: is there a way to mute things that are 'to: me' in gmail ? [09:49] poolie: (e.g. all lp bug fail) [09:49] are there any PPAs which have packages of the 2.0.x release branch in them? [09:51] OOPS-1565CEMAIL38 [09:51] https://lp-oops.canonical.com/oops.py/?oopsid=1565CEMAIL38 [09:52] mthaddon: (losa) ^ - looks like we're still failing on merge emails [09:56] It seems that all of the PPAs listed in http://wiki.bazaar.canonical.com/DistroDownloads don't contain the 2.0.x branch. [09:59] * vila back online and from dentist [09:59] hi all [10:03] how do I convert a lightweight checkout to a heavy checkout? [10:04] heya vila! [10:04] edgimar: reconfigure [10:04] bialix, ok thanks. [10:05] bialix: _o/ [10:07] bialix, tried 'bzr reconfigure --checkout', and got "bzr: ERROR: exceptions.AssertionError: is not a RemoteBzrDir" [10:08] (using 2.1.1) [10:09] this is a bug [10:10] edgimar: please, file a bug [10:10] how can it be that this kind of bug has not already occurred (or has it?)-- no one else has ever tried converting a lightweight to a heavy checkout?? [10:11] edgimar: I suspect the problem here is that your master branch is not local one [10:11] Yes, that is correct. [10:11] I believe it works correctly if master branch is local one [10:11] and I think there is even test for the local case [10:12] just somewhere between here and there something become broken, and there is no corresponding test, so core devs won't noticed it [10:13] as a workaround you'll need create heavy checkout from scratch :-/ [10:13] ok fine -- but I guess that for every test which is performed on local branches, it should also be tested on remote branches, right? [10:13] anyway you'll need to download entire branch history from remote location [10:13] vila: ^ [10:14] edgimar: in theory -- yes. in practice??? [10:14] bialix: you're correct, filing a bug will help find the hole in the test coverage [10:15] I'm sure that doing the testing in both scenarios for all tests can be automated/encapsulated. [10:15] anyway, I'll file the bug... [10:16] edgimar: thanks === vila changed the topic of #bzr to: Bazaar version control | try https://answers.launchpad.net/bzr for more help | http://irclogs.ubuntu.com/ | Patch pilot: spiv | bzr 2.1.0 is out === vila changed the topic of #bzr to: Bazaar version control | try https://answers.launchpad.net/bzr for more help | http://irclogs.ubuntu.com/ | Patch pilot: spiv | bzr 2.1.1 is out [10:18] indeed 2.1. is out [10:22] But filed at: https://bugs.launchpad.net/bzr/+bug/562896 [10:22] Launchpad bug 562896 in bzr "crash when converting lightweight to heavy checkout with remote master branch" [Undecided,New] [12:20] vila: http://pqm.bazaar-vcs.org/ [12:20] vila: check the pqm message out :P [12:21] lifeless: even better ! the leading 'success' is new right ? (the running [ x%] was there for a couple of days already) [12:21] vila: the success is not new [12:21] lifeless: what is then ? [12:21] the test ids ? [12:22] vila: the (lifeless) and (Parth Malwankar) [12:22] haaa, the *message* ! I had a crash last day and forgot to file the bug after that [12:23] lifeless: where did you fix that ? [12:23] hydrazine [12:24] bzr+ssh://bazaar.launchpad.net/~lifeless/hydrazine/cron/ [12:24] thumper: why you get up, why do new queued proposals get queu position *68* ? it seems oddly high,,, [12:25] And highly odd. [12:25] But also even. [12:26] 2 * 2 * 17, perfectly regular [12:27] spiv: even if its highly odd, its still oddly high [12:27] lifeless: :) [12:30] lifeless: where is this queue coming from ? [12:30] lp [12:30] vila: if you use my branch and set a message, and submit them, I'll do the signing. [12:31] meh [12:31] you mean your cron job will do the actual pqm submission ? [12:32] is there an url on lp where I can see this queue ? [12:38] lifeless: ^ ? [12:38] vila: oh, no. [12:38] not wanting to wake up fullermd by fixing typos but did you mean "it'll do the signing" ? [12:42] my gpg key [12:42] so 'me' [12:47] lifeless: *I* do a submission and *you* sign it ? [12:47] yes [12:47] decoupled 'queue' and 'tell pqm' [12:48] anyone with review approval permissions can land stuff now, using my hydrazine cron branch [12:50] ha ok, I'm not the primary target. So, that will avoid ready-to-land patches languishing but I'm a bit uncomfortable with automated landing... (I already make mistakes manually :) [12:50] vila: its no more automated than what you're using [12:50] someone has to say 'land this now', by hand. [12:52] lifeless: ok, I was confused by 'cron', you still need to run feed-pqm [12:52] right [12:52] you run feed-pqm [12:52] and it sets the metadata [12:52] then my feed-pqm picks it up and sends the email [12:54] ok, ok, there is a race condition of course but I doubt we run into it soon, may be the patch pilot should be the one to run with --cron [12:55] vila: race condition ? [12:55] if two people try to send_mp() [12:55] no race [12:56] or rather, no worse than bzr-pqm's pqm-submit command has === mrevell is now known as mrevell-lunch [12:59] lifeless: hmm, in queue_mp(), you have setStatus(status='Approved') followed by setStatus('Queued') [13:01] lifeless: then in send_mp, you check that the mp has already been sent and return early, if two users come here they will both miss the other submission and will both submit no ? [13:02]