[00:22] <igc> morning
[00:24] <lifeless> poolie: thumper asked that we not use the queue feature for the moment
[00:24] <lifeless> hi igc
[00:25] <igc> hi lifeless
[00:25] <lifeless> poolie: 'e' for email is the new command in hydrazine
[00:26] <poolie> yeah
[00:26] <poolie> you broke my muscle memory a bit there
[00:26] <poolie> hello igc!
[00:26] <igc> hi poolie!
[00:27] <lifeless> poolie: sorry!
[00:27] <lifeless> poolie: if you pull my hydrazine/cron, it will show you the last comment
[00:27] <lifeless> so you can see if its been sent to pqm or whatever
[00:28] <lifeless> ok, first pass reviews done
[00:29] <lifeless> I'm going to see if I can finish a spike I started in the weekend to make completing proposed things easier.
[00:44] <lifeless> losa - did something kill stuff on balleny ? I just got an interrupted test run...
[00:51] <lifeless> spm: ^?
[03:26] <AfC> bzr: ERROR: No such file: u'/home/andrew/vcs/java-gnome/.bzr/repository/obsolete_packs/': [Errno 2] No such file or directory: '/home/andrew/vcs/java-gnome/.bzr/repository/obsolete_packs/'
[03:26] <AfC> is that normal?
[03:37] <lifeless> you've deleted that directory.
[03:37] <lifeless> Don't do that.
[03:37] <lifeless> and you should recreate it.
[04:23] <lifeless> jam: thnaks
[04:28] <AfC> lifeless: maybe bzr could create the directory if it's missing?
[04:28] <lifeless> AfC: we felt it was better not to
[04:28] <AfC> lifeless: hm
[04:28] <lifeless> directories going missing is a pretty nasty event for the filesystem to do to us
[04:29] <AfC> lifeless: and that error saying the same thing twice is better?
[04:29] <AfC> lifeless: yes, I can understand that
[04:29] <AfC> although we've been telling people to "delete the obscelete_packs directory" and if you're saying the nuance is to delete it's *contents* perhaps it's not uncommon to lose the directory?
[04:33] <GaryvdM> Morning all
[04:34]  * GaryvdM having a sleep fail :-(
[04:41] <lifeless> :<
[04:42] <GaryvdM> Reading bug reports - sleep fail over :-)
[04:43] <lifeless> AfC: we've not been telling people to do that. Some folk have said it, I certainly never havwe.
[04:43] <AfC> lifeless: oh
[04:44] <AfC> I must have misread what I found in the list archive then
[04:45] <lifeless> there are people that run pack, for no particularly good reason
[04:45] <lifeless> except that you had to in git to get performance, at one point
[04:45] <lifeless> and pack keeps the old files around because some fs's (NFS, ext4) are shonky
[04:45] <AfC> it must have been from when I had a 5GB .bzr directory for a 20 MB project.
[04:46] <lifeless> if people want to *immediately* gc, they can delete the contents of that dir
[04:46] <lifeless> but bzr will gc it after a few commits anyway
[04:46] <lifeless> (5 on average)
[09:56]  * igc dinner
[10:24] <kostja_osipov> hello. How do I find out the current format of my bzr repo?
[10:24] <kostja_osipov> I seem to be unable to find anything in the docs...
[10:25] <spiv> "bzr info -v"
[10:25] <kostja_osipov> thanks!
[10:28] <bialix> heya
[12:14] <quicksilver> bzr doesn't deal very elegantly with the case where a directory is removed in one branch and some of the files inside are modified in the other
[12:15] <quicksilver> it deals *correctly* just not very elegantly
[12:15] <quicksilver> the messages you get don't immediately tell you what's happened.
[12:16] <lifeless> please file a bug saying that
[12:16] <lifeless> gnight
[19:37] <Spyzer>  due to interruptions in my internet connection my downloading from bazaar "bzr branch lp:inkscape" is interrupted. How do i resume the downloading from the interrupted point???
[19:37] <Spyzer> please help
[19:42] <Spyzer> ??
[19:43] <Spyzer> is there no possible way of what i ask??
[19:44] <sebi`> hi, is there a command to blacklist a file (read: preventing a file to appear in the branch)?
[19:45] <sebi`> since I have a bunch of gen*.sh files, which are mostly useless to the enduser
[20:06] <Spyzer> how to ab\void downloading history in a bzr branch ??
[20:06] <Spyzer> *ab\void avoid
[20:10] <hno> Spyzer: currently only by having the history locally already...
[20:11] <hno> been discussions about supporting shallow branches / history horizons, but not sure what the current status of that is.
[20:11] <Spyzer> hno: if i do a bzr branch lp:inkscape and it interrupts in middle and if later on i use the option --use-existing-dir later on. Shall that resume the donwload the branch form the point it was terminated
[20:11] <Spyzer> ?
[20:12] <Spyzer> pls don't mind the typos
[20:13] <Spyzer> its saying bzr: ERROR: Already a branch: "inkscape"
[20:13] <hno> Spyzer: No idea, but I guess not.
[20:13] <Spyzer> how do is\relove this
[20:13] <Spyzer> oh man, is there no way to resume a download
[20:13] <Spyzer> ?
[20:17] <hno> It's only initial branch creation which is this heavy. If you already have branched some branch of the same project before then no need to download much again.
[20:19] <sebi`> anyone? ):
[20:20] <Kohlrabi> sebi`: bzr ignore?
[20:20] <sebi`> Right, thanks.
[20:47] <mkanat> spm: Did codebrowse get updated to trunk after I mentioned I fixed the fix better, the other day? (A few weeks ago.)
[20:56] <lifeless> moin
[20:57] <lifeless> jam: would like to talk network fetch efficiency if you're around
[20:58]  * hno wonders if bzr wouldn't benefit from a general cache...
[21:00] <GaryvdM> vila: ping
[21:03] <TresEquis> hno:  you have a cache lying around ;)?
[21:03] <jam> lifeless: morning. I'm trying to get reviews done, but otherwise I'd appreciate the discussion
[21:04] <lifeless> jam: code or peers ?
[21:04] <lifeless> peers we have until june 11 I think
[21:05] <jam> peers
[21:06] <lifeless> so tell me when you have a break
[21:06] <jam> lifeless: did you make the peer requests, or was it done for you?
[21:06] <lifeless> you make them yourself - on the left bottom there is 'add peer'
[21:06] <lifeless> or something to that effect
[21:07] <lifeless> we had to have the mgr review sent in monday, then there is a gap
[21:08] <hno> TresEquis: Non-obvious how to make a cache to bzr with all the smartcalls around..
[21:36] <jam> lifeless: so what's up?
[21:36] <lifeless> well
[21:36] <lifeless> we seem to be sending 2-3 times the repo size on smart push of a full branch
[21:36] <lifeless> 200MB to push bzr.dev to launchpad when its not stacked
[21:36] <lifeless> I struggle to believe that its the encoding overhead
[21:37] <lifeless> and, possibly related, we're not saturing the network at all effectively
[21:37] <lifeless> I'm patch pilot this week, so not all that well placed to just nose down and drill into this.
[21:37] <lifeless> s/uring/urating/
[21:38] <lifeless> I'm hoping that you or spiv will be between patches and able to look deeply into it
[21:39] <mkanat> lifeless: I think a smart pull of a full branch is the same, as well.
[21:39] <jam> lifeless: I'm 75% confident that it is the "split to organize 'properly'" code, and dependent on how the sort order turns up. If you want a quick and dirty test, just change the fetch order to 'unordered' and see if it doesn't drop the bytes transferred.
[21:39] <lifeless> mkanat: yes, I'd expect that
[21:39] <mkanat> lifeless: Oh, yeah, I suppose because of the indexes and all.
[21:40] <jam> mkanat: no indices for smart pull, but push should transfer ~ the same as pull
[21:40] <lifeless> mkanat: just because push and pull on the smart server are symmetrical, its just which end does the work
[21:41] <lifeless> jam: so I can do quick tests
[21:41] <lifeless> jam: what I want, is to know that someone else is going to run it down to ground ;)
[21:42] <jam> lifeless: I've looked at this a few times, and mentioned it, and didn't get much traction anywhere...
[21:42] <lifeless> jam: traction with people or code ?
[21:42] <jam> people
[21:42] <jam> If we really want minimal transfer then we need to compress on the server isde
[21:42] <jam> side
[21:42] <lifeless> ok
[21:42] <jam> IMO
[21:42] <lifeless> makes sense to me
[21:43] <lifeless> I've been saying lp needs to scale with multiple front-end CPU boxes for a while, so it won't be a total suprise there ;)
[21:43] <jam> I think we were avoiding it because of wanting to avoid load on a central location
[21:43] <jam> though still a question as to whether push or pull is more common :)
[21:43] <jam> most likely fetching from lp is the most common.
[21:43] <lifeless> I don't recall any decision to avoid it
[21:44] <jam> lifeless: we've certainly discussed wanting clients to do a bit more work than a server, since you usually have N clients and 1 server.
[21:44] <lifeless> thats true
[21:44] <lifeless> but, if wishes were horses ;)
[21:45] <lifeless> are there any stats we should be gathering from the server to tell if this is the case ?
[21:45] <jam> lifeless: the "split group" comments would be a likely point
[21:45] <lifeless> ok
[21:45] <jam> since it is taking existing groups, and splitting them into less efficient ones
[21:45] <mkanat> jam: If there are no indices for smart pull, then what makes the pull so large?
[21:45] <jam> the big thing that I've seen
[21:45] <lifeless> well something I can follow through is getting some better logging and stats
[21:45] <lifeless> mkanat: bugs ! :P
[21:46] <mkanat> lifeless: Hahaha, okay. :-)
[21:46] <jam> is when it cycles between 2 groups over and over
[21:46] <jam> taking some bits from A, then B, then A, then B
[21:46] <jam> those are definitely going to put a lot more data on the wire, as each split == fulltext
[21:47] <lifeless> so, CHKRepo format is 'unordered' already
[21:47] <lifeless> and StreamSource is groupcompress
[21:48] <jam> bzrlib/repo_fmt/groupcompress_repo.py line 1025
[21:48] <jam> set that to 'unordered'
[21:48] <lifeless> yeah
[21:49] <lifeless> vila: thanks - Working tree "/home/robertc/source/baz/bzr-test-fixes/" has uncommitted changes (See bzr status). Uncommitted changes will not be pushed. - much better.
[21:51] <lifeless> ok, finding revisions taking a bit
[21:52] <jam> lifeless: so I just pushed on the local network, and saw 101.5MB to transfer a random bzr branch.
[21:52] <jam> size on disk when finished was 36MB
[21:52] <lifeless> 3x
[21:52] <jam> well, size in 'upload'
[21:52] <jam> no indices written yet
[21:52] <lifeless> yes
[21:52] <jam> still spinning on the 'commit_write_group' I think
[21:52] <lifeless> yes, we need to address that too
[21:54] <jam> lifeless: one other thing to 'address' is that I think the 'should this be rebuilt' heuristic is wrong for chk pages. As it checks to see if a group is 'full enough' and if not, it rebuilds it. But chk groups got better compression by logical grouping than trying to make the groups max size
[21:54] <lifeless> ok
[21:55] <jam> hmm... I just waited maybe 5-10 min after pushing, and now it is on Missing Keys. Is it going to be doing the check *yet again* ...
[21:55] <lifeless> would that affect the A<->B<->A behaviour you mentin abve ?
[21:55] <jam> lifeless: I don't think so
[21:55] <jam> I'm guessing the A B A stuff is more because of inexact sorting
[21:55] <jam> The sort is stable if you have the same set of revisions
[21:55] <lifeless> ok, streaming @ 70KB
[21:56] <jam> but I don't know how stable it is when new stuff is added.
[21:56] <lifeless> I'm on adsl, so thats maybe 1/3 of my available b/w
[22:00] <jam> lifeless: well, I'm testing on the home network, and certainly not saturated.
[22:00] <jam> though I'm in the basement with the WAP 2 stories up
[22:02] <jam> lifeless: sending from a freshly populated repo, which only has the content in question
[22:02] <jam> and I see 52.9MB transferred
[22:02] <jam> note that size on disk post indicies is 50MB
[22:02] <jam> 12M in indices, 39M in packs
[22:05] <jam> lifeless: in the latter case, I see 18: "creating new compressed block" lines in .bzr.log, in the former, it is closer to 3,300
[22:12] <jam> lifeless: case 3, push from unorganized repo, using 'unordered' texts, shows about 73.2MB transferred
[22:14] <lifeless> I see many pages
[22:14] <lifeless> not 3.3K worth
[22:14] <lifeless> 12 pages
[22:14] <lifeless> 25 per page - 300
[22:15] <lifeless> 1082.823  Transferred: 68957kB (63.8kB/s r:2368kB w:66588kB)
[22:16] <lifeless> that was with unordered
[22:16] <lifeless> so I image it would be considerably worse without
[22:17] <lifeless> I can sftp upload a file at roughly the same speed
[22:17] <lifeless> +- < 10%
[22:36] <GaryvdM> vila: I've merge lp:~garyvdm/qbzr/annotate_find into lp:qbzr
[22:38] <jam> lifeless: @64kB?
[22:38] <jam> lifeless: so your unordered and my unordered seem on par
[22:38] <guijemont> GaryvdM: hi, i have a qlog question
[22:38] <lifeless> yeah, 68, 71, etc
[22:38] <GaryvdM> Hi guijemont.
[22:38] <lifeless> jam: no, unordered is ~= SFTP
[22:39] <lifeless> using lftp to upload a single pack
[22:39] <jam> lifeless: I mean that uploading via lftp you get 64kB/s
[22:39] <guijemont> GaryvdM: when expanding a node, some other nodes (with links to the expanded stuff) are expanded
[22:39] <lifeless> so, I'm getting wire speed with unordered
[22:39] <lifeless> or at least, wire + latency to london
[22:39] <jam> ... Transferred ... (63.8kB/s ...)
[22:40] <guijemont> but there seems to be a limit on the quantity of branches that is expanded
[22:40] <guijemont> but I don't get what that limit is nor how/where it is implemented
[22:40] <lifeless> `../.bzr/repository/packs/2fc2a401beb2fd09d041842e3c27ebb7.pack' at 101785571 (76%) 61.4K/s eta:7m [Sending data]
[22:40] <GaryvdM> guijemont: ah - let me have a look at the code.
[22:41] <guijemont> what I was looking at is colapse_expand_rev(), but I'm not sure it's the right place
[22:41] <GaryvdM> guijemont: That is the right place
[22:41] <guijemont> esp. since it seems ot be called with visible=True, which results to  a quite simple codepath
[22:46] <GaryvdM> guijemont: It starts by looking at rev.twisty_branch_ids
[22:47] <GaryvdM> guijemont: rev.twisty_branch_ids gets updated each time compute_graph_lines runs
[22:52] <guijemont> hmm, and does that explain how some parents get expanded, and some other not (apparently, with lines drawn with dots)
[22:52] <guijemont> ?
[22:54] <GaryvdM> guijemont: It remebers what caused a branch to expand.
[22:54] <GaryvdM> guijemont: e.g.: http://pastebin.org/276155
[22:55] <guijemont> oh, ok, and it doesn't go more than a given number of recursions into that?
[22:56] <GaryvdM> guijemont: if you expand 4, and then 3 and then collapse 3, branch line 1.1 will stay open, because 4 expanded it
[22:56] <guijemont> yeah, i noticed that
[22:57] <GaryvdM> guijemont: but if you expand 3 and then 2.1.1, then collapse 3, branch line 1.1 will collapse
[22:57] <GaryvdM> guijemont: This is what self.branch_lines[branch_id].expanded_by is for
[22:59] <GaryvdM> guijemont: If collapsing, the we 'recurse' into branch_line.merges
[22:59] <guijemont> ok
[22:59] <guijemont> not exactly my issue though
[22:59] <guijemont> hold on
[23:00] <guijemont> making a screenshot
[23:01] <GaryvdM> guijemont: processed_branch_ids is to track what we have looked at so that we don't do a branch more than once, which prevents a infinite 'recursion'/loop.
[23:02] <guijemont> GaryvdM: http://emont.org/dots.png
[23:02] <guijemont> there, some lines are drawn with dots
[23:02] <guijemont> and some of them go to the "merge level 0 ancestor" of the bzr parent
[23:02] <guijemont> without expanding happening
[23:03] <GaryvdM> guijemont: hmm...
[23:04] <GaryvdM> guijemont: the dots indicate that the line is going to a non intimidate ancestor.
[23:05] <GaryvdM> guijemont: what happens if you click on say 2972.4.2, and then click on the other parent in the lower plane?
[23:06] <guijemont> hold on, i closed the window...
[23:07] <guijemont> and what's a "non intimidate ancestor"?
[23:07] <GaryvdM> guijemont: The line layout algorithm I use dose not suit the mysql dags well. :-(
[23:08] <guijemont> well, I guess mysql is a good test
[23:08] <guijemont> since I have that bug with it with my change to viz
[23:09] <guijemont> because my method to expand to the parents
[23:09] <guijemont> is to do it one level at a time
[23:09] <guijemont> in a callback that gets called when a row is expanded
[23:10] <guijemont> so, for each new row that I expand, the callback gets called again
[23:10] <guijemont> but in the case of mysql, the depdth at which you go with that is too big
[23:10] <guijemont> and I end up with a maximum recursion exception
[23:11] <guijemont> so i wanna try to do that iteratively instead
[23:11] <GaryvdM> guijemont: e.g.: http://pastebin.org/276155 - if branch line (bl) 1.1 is visible, and bl 2.1 was hidden, we would show a dotted line from 3 to 1.1.1. 1.1.1 is not a parent of 3, but rather a distant ancestor...
[23:11] <GaryvdM> =  "non intimidate ancestor"
[23:12] <guijemont> GaryvdM: do you know of a document that explains that kind of vocabulary?
[23:15] <fullermd> It's not polite to intimidate your ancestors.
[23:15] <GaryvdM> guijemont: no sorry, these are things that I've kind of made up. I'm sure there are definition of these things in formal graph theory, which I have unfortunately not studied.
[23:16] <guijemont> ok
[23:16] <fullermd> ITYM "immediate"...
[23:16] <GaryvdM> fullermd: yes - thanks
[23:17] <guijemont> ahh
[23:17] <guijemont> oh
[23:17] <guijemont> ok
[23:17] <guijemont> might be clearer with "immediate"
[23:17] <GaryvdM> jam is seems very knowledgeable about graph theory. Maybe he can give us the correct terms.
[23:18]  * fullermd thinks there need to be more terms like "weird uncle".
[23:19] <GaryvdM> guijemont: In the code, I refer to non immediate ancestor as direct = False
[23:20] <fullermd> How about 'indirect'.
[23:20] <GaryvdM> direct = False / indirect = True
[23:20] <fullermd> I mean as a term.
[23:21] <guijemont> I'm sure I *knew* all this graph theory vocabulary well
[23:21] <fullermd> Makes sense as the opposite of direct.
[23:21] <guijemont> though I knew it in french...
[23:22] <GaryvdM> guijemont: It also does not help that I'm terrible at spelling :-(
[23:23] <guijemont> hu
[23:24] <guijemont> btw, I think colapse_expand_rev() should be called collapse_expand_rev() ;)
[23:24] <lifeless> yes
[23:24] <GaryvdM> yes - that is one of my spelling mistakes....
[23:24] <guijemont> anyway, I think I get how the dotted lines work now
[23:25] <guijemont> at least the idea, which is what interests me now
[23:25] <guijemont> GaryvdM: thanks!
[23:25] <GaryvdM> guijemont: Pleasure.
[23:34] <GaryvdM> Spelling of 'collapse' fixed. :-)
[23:40] <guijemont> cool, so I did something useful today, i reported a bug that got fixed :)
[23:41] <guijemont> now I feel productive
[23:41] <guijemont> I can go to bed
[23:43] <lifeless> mwhudson: hi
[23:43] <mwhudson> lifeless: hello
[23:43] <lifeless> mwhudson: I'd like to get stats on bzr smart server stuff on production
[23:43] <lifeless> I'm sure you've considered this.
[23:43] <lifeless> care to tell me the shortest path ? :)
[23:47] <mwhudson> lifeless: 'stats' ?
[23:47] <lifeless> how many split groups per push/pull
[23:47] <lifeless> vs unsplit groups
[23:47] <lifeless> total bytes sents
[23:48] <lifeless> possibly drilling in further
[23:48] <mwhudson> lifeless: i guess something like:
[23:48]  * mwhudson thinks a bit more
[23:48] <lifeless> like
[23:48] <lifeless> I'm not scared if you say 'change bzr, change bzr-lp-serve, change oops, and change this odd crnoscript
[23:48] <mwhudson> lifeless: i guess something like: arrange for the bzr serve processes to log somewhere more obvious than ~/.bzr.log
[23:49] <mwhudson> lifeless: then make sure that the bzr serve processes log the information you want to see (possibly with a -D style flag i guess?  maybe this already exists)
[23:49] <mwhudson> lifeless: arrange with sysadmins to copy log files somewhere you can see them
[23:50] <mwhudson> lifeless: write scripts to parse said log files
[23:50] <mwhudson> lifeless: none of these steps seems particularly hard
[23:50] <lifeless> ok
[23:50] <lifeless> whats unobvious about ~/.bzr.log ?
[23:51] <mwhudson> i got a bit frustrated at one point because i wanted to have the bzr serve process log to a file like bzr-serve-$PID.log but by the time you know the pid it's too late to set $BZR_LOG_FILE (or whatever that's called)
[23:51] <mwhudson> lifeless: well, perhaps 'obvious' was the wrong word
[23:51] <lifeless> ok
[23:51] <lifeless> what would a better word be?
[23:52] <mwhudson> lifeless: it has the output of all the bzr serve processes jumbled up in it, it's not synced anywhere, and it gets rotated by bzr's policy not the sysadmin's
[23:52] <lifeless> ok, the first point is fine (we put pids in), the sync issue we can fix, the rotation however - we should fix that in bzr.
[23:53] <lifeless> log_rotation = None in ~/.bazaar.conf ?
[23:53] <lifeless> .bazaar/bazaar.conf, I mean
[23:53] <mwhudson> i guess that works
[23:53] <lifeless> We can work up to other things
[23:59] <lifeless> alternatively
[23:59] <lifeless> perhaps
[23:59] <lifeless> log_file = ~/.bazaar/logs/$time-$pid.log
[23:59] <lifeless> in bazaar.conf