[00:00] mwhudson: Looks indeed like I was getting the assertion because of actually closing a connection before data came out of it. [00:02] mkanat: oh right [00:42] bzr missing returns a shell exit code 1 if there is branche divergence? [00:43] phoenixz: that sounds right [00:44] spiv: ah, since it was not documented in bzr help missing.. I was already wondering about this behaviour [00:44] we should doc it [00:44] Similar to how diff will return non-zero if there are differences. [00:44] The docs should say so, though. [00:45] spiv: what manual? [00:45] spiv: bzr help missing doesnt have it documented.. [00:46] spiv: http://doc.bazaar-vcs.org/bzr-0.10/bzr_man.htm also doesn't show [00:46] phoenixz: bzr help missing probably should, and maybe other places too. [00:46] Wow, 0.10 is seriously old. [00:47] Is that really the version you are using? [00:47] phoenixz: by "should" I mean, "probably doesn't yet, but that is something for us to fix" [00:47] spiv: eh, no, its just a manual I quickly looked up :) [00:50] lifeless: nice page about pmtu [00:51] the historic problem is what i thought it was, but the auto adaption is new to me, and clever [00:51] also " Just remember: The glass is neither half full nor half empty, it is merely the wrong size. " sounds like you :) [00:54] but him spelling 'microseconds' as 'uS' is like nails on a blackboard [00:56] ;) [00:56] http://en.wikipedia.org/wiki/Siemens_(unit) [00:58] lifeless: more to the point: if you suspect bad window behaviour with streaming, it would be interesting to capture the packets and graph/print/analyse the window flags [01:02] poolie: yes, spm has tools to do this [01:02] it seems like we should be able to do it at our end? [01:03] tcmpdump + tcpflow + xplot ftw [01:03] garh. s/tcpflow/tcptrace/ [01:03] rly tcmpdump? [01:03] man. I so can't type today... tcpdump :-) [01:04] wireshark does the window graphing; but I find the above more versatile and ... obvious [01:05] poolie: eg: http://tcptrace.org/manual/node12_mn.html [01:08] hi poolie [01:08] hi spiv [01:56] mkanat: are you on ubuntu one? [01:56] mwhudson: Nope. [01:56] oh, this log chunk is only 28k [01:57] mkanat: what's your email address? [01:57] mwhudson: mkanat at everythingsolved [01:57] dooot coooooom [01:57] ♫ [02:00] mkanat: you should have mail [02:00] Sweet. [02:00] Indeeed. [02:19] Hello, [02:19] I'm interested in squashing some documentation bugs [02:20] I need some advice though [02:21] First, where should I pull a branch down from? bzr or bzr-alldocs? [02:21] igc: ^ [02:21] and should I just pull one down and hack away? or should I talk some things over with a maintainer first? [02:22] rubbs: hi [02:22] rubbs: which docs did you want to fix first? [02:22] igc: hello are you Mr. Clatworthy? [02:22] rubbs: I am [02:23] igc: cool I'm Pat Regan, I was the newbie on the mailing list who expressed intrest... [02:23] the first one I was thinking was bug 59608 [02:23] Launchpad bug 59608 in bzr "'bzr ignore' undocumented behaviour and misleading tutorial" [Medium,Confirmed] https://launchpad.net/bugs/59608 [02:23] but I'm thinking it might have already been taken care of. [02:24] rubbs: most of the doc is maintained inside the main bzr code base [02:24] mkanat: fwiw; that's an extract from a fail that starts from 20091118-05:21:13.665; next restarted @ 20091118-05:42:53.336. We have others :-) lots of others! but that's the smallest to start with. [02:24] mkanat: and hi, btw :-) [02:24] spm: Thanks! [02:25] spm: Hello. :-) [02:25] rubbs: so you want to grab lp:bzr [02:25] Ok, got one grabbed [02:25] rubbs: are you using explorer or the command line tool? [02:25] mkanat: fwiw, if you need any further assist; ping me or a general 'losa ping' to summon myself or one of my fellows [02:25] igc: mostly commandline, but I've got qbzr tools installed too [02:25] spm: Awesome. [02:25] rubbs: cool [02:26] rubbs: the doc for each command is inside bzrlib/builtins.py [02:26] well, most commands at least [02:26] igc: ok [02:26] look for cmd_ignore (say) [02:26] spm: I'm hoping that I can reproduce the problem locally, but if I can't, then those logs are going to be critical, for sure. [02:27] sure. at this stage all that's beeen done is a purely random IP s/r. [02:27] s/r = search and replace? [02:27] rubbs: the tutorials are under the doc/en directory [02:27] igc: k, found it [02:27] sorry - yes. [02:28] spm: No problem. :-) [02:28] igc: ok, found those too. [02:28] rubbs: have you submitted a merge proposal using Launchpad before? [02:29] igc: no I haven't. This would be my first ever contribution to anything really. [02:29] rubbs: no problem. We're here to help [02:29] great! [02:29] rubbs: the basic summary is ... [02:29] (1) make you changes [02:29] mkanat: fwiw, had another restart just a few ago. this one actually gave us a debug return: http://pastebin.ubuntu.com/322088/ so you can see some of the requests... hang. fwiw. [02:29] (2) commit [02:30] (3) push the branch to Launchpad [02:30] (4) submit a merge proposal [02:30] spm: So these are happening pretty frequently then. [02:30] igc: ok, that makes sense. Not much different than a centralized with local commits workflow then [02:31] mkanat: :-) you *could* say that. that codebrowse is referred to by us sysadmins as codebounce, is another clue ;-) [02:31] LOL [02:31] Interesting that there are so many /atom requests for exactly two projects right before the hang. [02:31] right [02:32] where it gets fun is when our nagios check url gets hungup as well. [02:32] Hahahaha, omg. [02:32] rubbs: I need to grab some lunch right now. If you have questions, spiv and many others can help so just ask [02:33] bbiab [02:33] igc: great thanks for all your help [02:33] mkanat: sometimes some requests that usually take a few seconds hang around for minutes [02:33] mkanat: i have basically no idea why [02:33] mkanat: note that our loggerhead access branches over http [02:33] mwhudson: In terms of the port being open, or what? [02:33] not the local filesystem [02:33] Ohh, interesting. [02:34] So there's going to be a lot more latency than on my systems. [02:34] I didn't even know loggerhead could do that. [02:34] but i've never pinned this down to a real problem [02:34] mkanat: well, not much more latency, it accesses the branches over gigE (i think) to apache serving static files [02:34] * mkanat nods. [02:34] There's still pretty good latency in general over the http protocol. [02:35] mkanat: as to it being possible, bzr is awesome sometimes :) [02:35] open branch, give it to loggerhead [02:35] loggerhead doesn't care about where the branch is from [02:35] That's pretty awesome. [02:35] Yeah, bzr's architecture seems pretty sweet, generally, from the outside. [02:35] thanks [02:35] :-) [02:36] heh [02:37] Hmm, on the memory leak one, I'm actually not currently getting it to go over a certain fixed size. [02:37] I'll have to try other things. [02:38] * mkanat is also doing a Bugzilla release simultaneously, right now. [02:40] mwhudson: I was noticing that a few things seem slower than necessary. [02:40] mwhudson: Though I can't quite pin that down either. [02:40] * mkanat just got a pause on mine, too, even on localhost. [02:40] Was pretty brief, though. [02:51] mwhudson: Okay, I can at least see which threads are hanging, from the logs. [02:51] mkanat: cool [02:51] mwhudson: I can see the "Getting information for" but no "Rendering" for certain requests right before the hang. [02:51] mkanat: are they building revision caches? [02:51] mwhudson: Yeah, two of them are. [02:51] I'll check the second hang. [02:51] cool [02:53] The second one's more confusing. :-) [02:53] There's a lot happening right before that hang. [02:56] mwhudson: And yeah, I see what you mean about the long requests. Getting information for RevisionUI: 201.69120788574219 secs [02:56] mwhudson: Here's an interesting thing. All requests are fast shortly before the hang. [02:56] I mean, *until* shortly before the hang. [02:57] so "something" happens, and then suddenly requests are slow, and then we're in a death spiral? [02:57] sounds about right [02:57] mwhudson: Yeah, that seems like what's going on. [02:58] mwhudson: And it's never rendering that takes a long time, it's always getting information, from what I can see. [02:58] mkanat: now, please tell me what it is that happens [02:58] :-) [02:58] LOL [02:58] mkanat: my /theory/ is that it's basically thrashing [02:59] too much going on at once, too many threads competing for cpu, performance falls into the toilet [02:59] it would be good to confirm/deny [02:59] mkanat: if you want to enhance logging, we can get new code into production pretty fast for this [03:00] (<24 hrs) [03:00] mwhudson: Okay, awesome. [03:00] mwhudson: I might want to do that for the memory leak if I don't get anything. [03:00] mwhudson: Just like a "process increased by X size after this request" sort of thing. [03:00] mkanat: go wild :) [03:01] we could even try getting meliae dumps i guess [03:01] but they will be, uh, moderately large [03:01] Hahahaha. [03:01] Yeah, I would imagine. [03:02] I think it might be better to just narrow down where it's happening first than try to look through a memory analyzer or profiler for the whole process over like, days and days. [03:03] from my side, i view the hangs as a much more serious and mysterious problem than the leak [03:05] Okay. [03:05] I'll focus on that. [03:05] * mkanat had just started to anyway. [03:05] And the htdig that I'm running against my local repo to try to reproduce the memory leak could go on for hours or days before I get anything, anyhow. [03:06] mwhudson: BTW, as far as a cache that everything could access, memcached seems like a reasonable possibility too. [03:06] * mkanat hasn't played much with CouchDB though. [03:06] mkanat: yeah, possibly [03:07] mkanat: a disk backend would be plenty fast enough though [03:07] Anyhow, back to analysis. [03:07] mwhudson: Okay. :-) [03:07] mkanat: a more serious problem is the serialization of the graph [03:07] mkanat: we basically deserialize the graph in one lump [03:07] and that's crack, it would be really good to stop doing that [03:08] mkanat: i guess the memory leak problem will in all likelyhood come back into prominence for us if we fix the hanging problem :-p [03:10] mwhudson: Wow, yeah, I imagine that kills performance for huge projects. [03:10] * mkanat nods. [03:10] mkanat: i marshal the data and gzip it [03:10] mkanat: loading it for launchpad takes ~0.2s [03:10] so it's not terrible [03:11] Okay. [03:11] but still crack :) [03:13] I can make it slower if you want. >.> [03:13] mwhudson: If I see one thread do "Getting information" on an identical request three times in a row with no Rendering, does that mean the connection broke before it rendered? [03:13] mkanat: not necessarily [03:14] it means the requests are piling up :( [03:14] mwhudson: Hmmm. Maybe I'm missing a bit on the architecture here. How does a thread get multiple "Getting information" requests without rendering the previous requests? [03:15] mkanat: er [03:15] mkanat: can you point me to a line number in the logfile? [03:15] mwhudson: INF [20091118-05:42:04.440] [1084594512] loggerhead.~munin-custom-plugins/munin/munin-plugins-tomcat: Getting information for ChangeLogUI: 3.7736661434173584 secs [03:15] mwhudson: That's the line. I have the log file a bit hacked up at the moment. [03:15] mwhudson: At least, that's where it starts. [03:17] mkanat: the requests are being handled by different threads i think [03:17] mwhudson: Oh, are the numbers in brackets not thread ids? [03:18] INF [20091118-05:42:07.532] [1093282128] loggerhead.~munin-custom-p... [03:18] ^^^^^^^^^^ thread id doing the logging [03:18] mwhudson: Yeah, that's what I thought. [03:18] * mwhudson looks a bit closer [03:18] mwhudson: I know that all those in a row aren't the same thread, but if you highlight the thread id, you'll see that it makes several information requests and doesn't render them all, and it doesn't really do it in order either. [03:19] mwhudson: I can investigate this on my own too, if I'm taking up too much of your time. [03:19] mwhudson: I'm perfectly capable of figuring out everything by digging through the code. :-) [03:19] mkanat: that's pretty whack [03:19] Yeah! [03:19] That's what I'm saying! :-D [03:22] mwhudson: Another hung thread does the same thing. [03:22] 1088797008 [03:23] mkanat: i would _guess_ that this perhaps means that the client is no longer listening [03:23] mwhudson: Yeah, some of those "Getting information" requests take so long, there could be a timeout. [03:23] (we run loggerhead behind apache, which might make a difference here) [03:23] mwhudson: I was just about to ask. [03:23] mkanat: oh right, yes, so you should know this: our ProxyTimeout is 20s i think [03:23] (maybe 30) [03:24] mwhudson: Ah, okay. :-) [03:24] mwhudson: So all these bad threads are getting SIGPIPEd or something. But apparently they keep on living. [03:24] well python ignores sigpipe [03:24] but you should get an error writing to the socket sure [03:24] * mkanat nods. [03:25] (and i think you do, sometimes) [03:25] I've seen it in my own logs, yeah. [03:25] maybe one or more of: paste, apache, python, linux, the universe is screwing us here [03:25] LOL [03:25] Yeah, it's possible. [03:26] mwhudson: So python ignores the broken socket, and the thread then...? I guess I should go look at the code at this point. [03:26] mkanat: if you want to enhance logging, we can get new code into production pretty fast for this <== actually TBH, us losas will fall all over each other to speedily get ANY changes in [03:26] spm: Hahahaha, okay. :-) [03:26] mkanat: what i would have written by now if i had an infinite amount of time would be a stateful log parser [03:27] mwhudson: I could probably write that pretty quickly. That's a Perly sort of job. [03:27] And I've written so many state-machine parsers now... [03:27] mkanat: most request go through a predictable pattern (starting to process, using branch url, ..., rendering) and it would be cool to be able to say how many threads at a particular time were in a particular state [03:27] mwhudson: Oh, that would be interesting for sure. [03:27] and also find threads that don't go through that pattern [03:27] mwhudson: I'll see if I can put together a little script here. [03:28] mkanat: that'd be awesome [03:28] then maybe we can fling existing logs at it rather than having to ship them to you and see if any interesting patterns show up [03:28] wfm [03:29] mwhudson: Well, I don't know how consumable by the external world said parser will be, but if it's going to be so consumed, I will consider that. :-) [03:29] mkanat: we don't have to tell anyone else about it :-) [03:29] lol, okay. [03:30] It'll be easier than parsing text stack traces. :-) [03:30] (That's the last stateful parser I wrote.) [03:31] should be [03:32] mwhudson: BTW, what log is this that I'm getting? Because it looks different than my normal loggerhead logs. [03:33] mkanat: the entry point we use and all the log setup is in the launchpad-loggerhead branch [03:33] mwhudson: Ahh, okay. [03:33] So this will be a launchpad-loggerhead log parser, then. [03:34] Not for the standard loggerhead logs. [03:34] yes [03:34] Works for me. [03:34] some of the goodnees in launchpad-loggerhead should probably be moved to loggerhead [03:34] not sure that the log setup counts as goodness mind... [03:34] I dunnow, these logs are more readable, at least. [03:42] OK, dumb question. I've just made some changes for some documentation, how do I push it to LP so I can ask for a merge review? [03:43] bzr push lp:~username/project/branchname [03:43] * rubbs slaps forehead. should have remembered that. thanks [03:47] Ok, so I've uploaded the branch. And I see where to click for a merge proposal. Is their any sort of standard way of going about this? Should I talk it over with people or just post and let people see it? [03:48] Sorry for all the newbie questions, I've just never done this before. [03:48] rubbs: feel free to just propose it [03:48] rubbs: no, please ask any questions, that's totally fine [03:48] spiv: great thanks! [03:48] rubbs: if you're unsure you can ask here or on the mailing list before proposing [03:49] spiv: great! thanks for being patient. I've always good help from this channel. [03:49] rubbs: but generally a merge proposal is a great venue for a conversation about a branch :) [03:49] spiv: great to know [03:51] Hey folks, I have a working redmine setup on my web server. I am using it to display my repository status. If I have nothing on the server then it's fine to do a push, and everything works... however I can't then redo a push because I get told transport does not support update; and that branches have diverged. How do I handle this? I just want to put up a copy of my repo, at whatever state it's in, with it's revision history. Do I have to delete t [03:53] distatica: you got cut off [03:53] what point? [03:53] "to delete t" [03:54] But you probably don't want a working tree on the server. [03:54] Do I have to delete the directory completely on the server and push again every time?Do I have to delete the directory completely on the server and push again every time? [03:54] The revision history is all present in the .bzr directory. [03:54] then I'm not sure what I want to do, just upload the whole directory via scp? [03:54] If you want to upload a copy of the current state of the tree to a server, the bzr-upload plugin is a good way to do that. [03:54] distatica: just "bzr push" [03:55] spiv: thanks for all your help. I've done a merge proposal. [03:55] igc: thanks for all your help too. [03:55] distatica: and probably "bzr remove-tree" once on the server to get stop the warnings [03:55] spm: Are those log times in UTC? [03:55] that's what I'm using, for the first time, but then I have to login and delete or else my client complains [03:55] hmm [03:56] mkanat: good Q; that server is on BST; but that may == UTC atm. looking.... [03:56] I'm curious about how you created the copy of the branch on the server in the first place though, "bzr push" to a remote host won't create working trees. [03:56] mkanat: yes. atm, that's UTC [03:56] spm: Okay. [03:57] spiv: I use --create-prefix with push, that does it doesn't it? [03:57] No, that's something else. [03:57] That's about whether you can push to a/b/c if the directory a/b doesn't exist yet. [03:58] bzr push --create-prefix sftp://name@host.net/~/public_html/code/myproj [03:58] that's what I use, myproj doesn't exist. [03:58] then in redmine, I point it to there for the repo. [03:58] That seems fine (although if code/ exists then --create-prefix is redundant but harmless) [03:58] code can't exist [03:58] because if it does, I get the errors [04:00] oh interesting [04:00] if I use that full line, it works [04:00] if I just do 'bzr push' it doesn't. [04:00] I thought push just used my last information.. [04:01] It uses the "remembered" location. [04:01] I'm confused [04:01] The first time you push a branch, it'll remember that location. But an already remembered location won't be overwritten unless you specify --remember [04:02] oh, how can I see what I'm attemptign to push to? [04:02] bzr will tell you when it is using a remembered location, it'll say something like "Using saved push location: ..." [04:02] You can also run "bzr info" [04:03] aha [04:03] that's it, pushing to another directory, heh [04:04] oh beautiful, now everything is working. Thank you very much. [04:06] Not a problem. [04:16] rubbs: oh, I almost forgot: please fill out the contributor agreement: . I don't remember if it's a requirement for docs rather than code, but if you don't object then it's probably simplest. [04:33] spiv: to check if ~FOO has signed, are you started at ~FOO and their groups, or at the group we care about? [04:38] lifeless: I look at ~FOO and check their groups, but I also check the wiki page [04:44] mwhudson: I have the parser, now it's just a matter of output and what we do with the parsed input. [04:45] mkanat: cool [04:46] mkanat: well, how about "what state are all the threads in when codebrowse was restarted" ? [04:46] mwhudson: Yeah, that can be done. [05:14] mwhudson: Here's the fully-parsed log that you sent me: http://pastebin.ubuntu.com/322153/ [05:14] mwhudson: It doesn't include lines that don't involve some sort of action. [05:15] mwhudson: It lists threads in reverse time order, but then the actions within the threads in forward time order. [05:16] mwhudson: I did it all by creating objects, so changing the output format should be pretty easy. [05:17] mkanat: cool [05:17] mkanat: my brain has stopped working for the day, btw :) [05:17] mwhudson: Hahahaha, no problem. [05:18] mwhudson: It seems to at least possibly hold up your theory. You can see that a lot of threads built a revision graph cache as one of their last actions. [05:19] mwhudson: Though I suppose those completed, so that doesn't help. :-) [05:19] mkanat: yeah [05:23] mwhudson: Code's here: bzr://bzr.everythingsolved.com/loggerparse/trunk FWIW. [05:24] cool [05:24] i'll look tomorrow :) [05:24] * mwhudson runs away from the computer [05:24] Okay. [05:24] LOL [05:24] night mwhudson! [05:24] Night mwhudson. :-) [05:41] Hey folks, I've been working on a little problem for a bit now, could use some advice. When I redirect from one of my controllers I get the error that headers are sent. I've checked now for whitespace and debug statements and have solved those. I am using Searchable Behavior, and I have various models to be searched (LocalListing, Classified, etc) [05:41] I have one search controller with one action (currently) to handle the request. If the $type and $query vars (and corresponding post vars) are empty I redirect to /search/ this is where my error occurs. I have narrowed this as close as var $uses = array('LocalListing') located in my search controller. I understand that I should include my controllers model name in that array, however I don't have one. Commenting out this statement fixes things (e [05:42] distatica: Are you sure this is the right channel? [05:42] It should be noted, when debug=0 the problems disappear. [05:43] oh goodness [05:43] sorry [05:43] no worries :-) [05:47] Besides lp, are there any publicly-available bzr hosting solutions? [05:48] JohnAlbin, sourceforge does, and I believe savannah as well [05:48] bzr hosting? like launchpad? [05:49] JohnAlbin, and anything that support sftp/ftp can be used (although a bit slower than the smart server) [05:49] depends what 'like launchpad' [05:49] means [05:50] I wasn't really sure, I thought John was looking for a place to host bzr projects. I use sftp+redmine as well, and that works on many web hosts. [05:50] oh geez [05:51] I completely missed the lp, I'm on a role. :/ [05:51] thought it was a nick [05:52] * JohnAlbin is googling savannah and redmine, atm. [05:52] thanks! [05:53] redmine you have to run though, it's not hosted. You need a web host that will let you do a little compiling and supports ruby [05:54] ah, ok. === JohnAlbin_ is now known as JohnAlbin [06:29] spm: Can I get more logs from around restarts? [06:30] spm: It's hard to say that something is a pattern with just two logs. :-) [06:30] But I see a few patterns. [06:30] mkanat: heh. no worries. will do; what are you after; cause the others are possibly quite large? [06:30] spm: Well, mostly I'd just like to see the last five minutes before the hang. [06:31] s/crash/hang/ [06:31] oki; that's easy. will round up to nearest 10 I 'spect; easier on my aching regexified eyes [06:31] That's fine. [06:33] oki, gimme a few and I'll grep that for you [06:34] Okay. :-) [06:36] Also, wow, loggerhead is not coping with all the requests it's getting. [06:36] DEB [20091118-05:42:40.758] [47447192702192] paste.httpserver.ThreadPool: Added task (14 tasks queued) [06:37] harumph. file size == 0. messed something up.... [06:47] spm: If you'd rather have it crash than hang, there's a max_zombie_threads_before_die option for Paste that might help. [06:48] oh man, if you can belt paste into being a bit more sane that would also be good [06:48] (or suggest a better container) [06:48] Yeah, there's also hung_thread_limit. [06:49] paste is really not expecting threads to be cpu bound though [06:49] and the default zombie_thread_timeout is like 1800 seconds [06:49] crash/hang - all equally bad from our perspective. no real preference [06:49] hung_thread_limit might help--it's supposed to be used to kill threads when they do a job for longer than X seconds. [06:49] Though that would screw with building the revision graph. [06:50] mkanat: indeed [06:50] mkanat: we kill threads after 300 s [06:50] ~ish [06:50] 300s of inactivity [06:51] A workaround for the hang might be to set hung_thread_limit to 300s. [06:51] Although you might still get 300s hangs. [06:51] i think we do? [06:52] [mkanat@es-compy loggerhead]$ grep -R hung_thread_limit * [06:52] [mkanat@es-compy loggerhead]$ [06:52] mkanat: not loggerhead [06:52] launchpad-loggerhead [06:52] mwhudson: Ohh. [06:52] mkanat: and in fact we sometimes do recover from the hangs [06:53] mkanat: but far too slowly [06:53] way too slowly [06:53] Ahh, okay. [06:53] and usually die "harder" within the next 30 mins or so fwiw. unscientific observation [06:53] mkanat: i had this vague idea that you could push off the revcache generation, not just caching to another process [06:54] then you could just kill threads after 30s and be fine with that [06:54] mwhudson: That would be a good workaround, yeah. [06:55] mwhudson: What's weird is that everything gets slow as soon as the revcache generation starts, and then it's all slow after that, even though they've been built. [06:55] I'm mostly just musing aloud, I'm still investigating. [06:55] mkanat: hm [06:56] Hmm… looks like SourceForge is on 1.10. Guess I'll use lp. :-) [06:57] mwhudson: It's only RevisionUI that builds the graph, right? [06:57] mkanat: no [06:57] basically any branch access will [06:57] mkanat: look at history,py [07:00] mwhudson: Thanks. [07:12] * spiv is done for the day [07:13] g'night spiv! [07:22] What I really wish I could get is a traceback of every thread when it's hung. [07:23] that... that's achievable. gdb attach dump goodness I assume? [07:23] spm: Well, it's in Python, so it won't be quite the same. [07:24]