=== medberry is now known as med_out === AuroraBorealis is now known as aurora|afw === med_out is now known as medberry === medberry is now known as med_out === yofel_ is now known as yofel [07:00] hi all ! [07:01] hi there, nice to see you! [07:01] i'm just reviewing your possible_transports patch [07:02] oh great ! [07:04] done [07:04] wow look at all these patches [07:05] morning! [07:05] hi mgz [07:07] mgz: _o/ [07:09] poolie: 'feel free to land with post review' ? I can't parse that, especially the 'with post review' part ;) [07:09] ah i mean [07:09] you don't need to wait for review [07:09] oh [07:09] i do appreciate you making an mp, so people can see the diff [07:10] yeah, I intended to deploy it but wanted to chit-chat with jml... can't remember exactly why though :) [07:13] ha, good analyze-log script :) 22h20 is how long it takes to catchup the imports for chromium-browser (still failing though) [07:17] ha crap, jml refactoring broke *all* scripts, the modules are not found anymore [07:17] yeah i see [07:17] yeah, I wondered about that yesterday when trying to run the importer locally [07:17] but i thought they were all already on jubany [07:18] i checked it was up to date with trunk the other day [07:18] thinking he was the one infecting me with TDD ;) [07:18] but perhaps they had not yet been merged [07:18] moving the script out of the base dir means "udd" is no longer on the path [07:18] so you can't run without installing (or setting PYTHONPATH or something) [07:18] mgz: exactly but not only, jquery.flot.js not found too [07:19] I just created bin/udd -> ../udd but that's not enough [07:19] (not to mentioned hackish ;) [07:19] ehee, that's even worse than most symlink hacks :) [07:21] ok, I'll revert it and comment on the mp, too much to fix without guarantee I get them all [07:21] could revert that change for now, and only land it w... [07:21] ...hen there's a setup.py script to go with it [07:21] :) [07:21] oh, setup.py, worth thinking about it [07:22] but we run from source and it's also a useful thing [07:23] bah, let's try a bit harder, reverting is not trivial either and is going in the wrong direction anyway [07:23] vila, mgz, i think it's just us this week so we could talk whenever you like [07:24] comfy :) [07:24] Riddell is not here either ? [07:33] i might be confused but i thought he was going to be en route to uds this week? [07:33] hmm, by boat ? :-D [07:34] exactly, kayak [07:34] wow [07:34] not really [07:36] hehe [07:39] ok, some sanity restored, categorise-failures works [07:41] etc scripts 101: you can't remove PATH or you won't do a lot [08:31] gz you're in ~bzr-explorer-dev now [08:47] poolie: cool, will land that branch [08:47] feel free to also add a button to lp! :) [08:48] i'm seriously tempted to do it myself [08:48] vila: so, if you could walk me through what to do to deploy the latest bzr-builddeb in jubany and requeue one of the failing packages some time today [08:49] mgz: sure thing ! [08:49] poolie: but doing it manually is such fun... [08:49] I've just fixed the last glitch and the importer is running again [08:50] I'm waiting a bit before calling it a win ;) [08:50] ... and stopping it before requeuing chromium-brower... [08:51] so, let's deploy builddeb as soon as you're ready [08:51] I'm guessing the procedure is basically stop importer, pull, start, then... tell it what package somehow? [08:51] yup [08:51] so, first thing, read README_DEPLOYMENT to make sure it's accurate [08:52] which lives...? [08:52] in lp:udd :) [08:52] with a symlink also in place in pkg_import@jubany:/srv/package-import.canonical.com/new/scripts [08:53] okay, that tells me the dir. [08:53] mgz: you know how to connect there ? [08:53] yep, I'm there [08:53] via ssh + sudo su - pkg_import ? [08:53] k [08:54] hmm './' is probably missing in front of the etc-init.d-mass-import examples [08:54] can you confirm that ? [08:54] yup. [08:55] ok, fixed locally (I'll take the notes, you do the work ;) [08:55] and in fact, when I stopped it last time, used the script in /etc/init.d/ [08:56] we should resync them but better use the one in the branch [08:56] which is I guess just the same thing, or an older copy, or a symlink [08:56] an older copy [08:56] you haven't stopped it yet right ? [08:56] yup, has the wrong path [08:56] use graceful-stop [08:56] not stopped yet. [08:56] k [08:56] doing now [08:57] yup seeing it [08:57] oh, I should mention that, [08:58] how do you monitor a running importer? [08:58] I usually have 3 connections to jubany: [08:58] oh failure ! [08:59] crap dpkg-mergechanglogs is not in PATH anymore [08:59] stopped nevertheless [09:00] there's an old import_package.pyc and .py~1~ in here for some reason... [09:00] mv dpkg-mergechangelogs bin/ should take care of that but we need a more formal way to track it [09:00] someone did a revert probably [09:00] back to monitoring: [09:01] tail some log? [09:01] 1) tail -F logs/driver/progress_log [09:02] 2) crap, broken now, just a sec [09:03] 2) export PYTHONPATH=${HOME}/src/udd/trunk ; (ssh -AY jubany.canonical.com tail -F /srv/package-import.canonical.com/new/logs/driver/progress_log) | ~/src/udd/trunk/bin/analyze-log - [09:04] 3) in .../scripts: '. ./fixit.sh' to get some shorcuts and setup the env [09:04] okay. [09:05] so, is lp:udd in a usable state now? should I pull in the latest bzr-builddeb? [09:06] lp:udd is deployed, so, yes, pull the latest builddeb, check that no uncommitted changes are there though [09:06] will do. [09:07] one shelve. :) [09:07] yeah, looking at it ;) [09:07] it's an old one, should be safe to delete but doesn't hurt to keep it until we fix the dpkg-mergechangelog for good [09:08] okay, that's got the fix in question [09:08] start time? [09:08] yup [09:10] started ok [09:10] notice that it says ' Read 4 in /srv/package-import.canonical.com/new/max_threads' [09:11] I changed from the default 8 while debugging the scripts issues [09:11] so, how are you running the scripts in bin? I need to do requeue-package next [09:11] ^right [09:11] I use '. ./fixit.sh' first [09:11] k [09:11] it provides 'requeue' as a shorcut to requeue-package [09:12] done. [09:12] failure [09:12] I see it. [09:12] run 'categorise-failures' to force the status page generation [09:13] stop importer first? [09:13] yup [09:13] err [09:13] no they are independent [09:13] it's useful when you don't want to wait for the crontab to run it [09:14] every 5 minutes (see importer.crontab in lp:udd) [09:14] done [09:14] all looks like the same issue in import_dsc [09:14] but you can stop the importer too as the failure seem to spread [09:15] you'll be able to requeue all the related failures by doing 'requeue --all ' [09:15] good thing I didn't requeue chromium :) [09:16] one sucessful import still ;) [09:16] looking for blame. [09:16] hehe [09:17] now, the usual choice is to either revert to a known-working rev and restart or fix before restarting [09:18] it's ok to fix locally and only then do a merge proposal because the importer is stopped and we don't want to accumulate too much stuff [09:18] probably in lp:udd [09:22] hm, nope, the singular->plural was a builddeb change [09:23] vila: bzr diff to see possible fix [09:24] +1 [09:24] starting [09:24] will let a few go past before requeuing [09:24] oh, I couldn't commit on jubany last time I tried and didn't investigate a workaround (just so you won't be surprised) [09:25] nope, try requeuing one that failed instead [09:25] with: [09:25] requeue --priority paraview [09:25] --priority will ensure it's run asap [09:27] done [09:27] if you run categorize-failures now it should appear in outstanding jobs [09:28] run, but, where am I looking? [09:29] oh, that was related to multiple tarballs, we should check with jelmer telling him we've deployed that [09:29] which one did you requeue ? [09:29] paraview, was the last one in the log [09:29] oh, sorry, that generates the status page [09:29] currently pending [09:30] running already [09:30] I must be being blind then, didn't see current jobs on status [09:30] *running, even, yeah [09:30] top of the page [09:30] 4 running [09:30] 30 outstanding jobs [09:31] and in the last 50 failures you've got the ones related to extract_upstream_tree [09:31] ...I'm now not sure if I needed that refresh or if I was just being blind [09:31] go with refresh ;) [09:31] tail is looking happy. [09:31] the fact that this page also refreshes itself from time to time doesn't help ;) [09:32] will let paraview go through, then requeue the one I wanted and all those that hit that issue [09:32] yup, that's the idea [09:33] you'll requeue them with: 'requeue --all polyglot' [09:33] adding --priority as you see fit [09:33] --all will requeue all packages that share the same failure signature [09:34] ok, i should go, good night [09:34] night poolie! [09:34] poolie: g'night ! [09:34] paraview done. [09:36] \o/ [09:36] okay, everything set up I think. [09:36] bump max_threads to 8 (echo 8 >../max_threads) [09:36] done. [09:38] categorize-failures again to see the effect (or look in the logs) [09:38] yeah, following log [09:38] k [09:39] I'll requeue the ones related to dpkg-merge... [09:40] ...and I'll requeue chromium-browser once the outstanding jobs are (will be ?) done [09:40] ha great lxc fails [09:40] ...lock contention on lxc? [09:40] did you see that ? [09:40] yup [09:41] ...great as in, really was good, or great as in, annoying? [09:41] in some cases (for yet unknown reasons) an exception manage to escape the finally clause where the lock is released [09:42] great to explain the issue and how I resolve it (hackish, needs to investigate one day and make it more robust, should probably file a bug) [09:42] hey, I think my test package went through okay while I was distracted [09:42] how do we see sucesses? (...apart from in the log) [09:42] to resolve: look at the traceback, it gaves hints about where the lock is still pending, in this case 'precise' [09:43] you don't, it's a bug I'm working on, let me find the number [09:43] bug #831699 [09:43] Launchpad bug 831699 in Ubuntu Distributed Development "no report/log of successful packages" [High,In progress] https://launchpad.net/bugs/831699 [09:44] so, cd ../updates/lxc/precise [09:44] pygobject also had an unhappy lock experience [09:44] yup, same issue [09:44] bzr info -v [09:44] it's a local lock so 'bzr breack-lock' [09:44] okay, I'll requeue all the other UnicodeDecodeError packages, and hopefully a couple of them will fail with new nicer errors [09:45] two locks broken [09:45] sometimes it's a remote one and I do 'bzr break-lock :push' [09:45] libproxy is a different issue [09:46] two broken locks for pygobject too [09:46] and requeued [09:47] rats another failure ;-/ [09:48] * vila blinks, that's an unknown one [09:48] all unicodedecodeerror packages queued [09:48] will need to repro locally [09:48] can you land your fix on lp:udd ? [09:49] 51, 10, 5, 3 [09:50] ^will do [09:50] hmm, the lxc failure spreads [09:51] something changed in the merge hook [09:51] yeah seems like it [09:52] probably worth stopping the importer when all your jobs have passed [09:53] huh, it's doing paraview again [09:53] ha [09:54] there is a bug there (not filed), some jobs seem to be re-processed for no good reasons [09:54] oh, probably because I did requeue --all on a similarly failing package after it had succeeded? [09:54] nope, it shouldn't it's more obscure than that [09:55] I've seen jobs being re-processed without being involved in anything recent [10:02] mgz: mwhahaha, conflicts in bzr-builddeb locally [10:02] mgz: import pdb; pdb.set_trace() in bzrtools_import.py getmembers :) [10:03] hehe [10:05] I went there trying to diagnose some unicode issue at one point ;) I should have tried harder ;) [10:07] mgz: did you note the revno before upgrading builddeb on jubany ? [10:08] should I just push to lp:udd? [10:08] vila: ...no, but I think you told me it yesterday [10:08] r613? [10:08] you rock ! [10:09] yup 613 [10:11] hm, a bunch of these current packages aren't doing anything new [10:12] maybe should stop, fix the merge hook issue, then see if I can get some non-utf8 filename failures [10:21] mgz: yup, I'm trying to reproduce locally with lxc buno success so far [10:21] s/buno/but no/ [10:22] mgz: you just got one ! [10:22] 2 even :) [10:23] make that 3 [10:23] of, what's you trick ? ;-P [10:23] s/of/ok/ [10:23] s/you/your/ :-( [10:24] woho! [10:24] bah, can't reproduce locally as it's code involved when pushing only ;_/ [10:24] what the hell paraview is going through for like the fourth time [10:25] yeah and outstanding jobs up to 120 (was down to 70) [10:26] so, categorize_failures requeue some but I suspect some cruft in the db [10:26] so, these will be packages with real filename encoding issues: [10:26] real as in ? [10:26] as in, not utf-8 [10:26] will never be able to encode ? [10:27] most of the ~70 failures should now pass [10:27] and the subset that don't will be listed there [10:28] basically, bzr core needs a strategy for handling that edge case, so the package importer failing is reasonable [10:29] the only other option would be to have a way of attaching a different encoding to the package metadata if they're using a valid, but non-utf-8, encoding [10:29] but I suspect that won't be worthwhile [10:29] ..and may not be enough (I suspect this should be attached to each file in some cases) [10:30] libperl5i-perl also enjoys being processed [10:30] qcomicbook too [10:31] eerk and libjgrapht0.6-java enve more ! [10:31] ok, so, this issue is now really bad [10:32] I have run categorise-failures several times, will stop doing that [10:32] down to 69 outstanding jobs, don't run.. yeah me too, let's stop :) [10:33] there are some basic tests for db now and one FIXME is mentioning something related [10:33] * vila back to the merge hook issue [10:36] ok, I think I got it [10:41] mgz: two bugs: dpkg-merchangelogs was not in PATH anymore, the merge hook can't find it but returns a single value instead of 2 [10:41] mgz: we can stop the importer ;) [10:41] DONE [10:42] what happens to the current queue when it's stopped? [10:42] it's in the db (meta.db JOBS_TABLE) [10:43] cool. [10:43] 4 more failures for you :) [10:43] err 3 [10:44] yep, seeing them [10:44] perhaps a latin-1 fallback in the importer wouldn't be so bad. [10:45] yeah, add an NFD too ;) [10:45] I forgot which package, but I'm pretty sure there is one with a contrib/docs/xxxx.pdf where xxx is in NFD [10:45] heh [10:46] some db related one [10:46] aaaa, that error message is so much nicer [10:46] reminds me, I should improve UnicodeError display is bzrlib [10:47] mgz: bzr diff on builddeb for the fix [10:47] have we got a bug for it? we've talked about it before [10:47] mgz: oh yes please ! [10:47] I don't think so, [10:47] but there should a place where all unicode exceptions can be trapped their *args* put to good use [10:48] i.e. the broken string is args[0] IIRC [10:48] but it's not displayed by default (strange idea) [10:48] diff looks reasonable, is this the right end of the interface to change? [10:49] I'm 90% sure as it's a recent change and match the error we see [10:50] yep, looks more unambiguously correct with more context [10:50] goforit [10:53] hmpf, it was still running :-} [10:54] the last job was being slow :) [10:57] yeah, chromium would have been even slower :) [10:57] openclipart... I fear a little for the contents of that branch [10:57] oh crap, I pushed to lp:bzr-builddeb :-( [10:58] quick, uncommit! :) [10:58] jelmer: so sorry about that :-/ [10:58] jelmer: the fix is trivial though, shouldn't be a problem ? [10:58] ETOOMANYBRANCHES [10:59] mgz: anyway, in this case, 4 fourth terminal with: 'tail -F logs/packages/openclipart' :) [11:00] paraview is in the queue again... [11:00] * mgz switches tails [11:06] * vila switches to lunch ;) [11:07] mgz: this import looks like it will take some time to finish anyway [11:07] :) [11:08] I'll restart if I notice it's done before you return [11:08] mgz: I have some shopping to do after lunch, I'll ping you when I'm back, feel free to play around as you like (my fix needs to be deployed first though) [11:08] k [11:10] ...bzr-builddeb just needs to be pulled, right? [11:14] taking care about the pending change yes [11:50] vila: all done, also requeued the packages with affected by the tuple unpacking issue [11:50] mgz: yup, still there for a couple of minutes, they fail with lockcontention :-0 [11:50] ...but they seem to now all be failing with the lock... [11:50] probably time to find a better way to addres that :) [11:52] I'll look into it when I get back [12:08] delayed again :) [12:23] huho [12:24] mgz: panic in the driver we may need to kill hard [12:25] woha! [12:26] ha, normalisation kills the world [12:27] did a hard stop [12:27] mee too :-/ But it seems I'm still receiving output from tail -F [12:28] there was a *lot* [12:28] it's basically in infinte log loop [12:29] bug in mass-import then ? [12:29] printing the failure triggered a complaint about normalisation caused a failure... [12:29] yeah [12:29] bad bad bad [12:30] good we encountered it while online [12:30] was going well till that, a bunch of packages imported, and a bunch more failed with useful errors [12:31] the invalid normalisation error class in bzr should really use repr [12:31] 326570836 is a lot of bytes of log [12:32] manually truncating that a bit wouldn't hurt [12:32] * mgz logs out of jubany for a short food break [12:35] no, please leave it there as evidence for forensics, it will be log'rotated when needed anyway [12:35] I'm already rsync'ing it [12:36] we had cases like that long ago, it's not an issue in itself [12:57] Hi all === yofel_ is now known as yofel [12:58] Is there a hook triggering once a checkout is done completely ? [12:58] I have tried post_pull, post_branch_change, but none of them is triggered after the .bzr/checkout directory is made [13:03] rom1: I don't think there is such a hook yet, patches or bug welcome ! [13:03] that's what i thought : thx vila ! [13:04] mgz: http://paste.ubuntu.com/718773/ should fix the issue, weird that we never encountered the case though [13:05] vila: the replace is redundant [13:05] otherwise looks about right [13:05] meh ? The one I delete ? [13:06] * mgz bops self on head [13:06] bah, sorry you don't have the context here [13:06] yes, the code you're deleting is wrong [13:08] that it triggered a loop though is still a bit concerning === med_out is now known as medberry [13:09] mgz: look at the method itself, this was the last place we use unicode_output [13:09] mgz: I suspect some similar case happened in the paste leading to the introduction of ascii_output to avoid the problem [13:10] s/paste/past/ :) [13:10] re-starting with max_threads = 2 [13:10] yeah, seen [13:10] hu ho [13:13] dandling scripts/main_lock, never see that before [13:13] wow, we'll now soon enough, praat requeued [13:15] hmm, otrs2 and console-tools didn't finish, they are still running [13:17] one more trick to monitor: 5) run top ordered by MEM used [13:18] that's how I found back otrs2 and console-tools (you need to add cmdline to the displayed columns) [13:18] those two orphan imports... [13:19] vila: sorry [13:19] ... will finish... eventually... but won't be marked as such in the db ??? May be another cause of those random imports coming out of sequence [13:20] jml: hehe, too soon to be sorry, join the fun fixing the fallouts ;-p [13:20] jml: kidding, I think we nailed them down [13:20] vila: cool. thanks. [13:21] jml: your way to remind me why you TDD-infected me is naughty though ;-D [13:23] mgz: in the mean time, the importer fail to start the already running imports so we should be safe... [13:23] bbiab [13:24] okay, first one back can try again [13:24] * vila nods [13:25] vila: :) [13:25] ^I do 5) all the time on this box :) [13:31] can anyone help me with bzr-git failing? http://paste.pocoo.org/show/497924/ [13:34] seems like https://bugs.launchpad.net/bzr-git/+bug/706990 but that should be fixed in bzr-git I have [13:34] Ubuntu bug 706990 in Bazaar Git Plugin "git-import fails on remote repository with AttributeError: 'RemoteGitRepository' object has no attribute '_git'" [Critical,Fix released] [13:38] ccxCZ: the traceback *is* different [13:39] ccxCZ: bug 849250 [13:39] Launchpad bug 849250 in Bazaar Git Plugin "pull into remote git branch fails" [Low,Fix committed] https://launchpad.net/bugs/849250 [13:39] you need 0.6.3 [13:39] seems if I just do branch, it works [13:40] ...which I'm not sure when jelmer plans to release [13:41] +affectsmetoo that bug at least. [13:41] okay, now I need to get going. [13:43] seems eadd fails consistently though [13:46] doesn't seem to be the same bug to me [13:47] ccxCZ: the line in question with the bug is that one [13:49] yeah, in the end it ends up in the same function, failing on the same row, should I post my traceback there, or just +1 it? [13:52] does lp allow some formatting? I guess the traceback should at least be monospaced === zyga is now known as zyga-afk [14:15] mgz, would you have a minute to review https://code.launchpad.net/~james-w/udd/config-location/+merge/80337 ? [14:15] vila, you've experience related to ^ I think? === aurora|afw is now known as AuroraBorealis [15:34] vila, https://code.launchpad.net/~james-w/udd/base_dir-config/+merge/80350 removes one of your FIXMEs if you would take a look === zyga-afk is now known as zyga [15:40] Mcough> finally back :-} [15:47] gra, that was annoying [15:47] will have a look at those udd branches unless vila gets there first [15:52] james_w: I won't be able to look at them today, but probably tomorrow [15:53]