[11:08] * gmb -> lunch [12:08] bac benji frankban gmb call in 2 [12:09] I'll start hangout [12:09] k [12:10] bac benji frankban gmb https://plus.google.com/hangouts/_/900b5fed14faeb2a94546bafa8ba4a9da566e8f4 [12:40] gary_poster: i've looked and cannot find the email from robert you mentioned. to whom was it sent? [12:41] looking [12:45] bac, it was sent to yellow. I wonder if it didn't make it through the list? I just forwarded it to you [12:48] bac, benji, frankban: Hangout for redirection stuff is here: https://plus.google.com/hangouts/_/8a02312c16f83738dc5c4993c6fc01d3723cf824 [12:50] gmb: I doubt we need four people on it quite yet, so I'll let frankban have the pleasure [12:50] Righto. [13:13] gary_poster: my ec2 instance is up and just started a build [13:13] cool, thanks bac [13:49] gary_poster: lots of failures related to non-existent /var/tmp/bazaar.launchpad.dev/mirrors. [14:05] bac :-( [14:05] bac that's the make clean thing, right? [14:07] gary_poster: yeah [14:08] bac, I assume you are investigating; lemme know when you discover anything, and if you want another investigator [14:08] it was my understanding that the codehosting tests created those directories on demand but that doesn't look to be true [14:08] ok [14:08] bac, they are at least not owned by root now, right? [14:09] because the parent directory is maintained [14:09] make inplace might be sufficient [14:09] rather than make build [14:09] gary_poster: don't know. is it possible to gain access to the temporary lptest containers? [14:10] nm, they are gone now [14:10] bac, you can look at them while they are running. However, IME the problem is not in the temporary containers but in the root container. In any case, the solution to both situations is the same: modify /etc/shadow in the base container before the ephemeral containers start, and then you can access all of them [14:10] great [14:54] gary_poster, good news. the lp-setup build recipe was not run so the testing happened with the old build script. i've requested a build now. [14:54] bac, ah, excellent [14:54] frankban, any reason the lpsetup build recipe cannot be set to automatically build? [14:54] gary_poster, will be able to retry in 15 minutes [14:54] bac: you mean daily build? [14:54] fantastic [14:55] frankban, yes [14:55] it is 'build on request' now [14:59] bac, I usually manually build lpsetup when I know the current revision works correctly. This lets us commit incremental changes whitout breaking the parallel testing story. [16:18] ok, frankban, I spent too much time on this, but I had to think through some things. [16:18] https://dev.launchpad.net/yellow/lp-setup-lep-draft#preview [16:19] Note that I'm suggesting that everything in "Thoughts" right now be shuffled out onto separate pages [16:19] linked from the main one [16:19] We should not delete Robert's input, for instance, but I think it is distracting if we have already tried to accommodate it [16:20] And similarly, a lot of your notes verge more into implementation, and the LEP is supposed to be about goals only [16:20] unfortunately we're leaning a bit towards design still with what I wrote [16:20] It should arguably be cleaned up even more [16:20] but I'm tired of it :-P [16:20] and I need to start working on something else [16:21] :-), thanks gary_poster [16:21] :-) welcome [16:22] gary_poster: I will update the real LEP and make a separate page for thoughts tomorrow morning if you agree [16:22] frankban, +1 [16:22] frankban, and please feel free to make changes on what I did [16:31] gary_poster or benji, since you already know the fix, could please take a look at https://code.launchpad.net/~frankban/launchpad/bug-1003040/+merge/108005 ? [16:31] * benji looks [16:31] * gary_poster therefore stops looking :-) [16:32] * gary_poster lied. but now is really stopping. [16:32] heh [16:33] bac, how's buildbot looking? any chance of a run? [16:33] the biweekly summary will be much happier with a successful test run at the end of it. :-) [16:36] frankban: approved [16:38] benji: thank you [16:38] np [16:41] gary_poster, not just yet [16:49] bac, are we still waiting on the lpsetup build? [16:50] gary_poster, no it built and is being used but i'm still seeing that dir owned by root [16:50] bac :-( ok [16:50] thank you for working on it [16:51] * gary_poster needs some lunch [16:51] it may be the branch rewrite script in play [16:51] will be back in a bit [16:51] ack [17:20] gary_poster, waiting for a run to finish so i can try again. even though i installed the new lpsetup package i'd forgotten that it doesn't update the scripts in /usr/local/bin. they get installed when lpsetup is run not installed. thus,my previous test was still using the old code. [17:36] oh ok, good news for some definition of good news [17:43] gary_poster, tests running. the directories are correct [17:43] great, bac [18:16] gary_poster, still running. 10,500 test done w/4 fail & 3 errs [18:16] bac, could you give me url? [18:16] sure [18:17] http://ec2-23-22-27-123.compute-1.amazonaws.com:8010/waterfall [18:17] ty [18:21] bac, interesting. the failures are instances of the tests that have the same name, did you notice? [18:21] the first set of three passed on lpbuildbot; checking the second. [18:21] yeah, the second set of two passed also [18:22] and for the third set... [18:23] lp.testing.layers.RabbitMQLayer:setUp failed with a timeout again too. [18:23] So, mostly new errors. [18:27] I'm trying them locally [18:31] i hadn't noticed that gary_poster [18:46] gary_poster, i did not apply the stub patch before running the tests. could it be the cause of the db-related ProgrammingError failures? [18:46] no bac, that's a different error [18:47] (a stemming thing) [18:50] bac, or anyone else who might know, my db is not initializing properly in my local lxc [18:50] I'm getting this error: [18:50] $ sudo utilities/launchpad-database-setup gary [18:50] Creating Launchpad database for gary [18:50] grep: /etc/postgresql/9.1/main/start.conf: No such file or directory [18:51] then it goes further south from there [18:51] have you seen that, and know the cause? [18:51] I've done various things [18:51] dpkg-reconfigure posgresql-9.1 [18:52] dpkg --search /etc/postgresql/9.1/main/start.conf [18:52] (that says that no package installed it :-( ) [18:52] gary_poster, i'm not sure who creates that file [18:52] but it only has one non-commented line: [18:52] auto [18:53] um ok [18:54] it looks like launchpad-database-setup creates it [18:54] oh [18:54] bac, that's what I was running :-P [18:55] but I tried making the file... [18:55] oh, right, it checks for it [18:55] seemed to work [18:55] i'm curious if other things are missing, though [18:55] then make schema seems to work but complains, menacingly: [18:55] * Loading base database schema [18:55] psql:launchpad-2209-00-0.sql:17: ERROR: language "plpgsql" already exists [18:55] psql:launchpad-2209-00-0.sql:2646: ERROR: could not access file "$libdir/plpython": No such file or directory [18:56] seeing if test runs now [18:56] it did [18:57] and passed [18:57] so I have no freaking idea what the problem is :-/ [18:57] ugh, this will be our first not-so-good biweekly report :-/ [18:57] oh well, getting to it... [18:58] thank you bac, your help got me past the hurdle [19:01] bac, do you mind if I start another run on that machine to see if the errors are reliable or not? [19:02] nope [19:02] i may fire up another just so i can see lpsetup working without intervention [19:17] gary_poster: are we not having the project status meeting today? [19:17] benji, we don't have a time set up, now [19:18] bac, I'll leave this one for your work then [19:18] it's not worth it [19:18] to hinder you [19:19] as long as you can run one more test run today, bac? [19:19] ok [19:19] gary_poster, but they can be done independently if you want to use this instance [19:23] bac, oh ok [19:23] will do thx [19:23] started [19:56] gary_poster, i was wrong, juju only lets you spin up one env. let me know when you're done with the instance so i can tear it down [19:57] bac oh ok [19:57] i see it is 1/2 don [19:57] done [19:57] right [19:58] bac, could you get a copy of the failures, and maybe a copy of the pertinent worker logs, from the previous run? looks like it will be the same [19:58] that permission denied is from the db is so weird [19:58] once the tests fail the same way and you have those files, that will be done [19:59] ok [20:42] bac, do you remember off-hand how long the timeout was that you set up for RabbitMQ https://code.launchpad.net/~benji/launchpad/bug-1002820/+merge/106851 bug 1002820 [20:42] <_mup_> Bug #1002820: lp.testing.layers.RabbitMQLayer:setUp times out rarely/intermittently in parallel tests

< https://launchpad.net/bugs/1002820 > [20:43] (It was either not long enough or we have to find another solution, given what we saw today) [20:44] let me look [20:45] thank you [20:45] gary_poster, 90 seconds [20:46] bac, wow. ok thanks. I'll update the bug and move the card back. [20:46] I wouldn't mind trying 300 myself [20:46] but perhaps cooler heads should prevail [20:49] there are two timeouts. the first one is 90 seconds (up from 15) [20:49] oh! [20:49] the second one is the max of whats left from the 90 or 60 (up from 5) [20:49] huh [20:49] yeah, a little overly complicated