[01:03] morning all [01:06] morning axw :) [01:06] well github is rather verbose [01:06] how's bug land? [01:07] not bad, since I haven't been looking at the horrible mongo-related bugs lately [01:13] ok guys, EOD [01:26] wallyworld: got time to chat? [01:27] sure [01:27] wallyworld: https://plus.google.com/hangouts/_/gw7us4q4lhcyws5ksu7ipthn6ia?hl=en [02:13] wallyworld: there is already a canonistack slave [02:13] oh [02:13] but... running tests on there may interfere with CI [02:14] yeah [02:16] I'm going to try adding another one in that we can use [02:21] axw: the current script used by jenkins for the github landing fails with an unexpected EOF. i can't spot the issue, here's a pastebin - it's the last bit that's bad. can you see the error? http://pastebin.ubuntu.com/7577197/ [02:23] wallyworld: nothing jumps out [02:24] ok, ta [02:26] wallyworld: the "}" needs to be preceded by a newline or semicolon [02:26] ok, will try that [02:27] \o/ thanks, that was it [02:29] nps [02:38] wallyworld, mgz, canonistack is under resources, our aws account is reaching saturation. joyent and hp cloud have lots of cpus/instances for us to test/build with [02:39] * sinzui will look to move some function tests and series tests to hp this week [02:39] sinzui: ok, we may need to try hp cloud. i'm testing now with a nailed up ec2 instance i was using for mongo testing in order to save time [02:39] wallyworld, mgz, will the lander be running unit tests for more than 1 series [02:39] ? [02:40] sinzui: not initially since we are just going trunk at first, but it will [02:40] wallyworld, I need to teach the default run-unit-tests about nova. At this time we can run ec2 or any existing host. Teaching the setup about joyen would be goos too [02:41] oh, and azure was saturated last December, not that you wanted to wait 25 minutes to bring up a instance [02:41] sinzui: at the moment, the unit tests the lander runs fail every time using an ephemeral ec2 instance [02:42] using a nailed up instance seems better [02:42] wallyworld, bad script, unbound variable [02:42] i have a couple of tests fail due to build errors but everything else passes [02:43] thats the problem with cargo culting a script, you loose the context of what the required setup is [02:43] wallyworld, I am adding set -eux to make it easier to debug [02:44] ok [02:46] wallyworld, I think you are going to get a pass this round [02:46] i hope so [02:47] well, except for the build error [02:47] sinzui: did you change the test-merge-git script? [02:48] wallyworld, no, I think someone fixed the unbound var from run #2 [02:48] wallyworld, sorry, a failure [02:49] oh, is this trusty? [02:49] yeah but at least it was due to a build failure [02:49] i think so [02:50] this is one of the common intermittent bugs. [02:50] oh, i just saw the replicaset failure also [02:50] maybe we want to call make check a few times to replay intermittent faules [02:51] make check || make check || make check [02:51] ^ three tries [02:51] on canonistack with tarmac, we seem to have eliminated most of the failures [02:51] seems more flakey on ec2 [02:51] but yes, i guess we can do 3 retries [02:51] wallyworld, This test always gets current packages before starting [02:52] and I think you were using precise which had fewer changes [02:52] wallyworld, may, I make some quick changes to remove the unsed parts of the script and make it retry? [02:52] yes please [02:53] we only have a small window left to get this working [02:53] it will still fail due to the build errors though [02:54] looks like the import pulled in a deleted package [02:54] cmd/jujuc [02:59] sinzui: those obsolete files which cause the compile errors are now gone. [02:59] okay [02:59] are you finished with the script? [02:59] can we start another run? [03:01] thumper: ping [03:02] waigani: in the middle of something right now [03:02] thumper: okay, np [03:12] wallyworld, I see this script was restarted. I just save my changes. I removed the non-ephemeral conditions because this use is always ephemeral. I removed the lines that built tools. I need to do the same to the real scripts because we now do a proper isolated build for each test deb and tool [03:13] sinzui: i restarted just to get a ahead start on whether my build fixes were ok, i'll re-run wth your changes also [03:14] okay [03:14] sinzui: my main concern is that now the landing process takes a fair bit longer [03:14] I just discovered that I cannot build juju packages and run unit tests on the same machine [03:14] wallyworld, yep [03:15] so i'm thinking we do want to move to a nailed up instance at some point post cutover [03:16] you can make it go much faster with provisioned instance or one that you start and stop. The deps phases of the script are essentially skipped [03:17] yep [03:17] i think having one that we start and stop will be best [03:17] I am pondering having a jenkins slave for each crucial series+arch, and using the instance as a dedicated builder and running of local-provider tests [03:18] sounds reasonable [03:25] sinzui: yay, that run worked [03:25] i'll try another with your changed script active [03:26] excellent, now we get to see if I can hack a bash script without syntax highlighting [03:28] wallyworld: it says #18 passed, but in the logs the replicaset package's tests failed [03:28] oops [03:28] wrong tab :) [03:28] phew [03:28] wallyworld: why did the PR not get updated? [03:29] it should have i think, not sure [03:44] * thumper grumbles [03:51] * thumper makes unpleasent noises [03:51] I've worked out why this test is failing [03:52] it is because JujuConnSuite bootstraps the environment for you [03:52] and I've changed the base test suite for this test [03:52] to something much simpler, [03:52] hence, broken test [03:52] now how to just write out some jenv stuff... [03:53] wallyworld: I was waiting for a LGTM, then I would do $$merge$$ - that's still the protocol right? [03:53] axw: yeah, i'm experimenting trying to get the bot to pick it up [03:53] the current core will be blown away and replaced with the trunk snapshot [03:53] ok [03:53] axw, wallyworld: will we use codereview at all anymore? [03:54] waigani: no [03:54] nope :-D [03:54] and good riddence [03:54] big changes, okay [03:54] wallyworld: the lander didn't pick up the $$merge$$ on your other PR? [03:55] axw: nope :-( [03:55] there's something wrong clearly [03:55] man git is spamming my inbox! [03:55] there's a cron job, but i also ran it by hand [03:55] wallyworld: are you running it on your machine now? [03:55] mk [03:55] axw: nope, i've sshed into jenkins [03:59] wallyworld: your membership in the juju org needs to be made public [03:59] ok, i didn't realise it wasn't [04:01] ah, that has fixed it === lazypower-travel is now known as lazyPower [04:01] wallyworld: I'll reply to the list about that [04:01] axw: that's a good pickup because a bunch of others are also private [04:01] yup [04:08] axw: so now with the ephemeral instances we waste about 6-7m getting everything set up to run the tests. not great, but not a show stopper for cut over [04:09] wallyworld: yeah doesn't seem too terrible for a first cut [04:09] axw: the thing now is the extra occurrences of these fraking timeout errors etc which break the tests [04:10] yep... :/ [04:10] just got one now in the current run [04:10] i could have sworn that it's worse on ec2 [04:13] wallyworld: I wonder if pre-warming the disk would be of benefit [04:13] * axw looks into what we can do there [04:13] axw: yeah, i was wondering that also [04:18] nailed it! [04:22] coffee time [04:29] WTF... [04:29] lbox still trying to work out the diff [04:29] * thumper goes to make the coffee [04:30] wallyworld: one option is to try using the ephemeral storage, in /mnt [04:31] wallyworld: I think that should be considerably faster [04:31] and we don't need the persistence... [04:31] omg, that grew a bit... https://code.launchpad.net/~thumper/juju-core/user-display-name/+merge/221823 [04:31] axw: can't hurt, looks like the current job will pass second time through. can you tweak the script to use/mnt? [04:31] wallyworld: even just putting TMPDIR in there [04:31] wallyworld: I'll give it a shot [04:32] ok, ta [04:32] TMPDIR sounds sensible also [04:32] menn0: https://codereview.appspot.com/102970043 - it grew more than expected [04:32] * thumper -> coffee [04:33] thumper: looking [04:53] menn0: I suggest that you, me and waigani chat tomorrow about it [04:53] as there are a number of things I want to talk through [04:53] * thumper heads out to local code craft meeting [04:54] thumper: ok. I'm almost done though and I think I understand most of the things you're going to talk about [04:54] :) [04:56] thumper: thanks for cleaning up some of my --password changes. Much better. [05:09] axw: that last one took all 3 attempts to merge. different error each time. the watcher / session closed issue seems to be more prevalent of late [05:11] wallyworld: I put in the $TMPDIR change, hopefully that makes a difference [05:12] axw: i'll trigger another merge and we'll see how it goes [05:12] ok [05:14] axw: is it work setting up a dir in /mnt and setting GOPATH to that? [05:14] worth [05:14] before i do another run [05:15] wallyworld: I don't think so, that's just for building the code [05:15] it might make building the code a bit quicker, I guess, but I don't think it's a bottleneck by a long shot === vladk|offline is now known as vladk [05:19] ok [05:24] wallyworld: https://github.com/juju/core/pull/4 [05:26] axw: looks like it's off and running [05:26] goodo [05:26] wallyworld: we should probably run that script before tests too [05:26] yeah [05:27] i think i mentioned that to martin but we never wrote it down [05:27] I'll add it in the job now [05:28] ok, ta === vladk is now known as vladk|offline [06:07] axw: a random thought - would running the tests with say "-p 4" help, since that may cut down on the thrashing of resources etc [06:08] wallyworld: umm, I think it's going to be 2 as it is [06:08] it's running on a dual core isn't it? [06:08] -p defaults to #CPU [06:08] not sure, i had thought it was higher [06:08] you may be right [06:08] wallyworld: I'll check, but I don't think it's higher than 4 [06:09] if it is 4, then may -p 2 [06:09] wallyworld: it has 4 [06:09] yeah, m1.xlarge [06:10] wallyworld: doesn't hurt to try [06:10] the latest run failed twice [06:10] yeah :/ [07:03] axw: sigh. latest builds not happy [07:03] go tool: no such tool "vet"; to install: [07:03] go get code.google.com/p/go.tools/cmd/vet [07:04] i have to head to soccer, are you able to have a look and i'll check later? [07:09] wallyworld: sorry, will fix [07:56] so... one hour to github D Day... how's it looking? [08:13] morning all [08:22] voidspace: morning [08:22] TheMue: o/ [08:44] wow, it took me 3 hours to actually get a bootable USB stick that works…. so hard, why is it so hard... [08:45] jam: I've always found it difficult too [08:45] jam: and I think in the end I discovered I could only make it work with *one* of my USB sticks [08:45] voidspace: yeah, there are 4 different master partition table types, and various flags that have to be set, and .... [08:45] and tools that you try to use that just blow up unless everything is already set up correctly [08:46] voidspace: so I formatted with MBR form on my Mac, took it over to Ubuntu and used fdisk to create 1 partition dos compatible, set it active and bootable, set the type to 0x0C, mkfs.fat -F 32, and then use Startup Disk Creator to copy the ISO onto the stick. [08:46] now *maybe* msdos formatting from gparted would have also worked (maybe) [08:47] heh, ouch [08:47] I tried a lot of other permutations in there [08:47] but it seems like a streamlined "here is an ISO, and do what you want to this stick to make it boot" [08:48] would be such a better experience [08:48] jam: dd usually works, no ? [08:49] I always used to use unetbootin on the Mac - it didn't work for me on Ubuntu last time I tried it [08:49] lifeless: morning [08:49] lifeless: so the Mac instructions have you convert the ISO into a DMG and then dd it, which worked once [08:49] but I accidentally used the 64 bit version for an old machine [08:49] so I had to fix it for 32 bit [08:49] and … yeah, a lot of stuff didn't work [08:49] I think the key was finding a way to get MSDos compatibility enough [08:51] lifeless: dd doesn't set the bootable or partition table, does it? [08:51] jam: it includes a partition table [08:51] jam: dd of=/deb/sdb [08:56] lifeless: the biggest problem I had was having the old machine find the USB device as bootable, I'm not sure if I tried just dd from the .iso [08:56] axw: ping [08:56] it wasn't in any of the recommended guides [08:56] all that just to run shred sanely on all my old hard drives before getting rid of old equipment [08:57] voidspace: pong [08:57] I'll try that once the shredding is done :) [08:59] lifeless: good to see you around, btw, haven't said hi in a while [08:59] jam: hi :) [08:59] jam: did you consider netbooting ? [09:00] jam: e.g. with maas and run shred from there? [09:01] or even perhaps maas should have a decommission-hardware facility, to make removing old content easy [09:05] lifeless: interesting thought, I had not considered it. The main idea was to not have / on the physical disks so that I could nuke everything without worrying about mounts. Because I *could* just boot the thing directly, though it was running Hardy [09:06] and I think the other machine is/was running Fedora Core *3* [09:06] axw: so we made a change to Initiate yesterday on the suspicion that the CI machine was just slow [09:06] axw: this is the bootstrap problem on precise amd64 [09:06] voidspace: I saw [09:06] axw: it did not fix the issue - but it made the problem a bit clearer [09:07] oh? [09:07] lifeless: one funny thing working here, is that I'm now the most senior person in all of Juju. Tim is here, but I predate him by a couple of months (since he actually originally applied for the Bazaar dev role) [09:07] axw: immediately after calling replSetInitiate we poll until CurrentStatus returns successfully [09:07] axw: on the failing machine that fails immediately (and persistently) with a "Closed explicitly" error [09:07] axw: that specific error seems to come from the mgo library when socket.Close() is called [09:08] yep [09:08] axw: I've read through the code from MaybeInitiate (I probably need to go further back in the code but we're running out of time) to see if I can find a race condition [09:08] on the other machines it comes back saying "it'll be ready soon" or something? [09:08] axw: yep [09:08] I thought that we just weren't waiting long enough before giving up on this machine [09:09] but that appears not to be the case [09:09] axw: I do have one question [09:09] inside replicaset/replicaset.Initiate [09:09] jam: wow, sounds like there was quite some turnover? [09:10] we call monotonicSession.SetMode(mgo.Monotonic, true) [09:10] lifeless: well, this is just Juju, there are still people at Canonical that predate me. [09:10] plus, I'm like #70 or so [09:10] so not a huge gap in front, and 600 people now at Canonical [09:10] so, I was in top 10% to start [09:10] axw: session.SetMode - the second parameter is for "refresh", which if true calls session.unsetSocket() [09:11] axw: do you know why it does this? [09:11] voidspace: I think the refresh is to ensure restarting of the consistency guarantee [09:11] I don't know any more than that [09:12] voidspace: rogpeppe or natefinch may know more about it [09:12] jam: sure, I was meaning in the juju context [09:12] axw: as far as I could tell this *doesn't* close the socket - although it does set a couple of sockets to nil [09:12] axw: but there's no finalizer that I could find [09:12] jam: like, I'd have thougt kapil or mramm joined juju before you [09:12] * axw looks [09:12] lifeless: I mean joined Canonical, where I predate mramm, but maybe not kapil [09:13] axw: and it calls socket.Unlock() - which I think comes from the fact that mgo.socket embeds sync.Mutex [09:13] he doesn't directly report on the same chain [09:13] axw: which also wouldn't call Close [09:13] for Juju proper, lots of people still working on the project longer than me [09:13] fwereade, mramm, rogpeppe, etc. [09:13] voidspace: unsetSocket does a socket.Release [09:13] lifeless: jam: mramm is relatively new - I predate him [09:14] voidspace: yeah, he's only about 2 years or so [09:14] but he's been working as part of Juju longer than me [09:14] axw: right, which does an Unlock() followed by a LogoutAll() [09:15] axw: none of which *seems* to call Close() [09:15] I guess that isn't it - it was just suspicious [09:16] axw: and I wondered if we *needed* that refresh [09:16] voidspace: hmm yeah, I think it just puts it back into a pool to be reused. [09:16] voidspace: that I do not know [09:17] axw: something is closing the socket the session is using - causing all further operations to fail [09:17] axw: but only on the CI build machine... [09:18] axw: I think this morning is the deadline though, we can't leave trunk broken any longer [09:18] even if it's just on one machine... [09:18] so we'll have to backout my changes and switch to a different strategy for setting the Mongo WMode (which was the purpose of enabling replica sets for local provider) [09:19] voidspace: bugger :( [09:19] not having direct access to the machine to try things on doesn't help [09:20] coffee [09:20] voidspace: it's the closing of the "Server" object that closes all the sockets [09:35] wow, driving it at 73MB/s sustained will still take 4 hours to do 1 pass over 1TB of data, and I have it set to do 3, guess I just come back tomorrow :) [09:37] jam: doing some map-reduce ? [09:37] jam: oh shred ;) [09:37] lifeless: shredding old disks before giving them away [09:38] well, gnight :) [09:38] lifeless: gnight [09:50] github.com/ju/ju/core [09:52] stub: hah, nice [09:52] we should do that [10:04] natefinch: morning [10:04] voidspace: morning [10:05] fwereade: quick poke about CodeNotFound vs CodeUnauthorized here. If you pass an environ we don't recognize, what code should we get? [10:05] We are still getting the error during Login, because of how round trips work, so Unauthed is somewhat reasonable [10:27] good morning [10:32] rogpeppe: jam: natefinch: you guys ok with githib.com/juju/juju-core ? seems to be the least hated option [10:32] wallyworld: omg, wtf,,, how could you say such a thing [10:32] wallyworld: I'm pretty easy going here. [10:32] me too [10:32] just trying to get consensus, whatever that means [10:33] I'd actually like to get rid of the "-core" part, because it feels like it is just baggage because there was a pyjuju that took 'juju' over for too long [10:33] yeah [10:33] that was my thought of why it should just be juju [10:33] if you "apt-get install juju" you get juju-core [10:33] so github.com/juju/juju then? [10:33] I agree that github.com/juju/juju/juju and github.com/juju/juju/cmd/juju.Juju are a bit silly, but the -core doesn't actually add any information. [10:34] yeah, it's the same stuttering, just with some garbage in the middle [10:34] i thought core because it distinguishes from logging testting etc [10:34] wallyworld: github.com/juju/juju is my #1, with github.com/juju/juju-core being #2, I guess [10:34] fair point, though I'd rather prefer we had libjuju and juju the cmdline tool :) [10:34] for the record, I still don't know what the juju-core/juju package is for, maybe we just need to rename that [10:34] or rename juju the team to juju-team :-) [10:35] github/juju-team/juju [10:35] +1 to renaming juju-core/juju [10:35] wallyworld: I actually don't like the testing package name, it's not descriptive on its own, and it should be, if it's in its own repo [10:35] s/package/repo/ [10:36] yeah, i haven't thought about that one, since i didn't set it up [10:36] natefinch: it is the entry point, AIUI, so juju/api.go etc. "juju.NewAPIClientFromName()" [10:36] natefinch: arguably we could have that at the top level [10:37] github.com/juju/juju.NewAPIClient() [10:37] jam: having something at the top level would be nice. I'd love to have cmd/juju at the top level, since that's what you'd normally "go get" even though we sadly don't support that [10:38] natefinch: IMO that is a clear use case for having 2 separate things, a 'libjuju' and a 'juju command line client" [10:38] that makes sense [10:39] perrito666: morning [10:39] what do other github team based projects do? [10:39] is team name normally = product name? [10:39] often times, yes [10:40] so i guess we should go with github/juju/juju [10:40] can we all live with that? [10:41] the only other person I heard respond was axw, who said he moderately preferred /juju/juju, I think [10:42] I prefer juju/juju [10:42] fwiw :-) [10:42] ok, the people have spoken :-) [10:42] Actually, rogpeppe responded, but sort of vaguely [10:42] i'm not that keen on juju/juju actually [10:43] because then we have a package github.com/juju/juju/juju [10:43] which seems like overkill [10:43] i actually think that core isn't too bad [10:43] but i'd be happy with juju/juju-core too [10:43] rogpeppe: we can just rename that package [10:44] as it distinguishes the genuinely juju-specific parts of github.com/juju [10:44] because actually most packages under github.com/juju are *not* juju specific [10:44] so having a juju- prefix to the name makes sense [10:44] that's why I like calling this repo juju :) [10:44] natefinch: yeah, but see above :-) [10:45] jujujujujuju [10:45] natefinch: my absolutely last ditch attempt to fix locl-deploy-precise-amd64 [10:45] natefinch: https://code.launchpad.net/~mfoord/juju-core/refresh-session/+merge/221857 [10:45] see above.... we should rename that package anyway [10:46] natefinch: just running all tests here to check it doesn't break anything [10:46] natefinch: should I propose it/ [10:46] ? [10:46] natefinch: well, the original intent of that package was that we'd have a "juju" package which would be the "face" of juju to external Go code [10:47] natefinch: i still think that's a reasonable intent [10:47] yay for bike-sheds [10:48] rogpeppe: rename it to libjuju, since juju is the command line client [10:48] natefinch: and if we were to export versioned APIs, maybe it might make sense to have a separate (versioned) repo named "juju" which would import juju-core [10:48] natefinch: libjuju? [10:48] vladk|offline, standup? [10:50] rogpeppe: maybe github/juju-team is better? [10:50] natefinch: probably [10:50] natefinch: but github/juju is fine too [10:50] natefinch: and github/juju/juju-core works pretty well, i think. [10:50] natefinch: and leaves the path open for other juju- packages [10:52] rogpeppe: juju-core was my second choice, I just don't think core adds much... and using /juju doesn't stop someone else from using juju-foo [10:53] * natefinch wants to make a juju-fu package now [10:55] natefinch: if the top level of the juju-core repo was a usable package, i'd agree that juju was a better name (it makes for a more guessable identifier) but as it is, i think i'm in favour of juju-core, or just core and people can rename it. tbh i don't care too much. [10:55] well, I'd love to put cmd/juju in the root so that the CLI client is the top level "thing that gets built"... but that's probably asking too much ;) [10:57] so do we want juju-core then? [10:58] It sounds like we have 4 votes for /juju and one for juju-core .... so I don't know where that puts us [10:58] i originally was +1 for juju-core if core were not suitabke [10:58] I'd have thought it was fairly clear where it puts us... [10:58] wallyworld: so 2 for juju-core. [10:59] voidspace: I'm actually not trying to make it a popularity contest [10:59] heh [10:59] I have decided we're going to use github.com/juju/justsomedamncode [10:59] lol [10:59] i just wish our team was called juju-team [10:59] it should be justsomedamnedcode [10:59] that would remove a lot of the stuttering [10:59] https://github.com/rails/rails [11:00] team and project name the same is a pretty common convention I would expect [11:00] for things that are actually worked on by outsiders, yes... there's github.com/dotcloud/docker for example [11:01] (for things that are more primarily worked on by the company) [11:37] natefinch: ping [11:51] natefinch: I'm going on lunch [11:51] natefinch: can you take a look at https://codereview.appspot.com/102980043/ [11:52] natefinch: and see if you think it's worth one last try, or if you have a better idea / modifications [11:52] natefinch: or just think it's a bad idea... [12:00] voidspace: I looked, it's worth a try. We should really get on the CI machine... sinzui tells me we can do that now. I forget exactly what he said but I can find it on the traceback [12:16] oh man, I missed the bikeshed [12:17] say guys, I have something to submit, but I am not sure were we stand since today is go to github day [12:18] perrito666: well they haven't said "we are on github now" so I think we're still on launchpad [12:18] jam: as far as we are not in between [12:19] perrito666: until they say stop, we might as well land where we land [12:19] well, until they say go, I guess [12:19] jam: ack [12:19] jam: btw https://codereview.appspot.com/103820044/ if you want to take a look [12:20] perrito666: do we need to concern ourselves about getting 127.0.0.1 or ::1 instead of "localhost" ? [12:20] for the moment I am getting spammed by gh I need to create a few new filters it seems [12:21] perrito666: fortunately, they are "lists" so gmail filters them ok, but yes, every inline comment = new email [12:21] jam: perrito666: there will be 30 minute's notice, so land away till then [12:21] a few prereq things are being finalised [12:22] jam: you mean why I actually look for localhost in the test? [12:23] perrito666: if strings.HasPrefix(addr, "localhost:") { [12:23]