/srv/irclogs.ubuntu.com/2014/06/09/#juju-dev.txt

waiganimenn0: can I run something by you?02:00
menn0sure02:00
thumperdavecheney: around?02:07
davecheneythumper: i am not, queens birthday today02:09
thumperdavecheney: ah, that's right02:09
thumperdavecheney: that'll explain it :)02:09
thumpernm02:09
thumpersee you tomorrow02:09
davecheneykk02:09
thumperdavecheney: you ozzies doing it a week after nz confuses me02:10
thumperit's almost as if you don't like doing stuff with us02:10
thumper:-)02:10
waiganithumper: 1:1 ?03:00
thumperyeah03:00
thumperwaigani: I'm in the hangout03:02
waiganithumper: trying to logon - network is being annoying03:02
thumperheh03:02
waiganiis there a way to check if a string contains a substring in the tests? c.Assert(stringVar, gc.MAGIC, subStringVar)03:50
waiganijc.Contains :)03:51
* thumper wrote that...04:01
thumperwaigani: also jc.HasPrefix and jc.HasSuffix04:01
waiganiah nice04:02
thumperaxw: do you have a different queens birthday too?04:12
axwthumper: we do indeed04:14
axw29 September04:14
thumpergeez, how many birthdays does a queen gt?04:15
thumperget04:15
waiganihaha04:23
wwitzel3man, before my long weekend I had 18 unread .. now 163 (after filters) .. awesome.04:34
waiganiwwitzel3: yep, just been there04:44
wwitzel3waigani: yeah, calling it a night and will have to tackle the rest in the monring04:56
waiganiwwitzel3: the rest + 100 more ;)05:03
=== vladk|offline is now known as vladk
jammorning dimitern06:59
dimiternmorning jam, brt07:00
rogpeppemornin' all07:34
dimiternhey rogpeppe07:34
rogpeppedimitern: hiya07:41
=== bodie_ is now known as Guest29235
voidspacemorning all08:51
dimiternmorning voidspace08:55
voidspacedimitern: o/08:57
=== bodie__ is now known as bodie_
=== bodie_ is now known as Guest28279
=== vladk is now known as vladk|offline
=== vladk|offline is now known as vladk
wwitzel3hello o/09:35
natefinchmorning all10:01
jamnatefinch: morning, be there in just a sec10:02
natefinchit's funny, I got a notification that the meeting was now, but on my calendar it still says it's in 2 hours.10:03
=== bodie__ is now known as bodie_
=== bodie_ is now known as Guest89303
voidspacenatefinch: morningg10:10
voidspacewwitzel3: morning10:12
wwitzel3voidspace: morning, how's it going?10:18
voidspacewwitzel3: not bad, you have a good weekend?10:26
wwitzel3voidspace: yeah, good, but very busy10:31
voidspacewwitzel3: did you go look at the house?10:33
wwitzel3voidspace: we did, sadly the old 1914 place is ruled out, the land didn't really work for our needs.10:33
voidspaceshame10:34
voidspacewwitzel3: did you see any other places whilst you were out there?10:34
wwitzel3voidspace: yeah, we viewed 12 and of those 2 stayed on the list.10:35
voidspacewow10:38
voidspaceyou were busy!10:39
natefinchwwitzel3: is this places to buy?10:39
wwitzel3natefinch: yep, in Raleigh10:40
natefinchwwitzel3: nice10:40
wwitzel3"in" Raleigh .. near there the two we still have on our list are 14 and 23 acres.10:41
natefinchwwitzel3: awesome10:42
wwitzel3natefinch: yeah, we are pretty excited to be getting out of FL, hopefully soon :)10:44
jamdimitern: vladk: just finishing up a conversation, will be in standup soon10:45
dimiternjam, sure, i'm ready10:46
natefinchwwitzel3: what are you going to do with the land?10:49
dimiternfwereade, will you join us for standup today?10:50
natefinchwwitzel3: besides not need to worry about neighbors :)10:50
wwitzel3natefinch: Jessa is going to run a CSA off it, we will probably do occasional farm to table dinner events, and Jessa also does event management / planning so she plans to setup a space for weddings, etc..10:51
natefinchwwitzel3: wow!  That is awesome :)10:53
natefinchwwitzel3: my 13 acres are mostly forest on a rocky hill, so finding space to plant stuff is a trick.10:54
wwitzel3natefinch: yeah, that is actually why we ruled out some properties.10:56
wwitzel3natefinch: fortunately we aren't really in a rush so we can be picky10:57
natefinchwwitzel3: that's great.  nice to be able to move wherever you want :)11:00
wwitzel3natefinch: indeed11:00
voidspacenatefinch: ping11:18
natefinchvoidspace: howdy11:19
voidspacenatefinch: did we come to any conclusions on the backup implementation strategy and division of labour11:19
voidspacenatefinch: I suggested that instead of returning a URL to a backup we serve backup files through the api11:20
voidspacenatefinch: which would need a "fetch backup api call / command"11:20
voidspacenatefinch: there isn't a ticket for this yet11:20
voidspacenatefinch: I can add one, if we've decided this is what we want to do11:21
natefinchvoidspace: yeah, that would be good11:21
voidspacenatefinch: which parts of backup can be parallelised?11:22
voidspacenatefinch: you were fleshing out the api - do you have a branch with that in progress?11:22
=== Ursinha-afk is now known as Ursinha
natefinchvoidspace: yes, trying to get things to build so I can push it up11:24
voidspacenatefinch: cool11:24
natefinchdammit git11:39
natefinchmade a branch on juju/juju not natefinch/juju  :/11:40
perrito666morning folks11:42
jammgz: poke about test suite failures11:55
mgzjam: hey11:55
jamhttps://github.com/juju/juju/pull/26 has failed 2 times with Bad Record MAC11:55
jamin 2 different test cases11:55
mgzhm, I can't actually find an open bug for that11:57
mgzbut it's not a new one11:58
jammgz: no, bad record MAC is thought to be an issue with upstream mongo + SSL/TLS12:01
jammgz: specifically: https://jira.mongodb.org/browse/SERVER-1180712:01
mgzjam: I think I'll reinstate the auto-retry12:01
jammgz: I'm not particularly happy with "just rerun the test suites until they pass" approach to landing code.12:02
mgzyeah, it's pretty sucky12:02
jammgz: as a for-example, if this is being a problem, can we just run 99% of the test suite with TLS disabled? We don't actually need to encrypt our data to the local mongo db12:02
jam(though I'd rather have tests that just don't need mongo at all for local testing)12:03
mgzideally that12:03
mgzbut... we've talked but not got as far as de-mongofying tests for a while12:03
perrito666could someone re-re-read this and give me a LGTM? is just docs and already passed 2 rounds of gramatical review but a final one wold be nice12:10
perrito666natefinch: did you manage to find out how to write a facade?12:12
* voidspace lunches12:12
natefinchperrito666: ish12:13
natefinchrogpeppe1: any idea how to fix this? testing/environ.go:109: undefined: "github.com/juju/testing".FakeHomeSuite12:13
perrito666natefinch: I am in the process of bash->go the actual backup logic, which is fairly short, then we can see how to stitch all together, sounds good to you?12:14
jamnatefinch: frankban just landed something that was talking about removing it12:15
natefinchjam: I just needed to update juju/testing12:15
natefinchjam: thought I'd just done that, but perhaps not12:15
jamnatefinch: ah, so it was failing in master, sure12:16
jamI thought maybe you needed to crib an example from how he removed it12:16
jamremoved or moved12:16
natefinchjam: nah... just merged from master and then my branch was broken12:17
jamnatefinch: yeah, I ran into that with the 6+ dependencies I had to go update12:18
natefinchvoidspace: when you get back, my backup api facade is at github.com/natefinch/juju  on the backup-api branch  under state/apiserver/backup12:18
voidspaceok, thanks12:22
* voidspace really leaves for lunch12:22
rogpeppe1natefinch: have you updated dependencies?12:29
rogpeppe1what do people think about moving testing/mgo.go into github.com/juju/testing?12:30
rogpeppe1the store tests need a mongo server to run against, so we need *some* code that starts mongod outside of core12:31
rogpeppe1but i'm swithering a bit about all the stuff that's really closely related to juju-core in there12:31
rogpeppe1for example the certificates - they really feel quite core-specific12:32
rogpeppe1i wonder if we made the certificates an argument to the testing functions, that might work ok and mean that we wouldn't need to move the cert code12:33
rogpeppe1anyway, gotta lunch now12:33
perrito666fwereade: tx :)12:39
natefinchwwitzel3, perrito666: today is Eric Snow's first day, I think he'll be coming on shortly... I'm unfortunately watching both my kids for a significant period of time this morning since my wife has to go to a doctor's appointment.  If one of you could help him when he gets on to get set up, that would be awesome12:50
perrito666natefinch: sure12:51
=== Guest89303 is now known as bodie_
bodie_morning all13:15
=== rogpeppe1 is now known as rogpeppe
lazyPowernatefinch: have you seen anything wrt 2014-06-09 12:35:57 INFO juju.worker runner.go:260 start "api"13:22
lazyPowerpanic: runtime error: invalid memory address or nil pointer dereference13:22
kikoupon startup, am getting a crash13:28
kiko2014-06-09 12:35:57 INFO juju.cmd supercommand.go:301 running juju-1.19.2-precise-amd64 [gc]13:28
kiko2014-06-09 12:35:57 INFO juju.cmd.jujud machine.go:151 machine agent machine-0 start (1.19.2-precise-amd64 [gc])13:28
kiko2014-06-09 12:35:57 DEBUG juju.agent agent.go:375 read agent config, format "1.18"13:28
kiko2014-06-09 12:35:57 INFO juju.worker runner.go:260 start "api"13:28
kikopanic: runtime error: invalid memory address or nil pointer dereference13:28
kiko[signal 0xb code=0x1 addr=0x0 pc=0x4643b2]13:28
kikocan anybody give me a hand figuring out what is wrong?13:28
natefinchlazyPower: ouch13:31
lazyPowernatefinch: yeah, as you can see that originated from Kiko13:31
lazyPowerI'm at a loss as to why it would be having memory pointer dereferencing issues13:31
lazyPoweri thought go used garbage collection so memory management wasn't a thing. but I really dont know much about Go so i'm not much help13:32
natefinchlazyPower: there's still pointers, which can be nil, and if you try to dereference the pointer, it'll panic like that13:32
lazyPoweryikes13:33
jamlazyPower: go just ensures you get a traceback/panic rather than undefined operation on whatever memory you happen to be pointing to13:33
natefinchit's also not as big a problem in Go as in some other languages, because most of the time you return both a pointer and an error, and if there's an error, you don't use the pointer.  it's a lot more obvious than in other languages where you might just do "GetFoo().Method()" or similar, and forget that GetFoo() might occasionally return a null pointer13:35
wwitzel3natefinch: re: Eric's first day, no problem13:35
natefinchwwitzel3: thanks13:35
lazyPowerok so we have no clue was to why kiko's deployment is panicing after a server reboot?13:36
bodie_I find that if I wrote code that has a nil dereference, it means that I need to put in checks for a nil value and they'll tell me where things are going wrong13:36
kikolazyPower, well, actually, it was a server reboot + juju client upgrade from 1.18.3 to 1.18.413:36
kikounfortunately because of other bugs (in upgrade-juju), our jujuds are all 1.1913:36
natefinchso, it seems likely that it's related to upgrading to 1.19 when it wasn't supposed to.... but it's hard to know.  Do you have the full stack trace from the panic?13:38
natefinchjam: yeah, that's the nice thing about Go's values always getting initialized to their zero value... pointers start out nil, not pointing at random spots in memory.13:40
natefinchSo, totally unrelated to anything else.... we just hatched our own chickens for the first time this morning.... like, toss some fertilized eggs under a chicken, wait 3 weeks, bam, baby chicks.  It's kind of amazing.13:42
perrito666natefinch: wwitzel3 voidspace what would be really nice is to know eric's irc nickname13:43
perrito666:p13:43
kikonatefinch, yes, just a second13:43
voidspaceperrito666: heh, I don't know I'm afraid13:43
natefinchperrito666: when he tells us, we'll let you know13:43
voidspaceperrito666: I thought it was ericsnow though...13:44
kikonatefinch, http://paste.ubuntu.com/7618160/13:44
bodie_when someone has a spare minute to look at my pseudo-PR here -- I'm thinking about whether we should be using markdown in our doc now13:46
bodie_https://github.com/juju/juju/pull/4613:46
bodie_just want to get some feedback so I know whether I'm barking up the wrong tree :)13:46
perrito666bodie_: I, actually, github picks it up and renders it, which is nice13:47
bodie_yup, it's very nice13:47
bodie_There's also a [tag:asdf] tag which I think links to another piece of github but I'm not certain how to make it work13:49
=== ev_ is now known as ev
bodie_but, it could be used for example as [tag:newbies] or something -- linking to a sticky post for newcomers -- etc13:50
voidspacegrabbing coffee13:50
voidspacemaybe a coupla minutes late to standup13:51
perrito666grabbing a hangout compatible computer13:51
jamkiko: so the traceback you pasted says that the agent is missing the "*api.Info" it needs to actually connect to the API13:55
jamso you can look for a /var/lib/juju/agents/machine-?/agent.conf file13:55
jamoffhand, I don't know how that would happen, given it is a pretty vital piece of information13:56
kikojam, well, the machine-0/agent.conf file is in ~/.juju/local13:56
kikojam, it contains apiaddresses, apipassword and apiport13:57
bodie_hm... I think my upstream is failing tests too13:58
natefinchvoidspace, perrito666, wwitzel3: I think I have to miss the standup today. My wife is leaving right now, so I gotta watch the kids.  I'll try to pop back in off and on.  Try to coordinate amongst yourselves if someone doesn't have something to do14:01
bodie_take care natefinch14:01
kikojam, the permissions had it originally only readable by root, but I chowned it to juju14:03
kikounfortunately I still get the same traceback14:03
kikojam, what am I missing explicitly?14:03
jamkiko: still digging into the traceback, I had to pull out the 1.19.2 source code and find the exact line, because it actually looks like we don't have a config object at all, but I can't figure out from tracing various paths how that object could be nil14:04
kikojam, could it be the file could not be opened?14:04
kikojam, because I remember being told that there was a permissions change14:04
perrito666voidspace: whenever you want, wwitzel3 and I are on the call14:04
jamkiko: the agent on machine-0 runs as root, so you can't stop it from opening anything14:04
bodie_http://paste.ubuntu.com/7618248/14:04
kikojam, I see -- what about the location where the file is?14:05
kikojam, it used to be in /var/lib/juju indeed, but I was told to move it to ~/.juju/local14:05
bodie_"not okForStorage"?14:05
bodie_"cannot update uploaded charm in state" -- is this because I've altered the contents of my testing/repo/quantal/dummy charm?14:06
kikojam, moving it back makes no difference though either14:06
jamkiko: hm. I hadn't heard of that, as for location, I would have thought it would fail far earlier with:14:06
jam        if err := a.ReadConfig(a.Tag()); err != nil {14:07
jam                return fmt.Errorf("cannot read agent configuration: %v", err)14:07
jam        }14:07
kikojam, yeah, that sounds right14:07
jamkiko: if it isn't in /var/lib/juju then I would expect your /etc/init/juju...conf upstart file will be passing a "--data-dir=" that would tell us where that file should be. Not that I think that must be the problem, because I *think* you would see a different error.14:09
jamkiko: so... another thought. There is another pointer in there "apiDetails"14:09
jamjust a sec14:09
jamkiko: is there a line in there called "APIAddresses" ?14:11
jamin the agent.conf file?14:11
kiko<kiko> jam, it contains apiaddresses, apipassword and apiport14:11
kikoone sec14:11
jamyeah, I just found that in traceback14:11
jam        if len(configParams.APIAddresses) > 0 {14:12
jam                config.apiDetails = &connectionDetails{14:12
jam                        addresses: configParams.APIAddresses,14:12
jam                }14:12
jam        }14:12
jamso it has to have contents as well14:12
jamit should be a list14:12
jammine looks like this:14:13
jampiaddresses:14:13
jam- 10.0.3.1:1707014:13
jamwell, with an 'a' there14:13
jamso it wraps to the next line and has a list that starts with "- "14:14
kikojam, my file is here: http://paste.ubuntu.com/7618297/14:14
jamkiko: there is no apiaddresses in that file14:14
jamkiko: my suggestion, is to put one in with "apiaddresses:\n- localhost:17070\n"14:15
kikohmm14:15
kikojam, my file also missing a sharedsecret entry14:17
kikojam (am comparing it with a freshly bootstrapped 1.18)14:17
kikojam, it runs14:18
kikojam, was that not required pre-1.18.4?14:18
jamkiko: both were introduced in 1.1914:19
kikojam, I see14:19
kikojam, and I guess the upgrade isn't being done properly?14:19
kikojam, is the lack of sharedsecret going to be a problem, and if so, how do I get one14:20
jamkiko: upgrades *to* dev versions aren't really supported, I'm really unclear how it actually triggered.14:20
kikojam, it's another bug I filed, let me find it14:20
jamkiko: it is a random string used for when you're going into HA/--replicaSet mode.14:20
kikojam, I see, so not that important14:21
kikojam, https://bugs.launchpad.net/juju-core/+bug/132503414:22
_mup_Bug #1325034: juju upgrade-juju on 1.18.3 upgraded my agents to 1.19.2 <upgrade-juju> <juju-core:Triaged> <https://launchpad.net/bugs/1325034>14:22
jamkiko: so *if* all upgrade steps were run, then we might need to be passing it to mongo, since *none* of them run, we don't actually need it. If we do, we can just put random data in there.14:22
jamkiko: so as I understand it, the upgrade logic is pinned on "stable" versions, so it says "oh you are upgrading from version 1.OLD to 1.NEW, I'll run these steps" but those steps are targetted at the "next stable" release, which would have been 1.20.14:22
jamsince 1.19.2 isn't 1.20, nothing gets run14:22
jamwallyworld knows more about the upgrade logic than I do, though.14:22
kikoyeah14:22
kikoall my agents are missing apiaddresses entries14:22
kikoshucks14:22
jamkiko: so the ones that aren't machine-0 need machine-0's address14:23
kikojam, they are all tracebacking identically, yeah14:23
jamkiko: sure, I just wanted to be clear that they shouldn't have "localhost" in there14:24
kikoah14:24
jamkiko: do you know where you got 1.18.3 from? ppa:juju/stable?  (it doesn't seem to be in cloud-archive:tools or Trusty archive because of problems landing updates in utopic)14:25
kikojam: yes, that's where it came from originally14:25
kikojam: I have the debs here if it helps, because I couldn't find them anywhere today either14:25
jamkiko: yeah, ppa's don't keep history so there is only 1.18.4 there now.14:27
jamanyway, I'm way past EOD and have to make dinner for my son, hopefully we've moved you forward a bit, and I'll try to follow up with other people.14:27
jamnatefinch: ^^ can you help kiko if he needs anything else?14:27
kikothanks jam14:28
natefinchjam: I'll do what I can.  I'm a little distracted by two kids, but I hope I;ll be able to help14:28
kikoheh14:29
natefinchsinzui: have you seen any problems upgrading from 1.18.3?  looks like kiko got upgraded to 1.19.2 somehow, and it seems to have messed up his environment14:32
sinzuinatefinch, I have never seen that issue14:33
kikoyeah, it messed it us great14:35
=== makyo_ is now known as Makyo
alexisbwelcome jheroux !15:16
alexisbfancy seeing you here15:16
dimiternjam, vladk, fwereade, others? - a mostly automatic refactoring to move network-related stuff from instance to environ/network package, i'd appreciate a review https://github.com/juju/juju/pull/4915:16
bodie_anyone able to grok the difference between json-schema core and json-schema hyperschema?15:17
bodie_I'm trying to write a mini validator that validates a hypothetical json-schema against the spec15:18
bodie_I'm just not sure if I should be validating against json-schema or json-schema hyperschema15:18
rogpeppeanyone else seen this log message "2014/06/09 16:19:30 http: TLS handshake error from 127.0.0.1:58318: tls: first record does not look like a TLS handshake"15:19
rogpeppebodie_: ha, let me just look at the description again. i *thought* i understood the difference once...15:20
rogpeppebodie_: from a naive reading of the description on the home page, json schema is what we need15:21
bodie_it would make sense that hyperschema is the schema that defines json-schema itself15:22
bodie_however, I don't think that's the case, or I'm just really confused, which is probably the case :)15:22
rogpeppebodie_: yeah, that's what i thought, but it doesn't look like it15:24
rogpeppebodie_: i guess that would actually be a "meta-schema"15:25
bodie_some people get those words mixed up :P15:25
rogpeppebodie_: from the json schema home page:15:25
rogpeppeJSON Schemadescribes your JSON data format15:25
rogpeppeJSON Hyper-Schematurns your JSON data into hyper-text15:25
bodie_ahhh15:25
bodie_somehow missed that15:25
bodie_thanks!15:25
rogpeppebodie_: the specs are not the clearest in the world15:25
mattywnatefinch, fwereade do either of you have some work I'm able to pick up?15:37
natefinchmattyw: not a ton for me right now.  Wrangling a couple kids this morning, so can't do much right now.  I can try to find something later today if you don't have anything else by then.15:40
rogpeppefwereade: i made a couple of improvements to the filetesting package. review appreciated: https://github.com/juju/testing/pull/1416:12
rogpeppeor anyone else, please: ^16:14
rogpeppefrankban, natefinch, bodie_, mgz: ^16:14
jherouxalexisb: hey, I've been been asked to look at OpenStack Heat & Juju -16:23
alexisbjheroux, cool16:24
alexisbwe should schedule some time to meet16:24
jherouxalexisb: sure16:24
alexisbare you jheroux@us.ibm.com?16:25
jherouxI don't fully understand all of the requirements yet, will be talking to SWG, and other IBM teams shortly16:25
jherouxyup, that's me16:25
alexisbI will start by inviting you to our weekly interlock16:25
jherouxok, makes sense16:25
alexisband then you and can meet 1x1 and I can give you some back ground16:26
jcw4marcoceppi: for juju v.next documentation... should I propose against dev branch?16:26
jherouxok, I'll look for the invite16:26
marcoceppijcw4: we're going to be making a juju-docs repo soon, which has master being the dev, then each version of the release by branch (ie 1.18 branch, etc) it's just not there yet. Right now master is the stable version of the docs on the repo and dev is a bit behind16:27
marcoceppileave the mp as it is, I'll chat with nick about how soon we can get the new repo setup16:27
jcw4cool thanks marcoceppi16:27
perrito666is it possible to change the libjuju path from /var/lib/juju?16:45
jcastroMy juju seems to only launch containers for the bootstrap, for everything else they just get stuck on "pending", ideas?16:50
natefinchjcastro: might be a bit late, but it's usually an LXC problem, rebooting may help, believe it or not17:09
jcastrough17:10
natefinchjcastro: you can try killing all lxc processes17:10
natefinchjcastro: sometimes that'll do it17:10
jcastrook I'll give it a shot17:10
voidspaceright, I'm off to Krav Maga17:14
voidspaceg'night all17:14
voidspaceEOD17:14
wwitzel3night voidspace17:16
perrito666I would really love to take classes of one of these martial arts if it didn't involve social interaction17:17
natefinchhaha17:18
natefinchI don't know, getting punched in the face isn't that social17:19
perrito666natefinch: I think that the whole non punching time includes social interaction17:19
natefinchperrito666: if you're not constantly punching or getting punched in the face, you're doing it wrong ;)17:25
perrito666we call that public transportation here17:26
natefinchlol nice17:26
bodie_perrito666, the fun thing about most martial arts is that you only have to socially interact with other nerds.17:28
bodie_MMA is probably the exception17:28
perrito666bodie_: oh no, here nerds keep their place appointed by birth and are as physically unfit as possible17:29
bodie_I guess it's the same catch-22 as going to the gym.  I'm puny, so I don't go to the gym to stop being puny... haha17:30
perrito666The only reason I get to have some fitness is because I was born in a ruralish town where you are forced to practice sports17:31
bodie_but, generally speaking I've had good experiences with martial arts -- a good dojo teaches humility and mentorship -- so if people there aren't humble (especially the sensei), you're at the wrong dojo!17:31
natefinchI think you'll find that most people at the gym will look at the puny guy or the fat guy or the old guy and think "Good for him!  He's working to make himself more fit!"17:31
bodie_:)17:32
bodie_unless he's obama.. then they make fun of him to the whole internet.  lol17:32
perrito666I try to exercise on my own, I do a lot of biking17:32
perrito666and recently a friend got me for my bd this step counting bracelet, which I find very fun and helpful17:33
rogpeppefrankban: a slightly more substantial review for you: https://github.com/juju/utils/pull/217:33
* lazyPower is always the fat guy at the gym17:33
rogpeppenatefinch: fancy reviewing a recursive file copy implementation?17:33
bodie_perrito666, you should check out beeminder -- they have an integration with those bracelets17:33
rogpeppenatefinch: if so... https://github.com/juju/utils/pull/217:33
rogpeppe+ anyone else17:33
natefinchrogpeppe: sure17:34
jcw4rogpeppe: looks good! :)17:41
natefinchrogpeppe: do we care about maintaining stuff like owner, last modified date, etc?17:45
rogpeppenatefinch: nope17:46
rogpeppenatefinch: i'm just interested in doing a rough substitute for cp -r17:46
* rogpeppe is done for the day17:47
=== alexisb is now known as alexisb_lunch
natefinchericsnow: welcome!18:45
perrito666hey ericsnow hi18:46
ericsnowhi there18:46
* perrito666 finds the doc for tar.FileInfoHeader a bit confusing18:48
=== hatch__ is now known as hatch
ericsnowso I spammed the CDO list with an intro email, but it was waiting for moderator approval the last I checked18:54
ericsnowanyway, glad to be here18:55
natefinchGlad to have you here.18:55
perrito666ericsnow: I think you need to subscribe to that18:56
natefinchperrito666 is Horacio, from Argentina (UTC-3 right now).  mgz is Martin, from the UK.    You evidently know wwitzel3.  fwereade is William, the Juju architect, who is in the UK18:56
jcw4ericsnow: welcome!18:57
natefinchrogpeppe is Roger, also in the UK (there's a theme here....)18:57
perrito666natefinch: interesting random slice of people18:57
natefinchthese are some of the juju devs that are online during normal daylight hours18:58
jcw4I thought fwereade was in Malta :)18:58
ericsnowcool18:58
natefinchjcw4: yes, I'm dumb18:58
perrito666jcw4: I think malta is UK18:58
perrito666isnt it?18:58
jcw4republic of malta...18:59
natefinchit's certainly not NEAR the UK.  The politics may be different18:59
perrito666oh I am like 60 year old in my info18:59
perrito666lol18:59
jcw4:D18:59
ericsnowperrito666: vivi en Buenos Aires 2 anos (hace 19 anos)19:01
natefinchtopical - countries the UK has NOT invaded: http://i.imgur.com/yKK2yrc.gif19:01
perrito666natefinch: seems to be counting unsuccessful invasions too19:02
natefinchperrito666: perhaps :)19:02
perrito666ericsnow: you managed to escape on time :p19:02
jcw4natefinch: and we wonder why engilish is the language of international business...19:04
jcw4English even19:04
jcw4(American notwithstanding)19:04
natefinchindeed19:04
ericsnowI think I've signed up for all the right stuff, but if you see me missing from anyway, just let me know19:05
ericsnows/anyway/anywhere/19:05
=== alexisb_lunch is now known as alexisb
perrito666jcw4: its a reasonable language for that, simple enough and lacks one of the things most people find confusing, which is having a gender for each object19:07
alexisbwelcome ericsnow !19:07
ericsnowthanks19:07
jcw4perrito666: +119:07
=== hatch__ is now known as hatch
jcw4mgz: if the tests fail in CI, is it safe to push updates to the PR branch before it officially reports failure?19:08
alexisbericsnow, natefinch: I assume you guys are all set and don't need me fore anything atm19:09
natefinchalexisb: as far as I am concerned, yeah'19:09
ericsnowalexisb: same here19:09
alexisbcool, if that changes just ping me19:10
ericsnowwhat's the typical meeting schedule?19:12
ericsnowis there a team calendar somewhere?19:13
natefinchyeah, there's a google calendar... can you get on canonical email?19:16
ericsnowyep19:18
natefinchso.... I'll ping the the other team leads, since I don't have rights to add anyone to the team calendar... but there's a team calendar with our standups and weekly team meeting in it19:19
ericsnowcool19:19
ericsnowdoes it also include events (like sprints)19:19
natefinchwe have daily standups with our squads (a squad is just a small team in the larger juju core team)19:19
ericsnowgot it19:20
natefinchthe sprints often get added to it by someone on the team, once we know when they are19:20
natefinchbut they're not added automatically or anything19:21
perrito666ericsnow: you will get notified anyway19:23
ericsnowcool19:24
natefinchwe have a weekly meeting on ~thursday (it rotates by 8 hours within the UTC day, which means one of the rotations ends up with the meeting late Wednesday night for people in the US)19:24
natefinchwe have people all over the globe, so we started rotating the meeting so it wouldn't suck for the same people every week.  But it means some of the time it might be really sucky.... we only really expect people to make 2/3 of the times (more is better, but we understand the meetings might be at like 3am for some people)19:25
ericsnowsounds good19:26
natefinchhaving IRC up all day while you're working is a good idea... we use it for asking for reviews, asking for help, helping others that need help (both on juju-core and not)19:28
natefinch#juju-dev is generally for people who write code for juju, #juju is for users of juju as well, so it's good to keep an eye out there, though the juju solutions and community guys are more responsible for fielding the easy questions, and will ping us if they have harder questions.19:29
ericsnowI'm on both19:30
natefinchyep, good19:30
natefinchsinzui is our intrepid QA-master who sets up our CI system19:30
natefinchoh btw, huge help - at directory.canonical.com you can search for people by irc nickname19:31
ericsnownivr19:31
ericsnownice19:31
natefinchso like, when you're thinking "Wait, who the hell is talking to me?"  You can go find out pretty easily :)19:32
natefinchwe have a kanban board here: https://canonical.leankit.com/Boards/View/103148069#workflow-view    if you can't log in there, we can figure out how to add you19:34
ericsnoware there any particular projects I should join on launchpad?19:34
natefinchjuju-core, definitely19:35
ericsnowam I supposed to have an account for that site?19:35
natefinchyes19:35
natefinchyou can set one up yourself for the most part19:35
natefinchThat's a good thing to do right now if you haven't already19:36
ericsnowyeah, navigating that now19:36
natefinchyou'll want to give your account an ssh key, we actually use that for some stuff19:36
natefinch(associated with your canonical email address)19:37
ericsnowk19:37
perrito666I think most of that is on the hanbook and the getting started page, ericsnow make sure to read that one too as its more reliable than natefinch's memory19:38
natefinchyep19:39
ericsnowdid that this morning19:39
perrito666ericsnow: so most likely you already added your pubkey everywhere19:39
ericsnowjust about19:40
ericsnowam I missing something or is setting up an account for the Kanban site not trivial?19:46
rick_h_ericsnow: you have to have it setup for you19:46
rick_h_arosales: is a master of it ^19:47
rick_h_ericsnow: email him a request with your email addr and wait a bit19:47
ericsnowthat makes *much* more sense :)19:47
rick_h_not sure if alexisb has super powers or not19:47
arosalesericsnow: I can get you set up.19:47
perrito666ericsnow: either someone does it for you or it was such a traumatic experience that I forgot how I did it19:47
ericsnowarosales: sweet!19:48
alexisbrick_h_, I do not, I need to get super powers though as this is the second time it has come up and I havent been able to help[19:48
rick_h_ericsnow: and welcome to the party19:48
alexisbarosales, thank you for helping out19:48
ericsnow:)19:48
rick_h_alexisb: yea, I just keep pushing it to arosales until he gets fed up one day and makes me figure out how to get super powers19:48
alexisblol19:48
alexisbor I can do that :)19:48
rick_h_works for me :)19:49
ericsnowarosales: you need my email?19:49
ericsnowdo I also need to get put on any launchpad teams (e.g. "juju hackers")?19:52
rick_h_ericsnow: yep, hopefully if you get put into your squad team you'll get into most of what you need, though you'll have a bunch of mailing lists to sign up for19:53
rick_h_ericsnow: bug your team lead on that front19:53
arosalesnatefinch:  ^19:54
natefinchjust did :)19:54
natefinchdidn't realize I had been granted that power19:54
arosalesericsnow: natefinch and alexisb may have a few getting started tasks for you to run though that may fill a lot of your access needs19:54
ericsnowcool, thanks!19:55
arosalesericsnow: welcome to the Juju team :-)19:55
natefinchericsnow: you're the first person I've onboarded, and I've only been here 11 months, so bear with me ;)19:55
ericsnow:)19:55
ericsnowno worries19:55
perrito666natefinch: you made a great onboarding :)19:57
natefinchso, juju just moved from launchpad.net/juju-core to github.com/juju/juju .... and by just, I mean the switchover was.... Thursday.  So things are still in a little bit of flux.  The code will live on github, but we're using launchpad as the canonical place for bug tracking19:58
ericsnowso git instead of bzr?19:59
perrito666yup19:59
natefinchcorrect... there's still some packages that are on launchpad, which may get moved eventually but are not yet moved19:59
natefinchstuff like the wrappers for MaaS, Azure, AWS, etc19:59
perrito666ericsnow: the contributing doc on juju's git is quite good explaining the base setup20:00
ericsnowk20:00
natefinchericsnow: are you running ubuntu?20:00
ericsnowsort of20:00
ericsnow:)20:00
natefinchheh20:00
natefinchVM?20:00
ericsnowI have a headless box running 12.0420:00
natefinchoh right, you said that20:00
ericsnowlaptop arriving tomorrow20:01
natefinchahh, ok20:01
natefinchwhat laptop did you end up choosing?20:01
perrito666rogpeppe: are you still around?20:01
ericsnowXPS 1520:01
natefinchmoce20:01
natefinchnice too20:01
natefinchok, so I think we'll skip setting up repos and stuff today, since we'd have to redo it tomorrow20:02
ericsnow:)20:02
natefinchI forget, how's your Go?20:02
perrito666natefinch: actually tar -cf - /* | ssh newbox tar -xf -20:02
perrito666works like a charm20:02
natefinchperrito666: yeah, I guess20:03
natefinchperrito666: you end up having to set up git and bzr and all that though20:03
ericsnowit's new to me, but I've been reading up20:03
fwereadeericsnow, welcome aboard20:03
ericsnowit's not too bad20:03
ericsnowthanks20:03
fwereadeericsnow, in a spirit of pedantry I should point out that I live in malta20:04
natefinchfwereade: someone else already pointed out my boneheaded mistake20:04
perrito666natefinch: nah, I just actually ran that command for all the files on my home and got my whole setup moved :p20:04
natefinchperrito666: I guess that's a good point20:04
fwereadenatefinch, as it happens malta is disturbingly similar to england anyway :)20:04
natefinchfwereade: heh.  Are you from there originally, or just like the weather?20:05
fwereadenatefinch, uk originally, moved here a while ago... will be 5 years in december actually20:06
natefinchfwereade: sweet, I was only half wrong :)20:06
fwereade:D20:06
perrito666natefinch: if we where in 1963 you would have been fully right20:06
natefinchericsnow: have you gone through the tour of Go yet?  It's a really useful intro to the language - http://tour.golang.org/#120:06
natefinchperrito666: lol, finally looked it up on wikipedia huh?20:06
ericsnowyeah, did the tour last week20:07
natefinchcool cool20:07
perrito666natefinch: I looked up when you first said it20:07
natefinchericsnow: sweet, so you know the basics.  And luckily, with Go, that's sorta all there is to it.  Learning the Juju codebase is like 100 times as hard as learning Go :)   Which is not to say the codebase is bad, there's just a lot to it.20:08
* perrito666 looks at fwereade that is que cue for your doc/ line20:09
ericsnowmy kind of code base :)20:09
fwereadeericsnow, yeah, there is at least some good stuff in the doc/ directory20:09
natefinchdo you have Go installed?20:09
natefinchwe can go get the code so you can look at the docs etc... I guess it can't hurt to set things up a little bit :)20:10
ericsnowI'm set on go20:10
natefinchcool cool, so just go get github.com/juju/juju/...20:11
natefinchthe path is actually a pattern match, if you put ... in the path, it's a wildcard.  In this case it means get all the packages under github.com/juju/juju  (you have to do it that way since the root package doesn't actually have code in it that pulls in all the rest of the code)20:12
natefinchgo get will get the code and all the code that it depends on and dump it in your gopath20:12
perrito666natefinch: heh, I wish I had that clarification the first time I tried that, I thought ... was just abreviation for the rest of the path20:13
natefinchperrito666: yeah..... took me a long time to figure out that you could do stuff like go test ./...20:14
natefinchoh yeah, you need a GOPATH.  I recommend $GOPATH=$HOME  but you can put it wherever you want20:14
natefinchin theory GOPATH can be a list of paths, but it's way easier just to use one20:15
ericsnowgo-getting now...20:15
natefinchit's a little slow because we imported the whole of juju history into github, so there's quite a bit of code to download20:15
ericsnowin the meantime what about reviews and CI?20:16
perrito666ericsnow: you mean code reviews?20:17
ericsnowyeah20:17
perrito666code reviews are explained in CONTRIBUTING with detail, I recommend you read those there, I can omit something If I try to tell you the process here20:18
natefinchsure, so we do reviews for github stuff on github.  You fork github.com/juju/juju, make your changes, submit a pull request, and then ping people on here to review it.  Once someone gives you a LGTM, you put a comment with the text $$merge$$  and the merge bot will pull your branch and test it after merging it to main to make sure the tests pass, then it'll merge it automatically into main20:18
* perrito666 was trying to avoid that ^^ clutter20:19
ericsnowgot it20:19
ericsnow:)20:19
natefinchheh20:19
natefinchgenerally the idea with reviews is that the reviewer is careful about when to give LGTM.  They'll only give it if they mean it.  If they give it *and* give you something to fix, it means "these are minor fixes that I trust you can make, and after that, go ahead and merge"20:20
=== vladk is now known as vladk|offline
natefinchoften times people will say something like "this looks ok, but you should have someone else look over it".20:21
ericsnowsounds right (good practice in my experience)20:22
natefinchyep20:22
natefinchwe review every last commit.  Tests are important. We don't really check coverage, but you generally need tests for any new functionality, and preferably for fixes to old functionality (if the old tests didn't catch the bug you fix)20:23
natefinchericsnow: CI is at http://juju-ci.vapour.ws:8080/20:25
natefinchfyi, I have to run in a half hour on the dot20:30
natefinchman, we need to put the dependency management stuff way at the top of CONTRIBUTING20:37
perrito666natefinch: you could actually add a "read this to the end"20:38
natefinchno, because it should be in chronological order20:39
perrito666heh "you might be wondering why the above steps did not work"20:41
natefinchI'm fixing it20:41
perrito666mm if I want the leading slashes removal from the header names on a tar file created with go's tar I should do it by hand on the Name of the file's header, right?20:43
ericsnowokay, I'm going to work my way through the readme and contributing docs20:48
natefinchperrito666: I have no idea what you're talking about :)20:51
natefinchericsnow: cool, I just updated the contributing doc a little20:51
perrito666natefinch: mmm, I am trying to tar -cf blah.tar /etc/something/file.conf which will remove the leading /20:52
perrito666so untar that blah.tar will yield etc/something/file.conf20:52
perrito666but I am doing that in go, and i am not sure how the spec of that goes20:53
natefinchthumper: this is ericsnow, he's new.  He's a couple hours behind me, so I gotta run, but he may have more questions20:53
thumpero/ ericsnow20:54
thumpernatefinch: ack20:54
ericsnowhowdy20:54
jcastrowelcome ericsnow!20:54
ericsnowthanks Nate20:54
natefinchericsnow: thumper is one of our team leads, in New Zealand20:54
thumpercold wet and rainy NZ at the moment20:54
natefinchjcastro is uhh... what is your title?  Cloud Community Liaison?    Man, there's a BS title if I ever heard one ;)20:56
natefinchjcastro basically does a little bit of everything and tries to make sure people love us20:56
natefinchwhich is sometimes a tough job ;)20:57
ericsnow:)20:57
natefinchok, I gotta run... have fun everyone.   ericsnow, I'm usually on pretty early20:57
ericsnowk20:57
natefinchbut get on whenever is convenient for you20:57
jcw4ericsnow: what is your timezone?20:57
ericsnow-060020:58
jcw4MST?20:58
natefinchmountain time20:58
jcw4k20:58
ericsnowwell, MDT right now :)20:58
natefinchmountain time just sounds cool20:58
jcw4I *always* get confused with that20:58
natefinchok, really going20:58
ericsnowwe all do :)20:58
thumperericsnow: FYI, NZ is UTC+1220:59
ericsnowgot it20:59
perrito666ericsnow: you are then 3 hs after me i thin21:00
perrito666think21:00
perrito666ericsnow: what time is it there :p21:00
ericsnow15:0021:00
thumpermramm: hey there, we have a scheduled meeting, are you around?21:01
mrammyep21:01
mrammon my way21:02
mrammjust finishing up with alexis/antonio/rick21:02
thumperack21:02
perrito666ok, is the standup time something you can reach?21:04
perrito666that would be 8AM for you21:04
ericsnowshould I expect to see errors when running "go install github.com/juju/juju/..."?21:08
ericsnowI see a bunch of "not a package file" and a couple "undefined:" in go.crypto21:08
perrito666ericsnow: you should not21:14
perrito666at least I dont21:14
jcw4ericsnow: you should run godeps -u dependencies.tsv21:15
jcw4and you may have to go get -u missing repos depending on what godeps tells you21:15
jcw4(if you don't have godeps install with: go get -u code.google.com/p/rog-go/cmd/godeps21:17
jcw4)21:17
ericsnowwas just about to ask that :)21:17
jcw4hmm, but I actually have one on launchpad too... that might be the one to install21:18
jcw4go get -u launchpad.net/godeps21:18
jcw4gah... I can't tell which one... rogpeppe which version of godeps should we install?21:20
jcw4ericsnow: its in CONTRIBUTIING sorry21:21
jcw4install the launchpad version21:21
ericsnowah, hadn't gotten there yet (ran into the errors in README)21:22
jcw4:D21:22
jcw4ericsnow: you know... the new guy always gets asked to update the getting started docs...21:22
ericsnowjcw4: :)21:23
perrito666jcw4: I think natefinch just proposed that fix21:24
perrito666sadly21:25
jcw4haha21:25
perrito666ericsnow: can be worse, I got asked to fix "a small bug" and am right now, a few months after, re-writing the whole feature21:25
* ericsnow didn't see anything21:26
perrito666ericsnow: I think that nate did the fix and forgot to propose it21:27
jcw4I saw a push directly to juju/juju master...21:27
jcw4skipped the review process all together21:27
jcw4:)21:27
perrito666mm... that does sounds like nate confusing origin and upstream again21:28
jcw4Isn't there a git config you can do to prevent that?21:28
jcw4I was trying to figure out how to prevent pushing to my own master branch this morning21:29
perrito666jcw4: a hook perhaps21:30
jcw4yeah...21:30
ericsnowgo is happier after I deleted $GOPATH/pkg/*21:44
perrito666ericsnow: yes, go sometimes has tis issue21:45
perrito666are you kidding me? time.Format takes an... example? as layout?22:07
jcw4perrito666: yep.22:07
jcw4actually pretty cool when you use it for a while22:07
jcw4:)22:08
perrito666jcw4: how does it handle things like yyyymmdd¿?22:08
jcw4forget the exact date but --- 20060102??22:10
jcw4that is the beauty of the template approach... each number in the template is unique so 02 always refers to the day22:13
jcw401 always refers to the month22:13
jcw42006 year22:13
jcw4etc.22:13
ericsnowhow much work should I expect outside https://github.com/juju/juju?22:18
rick_h_ericsnow: ? how much work?22:19
ericsnowright, it seems like there are a bunch of repos (but I expect that most of my time I'll spend in the main juju repo)22:20
rick_h_right now there's some work on splitting the core work into more isolated reusable modules22:20
rick_h_so right now there's not a ton, but I'd expect that to grow over time22:20
* rick_h_ says as someone that's not working on juju core :)22:20
ericsnowexactly what I needed to know :)22:20
perrito666ericsnow: expect to be around github.com/juju/*22:20
perrito666but ymmv22:21
perrito666ok, went far enough with my code today, EOD22:30
perrito666see you all tomorrow cheers22:30
ericsnowperrito666: thanks for your help today22:31
perrito666ericsnow: yw, bye22:31
wallyworldthumper: have you seen this before - adding a lxc container seems to get stuck in a loop doing lxc-ls http://paste.ubuntu.com/7620472/22:32
wallyworldthumper: you around?22:44
thumperwallyworld: hi, yes but otp22:45
wallyworldsure, ok22:45
thumperwallyworld: ok, off calls now23:23
thumperwallyworld: it isn't stuck23:24
wallyworldthumper: so cjohnston  can't start up a container with local provider. logs seems to show a lxc-ls loop? http://paste.ubuntu.com/7620472/23:24
thumperwallyworld: that is the machine agent I thought23:24
thumperwallyworld: I have found a horrible bug in the local provider23:24
wallyworldi would have expected to see lxc-start etc23:24
thumperwhat is the bug cjohnston hits exactly?23:24
thumperit may be it23:24
wallyworldin the logs23:24
wallyworldlet me paste the bug23:25
wallyworldbug 131994723:25
_mup_Bug #1319947: LXC local provider fails to provision precise instances from a trusty host - take 2 <juju-core:Confirmed> <https://launchpad.net/bugs/1319947>23:25
thumperugh...23:25
* thumper guesses...23:25
wallyworldi don't think it's a precise or trusty thing necessarily23:25
thumperno, it is a terrible communication problem23:26
thumperthere is a lock file created when the precise template is being written23:26
thumperit takes quite a while the first time23:26
thumperand people have been known to kill juju while it is doing that23:26
thumperthen they try to start again23:26
thumperbut the lock file is still there23:26
thumperand it is all a bit shit23:27
wallyworldah, that rings a bell23:27
wallyworldso he just needs to remove the lock file?23:27
thumpereasiest solution is to remove the precise template, delete the lock file, and give it 5 minutes the first time23:27
wallyworldremind me the lock file path23:27
thumperthis is why I wanted to do the juju-local plugin to make the template image23:27
* thumper looks in the code23:27
wallyworldsorry, i could have done that too23:27
wallyworldthought you might know ottoyh :-)23:28
cjohnston/var/lock/lxc/var/lib/lxc/juju-precise-template ?23:28
thumper/var/lib/juju/locks23:28
waiganithumper: did you work out notifications for landing PRs?23:28
thumperwaigani: no23:28
waiganiokay, I'm keen to know too :)23:28
wallyworldcjohnston: so can you try what thumper says above and let us know?23:28
thumpercjohnston: I think /var/lib/juju/locks/juju-precise-template23:28
cjohnstonack.. removing the lock.. how to I remove the template?23:29
wallyworlddelete lock file and precise template23:29
thumpercjohnston: I appologise for the shitty experience23:29
thumpersudo lxc-destroy -n juju-precise-template23:29
cjohnston:-)23:29
thumpercjohnston: although I'm assuming you aren't using btrfs23:29
cjohnstonno23:29
thumperalthough that shouldn't matter if you were23:30
thumperas the fs should handle the snapshotting IMO23:30
cjohnstonthumper: http://paste.ubuntu.com/7620671/23:30
thumperwtf?23:31
cjohnstonI try hard :-)23:31
cjohnston(to break things)23:31
rick_h_cjohnston: btrfs?23:32
rick_h_cjohnston: I had that once before due to some btrfs issue23:32
cjohnstonnope23:32
thumpercjohnston: I have no idea why that would fail, I've never had that before23:32
wallyworldrm -rf :-)23:32
thumpercjohnston: it isn't running is it?23:32
thumperalthough should have had a different error if it was23:32
thumpersudo lxc-ls --fancy23:32
cjohnstonSTOPPED23:33
thumperhmm23:33
thumperand lxc-destroy fails?23:33
cjohnstonhrm.. the second time it worked23:33
thumper\o/23:34
thumperkinda23:34
cjohnstonthumper: just do a juju add-machine now and let it hang out for a while?23:35
thumpercjohnston: yep23:35
thumpercjohnston: one thing you could do23:35
thumperis look at the running cloud init23:35
thumperwhich should be in...23:36
thumper/var/lib/juju/containers/juju-precise-template/console.log I think23:36
thumperyep, that one23:37
thumperalthough owned only readable by root23:37
cjohnstonwhat am I looking for there?23:37
thumpercloud-init boot finished at Wed, 05 Mar 2014 21:37:44 +0000. Up 417935.05 seconds23:38
thumperfollowed shortly after by:23:38
thumper * Will now halt23:38
thumperonce that happens, the template has been created, and will then be cloned23:38
cjohnstonok23:38
thumperand expect a few seconds of intense disk I/O23:38
thumperas the new container starts up23:38
thumpershould be much faster after that23:38
cjohnston"1" agent-state: started!23:41
cjohnstonthumper: fwiw, root root 0 May  9 12:09 /var/lib/juju/containers/juju-precise-template/console.log23:42
thumperhmm...23:46
* thumper thinks23:46
thumperI may be looking at an older log file...23:46
thumperI recall doing something strange with it, will have to look23:46
thumpercjohnston: but it is working?23:46
cjohnstonI don't know how much it matters, as things appear to be working now23:46
cjohnstonthanks much for your help thumper and wallyworld !23:49
cjohnstonjcastro: if you are still having problems, read the backscroll23:49
wallyworldthumper did most of it23:49
* wallyworld will close the bug23:50
* thumper goes to the gym23:50
* cjohnston goes to celebrate23:50

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!