[08:09] <rogpeppe2> mornin' all
[14:02] <rogpeppe> can anyone else bootstrap the local environment using trunk?
[14:04] <rogpeppe> i get "restart: Unknown job: rsyslog" in /home/rog/.juju/local/log/cloud-init-output.log
[14:19] <rogpeppe> _thumper_: hiya
[14:19] <_thumper_> oh hai
[14:19] <rogpeppe> _thumper_: the local provider seems broken for me currently on trunk BTW
[14:19] <_thumper_> awesome
[14:19] <rogpeppe> _thumper_: i can't get it to bootstrap at all
[14:19] <_thumper_> wasn't me
[14:19] <thumper> how are you bootstrapping?
[14:20] <rogpeppe> thumper: juju bootstrap -e local --debug
[14:20] <thumper> that should work
[14:20] <rogpeppe> thumper: indeed :-)
[14:20] <thumper> what's the error?
[14:21] <rogpeppe> thumper: 2014-02-03 14:10:34 ERROR juju.cmd supercommand.go:294 exit status 1
[14:21] <rogpeppe> :-)
[14:21] <thumper> that sounds very familiar
[14:21] <thumper> I think I hit that too
[14:21]  * thumper thinks
[14:21] <rogpeppe> thumper: looking in cloud-init-output.log, i see this:
[14:22] <rogpeppe> restart: Unknown job: rsyslog
[14:22] <rogpeppe> thumper: which i think is probably the cause
[14:22] <thumper> wat?
[14:22] <thumper> you should have rsyslog running
[14:22] <thumper> it is part of the OS
[14:22] <thumper> sudo status rsyslog
[14:23] <rogpeppe> thumper: status: Unknown job: rsyslog
[14:23] <natefinch> well there's you're problem
[14:23] <natefinch> your
[14:23] <thumper> haha
[14:23] <thumper> rogpeppe: go
[14:23] <thumper> sudo start rsyslog
[14:23] <thumper> unknown job
[14:23] <thumper> ?
[14:24] <rogpeppe> thumper: yeah
[14:24] <thumper> do you have rsyslog installed?
[14:24] <rogpeppe> start: Unknown job: rsyslog
[14:24] <rogpeppe> thumper: yes
[14:24] <rogpeppe> thumper: and there's a /etc/rsyslog.d directory
[14:24] <rogpeppe> thumper: containing a 25-juju-rog-local.conf file
[14:24] <thumper> rogpeppe: is there /etc/init/rsyslog.conf?
[14:25] <rogpeppe> thumper: yes
[14:25] <thumper> plz run rsyslog
[14:25] <thumper> sudo start rsyslog
[14:25] <rogpeppe> thumper: that doesn't work, as i pasted above
[14:25] <thumper> um...
[14:25] <thumper> bollocks
[14:26] <thumper> it is lying
[14:26] <rogpeppe> thumper: i could try running rsyslogd directly, i suppose
[14:26] <thumper> I wouldn't
[14:26] <thumper> upstart should start it
[14:26] <thumper> actually
[14:26] <thumper> yes
[14:26] <thumper> run it manually
[14:26] <thumper> if it is failing to start, it should tell you why
[14:26] <thumper> but rsyslog should be running
[14:27] <rogpeppe> thumper: rsyslogd is already running
[14:28] <thumper> but upstart doesn't think so?
[14:28] <thumper> that's problematic
[14:28] <rogpeppe> thumper: initctl --list doesn't list it
[14:28] <rogpeppe> thumper: but service --status-all does
[14:28] <thumper> you need help from someone that understands
[14:28]  * rogpeppe doesn't know how initctl (upstart) and service interact
[14:29] <thumper> the fundamental problem is that "sudo restart rsyslog" fails
[14:29] <rogpeppe> thumper: yeah
[14:30]  * rogpeppe straces start rsyslog
[14:32] <rogpeppe> i suppose it might just be trusty brokenness
[14:33] <natefinch> rogpeppe: works for me on trusty, but maybe there's something else going on
[14:34] <rogpeppe> ha, this is interesting:
[14:34] <rogpeppe> % sudo initctl --system start rsyslog
[14:34] <rogpeppe> initctl: Job is already running: rsyslog
[14:34] <rogpeppe> % sudo initctl start rsyslog
[14:35] <rogpeppe> initctl: Unknown job: rsyslog
[14:35] <natefinch> *twilight zone music*
[14:35] <rogpeppe> "initctl --system list" prints loads more jobs
[14:36] <rogpeppe> i wonder what the default, without --system, is doing
[14:37] <rogpeppe> perhaps i've got two /etc/init instances running
[14:37] <rogpeppe> hmm, i have
[14:38] <rogpeppe> /sbin/init and init --user
[14:38] <rogpeppe> i bet that's the issue here
[14:39] <rogpeppe> the init --user is started by lightdm, whatever that is
[14:39]  * rogpeppe googles it
[14:40] <rogpeppe> i wonder if it's related to this: http://lwn.net/Articles/547422/
[14:42] <rogpeppe> natefinch: have you also got an "init --user" process running?
[14:43] <natefinch> rogpeppe:  looking
[14:44] <natefinch> rogpeppe: how can I see what the actually command line was?
[14:44] <rogpeppe> natefinch: i did "ps alxw | grep init"
[14:44] <natefinch> rogpeppe: yep, I have both
[14:45] <rogpeppe> natefinch: hmm, and "initctl list" and "initctl --system list" print the same thing for you?
[14:46] <natefinch> rogpeppe: the --system one lists a hell of a lot more, if that's what you mean
[14:46] <rogpeppe> natefinch: right
[14:46] <rogpeppe> natefinch: what's the output of "initctl list" for you?
[14:47] <natefinch> rogpeppe: http://pastebin.ubuntu.com/6867463/
[14:48] <rogpeppe> natefinch: and "initctl start rsyslog" works for you?
[14:48] <rogpeppe> natefinch: sorry, sudo initctl start rsyslog, of course
[14:48] <natefinch> rogpeppe:  I was gonna say, not without sudo.... yes, with sudo it works fine
[14:49] <rogpeppe> natefinch: weird
[14:49] <rogpeppe> natefinch: what's the output of "sudo strace initctl start rsyslog" for you?
[14:50] <natefinch> rogpeppe: http://pastebin.ubuntu.com/6867473/
[14:51] <rogpeppe> natefinch: thanks
[14:51] <natefinch> rogpeppe: welcome.  Gotta step away from the keyboard for a bit.
[14:51] <rogpeppe> natefinch: k
[14:54] <rogpeppe> natefinch: interesting, yours is opening a different unix socket from mine
[14:54] <rogpeppe> natefinch: yours opens /com/ubuntu/upstart; mine opens /com/ubuntu/upstart-session/1000/3730
[15:12] <rogpeppe> natefinch: issue solved
[15:13] <rogpeppe> local environment bootstrapped, phew
[15:36]  * rogpeppe gets lunch
[16:15] <rogpeppe> natefinch: does the local provider work for you (on tip) ?
[16:15] <rogpeppe> natefinch: i can bootstrap, but not call juju status after that
[16:22] <natefinch> rogpeppe:  lemme try
[16:22] <rogpeppe> natefinch: thanks
[16:26] <natefinch> rogpeppe: ug, annoying.... evidently we now have a check to make sure mongod is in /usr/bin/mongod?  That's not where my mongo is :/
[16:26] <rogpeppe> natefinch: ha ha
[16:26] <natefinch> rogpeppe: I can symlink it, but still, annoying
[16:26] <rogpeppe> natefinch: does it work if you symlink it
[16:26] <rogpeppe> ?
[16:26] <natefinch> rogpeppe: we'll see
[16:32] <natefinch> rogpeppe: sonofa.... man, we gotta fix the local provider.  It's so so so configuration specific.  I forgot it always has problems building jujud because root doesn't have go in its path
[16:32] <rogpeppe> natefinch: you shouldn't run it with sudo now
[16:33] <natefinch> rogpeppe: tried that first - ERROR juju supercommand.go:282 bootstrapping a local environment must be done as root
[16:34] <rogpeppe> natefinch: what revision are you using?
[16:34] <natefinch> rogpeppe: tip.. just pulled and everything
[16:34] <rogpeppe> natefinch: i suspect you haven't go installed everything
[16:35] <rogpeppe> natefinch: because on my machine (rev 2291) it works ok
[16:35] <natefinch> rogpeppe: what besides juju do I need to go install?
[16:35] <natefinch> cmd/juju tha tis
[16:35] <rogpeppe> natefinch: i generally do go install ./cmd/...
[16:35] <rogpeppe> natefinch: or everything
[16:36] <natefinch> rogpeppe: ok, good to know
[16:37] <natefinch> rogpeppe: that'll probably fix the "can't build jujud" thing
[16:37] <rogpeppe> natefinch: maybe
[16:37] <rogpeppe> natefinch: the local provider has a very weird way of finding jujud
[16:38] <rogpeppe> natefinch: it finds juju and then looks in the same directory as that for jujud
[16:38] <rogpeppe> natefinch: so unless they're in the same directory, it won't work
[16:38] <natefinch> rogpeppe: yeah that fixed the "need sudo" problem and thus it was able to build jujud, even though it shouldn't have needed to, since it's in the path
[16:38] <natefinch> rogpeppe: spoke too soon... still ends with needs root
[16:38] <rogpeppe> natefinch: so... can you run "juju status" successfully now?
[16:39] <natefinch> rogpeppe: I can't bootstrap still
[16:40] <rogpeppe> natefinch: are you absolutely sure you're running the right juju binary? (perhaps put a logger.Infof in localEnviron.Bootstrap to make sure)
[16:40] <rogpeppe> natefinch: in the code i see, it calls ensureNotRoot to make sure that you are not running as root
[16:40] <natefinch> rogpeppe: ahh, no, there's a juju in /bin/juju somehow
[16:40] <rogpeppe> natefinch: right
[16:40] <natefinch> rogpeppe: I don't know how that got there
[16:41] <rogpeppe> natefinch: when was it created?
[16:41] <rogpeppe> natefinch: hmm, /bin, not /usr/bin ?
[16:41] <natefinch> rogpeppe: yeah, just /bin/juju  .... created Dec 5th
[16:42] <rogpeppe> natefinch: weird.
[16:42] <natefinch> yep
[16:42] <rogpeppe> natefinch: well, remove it :-)
[16:43] <natefinch> wtf?  Why is bash still trying to run juju from /bin/juju if which juju returns $GOPATH/bin/juju?
[16:45] <rogpeppe> natefinch: run hash -r
[16:45] <rogpeppe> natefinch: it's an ancient (and fucked up) shell misfeature
[16:45] <natefinch> rogpeppe: yeah, I just made a new terminal and it's fine
[16:45] <rogpeppe> natefinch: and the source of endless wtfs in the past
[16:46] <rogpeppe> natefinch: it's a premature optimisation
[16:46] <natefinch> man, that is dumb
[16:46] <natefinch> rogpeppe: I mean, I get it, but.... yeah
[16:47] <rogpeppe> natefinch: blame csh back in the 80s
[16:47] <natefinch> rog ERROR juju.cmd supercommand.go:294 Get http://10.0.3.1:8040/provider-state: dial tcp 10.0.3.1:8040: connection refused
[16:49] <rogpeppe> natefinch: you're best deleting any existing stuff. rm -r ~/.juju/environments/local.jenv ~/.juju/local
[16:49] <rogpeppe> natefinch: that behaviour will get better, hopefully
[16:49] <rogpeppe> natefinch: axw has a CL waiting
[16:50] <natefinch> rogpeppe: and finally... it seems to work fine
[16:50] <natefinch> rogpeppe: juju status that is .... er, agent state is down, hmm
[16:50] <rogpeppe> natefinch: juju status works ok, not running as root?
[16:51] <natefinch> rogpeppe: I was too quick for it, now it's up.   Yes, not running as root
[16:51] <rogpeppe> natefinch: i get this: ERROR failed getting all instances: error executing "lxc-ls": lxc: Permission denied - opendir on lxcpath; Traceback (most recent call last):;   File "/usr/bin/lxc-ls", line 200, in <module>;     for container_name in lxc.list_containers(config_path=lxcpath):;   File "/usr/lib/python3/dist-packages/lxc/__init__.py", line 390, in list_containers;     config_path=config_path); ValueError: failure to list containers
[16:52] <natefinch> rogpeppe: what happens if you just run lxc-ls?
[16:52] <rogpeppe> natefinch: yeah, just tried that - that fails too
[16:52] <natefinch> rogpeppe: that works for me (well, returns with no output, since I have no containers yet)
[16:53] <rogpeppe> natefinch: it would be wonderful if it actually printed the path in question :-)
[16:54] <natefinch> rogpeppe: that's one of the things I like best about go... all those type of errors actually include the path
[16:56] <rogpeppe> natefinch: ah, i guess i've found the reason
[16:56] <rogpeppe> natefinch: /var/lib/lxc is mode 700
[16:57] <natefinch> rogpeppe: uhh.... that does seem like a problem :)
[16:57] <rogpeppe> natefinch: and... now it works
[16:57] <rogpeppe> natefinch: i wonder how that happened
[16:58] <natefinch> rogpeppe: actually, mine is also 700
[16:58] <rogpeppe> natefinch: oh, maybe i shouldn't have chmodded mine then
[16:58] <rogpeppe> sigh
[17:02] <rogpeppe> WTFs/minute is running rather high currently
[18:47] <natefinch> rogpeppe: got a second? I have a question about the simpleworker that you wrote...
[18:47] <rogpeppe> natefinch: sure
[18:47]  * rogpeppe tries to remember what "simpleworker" was
[18:47] <natefinch> rogpeppe:  I can pastbin
[18:47] <rogpeppe> natefinch: please
[18:48] <natefinch> rogpeppe: http://pastebin.ubuntu.com/6868675/
[18:49] <rogpeppe> natefinch: ah yes, i remember now
[18:49] <rogpeppe> natefinch: it is perhaps a little over-simplistic
[18:49] <rogpeppe> natefinch: it might need to cache the error, for example
[18:50] <natefinch> rogpeppe: the kill function, as written, doesn't compile, but I'm not clear on what is supposed to be there
[18:50] <rogpeppe> natefinch: close(w.stopc)
[18:51] <natefinch> rogpeppe: ok... I was trying to follow it, but it was all a little too self-referential and relying on the behavior of outside stuff that I didn't understand
[18:52] <rogpeppe> natefinch: it's just a way of writing a lightweight worker without making a new type
[18:52] <rogpeppe> natefinch: i should probably have written some comments :-)
[18:54] <natefinch> rogpeppe: heh...
[18:58] <rogpeppe> natefinch: this is probably a little better: http://play.golang.org/p/zGQUsdR5Yy
[18:59] <rogpeppe> natefinch: (it makes both Wait and Kill idempotent)
[19:05] <natefinch> rogpeppe: so in that code, the fact that done is a channel doesn't really matter, right?  Since nothing is ever sent on it, it just makes Wait() block until done is closed
[19:05] <rogpeppe> natefinch: that's right
[19:06] <rogpeppe> natefinch: it's a fairly idiomatic use of a chan struct{}
[19:08] <natefinch> rogpeppe: yeah. I
[19:09] <natefinch> rogpeppe: I'm a little confused, because stopc is never used
[19:09] <natefinch> rogpeppe: I mean, we send it to the function, but we don't send or receive on it ourselves
[19:09] <rogpeppe> natefinch: that's the point
[19:09] <rogpeppe> natefinch: the function can use it to find out when it's being stopped
[19:10] <natefinch> rogpeppe: oh, I see, I missed that we're still closing it.  ok
[19:10] <rogpeppe> natefinch: and ISTR that the code i pasted to did
[19:11] <rogpeppe> s/pasted to did/pasted did that/
[19:13] <natefinch> rogpeppe: yeah.... ok. It's making more sense now.  your new code is a little easier for me to follow
[19:13] <rogpeppe> natefinch: the comments help, right?
[19:13] <rogpeppe> natefinch: (the code is actually more complex than it was before)
[19:15] <natefinch> rogpeppe: I think it was mostly the    w.done <- start(w.stopc)      line of the old code that was confusing me... calling a function passed into this function in a goroutine, passing it a channel and sending what that function returns into another channel....
[19:16] <natefinch> rogpeppe: cognitively dense
[19:17] <natefinch> rogpeppe: knowing that I can just ignore the channels except as signaling devices also helped
[19:17] <rogpeppe> natefinch: good thing we're not using haskell :-)
[19:17] <natefinch> rogpeppe: I'd be working somewhere else if we did :)
[19:23]  * rogpeppe finishes for the day
[19:23] <rogpeppe> natefinch: g'night
[19:24] <natefinch> rogpeppe: g'night.  Thanks for the help.