[01:11] <axw> hey thumper, have a nice break?
[01:11] <axw> anything I can help with on the maas thing?
[01:12] <thumper> axw: yeah, nice to get that mental break
[01:13] <thumper> axw: I'm trying to work out why maas is having bootstrap issues, and ec2 and openstack not
[01:14] <axw> thumper: I guess I won't be of use without maas knowledge, but give me a yell if there's something I can do.
[01:14] <thumper> sure
[01:14] <bigjools> this is a bit of a critical problem
[01:14] <thumper> bigjools: right
[01:14] <bigjools> thumper: forwarding you another email
[01:14] <thumper> bigjools: do you have any logs of a failed bootstrap?
[01:15] <bigjools> no, I am going to start poking soon
[01:15] <thumper> in particular, I want everything related to the cloud-=init
[01:15] <bigjools> did rvb's email not cover it?
[01:15] <bigjools> send you two emails
[01:16] <bigjools> sent*
[01:16] <bigjools> there's two problems I see at the moment
[01:16] <thumper> bigjools: the pastebin (http://paste.ubuntu.com/6226420/) had  two lines but I don't know what from
[01:16] <bigjools> 1. bootstrap failing as per rvb
[01:16] <bigjools> it's the QA lab environment
[01:16] <bigjools> 2. destroy-env gets its knickers in a twist
[01:17] <bigjools> I reckon the pastebin shows an invalid command line quite frankly
[01:17] <bigjools> --constraints is not getting any arg
[01:17] <thumper> locally, it gets '' if not set
[01:17] <thumper> I'm not sure how it ends up getting missed on the command line
[01:17] <thumper> I can submit a fix *right now* for the empty one
[01:18] <thumper> but I was really wanting to know why we didn't see issues on the ec2/openstack side
[01:18] <bigjools> NFI
[01:18] <thumper> yeah
[01:18] <thumper> me neither
[01:18]  * thumper bootstraps an ec2
[01:19] <bigjools> something is causing --constraints to get passed in with no arg, but I don't know what would do that
[01:19] <bigjools> passed in to jujud that is
[01:19] <thumper> hmm...
[01:19] <thumper> well,
[01:19] <thumper> when we generate the cloudinit
[01:20] <thumper> it makes" --constraints ''   " if empty
[01:20] <thumper> so
[01:20]  * thumper shrugs
[01:20] <bigjools> ok I am going to recreate locally
[01:20] <thumper> why is the version of in lp:juju-core/1.16 1.15?
[01:21] <bigjools> ?
[01:21] <bigjools> I have a sneaking suspicion that the tools are mismatched somewhere
[01:21] <thumper> nah
[01:21] <thumper> well...
[01:21] <thumper> actually
[01:23] <thumper> hmm...
[01:23] <thumper> ah...
[01:23] <thumper> nope, my fault there
[01:24]  * thumper bootstraps again
[01:24] <thumper> bigjools: got time for a hangout?
[01:24] <thumper> bigjools: is your shitty connection up to it?
[01:24] <bigjools> yeah it's ok now
[01:25] <bigjools> I finally plugged in my ADSL filter and it's better :)
[01:25] <thumper> haha
[01:25] <thumper> bigjools: hangout?
[01:25] <bigjools> yea just call me
[01:25] <thumper> kk
[01:27] <axw> thumper: re maas vs. ec2/openstack, I *suspect* it's due to https://codereview.appspot.com/13802045/patch/36001/37002
[01:27] <axw> see IsEmpty
[01:27]  * thumper looks
[01:27] <axw> and parseTags
[01:27] <axw> bit of a wild guess tho
[01:29] <axw> hmm tags= shouldn't be in the string tho, so maybe not
[01:41] <thumper> this isn't about tags, but constraints
[01:43] <axw> thumper: yeah... there's tags in constraints now. was thinking it might've affected whether the constraints are considered empty or not
[02:48] <bigjools> thumper: https://bugs.launchpad.net/juju-core/+bug/1239496
[02:48] <_mup_> Bug #1239496: Packaged 1.16 juju has oauth errors talking to maas <juju-core:New> <https://launchpad.net/bugs/1239496>
[02:48] <bigjools> rather critical
[02:52] <bigjools> thumper: http://pastebin.ubuntu.com/6234274/
[02:52] <bigjools> wallyworld_'s fault you say?
[02:52] <wallyworld_> wot
[02:53] <bigjools> hang on
[02:54] <wallyworld_> you are trying to bootstrap using a 1.9 client?
[02:54] <bigjools> trying to work out what TF I am building
[02:54] <bigjools> I rebuilt and it said I am using 1.9
[02:55] <wallyworld_> pbkac
[02:55] <bigjools> trunk branch moved ...
[02:55] <bigjools> old checkout
[03:05] <thumper> axw: you could look at this https://codereview.appspot.com/14494054
[03:06] <axw> thumper: will do
[03:06] <thumper> hang on
[03:06] <thumper> that's bollocks
[03:06] <thumper> WTF went on there then
[03:06] <thumper> I branched from 1.16
[03:06] <thumper> and proposed to trunk
[03:09] <bigjools> ok oauth in latest core is bollixed
[03:09] <bigjools> https://bugs.launchpad.net/juju-core/+bug/1239496
[03:09] <_mup_> Bug #1239496: 1.16 juju and above has oauth errors talking to maas <juju-core:New> <https://launchpad.net/bugs/1239496>
[03:16] <davecheney> bigjools: which series are you running
[03:16] <davecheney> i wonder if this is 'compiled against an old versino of go' again
[03:16] <bigjools> saucy
[03:16] <davecheney> hmm
[03:16] <davecheney> nope, not that the
[03:16] <davecheney> then
[03:16] <thumper> davecheney: bigjools says the package version is poked
[03:17] <bigjools> package in saucy itself is bollixed
[03:17] <davecheney> bigjools: ok
[03:17] <davecheney> what if you build from the 1.16.0 tarball
[03:17] <davecheney> download it
[03:17] <davecheney> set yoru GOPATH to the root of the tar
[03:17] <davecheney> go install ./...
[03:17] <davecheney> that is the litmus test
[03:17] <bigjools> where is it again?
[03:17] <davecheney> gotta go to the 1.16 series
[03:18] <davecheney> https://launchpad.net/juju-core/1.16/1.16.0/+download/juju-core_1.16.0.tar.gz
[03:18] <bigjools> ta
[03:18] <thumper> axw: https://codereview.appspot.com/14494054/ diff is now right
[03:18] <axw> thumper: ta, reading
[03:19] <thumper> but still need to target one against 1.16
[03:19] <axw> ok
[03:19] <axw> bigjools: did you also notice that the realm is different in the working/non-working tests?
[03:19] <bigjools> axw: yes
[03:19] <bigjools> maas has a log entry saying "invalid consumer"
[03:20] <thumper> axw: is the realm used in anyway to change the header values?
[03:21] <bigjools> src/code.google.com/p/go.net/ipv6/sockopt_rfc3542_linux.go:33: too many errors
[03:21] <bigjools> nice
[03:21] <axw> thumper: not that I've noticed so far
[03:21] <axw> thumper: I mean, apart from the "realm" part in the HTTP header
[03:22] <axw> one has realm="", one has realm="MAAS+API"
[03:22] <bigjools> davecheney: still got a 401 with that tarball
[03:22] <axw> which is URL-encoded (in code it's "MAAS API")
[03:23] <bigjools> davecheney: it prompts another question - why did Mark's package work
[03:25]  * bigjools grabs food
[03:26] <davecheney> bigjools: is he using saucy ?
[03:26] <axw> thumper: lgtm
[03:26] <thumper> axw: ta
[03:47] <bigjools> davecheney: yes from what I can tell, his juju --version showed 1.16.0-saucy-amd64
[03:51] <davecheney> ok
[03:51] <davecheney> just checking
[03:51] <davecheney> in case he was running precise
[03:53] <bigjools> thumper: did you end up uploading a new binaryu for me?
[03:53] <thumper> bigjools: yes, I copied the one I build into the ~/tim dir
[03:53] <bigjools> ok ta let's try it
[03:54] <bigjools> it gets the 401 too
[03:55] <thumper> wtf?
[03:56]  * thumper goes to do some bzr magic to see the changes to the maas provider between 1.14 and 1.16
[03:56] <thumper> bigjools: could it be gomaasapi?
[03:56] <thumper> davecheney: do we know which revisions of gomaasapi were used for 1.14?
[03:57] <thumper> davecheney: we have tarballs, right?
[03:57] <davecheney> thumper: it's encoded in the release-tarball script on the 1.14 branch
[03:57] <bigjools> thumper: not sure it changed.  Oauth stuff is handled in Go's library
[03:58] <davecheney> there will also be a tag on gomaasapi
[03:58]  * thumper grabs both release tarballs
[03:58] <thumper> bigjools: wondering why sabdfl didn't hit that problem...
[03:59] <bigjools> thumper: exactly
[03:59] <bigjools> thumper: nor in the lab
[03:59] <bigjools> is it only on my box?  if so why does an old version work
[04:05] <thumper> bigjools: yes, very weird
[04:05] <thumper> nothing oauth related has changed in gomaasapi nor provider/maas in the 1.14.1 -> 1.16 change
[04:05] <thumper> we need another maas to test against
[04:05] <bigjools> I was gonna say
[04:06] <bigjools> easy for you to do that
[04:06] <bigjools> you don't need nodes
[04:06] <thumper> bigjools: how?
[04:07] <bigjools> run it from a branch or a package; all you need to do is "juju destroy-environment"
[04:07] <bigjools> and if buggered you get a 401
[04:07] <bigjools> you can even do it on canonistack
[04:07] <thumper> say what?
[04:20] <thumper> axw: and now the 1.16 branch version, with other bug references: https://codereview.appspot.com/14516056/
[04:20] <axw> thumper: just lgtm'd
[04:20] <thumper> ta
[04:20] <axw> thanks for logging the bugs
[04:21] <thumper> np
[05:05] <bigjools> juju is now an easier interface to openstack than the other command line tools
[05:07] <davecheney> :)
[05:07] <davecheney> juju deploy -n 1000 ubuntu
[05:07] <bigjools> well even for adding arbitrary instances
[05:07] <bigjools> I just use bootstrap and abuse it, or add-machine
[05:10] <bigjools> davecheney: on that note, how do I force a particular distroseries with add-machine?
[05:11] <bigjools> oh --series ... duh
[06:04] <bigjools> davecheney: so for some odd reason that oauth problem is only on my maas test box at home
[06:22] <davecheney> does oauth use the hostname of the server ?
[06:24] <bigjools> no idea
[06:24] <bigjools> the server is called "maas" so if that's causing problems I think we can all quit now
[06:25]  * davecheney packs his bags
[06:34] <bigjools> axw, davecheney: if you can think of anything, anything at all, why the Auth header would be screwy only one one system.... I would be rather grateful.
[06:39] <axw> bigjools: what's the diff in Auth header with the same juju, in the two envs?
[06:39] <bigjools> axw: https://bugs.launchpad.net/juju-core/+bug/1239496
[06:39] <_mup_> Bug #1239496: 1.16 juju and above has oauth errors talking to maas <juju-core:New> <https://launchpad.net/bugs/1239496>
[06:40] <bigjools> older juju binaries work fine
[06:40] <bigjools> so something that is compiled into it, possibly from the stdlib, is behaving differently because of some environmental difference
[06:41] <axw> bigjools: do you have a capture of the auth header from an older juju?
[06:41] <bigjools> I'll get one hang on
[06:42] <bigjools> oh crap overwrote the old binary
[06:42] <bigjools> I'll grab a tarball and rebuild
[06:43] <axw> bigjools: you didn't happen to edit environments.yaml after bootstrapping, did you?
[06:43] <bigjools> nope
[06:43] <axw> k
[06:43] <axw> just asking because environments.yaml gets cached in these .jenv files now
[06:44] <bigjools> O_o
[06:44] <axw> e.g. ~/.juju/environments/maas.jenv
[06:44] <bigjools> what's that for?
[06:44] <axw> so we can auto generate stuff like admin-secret
[06:44] <axw> not sure what else tbh
[06:45] <axw> did you say you got this error when you did a destroy-environment, without first bootstrapping?
[06:45] <axw> or did I imagine that
[06:48] <bigjools> yeah
[06:48] <bigjools> an easy way to test
[06:48] <bigjools> it tries to list nodes and gets a 401
[06:48] <bigjools> but any command gets the same response
[06:52] <bigjools> axw, davecheney: so I can recreate with the 1.15 release, 1.13 does not have  the problem, let me just find the header
[06:54] <bigjools> axw: I updated the bug
[06:54] <axw> ta
[06:56] <axw> bigjools: the only thing that jumps out at me is the non-empty realm in 1.16
[06:56] <axw> both others have realm=""
[06:56] <bigjools> axw: see the latest one
[06:57] <axw> ok
[06:58] <bigjools> axw: ah sorry ignore the realm=""
[06:58] <bigjools> I posted the wrong request on the comment
[06:59] <axw> ok, no worries
[07:00] <bigjools> the ones that don't work use completely the wrong token
[07:01] <bigjools> axw: how do I clear that jenv?
[07:02] <bigjools> axw: because I can see from examining it the oauth is different ...
[07:02] <axw> bigjools: it gets created when you bootstrap
[07:02] <bigjools> so it looks like I did change the envionments.yaml at some point
[07:02] <axw> ok, that would explain it
[07:02] <bigjools> this is a bug in juju IMO - I should be able to change the credentials
[07:02] <bigjools> if my maas server was compromised, I need to change them
[07:03] <axw> yeah, I'm not sure what the supported method of doing that is
[07:04] <axw> you'd need to update the state db I think, via set-environment
[07:05] <axw> whatever's in the .jenv should just be used for connecting to the API server or state server I think.
[07:05] <bigjools> I hacked the jenv
[07:05] <bigjools> works that w3ay
[07:05] <bigjools> not ideal if you don't know about it :/
[07:05] <axw> yeah, I was banging my head against that for a while when dev/testing the null provider :)
[07:06] <axw> perhaps we should check contents of environments.yaml and .jenv, and make sure there's no changes
[07:07] <bigjools> good idea
[07:07] <axw> I'll log a new bug
[07:07] <bigjools> thank you
[07:08] <bigjools> and thanks for the help
[07:08] <axw> no worries
[07:09] <rvba> thumper-afk: I'm testing your fix for the empty constraints stuff in the lab now.
[07:10] <bigjools> I am re-creating that here as well
[07:11] <rvba> thedac: Looks like the fix for that worked (no empty --constraints any more) but jujud is still not starting properly: http://paste.ubuntu.com/6234820/
[07:12] <rvba> arg, sorry thedac, I meant thumper-afk.
[07:37] <bigjools> --upload-tools seems to cause bootstrap to break, at least on maas
[07:37] <bigjools> http://paste.ubuntu.com/6234893/
[07:39] <davecheney> bigjools:  i don't see anythig in that log that it is actually uploading anything
[07:39] <davecheney> that would be the problem
[07:39] <rvba> fwiw, I've this error (EOF) happening from time to time, sometimes when fetching the tools from aws.  But it was always transient.  Any idea?
[07:39] <bigjools> davecheney: the problem is that bootstrap has not done anything!
[07:39] <bigjools> getting that EOF error every time
[07:42]  * thumper looks at rvba's pastebin
[07:42] <rogpeppe> mornin'
[07:42] <rogpeppe> thumper: hiya
[07:42] <rvba> thumper: see my most recent email to the ML to have all the logs.
[07:42] <rvba> Morning rogpeppe.
[07:42] <thumper> rvba: ack
[07:42] <thumper> hi rogpeppe
[07:43] <rogpeppe> rvba: yo!
[07:43] <davecheney> i thin that eof indicates it ran off the end of whatever it was asking maas to list
[07:43] <bigjools> https://bugs.launchpad.net/juju-core/+bug/1239558
[07:43] <_mup_> Bug #1239558: --upload-tools failure preventing bootstrap completing <juju-core:New> <https://launchpad.net/bugs/1239558>
[07:43] <thumper> rvba: do you have access to the userdata.txt for the cloud image boot?
[07:43] <thumper> rvba: on ec2 found it in /var/cloud/... somewhere
[07:43] <thumper> sorry
[07:43] <thumper> /var/lib/cloud
[07:44] <rvba> thumper: sure, let me get that for you…
[07:44] <davecheney> /var/lib/cloud/instance/
[07:44] <davecheney> one of those is the text
[07:44] <thumper> rvba: also, log for mongod
[07:44] <thumper> rvba: the one that is the juju one
[07:45]  * thumper looks for the init script name
[07:45] <bigjools> eek, why are tools for quantal getting uploaded by default
[07:45] <rvba> /var/lib/cloud/instance/cloud-config.txt: http://paste.ubuntu.com/6234904/
[07:45] <rvba> /var/log/mongodb/ is empty (!)
[07:46]  * bigjools heading out to eat, back later
[07:46] <rvba> ps aux | grep mongo → zero
[07:46] <thumper> WTF?
[07:46] <thumper> rvba: it seems that we have jujud trying to start before mongod, that would certainly be an error :)
[07:46] <rvba> http://paste.ubuntu.com/6234911/
[07:47] <rvba> Mongodb is not installed
[07:47] <thumper> rvba: what about mongodb-server
[07:47] <rvba> 1:2.0.4-1ubuntu2.1 is installed
[07:48] <thumper> hmm..
[07:48] <thumper> that is the wrong version
[07:48] <rvba>  sudo service mongodb status → mongodb stop/waiting
[07:48] <thumper> the cloud-init has removed the stable ppa that has the mongodb-server package
[07:49] <thumper> rvba: what is the install source of the mongodb server package
[07:49] <thumper> ?
[07:50] <rvba> thumper: http://paste.ubuntu.com/6234922/
[07:50] <thumper> rvba: the juju mongo service is "juju-db"
[07:50] <rvba> arg, one sec…
[07:50] <rvba> thumper: http://paste.ubuntu.com/6234924/
[07:50] <rvba> thumper: ah ok
[07:50] <thumper> rvba: but it will be trying to start with ssl
[07:51] <thumper> and failing
[07:51] <rvba> $ sudo service juju-db status → juju-db stop/waiting
[07:51] <davecheney> rvba: did you see my reply ?
[07:51] <davecheney> Get:17 http://archive.ubuntu.com/ubuntu/ precise-updates/universe mongodb-clients amd64 1:2.0.4-1ubuntu2.1 [16.5 MB]
[07:51] <davecheney> Get:18 http://archive.ubuntu.com/ubuntu/ precise-updates/universe mongodb-server amd64 1:2.0.4-1ubuntu2.1 [4167 kB]
[07:51] <davecheney> ^ ppa:juju/stable is missing from this bootstrap node
[07:51] <davecheney> it will never bootstrap properly
[07:51] <thumper> rvba: and it seems to be logging to syslog
[07:51] <thumper> davecheney: agreed
[07:51] <rvba> davecheney: ah ok
[07:51]  * thumper looks at the 1.16 cloud-init work
[07:52] <davecheney> did someone try to remove ppa:juju/stable from our cloud-init scripts again ?
[07:52] <davecheney> i recall a change recently
[07:52] <thumper> yeah, me too
[07:52] <thumper> that was to add the cloud archive
[07:52] <thumper> I wonder if they removed the old
[07:52]  * thumper greps the bzr logs
[07:53]  * davecheney flips a table
[07:53] <davecheney> jamespage: is the mongodb update available in the cloud tools pocket ?
[07:53] <thumper> ok found the change
[07:53] <thumper> axw: ping
[07:54] <axw> thumper: yo
[07:54] <thumper> hey
[07:54] <thumper> axw: r1948 where you added cloud-tools
[07:54] <thumper> you modified the apt-repository bit for the ppa/stable
[07:55]  * axw brings up the diff
[07:55] <davecheney> thumper: axw just get the mongo depdency installed into the cloud-tools pocket
[07:55] <davecheney> that'll fix it today
[07:55] <davecheney> no release required
[07:56] <axw> last I checked, mongo was in cloud-tools
[07:56] <jamespage> davecheney, its already in the cloud-tools pocket
[07:56] <jamespage> but the cloud archive is just for 12.04
[07:56] <thumper> axw: http://paste.ubuntu.com/6234904/
[07:56] <thumper> axw: the cloud-init userdata doesn't add ppa:juju/stable
[07:56] <thumper> axw: also, it uses mongodb-server
[07:56] <thumper> not mongo
[07:56] <TheMue> morning
[07:57] <thumper> hmm...
[07:57] <thumper> mongodb-server is there
[07:58] <axw> thumper: I tested this live
[07:58] <axw> it's in  both ppa and pocket
[07:58]  * thumper wonders what was booted with maax
[07:59] <thumper> jamespage: it appears to be precise
[07:59]  * thumper ponders
[07:59] <thumper> ah...
[07:59] <thumper> here it goes
[07:59] <thumper> W: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages  403  Forbidden
[07:59] <thumper> E: Some index files failed to download. They have been ignored, or old ones used instead.
[07:59] <axw> yeah just saw that
[08:00] <jamespage> thumper, some sort of proxy blocking access?
[08:00] <axw> what's up with that?
[08:00] <thumper> rvba: can the machine see it?
[08:00]  * rvba tests.
[08:00] <thumper> jamespage: qa lab is firewalled off I think
[08:00] <thumper> rvba: you'll need to get a hole blased to that location
[08:00] <rvba> thumper: that machine can access the file all right.
[08:01] <rvba> (i.e. wget http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages works)
[08:01] <thumper> rvba: can you do an update, upgrade?
[08:01] <thumper> intermittant blockage?
[08:02] <rvba> hum, apt-get update → Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages
[08:02] <thumper> yeah... that'll bit it :)
[08:02] <thumper> jamespage: do you know what has changed with the cloud-init process?
[08:02]  * rvba wonders why wget works.
[08:03] <thumper> something has changed to how it executes the scripts
[08:03] <thumper> in particular the --constraints '' used to work
[08:03] <thumper> and now it doesn't
[08:03] <thumper> the empty two single quote param was getting dropped somewhere
[08:04] <thumper> hmm...
[08:04] <thumper> or perhaps...
[08:04] <thumper> it is just how it is logged out
[08:04] <thumper> and copying that got a different problem
[08:04] <thumper> that would kinda explain it
[08:04] <jamespage> thumper, sorry - no I don't
[08:04] <rvba> Here is goes, apt is configured to use the region's squid as a proxy and that is failing to fetch things from ubuntu-cloud.archive.canonical.com.
[08:05] <thumper> jamespage: doesn't matter, I think I explained it to myself
[08:13] <rvba> thumper: okay, I fixed the proxy in the lab, let's try again to bootstrap a node…
[08:13] <rvba> (with your empty constraints fix)
[08:14] <thumper> rvba: I'd be curious to see it without my fix
[08:14] <thumper> but lets try with first :)
[08:20] <rvba> thumper: node got bootstrapped okay.
[08:22] <bigjools> I recreated it without any fix
[08:22] <thumper> rvba: \o/
[08:24] <thumper> bigjools: did you say you got one bootstrapping without the fix?
[08:24] <bigjools> thumper: I recreated the bug
[08:24] <bigjools> ie. fail
[08:24] <thumper> oh
[08:24] <bigjools> I can retry with the fix
[08:24] <thumper> bigjools: recreated the bug locally?
[08:25] <rvba> thumper: testing again without your fix, just to confirm.
[08:25] <bigjools> thumper: yes
[08:25] <thumper> bigjools: is it the same? no access to the cloud-tools ?
[08:25] <thumper> or the bad command line parsing?
[08:25] <thumper> I'm confused as to how the command-line parsing is different
[08:25] <bigjools> thumper: the command line thing with the --constraint
[08:25] <thumper> really?
[08:25] <thumper> wow
[08:26] <thumper> how does the cloud-init get run in the maas environment?
[08:26] <bigjools> something is arsed in the core
[08:26] <bigjools> it runs as part of the image
[08:26] <bigjools> takes params from kernel cmd line
[08:26] <bigjools> then downloads stuff from the metadata service
[08:26] <thumper> bigjools: no, the cloud-init data is good
[08:27] <thumper> bigjools: something has changed in how it is run
[08:31] <thumper> bigjools: for the failing bootstrap with the constraint error, can I see the cloud-init userdata file?
[08:32] <bigjools> thumper: we're on our call, hang on
[08:32] <rogpeppe> thumper: what's the problem here?
[08:33] <thumper> rogpeppe: it seems that the maas execution of the cloud-init user data was dropping an empty string param
[08:33] <thumper> rogpeppe: causing the jujud bootstrap-state command to fail
[08:33] <thumper> rogpeppe: I'm gathering data
[08:34] <thumper> rogpeppe: landed a fix for it by not passing constraints where there are none
[08:34] <rogpeppe> thumper: this doesn't happen on ec2?
[08:34] <thumper> but I'd like to know why it is happening
[08:34] <thumper> no, ec2 is fine
[08:34] <rogpeppe> thumper: hmm, odd.
[08:34] <thumper> yeah
[08:35] <rvba> thumper: the node bootstrapped okay *without* your fix this time… I'm pretty confused…  maybe the wrong juju was used somehow…
[08:35] <thumper> hmm…
[08:35] <thumper> even more curious about bigjools's cloud-init issue now
[08:35] <rogpeppe> it's a pity that set -x doesn't print quoted commands correctly
[08:36] <thumper> yeah
[08:36] <rogpeppe> thumper: still, you should see an extra space in the log
[08:36] <rogpeppe> thumper: if the command was really there
[08:37] <rogpeppe> thumper: actually, another possible fix: use --constraints=foo rather than --constraints foo
[08:37] <rogpeppe> thumper: well, it would be interesting to see if that made any difference, anyway
[08:37]  * thumper shrugs
[08:37] <thumper> slightly cleaner not to pass through nothing IMO
[08:37] <rogpeppe> thumper: and therefore where it was a quoting issue or not
[08:38]  * thumper nods
[08:38] <thumper> I'm hesitant to think it is a quoting issue
[08:38] <thumper> I'm thinking user error, but want more data :)
[08:39] <rogpeppe> thumper: why do you think the problem was a dropped empty string param?
[08:40] <thumper> rogpeppe: because I didn't read the email properly
[08:40] <thumper> :)
[08:40] <rogpeppe> thumper: heh
[08:40] <thumper> perhaps not enough coffee
[08:40] <thumper> or alcohol
[08:40] <thumper> or something
[08:41] <rogpeppe> thumper: what kind of user error do you think might have caused the problem?
[08:45] <bigjools> thumper: ok sorry I destroyed-env before you asked :(
[08:46] <thumper> :(
[08:46] <thumper> bigjools: do it again :)
[08:46] <bigjools> can do, will leave it running while I deal with kids bedtime
[08:46] <thumper> ta
[08:46] <bigjools> bloody annoying that I get that EOF when using --upload-tools
[08:47] <bigjools> can you fix that too, kthxbai
[08:47] <thumper> wallyworld_: ^^
[08:48] <bigjools> at least is there a way to stop it downloading the world's tools?
[08:48] <bigjools> I only need one set
[08:57] <wallyworld_> reading backscroll, what's the issue?
[08:58] <wallyworld_> EOF uploading tools?
[09:00] <wallyworld_> that usually means the source tools can't be found
[09:00] <wallyworld_> which means the compile failed perhaps
[09:00] <wallyworld_> i'd need to see some logs etc
[09:03] <bigjools> wallyworld_: https://bugs.launchpad.net/juju-core/+bug/1239558
[09:03] <_mup_> Bug #1239558: --upload-tools failure preventing bootstrap completing <juju-core:New> <https://launchpad.net/bugs/1239558>
[09:04] <bigjools> thumper: ok where's the file you want?
[09:05] <wallyworld_> bigjools: it's not upload tools per se - it;s the maas storage provider returning an EOF when a file is requested
[09:06] <wallyworld_> sorry, i mean a list() is performed
[09:06] <wallyworld_> the expected behaviour for ec2/openstack etc is to return an empty list if there are no such files
[09:06] <wallyworld_> i guess maas is different :-(
[09:06] <bigjools> \o/
[09:07] <bigjools> maas itself returns an empty list
[09:07] <bigjools> so I guess the provider needs changing
[09:07] <wallyworld_> i'll have to look a little closer into it
[09:07] <wallyworld_> upload-tools han't changed really
[09:08] <wallyworld_> i wonder how it worked before
[09:08] <bigjools> pixie dust
[09:08] <wallyworld_> the main change is that upload-tools runs automatically if tools cannot be found
[09:08] <bigjools> and tears drawn from the cheeks of juju developers
[09:09] <wallyworld_> bigjools: you running this against your local maas?
[09:09] <bigjools> wallyworld_: Yes. I just wanted to avoid the long wait while it downloads 8 sets of tools
[09:09] <bigjools> thumper: speak to me!
[09:10] <wallyworld_> bigjools: maybe you can slip me some creds and i'll try against that too?
[09:10] <bigjools> slip you what?
[09:10] <wallyworld_> whatever creds i need to add to my env.yaml
[09:10] <wallyworld_> so i can try bootstrapping on your maas
[09:10] <bigjools> that's not going to help, you need ssh access to my server
[09:10]  * thumper looks up at bigjools
[09:11] <thumper> bigjools: the user data from /var/lib/cloud
[09:11] <bigjools> which I can give you, I'll add your ssh key in one moment once I've given thumper my load
[09:11] <thumper> I don't know exactly where it is
[09:11] <wallyworld_> bigjools: so i need to ssh in and run juju from there?
[09:12] <bigjools> wallyworld_: yep
[09:12] <wallyworld_> ok, so any debugging, i need to checkout juju src etc without my lovely ide :-(
[09:12] <bigjools> thumper: the actual data has this:
[09:12] <bigjools> --constraints '' --debug
[09:13] <thumper> right
[09:13] <bigjools> thumper: so I call cloud-init bug
[09:13] <thumper> bigjools: but cloud-init fails when running it?
[09:13] <bigjools> however I suggest you use --constraints=''
[09:13] <bigjools> thumper: yeah the '' is lost
[09:13] <bigjools> wallyworld_: fraid so
[09:13] <thumper> bigjools: I just don't set --constraints now if empty
[09:13] <wallyworld_> ok
[09:14] <bigjools> wallyworld_: ha apparently I already gave you access once before
[09:14] <wallyworld_> lookslike i never used it :-)
[09:15] <wallyworld_> bigjools: are there live tests for the maas storage provider (save me looking up the code)
[09:15] <wallyworld_> bigjools: cause i need to discover why it's giving an EOF
[09:15] <bigjools> wallyworld_: define live
[09:15] <bigjools> it's all against a crappy mock IIRC
[09:15] <wallyworld_> tests which are run against a live maas instance rather than a test double
[09:16] <bigjools> only place is the QA lab where we run stuff daily
[09:16] <wallyworld_> bollocks
[09:16] <wallyworld_> looks like i'll need to figure out how to get that all set up
[09:17] <wallyworld_> or i may just experiment on your maas
[09:17] <thumper> bigjools: can I get you to email me the cloud-init and the log output for the cloud-init?
[09:18] <bigjools> thumper: one sec
[09:23] <bigjools> thumper: logs are in your "tim" directory on my server
[09:23] <thumper> bigjools: ta
[09:23] <bigjools> thumper: I want to destroy env now, ok?
[09:23] <bigjools> log and cloud-data is there, I mean
[09:23] <thumper> bigjools: gimmie a minute
[09:26] <thumper> bigjools: you don't have a problem with --constraints either
[09:26] <thumper> bigjools: your problem is: 2013-10-14 08:58:18,294 - cc_apt_update_upgrade.py[WARNING]: Source Error: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main:failed to get key from keyserver.ubuntu.com
[09:27] <thumper> bigjools: when the commandline is echoed in the log file
[09:27] <thumper> bigjools: it doesn't quote the parameters
[09:27] <thumper> bigjools: not a cloud-init bug
[09:27] <bigjools> hahaha
[09:28] <rvba> thumper: yeah, the --constraints stuff is not a bug… I thought it was because I copied the output I found in cloudinit's log.
[09:28] <thumper> rvba: bigjools copied you too
[09:29] <bigjools> thumper: so if the keyserver is down, we can't deploy ...
[09:29] <bigjools> \m/
[09:29] <thumper> bigjools: seems a bit weird huh
[09:30] <thumper> I wonder if there is a way to encode the key in cloud-init
[09:30] <bigjools> my nodes don't have internet access BTW
[09:30] <bigjools> can just talk to the proxy on the maas host
[09:31] <bigjools> thumper: that was only a warning, it's not fatal
[09:31] <bigjools> thumper: I still think the error is the --constraints
[09:31] <bigjools> because of the error text
[09:33] <bigjools> thumper: also the connection failed thing
[09:33] <thumper> bigjools: no
[09:33] <thumper> bigjools: it isn't
[09:34] <bigjools> mongo is not running
[09:34] <thumper> correct
[09:34] <thumper> because you have the wrong version of mongodb-server
[09:34] <bigjools> that seems to fairly fatal
[09:34] <thumper> so it isn't running with ssl
[09:34] <thumper> so the client can't connect
[09:34] <bigjools> hurray!
[09:34] <bigjools> why is it wrong?
[09:34] <thumper> because it didn't get the mongodb-server from the cloud-tools archive
[09:34] <bigjools> this is all installed from the archive....
[09:34] <thumper> it got the default one, which is old
[09:35] <bigjools> I thought the ssl-based on was in the ubuntu archive now?
[09:35] <bigjools> one*
[09:35] <thumper> it is, for saucy and raring
[09:35] <thumper> but not precise
[09:35] <thumper> hence the cloud-tools archive
[09:35] <thumper> which it didn't use
[09:35] <thumper> because it wasn't authorized
[09:35] <thumper> anyway
[09:35] <thumper> I have my iom-maas group access now
[09:35] <thumper> will test in the morning
[09:35] <thumper> it is 22:35 here now, and I'm going to have a cuppa
[09:36] <thumper> night all
[09:36] <bigjools> so bootstrapping on precise is basically borked
[09:37] <bigjools> rvba: ^
[09:37] <rvba> bigjools: I didn't get the logs… can you send them to me?
[09:37] <bigjools> rvba: no need - read scrollback
[09:37] <rvba> bigjools: I just got a node bootstrapped with precise in the lab.
[09:37] <bigjools> !
[09:38] <bigjools> rvba: in cloud-init-output.log, where is it downloading mongodb-server from ?
[09:38] <bigjools> rvba: I see Get:18 http://archive.ubuntu.com/ubuntu/ precise-updates/universe mongodb-server amd64 1:2.0.4-1ubuntu2.1 [4167 kB]
[09:40] <rvba> bigjools: No, it was downloading it from the right place (I don't have the logs handy, I'm recreating the env right now).
[09:41] <bigjools> rvba: so it means that the cloud-archive was not added to cloud-init....
[09:41] <davecheney> bigjools: what does /etc/apt.d/sources.list.d/ say ?
[09:42] <bigjools> no cloud archive
[09:42] <davecheney> well, shit
[09:43] <rvba> bigjools: can you share the cloudinit config?
[09:43] <bigjools> and I see this in cloud-data
[09:43] <davecheney> is that 'cos of the proxy issue ?
[09:43] <bigjools> apt_sources:
[09:43] <bigjools> - source: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools
[09:44] <bigjools> Oct 14 08:58:18 gxx4a [CLOUDINIT] cc_apt_update_upgrade.py[WARNING]: Source Error: deb http://ubuntu-cloud.archive.canonical.com/ubuntu precise-updates/cloud-tools main:failed to get key from keyserver.ubuntu.com
[09:44] <bigjools> not sure if relevant
[09:44] <davecheney> bigjools: yup
[09:45] <rvba> It is!
[09:45] <bigjools> not necessarily
[09:45] <davecheney> you need to setup the proxy to allow keyserver.ubuntu.com
[09:45] <bigjools> it can continue w/o the key
[09:45] <davecheney> it could
[09:45] <davecheney> but it doesn't
[09:45] <bigjools> but I guess you fixed this already rvba?
[09:45] <rvba> bigjools: In the lab, I fixed it yes.
[09:46] <bigjools> rvba: squid-deb-proxy?
[09:46] <rvba> bigjools: yes.
[09:46] <bigjools> diff please? :)
[09:46] <bigjools> rvba: we need to fix this in the release packaging
[09:46] <rvba> bigjools: hang on, this is only related to the lab's config.
[09:47] <rvba> bigjools: I fixed the config of the lab's proxy.
[09:48] <rvba> bigjools: I'm not talking about the proxy installed on the region, I'm talking about the proxy (installed on the lab's machine) that the region's proxy is setup to use.
[09:48] <rvba> bigjools: does that make sense.
[09:48] <rvba> bigjools: in short, my issue was completely specific to the lab's setup.
[09:48] <bigjools> rvba: we also have a proxy on the region that will disallow access to both the keyserver and the cloud archive
[09:49] <rvba> bigjools: well, apparently not because I didn't have to tweak that at all.
[09:49] <bigjools> actually the latter is ok because of ".archive.canonical.com"
[09:50]  * bigjools bootstraps again
[10:22] <axw> bigjools: is there a session agenda somewhere I can see?
[10:22] <axw> for next week
[10:22] <axw> (just saw your email)
[11:22]  * TheMue => lunch
[11:23]  * rogpeppe is now really quite damp
[11:31] <mgz> rogpeppe: you were going *on* the roof?
[11:31] <rogpeppe> mgz: just out in the rain to look at the roof with the potential contractor
[11:31] <rogpeppe> mgz: it was absolutely chucking it down
[12:43] <rogpeppe> prompted by thumper's remarks this morning, a change to environs/cloudinit tests: https://codereview.appspot.com/14665043
[12:45] <rogpeppe> mgz, TheMue, wallyworld, davecheney: ^
[12:46] <rogpeppe> mattyw: did you get any further with your issues?
[12:59] <TheMue> rogpeppe: nice one. one question: in case of multiple non-continuous lines I'm interested in, can I use patterns there?
[12:59] <rogpeppe> TheMue: i'm not sure what you mean
[13:05] <TheMue> rogpeppe: e.g. I'm expecting foo \n something uninteresting \n bar \n
[13:05] <TheMue> rogpeppe: foo and bar is interesting, but not the part in the middle
[13:05] <rogpeppe> TheMue: yes, sure, that works
[13:05] <TheMue> rogpeppe: ah, fine, expected nothing else, but wanted to get sure
[13:05] <rogpeppe> TheMue: (i tried to say that in the comment on inexactMatch, but i guess i didn't succeed)
[13:06] <rogpeppe> TheMue: the logic in the function should make it fairly clear that that works too
[13:06] <rogpeppe> TheMue: assertScriptMatch, that is
[13:07] <rogpeppe> TheMue: thanks for the review, anyway
[13:09] <TheMue> rogpeppe: yeah, seen the func and thought I mostly got it, but not in total
[13:09] <TheMue> rogpeppe: missing some commments ;)
[13:17] <rogpeppe> mgz: ping
[13:21] <rogpeppe> mgz: ping
[13:21] <mgz> rogpeppe: hey
[13:21] <rogpeppe> mgz: fancy doing some stuff on addresses?
[13:22] <mgz> rogpeppe: lets, give me five then I'll join the hangout
[13:22] <rogpeppe> mgz: tell you what, gimme 15 mins and i'll have a bite to eat first
[13:24] <mgz> rogpeppe: sure
[13:41] <mattyw> rogpeppe, same again, http://paste.ubuntu.com/6235841/ I was looking at other stuff this morning, looking again now
[13:49] <rogpeppe> mattyw: i'm just about to start on some other stuff, but it would be good to find out more about what's going on.
[13:50] <rogpeppe> mgz: https://plus.google.com/hangouts/_/b9aefae7756c493cf05ba17f092adfe125b6305d?authuser=1
[13:51] <mgz> rogpeppe: ta
[13:54] <rogpeppe> mattyw: perhaps tomorrow morning?
[13:55] <mattyw> rogpeppe, sounds good, I'm going to see if I can get it going on my main machine so I can carry on, but I'll leave that vm as it is and sure, let's look tomorrow morning
[13:56] <mattyw> rogpeppe, is william on holiday?
[13:56] <mgz> mattyw: for a few days, yeah
[15:52] <mattyw> rogpeppe, trying it out on my main machine I get similar errors - not exactly the same, but lots of sockets left open
[15:53] <rogpeppe> mattyw: that's, erm, good
[15:56] <mattyw> rogpeppe, it's certainly "information" but I don't know what it means :)
[17:44] <hazmat> anyone ever tried juju with gccgo?
[17:47] <rogpeppe> hazmat: i haven't tried gccgo period...
[17:47] <rogpeppe> hazmat: i'd like to know what happens :-)
[17:47] <hazmat> rogpeppe, yeah.. it seems  more than a little flakey
[17:48] <hazmat> rogpeppe, i'm trying with just a single go file, and the resulting exec is tossing up errors
[17:48] <hazmat> the question came up wrt to juju portability
[17:48] <rogpeppe> hazmat: hmm. what errors are you seeinG?
[17:48] <sidnei> hazmat: is that related to the arm thread?
[17:48] <hazmat> sidnei, not arm
[17:48] <sidnei> k
[17:49] <hazmat> rogpeppe, http://pastebin.ubuntu.com/6236943/
[17:50] <hazmat> looks like it got to upstream.. http://gcc.gnu.org/bugzilla/show_bug.cgi?id=57194
[17:50] <sidnei> hazmat: first hit on google: http://gcc.gnu.org/bugzilla/show_bug.cgi?id=57194
[17:50] <sidnei> aha
[17:50] <hazmat> :-)
[17:56] <hazmat> looks like it barfs on compiling juju (gccgo 4.8) http://paste.ubuntu.com/6236975/
[17:56] <hazmat> some sort of inter-package dep issue
[17:59] <rogpeppe> hazmat: how are you trying to compile juju there?
[17:59] <hazmat> are we still targeting go 1.0? or have we moved on to go1.1
[17:59] <hazmat> rogpeppe, go install -compiler=gccgo launchpad.net/juju-core/...
[17:59] <rogpeppe> hazmat: we've moved on to 1.1.2
[18:04] <rogpeppe> hazmat: where did you get your gcc from
[18:06] <hazmat> rogpeppe, saucy pkg .. apt-get install gccgo == 4.8.1
[18:06] <rogpeppe> hazmat: i think you'll need to use 4.8.2
[18:07] <hazmat> rogpeppe, it feels like it might be a tool issue.. but i'll give it a whirl
[18:07] <rogpeppe> hazmat: you'll probably need to compile from source
[18:07] <rogpeppe> hazmat: 4.8.1 doesn't support go1.1
[18:08] <hazmat> rogpeppe, sure.. but its only the inter-pkg links that barfs.. 1.1 has prelim support
[18:08] <rogpeppe> hazmat: (i just confirmed that with iant on #go-nuts)
[18:08] <hazmat> in go 4.8.1
[18:08] <hazmat> k
[18:08] <hazmat> rogpeppe, i'll give it a whirl, thanks
[18:09] <hazmat> rogpeppe, is 4.8.2 released?
[18:09] <rogpeppe> hazmat: very soon, apparently
[18:10]  * rogpeppe wonders if he's got enough resources on his laptop to build gcc
[18:16] <rogpeppe>  /me is done for the day
[18:16] <rogpeppe> g'night all
[18:17] <rogpeppe> mgz: quite a few test errors from that branch (not really surprising)
[21:01] <thumper> mramm2: do we have a chat today?
[21:01] <mramm2> I can chat on the phone or skype
[21:02] <mramm2> or on IRC
[21:02] <mramm2> but am traveling today
[21:02] <mramm2> (driving to florida -- national holiday here)
[21:02] <thumper> ah
[21:03] <thumper> well, I don't have anything too urgent I suppose
[21:03] <mramm2> I am currently not driving
[21:03] <thumper> I didn't think you would be
[21:03] <mramm2> my only big thing is making sure we are ready to go for the NYC and SF sprints
[21:03] <thumper> not on irc :)
[21:03] <mramm2> haha
[21:03]  * thumper nods
[21:04] <thumper> I've been looking into the maas issues
[21:04] <thumper> we have one clear deficiency
[21:04] <thumper> and that is if we fail during startup of the bootstrap node
[21:04] <thumper> we have no error reporting mechanism
[21:04] <thumper> like not getting the right mongodb-server
[21:04] <thumper> which means we can never status
[21:04] <thumper> nor destory
[22:30] <hazmat> davecheney, ping
[22:30] <hazmat> davecheney, trying out gccgo (per roger got 4.8.2) but the fundamental issue is the tooling seems to be broken for multi-package compilations
[22:35] <hazmat> aha.. got it
[22:35] <hazmat> needed  a pkg symlink
[22:46] <davecheney> hazmat: i don't know what you're doing
[22:46] <davecheney> but i don't think a symlink is the solution
[22:46] <davecheney> hazmat: rogpeppe related, http://gcc.gnu.org/ml/gcc/2013-10/msg00085.html
[22:48] <hazmat> davecheney, i'm using that fwiw
[22:48] <davecheney> cool
[22:48] <hazmat> davecheney, gcc took forever to compile
[22:48] <hazmat> davecheney, my underlying issue was the way inter package lookups were being done
[22:49] <hazmat> for linking purposes
[22:49] <hazmat> default compilation was dropping them into $GO_HOME/lib/gccgo but lookup had an architecture suffix
[22:49] <hazmat> a simple symlinked fixed it
[22:50] <hazmat> running through the core test suite atm
[22:50] <davecheney> i rememver seeing a patch about that recently
[22:50] <hazmat> davecheney, i'm on golang trunk also re tooling
[22:51] <hazmat> getting an env to bootstrap on this might be a bit tricky.. haven't tried the static compilation options yet
[22:53] <davecheney> yup, from what I know
[22:53] <davecheney> gccgo binaries need libgo.a on the target
[22:54] <thumper> time to go hit something
[22:54]  * thumper -> gym
[22:55] <hazmat> davecheney, it can be statically linked but running into some issues with that
[22:55] <hazmat> davecheney, ala -gccgoflags '-static -static-libgo'
[22:57] <bigjools> thumper (or anyone): how do you deal with bootstrap failures on other providers then?
[22:57] <davecheney> bigjools: we don
[22:57] <davecheney> dont
[22:57] <bigjools> cloud-init's failure to add the cloud-archive and then silently install the wrong mongo is pretty shite
[22:57] <bigjools> davecheney: ok
[22:58] <davecheney> bigjools: i think this would be the 4th time it's broken
[22:58] <davecheney> i cant break for other reasons, including when cloud-init is setup properly
[22:58] <davecheney> rvba hit one with his proxy
[22:59] <davecheney> if the mongo we needed was SRU'd into precise
[22:59] <davecheney> this would solve the problem
[22:59] <bigjools> davecheney: indeed.  I've no idea why why setup won't install the cloud archive - it can't get to the keyserver for some reason.  Perhaps the apt proxy doesn't get used for that.
[23:00]  * bigjools experiments...again
[23:00] <bigjools> wallyworld_: are you using my maas server?
[23:00] <wallyworld_> not yet
[23:00] <hazmat> davecheney, bigjools we should be doing the cloud-archive by hand per cloud-archive instructions instead of using the ppa facility of cloudinit
[23:01] <hazmat> that will fail more obviously
[23:01] <hazmat> and also remove duplicate work against manual provider for the cloud archive install
[23:01] <hazmat> since its not using cloudinit
[23:02] <davecheney> 1276734
[23:02] <davecheney> lp#1276734
[23:02] <hazmat> bug 1276734
[23:02] <davecheney> no mup ?
[23:02] <hazmat> mup.. where'd you go
[23:02] <davecheney> canonistack strikes again
[23:03] <bigjools> wallyworld_: ok let me know when you might as I am doing some experimentation on it
[23:03] <wallyworld_> ok
[23:03] <kurt_> Hi Guys
[23:04] <kurt_> I discovered bug 1276734 was actually not fixed in 1.16.0 as it said it was.  I noted this in the bug tracker, but no one has responded.  What can I do about this?
[23:04] <bigjools> hazmat: it's using the apt sources directive of cloud-init - what do you mean "by hand"?
[23:05] <kurt_> Oh, I see its the topic du jour already
[23:05] <hazmat> bigjools, use the script part to install the archive
[23:07] <bigjools> add-apt-repo?
[23:19] <hazmat> bigjools, yeah
[23:19] <hazmat> bigjools, see cloud archive install instructions
[23:19] <bigjools> ok ta
[23:19] <hazmat> its not just a ppa
[23:19] <hazmat> np