[01:11] hey thumper, have a nice break? [01:11] anything I can help with on the maas thing? [01:12] axw: yeah, nice to get that mental break [01:13] axw: I'm trying to work out why maas is having bootstrap issues, and ec2 and openstack not [01:14] thumper: I guess I won't be of use without maas knowledge, but give me a yell if there's something I can do. [01:14] sure [01:14] this is a bit of a critical problem [01:14] bigjools: right [01:14] thumper: forwarding you another email [01:14] bigjools: do you have any logs of a failed bootstrap? [01:15] no, I am going to start poking soon [01:15] in particular, I want everything related to the cloud-=init [01:15] did rvb's email not cover it? [01:15] send you two emails [01:16] sent* [01:16] there's two problems I see at the moment [01:16] bigjools: the pastebin (http://paste.ubuntu.com/6226420/) had two lines but I don't know what from [01:16] 1. bootstrap failing as per rvb [01:16] it's the QA lab environment [01:16] 2. destroy-env gets its knickers in a twist [01:17] I reckon the pastebin shows an invalid command line quite frankly [01:17] --constraints is not getting any arg [01:17] locally, it gets '' if not set [01:17] I'm not sure how it ends up getting missed on the command line [01:17] I can submit a fix *right now* for the empty one [01:18] but I was really wanting to know why we didn't see issues on the ec2/openstack side [01:18] NFI [01:18] yeah [01:18] me neither [01:18] * thumper bootstraps an ec2 [01:19] something is causing --constraints to get passed in with no arg, but I don't know what would do that [01:19] passed in to jujud that is [01:19] hmm... [01:19] well, [01:19] when we generate the cloudinit [01:20] it makes" --constraints '' " if empty [01:20] so [01:20] * thumper shrugs [01:20] ok I am going to recreate locally [01:20] why is the version of in lp:juju-core/1.16 1.15? [01:21] ? [01:21] I have a sneaking suspicion that the tools are mismatched somewhere [01:21] nah [01:21] well... [01:21] actually [01:23] hmm... [01:23] ah... [01:23] nope, my fault there [01:24] * thumper bootstraps again [01:24] bigjools: got time for a hangout? [01:24] bigjools: is your shitty connection up to it? [01:24] yeah it's ok now [01:25] I finally plugged in my ADSL filter and it's better :) [01:25] haha [01:25] bigjools: hangout? [01:25] yea just call me [01:25] kk [01:27] thumper: re maas vs. ec2/openstack, I *suspect* it's due to https://codereview.appspot.com/13802045/patch/36001/37002 [01:27] see IsEmpty [01:27] * thumper looks [01:27] and parseTags [01:27] bit of a wild guess tho [01:29] hmm tags= shouldn't be in the string tho, so maybe not [01:41] this isn't about tags, but constraints [01:43] thumper: yeah... there's tags in constraints now. was thinking it might've affected whether the constraints are considered empty or not [02:48] thumper: https://bugs.launchpad.net/juju-core/+bug/1239496 [02:48] <_mup_> Bug #1239496: Packaged 1.16 juju has oauth errors talking to maas [02:48] rather critical [02:52] thumper: http://pastebin.ubuntu.com/6234274/ [02:52] wallyworld_'s fault you say? [02:52] wot [02:53] hang on [02:54] you are trying to bootstrap using a 1.9 client? [02:54] trying to work out what TF I am building [02:54] I rebuilt and it said I am using 1.9 [02:55] pbkac [02:55] trunk branch moved ... [02:55] old checkout [03:05] axw: you could look at this https://codereview.appspot.com/14494054 [03:06] thumper: will do [03:06] hang on [03:06] that's bollocks [03:06] WTF went on there then [03:06] I branched from 1.16 [03:06] and proposed to trunk [03:09] ok oauth in latest core is bollixed [03:09] https://bugs.launchpad.net/juju-core/+bug/1239496 [03:09] <_mup_> Bug #1239496: 1.16 juju and above has oauth errors talking to maas [03:16] bigjools: which series are you running [03:16] i wonder if this is 'compiled against an old versino of go' again [03:16] saucy [03:16] hmm [03:16] nope, not that the [03:16] then [03:16] davecheney: bigjools says the package version is poked [03:17] package in saucy itself is bollixed [03:17] bigjools: ok [03:17] what if you build from the 1.16.0 tarball [03:17] download it [03:17] set yoru GOPATH to the root of the tar [03:17] go install ./... [03:17] that is the litmus test [03:17] where is it again? [03:17] gotta go to the 1.16 series [03:18] https://launchpad.net/juju-core/1.16/1.16.0/+download/juju-core_1.16.0.tar.gz [03:18] ta [03:18] axw: https://codereview.appspot.com/14494054/ diff is now right [03:18] thumper: ta, reading [03:19] but still need to target one against 1.16 [03:19] ok [03:19] bigjools: did you also notice that the realm is different in the working/non-working tests? [03:19] axw: yes [03:19] maas has a log entry saying "invalid consumer" [03:20] axw: is the realm used in anyway to change the header values? [03:21] src/code.google.com/p/go.net/ipv6/sockopt_rfc3542_linux.go:33: too many errors [03:21] nice [03:21] thumper: not that I've noticed so far [03:21] thumper: I mean, apart from the "realm" part in the HTTP header [03:22] one has realm="", one has realm="MAAS+API" [03:22] davecheney: still got a 401 with that tarball [03:22] which is URL-encoded (in code it's "MAAS API") [03:23] davecheney: it prompts another question - why did Mark's package work [03:25] * bigjools grabs food [03:26] bigjools: is he using saucy ? [03:26] thumper: lgtm [03:26] axw: ta [03:47] davecheney: yes from what I can tell, his juju --version showed 1.16.0-saucy-amd64 [03:51] ok [03:51] just checking [03:51] in case he was running precise [03:53] thumper: did you end up uploading a new binaryu for me? [03:53] bigjools: yes, I copied the one I build into the ~/tim dir [03:53] ok ta let's try it [03:54] it gets the 401 too [03:55] wtf? [03:56] * thumper goes to do some bzr magic to see the changes to the maas provider between 1.14 and 1.16 [03:56] bigjools: could it be gomaasapi? [03:56] davecheney: do we know which revisions of gomaasapi were used for 1.14? [03:57] davecheney: we have tarballs, right? [03:57] thumper: it's encoded in the release-tarball script on the 1.14 branch [03:57] thumper: not sure it changed. Oauth stuff is handled in Go's library [03:58] there will also be a tag on gomaasapi [03:58] * thumper grabs both release tarballs [03:58] bigjools: wondering why sabdfl didn't hit that problem... [03:59] thumper: exactly [03:59] thumper: nor in the lab [03:59] is it only on my box? if so why does an old version work [04:05] bigjools: yes, very weird [04:05] nothing oauth related has changed in gomaasapi nor provider/maas in the 1.14.1 -> 1.16 change [04:05] we need another maas to test against [04:05] I was gonna say [04:06] easy for you to do that [04:06] you don't need nodes [04:06] bigjools: how? [04:07] run it from a branch or a package; all you need to do is "juju destroy-environment" [04:07] and if buggered you get a 401 [04:07] you can even do it on canonistack [04:07] say what? [04:20] axw: and now the 1.16 branch version, with other bug references: https://codereview.appspot.com/14516056/ [04:20] thumper: just lgtm'd [04:20] ta [04:20] thanks for logging the bugs [04:21] np === thumper is now known as thumper-afk [05:05] juju is now an easier interface to openstack than the other command line tools [05:07] :) [05:07] juju deploy -n 1000 ubuntu [05:07] well even for adding arbitrary instances [05:07] I just use bootstrap and abuse it, or add-machine [05:10] davecheney: on that note, how do I force a particular distroseries with add-machine? [05:11] oh --series ... duh === Guest99228 is now known as amithkk [06:04] davecheney: so for some odd reason that oauth problem is only on my maas test box at home [06:22] does oauth use the hostname of the server ? [06:24] no idea [06:24] the server is called "maas" so if that's causing problems I think we can all quit now [06:25] * davecheney packs his bags [06:34] axw, davecheney: if you can think of anything, anything at all, why the Auth header would be screwy only one one system.... I would be rather grateful. [06:39] bigjools: what's the diff in Auth header with the same juju, in the two envs? [06:39] axw: https://bugs.launchpad.net/juju-core/+bug/1239496 [06:39] <_mup_> Bug #1239496: 1.16 juju and above has oauth errors talking to maas [06:40] older juju binaries work fine [06:40] so something that is compiled into it, possibly from the stdlib, is behaving differently because of some environmental difference [06:41] bigjools: do you have a capture of the auth header from an older juju? [06:41] I'll get one hang on [06:42] oh crap overwrote the old binary [06:42] I'll grab a tarball and rebuild [06:43] bigjools: you didn't happen to edit environments.yaml after bootstrapping, did you? [06:43] nope [06:43] k [06:43] just asking because environments.yaml gets cached in these .jenv files now [06:44] O_o [06:44] e.g. ~/.juju/environments/maas.jenv [06:44] what's that for? [06:44] so we can auto generate stuff like admin-secret [06:44] not sure what else tbh [06:45] did you say you got this error when you did a destroy-environment, without first bootstrapping? [06:45] or did I imagine that [06:48] yeah [06:48] an easy way to test [06:48] it tries to list nodes and gets a 401 [06:48] but any command gets the same response [06:52] axw, davecheney: so I can recreate with the 1.15 release, 1.13 does not have the problem, let me just find the header [06:54] axw: I updated the bug [06:54] ta [06:56] bigjools: the only thing that jumps out at me is the non-empty realm in 1.16 [06:56] both others have realm="" [06:56] axw: see the latest one [06:57] ok [06:58] axw: ah sorry ignore the realm="" [06:58] I posted the wrong request on the comment [06:59] ok, no worries [07:00] the ones that don't work use completely the wrong token [07:01] axw: how do I clear that jenv? [07:02] axw: because I can see from examining it the oauth is different ... [07:02] bigjools: it gets created when you bootstrap [07:02] so it looks like I did change the envionments.yaml at some point [07:02] ok, that would explain it [07:02] this is a bug in juju IMO - I should be able to change the credentials [07:02] if my maas server was compromised, I need to change them [07:03] yeah, I'm not sure what the supported method of doing that is [07:04] you'd need to update the state db I think, via set-environment [07:05] whatever's in the .jenv should just be used for connecting to the API server or state server I think. [07:05] I hacked the jenv [07:05] works that w3ay [07:05] not ideal if you don't know about it :/ [07:05] yeah, I was banging my head against that for a while when dev/testing the null provider :) [07:06] perhaps we should check contents of environments.yaml and .jenv, and make sure there's no changes [07:07] good idea [07:07] I'll log a new bug [07:07] thank you [07:08] and thanks for the help [07:08] no worries [07:09] thumper-afk: I'm testing your fix for the empty constraints stuff in the lab now. [07:10] I am re-creating that here as well [07:11] thedac: Looks like the fix for that worked (no empty --constraints any more) but jujud is still not starting properly: http://paste.ubuntu.com/6234820/ [07:12] arg, sorry thedac, I meant thumper-afk. [07:37] --upload-tools seems to cause bootstrap to break, at least on maas [07:37] http://paste.ubuntu.com/6234893/ [07:39] bigjools: i don't see anythig in that log that it is actually uploading anything [07:39] that would be the problem [07:39] fwiw, I've this error (EOF) happening from time to time, sometimes when fetching the tools from aws. But it was always transient. Any idea? [07:39] davecheney: the problem is that bootstrap has not done anything! [07:39] getting that EOF error every time === thumper-afk is now known as thumper [07:42] * thumper looks at rvba's pastebin [07:42] mornin' [07:42] thumper: hiya [07:42] thumper: see my most recent email to the ML to have all the logs. [07:42] Morning rogpeppe. [07:42] rvba: ack [07:42] hi rogpeppe [07:43] rvba: yo! [07:43] i thin that eof indicates it ran off the end of whatever it was asking maas to list [07:43] https://bugs.launchpad.net/juju-core/+bug/1239558 [07:43] <_mup_> Bug #1239558: --upload-tools failure preventing bootstrap completing [07:43] rvba: do you have access to the userdata.txt for the cloud image boot? [07:43] rvba: on ec2 found it in /var/cloud/... somewhere [07:43] sorry [07:43] /var/lib/cloud [07:44] thumper: sure, let me get that for you… [07:44] /var/lib/cloud/instance/ [07:44] one of those is the text [07:44] rvba: also, log for mongod [07:44] rvba: the one that is the juju one [07:45] * thumper looks for the init script name [07:45] eek, why are tools for quantal getting uploaded by default [07:45] /var/lib/cloud/instance/cloud-config.txt: http://paste.ubuntu.com/6234904/ [07:45] /var/log/mongodb/ is empty (!) [07:46] * bigjools heading out to eat, back later [07:46] ps aux | grep mongo → zero [07:46] WTF? [07:46] rvba: it seems that we have jujud trying to start before mongod, that would certainly be an error :) [07:46] http://paste.ubuntu.com/6234911/ [07:47] Mongodb is not installed [07:47] rvba: what about mongodb-server [07:47] 1:2.0.4-1ubuntu2.1 is installed [07:48] hmm.. [07:48] that is the wrong version [07:48] sudo service mongodb status → mongodb stop/waiting [07:48] the cloud-init has removed the stable ppa that has the mongodb-server package [07:49] rvba: what is the install source of the mongodb server package [07:49] ? [07:50] thumper: http://paste.ubuntu.com/6234922/ [07:50] rvba: the juju mongo service is "juju-db" [07:50] arg, one sec… [07:50] thumper: http://paste.ubuntu.com/6234924/ [07:50] thumper: ah ok [07:50] rvba: but it will be trying to start with ssl [07:51] and failing [07:51] $ sudo service juju-db status → juju-db stop/waiting [07:51] rvba: did you see my reply ? [07:51] Get:17 http://archive.ubuntu.com/ubuntu/ precise-updates/universe mongodb-clients amd64 1:2.0.4-1ubuntu2.1 [16.5 MB] [07:51] Get:18 http://archive.ubuntu.com/ubuntu/ precise-updates/universe mongodb-server amd64 1:2.0.4-1ubuntu2.1 [4167 kB] [07:51] ^ ppa:juju/stable is missing from this bootstrap node [07:51] it will never bootstrap properly [07:51] rvba: and it seems to be logging to syslog [07:51] davecheney: agreed [07:51] davecheney: ah ok [07:51] * thumper looks at the 1.16 cloud-init work [07:52] did someone try to remove ppa:juju/stable from our cloud-init scripts again ? [07:52] i recall a change recently [07:52] yeah, me too [07:52] that was to add the cloud archive [07:52] I wonder if they removed the old [07:52] * thumper greps the bzr logs [07:53] * davecheney flips a table [07:53] jamespage: is the mongodb update available in the cloud tools pocket ? [07:53] ok found the change [07:53] axw: ping [07:54] thumper: yo [07:54] hey [07:54] axw: r1948 where you added cloud-tools [07:54] you modified the apt-repository bit for the ppa/stable [07:55] * axw brings up the diff [07:55] thumper: axw just get the mongo depdency installed into the cloud-tools pocket [07:55] that'll fix it today [07:55] no release required [07:56] last I checked, mongo was in cloud-tools [07:56] davecheney, its already in the cloud-tools pocket [07:56] but the cloud archive is just for 12.04 [07:56] axw: http://paste.ubuntu.com/6234904/ [07:56] axw: the cloud-init userdata doesn't add ppa:juju/stable [07:56] axw: also, it uses mongodb-server [07:56] not mongo [07:56] morning [07:57] hmm... [07:57] mongodb-server is there [07:58] thumper: I tested this live [07:58] it's in both ppa and pocket [07:58] * thumper wonders what was booted with maax [07:59] jamespage: it appears to be precise [07:59] * thumper ponders [07:59] ah... [07:59] here it goes [07:59] W: Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages 403 Forbidden [07:59] E: Some index files failed to download. They have been ignored, or old ones used instead. [07:59] yeah just saw that [08:00] thumper, some sort of proxy blocking access? [08:00] what's up with that? [08:00] rvba: can the machine see it? [08:00] * rvba tests. [08:00] jamespage: qa lab is firewalled off I think [08:00] rvba: you'll need to get a hole blased to that location [08:00] thumper: that machine can access the file all right. [08:01] (i.e. wget http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages works) [08:01] rvba: can you do an update, upgrade? [08:01] intermittant blockage? [08:02] hum, apt-get update → Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/precise-updates/cloud-tools/main/binary-amd64/Packages [08:02] yeah... that'll bit it :) [08:02] jamespage: do you know what has changed with the cloud-init process? [08:02] * rvba wonders why wget works. [08:03] something has changed to how it executes the scripts [08:03] in particular the --constraints '' used to work [08:03] and now it doesn't [08:03] the empty two single quote param was getting dropped somewhere [08:04] hmm... [08:04] or perhaps... [08:04] it is just how it is logged out [08:04] and copying that got a different problem [08:04] that would kinda explain it [08:04] thumper, sorry - no I don't [08:04] Here is goes, apt is configured to use the region's squid as a proxy and that is failing to fetch things from ubuntu-cloud.archive.canonical.com. [08:05] jamespage: doesn't matter, I think I explained it to myself [08:13] thumper: okay, I fixed the proxy in the lab, let's try again to bootstrap a node… [08:13] (with your empty constraints fix) [08:14] rvba: I'd be curious to see it without my fix [08:14] but lets try with first :) [08:20] thumper: node got bootstrapped okay. [08:22] I recreated it without any fix [08:22] rvba: \o/ [08:24] bigjools: did you say you got one bootstrapping without the fix? [08:24] thumper: I recreated the bug [08:24] ie. fail [08:24] oh [08:24] I can retry with the fix [08:24] bigjools: recreated the bug locally? [08:25] thumper: testing again without your fix, just to confirm. [08:25] thumper: yes [08:25] bigjools: is it the same? no access to the cloud-tools ? [08:25] or the bad command line parsing? [08:25] I'm confused as to how the command-line parsing is different [08:25] thumper: the command line thing with the --constraint [08:25] really? [08:25] wow [08:26] how does the cloud-init get run in the maas environment? [08:26] something is arsed in the core [08:26] it runs as part of the image [08:26] takes params from kernel cmd line [08:26] then downloads stuff from the metadata service [08:26] bigjools: no, the cloud-init data is good [08:27] bigjools: something has changed in how it is run [08:31] bigjools: for the failing bootstrap with the constraint error, can I see the cloud-init userdata file? [08:32] thumper: we're on our call, hang on [08:32] thumper: what's the problem here? [08:33] rogpeppe: it seems that the maas execution of the cloud-init user data was dropping an empty string param [08:33] rogpeppe: causing the jujud bootstrap-state command to fail [08:33] rogpeppe: I'm gathering data [08:34] rogpeppe: landed a fix for it by not passing constraints where there are none [08:34] thumper: this doesn't happen on ec2? [08:34] but I'd like to know why it is happening [08:34] no, ec2 is fine [08:34] thumper: hmm, odd. [08:34] yeah [08:35] thumper: the node bootstrapped okay *without* your fix this time… I'm pretty confused… maybe the wrong juju was used somehow… [08:35] hmm… [08:35] even more curious about bigjools's cloud-init issue now [08:35] it's a pity that set -x doesn't print quoted commands correctly [08:36] yeah [08:36] thumper: still, you should see an extra space in the log [08:36] thumper: if the command was really there [08:37] thumper: actually, another possible fix: use --constraints=foo rather than --constraints foo [08:37] thumper: well, it would be interesting to see if that made any difference, anyway [08:37] * thumper shrugs [08:37] slightly cleaner not to pass through nothing IMO [08:37] thumper: and therefore where it was a quoting issue or not [08:38] * thumper nods [08:38] I'm hesitant to think it is a quoting issue [08:38] I'm thinking user error, but want more data :) [08:39] thumper: why do you think the problem was a dropped empty string param? [08:40]