=== Aaton is now known as Aaton_off [02:51] ping me please if anyone knows why i cannot get past this. ssh key added via web ui. [02:51] $ juju -v status [02:51] 2012-06-06 22:49:17,153 DEBUG Initializing juju status runtime [02:51] 2012-06-06 22:49:17,162 INFO Connecting to environment... [02:51] 2012-06-06 22:49:17,227 DEBUG Connecting to environment using node-e4115b138176.local... [02:51] 2012-06-06 22:49:17,227 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node-e4115b138176.local" remote_port="2181" local_port="37322". [02:51] 2012-06-06 22:49:17,312 ERROR Invalid SSH key [02:59] . [02:59] disregard. i think i know what's wrong [03:00] pmatulis: do tell, for future reference [03:01] bigjools: well, i've been having a lot of WOL trouble on my nodes. i re-read the docs (wiki) and it does say that after 'juju bootstrap' WOL is supposed to kick in so ubuntu gets installed, *then* i do 'juju status' [03:02] yeah, it takes a while to install [03:02] but i don't even get that far. WOL doesn't work for me [03:03] that's a bizarre error from juju status then [03:04] unless i'm not reading the docs right, does WOL kick in after 'juju bootstrap' or does the OS get installed right away? [03:04] and then WOL [03:04] bootstrap will power up a node and then install ZK on it [03:05] so by power up you mean WOL? [03:05] and what is ZK? [03:05] WoL if you are using it yes. ZK is zookeeper [03:06] k [03:06] well, the OS is installed first [03:06] can take an hour or so :( [03:06] I got it down to 10m with an SSD [03:06] an hour?? [03:07] you have SSD on your nodes? what h/w is that? [03:07] the dev environment on my laptop :) [03:07] ah [03:07] I use VMs to test [03:08] i didn't realize you could do that [03:08] yeah it needs some trickery to set up [03:08] feel like sharing? [03:09] there's a vdenv directory in the source tree and it contains a readme [03:09] it pokes stuff directly into cobbler [03:09] you can use that or use it as inspiration [03:11] thanks. i'll consider that === Aaton_off is now known as Aaton === Aaton is now known as Aaton_off === matsubara is now known as matsubara-lunch === matsubara-lunch is now known as matsubara [13:01] so the server team wiki says 'juju bootstrap' will install ubuntu on a node yet after following the doc very closely all my nodes in 'Ready' state (prior to bootstrap) already have ubuntu installed [13:03] going ahead anyway, 'juju bootstrap' does not 'boot the nodes' (why plural?) and says if WoL is not working i should reset the power manually, which i did but to no avail. nothing seems to happen and 'juju status' still give errors: [13:04] 2012-06-07 08:40:55,867 DEBUG Initializing juju status runtime [13:04] 2012-06-07 08:40:55,875 INFO Connecting to environment... [13:04] 2012-06-07 08:40:55,992 DEBUG Connecting to environment using node-e4115b138176.local... [13:04] 2012-06-07 08:40:55,993 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node-e4115b138176.local" remote_port="2181" local_port="47339". [13:04] 2012-06-07 08:40:56,690 ERROR Invalid SSH key [13:05] pmatulis: is this the same cluster alexmoldovan was using yesterday? [13:05] roaksoax: yup [13:05] pmatulis: so I believe he got them installed without using juju [13:05] to try to gfix the wol issue [13:06] roaksoax: yes, like i said above, ubuntu gets installed on all nodes [13:06] pmatulis: right, but juju bootstrap only selects 1 node and sends a message to only 1 node and that's it [13:07] pmatulis: it doesn't WoL all the nodes [13:08] roaksoax: that's fine, it still doesn't work [13:09] pmatulis: is cloud init correctly importing the keys on the node? [13:09] roaksoax: no idea [13:09] pmatulis: well that might be the problme then [13:09] roaksoax: how to check? [13:10] pmatulis: ssh into the node and check cloud-init logs in /var/log [13:10] roaksoax: credentials? [13:11] pmatulis: did you guys set up an SSH key inMAAS? [13:11] roaksoax: yes [13:11] smoser: MAAS still has the bug for which it doesn't update cloud-init meta-server right? [13:12] pmatulis: so if you added a public ssh key in MAAS, then you should be able to ssh using that (from the node the private key is of course) [13:13] roaksoax, i dont know. we saw that in the ephemeral (commissioning) environment [13:13] but surely it does it on deployment [13:13] as because up until the deployment request, the user to do that wasn't yet known [13:14] smoser: right, but remember that during the demo, if we added the ssh public key in maas *after* enlisting a node, once it got deployed, it didn't obtain the ssh key [13:14] roaksoax: after grepping syslog for dhcp assignment (ip address), dns is broke, and trying ssh i am prompted for a p/w. is the node user 'ubuntu'? [13:14] so we had to add the public key to maas before enlistment [13:14] roaksoax, that was the commissioning environment i think. [13:15] pmatulis: yes [13:15] otherwise,t hats completely broken [13:15] smoser: nope, that was maas not updating the meta server [13:15] roaksoax: add the ssh key before enlistment? [13:16] well, roaksoax yes, but the meta-data for the commissioning environment. not for the install. [13:16] otherwise it is 100% completely and stupidly broken [13:16] as how could it possibly guess which user is going to deploy? [13:16] or i'm just missing something [13:16] smoser: ubuntu? [13:17] no. the MAAS user [13:17] a maas user has ssh keys [13:17] ssh keys are in the metadata provided to the instance after it iinstalls [13:17] so maas has to present the ssh keys of the correct maas user [13:18] smoser: right, so that's why each node is assigned to a maas user right? [13:20] roaksoax, right. [13:20] but when does that assignment happen? [13:20] and how does it happen? [13:21] alexmoldovan: hi [13:21] smoser: idk if this was removed, but you can manually do that in the WebUI, and if no maas user was selected, defaults to the admin user [13:22] pmatulis: so we found an isue on which the meta-data server was not being updated correctly, I just can't recall this affect the ssh keys, but we had to made changes in the meta-data server first, and then enlist so that these get to the machines when commissioned [13:25] pmatulis: hi [13:26] pmatulis, if you have the ability, you could re-install setting the password as i suggested a couple days ago [13:26] so it is 'ubuntu' and then ssh in that way [13:26] pmatulis: yeah you can also do that [13:31] pmatulis, i'm suggesting that not as a fix. [13:31] but as a way to figure out what is going wrong. [13:32] because i thikn this is a serious issue and want it to be fixed. [13:32] smoser: acknowledged [14:20] question: where does the ssh key that is embedded into the node originate? I am getting 'Invalid SSH Key' on juju status even though I have ssh keys generated; when I get into the node I see no entries in ubuntu user's ~/.ssh/authorized_keys file. [14:37] hello [14:37] hey.. look at that! [14:45] cheez0r, hi [14:46] cheez0r, if you have a moment, i'd like to debug that with you [14:46] hazmat: I do. [14:46] cheez0r, if your on the machine, do you see any files in /var/log/cloud [14:46] cheez0r, cloudinit is what actually puts the keys into place [14:46] What I'm doing now is that I've populated an SSH key in the MAAS gui for my user, and I'm about to go and re-commission all of my nodes in an attempt to make it embed that key in them. [14:47] cheez0r, it would be nice to look at those files to understand why its not able to put the files into place [14:47] cheez0r, if you could pastebin those files that would helpful [14:47] on the MAAS node, there is no /var/log/cloud [14:47] the bootstrap node is rebooting. [14:47] sudo apt-get install pastebin && pastebin /var/log/cloud-init.log [14:48] and pastebin /var/log/cloud-init-output.log [14:48] sorry its not /var/log/cloud .. there just files in /var/log [14:48] neither of those files exist on the MAAS node, I'll look on the bootstrap node when it's done rebooting [14:48] yeah.. it won't be on the maas node, but the nodes maas boots as a result of juju commands [14:49] cheez0r, thanks [14:49] no, thank you ;) [14:53] hazmat's da man [14:54] i have to step out for 15m, my dog desperately needs a walk, back soon [14:54] have fun with that ;) [14:58] smoser: the commissioning image now cleanly shutdowns the servers right? [14:59] smoser: or was that a hack? [14:59] http://pastebin.ubuntu.com/1028750 is the cloud-init.log [14:59] roaksoax, it does a shutdown. [15:00] pmatulis: so after commissioning, it should WoL just fine ^^ [15:00] pmatulis: since it does a shutdown [15:00] as stated above [15:01] hazmat, well, its not getting the key. [15:01] and we can't see that (unfortunately) in this output [15:01] because i suspect its just not there, and cloud-init is allowing that. [15:01] where does cloud-init pull the key from? [15:01] now.... juju would insert the keys itself, so that would work around the issue. [15:01] cheez0r, it pulls it from the metadata service. [15:01] smoser: juju bootstrap is not inserting my keys at all [15:01] smoser: how do they get populated there? [15:02] and you can get that data from somewhere other than the node, but you ahve to get the nodes ccredentials [15:02] which rae in the db [15:02] its less than simple [15:02] yeah I can dig that [15:02] ok. [15:02] I'm just trying to figure out in the standard maas build-up process where that ssh key originates from [15:02] well, the node has some Oauth creds associated with it. [15:02] those creds are given to it, and then it uses it to get userdata and metadata (including ssh keys) [15:02] the step-by-step just says run ssh-keygen and then juju bootstrap and that's the sum total of work to embed the ssh key [15:03] well, if you used juju, then it doesn't need ssh keys from maas [15:03] how does juju embed them, then? [15:03] it shoves them into user-data [15:03] my authorized_keys file on the target node remains empty [15:03] in which case, if the node you showed cloud-init log of was delpoyed by juju i would have thought it would get your ssh keys. [15:04] that's what the pastebin is- a juju bootstrapped node [15:04] cheez0r, /home/ubuntu/.ssh/authorized_keys is empty? [15:04] yes [15:04] can you pastebin the /var/log/cloud-init-output.log ? [15:04] there isn't one. [15:04] only cloud-init.log exists [15:06] if there isnt one, then it did not get user-data from maas. [15:06] so the debugging path here then is to get the oauth creds fro that node from the maas server [15:06] and then use cloud-init's MAAS datasource as a client to see what it is seeing [15:06] ok so I just rebooted the bootstrapped node [15:06] and it seems to me that it is not seeing anything :) [15:06] and cloud-init-output.log exists [15:07] roaksoax, ^ [15:07] was it you, cheez0r that aw this "reboot fixes it" issue? [15:07] before? [15:07] someone did. maybe it was pmatulis [15:07] the total content of the file, however, is "cloud-init boot finished at Thu, 07 Jun 2012 15:06:34 +0000. Up 15.03 seconds [15:07] " [15:07] yeah it was me, but it only did once [15:08] authorized_keys remains empty [15:08] cheez0r: pastebin apache error and access log [15:08] from maas node? [15:08] yep [15:08] working [15:10] access.log is 45k lines [15:10] want all of that? [15:10] back [15:10] so we should look at the output of the metadata service then [15:10] to verify what cloud-init is seeing [15:10] or poke at the instance user-data on disk [15:11] smoser, does the maas data source cache that to /var/lib/cloud/instance/user-data? [15:11] hazmat, it shoudl, but that will be empty. [15:11] oh, cheez0r i forgot that you can get to the instance (duh, you were pastebining from it) [15:11] yeah.. its at http://192.168.1.1/MAAS/metadata//2012-03-01/meta-data/instance-id [15:12] we still after the apache2 logs then? [15:12] well, its not really at that. [15:13] cheez0r, can you. [15:13] smoser, well there's a user-data and public-keys at the same address [15:13] smoser, that's how cloudinit gets the data from my read of maasdatasource [15:13] hazmat, they're oauth protected [15:13] * open up the file in /etc/cloud/cloud.cfg.d/91*.cfg [15:13] * there is a 'url' setting there [15:13] http://paste.ubuntu.com/1028772 and http://paste.ubuntu.com/1028774 for the apache2 access.log and error.log [15:14] * run: python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/91*.cfg crawl [15:14] or even just "get" on that url (it should show metadata and userdata) [15:14] and then you can 'get' the same url with those appended [15:14] but i'm pretty sure you're not getting anything there. [15:17] so from the bootstrapped node, I open that file, then run that python command against the URL? [15:17] there is no 91*.cfg in that directory [15:17] what files are in that directory? [15:18] this is wierd, btw :) [15:18] 05_logging.cfg; 90_dpkg.cfg; 90_dpkg_local_cloud_config.cfg; 90_dpkg_maas.cfg [15:18] README [15:18] that's it [15:18] one of the other 90_. probably _dpkg-maas [15:18] sorry [15:18] yes, I agree, since I'm following the most basic process possible to build up this config [15:18] i was just going from memory [15:18] look at those, one of them will have yaml with oauth creds [15:19] there's a consumer_key metadata_url token_key and token_secret in the file [15:20] so I ran "python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg crawl http://192.168.1.1/MAAS/metadata/ [15:20] " [15:21] response was "== http://192.168.1.1/MAAS/metadata/2012-03-01 == [15:21] 2012-03-01 [15:21] latest [15:21] [15:21] == http//192.168.1.1/MAAS/metadata/latest == [15:21] 2012-03-01 [15:21] latest [15:21] [15:21] " [15:25] that's a serious bug in maas then [15:25] smoser, what's the difference between crawl and check-seed? [15:26] should the ssh key be in the cobbler system variables? [15:26] I don't see anything regarding SSH in it. [15:28] cheez0r, could you run the same command with check-seed instead of crawl [15:28] ie. python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg check-seed http://192.168.1.1/MAAS/metadata/ [15:28] k [15:28] let me pastebin output, more significant [15:29] crawl would potentiall do more. [15:29] check-seed would only verify that the data it wants is there. [15:29] smoser, well thats sort of what i want to see [15:29] right. i just hadn'tremembered the correct usage of that tool [15:30] I see the public-keys in the output [15:30] cheez0r, can you pastebin the output of that command [15:30] hm.. [15:30] the check-seed one [15:30] http://pastebin.ubuntu.com/1028800 [15:30] yeah, that's it [15:31] that's my correct ssh key at the bottom [15:31] hmm [15:31] okay.. so the data is there [15:32] that's good [15:32] so if the data is there, what would make it not be utilized by cloud-init? [15:32] cheez0r, a bug in cloud-init [15:33] hrm, ok [15:34] smoser, can we get anymore verbose out of cloud-init for this [15:34] well... [15:34] kindo f no [15:34] because i'm suspecting right now [15:34] that if he reboots [15:34] if cheez0r does: [15:34] * rm -Rf /var/lib/cloud [15:34] * sudo reboot [15:34] smoser, but before he reboots can we determine what went wrong [15:35] then it will "just work" [15:35] smoser, sure, that will make it work.. [15:35] what i think is wrong is that nothing was there. [15:35] and now it is. [15:35] smoser, but how do we figure out what's wrong [15:35] smoser, yeah.. that would do it [15:35] smoser, it would be nice if cloud init captured that data that its using to disk [15:35] smoser, i thought it did that into /var/lib/cloud/instance [15:35] well, it does [15:35] :) [15:36] smoser, doh [15:36] and it captured a nice empty cloud-config, right? [15:36] smoser, we haven't verifeid tyat [15:36] cheez0r, do you have a /var/lib/cloud/instance dir ? [15:36] cheez0r, could you pastebin /var/lib/cloud/instance/user-data.txt [15:36] you'll probably need a sudo on that [15:37] actually. thi sis really wierd. [15:37] just for fun, can you also tell me what is in 'date -R' on the node? [15:37] cheez0r, ^ [15:37] (and compared against that on the maas server) [15:37] give me a sec [15:38] shows current date in EDT [15:38] the issue i'm thinking of (bug 978127) doesn't usually expose itself here. [15:38] Launchpad bug 978127 in MAAS "incorrect time on node causes failed oauth" [High,Triaged] https://launchpad.net/bugs/978127 [15:38] but rather in commissioning. [15:38] ChanServ, could you pastebin user-data.txt [15:38] cheez0r, i'm interested in seeing if they're off [15:39] so 'date -R' on one and 'date -R' on the other. [15:39] smoser, let's see the data first [15:39] to see if they're reasonably in sync [15:41] smoser, date out of sync would have hit us running the datasource by hand as well [15:41] yes, the bootstrapped node is in EDT and the MAAS node is in EDT [15:41] err. [15:41] CDT. [15:42] bootstrapped node == EDT; MAAS node == CDT [15:42] nah. i'm not worried about timezone [15:42] thats why i said date -R [15:42] cheez0r, please pastebin /var/lib/cloud/instance/user-data.txt [15:42] smoser, it got some of the data, it picked up hostname for example and set that [15:42] smoser, per the cloud-init output [15:42] that is output of date -R [15:42] $ date -R [15:42] Thu, 07 Jun 2012 11:42:53 -0400 [15:43] thats what my date -R shows. [15:43] there is one hour difference, MAAS node is accurate to CDT, bootstrapped node is accurate to EDT [15:43] but yes, plesae go ahead and pastebin what hazmat is asking for. [15:43] MAAS node == -0500; bootstrapped node == -0400 [15:43] cheez0r, ok. thats good enough then, that they're reasonably accurate. (they need to be within 5 minutes i think) [15:44] they're an hour different but the same UTC [15:44] ok. then, yeah, get what hazmat asked for. [15:44] date -u shows same on both [15:44] thakns. [15:45] accurate to within 10 sec [15:45] http://pastebin.ubuntu.com/1028816 [15:45] yeah.. instead of playing smoke and mirrors games theorizing about time, we already know the dates are in sync because we can fetch the data in the first place. so please see what its actually running first.. [15:45] so its a different problem [15:46] hazmat, well, not really. [15:46] smoser, so that data is correct [15:46] something could have fixed the time since [15:46] smoser, not if the user-data.txt is in place [15:46] correct. [15:46] then i'm really confused. [15:47] smoser, did running the datasource overwrite the instance cache? [15:47] no. its just an oauth client [15:47] well, the preseed does set an NTP server [15:47] but, sure, cheez0r can you verify the timestamps on that file? [15:47] and it is accessible from the bootstrapped node [15:47] actually.. [15:47] today at 11:06am [15:48] that's 41 minutes ago [15:48] cheez0r, that's roughly when you commissioned it? [15:48] could you : [15:48] sudo ls -lR /var/lib/cloud/ | pastebinit [15:48] yeah, roughly [15:48] cheez0r, but http://pastebin.ubuntu.com/1028750/ [15:48] says that stuff happened on Jun 6 [15:48] (i htink that was your pastebin of /var/log/cloud-init.log [15:49] http://pastebin.ubuntu.com/1028826 [15:49] smoser, ah.. [15:49] the reboot already happend. [15:49] its a recomissioned node maybe? [15:49] -rw-r--r-- 1 root root 13 Jun 7 11:06 config-scripts-per-boot.always [15:49] -rw-r--r-- 1 root root 14 Jun 6 16:09 config-scripts-per-once.once [15:50] I can destroy-environment and rebootstrap if you like [15:50] and it never cleared out the old data when it recomissions? [15:50] i thought maas would do a fresh install on a recommissioned node [15:50] it was never recommissioned. [15:50] it was bootstrapped and then rebooted. [15:50] I think, at least. [15:51] ic [15:54] ok [15:54] so here is my hypothesis: [15:55] * on initial first boot (yesterday) instance got no user-data from maas server, so it came up happily and did nothing. [15:56] * in order to remove the dependency on the maas MD server on every boot, maas configures cloud-init to only ever look on the first boot [15:56] hm.. [15:56] nah [15:56] my hypothesis is flawed. [15:56] i'm confused. [16:05] smoser, it seems to be me that the issue is basically what your surmise.. effectively cloud-init is being rerun with existing contents of /var/lib/cloud and only parts are executing because of the pre-existing state [16:05] but specifically parts like putting keys into place and user scripts are not being run [16:06] oh. [16:06] crap [16:06] yeah. [16:06] you're right. [16:06] so the first time it came up [16:06] it got its instance ID [16:06] because that is constant for the node (it is not per "instance") [16:06] then, second boot, it came up and got different user data [16:06] but it had already run that stuff once per the given instance-id [16:06] so it did not run again [16:07] i'm still kind of confused as to why we dont see evidence of cloud-init running *at all* on second boot in http://pastebin.ubuntu.com/1028750/ [16:13] smoser, that is curious [16:13] it should still have out there just running through the handlers [16:13] smoser, hmm.. well [16:13] smoser, what if it never actually rebooted the machine [16:14] and actually, we have evidence that it *did* [16:14] in the timestamps [16:14] http://pastebin.ubuntu.com/1028826/ [16:14] hmm.. yeah [16:14] see '.always' timestamps [16:14] so i'm jsut really confused [16:16] cheez0r, one more thing... [16:17] acutally never mind. [16:17] cheez0r, is there additional content at bottom of /var/log/cloud-init.log [16:17] i'm almost certain maas gave cloud-init empty user data on first boot [16:17] that didn't get to the pastebin? [16:18] smoser, so what's the maas usage here.. does it clean out the disk/reinstall when we destroy/bootstrap again? [16:18] destroy-environment that is [16:18] hazmat, yes, its a full install. [16:18] but it doesn't do that when doing the first boot dance [16:18] its slow as molasses [16:19] so the first time you setup, the nodes have pre-existing state [16:19] and never fully initialize with cloud-init [16:19] but subsequent to destroy-environment, bootstrap it gets a clean disk [16:20] adam_g, ping [16:20] no. there is never any pre-existing state. [16:21] smoser, so when/why is it doing a reboot? [16:21] that was human [16:21] actually, nothing in maas has the ability to reboot :) [16:22] ah, ic [16:25] smoser, so our working hypothesis atm then is that when it first comes up it doesn't get any init metadata [16:25] but subsequently it does [16:25] but its too late, since cloud-init already ran with a mostly empty metadata [16:26] cheez0r, smoser okay.. i think we've exhausted the debug info we can get from that system [16:27] cheez0r, you should be able to destroy-environment/bootstrap and have things work [16:27] hazmat, yeah. [16:27]