=== Aaton is now known as Aaton_off
pmatulisping me please if anyone knows why i cannot get past this.  ssh key added via web ui.02:51
pmatulis$ juju -v status02:51
pmatulis2012-06-06 22:49:17,153 DEBUG Initializing juju status runtime02:51
pmatulis2012-06-06 22:49:17,162 INFO Connecting to environment...02:51
pmatulis2012-06-06 22:49:17,227 DEBUG Connecting to environment using node-e4115b138176.local...02:51
pmatulis2012-06-06 22:49:17,227 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node-e4115b138176.local" remote_port="2181" local_port="37322".02:51
pmatulis2012-06-06 22:49:17,312 ERROR Invalid SSH key02:51
pmatulisdisregard.  i think i know what's wrong02:59
bigjoolspmatulis: do tell, for future reference03:00
pmatulisbigjools: well, i've been having a lot of WOL trouble on my nodes.  i re-read the docs (wiki) and it does say that after 'juju bootstrap' WOL is supposed to kick in so ubuntu gets installed, *then* i do 'juju status'03:01
bigjoolsyeah, it takes a while to install03:02
pmatulisbut i don't even get that far.  WOL doesn't work for me03:02
bigjoolsthat's a bizarre error from juju status then03:03
pmatulisunless i'm not reading the docs right, does WOL kick in after 'juju bootstrap' or does the OS get installed right away?03:04
pmatulisand then WOL03:04
bigjoolsbootstrap will power up a node and then install ZK on it03:04
pmatulisso by power up you mean WOL?03:05
pmatulisand what is ZK?03:05
bigjoolsWoL if you are using it yes.  ZK is zookeeper03:05
bigjoolswell, the OS is installed first03:06
bigjoolscan take an hour or so :(03:06
bigjoolsI got it down to 10m with an SSD03:06
pmatulisan hour??03:06
pmatulisyou have SSD on your nodes?  what h/w is that?03:07
bigjoolsthe dev environment on my laptop :)03:07
bigjoolsI use VMs to test03:07
pmatulisi didn't realize you could do that03:08
bigjoolsyeah it needs some trickery to set up03:08
pmatulisfeel like sharing?03:08
bigjoolsthere's a vdenv directory in the source tree and it contains a readme03:09
bigjoolsit pokes stuff directly into cobbler03:09
bigjoolsyou can use that or use it as inspiration03:09
pmatulisthanks.  i'll consider that03:11
=== Aaton_off is now known as Aaton
=== Aaton is now known as Aaton_off
=== matsubara is now known as matsubara-lunch
=== matsubara-lunch is now known as matsubara
pmatulisso the server team wiki says 'juju bootstrap' will install ubuntu on a node yet after following the doc very closely all my nodes in 'Ready' state (prior to bootstrap) already have ubuntu installed13:01
pmatulisgoing ahead anyway, 'juju bootstrap' does not 'boot the nodes' (why plural?) and says if WoL is not working i should reset the power manually, which i did but to no avail.  nothing seems to happen and 'juju status' still give errors:13:03
pmatulis2012-06-07 08:40:55,867 DEBUG Initializing juju status runtime13:04
pmatulis2012-06-07 08:40:55,875 INFO Connecting to environment...13:04
pmatulis2012-06-07 08:40:55,992 DEBUG Connecting to environment using node-e4115b138176.local...13:04
pmatulis2012-06-07 08:40:55,993 DEBUG Spawning SSH process with remote_user="ubuntu" remote_host="node-e4115b138176.local" remote_port="2181" local_port="47339".13:04
pmatulis2012-06-07 08:40:56,690 ERROR Invalid SSH key13:04
roaksoaxpmatulis: is this the same cluster alexmoldovan was using yesterday?13:05
pmatulisroaksoax: yup13:05
roaksoaxpmatulis: so I believe he got them installed without using juju13:05
roaksoaxto try to gfix the wol issue13:05
pmatulisroaksoax: yes, like i said above, ubuntu gets installed on all nodes13:06
roaksoaxpmatulis: right, but juju bootstrap only selects 1 node and sends a message to only 1 node and that's it13:06
roaksoaxpmatulis: it doesn't WoL all the nodes13:07
pmatulisroaksoax: that's fine, it still doesn't work13:08
roaksoaxpmatulis: is cloud init correctly importing the keys on the node?13:09
pmatulisroaksoax: no idea13:09
roaksoaxpmatulis: well that might be the problme then13:09
pmatulisroaksoax: how to check?13:09
roaksoaxpmatulis: ssh into the node and check cloud-init logs in /var/log13:10
pmatulisroaksoax: credentials?13:10
roaksoaxpmatulis: did you guys set up an SSH key inMAAS?13:11
pmatulisroaksoax: yes13:11
roaksoaxsmoser: MAAS still has the bug for which it doesn't update cloud-init meta-server right?13:11
roaksoaxpmatulis: so if you added a public ssh key in MAAS, then you should be able to ssh using that (from the node the private key is of course)13:12
smoserroaksoax, i dont know. we saw that in the ephemeral (commissioning) environment13:13
smoserbut surely it does it on deployment13:13
smoseras because up until the deployment request, the user to do that wasn't yet known13:13
roaksoaxsmoser: right, but remember that during the demo, if we added the ssh public key in maas *after* enlisting a node, once it got deployed, it didn't obtain the ssh key13:14
pmatulisroaksoax: after grepping syslog for dhcp assignment (ip address), dns is broke, and trying ssh i am prompted for a p/w.  is the node user 'ubuntu'?13:14
roaksoaxso we had to add the public key to maas before enlistment13:14
smoserroaksoax, that was the commissioning environment i think.13:14
roaksoaxpmatulis: yes13:15
smoserotherwise,t hats completely broken13:15
roaksoaxsmoser: nope, that was maas not updating the meta server13:15
pmatulisroaksoax: add the ssh key before enlistment?13:15
smoserwell, roaksoax yes, but the meta-data for the commissioning environment. not for the install.13:16
smoserotherwise it is 100% completely and stupidly broken13:16
smoseras how could it possibly guess which user is going to deploy?13:16
smoseror i'm just missing something13:16
roaksoaxsmoser: ubuntu?13:16
smoserno. the MAAS user13:17
smosera maas user has ssh keys13:17
smoserssh keys are in the metadata provided to the instance after it iinstalls13:17
smoserso maas has to present the ssh keys of the correct maas user13:17
roaksoaxsmoser: right, so that's why each node is assigned to a maas user right?13:18
smoserroaksoax, right.13:20
smoserbut when does that assignment happen?13:20
smoserand how does it happen?13:20
pmatulisalexmoldovan: hi13:21
roaksoaxsmoser: idk if this was removed, but you can manually do that in the WebUI, and if no maas user was selected, defaults to the admin user13:21
roaksoaxpmatulis: so we found an isue on which the meta-data server was not being updated correctly, I just can't recall this affect the ssh keys, but we had to made changes in the meta-data server first, and then enlist so that these get to the machines when commissioned13:22
alexmoldovanpmatulis: hi13:25
smoserpmatulis, if you have the ability, you could re-install setting the password as i suggested a couple days ago13:26
smoserso it is 'ubuntu' and then ssh in that way13:26
roaksoaxpmatulis: yeah you can also do that13:26
smoserpmatulis, i'm suggesting that not as a fix.13:31
smoserbut as a way to figure out what is going wrong.13:31
smoserbecause i thikn this is a serious issue and want it to be fixed.13:32
pmatulissmoser: acknowledged13:32
cheez0rquestion: where does the ssh key that is embedded into the node originate? I am getting 'Invalid SSH Key' on juju status even though I have ssh keys generated; when I get into the node I see no entries in ubuntu user's ~/.ssh/authorized_keys file.14:20
m_3hey.. look at that!14:37
hazmatcheez0r, hi14:45
hazmatcheez0r, if you have a moment, i'd like to debug that with you14:46
cheez0rhazmat: I do.14:46
hazmatcheez0r, if your on the machine, do you see any  files in /var/log/cloud14:46
hazmatcheez0r, cloudinit is what actually puts the keys into place14:46
cheez0rWhat I'm doing now is that I've populated an SSH key in the MAAS gui for my user, and I'm about to go and re-commission all of my nodes in an attempt to make it embed that key in them.14:46
hazmatcheez0r, it would be nice to look at those files to understand why its not able to put the files into place14:47
hazmatcheez0r, if you could pastebin those files that would helpful14:47
cheez0ron the MAAS node, there is no /var/log/cloud14:47
cheez0rthe bootstrap node is rebooting.14:47
hazmatsudo apt-get install pastebin && pastebin /var/log/cloud-init.log14:47
hazmatand pastebin /var/log/cloud-init-output.log14:48
hazmatsorry its not /var/log/cloud .. there just files in /var/log14:48
cheez0rneither of those files exist on the MAAS node, I'll look on the bootstrap node when it's done rebooting14:48
hazmatyeah.. it won't be on the maas node, but the nodes maas boots as a result of juju commands14:48
hazmatcheez0r, thanks14:49
cheez0rno, thank you ;)14:49
m_3hazmat's da man14:53
hazmati have to step out for 15m, my dog desperately needs a walk, back soon14:54
cheez0rhave fun with that ;)14:54
roaksoaxsmoser: the commissioning image now cleanly shutdowns the servers right?14:58
roaksoaxsmoser: or was that a hack?14:59
cheez0rhttp://pastebin.ubuntu.com/1028750 is the cloud-init.log14:59
smoserroaksoax, it does a shutdown.14:59
roaksoaxpmatulis: so after commissioning, it should WoL just fine ^^15:00
roaksoaxpmatulis: since it does a shutdown15:00
roaksoaxas stated above15:00
smoserhazmat, well, its not getting the key.15:01
smoserand we can't see that (unfortunately) in this output15:01
smoserbecause i suspect its just not there, and cloud-init is allowing that.15:01
cheez0rwhere does cloud-init pull the key from?15:01
smosernow.... juju would insert the keys itself, so that would work around the issue.15:01
smosercheez0r, it pulls it from the metadata service.15:01
cheez0rsmoser: juju bootstrap is not inserting my keys at all15:01
cheez0rsmoser: how do they get populated there?15:01
smoserand you can get that data from somewhere other than the node, but you ahve to get the nodes ccredentials15:02
smoserwhich rae in the db15:02
smoserits less than simple15:02
cheez0ryeah I can dig that15:02
cheez0rI'm just trying to figure out in the standard maas build-up process where that ssh key originates from15:02
smoserwell, the node has some Oauth creds associated with it.15:02
smoserthose creds are given to it, and then it uses it to get userdata and metadata (including ssh keys)15:02
cheez0rthe step-by-step just says run ssh-keygen and then juju bootstrap and that's the sum total of work to embed the ssh key15:02
smoserwell, if you used juju, then it doesn't need ssh keys from maas15:03
cheez0rhow does juju embed them, then?15:03
smoserit shoves them into user-data15:03
cheez0rmy authorized_keys file on the target node remains empty15:03
smoserin which case, if the node you showed cloud-init log of was delpoyed by juju i would have thought it would get your ssh keys.15:03
cheez0rthat's what the pastebin is- a juju bootstrapped node15:04
smosercheez0r, /home/ubuntu/.ssh/authorized_keys is empty?15:04
smosercan you pastebin the /var/log/cloud-init-output.log ?15:04
cheez0rthere isn't one.15:04
cheez0ronly cloud-init.log exists15:04
smoserif there isnt one, then it did not get user-data from maas.15:06
smoserso the debugging path here then is to get the oauth creds fro that node from the maas server15:06
smoserand then use cloud-init's MAAS datasource as a client to see what it is seeing15:06
cheez0rok so I just rebooted the bootstrapped node15:06
smoserand it seems to me that it is not seeing anything :)15:06
cheez0rand cloud-init-output.log exists15:06
smoserroaksoax, ^15:07
smoserwas it you, cheez0r that aw this "reboot fixes it" issue?15:07
smosersomeone did. maybe it was pmatulis15:07
cheez0rthe total content of the file, however, is "cloud-init boot finished at Thu, 07 Jun 2012 15:06:34 +0000. Up 15.03 seconds15:07
cheez0ryeah it was me, but it only did once15:07
cheez0rauthorized_keys remains empty15:08
roaksoaxcheez0r: pastebin apache error and access log15:08
cheez0rfrom maas node?15:08
cheez0raccess.log is 45k lines15:10
cheez0rwant all of that?15:10
hazmatso we should look at the output of the metadata service then15:10
hazmatto verify what cloud-init is seeing15:10
hazmator poke at the instance user-data on disk15:10
hazmatsmoser, does the maas data source cache that to /var/lib/cloud/instance/user-data?15:11
smoserhazmat, it shoudl, but that will be empty.15:11
smoseroh, cheez0r i forgot that you can get to the instance (duh, you were pastebining from it)15:11
hazmatyeah.. its at
cheez0rwe still after the apache2 logs then?15:12
smoserwell, its not really at that.15:12
smosercheez0r, can you.15:13
hazmatsmoser, well there's a user-data and public-keys at the same address15:13
hazmatsmoser, that's how cloudinit gets the data from my read of maasdatasource15:13
smoserhazmat, they're oauth protected15:13
smoser * open up the file in /etc/cloud/cloud.cfg.d/91*.cfg15:13
smoser * there is a 'url' setting there15:13
cheez0rhttp://paste.ubuntu.com/1028772 and http://paste.ubuntu.com/1028774 for the apache2 access.log and error.log15:13
smoser * run: python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/91*.cfg crawl <htat-url>15:14
smoseror even just "get" on that url (it should show metadata and userdata)15:14
smoserand then you can 'get' the same url with those appended15:14
smoserbut i'm pretty sure you're not getting anything there.15:14
cheez0rso from the bootstrapped node, I open that file, then run that python command against the URL?15:17
cheez0rthere is no 91*.cfg in that directory15:17
smoserwhat files are in that directory?15:17
smoserthis is wierd, btw :)15:18
cheez0r05_logging.cfg; 90_dpkg.cfg; 90_dpkg_local_cloud_config.cfg; 90_dpkg_maas.cfg15:18
cheez0rthat's it15:18
smoserone of the other 90_. probably _dpkg-maas15:18
cheez0ryes, I agree, since I'm following the most basic process possible to build up this config15:18
smoseri was just going from memory15:18
smoserlook at those, one of them will have yaml with oauth creds15:18
cheez0rthere's a consumer_key metadata_url token_key and token_secret in the file15:19
cheez0rso I ran "python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg crawl
cheez0rresponse was "== ==15:21
cheez0r 15:21
cheez0r== http// ==15:21
cheez0r 15:21
hazmatthat's a serious bug in maas then15:25
hazmatsmoser, what's the difference between crawl and check-seed?15:25
cheez0rshould the ssh key be in the cobbler system variables?15:26
cheez0rI don't see anything regarding SSH in it.15:26
hazmatcheez0r, could you run the same command with check-seed instead of crawl15:28
hazmatie. python /usr/share/pyshared/cloudinit/DataSourceMAAS.py --config /etc/cloud/cloud.cfg.d/90_dpkg_maas.cfg check-seed
cheez0rlet me pastebin output, more significant15:28
smosercrawl would potentiall do more.15:29
smosercheck-seed would only verify that the data it wants is there.15:29
hazmatsmoser, well thats sort of what i want to see15:29
smoserright. i just hadn'tremembered the correct usage of that tool15:29
cheez0rI see the public-keys in the output15:30
hazmatcheez0r, can you pastebin the output of that command15:30
hazmatthe check-seed one15:30
cheez0ryeah, that's it15:30
cheez0rthat's my correct ssh key at the bottom15:31
hazmatokay.. so the data is there15:31
hazmatthat's good15:32
cheez0rso if the data is there, what would make it not be utilized by cloud-init?15:32
hazmatcheez0r, a bug in cloud-init15:32
cheez0rhrm, ok15:33
hazmatsmoser, can we get anymore verbose out of cloud-init for this15:34
smoserkindo f no15:34
smoserbecause i'm suspecting right now15:34
smoserthat if he reboots15:34
smoserif cheez0r does:15:34
smoser * rm -Rf /var/lib/cloud15:34
smoser * sudo reboot15:34
hazmatsmoser, but before he reboots can we determine what went wrong15:34
smoserthen it will "just work"15:35
hazmatsmoser, sure, that will make it work..15:35
smoserwhat i think is wrong is that nothing was there.15:35
smoserand now it is.15:35
hazmatsmoser, but how do we figure out what's wrong15:35
hazmatsmoser, yeah.. that would do it15:35
hazmatsmoser, it would be nice if cloud init captured that data that its using to disk15:35
hazmatsmoser, i thought it did that into /var/lib/cloud/instance15:35
smoserwell, it does15:35
hazmatsmoser, doh15:36
smoserand it captured a nice empty cloud-config, right?15:36
hazmatsmoser, we haven't verifeid tyat15:36
smosercheez0r, do you have a /var/lib/cloud/instance dir ?15:36
hazmatcheez0r, could you pastebin  /var/lib/cloud/instance/user-data.txt15:36
hazmatyou'll probably need a sudo on that15:36
smoseractually. thi sis really wierd.15:37
smoserjust for fun, can you also tell me what is in 'date -R' on the node?15:37
hazmatcheez0r, ^15:37
smoser(and compared against that on the maas server)15:37
cheez0rgive me a sec15:37
cheez0rshows current date in EDT15:38
smoserthe issue i'm thinking of (bug 978127) doesn't usually expose itself here.15:38
ubot5Launchpad bug 978127 in MAAS "incorrect time on node causes failed oauth" [High,Triaged] https://launchpad.net/bugs/97812715:38
smoserbut rather in commissioning.15:38
hazmatChanServ, could you pastebin user-data.txt15:38
smosercheez0r, i'm interested in seeing if they're off15:38
smoserso 'date -R' on one and 'date -R' on the other.15:39
hazmatsmoser, let's see the data first15:39
smoserto see if they're reasonably in sync15:39
hazmatsmoser, date out of sync would have hit us running the datasource by hand as well15:41
cheez0ryes, the bootstrapped node is in EDT and the MAAS node is in EDT15:41
cheez0rbootstrapped node == EDT; MAAS node == CDT15:42
smosernah. i'm not worried about timezone15:42
smoserthats why i said date -R15:42
hazmatcheez0r, please pastebin  /var/lib/cloud/instance/user-data.txt15:42
hazmatsmoser, it got some of the data, it picked up hostname for example and set that15:42
hazmatsmoser, per the cloud-init output15:42
cheez0rthat is output of date -R15:42
smoser$ date -R15:42
smoserThu, 07 Jun 2012 11:42:53 -040015:42
smoserthats what my date -R shows.15:43
cheez0rthere is one hour difference, MAAS node is accurate to CDT, bootstrapped node is accurate to EDT15:43
smoserbut yes, plesae go ahead and pastebin what hazmat is asking for.15:43
cheez0rMAAS node == -0500; bootstrapped node == -040015:43
smosercheez0r, ok. thats good enough then, that they're reasonably accurate. (they need to be within 5 minutes i think)15:43
cheez0rthey're an hour different but the same UTC15:44
smoserok. then, yeah, get what hazmat asked for.15:44
cheez0rdate -u shows same on both15:44
cheez0raccurate to within 10 sec15:45
hazmatyeah.. instead of playing smoke and mirrors games theorizing about time, we already know the dates are in sync because we can fetch the data in the first place. so please see what its actually running first..15:45
hazmatso its a different problem15:45
smoserhazmat, well, not really.15:46
hazmatsmoser, so that data is correct15:46
smosersomething could have fixed the time since15:46
hazmatsmoser, not if the user-data.txt is in place15:46
smoserthen i'm really confused.15:46
hazmatsmoser, did running the datasource overwrite the instance cache?15:47
smoserno. its just an oauth client15:47
cheez0rwell, the preseed does set an NTP server15:47
smoserbut, sure, cheez0r can you verify the timestamps on that file?15:47
cheez0rand it is accessible from the bootstrapped node15:47
cheez0rtoday at 11:06am15:47
cheez0rthat's 41 minutes ago15:48
hazmatcheez0r, that's roughly when you commissioned it?15:48
smosercould you :15:48
smoser sudo ls -lR /var/lib/cloud/ | pastebinit15:48
cheez0ryeah, roughly15:48
smosercheez0r, but http://pastebin.ubuntu.com/1028750/15:48
smosersays that stuff happened on Jun 615:48
smoser(i htink that was your pastebin of /var/log/cloud-init.log15:48
hazmatsmoser, ah..15:49
smoserthe reboot already happend.15:49
hazmatits a recomissioned node maybe?15:49
smoser-rw-r--r-- 1 root root 13 Jun  7 11:06 config-scripts-per-boot.always15:49
smoser-rw-r--r-- 1 root root 14 Jun  6 16:09 config-scripts-per-once.once15:49
cheez0rI can destroy-environment and rebootstrap if you like15:50
hazmatand it never cleared out the old data when it recomissions?15:50
hazmati thought maas would do a fresh install on a recommissioned node15:50
cheez0rit was never recommissioned.15:50
cheez0rit was bootstrapped and then rebooted.15:50
cheez0rI think, at least.15:50
smoserso here is my hypothesis:15:54
smoser * on initial first boot (yesterday) instance got no user-data from maas server, so it came up happily and did nothing.15:55
smoser * in order to remove the dependency on the maas MD server on every boot, maas configures cloud-init to only ever look on the first boot15:56
smosermy hypothesis is flawed.15:56
smoseri'm confused.15:56
hazmatsmoser, it seems to be me that the issue is basically what your surmise.. effectively cloud-init is being rerun with existing contents of /var/lib/cloud and only parts are executing because of the pre-existing state16:05
hazmatbut specifically parts like putting keys into place and user scripts are not being run16:05
smoseryou're right.16:06
smoserso the first time it came up16:06
smoserit got its instance ID16:06
smoserbecause that is constant for the node (it is not per "instance")16:06
smoserthen, second boot, it came up and got different user data16:06
smoserbut it had already run that stuff once per the given instance-id16:06
smoserso it did not run again16:06
smoseri'm still kind of confused as to why we dont see evidence of cloud-init running *at all* on second boot in http://pastebin.ubuntu.com/1028750/16:07
hazmatsmoser, that is curious16:13
hazmatit should still have out there just running through the handlers16:13
hazmatsmoser, hmm.. well16:13
hazmatsmoser, what if it never actually rebooted the machine16:13
smoserand actually, we have evidence that it *did*16:14
smoserin the timestamps16:14
hazmathmm.. yeah16:14
smosersee '.always' timestamps16:14
smoserso i'm jsut really confused16:14
smosercheez0r, one more thing...16:16
smoseracutally never mind.16:17
hazmatcheez0r, is there additional content at bottom of /var/log/cloud-init.log16:17
smoseri'm almost certain maas gave cloud-init empty user data on first boot16:17
hazmatthat didn't get to the pastebin?16:17
hazmatsmoser, so what's the maas usage here.. does it clean out the disk/reinstall when we destroy/bootstrap again?16:18
hazmatdestroy-environment that is16:18
smoserhazmat, yes, its a full install.16:18
hazmatbut it doesn't do that when doing the first boot dance16:18
smoserits slow as molasses16:18
hazmatso the first time you setup, the nodes have pre-existing state16:19
hazmatand never fully initialize with cloud-init16:19
hazmatbut subsequent to destroy-environment, bootstrap it gets a clean disk16:19
hazmatadam_g, ping16:20
smoserno. there is never any pre-existing state.16:20
hazmatsmoser, so when/why is it doing a reboot?16:21
smoserthat was human16:21
smoseractually, nothing in maas has the ability to reboot :)16:21
hazmatah, ic16:22
hazmatsmoser, so our working hypothesis atm then is that when it first comes up it doesn't get any init metadata16:25
hazmatbut subsequently it does16:25
hazmatbut its too late, since cloud-init already ran with a mostly empty metadata16:25
hazmatcheez0r, smoser okay.. i think we've exhausted the debug info we can get from that system16:26
hazmatcheez0r, you should be able to destroy-environment/bootstrap and have things work16:27
smoserhazmat, yeah.16:27
smoserthats my hypothesis.16:27
smosercloud-init isn't very friendly to maas MD either.16:28
smoserit doesn't even retry.16:28
smoserbut it sure seems that it must have gotten a 200 response with empty data to me16:28
hazmatsmoser, for juju's purposes, it would be fine for it to re run the user data16:28
smoserwell, thats neither here nor there.16:29
smoserruncmd by design is run only once per instance.16:29
hazmatwe'd have to make one minor adjustment to the initialize stuff, but then it would be idempotent16:29
cheez0rsorry guys I had to step away16:35
cheez0rwill read scroll and see what's doing in a bit16:36
smoserhazmat, even if it was idempotent, you wouldn't want to do it.16:40
smoseryou dont want to apt-get update && apt-get install on every boot16:40
smoserthats just a silly waste of time16:41
hazmatsmoser, i sent out a reply16:52
hazmatsmoser, i'm pretty sure that the root of this is a timing issue getting metadata to the api within maas16:52
hazmatand i wonder if it works subsequently because its returning old metadta16:53
hazmati'd have to dig into the code of maas to verify though16:53
smoserah. you're right16:56
smoserits race16:56
hazmatsmoser, going through maas code its not clear why16:57
hazmatit updates the db before it starts the nodes16:57
hazmatso the userdata is in place16:57
hazmatare there any maas developers up?16:57
smoser "before it starts the nodes"16:57
hazmatallenap, ping16:58
hazmatroaksoax, ping16:58
hazmatthe more the merrier16:58
roaksoaxhazmat: pong16:59
hazmategads i hate brighttalk for videos.. why should folks have to signup17:02
hazmatroaksoax, trying to understand why there might be a delay for userdata showing up for a node17:02
hazmatroaksoax, we're trying to debug the invalid ssh key thing17:02
hazmatroaksoax, i just sent email to the maas-dev list which outlines our hypothesis17:02
hazmatroaksoax, looking at the code i'm not sure how that could be the case17:03
hazmatit looks like the api will update the userdata before launching the node17:04
hazmatis there some celery task that will try to overwrite that? or i'm misunderstanding this bit..17:04
roaksoaxhazmat: in precise maas we are not using celery17:04
hazmatroaksoax, gotcha..17:05
hazmatroaksoax, but does delay availability of metadata sound familiar or possible?17:05
adam_ghazmat: pong17:05
roaksoaxhazmat: sounds familiar. I've been in a situation on which a node is enlisted, and then a ssh public key gets added and the node should have had the meta-data update so than on deployment it would have imported that ssh key. however, it didn't happen, so I had to enlist the nodes after the key was added.17:06
hazmatadam_g, there's a question up on stack overflow/askubuntu.. which i think you might be the only who can respond effectively..17:06
hazmatadam_g, http://askubuntu.com/questions/141552/creating-volume-group-in-nova-volume-juju-charm17:06
roaksoaxhazmat: while it is not the same issue (and this is a MAAS SSH key), it might be related17:06
adam_ghazmat: ill take a look, thanks17:07
Davieyhazmat: metadata issues?  check the server apache logs.17:12
hazmatDaviey, does it print out the full response?17:13
DavieyThe common hunch that has hit most of us so far, is bad localtime offset.. meaning that the oauth token was expired17:13
Davieyhazmat: you can ask apache nicely to do that.17:13
smoserDaviey, that might be useful to help prove this.17:16
smoseradam_g, i'm interested in seeing your answer to that question17:16
hazmatsmoser, Daviey so how does the ntp get setup?17:18
smoserit does not.17:18
smosernothing sets up ntp17:18
smoserthis is not a time related issue.17:18
smoseri'm virtually certain of that.17:18
adam_gsmoser: yeah, me too. :) i need to setup a local provider to test this. you added the loopback stuff to that charm, wasn't there some namespacing / insmod'ing that needed to be done on the host before LVM would work?17:19
smoseradam_g, you can't do it.17:20
smoserthere is no device name space17:20
smoserso you have to grant the container access to loopback devices17:20
smoserits a rathole17:20
adam_gsmoser: right, the loopback as the PV worked IIRC, but i thought it required something to be done on the host too17:20
smoseryou have to allow the lxc  container to write to /dev/loop-control17:22
smoserand thats fine... maybe that even makes nova-volume charm work17:22
smoserbut the issue just keeps popping its head up.17:22
smoseryou have to basically turn lxc into chroot if you want it to work. which defeats most of the purpose of lxc17:23
pmatulisi just juju-deployed the mysql/wordpress stuff and pulling up wordpress site yields an error.  it's looking for file /etc/wordpress/config-localhost.php but i see that i have /etc/wordpress/config-node-e4115b137b36.local.php .17:53
=== matsubara is now known as matsubara-afk
cheez0rsmoser/hazmat/roaksoax: I'm re-commissioning my nodes- I went ahead and populated a SSH public key for my user in MAAS and I want to see if maybe that's what I'm missing.18:47
adam_gsmoser: jeez, i can't even get loop-control allowed into the container anymore18:58
smoseryou mgiht be fighting app armour now?18:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!