/srv/irclogs.ubuntu.com/2017/06/01/#cloud-init.txt

=== sambetts|afk is now known as sambetts
=== shardy is now known as shardy_afk
=== shardy_afk is now known as shardy
tellingHi guys, I have an issue where the pubkey i specify for my ec2 instance doesnt get installed on the instance. The right key is available from the metadata, but doesnt get installed. I use cloud-init to provision my base AMI, might there be something I'm missing to "reset" for the keys to be installed?12:09
tellingAlso can I create PRs for the docs? Not familiar with launchpad12:32
smosertelling, yes you can do "merge proposals" for the docs.12:47
smoserfollow http://cloudinit.readthedocs.io/en/latest/topics/hacking.html12:47
smoserto do doc you run 'tox -e doc'12:47
smoserand, yes the key should get pulled in.12:48
smosercan you pastebin a /var/log/cloud-init.log from an instance ? this does onyl happen once per instance, though. not ever boot.12:48
tellingsmoser: right, so the issue is I dont clean up the instances dir on my base ami?12:48
tellingThis is my cloud-init log: https://ncry.pt/p/nxDn#FbcKAdDqbLBqKIqASdwzhw3PK6FtHibonWeX3zzIr0I12:51
tellingAs you can see it contains the base-ami log and the newly created jenkins instances log :)12:52
smosertelling, you should not have to clean up anything on an ami12:56
smoserthats the goal at least.12:56
tellingRight, i thought so initially12:56
tellingJust cant figure this out :)12:56
smoserhttp://paste.ubuntu.com/24737229/12:58
smosertelling, hm..13:00
smosercan you share the cloud-config that you're giving ? it seems like you've at least modified the default user13:01
tellingsmoser: yes, it's here: https://ncry.pt/p/pxDn#MpKZXCM083Hu_imxWbRE8mrPj1gbPzNYqbNhJblMgNs13:03
smosertelling, http://cloudinit.readthedocs.io/en/latest/topics/examples.html?highlight=users13:06
smoserso what is happening...13:06
smoseris that you are overriding the default 'users' list13:06
smoserand defining one without a 'default' entry13:06
smoserand the ssh keys from the metadata service go into which ever user is 'default'13:06
smoserthere may be a way for you to tag your jenkins user in the list13:07
smoserlet me check13:07
tellingNo13:07
tellingThats not what i want, i want it to be the ubuntu user as normal13:07
tellingI just want cloud-init to create a jenkins user, not for it to be default13:07
tellingCurrently it seems, no user gets the key. I feel like this can be cause of trouble13:11
tellingI can only access my instance because I have it provisioned with a master key in the base ami (Which im still debating with myself if I can justify or not)13:12
smoserah.13:15
smoserjust add an entry 'default'13:15
smoserin your users array13:15
tellingBut you would've expected my jenkins user in this case to get the key? Or whats to be expected?13:17
tellingBut that did indeed fix it, thanks a bunch smoser.13:22
smosertelling, well, the user that gets the key is the 'default' user (as described in the system_info)13:22
smoserbut that user is not modified/created if they are not int he 'users' array13:23
=== rangerpbzzzz is now known as rangerpb
smoserrharper, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324948 quickly ?14:43
smoseror blackboxsw14:45
blackboxswmornin'14:45
blackboxswsmoser: I don't see how that kwarg specified will not break on old  versions.14:49
blackboxswsmoser: nevermind, dumb14:50
blackboxswreordered call args +114:50
smoserthanks14:54
ragechinHey smoser, what's the proper procedure to formally objecting to the behavior of a module and discussing changing it?15:03
ragechinSomething in a github issue or whatever?15:03
Odd_Blokesmoser: So have we talked to SL about identifying information before?15:12
Odd_BlokeI'm about to send them an email asking, and want to make sure I'm not repeating a question we've already asked.15:13
smoseri'im sure you are :)15:19
Odd_Bloke:)15:19
smoserblackboxsw, http://paste.ubuntu.com/24738693/15:55
smoserthose on top of your15:55
smoserhttps://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/32487515:55
smoserand i'm good to pull15:55
smoser(not that i'm workign on that rather than other things i should be working on)15:55
blackboxswsmoser: +1 you need me to apply it?15:56
blackboxswsmoser: I know your listening15:56
smoseri'll just merge it16:01
smoseryou can ACK in that MP16:01
smoser(i commented there too)16:01
blackboxswIpushed16:02
blackboxswsmoser: sorry pushed16:02
smoserblackboxsw, fudge16:16
smoser    dmi_chassis_asset_tag_matches "${azure_chassis}" && return $DS_FOUND16:16
smoser    check_seed_dir azure ovf-env.xml && return ${DS_FOUND}16:16
smosermy feeling is that always seed dir indicates datasource.16:17
smoserif you put that stuff in, you are disabling all other checks. so i think we want/wanted those lines swapped16:17
blackboxswsmoser: don't we want to ds-identify to exit on first/cheapest check for a given datasource?16:18
smoserbah.16:19
smoserand i think in my squashing i duplicated a line too16:20
smoser /o\16:20
blackboxswhmm, I thought generally we wanted to claim success (DS_FOUND) as soon as possible for each datasource in ds-identify. Looking back over the script. I'm on the cloud-init hangout if you want to chat for faster turnaround16:22
smoserblackboxsw, well, the idea is that if you write /var/lib/cloud/seed/<datasource>/ you're feeding information to cloud-init. it overrides any checks. you're telling it "USE THIS DATA!".16:22
smosersure16:22
=== sambetts is now known as sambetts|afk
erick3kcan someone please help me17:07
erick3kam using debian 8 and cloud-init 0.7.7 and the root partition is not resizing17:08
erick3khttps://docs.google.com/document/d/1nAiVAt0rIG6Fl-4vEtvOpHR2e2F5W3j5OEO3POYhcKA/edit?usp=sharing17:08
blackboxswsmoser: I'm going through the DataSourceAzure code, and it looks like our seed dir ovf file can contain dscfg key in which the dscfg can override hostname_command, hostname_bounce.command, and agent_command with a custom command that we could have injected in the seed dir, Wouldn't that get us past seed dir being broken? It seems like setting dscfg['agent_command'] = ["not__builtin__"]  would let us fall through an pull17:10
blackboxswssh keys out of get_metadata_from_agent (instead of fabric)17:10
blackboxswerick3k: I wonder if that's related to https://bugs.launchpad.net/cloud-init/+bug/168486917:10
ubot5Ubuntu bug 1684869 in cloud-init "growing root partition does not always work with root=PARTUUID=" [Medium,Confirmed]17:10
blackboxswchecking your doc17:10
erick3ki see resize module not found and then found17:10
erick3kblackboxsw thank you17:10
blackboxswhmm, so I'm seeing "Jun  1 08:49:53 newdebian8 [CLOUDINIT] util.py[DEBUG]: Running command ('resize2fs', '/dev/vda1') with allowed return codes [0] (shell=False, capture=True)17:17
blackboxswJun  1 08:49:53 newdebian8 [CLOUDINIT] util.py[DEBUG]: Resizing took 0.007 seconds17:17
blackboxswJun  1 08:49:53 newdebian8 [CLOUDINIT] cc_resizefs.py[DEBUG]: Resized root filesystem "17:17
blackboxswhmm yeah then some module not found errors17:18
erick3kanyway to fix that?17:22
smoserblackboxsw, yeah.. i gues maybe.17:25
blackboxswerick3k: I'm not sure why that log appears to be running the init modules so many time17:25
blackboxswerick3k: I'm not sure why that log appears to be running the init modules so many times17:25
smoserblackboxsw, it runs on every boot17:25
smoseras you can shut down an instance, grow its diskk and it wants to make that magic17:25
smosererick3k, i suspect htat you do not have growpart17:26
smoserbut i woudl have thought you'd have some log of an error17:26
erick3ki did just check17:27
erick3ksmoser it is installed :( https://i.imgur.com/HdSDob4.png17:27
smosererick3k, you're not running the growpart module17:29
smosererick3k, and fyi, 'pastebinit' is installable in debian and is fabulous17:29
erick3kkool17:29
smoserie, you run 'pastebinit /var/log/cloud-init.log'17:29
erick3khow can i try and run the module?17:29
smoseror 'dpkg-query --show | pastebinit'17:30
erick3kcool17:30
smoseryour /etc/cloud/cloud.cfg probaly has no 'growpart'17:30
erick3kumm17:30
smoserhttp://paste.ubuntu.com/24739577/17:30
erick3kit does have resizefs17:30
smoserresizefs resizes the filesystem17:30
smoserbut growpart grows the partition to use any space at the end.17:30
smoseryou can probably urn17:31
smosersudo cloud-init single --frequency=always --name=growpart17:31
erick3khttps://0bin.net/paste/dd5jh1SY5RJtBCn-#v0WV8Is-//g0CwlmdqSEjT33ny325sdWqdqTVkjGeQ417:31
erick3kthats my cloud.cfg17:31
erick3krun cloud-init single --frequency=always --name=growpart and reboot?17:33
erick3knice17:33
erick3kthat worked17:33
smoseradd 'growpart' before 'resizefs'17:33
smoserin cloud_init_modules17:33
erick3ksmoser that does not work17:40
smoserwhy17:40
erick3ki increased the size, deleted /var/lib/instances/xxxx17:40
erick3krebooted17:41
erick3kand still has the same size17:41
smoseri'd hope that /var/log/cloud-init.log has a WARN message17:43
smoserdoes17:43
erick3kchecking17:43
smosercan you pastebin:  sudo growpart --dry-run --update=on17:43
smosercan you pastebin:  sudo growpart --dry-run --update=on /dev/vda 117:43
smoser(where 1 is the partition of the device that needs resizing)17:43
smoserand /dev/vda is the device17:44
erick3khttps://0bin.net/paste/2q9nEwhsy5wlg0-b#NBF6dIc76qJkDmnWhnv7K6fTrkqLVVwYkZ1IkszmPxp17:44
erick3ki dont see the module invoked on the log17:45
erick3koh hold on17:45
erick3knvm17:47
erick3ksmoser it does work, looks like cloud.cfg with -growpart didnt save17:47
erick3khehe17:47
erick3kthank you very much17:47
erick3kyou are the man, owe you a beer17:47
dpb1you can pay me now and I'll buy smoser a beer17:48
dpb1:)17:48
erick3kxD17:52
RedcavalierHi, is there a way to prevent cloud-init from caching the user-data locally?18:02
RedcavalierMy main issue is that if, for some reason, user-data is not received by a VM after a reboot, the user-data is loaded locally and some stuff that normally only get executed on first boot, get run again.18:08
RedcavalierThat results in password getting changed and other fun stuff.18:09
dgarstangTrying to get fs_setup and mounts to work. Getting "Failed to make '/mysql_data' config-mount". Any ideas?18:38
rharperpowersj: http://paste.ubuntu.com/24740067/18:45
rharperend of my tox run18:45
rharperRedcavalier: you don't have user-data but somehow it's cached?18:46
rharperdgarstang: cloud-init.log and the boot syslog may be helpful (along with any user-data supplied)18:46
dgarstangrharper: The file /var/log/cloud-init.log is empty.18:48
Redcavalierrharper, I do have user-data, it get executed on first boot. However, sometimes after a reboot (1 in 100, roughly), the instance is unable to go and fetch that userdata and decides to use the local data. Now, for some reason, it all the commands, even though they already ran on first boot.18:48
dgarstangrharper: Which is weird because I haven't made any logging changes18:48
rharperdgarstang: suggests that cloud-init didn't run or something cleared it.18:48
rharperdgarstang: what about syslog? that also may have some cloud-init output18:48
dgarstangrharper: It ran because it executed scripts and it's logged "Failed to make '/redis_data' config-mount" to /var/log/cloud-init-output.log18:49
Redcavalierall the commands, even though they ran on first boot already, get executed again*18:49
rharperRedcavalier: hrm, that sounds like a network metadata source; it may be going down a NoDataSource path;18:49
dgarstangrharper: Same stuff logged to /var/log/messages for cloud-init as /var/log/cloud-init-output.log18:50
Redcavalierrharper, oh right, if I can change the way the no data source case behave, I might be able to fix this.18:50
dgarstangSo, basically, cloud-init isn't logging _anything_ except stdout18:50
rharperRedcavalier: in general, cloud-init does some things every boot, per-instance, or always;  most of the initialization items like passwords, are run per-instance, so unless you've wiped out /var/lib/cloud/instance (symlink to the instance-id dir)18:50
rharperit shouldn't re-run any of those per-instance configurations18:51
dgarstangDoes this look correct? https://gist.github.com/dgarstang/8352ad8c51d834fbf3f282eb300d83d718:51
Redcavalierrharper, indeed, hence why the fact it re-runs them makes no sense to me.18:51
powersjrharper: on what branch of yours? I've don't recall seeing that happen before18:51
rharperRedcavalier: your /var/log/cloud-init.log should help digest the logic, if you have it18:51
rharperdgarstang: looking18:52
dgarstangI wish I had a /var/log/cloud-init.log... :(18:52
* powersj goes to make lunch 18:52
rharperdgarstang: seems odd not to have one;  the user data in your paste doesn not reference /mysql_data18:52
dgarstangoh hm maybe xvdj0 should be xvdj18:52
rharperit has redis_data18:52
rharperI don't think so18:53
dgarstangThat was from an earlier exammple sorry18:53
rharperdoes it come partitioned ?18:53
dgarstangNow it's complaining about /redis_data18:53
dgarstangrharper: Partitioned...? No, it's an external EBS disk18:53
dgarstangMounted tho at /dev/xvdj18:53
dgarstangsorry attached I mean18:53
rharperok, just trying to line up the device name to the device name in the mounts18:53
Redcavalierrharper, yes I have it, that's how I know it runs again when it shouldn't. Basically I get : util.py[WARNING]: Getting data from <class 'cloudinit.sources.DataSourceOpenStack.DataSourceOpenStack'> failed . It's then followed by messages telling me that the commands that are supposed to only run once per instance are being executed again.18:53
dgarstangWell, yeah maybe it should be xvdj not xvdj0 in mounts18:54
dgarstangBut...18:54
dgarstanglemme check something18:54
rharperRedcavalier: so, if you fail to get hit the openstack metadata service, I suspect that undefined bad things may happen;  I'm not 100% sure where the instance-Id comes from on openstack instance types18:54
dgarstangNope... so the disk /dev/xvdj isn't formatted. I can't mount it. So, it skipped the fs_setup step as well18:54
rharperRedcavalier: you can also look in /var/lib/cloud/instances/*18:54
rharperdgarstang: well, if somethings' not right with the formatting in fs_setup, then it won't have a device to mount which would prevent the mounts section from being successful18:55
dgarstangrharper: Sure. Getting it to format would be a nice first step18:55
rharperdgarstang: it looks fine to me, but maybe double check the device name?  and are the EBS volumes attached prior to boot (I don't know)18:56
rharperpowersj: master with some local changes18:56
dgarstangThe EBS disk is attached. It's not even logging the fail to format. GRRR18:56
Redcavalierrharper, logically, I'm betting it comes from /var/lib/cloud/instances/iid-datasource-none/user-data.txt18:56
rharperRedcavalier: that's the fallback18:56
rharperso since it failed to connect to the metadata service, it runs with fallback instance-id, which forces a new instance-id (it doesn't know which one)18:57
rharperhowever, I wonder why it couldn't use the cached metadata ... maybe smoser knows more about that datasource18:57
Redcavalierrharper, actually I'm wrong, it can't come from /var/lib/cloud/instances/iid-datasource-none/user-data.txt since that file is empty. After all, my original commands are getting re-run again.18:58
rharperRedcavalier: right, it's related18:58
rharpercloud-init couldn't find the instance-id, the fallback one (none found) is a *new* instance id, which means cloud-init won't know it hasn't run any of those config modules before and re-runs them with defaults18:58
rharperie, it didn't know the difference between image booting in a new instance (ie, I want you to regenerate my keys and etc) vs.  this temporary failure to identify itself as the previous instance-id18:59
Redcavalierrharper, that makes sense. I do need to change the behaviour then. Could I put information in the fallback datasource so it doesn't use the original one again?19:02
rharperunstable metadata service seems like the critical fix19:03
rharperI don't know without looking if cloud-init can know to use a previous instance-id in the case that the currently expected datasource is failing19:03
rharperwhat, if for example there were 3 different instance-id's ?  is it always right to use the one from the previous boot?  or how does cloud-init know which one to use on failure ?19:04
Redcavalierrharper, it's clearly copying the previous one though. For example, I have 2 IDs right now, 25603f8d-e602-4cde-8a6b-09e7387e1512 and i-00000165. 25603f8d-e602-4cde-8a6b-09e7387e1512 is my original instance ID and the good userdata. i-00000165 was created when the metadata failed. It contains the same user-data as 25603f8d-e602-4cde-8a6b-09e7387e1512, which in this case is bad as it get executed again.19:07
rharperare both the user-data.txt files empty ?19:08
Redcavalierrharper, nope, thewy both have my original data when the VM was created. However, I suspect that something else might be happening.19:08
rharperI don't believe there is any copying, the metadata service failed, so it picks an instance-id (Not sure if that's harcoded since it's a failure path); and then runs normal boot sequence , but since the instance-id is *different*, it doesn't find a cached instance dir, and runs like first boot of an instance19:09
rharperwithin that instance-id dir, it writes out the various files it normally would, including sem dir which tracks when it last ran those config modules19:10
RedcavalierOpenstack has compatibility to both offer its own metadata source, but also EC2-like data. If cloud-init queries openstack for EC2-like data, it will get a reply. I wonder if this is what is happening.19:10
rharperthe two datasources for OpenStack that I'm aware of are the metadata-service URL (network based) or a ConfigDrive (vm local)19:11
Redcavalierbecause if it really didn't get any data, by second instance ID user-data would indeed be empty19:11
rharperI believe we well perfer a ConfigDrive since it's local and we can detect that before we bring up networking ;19:11
Redcavalierrharper, yes, I've always been in favor of configdrives. However, my hand was forced by the higher up, but that's irrelevant here.19:12
rharperbut given your error message about the OpenStack datasource failed (and your log file may include URL timeouts) I don't thing that'd be a conflicting datasource (config drive) path19:12
rharperRedcavalier: I didn't mean to indicate a perferred solution, only that cloud-init detects config drives before it attempts to hit the network URL for OpenStack19:13
Redcavalierrharper, right, it didn't load from configdrive here though.19:13
dgarstang"Failed to make '/redis_data' config-mount"... This is getting REALLY annoying19:15
dgarstangSeriously, what am I supposed to do without debug output?19:16
dgarstangHere's my latest attempt .. https://gist.github.com/dgarstang/a7d24e74c78d00a4f90a94019154dd4419:16
dgarstangI don't need to create the dir do I? I looked through the code and it looks like cloud-init does it. I did see something about selinux in there19:18
dgarstangThis error "Failed to make '/redis_data' config-mount" implies it can't even create a direcrory19:20
rharperlooking19:20
dgarstangWhy, when I google "cloud-init "Failed to make"" do I get nothing?19:21
dgarstang(except source...)19:21
dgarstangActually, it HAS created /redis_data19:24
dgarstangRunning "mkfs -t ext4 /dev/xvdj" manually works fine19:25
rharperdgarstang: you should be able to re-run the module like this:  cloud-init --debug --force --file test.yaml single --name disk_setup --frequency always --report19:25
rharperwher test.yaml is your user-data you pasted19:25
dgarstangchecking19:25
rharperthen replace disk_setup with mounts19:25
rharperto run the mounts section19:26
dgarstangOk, I ran it.. got a lot of data, nothing relevant. Not even a mention of 'redis'19:27
dgarstangMaybe it's because I manually formatted the disk. I'll start fresh19:28
blackboxsw38% on SRU bug templates smoser :). think that's worth a coffee break... back in a few19:28
dgarstangWould be nice if that debug got logged somewhere on boot19:28
rharperdgarstang: it does, /var/log/cloud-init.log; but you said it was empty; that's not normal19:30
dgarstangrharper: It indeed is empty19:30
dgarstangIt's just an official CentOS 6 AMI19:30
rharperhrm19:31
dgarstangFresh instance. Same issue. Same error. When I run the command above, still doesn't work and IT'S output doesnt got to cloud-init-output AT ALL19:32
rharperthe single command won't go to the file, it goes to stdout; but when the module runs during boot, all of cloud-init's logging should go to /var/log/cloud-init.log ;  if it's empty, that's possibly a logging config issue in /etc/cloud/cloud.cfg but I would have thought that an official AMI would have cloud-init logged somewhere properly19:33
dgarstangThe scripts that I am running from runcmd have "exec >> /var/log/ec2-bootstrap.log ; exec 2>&1" at the top. Maybe that's confusing cloud-init's logging19:33
rharperthat's definitely going to grab the runcmd commands outputs and redirectl them19:34
dgarstangWell sure, for my scripts, but nothing else.19:34
dgarstangCloud-init is still sending its output to /var/log/cloud-init-output.log, but /var/log/cloud-init.log is empty19:35
rharperit's possible that the CentOS 6 AMI has it configured to send logs somewhere else, I would think syslog woudl be the other choice19:37
dgarstangChecking with those exec lines removed19:37
blackboxswback19:58
* blackboxsw checks runcmd tests on ubutnu w/ 2>&1 just to be sure I'm seeing logs where I think I should20:00
dgarstangThis is my latest attempt... https://gist.github.com/dgarstang/0fecc2dc7baaf1a2272a250bfe4da828... Output is going to cloud-init-output.log, and it's still logging "Failed to make '/redis_data' config-mount" ... still nothing going to cloud-init.log. This is so frustrating!20:01
dgarstangI can't make the cloud-init user data any more simple20:01
dgarstang"mount -t ext4 /dev/xvdj /redis_data" fails when I run it, so cloud-init did NOT format the disk20:03
smoserdgarstang, its empty on rhel before we changed the logging to go directly to a file20:03
dgarstangsmoser: Egads. How'd you do that?20:04
smoserdgarstang, look in /etc/cloud/cloud.cfg.d/05_logging.cfg20:04
smosersee the line about 'syslog' (log_syslog) . just comment it out.20:04
dgarstangsmoser: yah... don't know how to read/update that file20:04
dgarstangah20:04
smoserthen logging goes right to the file, not to syslog20:04
smoserwhich was busted on rhel20:04
dgarstangWell it's not going to syslog either20:04
smoserbecause cloud-init *thought* syslog was hooked up, but it wasnt20:04
smoseranyway, thats the fix.20:04
dgarstangAll it's logging to syslog is the "Failed to make '/redis_data' config-mount" message20:04
dgarstangIt's not doing it's regular debut to syslog20:05
smoseryea, thats kind of expected. syslog actually isnt all the way up. just make it go right to the file.20:05
smoserthats the change that is in upstream now20:05
dgarstangGood grief. Thanks. Well maybe the ability to format and mount disks is also broken?20:06
dgarstangActually I don't have that line in 05_logging20:06
dgarstangI've got " - &log_syslog |" and " - [ *log_base, *log_syslog ]"20:07
dgarstangI'd rather fix the inability to mount disks rather than the logging tho. Maybe I should try CentOS 720:07
smosersorry20:09
smoseryou want to comment out a line like20:09
smoser- [ *log_base, *log_syslog ]20:09
smosercomment that out (#)20:09
smoserthen you'll get a log20:09
smoserand the log should have a WARN message20:10
dgarstangOk. Lemme try centos 7 first20:10
smoserblackboxsw, 44%. and i havent started my next one yet. (utlemming had done the DigitalOcean one)20:12
dgarstangLooks like the logging issue is fixed in CentOS 7 .... but not it's ability to mount disks! ARGH!20:19
smoserdgarstang, can i see /var/log/ ?20:19
smoservar/log/cloud-init.log20:19
dgarstangsure, hang on20:19
dgarstangHere ya go. https://gist.github.com/dgarstang/2d9c134b7f230c84a82ed64c34a82852 Not much useful stuff there20:20
dgarstangLines 110,11120:21
dgarstangCorresponding user-data with cloud-init https://gist.github.com/dgarstang/2fa1ed8b630117bdf472744d537d7d2820:22
smoserfor as much as gists are used for a pastebin youd think they would have an interest in just offering a pastebin solution20:24
dgarstang:-\20:25
dgarstangI can't see how to make my config any simpler20:25
smoser... i suspect that selinux is in play20:25
dgarstangI dunno. I read through the cloud-init code and saw something about that so I added it20:26
smoseras20:26
smoserJun 1 20:17:35 ip-172-31-7-213 cloud-init: 2017-06-01 20:17:35,363 - util.py[WARNING]: Failed to make '/redis_data' config-mount20:26
dgarstang^ Yep20:26
smoseris from util.ensure_dir() failing with that argument20:26
dgarstangBut, it does create the directory20:26
smoserhm.. oh.20:27
dgarstangYep20:27
smoserdid rharper already go over this with you?20:27
smoserapparently there is a bug in python-selinux that couldbe related.20:27
dgarstang:-O20:27
smoser(sorry if we're re-treading here)20:27
dgarstangI'm the first person in history to use the cloud-init disk mount feature?20:27
dgarstangSo, maybe when I build my AMI I should have it disable selinux in /etc/sysconfig/selinux20:28
smoserhttps://bugzilla.redhat.com/show_bug.cgi?id=140652020:28
ubot5bugzilla.redhat.com bug 1406520 in libselinux "calling libselinux python restorecon fails on /var/lib/nfs/rpc_pipefs" [High,Verified]20:28
smoserdgarstang, are you able to make a change easily and re-try ?20:30
smoseri'd like to see the exception that is raised20:30
dgarstangWell, someone earlier said I ran rerun with "cloud-init --debug --force --file data.yml single --name disk_setup --frequency always —report"20:31
dgarstangso, I can try that20:31
rharpersmoser: I didn't mention the selinux python issue yet20:32
smoseryeah. i'm surprised you're nto getting debug messages in that log20:32
smoserthis is also 0.7.520:32
smoserwhich... realkly old. and i know that it is what you have, but, really old20:33
smoserb20:33
rharpersmoser: re: selinux;  there was a set_enforce 020:33
rharperas a bootcmd20:33
rharperhowever, someone else mentioned that maybe bootcmd didn't run "early" enough w.r.t disk_setup stuff20:33
smoseryeah, but can you even do that ?20:33
rharperwhich still seems odd to me given my reading of the config module order20:33
rharperbut possibly things changed in trunk vs. 0.7.520:33
dgarstangI've disabled selinux in /etc/selinux/config... Rebooting... Will try again after back up20:34
dgarstangWell, I think I'm not getting the error now but "cloud-init --debug --force --file data.yml single --name disk_setup --frequency always —report" isn't causing it to mount still20:37
dgarstangSo, maybe selinux needs to be disabled before boot20:37
dgarstangUGH20:40
smoserdgarstang, we are working recently on getting cloud-init much better on centos20:40
smoserbut as your'e finding, its not as thouroughly tested as Ubuntu.20:41
smoserthe goal is to improve it and get automated tests in  place to keep the function there.20:41
dgarstangSadness20:46
rharperdgarstang: the single runs just one module, you will need to call it again with '--name mounts' to perform the mount section20:54
dgarstangIs this progress? Using CentOS 7, and selinux disabled on boot, I'm getting only "util.py[WARNING]: Activating mounts via 'mount -a' failed" now. The other error has gone. it's still not formatting it though20:54
smoserhm.20:55
dgarstangI noticed earlier too that when I rebooted, I lost the custom hostname I had set. I presume it's picking that from DHCP20:56
dgarstangWait wait. I am still seeing "Failed to make '/redis_data' config-mount" ... missed it earlier20:56
dgarstangSo, I can probablt assume that the latest version of CentOS with Cloud-init can't format and mount disks20:56
dgarstangSighj20:58
smoserblackboxsw, i have to run. even 50% right now.21:03
blackboxswsmoser: make that 56%21:04
dgarstangI'm screwed21:12
dgarstangHow could I get cloud-init 0.7.9 onto centos 7?21:26
dgarstangMight 0.7.9 potentially fix my disk mount issue?21:28
rharperdgarstang: we're still working on getting daily rpms built properly; that's coming soon, in the mean time, I've some hand-built ones I;m testing with which may help at least determining if there are other issues in play:  http://people.canonical.com/~rharper/cloud-init/rpms/centos/7/cloud-init-0.7.9+123.g8ccf377-1.el7.centos.noarch.rpm21:35
rharperI think you can yum install that URL;  you'll have to accept the unsigned rpm21:35
rharperthat's trunk from a few weeks ago plus a few network configuration related fixes, but should be good enough to check that the disk_setup/mount stuff works (or fails in the same way) in which case, we should file a bug against cloud-init with your test user-data so we can get that resolved21:36
dgarstangOk, thanks. Tried with that RPM... same issue21:39
dgarstang"cloud-init --debug --force --file data.yml single --name mounts --frequency always —report" right?21:40
rharperdoes that show errors, or you just don't get mounts ?21:41
=== rangerpb is now known as rangerpbzzzz
dgarstangHmmm I got this ... https://gist.github.com/dgarstang/4eb93fda29decf54762b2d9356d505dc21:43
rharperhrm, doesn't appear to have run the mounts, lemme make sure I can get mounts to run via single21:44
dgarstangfs_setup is supposed to format right?21:44
rharperyes21:44
dgarstangActually I removed mounts from data.yml. Was trying to simplify21:44
dgarstangIt's not formatting however, as a manual mount command fails. "mount -t ext4 /dev/xvdj /redis_data"21:45
rharperdo we know get debug in /var/log/cloud-init.log ?21:46
dgarstangchecking21:46
dgarstangYes21:46
rharperok, then maybe let's run the fs_setup one and mounts, and gist the cloud-init.log21:46
dgarstangI got debug since I went from centos 6.5 to centos 721:46
rharperand see what we can see21:46
rharperand the updated rpm will have the logging fix21:46
rharperheh, 'ignorming' type in the output there21:47
dgarstang"cloud-init --debug --force --file data.yml single --name fs_setup --frequency always —report" ?21:47
rharperno, disk_setup21:47
dgarstangjeez21:48
rharperdisk_setup is the module name, it reads fs_setup config as one of the different areas of disk setup21:48
dgarstanghang on21:50
rharpersure21:51
dgarstangWell here's the whole thing https://gist.github.com/dgarstang/91122f453f7c512e4b527fe8c73aa41d21:53
rharperhrm21:54
dgarstangDoes it matter that this is a t2.nano instance? It has a 8Gb EBS disk attached at /dev/xvdj21:55
dgarstangNo ephemeral21:55
rharperno21:56
rharperfs_setup needs to be a list21:56
rharperfs_setup:21:56
dgarstanglist in the yaml?21:56
dgarstang:-O21:57
rharperlike in your original user-data post:  https://gist.github.com/dgarstang/8352ad8c51d834fbf3f282eb300d83d721:57
dgarstanghow did that happen. :-\21:57
rharperand it's *terrible* that we don't log something like, got an fs_setup but no list ...21:57
dgarstangThat must have happened when I removed the label line to simplify21:58
dgarstangMade it a list... same deal21:58
rharperand can you look at /dev/disk/by-uuid and see if there is a symlink to the disk ?21:59
rharperit's possible it was successful and just doesn't output much21:59
dgarstangWell that's 0a84de8e-5bfe-43e7-992b-5bfff8cdce43 -> ../../xvda1 ... but xvda is the root disk22:00
rharperwhen the device isn't present, I get an error reported, Failed during disk check for /dev/xvdj22:00
rharperand in the log, I can see cc_disk_setup debug output:   http://paste.ubuntu.com/24741508/22:03
dgarstangrharper: on my output?22:04
rharperno, you said it was the same22:05
dgarstangright... but I don't see anything like the output you pasted22:05
rharperbut I would expect to see cc_disk_setup output if the user-data for fs_setup was fix; which is surprising22:05
rharperhere's my change to your yaml you pasted; that *should* show  the 'setting up filesystems: ' message in /var/log/cloud-init.log; http://paste.ubuntu.com/24741550/22:07
dgarstangone sec22:11
dgarstangrharper: Didn't work22:11
dgarstangwait wait22:12
dgarstangnah. I dunno. :(22:13
dgarstang"Unable to convert /dev/xvdj to a device"22:20
dgarstangok, it actually formatted it. Didn't mount it tho22:21
dgarstangWhat command would I run to mount? Does disk_setup mount?22:24
rharperdgarstang: that's progress;lemme look at the code22:26
rharperif disk_setup didn't return OK, then you won't be able to mount it (unless for some reason it's already formatted);22:27
rharperone sec22:27
dgarstangI'm gonna roll a new AMI with the newer cloud-init for a start22:27
rharperok, I've gotta step out for a bit more;   would you be able to paste: cat  /proc/partitions ?  I'll read the disk_setup code and see what that message is coming from, but I suspect that something's awry with the device22:29
dgarstangrharper: kk, thanks22:30
dgarstangLooks like using  http://people.canonical.com/~rharper/cloud-init/rpms/centos/7/cloud-init-0.7.9+123.g8ccf377-1.el7.centos.noarch.rpm breaks system boot. Public ssh key doesn't get installed properly. Can't ssh in23:15
rharperdgarstang: lemme see, it may have changed the default user where the keys are installed23:59

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!