smoseraskb, did larsks not respond?00:04
smoserblackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/32427400:47
askbsmoser, yes ... thanks for the update on the bug!01:21
askbsmoser, so if this is an issue with selinux, then disabling selinux should allow cloud-init to proceed with passwd lock issue correct ?01:22
askbsmoser, I could do another run trying to selinux disabled01:23
smoseraskb, i'd expect that disabled selinux would fix that yes, but the proper fix is to fix your image to have the correct filesystem settings.02:19
askbsmoser, we don't know what is overriding the /etc/shadow permissions, possibly nova-agent which brings up the service02:22
askbsmoser, any issue with the fs settings I may have missed out ?02:22
=== cpaelzer_ is now known as cpaelzer
=== sambetts|afk is now known as sambetts
=== rangerpbzzzz is now known as rangerpb
=== nacc_ is now known as nacc
Odd_Blokesmoser: rharper: How is (ds-identify|cloud-id) run in systemd?  (i.e. how can I work out when in boot it ran?)15:17
rharperit runs at generator time15:17
rharperwhich is prior to any units running (generators help determine if a unit should run or not)15:18
Odd_BlokeAha, OK.15:18
Odd_BlokeSo that would (necessarily?) be before some mounts, right?15:18
rharperso, ds-identify (cloud_id) will write a .unit file in /run/systemd/targets/multiuser.target/wants/cloud-init.target or something like that to indicate that cloud-init units should run15:19
rharperwell, rootfs has to be moutned for systemd to run15:19
Odd_BlokeIt looks like on Softlayer it's running before the config drive mount is in-place, so it isn't detecting ConfigDrive as an option.15:19
rharperwe don't mount config drive15:19
rharperConfigDrive checks udev fs labels IIRC15:19
Odd_BlokeIt checks FS labels, but does also check the path if an FS label isn't found.15:20
Odd_BlokeBecause some clouds (e.g., of course, Softlayer) don't present a standard FS label.15:20
rharperI'm not sure I understand what is different about softlayer's config drive15:21
rharpercloud_id will check it in the same way we always have;  that said, the config drive device may be *slow*, or *not present* at the time;  which is one of the concerns that smoser raised in the past15:22
Odd_BlokeSo there are two parts to the issue.  Firstly, Softlayer only presents a config drive if metadata is specified for the instance.15:22
Odd_BlokeSo we have to do boot-time determination of the appropriate DS to use.15:22
Odd_BlokeThe second problem is that the config drive doesn't present in a standard way, so ds-identify doesn't recognise it.15:23
rharperds choice is between config drive and URL ?15:24
Odd_BlokeConfigDrive and NoCloud.15:24
rharpera cloud using NoCloud15:24
rharperI suppose this is more bare metally though IIRC15:25
Odd_BlokeYep and yep.15:25
rharpernocloud is present in the filesystem they laydown; that's straight forward, I suppose if you have running instances with the configdrive, we can experiement with ds-identify on such a system15:26
rharperand expand/update the configdrive detection, or possible introduce a ds_check_softlayer with the differences contained therein15:27
Odd_BlokeSo the code in question is https://git.launchpad.net/cloud-init/tree/tools/ds-identify#n55715:27
Odd_Bloke561 onwards is intended to handle this case.15:27
Odd_BlokeBut I'm not sure it ever will anywhere, because of mount ordering.15:28
Odd_Bloke(Well, except in cases where people are baking config drive data in to their images, but that is not an interesting case. :p)15:28
rharpermount ordering15:30
rharperyou believe that the device won't be present ?15:30
Odd_BlokeThe device is present, but we check the actual paths.15:30
Odd_BlokeI believe it won't be mounted.15:30
rharperit doesn't have to be15:30
Odd_Bloke558-560 could be modified to include the Softlayer "METADATA" disk label.15:30
rharperFS labels are populated via blkid15:31
rharperso, if the block device is present, and readable, and it has a filesystem label that blkid can detect, it should get listed15:31
Odd_BlokeRight, agreed.15:31
Odd_BlokeBut that's the first three lines of the function.15:31
Odd_BlokeI'm talking about the rest of the function. :p15:31
rharperwell, the mount part is not related then15:32
rharperwe find all of the filesystem labels present and check for known config drive labels;  if there is a *new* one for softlayer, then I suppose we should add that15:32
rharperI see15:33
rharperthe expectation is that it is mounted to /config_drive .15:34
Odd_BlokeOr /var/lib/cloud/seed/config_drive15:34
rharperyes, I see; hrm;  I don't know without looking at the code what would mount it there15:34
rharperin the datasource code, there are some test mount paths15:35
Odd_BlokeBut nothing will be mounted at ds-identify time so the specific location is irrelevant, no?15:35
Odd_Bloke*nothing other than /15:35
rharperwell, the path that ds-identify checks matters; but if nothing is going to mount it then no it does not matter15:35
rharperhere's what happens, (Or can);  if the filesystem label is present) then we return that we've found a ConfigDrive and that enables cloud-init to run15:36
rharperwhen cloud-init runs, it has code that will do the mount and read the contents15:36
rharperfor config drives15:36
rharperit's not clear to me why ds-identify also has the same content checks but I suspect there may be some that populate a directory (but do not have a fs lable)15:37
rharpersmoser: may confirm15:37
rharperso for softlayer, it should be sufficient that they attach a config drive with the 'config-2' label; and ds-identify will see that from blkid label parsing, and *enable* cloud-init to run later;  when cloud-init runs it will mount and consume the datasource15:38
rharperif cloud-init isn't running when a config drive is present on softlayer, then we need to examine the filesystem labels that were found15:38
smoser"METADATA" is kind of garbage15:39
smoserwhich is why i've avoided it. its not at all unreasonable for someone to have a block device with a filesystem label of 'METADATA'15:39
Odd_Blokerharper: Yeah, the problem is that Softlayer _don't_ present a "config-2" label, they present a "METADATA" label.15:40
Odd_BlokeSorry, I should have been more explicit about that previously.15:41
smoserso... i think that extending the ConfigDrive to notice 'METADATA' as a valid filesystem label would probably work.15:42
smoserbut i really don't like that without some other piece of identification15:42
smoserjust because its such a generic term.15:43
smoserif i had to do ConfigDrive again (the openstack implementation) i would have picked os-cfg-2 or something as the label to namespace it some as even 'config-2' is arguably generic15:43
smosereven if we did etend config drive to support 'METADATA" then we'd still have a wierd situation wehre there were 2 valid datasources.... which should be preferred ?15:44
Odd_Blokesmoser: NoCloud is always put last, right?15:44
Odd_Bloke(That's the appropriate ordering on SL.)15:45
smoserwell, in the default list...15:45
smoserdatasource_list: [ NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Ec2, CloudStack, None ]15:45
smoserbut id' really rather not build any (more) dependency on order.15:46
smoserwe're *almost* to a point where these dont have to rely on order.15:46
smoserEc2 is the last big one... once it turns strict on, then order doesnt really matter.15:46
smoserand if i had the option, i think i'd give preference to things in /var/lib/cloud/seed over any other source...15:47
smoserjust because "well, don't put stuff there if you dont want it there."15:47
smoserit wasn't ever really meant to be a full scale "datasource", but more for testing and such.15:48
Odd_BlokeIf we can work out some way of identifying SL, we could extend check_configdrive_v2 to do 'if SL and METADATA FS label found: yes'.15:48
smoseryes. i think so.15:49
smoserthats essentially what we do for brightbox now15:49
smoser(with ec2)15:49
smoseri realize we want/need to support softlayer, but i dont want doing so to really screw me later.15:49
smoserand i'm kind of feeling like we're not in a good situation because of some decisions that were made in the past, and i dont want to make it worse.15:50
Odd_Blokesmoser: Are you saying that you're worried this path would screw you later, or you're saying that this is our only viable option?15:50
Odd_Bloke(Or both? >.<)15:50
smoserif we get a positive identification "running on softlayer", then we're in reasonable shape15:51
smoserif we don't get one, then there are a whole lot of heuristics that crossing our fingers and hoping for the best.15:51
Odd_Blokesmoser: Am I right in thinking that you have a script that will harvest a bunch of data that I can pick through?15:52
smoserwell if you look at /run/cloud-init/ds-identify.log you'll see what it collects now15:52
smosersudo sh -c 'cd /sys/class/dmi/id && grep -r . *'15:53
smoseri think they're xen15:53
Odd_BlokeThere isn't any DMI data.15:53
smoserso there might be some stuff in /sys/hypervisor15:53
smoserbut iirc i dont think there was15:53
Odd_Blokehttp://paste.ubuntu.com/24727582/ is what's in there at first-boot.15:54
smoserdisapointed in myself that i didnt grab that info when i had an instance up in bug 168989015:54
ubot5bug 1689890 in cloud-init "Unable to identify datasource in IBM Bluemix" [Medium,Confirmed] https://launchpad.net/bugs/168989015:54
smosersudo sh -c 'cd /sys/hypervisor && grep -r . *'15:55
smoser^ Odd_Bloke run that15:55
Odd_BlokeI have a meeting to get to.15:55
Odd_BlokeSo I'll have to pick this up again later.15:55
smoserec2 says that their uuid i 'uuid' there will start with 'ec2'15:56
smoseri'd accept the crappy solution that 'uuid' there will start with '367fb048-'15:56
rharperdidn't realize it was a xen stack15:59
rharperwonder if we should be doing xenstore-ls on xen platforms16:00
smoseri'm more willing to take low-chance-of-false-positive solutions like collision on the first 8 chars of a uuid on xen than i am on kvm.16:00
smoserdue to the built in future proofing of xen16:00
rharperwell, if they don't want to control their UUID space, they can inject an arbitrary key into the store16:00
rharperit would be nice to see what's currently in there16:01
rharperon softlayer16:01
rharperdo we have a dmesg from one ?16:02
smoseri dont have one.16:05
smoserand xenstore-ls (package xenstore-utils) is not in an Ubuntu image by default.16:06
=== sambetts is now known as sambetts|afk
smoserblackboxsw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/32464017:19
smoserthoughts on my response ?17:19
blackboxswsure smoser so what would the main run? Maybe move the tools/cloudconfig-schema functionality into schema.py?17:20
blackboxswso argparsing etc?17:21
smoserthats what i was saying, and then just a small entry point thing in tools/ that is something like:17:23
blackboxswI think that's what you meant. Yeah I'll hoist the main out of tools/cloudconfig-schema and into schema.py17:23
smoser http://paste.ubuntu.com/24728389/17:23
blackboxswopps too late17:23
smoseryeah, so cloudconfig-schema then ends up being like ^17:23
blackboxswyeah makes sense. I might add another unit test or two to handle arg parse behavior etc (as I left tools without coverage as it wasn't officially part of the delivered cloudinit package/modules)17:24
blackboxswjust put up the azure ds-id from chassis-asset-tag too. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324875   and was about to work on SRU templates17:25
blackboxswcleaning up the schema right now17:25
smoseryeah, was reading that now.17:28
Odd_Blokerharper: http://paste.ubuntu.com/24728642/17:58
Odd_Bloke^ dmesg from SL17:58
Odd_Blokesmoser: We do actually install xenstore-utils on our SL images.18:01
smoserOdd_Bloke, doesnt really help. if ds-identify or cloud-init used it... then reasonably we'd have to at least Recommends it.18:06
smoserfixing CPC provided Ubuntu images is only half a fix. Ubuntu should just work.18:07
smoserblackboxsw, reviewed chassis-asset-tag18:08
rharperOdd_Bloke: thanks, I see that xen procfs  ... not sure what that is, but maybe that has some info without using the tools18:17
smoserOdd_Bloke, can you ssh-import-id ?18:18
smoserif you have one up.18:18
Odd_Blokesmoser: rharper: You're both on root@
=== smatzek_ is now known as smatzek
rharperOdd_Bloke: thanks18:31
rharperso, xenstore has bios-string options, http://paste.ubuntu.com/24728907/18:32
rharperit's very possible that they could set Softlayer specific stuff in there that can be read via xenstore-read18:32
rharper the whole set of  key/values with this:  xenstore-ls /local/domain/`xenstore-read domid`18:32
rharper# xenstore-read /local/domain/`xenstore-read domid`/bios-strings/bios-vendor18:34
rharper 18:34
smoserwe should write up a "Why we recommend you identify your cloud platform, and how best to do it."18:41
mak2Hi !!18:51
mak2smoser i have problems with cloud-init, this my log http://paste.ubuntu.com/24729092/18:52
mak2I can not login on vm18:53
mak2My environment is openstack master i'm  trying perfomer a deploy with de network interface flat and/or neutron (ironic.conf)18:55
blackboxswthanks smoser on the asset-tag review just pushed the updates to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/32464018:58
blackboxswand additional unit tests18:58
mak2smoser this is cloud-init-output.log http://paste.ubuntu.com/24729173/19:02
mak2Can someone help me?19:02
smosermak2, this is 12.04 ?19:05
smosermak2, well, your image is configured to search only the Ec2 datasource, but your'e running on openstack19:06
smoseropenstack provides a compatible ec2 metadata service at
smoserso that *should* be ok19:06
smoserbut when cloud-init is trying to read from that url, its getting19:07
smoser2017-05-31 18:06:59,187 - url_helper.py[WARNING]: Calling '' failed [0/120s]: bad status code [500]19:07
smoseri suspect you can recreate that simply with19:07
smosercurl ''19:07
smoserand i suspect your nova api server has stack traces19:07
mak2is 14.0419:07
smoserblackboxsw, i thin i like iut.19:10
mak2smoser this comand into VM curl ''and return 500 Internal Server Error19:11
mak2smoser nova-placement-api-error.log http://paste.ubuntu.com/24729333/19:18
mak2why do you know the image is configured to search only the Ec2 datasource?19:25
mak2smoser ?19:26
smoseri dont know where you got your image from19:30
smoserbut it has (i think) config in /etc/cloud/cloud.cfg that says datasource_list: ['Ec2', 'None']19:30
mak2smoser I created the diskimage-builder :D19:31
smoserin theory that should work, as i said, openstack does provide a ec2 api endpoint.19:31
smoserbut your cloud platform is broken and gives 500 errors when someone tries to access it19:32
smoserso thats busted19:32
smoseralmost entirely unrelated, but i'd really suggest just using the ubuntu images (http://cloud-images.ubuntu.com/)19:32
smoserwhile disk image builder could/should/may work, the cloud-images are official ubuntu output, *and* you dont' have to bother building them.19:33
smosermy feeling on building your own images is similar to my feelings on building your own glibc, gcc or kernel.19:34
smosersure, you can do it. its probably mostly supported, but ... unless you really know why you want to do such a thing, you probably dont want to do such a thing.19:34
smoserall that make sense ?19:34
smoserblackboxsw, https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/433/console19:35
smoseryou understand that ?19:35
smoser(use six.StringIO)19:36
mak2ahhhhh great I trying using export DIB_CLOUD_INIT_DATASOURCES=OpenStack19:36
blackboxswsmoser from six import StringIO py3 changed locations used to be io.StringIO in py219:38
blackboxswsmoser: I'll put up a trivial19:39
blackboxswmy bad19:39
blackboxswthanks CI :)19:39
smosermak2, that said i really suspect that changing the image wont help in any way19:39
smoseryour openstack is busted.19:39
blackboxswwe probably could/should chanage tests/unittests/helpers.py to avoid the try/ImportError dance on that too19:40
mak2smoser thx :D19:41
smosermak2, if you just want to see things fail faster you can use cirros19:43
smoseri'm pretty sure it'd boot faster and show you this same error19:43
mak2smoser ok I will try it! :D19:47
blackboxswsmoser: here's the diff http://paste.ubuntu.com/24729567/   shall I put up another review for it or push to cc-ntp-schema-validation?19:48
blackboxswthere were no other abuses. just what I had added for the with_logs stuff19:48
smoserjust push is fine.19:52
smoserwhat is cStringIO versus StringIO19:52
smoser>>> import cStringIO19:53
smoser>>> cStringIO19:53
smoser<module 'cStringIO' from '/usr/lib64/python2.6/lib-dynload/cStringIO.so'>19:53
smoser>>> import StringIO19:53
smoser>>> StringIO19:53
smoser<module 'StringIO' from '/usr/lib64/python2.6/StringIO.pyc'>19:53
smoseri guess just performant.19:53
blackboxswsmoser: pushed19:55
smoserblackboxsw, https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/434/console19:58
smoserthat ran against af370e135e8b9873ac8182ab6250aca061b420d119:59
blackboxswsmoser:turns out you actually need to add the file20:04
blackboxswaf370e1..f85cd6c should have it20:04
blackboxswmeh one more fix.20:20
Odd_Blokesmoser: cStringIO is StringIO written in C, but with some limitations.20:26
Odd_Bloke(Historically including problems with UTF-8.)20:26
Odd_BlokeWhich is to say, "it's Python". :p20:26
blackboxsw:) . smoser ci test runs agree I'm officially done w/ that branch20:30
blackboxswalso officially done w/ the azure  instance20:30
blackboxswminor review comments on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/32467721:40
blackboxswand I was seeing merge conflicts when trying to merge w/ trunk21:40
akarissmoser hi! qq for you - do you know how ubuntu cloud-init handles the presence of multiple default routes? e.g. eth0 has a default route, eth1 has a default route. as the kernel will only accept one default route, do you guys configure PBR or something similar, or are you simply blindly implementing the multiple default routes in the configuration files and have the kernel reject all additional default routes?21:48
akaristo be more precise:   if I create a neutron network with a router and default route, and a second network with a default gateway as well, and I attach both to an instance .. IMO there are 3 ways to handle this: implement PBR (for the inbound case) and for traffic from the instance outbound "role the dice" ... reject multiple default routes with an error message ... accept multiple default routes but role the dice for the one that wins (which is the fir21:51
akarisst default rotue that gets configured)21:51
=== rangerpb is now known as rangerpbzzzz

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!