[00:04] <smoser> askb, did larsks not respond?
[00:47] <smoser> blackboxsw, https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324274
[01:21] <askb> smoser, yes ... thanks for the update on the bug!
[01:22] <askb> smoser, so if this is an issue with selinux, then disabling selinux should allow cloud-init to proceed with passwd lock issue correct ?
[01:23] <askb> smoser, I could do another run trying to selinux disabled
[02:19] <smoser> askb, i'd expect that disabled selinux would fix that yes, but the proper fix is to fix your image to have the correct filesystem settings.
[02:22] <askb> smoser, we don't know what is overriding the /etc/shadow permissions, possibly nova-agent which brings up the service
[02:22] <askb> smoser, any issue with the fs settings I may have missed out ?
=== cpaelzer_ is now known as cpaelzer
=== sambetts|afk is now known as sambetts
=== rangerpbzzzz is now known as rangerpb
[14:16] <smoser> o/
[14:19] <sauloaislan> Hi!!
=== nacc_ is now known as nacc
[15:17] <Odd_Bloke> smoser: rharper: How is (ds-identify|cloud-id) run in systemd?  (i.e. how can I work out when in boot it ran?)
[15:17] <rharper> it runs at generator time
[15:18] <rharper> which is prior to any units running (generators help determine if a unit should run or not)
[15:18] <Odd_Bloke> Aha, OK.
[15:18] <Odd_Bloke> So that would (necessarily?) be before some mounts, right?
[15:19] <rharper> so, ds-identify (cloud_id) will write a .unit file in /run/systemd/targets/multiuser.target/wants/cloud-init.target or something like that to indicate that cloud-init units should run
[15:19] <rharper> well, rootfs has to be moutned for systemd to run
[15:19] <Odd_Bloke> It looks like on Softlayer it's running before the config drive mount is in-place, so it isn't detecting ConfigDrive as an option.
[15:19] <rharper> we don't mount config drive
[15:19] <rharper> ConfigDrive checks udev fs labels IIRC
[15:20] <Odd_Bloke> It checks FS labels, but does also check the path if an FS label isn't found.
[15:20] <Odd_Bloke> Because some clouds (e.g., of course, Softlayer) don't present a standard FS label.
[15:21] <rharper> I'm not sure I understand what is different about softlayer's config drive
[15:22] <rharper> cloud_id will check it in the same way we always have;  that said, the config drive device may be *slow*, or *not present* at the time;  which is one of the concerns that smoser raised in the past
[15:22] <Odd_Bloke> So there are two parts to the issue.  Firstly, Softlayer only presents a config drive if metadata is specified for the instance.
[15:22] <Odd_Bloke> So we have to do boot-time determination of the appropriate DS to use.
[15:23] <Odd_Bloke> The second problem is that the config drive doesn't present in a standard way, so ds-identify doesn't recognise it.
[15:24] <rharper> ds choice is between config drive and URL ?
[15:24] <Odd_Bloke> ConfigDrive and NoCloud.
[15:24] <rharper> a cloud using NoCloud
[15:25] <rharper> I suppose this is more bare metally though IIRC
[15:25] <Odd_Bloke> Yep and yep.
[15:26] <rharper> nocloud is present in the filesystem they laydown; that's straight forward, I suppose if you have running instances with the configdrive, we can experiement with ds-identify on such a system
[15:27] <rharper> and expand/update the configdrive detection, or possible introduce a ds_check_softlayer with the differences contained therein
[15:27] <Odd_Bloke> So the code in question is https://git.launchpad.net/cloud-init/tree/tools/ds-identify#n557
[15:27] <Odd_Bloke> 561 onwards is intended to handle this case.
[15:28] <Odd_Bloke> But I'm not sure it ever will anywhere, because of mount ordering.
[15:28] <Odd_Bloke> (Well, except in cases where people are baking config drive data in to their images, but that is not an interesting case. :p)
[15:30] <rharper> mount ordering
[15:30] <rharper> you believe that the device won't be present ?
[15:30] <Odd_Bloke> The device is present, but we check the actual paths.
[15:30] <Odd_Bloke> I believe it won't be mounted.
[15:30] <rharper> it doesn't have to be
[15:30] <Odd_Bloke> 558-560 could be modified to include the Softlayer "METADATA" disk label.
[15:31] <rharper> FS labels are populated via blkid
[15:31] <rharper> https://git.launchpad.net/cloud-init/tree/tools/ds-identify#n183
[15:31] <rharper> so, if the block device is present, and readable, and it has a filesystem label that blkid can detect, it should get listed
[15:31] <Odd_Bloke> Right, agreed.
[15:31] <Odd_Bloke> But that's the first three lines of the function.
[15:31] <Odd_Bloke> I'm talking about the rest of the function. :p
[15:32] <rharper> well, the mount part is not related then
[15:32] <rharper> we find all of the filesystem labels present and check for known config drive labels;  if there is a *new* one for softlayer, then I suppose we should add that
[15:33] <rharper> I see
[15:34] <rharper> the expectation is that it is mounted to /config_drive .
[15:34] <Odd_Bloke> Or /var/lib/cloud/seed/config_drive
[15:34] <rharper> yes, I see; hrm;  I don't know without looking at the code what would mount it there
[15:35] <rharper> in the datasource code, there are some test mount paths
[15:35] <Odd_Bloke> But nothing will be mounted at ds-identify time so the specific location is irrelevant, no?
[15:35] <Odd_Bloke> *nothing other than /
[15:35] <rharper> well, the path that ds-identify checks matters; but if nothing is going to mount it then no it does not matter
[15:35] <rharper> practically
[15:36] <rharper> so
[15:36] <rharper> here's what happens, (Or can);  if the filesystem label is present) then we return that we've found a ConfigDrive and that enables cloud-init to run
[15:36] <rharper> when cloud-init runs, it has code that will do the mount and read the contents
[15:36] <rharper> for config drives
[15:37] <rharper> it's not clear to me why ds-identify also has the same content checks but I suspect there may be some that populate a directory (but do not have a fs lable)
[15:37] <rharper> label)
[15:37] <rharper> smoser: may confirm
[15:38] <rharper> so for softlayer, it should be sufficient that they attach a config drive with the 'config-2' label; and ds-identify will see that from blkid label parsing, and *enable* cloud-init to run later;  when cloud-init runs it will mount and consume the datasource
[15:38] <rharper> if cloud-init isn't running when a config drive is present on softlayer, then we need to examine the filesystem labels that were found
[15:39] <smoser> "METADATA" is kind of garbage
[15:39] <smoser> which is why i've avoided it. its not at all unreasonable for someone to have a block device with a filesystem label of 'METADATA'
[15:40] <Odd_Bloke> rharper: Yeah, the problem is that Softlayer _don't_ present a "config-2" label, they present a "METADATA" label.
[15:40] <rharper> =(
[15:41] <Odd_Bloke> Sorry, I should have been more explicit about that previously.
[15:42] <smoser> so... i think that extending the ConfigDrive to notice 'METADATA' as a valid filesystem label would probably work.
[15:42] <smoser> but i really don't like that without some other piece of identification
[15:43] <smoser> just because its such a generic term.
[15:43] <smoser> if i had to do ConfigDrive again (the openstack implementation) i would have picked os-cfg-2 or something as the label to namespace it some as even 'config-2' is arguably generic
[15:44] <smoser> even if we did etend config drive to support 'METADATA" then we'd still have a wierd situation wehre there were 2 valid datasources.... which should be preferred ?
[15:44] <Odd_Bloke> smoser: NoCloud is always put last, right?
[15:45] <Odd_Bloke> (That's the appropriate ordering on SL.)
[15:45] <smoser> well, in the default list...
[15:45] <smoser> datasource_list: [ NoCloud, ConfigDrive, OpenNebula, DigitalOcean, Azure, AltCloud, OVF, MAAS, GCE, OpenStack, CloudSigma, SmartOS, Bigstep, Ec2, CloudStack, None ]
[15:46] <smoser> but id' really rather not build any (more) dependency on order.
[15:46] <smoser> we're *almost* to a point where these dont have to rely on order.
[15:46] <smoser> Ec2 is the last big one... once it turns strict on, then order doesnt really matter.
[15:47] <smoser> and if i had the option, i think i'd give preference to things in /var/lib/cloud/seed over any other source...
[15:47] <smoser> just because "well, don't put stuff there if you dont want it there."
[15:48] <smoser> it wasn't ever really meant to be a full scale "datasource", but more for testing and such.
[15:48] <Odd_Bloke> If we can work out some way of identifying SL, we could extend check_configdrive_v2 to do 'if SL and METADATA FS label found: yes'.
[15:49] <smoser> yes. i think so.
[15:49] <smoser> thats essentially what we do for brightbox now
[15:49] <smoser> (with ec2)
[15:49] <smoser> i realize we want/need to support softlayer, but i dont want doing so to really screw me later.
[15:50] <Odd_Bloke> Agreed.
[15:50] <smoser> and i'm kind of feeling like we're not in a good situation because of some decisions that were made in the past, and i dont want to make it worse.
[15:50] <Odd_Bloke> smoser: Are you saying that you're worried this path would screw you later, or you're saying that this is our only viable option?
[15:50] <Odd_Bloke> (Or both? >.<)
[15:51] <smoser> if we get a positive identification "running on softlayer", then we're in reasonable shape
[15:51] <smoser> if we don't get one, then there are a whole lot of heuristics that crossing our fingers and hoping for the best.
[15:52] <Odd_Bloke> smoser: Am I right in thinking that you have a script that will harvest a bunch of data that I can pick through?
[15:52] <smoser> well if you look at /run/cloud-init/ds-identify.log you'll see what it collects now
[15:53] <smoser> sudo sh -c 'cd /sys/class/dmi/id && grep -r . *'
[15:53] <smoser> i think they're xen
[15:53] <Odd_Bloke> There isn't any DMI data.
[15:53] <Odd_Bloke> ;.;
[15:53] <smoser> so there might be some stuff in /sys/hypervisor
[15:53] <smoser> but iirc i dont think there was
[15:54] <Odd_Bloke> http://paste.ubuntu.com/24727582/ is what's in there at first-boot.
[15:54] <smoser> disapointed in myself that i didnt grab that info when i had an instance up in bug 1689890
[15:54] <ubot5> bug 1689890 in cloud-init "Unable to identify datasource in IBM Bluemix" [Medium,Confirmed] https://launchpad.net/bugs/1689890
[15:55] <smoser> sudo sh -c 'cd /sys/hypervisor && grep -r . *'
[15:55] <smoser> ^ Odd_Bloke run that
[15:55] <Odd_Bloke> http://paste.ubuntu.com/24727590/
[15:55] <Odd_Bloke> I have a meeting to get to.
[15:55] <Odd_Bloke> So I'll have to pick this up again later.
[15:56] <smoser> ec2 says that their uuid i 'uuid' there will start with 'ec2'
[15:56] <smoser> i'd accept the crappy solution that 'uuid' there will start with '367fb048-'
[15:59] <rharper> didn't realize it was a xen stack
[16:00] <rharper> wonder if we should be doing xenstore-ls on xen platforms
[16:00] <smoser> i'm more willing to take low-chance-of-false-positive solutions like collision on the first 8 chars of a uuid on xen than i am on kvm.
[16:00] <smoser> due to the built in future proofing of xen
[16:00] <rharper> well, if they don't want to control their UUID space, they can inject an arbitrary key into the store
[16:01] <smoser> yeah.
[16:01] <rharper> it would be nice to see what's currently in there
[16:01] <rharper> xenstore-ls
[16:01] <rharper> on softlayer
[16:02] <rharper> do we have a dmesg from one ?
[16:05] <smoser> i dont have one.
[16:06] <smoser> and xenstore-ls (package xenstore-utils) is not in an Ubuntu image by default.
=== sambetts is now known as sambetts|afk
[17:19] <smoser> blackboxsw, https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324640
[17:19] <smoser> thoughts on my response ?
[17:20] <blackboxsw> sure smoser so what would the main run? Maybe move the tools/cloudconfig-schema functionality into schema.py?
[17:21] <blackboxsw> so argparsing etc?
[17:23] <smoser> thats what i was saying, and then just a small entry point thing in tools/ that is something like:
[17:23] <blackboxsw> I think that's what you meant. Yeah I'll hoist the main out of tools/cloudconfig-schema and into schema.py
[17:23] <smoser>  http://paste.ubuntu.com/24728389/
[17:23] <blackboxsw> opps too late
[17:23] <smoser> yeah, so cloudconfig-schema then ends up being like ^
[17:24] <blackboxsw> yeah makes sense. I might add another unit test or two to handle arg parse behavior etc (as I left tools without coverage as it wasn't officially part of the delivered cloudinit package/modules)
[17:25] <blackboxsw> just put up the azure ds-id from chassis-asset-tag too. https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324875   and was about to work on SRU templates
[17:25] <blackboxsw> cleaning up the schema right now
[17:28] <smoser> yeah, was reading that now.
[17:58] <Odd_Bloke> rharper: http://paste.ubuntu.com/24728642/
[17:58] <Odd_Bloke> ^ dmesg from SL
[18:01] <Odd_Bloke> smoser: We do actually install xenstore-utils on our SL images.
[18:06] <smoser> Odd_Bloke, doesnt really help. if ds-identify or cloud-init used it... then reasonably we'd have to at least Recommends it.
[18:07] <smoser> fixing CPC provided Ubuntu images is only half a fix. Ubuntu should just work.
[18:08] <smoser> blackboxsw, reviewed chassis-asset-tag
[18:17] <rharper> Odd_Bloke: thanks, I see that xen procfs  ... not sure what that is, but maybe that has some info without using the tools
[18:18] <smoser> Odd_Bloke, can you ssh-import-id ?
[18:18] <smoser> if you have one up.
[18:18] <Odd_Bloke> smoser: rharper: You're both on root@169.53.54.118
=== smatzek_ is now known as smatzek
[18:31] <rharper> Odd_Bloke: thanks
[18:32] <rharper> so, xenstore has bios-string options, http://paste.ubuntu.com/24728907/
[18:32] <rharper> it's very possible that they could set Softlayer specific stuff in there that can be read via xenstore-read
[18:32] <rharper>  the whole set of  key/values with this:  xenstore-ls /local/domain/`xenstore-read domid`
[18:34] <rharper> # xenstore-read /local/domain/`xenstore-read domid`/bios-strings/bios-vendor
[18:34] <rharper> Xen
[18:34] <rharper>  
[18:41] <smoser> we should write up a "Why we recommend you identify your cloud platform, and how best to do it."
[18:44] <rharper> ack
[18:51] <mak2> Hi !!
[18:52] <mak2> smoser i have problems with cloud-init, this my log http://paste.ubuntu.com/24729092/
[18:53] <mak2> I can not login on vm
[18:55] <mak2> My environment is openstack master i'm  trying perfomer a deploy with de network interface flat and/or neutron (ironic.conf)
[18:58] <blackboxsw> thanks smoser on the asset-tag review just pushed the updates to https://code.launchpad.net/~chad.smith/cloud-init/+git/cloud-init/+merge/324640
[18:58] <blackboxsw> and additional unit tests
[19:02] <mak2> smoser this is cloud-init-output.log http://paste.ubuntu.com/24729173/
[19:02] <mak2> Can someone help me?
[19:05] <smoser> mak2, this is 12.04 ?
[19:05] <smoser> 14.04
[19:06] <smoser> mak2, well, your image is configured to search only the Ec2 datasource, but your'e running on openstack
[19:06] <smoser> openstack provides a compatible ec2 metadata service at 169.254.169.254
[19:06] <smoser> so that *should* be ok
[19:07] <smoser> but when cloud-init is trying to read from that url, its getting
[19:07] <smoser> 2017-05-31 18:06:59,187 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [0/120s]: bad status code [500]
[19:07] <smoser> i suspect you can recreate that simply with
[19:07] <smoser> curl 'http://169.254.169.254/2009-04-04/meta-data/instance-id'
[19:07] <smoser> and i suspect your nova api server has stack traces
[19:07] <mak2> is 14.04
[19:10] <smoser> blackboxsw, i thin i like iut.
[19:10] <smoser> it
[19:10] <smoser> biab
[19:11] <mak2> smoser this comand into VM curl 'http://169.254.169.254/2009-04-04/meta-data/instance-id'and return 500 Internal Server Error
[19:18] <mak2> smoser nova-placement-api-error.log http://paste.ubuntu.com/24729333/
[19:25] <mak2> why do you know the image is configured to search only the Ec2 datasource?
[19:26] <mak2> smoser ?
[19:30] <smoser> i dont know where you got your image from
[19:30] <smoser> but it has (i think) config in /etc/cloud/cloud.cfg that says datasource_list: ['Ec2', 'None']
[19:31] <mak2> smoser I created the diskimage-builder :D
[19:31] <smoser> in theory that should work, as i said, openstack does provide a ec2 api endpoint.
[19:32] <smoser> but your cloud platform is broken and gives 500 errors when someone tries to access it
[19:32] <smoser> so thats busted
[19:32] <smoser> almost entirely unrelated, but i'd really suggest just using the ubuntu images (http://cloud-images.ubuntu.com/)
[19:33] <smoser> while disk image builder could/should/may work, the cloud-images are official ubuntu output, *and* you dont' have to bother building them.
[19:34] <smoser> my feeling on building your own images is similar to my feelings on building your own glibc, gcc or kernel.
[19:34] <smoser> sure, you can do it. its probably mostly supported, but ... unless you really know why you want to do such a thing, you probably dont want to do such a thing.
[19:34] <smoser> all that make sense ?
[19:35] <smoser> blackboxsw, https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/433/console
[19:35] <smoser> you understand that ?
[19:36] <smoser> (use six.StringIO)
[19:36] <mak2> ahhhhh great I trying using export DIB_CLOUD_INIT_DATASOURCES=OpenStack
[19:38] <blackboxsw> smoser from six import StringIO py3 changed locations used to be io.StringIO in py2
[19:39] <blackboxsw> smoser: I'll put up a trivial
[19:39] <blackboxsw> my bad
[19:39] <blackboxsw> thanks CI :)
[19:39] <smoser> mak2, that said i really suspect that changing the image wont help in any way
[19:39] <smoser> your openstack is busted.
[19:40] <blackboxsw> we probably could/should chanage tests/unittests/helpers.py to avoid the try/ImportError dance on that too
[19:41] <smoser> yeah
[19:41] <mak2> smoser thx :D
[19:43] <smoser> mak2, if you just want to see things fail faster you can use cirros
[19:43] <smoser> i'm pretty sure it'd boot faster and show you this same error
[19:47] <mak2> smoser ok I will try it! :D
[19:48] <blackboxsw> smoser: here's the diff http://paste.ubuntu.com/24729567/   shall I put up another review for it or push to cc-ntp-schema-validation?
[19:48] <blackboxsw> there were no other abuses. just what I had added for the with_logs stuff
[19:52] <smoser> just push is fine.
[19:52] <smoser> what is cStringIO versus StringIO
[19:53] <smoser> >>> import cStringIO
[19:53] <smoser> >>> cStringIO
[19:53] <smoser> <module 'cStringIO' from '/usr/lib64/python2.6/lib-dynload/cStringIO.so'>
[19:53] <smoser> >>> import StringIO
[19:53] <smoser> >>> StringIO
[19:53] <smoser> <module 'StringIO' from '/usr/lib64/python2.6/StringIO.pyc'>
[19:53] <smoser> i guess just performant.
[19:55] <blackboxsw> smoser: pushed
[19:58] <smoser> blackboxsw, https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/434/console
[19:58] <smoser> :?
[19:59] <smoser> that ran against af370e135e8b9873ac8182ab6250aca061b420d1
[20:04] <blackboxsw> smoser:turns out you actually need to add the file
[20:04] <blackboxsw> af370e1..f85cd6c should have it
[20:10] <smoser> k
[20:20] <blackboxsw> meh one more fix.
[20:26] <Odd_Bloke> smoser: cStringIO is StringIO written in C, but with some limitations.
[20:26] <Odd_Bloke> (Historically including problems with UTF-8.)
[20:26] <Odd_Bloke> Which is to say, "it's Python". :p
[20:30] <blackboxsw> :) . smoser ci test runs agree I'm officially done w/ that branch
[20:30] <smoser> \o/
[20:30] <blackboxsw> also officially done w/ the azure  instance
[21:40] <blackboxsw> minor review comments on https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/324677
[21:40] <blackboxsw> and I was seeing merge conflicts when trying to merge w/ trunk
[21:48] <akaris> smoser hi! qq for you - do you know how ubuntu cloud-init handles the presence of multiple default routes? e.g. eth0 has a default route, eth1 has a default route. as the kernel will only accept one default route, do you guys configure PBR or something similar, or are you simply blindly implementing the multiple default routes in the configuration files and have the kernel reject all additional default routes?
[21:51] <akaris> to be more precise:   if I create a neutron network with a router and default route, and a second network with a default gateway as well, and I attach both to an instance .. IMO there are 3 ways to handle this: implement PBR (for the inbound case) and for traffic from the instance outbound "role the dice" ... reject multiple default routes with an error message ... accept multiple default routes but role the dice for the one that wins (which is the fir
[21:51] <akaris> st default rotue that gets configured)
=== rangerpb is now known as rangerpbzzzz