[00:26] <rharper> smoser@smoser1117x:~$ sudo blkid
[00:26] <rharper> /dev/sda1: LABEL="cloudimg-rootfs" UUID="e990f8b3-1d6b-4615-8280-8ead4ed2fe7c" TYPE="ext4" PARTUUID="2865a230-01"
[00:26] <rharper> /dev/sdb1: UUID="c2108ffd-a113-4c1f-add4-4d6c92c7bef9" TYPE="ext4" PARTUUID="9b4d08ab-01"
[00:26] <rharper> \o/
[00:33] <rharper> smoser: here's my changes to get it working: http://paste.ubuntu.com/23493164/
[00:58] <smoser> rharper, still there ?
[00:58] <rharper> y
[00:58] <smoser> realpath will work on the devpath even if its a link
[00:59] <smoser> exists does the right thing on a link or dangling link
[00:59] <rharper> you mean if it's not a link ?
[00:59] <smoser> right?
[00:59] <rharper> but we *want* to be sure it's a real device ?
[00:59] <rharper> I'm confused why you ask ?
[00:59] <rharper> if the link exists but the device doesn't then we can't do anything
[01:00] <smoser> ln -s /does-not-exist /tmp/foo
[01:00] <smoser> $ python3 -c 'import os; print(os.path.exists("/tmp/foo"))'
[01:00] <smoser> False
[01:00] <smoser> for this case, just checking if somethign exists, exists is enough even on a link.
[01:00] <smoser> the only rason we convert it to realpath was because that is what find_devs_with would respond with
[01:01] <rharper> realpath only resolves a symlink if it has a valid target
[01:01] <smoser> right.
[01:01] <smoser> but it has a valid target
[01:01] <rharper> sure
[01:01] <smoser> or os.path.exists("this symllink") will say no
[01:02] <rharper> it doesnt check if it exists (the target)
[01:02] <smoser> it does
[01:02] <smoser> see above
[01:02] <rharper> only if it's pointing to other links
[01:02] <smoser> no. i doubt it
[01:03] <smoser> stupid me and isfile
[01:03] <rharper> we went throught this in curtin;  don't we want to know if the target kernel device exists? (/dev/sdb1 ) ?
[01:03] <smoser> i just assumed "in unix everythign is a file"
[01:03] <smoser> we do, but it is sufficient to say "os.path.exists()"
[01:03] <smoser> as os.path.exists("dangling symlink") == false
[01:03] <smoser> and
[01:03] <smoser> os.path.exists("non existant file") == false
[01:04] <rharper> (foudres) tmp % ls -al invalid
[01:04] <rharper> lrwxrwxrwx 1 root root 8 Nov 17 19:00 invalid -> /dev/sdz
[01:04] <rharper> (foudres) tmp % test -e /dev/sdz; echo $?
[01:04] <rharper> 1
[01:04] <rharper> (foudres) tmp % python3 -c 'import os; print(os.path.realpath("invalid"))'
[01:04] <rharper> /dev/sdz
[01:04] <rharper> I don't have an sdz, but realpath tells me that it's resolved any relative paths
[01:04] <smoser> ok.
[01:05] <smoser> http://paste.ubuntu.com/23493164/
[01:05] <smoser> in line 9 there, you dont have to do that....
[01:05] <smoser> you could use your line 14 ahead of it
[01:05] <rharper> (foudres) tmp % ls -al invalid crazylink
[01:05] <rharper> lrwxrwxrwx 1 root root 7 Nov 17 19:05 crazylink -> invalid
[01:05] <rharper> lrwxrwxrwx 1 root root 8 Nov 17 19:00 invalid -> /dev/sdz
[01:05] <rharper> (foudres) tmp % python3 -c 'import os; print(os.path.realpath("crazylink"))'
[01:05] <rharper> /dev/sdz
[01:05] <smoser> sure.
[01:05] <smoser> but print(os.path.exists("crazylink")
[01:05] <smoser> it will be false
[01:05] <rharper> yes
[01:05] <rharper> which proves that realpath does not check for existance
[01:06] <smoser> :)
[01:06] <rharper> just resolvs symlinks
[01:06] <smoser> i agree
[01:06] <rharper> =)
[01:06] <smoser> but we dont care
[01:06] <rharper> why not?
[01:06] <smoser> in your pasete
[01:06] <rharper> how can you know you can reformat if the device doesn't exist ?
[01:06] <smoser> if you moved line 14 up
[01:06] <rharper> that's the part I'm confused about
[01:06] <smoser> then "device does not exist"
[01:06] <smoser> before we ever try to resolve the devpath
[01:06] <rharper> hrm
[01:07] <rharper> then why did you fail before ?
[01:07] <rharper> indeed, exists does a realpath (I suppose)
[01:07] <smoser> because we used os.path.isfile
[01:07] <rharper> ah
[01:07] <rharper> right
[01:07] <smoser> which is not true for a block device
[01:07] <rharper> and there is no path.isdevice
[01:07] <smoser> but /me just assumed it was
[01:07] <smoser> right
[01:07] <rharper> well, it's pretty crappy to not have that but I suppose that's a posix thingy (or even linux ism)
[01:08] <smoser> yeah. curtin probably has one
[01:08] <smoser> just stat and check its doable. but this is sufficient. we're reasonably sure that //dev/disk/cloud/azure_resource is not goign to point to a character device or a directory
[01:08] <smoser> or even a file
[01:08] <smoser> and if it does, then other crap is gonna fail
[01:09] <rharper> y
[01:09] <smoser> thank you for your help, rharper!
[01:09] <rharper> np
[01:14] <smoser> rharper, did you find evidence of /dev/sdb1 existing
[01:14] <smoser> when the /dev/disk/cloud/azure_resource did not ?
[01:15] <smoser> er... /dev/disk/cloud/azure_resource-part1 did not
[01:15] <smoser> i did consider it possible htat we could find existance of /dev/disk/cloud/azure_resource before kernel had probed the partitiones
[01:16] <rharper> smoser: no, but it's certainly possible w.r.t udev race with symlink
[01:16] <rharper> I thought it was the case
[01:16] <smoser> but a udevadm settle shoudl work
[01:16] <smoser> right ?
[01:16] <rharper> yes (empty the queue)
[01:16] <smoser> or maybe not
[01:16] <smoser> actually not though
[01:16] <smoser> because i guess the device could exist
[01:17] <smoser> but the kernel not have probed
[01:17] <smoser> then we see it exist
[01:17] <smoser> udevamd settle
[01:17] <smoser> and *then* kernel probe
[01:17] <smoser> possible ?
[01:17] <smoser> so i dont know what to do there...
[01:17] <rharper> well, the kernel will probe the device
[01:17] <rharper> for us
[01:17] <rharper> and reports the partition table (and emits two events)
[01:17] <rharper> oen for the the disk and one for the partition
[01:17] <smoser> right. but it might not have gotten around to it
[01:17] <rharper> so, the only tiem would be between the two
[01:17] <smoser> right.
[01:17] <rharper> ok, I've gotta run and get my son, bb in 30
[01:18] <smoser> ok.
[01:18] <smoser> thankds.
[15:23] <jgrimm> powersj, for awareness -> https://gist.github.com/smoser/29ea35a797c0df1fcb6ac875a024efa9
[15:24] <jgrimm> consider as part of scenarios we'd like to make sure covered in integration tests
[16:21] <powersj> jgrimm: thanks - I will add that to the docs of things we need to do
[16:22] <jgrimm> thanks
[16:41] <powersj> rharper: there was another document you put together that had like 8 steps for cloud-init that was a good test. I can't seem to find my bookmark for it.
[16:41] <powersj> Do you recall where that is as well?
[16:42] <rharper> yes
 but, really it was just a list of things to check (didn't discuss how to do that): Networking, User Creation, Import ssh keys, ssh to instance, sudo privs, resizes disk, “refreshes on boot”, install snaps
 I wanted to check to see if the tests I wrote either covered all that or I should write a custom case for that
 nice
 refreshes on boot == re-runs parts of cloud-init on a reboot?
 the "refreshes on boot" and "install snaps" is snappy specific,  resizes disks happens both in cloud-image and UC16 image but in different places (UC16 initramfs does it vs. cloud-init)
 no, rather snapd checks if there are new snaps (core, kernel) downloads, installs and schedules a reboot
 that can be put on-the-side for now (UC16 specific test)
 and unrelated to cloud-init other than, it should still work in UC16 + cloud-init (which it does, though testing that is hard since we don't control what's in the channel)
 ok thank you!
[16:58] <vans163> Fedora 24 cloud image. "calling 10.0.2.2/latest/meta-data/instance-id"  does this mean my cloud-init iso is invalid?
[16:58] <vans163> so its defaulting to trying cloudinit over the network?
[17:01] <vans163> is instance-id: a manditory field? i was missing it
[17:04] <vans163> Alot of cloud images do not have root user inmind.  I want to set root password the same on all cloud images (vs using their default users)
[17:04] <vans163> so I just did   chpasswd: list | root:12345678
[17:05] <vans163> maybe this is wrong?
[17:05] <rharper> vans163: if you provide a configdrive then you need an instance-id
[17:05] <rharper> if you only provide user-data, it is going to search for an instance id
[17:06] <rharper> vans163: you don't want a root password, really ;  cloud-init can configure sudo privs for a user (The ubuntu cloud-image does this for the default ubuntu user if you don't specify your own list)
[17:09] <vans163> rharper: my problem is this is for veterans to beginners of linux, the vets can use their own custom cloud-config, but beginners will have a hrd time trying new distros if every distro has their own default user
[17:10] <vans163> rhapher: let me try with adding instance-id
[17:10] <vans163> (kind of how digital ocean and vultr do it)
[17:10] <vans163> every flavor of linux has the same default user (root)
[17:13] <rharper> =(
[17:13] <rharper> sudo bash, gets you to root
[17:13] <rharper> without setting root password
[17:14] <rharper> you can define the name of the user the same across distros via the cloud-init user: config
[17:14] <rharper> http://cloudinit.readthedocs.io/en/latest/topics/examples.html   the 'users'
[17:22] <vans163> rharper: that is what I am confused about, root should automatically be defined so if I wanted to make it only SSH key accessible, I do users: - name: root ssh-authorized-keys: ...
[17:22] <vans163> and it worked for me on a simple config, duplicate sshkeys were ignored
[17:23] <rharper> root of course exists, but it doesn't have a password set
[17:23] <rharper> to enable root ssh access, there may be another switch
[17:23] <rharper> remote root via ssh is really bad too
[17:25] <vans163> hum.. neither is working for me now. it prints "cloud init 0.7.7 running init-local".. and stucks. Then after a while starts trying URL, then boots the cloud image having not ran anything.  Going to try I guess to pick apart the cloud-config I have?
[17:25] <vans163> and see what could be making it freeze like this
[17:25] <rharper> http://cloudinit.readthedocs.io/en/latest/topics/modules.html#ssh
[17:26] <rharper> it's not freezing, but likely looking for a network datasource which you don't have;  I suggest using cloud-localds  --help as a starting point for building a working local data source (configdrive)
[17:26] <vans163> using a simple config taht smoser linked me, i had no problems eirler
[17:27] <vans163> going to start from there i guess
[17:27] <vans163> instance-id does this have to be string actually?
[17:27] <vans163> i just put 1 for testing, maybe thats causing the issue?
[17:27] <rharper> not sure what the min is
[17:27] <rharper> but using `uuidgen` output is going to work fine
[17:28] <rharper> the example is i-abcdefg
[17:34] <vans163> rharper: currently trying this config http://paste.ubuntu.com/23486774/
[17:35] <vans163> can that instance-id go into the user-data tho?
[17:35] <rharper> no, instance-id is always part of meta-data
[17:35] <rharper> so, the configdrive is an iso of a directory structure
[17:35] <rharper> in there will be a metadata dir with the instance-id in it
[17:36] <rharper> % cat m/meta-data
[17:36] <rharper> instance-id 7c871218-4ba9-45ba-854a-2a0a41cb928f
[17:36] <vans163> ah
[17:47] <vans163> so using that links config, I can boot BUT the password for user fedora (the default fedora 24 user) is not passw0rd
[17:47] <vans163> also i see the cloud init stuff go off
[17:49] <vans163> it says I need to provide a http://cloudinit.readthedocs.io/en/latest/topics/datasources.html#no-cloud.
[17:49] <vans163> no-cloud datasource?
[17:49] <rharper> well, you did, if you created the iso file and passed it in
[17:49] <vans163> yea i see.  let me go back to trying the ubuntu image again and confirm the password for ubuntu changed
[17:50] <rharper> nocloud is embedded within the image, the configdrive is equivalent ;
[17:50] <rharper> looking at the fedora distro code in cloud-init, need to see where the default username is set
[17:50] <vans163> i jsut googled for fedora cloud image default username, seems to indicate fedora.
[17:50] <rharper> right
[17:51] <rharper> but cloud-init has a distro class which can set these values
[17:51] <rharper> I'm not sure what's set at the moment in your cloud-init rpm
[17:51] <vans163> ahh
[17:52] <vans163> i have cloud init 0.7.7 not sure what ver of cloud-localds
[17:53] <rharper> http://paste.ubuntu.com/23496463/
[17:55] <rharper> I think that should get you a fedora user for now;  harlowja had a merge proposal which would have broken out the default distro and user stuff  better;    you can look at /etc/cloud/cloud.cfg in your instance to see what the system_info dict looks like
[18:00] <vans163> tried ubuntu. I see the cloud init go off saying things like "resizing root " etc, but go to login screen and i cant login as ubuntu/passw0rd
[18:02] <vans163> let me try that fedora config, back to fedora
[18:03] <powersj> smoser: rharper: have time for int. test meeting?
[18:03] <rharper> y
[18:05] <jgrimm> powersj, interetsted too if you have a HO
[18:05] <vans163> hum  hostfwd=tcp::2222-:22
[18:05] <vans163> is this line critical?
[18:06] <vans163> i took that line out
[18:06] <vans163> dont see why the need to forward a port.. but cloud-localds shows that line in the example
[18:08] <rharper> that just lets  you ssh into your VM if you're running local
[18:08] <rharper> vs login via console
[18:08] <rharper> it forwards host port 2222 into the guest 22
[18:12] <vans163> yea i see, so not critical to cloudinit i tried to ssh to localhost 2222
[18:12] <vans163> fedora and pass passw0rd
[18:16] <vans163> Thisis what im doing  https://gist.github.com/anonymous/1406b7470736587554075f1f9be58734
[18:16] <vans163> maybe the config is broken so it stops parsing it without error?
[18:18] <rharper> vans163: not sure, if you cleanly shutdown, then you can mount it up and extract /var/log/cloud-init*.log so we can see what happened
[18:19] <vans163> okay il do that, let me try 1 last thing.  notice how I passed paths to the file, but the example clearly say user-data and meta-data
[18:20] <vans163> maybe the cloud-localds does not form the right paths?
[18:24] <vans163> okay rharper: so if I follow the instruction to a T it works
[18:24] <vans163> x.x
[18:24] <vans163> but how can I add more flexibilty
[18:25] <rharper> cloud-localds does the right thing
[18:25] <vans163> like I cannot call the files meta-data as tehre could be a case when I spin up 2 at once
[18:25] <rharper> not following, you can point to the same iso, it's read-only
[18:25] <vans163> actually there maybe 1 thing I missed  the two EOF's
[18:25] <rharper> if you want the same user-data into multiple instances, you can re-use the same iso
[18:26] <vans163> or is that just for the terminal.. sec
[18:26] <vans163> yea that EOF is just for the terminal
[18:26] <vans163> (it does not change the file)
[18:27] <vans163> oaky let me try something then
[18:30] <vans163> fedora still does not work. Ubuntu worked, fedora I see my script executed (because sshd restarted)
[18:31] <rharper> ok, there may be something not quite right with the default cloud.cfg in the fedora cloud-init;  is there a fedora cloud-image that is publicly available ?
[18:32] <vans163> https://getfedora.org/en/cloud/download/  i got it from here the raw image and made it into a qcow2
[18:32] <rharper> ok
[18:32] <rharper> I'll see what I can figure out in a bit
[18:33] <vans163> md5sum  672669a587d653ef4100bcd03cb4ce32  /cloud/image_mnt/Fedora-Cloud-Base-24-1.2.x86_64.qcow2
[18:34] <vans163> thanks
[18:42] <vans163> rharper: i fetched the cloud-init-output.log does not seem to have anything useful.  cloud-init.log is empty
[18:42] <rharper> that's odd
[18:42] <vans163> do you still need the *-output.log?
[18:42] <rharper> not really, if cloud-init.log is empty, I don't see why we'd have anything in -output.log (that's stderr/stdout from cloud-init )
[18:42] <vans163> ah I see a line like this:
[18:43] <rharper> doe syslog have an error ?
[18:43] <rharper> maybe like a python stacktrace ?
[18:43] <vans163> ci-info: no authorized ssh keys fingerprints found for use fedora.
[18:44] <vans163> cc_set_passwords.py [WARNING]: no default or designated user to change password work
[18:44] <vans163> (that was the password: passw0rd line probably outside the scope of users)
[18:44] <vans163> *work/for
[18:45] <vans163> it seems as if it ignored the users: field
[18:46] <vans163> or no thats not the case. otherwise it would not print that ci-info:  (the order of those 2 is reversed, first the cc_set_passwrd.py warn, then that other)
[18:46] <rharper> I think the cloud.cfg inside the image is probably still set to ubuntu, rather than fedora;  need to check that out...
[18:47] <vans163> where woudl the syslog be located?
[18:47] <rharper> var/log/syslog
[18:48] <vans163> its not there
[18:48] <vans163> i can dump you the /etc/cloud/cloud.cfg?
[18:49] <rharper> I'm opening it up now
[18:49] <vans163> ah
[18:52] <rharper> it's working for me here
[18:52] <rharper> the default user is fedora
[18:52] <rharper> so, you can leave the 'user:' key out
[18:55] <vans163> does your config look the same as this http://paste.ubuntu.com/23486774/?
[18:55] <vans163> let me try redownload the image again maybe i got the openstack one by accident, pretty certain i got raw tho.
[18:55] <rharper> yes
[18:56] <vans163> so you just downloaded the raw and did  xz Image.raw.xz  then used that?
[18:57] <rharper> I used the qcow2 file
[18:57] <rharper> it works fine
[18:57] <rharper> but the 0.7.7 I think has a bug that's been fixed upstream but not in there, yet
[18:57] <rharper> so, it's trying to load the user-data but failing
[18:57] <rharper> which is why you're seeing anything besides fedora: passw0rd not working
[19:01] <vans163> let me try.. also does that mean I wont be able to set a ssh key for a user other then fedora?
[19:01] <rharper> I think so, but I need to find out a bit more
[19:01] <rharper> I believe the cloud-init in the image is going to need a patch; but I need to find out the actual bug
[19:03] <vans163> say once it gets patched (if it does) would I be able to simply mount the image and apply the patch?
[19:03] <vans163> vs a more complex process
[19:05] <rharper> cloud-utils includes a tool called mount-image-callback which would aide in injecting an updated package (or applying a patch)
[19:06] <rharper> what does your meta-data look like ?
[19:07] <rharper> https://gist.github.com/anonymous/1406b7470736587554075f1f9be58734
[19:08] <vans163> hum wtf
[19:08] <vans163> that raw image is 500mb while the decompressed openstaack qcow2 is 1gb
[19:08] <rharper> that looks fine, you can drop the users dict;  then you can in-line a ssh-authorized key
[19:08] <vans163> let me try that other image
[19:09] <vans163> instance-id: i-abcdefg   is all thats inside meta-data
[19:10] <rharper> that's fine; I had a typo in mine
[19:10] <rharper> fixed that and the sample config works just fine
[19:10] <rharper> I'm testing adding an ssh public key to fedora user
[19:11] <vans163> going to brb in an hour or two
[19:11] <rharper> sure
[19:31] <rharper> vans163: so, things appear to be working fine for me:  http://paste.ubuntu.com/23496940/  is the template I'm using, I see a default password for user: fedora, and then import a ssh public key to them, I used 'users' to define a new user 'rharper' and import an ssh public key there.  That runs fine in the qcow2 image (I expect the raw to work just as well);  I can ssh in to either fedora or 'rharper' in that image after
[19:31] <rharper>  cloud-init runs.
[22:46] <vans163> rharper: I am back let me test
[22:47] <seanbright> hi. about to head out for the weekend, does everything look ok with this merge request:
[22:47] <seanbright> https://code.launchpad.net/~sbright/cloud-init/+git/cloud-init/+merge/311313
[22:48] <seanbright> first time submitting a LP patch
[22:48] <vans163> rharper: this is not working for me on ubuntu :( http://paste.ubuntu.com/23496940/
[22:49] <vans163> i replaced <ssh key here> with  wtff abababab ee
[22:49] <vans163> in both cases
[22:49] <vans163> if I only leave the top 4 lines of that paste, it works
[22:49] <vans163> otherwise even ubuntu users password is not passw0rd
[22:56] <rharper> vans163: if you don't want the ssh key, take the 3 lines out;  5-7, and 13-15; don't make them gibberish
[22:57] <vans163> the command works its starnge i just mounted the drive and the user has a home folder authorized keys are there
[22:57] <vans163> but at the login prompt i cant login as the user
[22:57] <vans163> (using vnc)
[22:58] <vans163> so cloud init successfully created everything
[22:59] <vans163> maybe some command is blocking..
[23:01] <vans163> i cant even ssh to it
[23:02] <vans163> as soon as i delete everything except the top 4 lines of that config file I can login as ubuntu/passw0rd
[23:02] <vans163> when I add the line to create a new user (leaving the top 4 as well)
[23:02] <vans163> the /home/jharper/.ssh/authorized_keys is there
[23:02] <vans163> but i cannot login as jharper/passw0rd nor ubuntu/passw0rd
[23:03] <vans163> also /var/log/cloud-init.log is quite populated
[23:27] <vans163> now even ubuntu stopped working
[23:27] <vans163> something is def wrong with my system i guess
[23:27] <vans163> and not cloud init
[23:29] <vans163> no nevermind i got confused, i just retried ubuntu with the top 4 lines of the config. everything works
[23:29] <vans163> as soon as i make config more complex. nothing
[23:48] <vans163> call it a bug or a feature but apparnetly passwords like passw0rd and toor dont work
[23:48] <vans163> using wtf12345! for example works
[23:49] <vans163> so maybe something inside cloud-init is checking the passwords complexity or something and silently failing
[23:49] <vans163> and a silent failure here seems to make the entire config fail or locks the system from login or something
[23:51] <vans163> fedora works finally!
[23:51] <vans163> so conclusion. 99% of ppl use cloud-init in openstack
[23:51] <vans163> and openstack generates the passwords by random
[23:51] <vans163> for ppl using cloud-init and setting a password of hello, it silently fails
[23:52] <vans163> thus this bug/feature was never reported
[23:52] <vans163> maybe this conclusion is wrong and the underlying OS is limiting it but i doubt it as i can set a password of 12345 for excample on fedora 24 wiht no problems
[23:52] <vans163> even for root