[13:37] <smoser> tribaal: on upgrade, that config file wont get re-written. on fresh install it will.
[13:37] <smoser> i'm pretty sure thats thecase at least.
[13:37] <smoser> so if you purged the package and then installed it, you should get the "fresh install" feel.
[13:37] <smoser> which is what will happen for new image builds
[13:38] <smoser> well... not a purge, but they actually *are* fresh installs
[14:05] <rharper> smoser: tribaal: thanks for the info; I was thinking that;  generally we wouldn't want an upgrade to migrate existing users to a new datasource;  but users could opt-in via the dpkg reconfigure;
[14:07] <smoser> i think what we'd really want to get to is no datasource_list int he packaging by default
[14:07] <smoser> and cloud-init just do the right thing
[14:07] <tribaal> yes, I guess trying / discovering datasources would make sense
[14:07] <rharper> smoser: right;  can you think of a blocker here ?
[14:08] <smoser> and then if the user *did* configure the list, then the would have effectively opted out of new datasources in ugprades which makes sense.
[14:08] <rharper> smoser: ideally it's empty; unless a user (or program) wrote a "local" override
[14:09] <smoser> rharper: i can't think of a blocker.
[14:09] <tribaal> one think that could be an isuue is for cases were ordering matters - for example if we maintain cloudstack compat and the cloudstack datasource is first to report "found". but that's not a blocker.
[14:09] <smoser> you just have to consider how upgrades work and such. and have the packaging do the right thing there.
[14:09] <tribaal> or say clouds migrating away from EC2 compatibility (which I assume exists)
[14:10] <smoser> tribaal: those points aren't reasons to drop datasource_list from the package i dont think.
[14:11] <smoser> they're valid things that we have to consider (and we have had bugs where the same instance swithced datasources on reboot)
[14:11] <tribaal> true
[14:11] <rharper> mmm, ibmcloud
[14:11] <smoser> but having "one more thing" (the packaging) holding that list is just a pita
[14:12] <smoser> dry - don't repeat yourself
[14:12] <smoser> cloud-init (thanks smoser) likes to have you repeat yourself ;)
[14:12] <rharper> smoser: I think we could drop it; ds-identify already provides the "default" list if nothing is present
[14:12] <rharper> would be worth experimenting with
[14:13] <smoser> yeah. i agree.
[14:13] <smoser> and event hen... i'd like to get rid of *that* list somehow.
[14:13] <smoser> but at least that isnt end-user configurable
[14:18] <rharper> hrm, yes;  and we still find them  anyhow rather than just accepting the names
[16:43] <rharper> blackboxsw: did we have general sru testing steps/language somewhere?  or did we say we wanted to write that down to point folks at it ?
[16:45] <blackboxsw> rharper: we don't but we said we wanted to create those docs.  I thought I added a trello card to todo soon, but I don't see it
[16:45] <blackboxsw> adding one now
[16:45] <rharper> ok
[16:46] <blackboxsw> https://trello.com/c/96vCbWgJ/1129-document-sru-manual-upgrade-testing-steps-for-cloud-init-for-the-community
[16:54] <rharper> thx
[16:54] <rharper> blackboxsw: was the bug verification script card updated with the 8 bugs for this SRU ?
[17:06] <blackboxsw> rharper: sorry, done https://trello.com/c/WzfgsM2l/9-write-verfication-scripts-in-ubuntu-sru-repository
[17:06] <blackboxsw> might still need pruning
[17:06] <blackboxsw> but just grabbed via git diff ubuntu/xenial..HEAD | log2dch --trello | grep pad.lv (for bug-related lines)
[17:07]  * blackboxsw has an errand. back in ~20
[17:07] <rharper> ok
[17:59] <blackboxsw> back
[18:08] <chillysurfer> i'm adding some code for a merge request to cloud-init, and i'm making the (abbreviated) call to cloudinit.log.getLogger().debug('my message here'), but in my testing that doesn't actually log to /var/log/cloud-init.log. am i doing something wrong? i see debug log messages in cloud-init.log so i think the level is fine
[18:10] <blackboxsw> hrm rharper I'm seeing strange behavior  trying to launch softlayer vms. The default user in cloud.cfg is ubuntu, yet that user is not being created. no tracebacks, and I see that the platform is also providing vendordata to reset root's password via a #!/bin/bash\necho 'root:$6$rounds=5000$5d6<redacted>' | chpasswd -e"
[18:10] <rharper> blackboxsw: eww
[18:10] <rharper> blackboxsw: let's collect-logs and look there
[18:10] <blackboxsw> this is OS-Code/Live
[18:10] <rharper> also check previous softlayer sru results ?
[18:11] <blackboxsw> rharper: yeah, will do on that
[18:11] <rharper> chillysurfer: deploying a newly installed patch, unittesting ?  can you provide more details on how you're testing ?
[18:11] <blackboxsw> chillysurfer: if you are actually talking about grabbing logs from unit tests, you'll need to set with_logs = True in the unit test
[18:11] <blackboxsw> then you can check self.logs.getvalue() in the unit tests
[18:12] <chillysurfer> rharper blackboxsw sorry i should've clarified. the code is written and i've generated a deb and testing it on a vm right now. not unit testing, actually running cloud-init through systemd
[18:13] <chillysurfer> i don't even really see where we wire up logging to /var/log/cloud-init.log. i see the def_log_file but i don't see us addHandler on it
[18:13] <rharper> chillysurfer: typically the module that logs will have something like:  from cloudinit import log as logging;  LOG = logging.getLogger(__name__)  ;
[18:14] <rharper> chillysurfer: the connection happens via /etc/cloud/cloud.cfg.d/05_logging.cfg
[18:14] <blackboxsw> chillysurfer: wiring logging comes config from /etc/cloud/cloud.cfg.d/05_logging.cfg
[18:14] <chillysurfer> rharper: yep that's exactly what i did (except i didn't pass in __name__)
[18:14] <blackboxsw> I lose :)
[18:14] <chillysurfer> rharper: ah nice :)
[18:14] <chillysurfer> in that case i should be logging
[18:14] <chillysurfer> really strange i'm not
[18:14] <chillysurfer> no need to flush or anything right?
[18:14] <rharper> no
[18:14] <rharper> so either the path isn't being taken
[18:15] <rharper> or something else isn't quite right;  do you have a diff of your changes you can share ?
[18:16] <chillysurfer> rharper: i don't have a diff at the moment. here's all i'm doing:
[18:16] <chillysurfer> from cloudinit import log as logging; _LOG = logging.getLogger(); _LOG.debug('hello world')
[18:16] <rharper> how is this module called ?
[18:17] <chillysurfer> rharper: i wired it up from cloudinit.cmd.main
[18:17] <rharper> cloud-init runs through cloudinit/cmd/main.py which will run through cloudinit/stages.py:Init() object which runs a sequence of "stages";
[18:17] <blackboxsw> rharper: looks like softlayer leaves around the /var/lib/cloud/seed/nocloud-net/user-data  which contains user: root definition, which overrides typical cloud-init behavior . That seed file is left over from earlier networklayer boot stages I think https://pastebin.ubuntu.com/p/SMWQvcZMhY/
[18:18] <chillysurfer> rharper: yep that's what i did. i added an arg, and then imported and run the action
[18:18] <rharper> chillysurfer: so depending on how early you are, logging may not be initialized;
[18:18] <rharper> if you're calling LOG. before setUpBasicLogging() in main is called
[18:18] <rharper> then you won't see that;  early main.py has it's own logging
[18:19] <rharper> which then gets replayed after setUpBasicLogging is called;  but previous calls to LOG before setup won't show up
[18:19] <chillysurfer> ahhhh wait yeah i see that we have `if args.debug: logging.setupBasiclLogging()`
[18:19] <rharper> yep
[18:19] <rharper> which will write to stderr
[18:19] <rharper> without config
[18:19] <rharper> but with the 05... it should show up in /var/log/cloud-init.log (but not in journalctl )
[18:20] <chillysurfer> rharper: oh so you're saying it should show up in cloud-init.log?
[18:20] <chillysurfer> even before setupBasicLogging?
[18:20] <rharper> no
[18:20] <chillysurfer> oh sorry got confused
[18:20] <chillysurfer> so you say this is expected that i'm seeing now logged messages
[18:20] <chillysurfer> *no
[18:20] <chillysurfer> *no logged messages
[18:20] <rharper> depending on where you are inserting your LOGs, it's possible.
[18:21] <Petaris> Is there anything special I need to do when using write_files with content that includes "#" in it?
[18:21] <chillysurfer> ah ok makes sense. i'll play around with this. thanks, rharper!
[18:21] <rharper> main.py has it's own logging functions before it gets to parsing /etc/cloud/* for log setup;  which eventually get's configured to write to /var/log/cloud-init.log;
[18:21] <blackboxsw> Petaris: you might want to base64 encode the content to avoid getting interpreted as yaml comments
[18:22] <blackboxsw> Petaris: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#writing-out-arbitrary-files
[18:23] <blackboxsw> rharper: per softlayer, this root behavior is there regardless of the current SRU. I'll spend some time trying to debug and file a separate bug  if needed
[18:23] <blackboxsw> it's happening on 19.1.1 on softlayer before the actual upgrade to 19.2.24
[18:23] <rharper> blackboxsw: hrm  I forget if softlayer has multiple datasources ...
[18:23] <rharper> that really smells wrong to me
[18:23] <rharper> I thought they were configdrive, not a nocloud
[18:23] <blackboxsw> rharper: agreed. I'll try the other 'flavors'
[18:24] <rharper> that seems like a metadata bug
[18:24] <blackboxsw> rharper: it does smell like metadata bug (or cloud-init not cleaning up earlier boot configs)
[18:24] <blackboxsw> since SL has multiple boot 'configurations
[18:25] <rharper> blackboxsw: right, there are like two stages
[18:25] <rharper> blackboxsw: but the seed file seems wrong, Ds is IBMCloud right ?
[18:25] <rharper> for softlayer ?
[18:26] <rharper> https://bugs.launchpad.net/cloud-init/+bug/1673637
[18:26] <rharper> blackboxsw: looks like it has some backgound
[18:28] <Petaris> blackboxsw, Thanks for the link.  I figured there would be need to either encode or escape them
[18:32] <rharper> https://paste.ubuntu.com/p/TK2ht5Vd8T/
[18:32] <rharper> Petaris: you don't need to encode for passing along # if you use the | marker in yaml , it will preserve the it as a string
[18:38] <blackboxsw> thx for the extra option rharper
[18:38] <Petaris> rharper, ok, cool
[18:38] <Petaris> Thanks
[18:38] <rharper> sure
[18:39] <blackboxsw> rharper: per SL, yes datasource is DataSourceIBMCloud [OS-Code/Live /dev/xvdh]
[18:40] <blackboxsw> will RTD on softlayer behavior again
[18:40]  * blackboxsw needs to swap that back into memory
[18:58] <faa> hello i'm FreeBSD tester and simple patch creator ;)
[18:59] <rharper> hiya
[19:00] <smoser> blackboxsw: what did you think was wrong ?
[19:00] <smoser> with ibmcloud
[19:01] <blackboxsw> smoser: the instances are coming up with root as the default user, instead of ubuntu (which is still configured default in /etc/cloud/cloud.cfg).
[19:01] <blackboxsw> smoser: and there is a v.l.c/seed/nocloud-net left around on the booted instance that is overriding user: root etc which is why my default ubuntu user isn't being set *I think*
[19:03] <smoser> blackboxsw: xenial or bionic
[19:03] <smoser> ?
[19:03] <blackboxsw> xenial: my expectation  was that I'd have an ubuntu user set up.
[19:03] <blackboxsw> but no ubuntu user, it's only trying to create root. root@SRU-worked-ibmcloud:~# grep User /var/log/cloud-init.log
[19:03] <blackboxsw> 2019-09-03 16:57:52,405 - __init__.py[INFO]: User root already exists, skipping.
[19:03] <blackboxsw> 2019-09-03 18:56:11,903 - __init__.py[INFO]: User root already exists, skipping
[19:04]  * blackboxsw tries bionic too
[19:04] <smoser> unless something has changed, the cloud team builds those images in some odd way.
[19:04] <blackboxsw> smoser: yeah I was double checking cpc
[19:04] <smoser> ibmcloud datasource ends up never being used.
[19:07] <smoser> i think if you poke around in /etc/cloud/ you'll see.
[19:07] <blackboxsw> smoser: looks like bionic works
[19:08] <smoser> and /etc/fstab and such
[19:08] <smoser> it is really bad
[19:08] <blackboxsw> just xenial falls over
[19:08] <smoser> and we decided not to fix it there.
[19:08] <smoser> or at least that it could happen later.
[19:10] <blackboxsw> so bionic is good (and has no /v/l/c/seed/nocloud-net config) root@SRU-worked-ibmcloud:~# egrep -r 'Adding|User' /var/log/cloud-init.log
[19:10] <blackboxsw> 2019-09-03 19:06:53,113 - __init__.py[DEBUG]: Adding user ubuntu
[19:10] <smoser> blackboxsw: iirc there is some udev hook that mounts the config disk to /var/lib/cloud/seed/nocloud
[19:10] <blackboxsw> ahh checking hooks
[19:10] <blackboxsw> thanks
[19:10] <smoser> i wrote all this down somewhere i thikn
[19:10] <smoser> probalb y  a google doc
[19:10] <blackboxsw> yeah /me was looking through transition google docs atm
[19:11] <blackboxsw> thank you scott
[19:11] <blackboxsw> and hackmd to cover our bases :)
[19:11] <smoser> https://hackmd.io/HQ4Brp_RTNyFPfopxtDFFw
[19:11] <smoser> is all i could find on hackmd
[19:11] <smoser> which mostly just points to / mirrors the datasource doc
[19:11] <smoser> (in DataSourceIBMCloud)
[19:12] <blackboxsw> https://hackmd.io/HQ4Brp_RTNyFPfopxtDFFw :)
[19:12] <blackboxsw> yep just found it
[19:12] <smoser> whcih only discusses how cloud-init works, not how you could modify an image with a bunch of changes so that cloud-init would be tricked
[19:13] <smoser> https://bugs.launchpad.net/cloud-init/+bug/1766401
[19:13] <smoser> https://git.launchpad.net/cloud-init/commit/cloud-init/sources?id=11172924a48a47
[19:15] <blackboxsw> ahh smoser ok, yet it seems on xenial we don't have all the magic we need to continue using the old datasource, as IBMCloud is being detected still
[19:15] <blackboxsw> ok I can file a bug on that and get it resolved
[19:16]  * blackboxsw tries to remember if we had a followup to try to activate IBMCloud ds on xenial.
[19:16] <smoser> hm..
[19:16] <smoser>  https://git.launchpad.net/cloud-init/commit/cloud-init/sources?id=11172924a48a47a7231d19d9cefe628dfddda8bf
[19:16] <smoser> looks broken
[19:16] <smoser> Diffstat shows 0 files changed
[19:16] <smoser> :-(
[19:17] <blackboxsw> ok xenial I do see the following though datasource_list: [ IBMCloud ] in etc/cloud/cloud.cfg.d/99_networklayer_common.cfg
[19:17] <smoser> well, that'd be a change
[19:18] <blackboxsw> yeah because I don't think that was set originally
[19:18] <blackboxsw> it may have been migrated back from bionic
[19:18] <smoser> yeah.
[19:32] <smoser> blackboxsw: so do you see two sets of logs ?
[19:33] <smoser> cloud-init logs / runs
[19:33] <smoser> can i ssh in?
[19:34] <blackboxsw> smoser: ok, there was a separate salesforce case requesting this change on xenial it seems. the shift from configdrive -> IBMCloud was requested by IBM on xenial
[19:34] <blackboxsw> and they desire root user there it seems.
[19:34] <smoser> because inconsistency makes sure people stay employed
[19:35] <smoser> so it is working as desired?
[19:35] <smoser> s/desired/"desired"/
[19:35] <blackboxsw> smoser: you can login if you like root@75.126.158.109
[19:35] <rharper> *sigh*
[19:35] <blackboxsw> "desired" for the moment, but I think we need to circle back w/ IBM/CPC on this
[19:36] <Odd_Bloke> My understanding is that root should be the default for all instances, so the ubuntu user may be an accident.
[19:44] <blackboxsw> Odd_Bloke: though I do wonder what the compelling need is to drop the ubuntu user altogether. I'm trying to get the background there on why that's a good thing to have the special case.
[19:45] <Odd_Bloke> Well, if the ubuntu user isn't cloud-init's default then it's going to have surprising behaviour compared to other places.
[19:45] <blackboxsw> though even in the Datasource itself IBM bakes in       {"cloud-init":"#!/bin/bash\necho 'root:$6$<snip>' | chpasswd -e"}
[19:45] <Odd_Bloke> Whereas if it doesn't exist at all, then people won't expect it to behave in any particular way.
[19:45] <blackboxsw> so we were already doing some funkiness for root user in IBM
[19:47] <smoser> wlel, generally speaking this looks like it is functioning as designed.
[19:48] <smoser> cpc could drop the hard coded datasource_list there. and i'd guess also the 'resize_rootfs' business
[19:49] <blackboxsw> smoser: yeah, I'm trying to shift back to the original datasource_list: [ConfigDrive, NoCloud]  to check on Xenial creating ubuntu user.
[19:49]  * blackboxsw needs to rebooit
[19:49] <blackboxsw> thx
[19:51] <smoser> so we would have to backport the ibmcloud packaging bits.
[19:53] <smoser> need abe6bcdfacdf2f6745e3acf5c76b332380b9c493 and 4f7bdee8cc37e0f36df888fbd79f0e68d6429431 back to xenial
[20:07] <blackboxsw> smoser: right, agreed, to enable IBM appropriately on new instances
[20:08] <blackboxsw> yeah, so looking like this particular case is IBM-desired root-only login on xenial. bionic++ looks like it will have ubuntu & root useres.
[20:08] <blackboxsw> users* even
[20:09] <smoser> i think that probably that is the way it has always been on xenial
[20:09] <smoser> and bionic changed to be more like ubuntu "standard"
[20:10] <smoser> so from that perspectrive... yeah, you want to keep the old behavior
[20:12] <rharper> blackboxsw: yay for no regression, and let's document that expection in the sru manual template , ?
[20:18] <smoser> oh. i forgot. xenial is reporting only still for ds-identify
[20:18] <blackboxsw> ok added https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1842510
[20:19] <smoser> blackboxsw: i'd say that is 'fix released' in cloud-init (Ubuntu)
[20:19] <smoser> not triaged
[20:19] <blackboxsw> still probably need to add IBMCloud to datasource_list config template even though image magic specifies an overried
[20:19] <blackboxsw> smoser: +1
[20:19] <blackboxsw> added confirmed to the xenial task
[20:21] <blackboxsw> ok /me gets back to softlayer SRU verification now :)
[22:06] <blackboxsw> hrm, probably about time for us to fix our rtd content generation.
[22:06] <blackboxsw> :/
[22:06] <blackboxsw> it seems we are still out of date.
[22:13] <blackboxsw> hrm github integrations seem successful
[22:13] <blackboxsw> last success 5 days ago, during our last commit
[22:13] <blackboxsw> yet cc_set_passwords documentation still seems out of date w/ tip
[22:38] <blackboxsw> there, fixed RTD for the moment. now why did I have to manually loging to readthedocs.io and click the build button. that should have been triggered automatically on doc updates
[22:43] <rharper> blackboxsw: yeah, it's not quite working, I have a script that we could cron
[22:44] <rharper> I think the challenge is the github mirror, I'm not sure it picks up new branches/commits;  I know when we create new tags, those are missing as well