[13:37] tribaal: on upgrade, that config file wont get re-written. on fresh install it will. [13:37] i'm pretty sure thats thecase at least. [13:37] so if you purged the package and then installed it, you should get the "fresh install" feel. [13:37] which is what will happen for new image builds [13:38] well... not a purge, but they actually *are* fresh installs [14:05] smoser: tribaal: thanks for the info; I was thinking that; generally we wouldn't want an upgrade to migrate existing users to a new datasource; but users could opt-in via the dpkg reconfigure; [14:07] i think what we'd really want to get to is no datasource_list int he packaging by default [14:07] and cloud-init just do the right thing [14:07] yes, I guess trying / discovering datasources would make sense [14:07] smoser: right; can you think of a blocker here ? [14:08] and then if the user *did* configure the list, then the would have effectively opted out of new datasources in ugprades which makes sense. [14:08] smoser: ideally it's empty; unless a user (or program) wrote a "local" override [14:09] rharper: i can't think of a blocker. [14:09] one think that could be an isuue is for cases were ordering matters - for example if we maintain cloudstack compat and the cloudstack datasource is first to report "found". but that's not a blocker. [14:09] you just have to consider how upgrades work and such. and have the packaging do the right thing there. [14:09] or say clouds migrating away from EC2 compatibility (which I assume exists) [14:10] tribaal: those points aren't reasons to drop datasource_list from the package i dont think. [14:11] they're valid things that we have to consider (and we have had bugs where the same instance swithced datasources on reboot) [14:11] true [14:11] mmm, ibmcloud [14:11] but having "one more thing" (the packaging) holding that list is just a pita [14:12] dry - don't repeat yourself [14:12] cloud-init (thanks smoser) likes to have you repeat yourself ;) [14:12] smoser: I think we could drop it; ds-identify already provides the "default" list if nothing is present [14:12] would be worth experimenting with [14:13] yeah. i agree. [14:13] and event hen... i'd like to get rid of *that* list somehow. [14:13] but at least that isnt end-user configurable [14:18] hrm, yes; and we still find them anyhow rather than just accepting the names [16:43] blackboxsw: did we have general sru testing steps/language somewhere? or did we say we wanted to write that down to point folks at it ? [16:45] rharper: we don't but we said we wanted to create those docs. I thought I added a trello card to todo soon, but I don't see it [16:45] adding one now [16:45] ok [16:46] https://trello.com/c/96vCbWgJ/1129-document-sru-manual-upgrade-testing-steps-for-cloud-init-for-the-community [16:54] thx [16:54] blackboxsw: was the bug verification script card updated with the 8 bugs for this SRU ? [17:06] rharper: sorry, done https://trello.com/c/WzfgsM2l/9-write-verfication-scripts-in-ubuntu-sru-repository [17:06] might still need pruning [17:06] but just grabbed via git diff ubuntu/xenial..HEAD | log2dch --trello | grep pad.lv (for bug-related lines) [17:07] * blackboxsw has an errand. back in ~20 [17:07] ok [17:59] back [18:08] i'm adding some code for a merge request to cloud-init, and i'm making the (abbreviated) call to cloudinit.log.getLogger().debug('my message here'), but in my testing that doesn't actually log to /var/log/cloud-init.log. am i doing something wrong? i see debug log messages in cloud-init.log so i think the level is fine [18:10] hrm rharper I'm seeing strange behavior trying to launch softlayer vms. The default user in cloud.cfg is ubuntu, yet that user is not being created. no tracebacks, and I see that the platform is also providing vendordata to reset root's password via a #!/bin/bash\necho 'root:$6$rounds=5000$5d6' | chpasswd -e" [18:10] blackboxsw: eww [18:10] blackboxsw: let's collect-logs and look there [18:10] this is OS-Code/Live [18:10] also check previous softlayer sru results ? [18:11] rharper: yeah, will do on that [18:11] chillysurfer: deploying a newly installed patch, unittesting ? can you provide more details on how you're testing ? [18:11] chillysurfer: if you are actually talking about grabbing logs from unit tests, you'll need to set with_logs = True in the unit test [18:11] then you can check self.logs.getvalue() in the unit tests [18:12] rharper blackboxsw sorry i should've clarified. the code is written and i've generated a deb and testing it on a vm right now. not unit testing, actually running cloud-init through systemd [18:13] i don't even really see where we wire up logging to /var/log/cloud-init.log. i see the def_log_file but i don't see us addHandler on it [18:13] chillysurfer: typically the module that logs will have something like: from cloudinit import log as logging; LOG = logging.getLogger(__name__) ; [18:14] chillysurfer: the connection happens via /etc/cloud/cloud.cfg.d/05_logging.cfg [18:14] chillysurfer: wiring logging comes config from /etc/cloud/cloud.cfg.d/05_logging.cfg [18:14] rharper: yep that's exactly what i did (except i didn't pass in __name__) [18:14] I lose :) [18:14] rharper: ah nice :) [18:14] in that case i should be logging [18:14] really strange i'm not [18:14] no need to flush or anything right? [18:14] no [18:14] so either the path isn't being taken [18:15] or something else isn't quite right; do you have a diff of your changes you can share ? [18:16] rharper: i don't have a diff at the moment. here's all i'm doing: [18:16] from cloudinit import log as logging; _LOG = logging.getLogger(); _LOG.debug('hello world') [18:16] how is this module called ? [18:17] rharper: i wired it up from cloudinit.cmd.main [18:17] cloud-init runs through cloudinit/cmd/main.py which will run through cloudinit/stages.py:Init() object which runs a sequence of "stages"; [18:17] rharper: looks like softlayer leaves around the /var/lib/cloud/seed/nocloud-net/user-data which contains user: root definition, which overrides typical cloud-init behavior . That seed file is left over from earlier networklayer boot stages I think https://pastebin.ubuntu.com/p/SMWQvcZMhY/ [18:18] rharper: yep that's what i did. i added an arg, and then imported and run the action [18:18] chillysurfer: so depending on how early you are, logging may not be initialized; [18:18] if you're calling LOG. before setUpBasicLogging() in main is called [18:18] then you won't see that; early main.py has it's own logging [18:19] which then gets replayed after setUpBasicLogging is called; but previous calls to LOG before setup won't show up [18:19] ahhhh wait yeah i see that we have `if args.debug: logging.setupBasiclLogging()` [18:19] yep [18:19] which will write to stderr [18:19] without config [18:19] but with the 05... it should show up in /var/log/cloud-init.log (but not in journalctl ) [18:20] rharper: oh so you're saying it should show up in cloud-init.log? [18:20] even before setupBasicLogging? [18:20] no [18:20] oh sorry got confused [18:20] so you say this is expected that i'm seeing now logged messages [18:20] *no [18:20] *no logged messages [18:20] depending on where you are inserting your LOGs, it's possible. [18:21] Is there anything special I need to do when using write_files with content that includes "#" in it? [18:21] ah ok makes sense. i'll play around with this. thanks, rharper! [18:21] main.py has it's own logging functions before it gets to parsing /etc/cloud/* for log setup; which eventually get's configured to write to /var/log/cloud-init.log; [18:21] Petaris: you might want to base64 encode the content to avoid getting interpreted as yaml comments [18:22] Petaris: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#writing-out-arbitrary-files [18:23] rharper: per softlayer, this root behavior is there regardless of the current SRU. I'll spend some time trying to debug and file a separate bug if needed [18:23] it's happening on 19.1.1 on softlayer before the actual upgrade to 19.2.24 [18:23] blackboxsw: hrm I forget if softlayer has multiple datasources ... [18:23] that really smells wrong to me [18:23] I thought they were configdrive, not a nocloud [18:23] rharper: agreed. I'll try the other 'flavors' [18:24] that seems like a metadata bug [18:24] rharper: it does smell like metadata bug (or cloud-init not cleaning up earlier boot configs) [18:24] since SL has multiple boot 'configurations [18:25] blackboxsw: right, there are like two stages [18:25] blackboxsw: but the seed file seems wrong, Ds is IBMCloud right ? [18:25] for softlayer ? [18:26] https://bugs.launchpad.net/cloud-init/+bug/1673637 [18:26] Launchpad bug 1673637 in cloud-init "cloud-init - Hosts in softlayer receiving warning" [High,Fix released] [18:26] blackboxsw: looks like it has some backgound [18:28] blackboxsw, Thanks for the link. I figured there would be need to either encode or escape them [18:32] https://paste.ubuntu.com/p/TK2ht5Vd8T/ [18:32] Petaris: you don't need to encode for passing along # if you use the | marker in yaml , it will preserve the it as a string [18:38] thx for the extra option rharper [18:38] rharper, ok, cool [18:38] Thanks [18:38] sure [18:39] rharper: per SL, yes datasource is DataSourceIBMCloud [OS-Code/Live /dev/xvdh] [18:40] will RTD on softlayer behavior again [18:40] * blackboxsw needs to swap that back into memory [18:58] hello i'm FreeBSD tester and simple patch creator ;) [18:59] hiya [19:00] blackboxsw: what did you think was wrong ? [19:00] with ibmcloud [19:01] smoser: the instances are coming up with root as the default user, instead of ubuntu (which is still configured default in /etc/cloud/cloud.cfg). [19:01] smoser: and there is a v.l.c/seed/nocloud-net left around on the booted instance that is overriding user: root etc which is why my default ubuntu user isn't being set *I think* [19:03] blackboxsw: xenial or bionic [19:03] ? [19:03] xenial: my expectation was that I'd have an ubuntu user set up. [19:03] but no ubuntu user, it's only trying to create root. root@SRU-worked-ibmcloud:~# grep User /var/log/cloud-init.log [19:03] 2019-09-03 16:57:52,405 - __init__.py[INFO]: User root already exists, skipping. [19:03] 2019-09-03 18:56:11,903 - __init__.py[INFO]: User root already exists, skipping [19:04] * blackboxsw tries bionic too [19:04] unless something has changed, the cloud team builds those images in some odd way. [19:04] smoser: yeah I was double checking cpc [19:04] ibmcloud datasource ends up never being used. [19:07] i think if you poke around in /etc/cloud/ you'll see. [19:07] smoser: looks like bionic works [19:08] and /etc/fstab and such [19:08] it is really bad [19:08] just xenial falls over [19:08] and we decided not to fix it there. [19:08] or at least that it could happen later. [19:10] so bionic is good (and has no /v/l/c/seed/nocloud-net config) root@SRU-worked-ibmcloud:~# egrep -r 'Adding|User' /var/log/cloud-init.log [19:10] 2019-09-03 19:06:53,113 - __init__.py[DEBUG]: Adding user ubuntu [19:10] blackboxsw: iirc there is some udev hook that mounts the config disk to /var/lib/cloud/seed/nocloud [19:10] ahh checking hooks [19:10] thanks [19:10] i wrote all this down somewhere i thikn [19:10] probalb y a google doc [19:10] yeah /me was looking through transition google docs atm [19:11] thank you scott [19:11] and hackmd to cover our bases :) [19:11] https://hackmd.io/HQ4Brp_RTNyFPfopxtDFFw [19:11] is all i could find on hackmd [19:11] which mostly just points to / mirrors the datasource doc [19:11] (in DataSourceIBMCloud) [19:12] https://hackmd.io/HQ4Brp_RTNyFPfopxtDFFw :) [19:12] yep just found it [19:12] whcih only discusses how cloud-init works, not how you could modify an image with a bunch of changes so that cloud-init would be tricked [19:13] https://bugs.launchpad.net/cloud-init/+bug/1766401 [19:13] Launchpad bug 1766401 in cloud-images "full config file wiped after apt-upgrade issued" [Undecided,New] [19:13] https://git.launchpad.net/cloud-init/commit/cloud-init/sources?id=11172924a48a47 [19:15] ahh smoser ok, yet it seems on xenial we don't have all the magic we need to continue using the old datasource, as IBMCloud is being detected still [19:15] ok I can file a bug on that and get it resolved [19:16] * blackboxsw tries to remember if we had a followup to try to activate IBMCloud ds on xenial. [19:16] hm.. [19:16] https://git.launchpad.net/cloud-init/commit/cloud-init/sources?id=11172924a48a47a7231d19d9cefe628dfddda8bf [19:16] looks broken [19:16] Diffstat shows 0 files changed [19:16] :-( [19:17] ok xenial I do see the following though datasource_list: [ IBMCloud ] in etc/cloud/cloud.cfg.d/99_networklayer_common.cfg [19:17] well, that'd be a change [19:18] yeah because I don't think that was set originally [19:18] it may have been migrated back from bionic [19:18] yeah. [19:32] blackboxsw: so do you see two sets of logs ? [19:33] cloud-init logs / runs [19:33] can i ssh in? [19:34] smoser: ok, there was a separate salesforce case requesting this change on xenial it seems. the shift from configdrive -> IBMCloud was requested by IBM on xenial [19:34] and they desire root user there it seems. [19:34] because inconsistency makes sure people stay employed [19:35] so it is working as desired? [19:35] s/desired/"desired"/ [19:35] smoser: you can login if you like root@75.126.158.109 [19:35] *sigh* [19:35] "desired" for the moment, but I think we need to circle back w/ IBM/CPC on this [19:36] My understanding is that root should be the default for all instances, so the ubuntu user may be an accident. [19:44] Odd_Bloke: though I do wonder what the compelling need is to drop the ubuntu user altogether. I'm trying to get the background there on why that's a good thing to have the special case. [19:45] Well, if the ubuntu user isn't cloud-init's default then it's going to have surprising behaviour compared to other places. [19:45] though even in the Datasource itself IBM bakes in {"cloud-init":"#!/bin/bash\necho 'root:$6$' | chpasswd -e"} [19:45] Whereas if it doesn't exist at all, then people won't expect it to behave in any particular way. [19:45] so we were already doing some funkiness for root user in IBM [19:47] wlel, generally speaking this looks like it is functioning as designed. [19:48] cpc could drop the hard coded datasource_list there. and i'd guess also the 'resize_rootfs' business [19:49] smoser: yeah, I'm trying to shift back to the original datasource_list: [ConfigDrive, NoCloud] to check on Xenial creating ubuntu user. [19:49] * blackboxsw needs to rebooit [19:49] thx [19:51] so we would have to backport the ibmcloud packaging bits. [19:53] need abe6bcdfacdf2f6745e3acf5c76b332380b9c493 and 4f7bdee8cc37e0f36df888fbd79f0e68d6429431 back to xenial [20:07] smoser: right, agreed, to enable IBM appropriately on new instances [20:08] yeah, so looking like this particular case is IBM-desired root-only login on xenial. bionic++ looks like it will have ubuntu & root useres. [20:08] users* even [20:09] i think that probably that is the way it has always been on xenial [20:09] and bionic changed to be more like ubuntu "standard" [20:10] so from that perspectrive... yeah, you want to keep the old behavior [20:12] blackboxsw: yay for no regression, and let's document that expection in the sru manual template , ? [20:18] oh. i forgot. xenial is reporting only still for ds-identify [20:18] ok added https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1842510 [20:18] Launchpad bug 1842510 in cloud-init (Ubuntu) "Xenial: Enable IBMCloud in default datasource_list template" [Undecided,Triaged] [20:19] blackboxsw: i'd say that is 'fix released' in cloud-init (Ubuntu) [20:19] not triaged [20:19] still probably need to add IBMCloud to datasource_list config template even though image magic specifies an overried [20:19] smoser: +1 [20:19] added confirmed to the xenial task [20:21] ok /me gets back to softlayer SRU verification now :) [22:06] hrm, probably about time for us to fix our rtd content generation. [22:06] :/ [22:06] it seems we are still out of date. [22:13] hrm github integrations seem successful [22:13] last success 5 days ago, during our last commit [22:13] yet cc_set_passwords documentation still seems out of date w/ tip [22:38] there, fixed RTD for the moment. now why did I have to manually loging to readthedocs.io and click the build button. that should have been triggered automatically on doc updates [22:43] blackboxsw: yeah, it's not quite working, I have a script that we could cron [22:44] I think the challenge is the github mirror, I'm not sure it picks up new branches/commits; I know when we create new tags, those are missing as well