| smoser | tribaal: on upgrade, that config file wont get re-written. on fresh install it will. | 13:37 |
|---|---|---|
| smoser | i'm pretty sure thats thecase at least. | 13:37 |
| smoser | so if you purged the package and then installed it, you should get the "fresh install" feel. | 13:37 |
| smoser | which is what will happen for new image builds | 13:37 |
| smoser | well... not a purge, but they actually *are* fresh installs | 13:38 |
| rharper | smoser: tribaal: thanks for the info; I was thinking that; generally we wouldn't want an upgrade to migrate existing users to a new datasource; but users could opt-in via the dpkg reconfigure; | 14:05 |
| smoser | i think what we'd really want to get to is no datasource_list int he packaging by default | 14:07 |
| smoser | and cloud-init just do the right thing | 14:07 |
| tribaal | yes, I guess trying / discovering datasources would make sense | 14:07 |
| rharper | smoser: right; can you think of a blocker here ? | 14:07 |
| smoser | and then if the user *did* configure the list, then the would have effectively opted out of new datasources in ugprades which makes sense. | 14:08 |
| rharper | smoser: ideally it's empty; unless a user (or program) wrote a "local" override | 14:08 |
| smoser | rharper: i can't think of a blocker. | 14:09 |
| tribaal | one think that could be an isuue is for cases were ordering matters - for example if we maintain cloudstack compat and the cloudstack datasource is first to report "found". but that's not a blocker. | 14:09 |
| smoser | you just have to consider how upgrades work and such. and have the packaging do the right thing there. | 14:09 |
| tribaal | or say clouds migrating away from EC2 compatibility (which I assume exists) | 14:09 |
| smoser | tribaal: those points aren't reasons to drop datasource_list from the package i dont think. | 14:10 |
| smoser | they're valid things that we have to consider (and we have had bugs where the same instance swithced datasources on reboot) | 14:11 |
| tribaal | true | 14:11 |
| rharper | mmm, ibmcloud | 14:11 |
| smoser | but having "one more thing" (the packaging) holding that list is just a pita | 14:11 |
| smoser | dry - don't repeat yourself | 14:12 |
| smoser | cloud-init (thanks smoser) likes to have you repeat yourself ;) | 14:12 |
| rharper | smoser: I think we could drop it; ds-identify already provides the "default" list if nothing is present | 14:12 |
| rharper | would be worth experimenting with | 14:12 |
| smoser | yeah. i agree. | 14:13 |
| smoser | and event hen... i'd like to get rid of *that* list somehow. | 14:13 |
| smoser | but at least that isnt end-user configurable | 14:13 |
| rharper | hrm, yes; and we still find them anyhow rather than just accepting the names | 14:18 |
| rharper | blackboxsw: did we have general sru testing steps/language somewhere? or did we say we wanted to write that down to point folks at it ? | 16:43 |
| blackboxsw | rharper: we don't but we said we wanted to create those docs. I thought I added a trello card to todo soon, but I don't see it | 16:45 |
| blackboxsw | adding one now | 16:45 |
| rharper | ok | 16:45 |
| blackboxsw | https://trello.com/c/96vCbWgJ/1129-document-sru-manual-upgrade-testing-steps-for-cloud-init-for-the-community | 16:46 |
| rharper | thx | 16:54 |
| rharper | blackboxsw: was the bug verification script card updated with the 8 bugs for this SRU ? | 16:54 |
| blackboxsw | rharper: sorry, done https://trello.com/c/WzfgsM2l/9-write-verfication-scripts-in-ubuntu-sru-repository | 17:06 |
| blackboxsw | might still need pruning | 17:06 |
| blackboxsw | but just grabbed via git diff ubuntu/xenial..HEAD | log2dch --trello | grep pad.lv (for bug-related lines) | 17:06 |
| * blackboxsw has an errand. back in ~20 | 17:07 | |
| rharper | ok | 17:07 |
| blackboxsw | back | 17:59 |
| chillysurfer | i'm adding some code for a merge request to cloud-init, and i'm making the (abbreviated) call to cloudinit.log.getLogger().debug('my message here'), but in my testing that doesn't actually log to /var/log/cloud-init.log. am i doing something wrong? i see debug log messages in cloud-init.log so i think the level is fine | 18:08 |
| blackboxsw | hrm rharper I'm seeing strange behavior trying to launch softlayer vms. The default user in cloud.cfg is ubuntu, yet that user is not being created. no tracebacks, and I see that the platform is also providing vendordata to reset root's password via a #!/bin/bash\necho 'root:$6$rounds=5000$5d6<redacted>' | chpasswd -e" | 18:10 |
| rharper | blackboxsw: eww | 18:10 |
| rharper | blackboxsw: let's collect-logs and look there | 18:10 |
| blackboxsw | this is OS-Code/Live | 18:10 |
| rharper | also check previous softlayer sru results ? | 18:10 |
| blackboxsw | rharper: yeah, will do on that | 18:11 |
| rharper | chillysurfer: deploying a newly installed patch, unittesting ? can you provide more details on how you're testing ? | 18:11 |
| blackboxsw | chillysurfer: if you are actually talking about grabbing logs from unit tests, you'll need to set with_logs = True in the unit test | 18:11 |
| blackboxsw | then you can check self.logs.getvalue() in the unit tests | 18:11 |
| chillysurfer | rharper blackboxsw sorry i should've clarified. the code is written and i've generated a deb and testing it on a vm right now. not unit testing, actually running cloud-init through systemd | 18:12 |
| chillysurfer | i don't even really see where we wire up logging to /var/log/cloud-init.log. i see the def_log_file but i don't see us addHandler on it | 18:13 |
| rharper | chillysurfer: typically the module that logs will have something like: from cloudinit import log as logging; LOG = logging.getLogger(__name__) ; | 18:13 |
| rharper | chillysurfer: the connection happens via /etc/cloud/cloud.cfg.d/05_logging.cfg | 18:14 |
| blackboxsw | chillysurfer: wiring logging comes config from /etc/cloud/cloud.cfg.d/05_logging.cfg | 18:14 |
| chillysurfer | rharper: yep that's exactly what i did (except i didn't pass in __name__) | 18:14 |
| blackboxsw | I lose :) | 18:14 |
| chillysurfer | rharper: ah nice :) | 18:14 |
| chillysurfer | in that case i should be logging | 18:14 |
| chillysurfer | really strange i'm not | 18:14 |
| chillysurfer | no need to flush or anything right? | 18:14 |
| rharper | no | 18:14 |
| rharper | so either the path isn't being taken | 18:14 |
| rharper | or something else isn't quite right; do you have a diff of your changes you can share ? | 18:15 |
| chillysurfer | rharper: i don't have a diff at the moment. here's all i'm doing: | 18:16 |
| chillysurfer | from cloudinit import log as logging; _LOG = logging.getLogger(); _LOG.debug('hello world') | 18:16 |
| rharper | how is this module called ? | 18:16 |
| chillysurfer | rharper: i wired it up from cloudinit.cmd.main | 18:17 |
| rharper | cloud-init runs through cloudinit/cmd/main.py which will run through cloudinit/stages.py:Init() object which runs a sequence of "stages"; | 18:17 |
| blackboxsw | rharper: looks like softlayer leaves around the /var/lib/cloud/seed/nocloud-net/user-data which contains user: root definition, which overrides typical cloud-init behavior . That seed file is left over from earlier networklayer boot stages I think https://pastebin.ubuntu.com/p/SMWQvcZMhY/ | 18:17 |
| chillysurfer | rharper: yep that's what i did. i added an arg, and then imported and run the action | 18:18 |
| rharper | chillysurfer: so depending on how early you are, logging may not be initialized; | 18:18 |
| rharper | if you're calling LOG. before setUpBasicLogging() in main is called | 18:18 |
| rharper | then you won't see that; early main.py has it's own logging | 18:18 |
| rharper | which then gets replayed after setUpBasicLogging is called; but previous calls to LOG before setup won't show up | 18:19 |
| chillysurfer | ahhhh wait yeah i see that we have `if args.debug: logging.setupBasiclLogging()` | 18:19 |
| rharper | yep | 18:19 |
| rharper | which will write to stderr | 18:19 |
| rharper | without config | 18:19 |
| rharper | but with the 05... it should show up in /var/log/cloud-init.log (but not in journalctl ) | 18:19 |
| chillysurfer | rharper: oh so you're saying it should show up in cloud-init.log? | 18:20 |
| chillysurfer | even before setupBasicLogging? | 18:20 |
| rharper | no | 18:20 |
| chillysurfer | oh sorry got confused | 18:20 |
| chillysurfer | so you say this is expected that i'm seeing now logged messages | 18:20 |
| chillysurfer | *no | 18:20 |
| chillysurfer | *no logged messages | 18:20 |
| rharper | depending on where you are inserting your LOGs, it's possible. | 18:20 |
| Petaris | Is there anything special I need to do when using write_files with content that includes "#" in it? | 18:21 |
| chillysurfer | ah ok makes sense. i'll play around with this. thanks, rharper! | 18:21 |
| rharper | main.py has it's own logging functions before it gets to parsing /etc/cloud/* for log setup; which eventually get's configured to write to /var/log/cloud-init.log; | 18:21 |
| blackboxsw | Petaris: you might want to base64 encode the content to avoid getting interpreted as yaml comments | 18:21 |
| blackboxsw | Petaris: https://cloudinit.readthedocs.io/en/latest/topics/examples.html#writing-out-arbitrary-files | 18:22 |
| blackboxsw | rharper: per softlayer, this root behavior is there regardless of the current SRU. I'll spend some time trying to debug and file a separate bug if needed | 18:23 |
| blackboxsw | it's happening on 19.1.1 on softlayer before the actual upgrade to 19.2.24 | 18:23 |
| rharper | blackboxsw: hrm I forget if softlayer has multiple datasources ... | 18:23 |
| rharper | that really smells wrong to me | 18:23 |
| rharper | I thought they were configdrive, not a nocloud | 18:23 |
| blackboxsw | rharper: agreed. I'll try the other 'flavors' | 18:23 |
| rharper | that seems like a metadata bug | 18:24 |
| blackboxsw | rharper: it does smell like metadata bug (or cloud-init not cleaning up earlier boot configs) | 18:24 |
| blackboxsw | since SL has multiple boot 'configurations | 18:24 |
| rharper | blackboxsw: right, there are like two stages | 18:25 |
| rharper | blackboxsw: but the seed file seems wrong, Ds is IBMCloud right ? | 18:25 |
| rharper | for softlayer ? | 18:25 |
| rharper | https://bugs.launchpad.net/cloud-init/+bug/1673637 | 18:26 |
| ubot5 | Launchpad bug 1673637 in cloud-init "cloud-init - Hosts in softlayer receiving warning" [High,Fix released] | 18:26 |
| rharper | blackboxsw: looks like it has some backgound | 18:26 |
| Petaris | blackboxsw, Thanks for the link. I figured there would be need to either encode or escape them | 18:28 |
| rharper | https://paste.ubuntu.com/p/TK2ht5Vd8T/ | 18:32 |
| rharper | Petaris: you don't need to encode for passing along # if you use the | marker in yaml , it will preserve the it as a string | 18:32 |
| blackboxsw | thx for the extra option rharper | 18:38 |
| Petaris | rharper, ok, cool | 18:38 |
| Petaris | Thanks | 18:38 |
| rharper | sure | 18:38 |
| blackboxsw | rharper: per SL, yes datasource is DataSourceIBMCloud [OS-Code/Live /dev/xvdh] | 18:39 |
| blackboxsw | will RTD on softlayer behavior again | 18:40 |
| * blackboxsw needs to swap that back into memory | 18:40 | |
| faa | hello i'm FreeBSD tester and simple patch creator ;) | 18:58 |
| rharper | hiya | 18:59 |
| smoser | blackboxsw: what did you think was wrong ? | 19:00 |
| smoser | with ibmcloud | 19:00 |
| blackboxsw | smoser: the instances are coming up with root as the default user, instead of ubuntu (which is still configured default in /etc/cloud/cloud.cfg). | 19:01 |
| blackboxsw | smoser: and there is a v.l.c/seed/nocloud-net left around on the booted instance that is overriding user: root etc which is why my default ubuntu user isn't being set *I think* | 19:01 |
| smoser | blackboxsw: xenial or bionic | 19:03 |
| smoser | ? | 19:03 |
| blackboxsw | xenial: my expectation was that I'd have an ubuntu user set up. | 19:03 |
| blackboxsw | but no ubuntu user, it's only trying to create root. root@SRU-worked-ibmcloud:~# grep User /var/log/cloud-init.log | 19:03 |
| blackboxsw | 2019-09-03 16:57:52,405 - __init__.py[INFO]: User root already exists, skipping. | 19:03 |
| blackboxsw | 2019-09-03 18:56:11,903 - __init__.py[INFO]: User root already exists, skipping | 19:03 |
| * blackboxsw tries bionic too | 19:04 | |
| smoser | unless something has changed, the cloud team builds those images in some odd way. | 19:04 |
| blackboxsw | smoser: yeah I was double checking cpc | 19:04 |
| smoser | ibmcloud datasource ends up never being used. | 19:04 |
| smoser | i think if you poke around in /etc/cloud/ you'll see. | 19:07 |
| blackboxsw | smoser: looks like bionic works | 19:07 |
| smoser | and /etc/fstab and such | 19:08 |
| smoser | it is really bad | 19:08 |
| blackboxsw | just xenial falls over | 19:08 |
| smoser | and we decided not to fix it there. | 19:08 |
| smoser | or at least that it could happen later. | 19:08 |
| blackboxsw | so bionic is good (and has no /v/l/c/seed/nocloud-net config) root@SRU-worked-ibmcloud:~# egrep -r 'Adding|User' /var/log/cloud-init.log | 19:10 |
| blackboxsw | 2019-09-03 19:06:53,113 - __init__.py[DEBUG]: Adding user ubuntu | 19:10 |
| smoser | blackboxsw: iirc there is some udev hook that mounts the config disk to /var/lib/cloud/seed/nocloud | 19:10 |
| blackboxsw | ahh checking hooks | 19:10 |
| blackboxsw | thanks | 19:10 |
| smoser | i wrote all this down somewhere i thikn | 19:10 |
| smoser | probalb y a google doc | 19:10 |
| blackboxsw | yeah /me was looking through transition google docs atm | 19:10 |
| blackboxsw | thank you scott | 19:11 |
| blackboxsw | and hackmd to cover our bases :) | 19:11 |
| smoser | https://hackmd.io/HQ4Brp_RTNyFPfopxtDFFw | 19:11 |
| smoser | is all i could find on hackmd | 19:11 |
| smoser | which mostly just points to / mirrors the datasource doc | 19:11 |
| smoser | (in DataSourceIBMCloud) | 19:11 |
| blackboxsw | https://hackmd.io/HQ4Brp_RTNyFPfopxtDFFw :) | 19:12 |
| blackboxsw | yep just found it | 19:12 |
| smoser | whcih only discusses how cloud-init works, not how you could modify an image with a bunch of changes so that cloud-init would be tricked | 19:12 |
| smoser | https://bugs.launchpad.net/cloud-init/+bug/1766401 | 19:13 |
| ubot5 | Launchpad bug 1766401 in cloud-images "full config file wiped after apt-upgrade issued" [Undecided,New] | 19:13 |
| smoser | https://git.launchpad.net/cloud-init/commit/cloud-init/sources?id=11172924a48a47 | 19:13 |
| blackboxsw | ahh smoser ok, yet it seems on xenial we don't have all the magic we need to continue using the old datasource, as IBMCloud is being detected still | 19:15 |
| blackboxsw | ok I can file a bug on that and get it resolved | 19:15 |
| * blackboxsw tries to remember if we had a followup to try to activate IBMCloud ds on xenial. | 19:16 | |
| smoser | hm.. | 19:16 |
| smoser | https://git.launchpad.net/cloud-init/commit/cloud-init/sources?id=11172924a48a47a7231d19d9cefe628dfddda8bf | 19:16 |
| smoser | looks broken | 19:16 |
| smoser | Diffstat shows 0 files changed | 19:16 |
| smoser | :-( | 19:16 |
| blackboxsw | ok xenial I do see the following though datasource_list: [ IBMCloud ] in etc/cloud/cloud.cfg.d/99_networklayer_common.cfg | 19:17 |
| smoser | well, that'd be a change | 19:17 |
| blackboxsw | yeah because I don't think that was set originally | 19:18 |
| blackboxsw | it may have been migrated back from bionic | 19:18 |
| smoser | yeah. | 19:18 |
| smoser | blackboxsw: so do you see two sets of logs ? | 19:32 |
| smoser | cloud-init logs / runs | 19:33 |
| smoser | can i ssh in? | 19:33 |
| blackboxsw | smoser: ok, there was a separate salesforce case requesting this change on xenial it seems. the shift from configdrive -> IBMCloud was requested by IBM on xenial | 19:34 |
| blackboxsw | and they desire root user there it seems. | 19:34 |
| smoser | because inconsistency makes sure people stay employed | 19:34 |
| smoser | so it is working as desired? | 19:35 |
| smoser | s/desired/"desired"/ | 19:35 |
| blackboxsw | smoser: you can login if you like root@75.126.158.109 | 19:35 |
| rharper | *sigh* | 19:35 |
| blackboxsw | "desired" for the moment, but I think we need to circle back w/ IBM/CPC on this | 19:35 |
| Odd_Bloke | My understanding is that root should be the default for all instances, so the ubuntu user may be an accident. | 19:36 |
| blackboxsw | Odd_Bloke: though I do wonder what the compelling need is to drop the ubuntu user altogether. I'm trying to get the background there on why that's a good thing to have the special case. | 19:44 |
| Odd_Bloke | Well, if the ubuntu user isn't cloud-init's default then it's going to have surprising behaviour compared to other places. | 19:45 |
| blackboxsw | though even in the Datasource itself IBM bakes in {"cloud-init":"#!/bin/bash\necho 'root:$6$<snip>' | chpasswd -e"} | 19:45 |
| Odd_Bloke | Whereas if it doesn't exist at all, then people won't expect it to behave in any particular way. | 19:45 |
| blackboxsw | so we were already doing some funkiness for root user in IBM | 19:45 |
| smoser | wlel, generally speaking this looks like it is functioning as designed. | 19:47 |
| smoser | cpc could drop the hard coded datasource_list there. and i'd guess also the 'resize_rootfs' business | 19:48 |
| blackboxsw | smoser: yeah, I'm trying to shift back to the original datasource_list: [ConfigDrive, NoCloud] to check on Xenial creating ubuntu user. | 19:49 |
| * blackboxsw needs to rebooit | 19:49 | |
| blackboxsw | thx | 19:49 |
| smoser | so we would have to backport the ibmcloud packaging bits. | 19:51 |
| smoser | need abe6bcdfacdf2f6745e3acf5c76b332380b9c493 and 4f7bdee8cc37e0f36df888fbd79f0e68d6429431 back to xenial | 19:53 |
| blackboxsw | smoser: right, agreed, to enable IBM appropriately on new instances | 20:07 |
| blackboxsw | yeah, so looking like this particular case is IBM-desired root-only login on xenial. bionic++ looks like it will have ubuntu & root useres. | 20:08 |
| blackboxsw | users* even | 20:08 |
| smoser | i think that probably that is the way it has always been on xenial | 20:09 |
| smoser | and bionic changed to be more like ubuntu "standard" | 20:09 |
| smoser | so from that perspectrive... yeah, you want to keep the old behavior | 20:10 |
| rharper | blackboxsw: yay for no regression, and let's document that expection in the sru manual template , ? | 20:12 |
| smoser | oh. i forgot. xenial is reporting only still for ds-identify | 20:18 |
| blackboxsw | ok added https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1842510 | 20:18 |
| ubot5 | Launchpad bug 1842510 in cloud-init (Ubuntu) "Xenial: Enable IBMCloud in default datasource_list template" [Undecided,Triaged] | 20:18 |
| smoser | blackboxsw: i'd say that is 'fix released' in cloud-init (Ubuntu) | 20:19 |
| smoser | not triaged | 20:19 |
| blackboxsw | still probably need to add IBMCloud to datasource_list config template even though image magic specifies an overried | 20:19 |
| blackboxsw | smoser: +1 | 20:19 |
| blackboxsw | added confirmed to the xenial task | 20:19 |
| blackboxsw | ok /me gets back to softlayer SRU verification now :) | 20:21 |
| blackboxsw | hrm, probably about time for us to fix our rtd content generation. | 22:06 |
| blackboxsw | :/ | 22:06 |
| blackboxsw | it seems we are still out of date. | 22:06 |
| blackboxsw | hrm github integrations seem successful | 22:13 |
| blackboxsw | last success 5 days ago, during our last commit | 22:13 |
| blackboxsw | yet cc_set_passwords documentation still seems out of date w/ tip | 22:13 |
| blackboxsw | there, fixed RTD for the moment. now why did I have to manually loging to readthedocs.io and click the build button. that should have been triggered automatically on doc updates | 22:38 |
| rharper | blackboxsw: yeah, it's not quite working, I have a script that we could cron | 22:43 |
| rharper | I think the challenge is the github mirror, I'm not sure it picks up new branches/commits; I know when we create new tags, those are missing as well | 22:44 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!