| _NiC | If I have an existing user 'debian' in my image, and in /etc/cloud/cloud.cfg I have system_info: -> default_user: -> name: debian, will the settings from cloud.cfg not be applied because the user already exists? Because the settings there are not applied. :-\ (debian 9, cloud-init 0.7.9-2) | 10:08 |
|---|---|---|
| _NiC | oh, perhaps it's skipped because I'm specifying another user in my user-data, and not also explicitly telling it to create the default user as well? | 10:23 |
| meena | _NiC: perhaps! is the user from user-data created? | 12:40 |
| _NiC | meena, yes. and I didn't have "- default" listed under users: | 12:40 |
| _NiC | meena, so from what I could gather, that meant to *not* care about the default_user as defined in cloud.cfg. So when I added "- default" it worked as expected again. :) | 12:41 |
| meena | 👍 | 12:41 |
| _NiC | When running the openstack create command, you get sent back an 'adminPass' though, and I tried to figure out if that could be injected somehow, but not easily it seems, so I ignored that as well. I have my custom user, so it's all good. :) | 12:42 |
| brandor5 | hello everyone: we've got a pipeline that builds custom cloud images for openstack... I've added centos8 support and I'm having trouble with adduser for our custom user... on all the other builds stamping in an updated cloud.cfg that replaces the centos user with our user works ok, but with centos8 it doesnt... that user never gets added and our pipeline fails... any ideas on whats going on? | 14:07 |
| hetii | Hello | 14:11 |
| hetii | I have a vmware instance where I spawn by terraform rancheros. The issue is that in this environment DHCP is disable so I need to provide network configuration. The question is should I do this by metadata or by cloud-config? What happens if I have defined in both? with one is taken? | 14:13 |
| onklmaps | Hi :) I want to deploy some cloud servers at Hetzner. I plan to run 3-4 PrestaShops on them. I think nginx+php-fpm is the best and then a separate mysql server. So my question: does anyone have a good cloud-init-file for nginx + php-fpm that would be suitable for running prestashop? PHP 7.2 is needed | 14:26 |
| _NiC | onklmaps, my knowledge of cloud-init is pretty limited, but I suspect you don't want to try to do *all* of your config using cloud-init | 14:41 |
| _NiC | onklmaps, rather use cloud-init to get a good base, and use something else to configure the rest. (such as ansible, puppet, chef, or similar) | 14:41 |
| onklmaps | Ahh. That makes sense.. You know any good base for nginx + php-fpm? | 14:42 |
| _NiC | I'm not familiar with prestashop either, but unless its requirements are very specific, any standard setup should work | 14:44 |
| _NiC | if you need high performance etc etc you should probably look into various forms of caching, but that's a topic for another channel I think.. :-) | 14:44 |
| onklmaps | It's pretty standard, yes..I have some custom things that needs to go into the server block, but other than that its pretty standard | 14:44 |
| onklmaps | Yeah - caching... I know.. It's a big subject | 14:45 |
| onklmaps | I just installed nginx as reverse proxy in front however. I might do static cache there? | 14:45 |
| _NiC | that's possible | 14:46 |
| onklmaps | PrestaShop is much easier to set up with apache.. I might just serve the prestashops using apache + php-fpm, then use nginx reverse proxy with static cache in front | 14:47 |
| onklmaps | What you think? | 14:47 |
| _NiC | Not sure I have an opinion about that :-) | 14:52 |
| wyoung | onklmaps: I think php is a bad idea. | 15:19 |
| blackboxsw | hey folks. let's do this.... | 17:23 |
| blackboxsw | #startmeeting Cloud-init bi-weekly status | 17:23 |
| meetingology | Meeting started Tue Feb 4 17:23:28 2020 UTC. The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology. | 17:23 |
| meetingology | Available commands: action commands idea info link nick | 17:23 |
| blackboxsw | morning, afternoon and evening folks. Time for another cloud-init community status meeting | 17:23 |
| blackboxsw | #chair rharper | 17:24 |
| meetingology | Current chairs: blackboxsw rharper | 17:24 |
| blackboxsw | #chair Odd_Bloke | 17:24 |
| meetingology | Current chairs: Odd_Bloke blackboxsw rharper | 17:24 |
| blackboxsw | Coud-init upstream uses this meeting as a platform for community updates, feature/bug discussions, and an opportunity to get some extra input on current development. | 17:24 |
| blackboxsw | The next scheduled status meeting is always listed in the topic of this channel, so feel free to drop in on next session if you miss this one | 17:24 |
| blackboxsw | while we're at it I'll update for next status meeting. | 17:25 |
| === blackboxsw changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting February 16 17:15 UTC | 19.4 (Dec 17) drops Py2.7 : origin/stable-19.4 | 20.1 (Feb 18) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw | 2 weeks from today, same bat time, same bat channel | 17:26 |
| blackboxsw | Our previous meeting minutes line here: | 17:26 |
| blackboxsw | #link https://cloud-init.github.io/ | 17:26 |
| blackboxsw | *live here* rather | 17:26 |
| blackboxsw | the topics we cover in this meeting are the following: Previous Actions, Recent Changes, In-progress Development, Community Charter, Upcoming Meetings, Office Hours (~30 mins). | 17:27 |
| blackboxsw | new topics or intejections are always welcome | 17:27 |
| blackboxsw | #topic Previous Actions | 17:27 |
| blackboxsw | From last meeting we had no unresolved actions so we can jump to the next section | 17:27 |
| blackboxsw | #topic Recent Changes | 17:27 |
| blackboxsw | found from tip of master with `git log --since 01/21/2020` | 17:29 |
| blackboxsw | - sysconfig: distro-specific config rendering for BOOTPROTO option (#162) | 17:29 |
| blackboxsw | [Robert Schweikert] (LP: #1800854) | 17:29 |
| blackboxsw | - cloudinit: replace "from six import X" imports (except in util.py) (#183) | 17:29 |
| blackboxsw | - run-container: use 'test -n' instead of 'test ! -z' (#202) | 17:29 |
| blackboxsw | [Paride Legovini] | 17:29 |
| blackboxsw | - net/cmdline: correctly handle static ip= config (#201) | 17:29 |
| blackboxsw | [Dimitri John Ledkov] (LP: #1861412) | 17:29 |
| ubot5 | Launchpad bug 1800854 in cloud-init "BOTOPROTO handling between RHEL/Centos/Fedora and SUSE distros is different" [Medium,Triaged] https://launchpad.net/bugs/1800854 | 17:29 |
| blackboxsw | - Replace mock library with unittest.mock (#186) | 17:29 |
| blackboxsw | - HACKING.rst: update CLA link (#199) | 17:29 |
| ubot5 | Launchpad bug 1861412 in cloud-init (Ubuntu) "cloud-init crashes with static network configuration" [Undecided,Fix committed] https://launchpad.net/bugs/1861412 | 17:29 |
| blackboxsw | - Scaleway: Fix DatasourceScaleway to avoid backtrace (#128) | 17:29 |
| blackboxsw | [Louis Bouchard] | 17:29 |
| blackboxsw | - cloudinit/cmd/devel/net_convert.py: add missing space (#191) | 17:29 |
| blackboxsw | - tools/run-container: drop support for python2 (#192) [Paride Legovini] | 17:29 |
| blackboxsw | - Print ssh key fingerprints using sha256 hash (#188) (LP: #1860789) | 17:29 |
| blackboxsw | - Make the RPM build use Python 3 (#190) [Paride Legovini] | 17:29 |
| ubot5 | Launchpad bug 1860789 in cloud-init (Ubuntu) "ssh_authkey_fingerprints must use sha256 not md5" [Undecided,Fix committed] https://launchpad.net/bugs/1860789 | 17:29 |
| powersj | thought we were going to use pastebin :P | 17:29 |
| blackboxsw | heh, that is a good point (I wondered if anyone would call me on that) | 17:29 |
| blackboxsw | #link https://paste.ubuntu.com/p/3jQdKZVPcM/ | 17:30 |
| blackboxsw | generally speaking, dropping use of six since our code based is not python3-only, tooling dropping py2, sysconfig rendering flavors for opensuse, doc fixes and read the docs fixups | 17:31 |
| blackboxsw | thanks all for the contributions over the last couple weeks | 17:32 |
| blackboxsw | #topic In-progress Development, | 17:32 |
| blackboxsw | #topic In-progress Development | 17:32 |
| blackboxsw | Any existing PRs are up for review at the following url: | 17:32 |
| blackboxsw | #link https://github.com/canonical/cloud-init/pulls | 17:33 |
| blackboxsw | generally speaking we are in the 'long tail' part of a couple of feature-sets: | 17:33 |
| blackboxsw | * we are trying to wrap up tooling for our automated CI, publishing processes and documentation for the shift to github from launchpad | 17:34 |
| blackboxsw | * we are in progress on cloud-init handling network hotplug for a couple of datasources | 17:34 |
| blackboxsw | * in progress on boot speed improvements for various platforms | 17:35 |
| blackboxsw | We also recently validated and released cloud-init v 19.4.33 to Xenial, Bionic and Eoan (1/9/2020) | 17:36 |
| ahosmanMSFT | Hi @blackboxsw I'm no longer in the provisioning team, but there's an urgency for the cloud test to be resilient. Have you looked at those issues, I can dedicate as much time as needed to this. If you have time, can we tackle this today? | 17:38 |
| blackboxsw | there are also a number of PRs in flight for FreeBSD,NetBSD, OpenSUSE and CentOS that need attention so we can better enable those distros | 17:38 |
| ahosmanMSFT | azurecloudtest that is | 17:38 |
| blackboxsw | hi ahosmanMSFT I can spend some time on office hours here to peek more at it. my individual runs didn't hit the timeouts again, so we might need a reproducer cmdline from you in a new bug maybe? | 17:39 |
| ahosmanMSFT | So your able to run all tests successfully without timeout and image not building> | 17:39 |
| blackboxsw | ahosmanMSFT: but yes I can spend a little time on this today. and I think ultimately we'll have to find the tox command line that exhibits this error. I'll go checkout my test run again and see. I don't think I saw the failure. but I might be invoking tests differently than you | 17:40 |
| ahosmanMSFT | blackboxsw: hmm that's interesting, thanks let me know | 17:41 |
| blackboxsw | same here ahosmanMSFT, can you file a bug with the traceback you see and the tox cmdline you are running? | 17:41 |
| blackboxsw | then I know exactly what to look for | 17:41 |
| ahosmanMSFT | Sure, will do now | 17:42 |
| blackboxsw | cool. | 17:43 |
| blackboxsw | ok next topic | 17:43 |
| blackboxsw | #topic Community Charter | 17:43 |
| blackboxsw | ok this section is reserved to raise general community work/goals. | 17:43 |
| blackboxsw | At last cloud-init summit we raised a couple of general themes of improvements cloud-init would like to achieve | 17:44 |
| blackboxsw | These themes fell into two categories for this year: datasource documentation updates and cloud-init json schema validation for the 50+ config modules in cloudinit/config/cc_*py so that we can better raise user-config errors and remove some of cloud-init's "sharp edges" | 17:45 |
| blackboxsw | we converted a number of these feature requests in into bugs which can be searched here: | 17:45 |
| blackboxsw | #link https://bugs.launchpad.net/cloud-init/+bugs?field.tag=bitesize | 17:46 |
| blackboxsw | tasks in this list should be fairly easy one-time bugs for folks with a little time available to help improve cloud-init. | 17:46 |
| blackboxsw | we'll revisit this set of bugs/features and the community charter goals near the end of 2020 at the next cloud-init summit | 17:47 |
| blackboxsw | #topic Office Hours (next ~30 mins) | 17:47 |
| blackboxsw | this time is spent with cloud-init upstream dev eyes on this channel for any cloud-init feature, bug or implementation discussions. In the absence of such discussions, we'll review the active PRs to try to tidy up the review queue and unblock developers | 17:48 |
| blackboxsw | for the moment, I'll look over some Azure test timeouts ahosmanMSFT is seeing | 17:49 |
| blackboxsw | any other topics, concerns, bugs, questions are welcome and someone should be around to field them | 17:49 |
| blackboxsw | ahosmanMSFT: so timeouts running integration tests, you said you are getting them about half the time? | 17:50 |
| ahosmanMSFT | blackboxsw: Yes, I tracked it down to platforms/instance._wait_for_system | 18:48 |
| ahosmanMSFT | I invoke it after initializing vm in platform/azurecloudtest/instance.start | 18:49 |
| ahosmanMSFT | when removed, everything works as expected | 18:49 |
| ahosmanMSFT | looks like it's needed for cloud tests so thought I'd leave it to you, since I don't know how ec2/lxd/... rely on | 18:50 |
| powersj | ahosmanMSFT, can you file a bug please with the cli example? | 18:50 |
| powersj | that would help us triage and make a proper decision on what change to make | 18:50 |
| ahosmanMSFT | powersj, yes, was in the middle of that side tracked by meeting. On it now | 18:51 |
| powersj | thanks! | 18:51 |
| blackboxsw | aaaand, I should probably wrap the meeting for the day. | 19:07 |
| blackboxsw | Thanks all for the time and energy you put into improving cloud-init! See you next time, or anytime in between | 19:08 |
| blackboxsw | #endmeeting | 19:08 |
| meetingology | Meeting ended Tue Feb 4 19:08:14 2020 UTC. | 19:08 |
| meetingology | Minutes: http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-02-04-17.23.moin.txt | 19:08 |
| ahosmanMSFT | blackboxsw, powersj: submitted the bug https://bugs.launchpad.net/cloud-init/+bug/1861921 | 19:27 |
| ubot5 | Ubuntu bug 1861921 in cloud-init "cloud tests ssh time out" [Undecided,New] | 19:27 |
| powersj | ahosmanMSFT, can you give us the full CLI you are using to launch? | 19:28 |
| ahosmanMSFT | sure | 19:29 |
| ahosmanMSFT | powersj: added full log to the bug | 19:36 |
| powersj | ahosmanMSFT, perfect thanks | 19:36 |
| ahosmanMSFT | powersj: I found it most reproducible when going Idle | 19:37 |
| ahosmanMSFT | for some reason that repro's it for me, for example launching the tests then checking on something else outside the vm where I am running it | 19:39 |
| powersj | ahosmanMSFT, "reproduce this is by going idle." what do you mean by that? | 19:39 |
| meena | i missed another one! | 20:31 |
| meena | this time my excuse is that I'm visiting family in Ireland | 20:31 |
| rharper | meena: =) | 20:34 |
| ahosmanMSFT | powersj, I run cloud tests on a vm, so when not on the vm it's self , ie use the browser. It typically hit's that time out error. This doesn't occur though when removing that wait function. | 20:35 |
| rharper | hrm, do you think the VM itself is suspending ? | 20:35 |
| rharper | the wait has a timeout which is likely what it's hitting | 20:36 |
| rharper | if you remove the timeout, then when the VM resumes, it just continues executing | 20:36 |
| rharper | when you resume, the VM time will catch up, that will trip timers , ie, if you've suspeneded for longer than the 300 seconds, then when it resumes the timer expires immediately | 20:37 |
| rharper | ahosmanMSFT: can you confirm that if you stay active in the VM that it does not fail ? | 20:37 |
| rharper | alternatively, launch an Ubuntu instance in Azure; and run cloud-tests from within the Azure instances (instead of a local VM) | 20:38 |
| ahosmanMSFT | This was on a ubuntu VM in Azure, rharper | 20:39 |
| ahosmanMSFT | but, yes, it's less likely to happen when staying on VM | 20:39 |
| rharper | quite strange | 20:39 |
| rharper | it still smells like some sort of suspend | 20:39 |
| rharper | you could spawn some other cpu bound task in the background ? | 20:40 |
| blackboxsw | meena: you have to disavow all family and friends like the rest of us... that's the only way this is going to work ;) | 21:13 |
| blackboxsw | yeah ahosmanMSFT my full ci tests against azure are still running and I'm not seeing any of those failures yet. but I'm running from an Ubuntu lxc on Ubuntu host OS. If memory serves you may be running on an Ubuntu lxc running on an OSX host system? How the 'vms' behave on OSX when focus shifts to other services (like browsers etc) could likely be the culprit | 21:17 |
| blackboxsw | and I'd follow rharper's suggestion of "alternatively, launch an Ubuntu instance in Azure; and run cloud-tests from within the Azure instances (instead of a local VM)" | 21:18 |
| rharper | blackboxsw: I think ahosmanMSFT is doing that | 21:19 |
| blackboxsw | oops just read that | 21:19 |
| rharper | blackboxsw: I suppose we could try that | 21:19 |
| rharper | but still not sure what "going idle" means from that perspective | 21:19 |
| blackboxsw | sure I can kick off an instance to do that. I was just thrown by the "browser" comment | 21:20 |
| rharper | running from within a screen/tmux or something else | 21:20 |
| rharper | blackboxsw: as was I | 21:20 |
| ahosmanMSFT | it's more random than anything | 21:20 |
| ahosmanMSFT | ssh timeout | 21:20 |
| rharper | so I suspect there's still something else tooling-wise were missing | 21:20 |
| rharper | ahosmanMSFT: sure, but there are really only two causes here; 1) bad connection between the cloud_test client and target instance 2) the instance is taking longer than the timeout to boot up to ssh | 21:22 |
| rharper | (1) and (2) can be intermittent | 21:22 |
| ahosmanMSFT | I agree, but the second is usually not the case. It usually never goes pass the boot time out of 300s | 21:23 |
| blackboxsw | just looking over those pastes more closely, I *am* running basically the same command as ahosmanMSFT : .tox/citest/bin/python -m tests.cloud_tests run --verbose --os-name bionic --data-dir results --preserve-data --platform azurecloud | 21:25 |
| rharper | ahosmanMSFT: do we not have the console-log configured ? | 21:27 |
| ahosmanMSFT | rharpher: can you elaborate | 21:27 |
| rharper | serial_console_log_blob_uri | 21:28 |
| ahosmanMSFT | that should be created | 21:28 |
| rharper | looks like we enable the serial console, so cloud_tests should capture the serial-console log on failure | 21:28 |
| rharper | I'd like to see the serial console log of the failing instance | 21:28 |
| rharper | and comparing the failing serial-console log to the passing one likely will reveal some differences which may explain what's going on | 21:29 |
| blackboxsw | I'm also a bit surprised to see a 3:45 min gap between the following logs | 21:29 |
| blackboxsw | 2020-02-04 19:11:03,628 - tests.cloud_tests - DEBUG - image not found, launching instance with base image | 21:30 |
| blackboxsw | 2020-02-04 19:14:44,950 - tests.cloud_tests - DEBUG - executing "wait for instance start" | 21:30 |
| ahosmanMSFT | That is stored in a storage account in your resource group when running, but the resource group is deleted after the cloud-tests | 21:30 |
| blackboxsw | in my runs, typically that gap should only be around the time to launch and get a response back from Azure API. which is about 30 seconds on 'az vm create' | 21:30 |
| rharper | ahosmanMSFT: ah you mentioned that bug; we should dump the value before tearing down | 21:31 |
| rharper | blackboxsw: the compute_client.images.get() may be taking a long time | 21:32 |
| ahosmanMSFT | preserve should keep it, but it doesn't working on fixing that too | 21:32 |
| blackboxsw | ahosmanMSFT: I think --preserve-instance and --preserve-data should keep that around post failure | 21:32 |
| rharper | depending on the REST query; I've seen *long* REST calls to list of images | 21:32 |
| blackboxsw | hrm :/ | 21:32 |
| rharper | though 3 minutes seems excessive | 21:32 |
| ahosmanMSFT | Out of three of these bugs, the current biggest flaw isn't even the timeout, it's that images aren't created properly due to azurecloud/image._img_instance staying None despite re-assignment, but this is more on my side, but can use your help. As the ssh timeout isn't 100%, this issue is. | 21:33 |
| ahosmanMSFT | the preserve call should be a simple fix | 21:33 |
| ahosmanMSFT | ssh timeout I think is more machine/system dependent now that I'm seeing your not getting it | 21:34 |
| rharper | ahasenack: the default timeout is 300, it would be a terrible thing to bump that on azure platform to say 600 and see if that runs more reliabling to completion | 21:35 |
| ahasenack | ahosmanMSFT: ^ | 21:36 |
| ahasenack | :) | 21:36 |
| rharper | ahasenack: sorry | 21:36 |
| ahasenack | np :) | 21:36 |
| rharper | lack of extra tab | 21:36 |
| ahosmanMSFT | rharper: I agree, I don't think timeout is the issue | 21:38 |
| ahosmanMSFT | default timeout that is | 21:38 |
| ahosmanMSFT | rharper, blackboxsw: Are you still in today? | 22:07 |
| rharper | yep | 22:07 |
| ahosmanMSFT | Mind taking a look at azurecloud/image._img_instance | 22:08 |
| ahosmanMSFT | it's not re-assigning correctly, which means images don't build correctly | 22:08 |
| ahosmanMSFT | it works on ec2, right? | 22:09 |
| ahosmanMSFT | I placed some print statements and it stays instantiated as None | 22:09 |
| rharper | ahosmanMSFT: sure, do you have more details on "images aren't created properly due to azurecloud/image._img_instance staying None"; | 22:10 |
| ahosmanMSFT | rharper: yes... | 22:12 |
| ahosmanMSFT | so when cloud-tests starts it hit's image.py checks if there is an image and if there isn't it creates one via streams parameters. Then in the next test it checks, is there a clean image, in our case the variable that specifies this never changes and always stays None for some reason | 22:13 |
| ahosmanMSFT | because of that, it always launches a base image, not a clean image | 22:14 |
| ahosmanMSFT | which negates the whole image process | 22:14 |
| ahosmanMSFT | of creating a clean image for all future tests | 22:14 |
| rharper | ahosmanMSFT: are you specifying a cloud-init.deb or is one getting built ? | 22:15 |
| ahosmanMSFT | One get's built, looking at the code, seems to be abstracted | 22:16 |
| ahosmanMSFT | The fault comes when _img_instance is used in snapshot, it never passes the initial conditional | 22:16 |
| ahosmanMSFT | https://paste.ubuntu.com/p/nwgm8KjXJk/ | 22:16 |
| ahosmanMSFT | rharper | 22:17 |
| rharper | ahosmanMSFT: just looking at some differences, in azurecloud/image.py:_instance(), you call platform.create_images(); but you don't start it; | 22:24 |
| ahosmanMSFT | rharper: hmm I changed it and added some logging and still get that it's None | 22:27 |
| ahosmanMSFT | https://paste.ubuntu.com/p/VPgqXZXkkX/ | 22:27 |
| rharper | ahosmanMSFT: so, if it's not started, then when snapshot gets called there's not self._img_instance set and it returns the AzureCloudSnapshot() rather than doing the boot clean script and such | 22:28 |
| ahosmanMSFT | rharper: running again, I'll show you the logs and all the logging I added when trying to debug | 22:30 |
| ahosmanMSFT | I modified _instance(), as shown above | 22:31 |
| rharper | right, so the call path goes from platform.get_image() -> AzureCloudImage() -> platform.get_snapshot() -> snapshot.launch(); where you create an instance, but don't start it either () | 22:31 |
| ahosmanMSFT | rharper: that might be it, I didn't spot that | 22:33 |
| ahosmanMSFT | I'll make the change and run it again, keep you update | 22:33 |
| rharper | yeah, I'm just walking through cloud_tests/platforms/__init__.py cloud_tests/bddeb.py:setup_build() and comparing platform/ec2/* to platform/azurecloud/* | 22:34 |
| ahosmanMSFT | are you running this in a container | 22:34 |
| rharper | no, I'm just reading the code | 22:40 |
| ahosmanMSFT | I mean where the tests are run | 22:41 |
| rharper | no, just under tox on an jenkins node; | 22:43 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!