| === vrubiolo1 is now known as vrubiolo | ||
| === Cadmus_ is now known as Cadmus | ||
| AnhVoMSFT | @smoser @Odd_Bloke that clock jump is expected for VMs that have been pre-provisioned. cloud-init can block "waiting for media state change" for hours or days, or in some extreme case weeks. Is there an undesirable behavior arising from the clock jump? | 14:25 |
|---|---|---|
| AnhVoMSFT | looking at the log you provided in the ticket I could confirm from our data that the VM indeed was pre-provisioned. The prediction isn't based on the user's particular usage but rather based on the overall demand for the particular vm size/region/image... | 14:29 |
| smoser | AnhVoMSFT: but it seems unlikely (to me at least) that that system was a pre-provision | 14:38 |
| Odd_Bloke | smoser: Didn't Anh just confirm that it was pre-provisioned? | 14:42 |
| Odd_Bloke | (And it sounds like the pre-provisioning doesn't happen per-user, but instead happens within a region and then the pre-provisioned instances are assigned to an account on "launch". Given that, no user can infer whether to expect pre-provisioned VMs or not.) | 14:44 |
| smoser | well, yes. but i didn't read that before typing. :)~ | 14:44 |
| AnhVoMSFT | yes, it was indeed a pre-provisioned VM (confirmed from cloud-init log itself, and from the backend, using the data available in cloud-init log) | 14:46 |
| * smoser downloads the 'logs.tgz.gz' file which is infact a gzipped gzipped tarball | 14:46 | |
| AnhVoMSFT | lol - i just clickeda couple times on 7z and finally got to the log. I did wonder why it was gzipped a couple times | 14:47 |
| smoser | i suspect that 'ubuntu-bug' is doing something generically now that it wasn't originally doing | 14:51 |
| smoser | or launchpad is magically gzipping files longer than X or something. | 14:51 |
| smoser | it feels almost too perfect to me | 14:54 |
| smoser | that the system did not generate *any* journal entries betwee May 17 at 17:03 and May 26 at 13:52 | 14:55 |
| Odd_Bloke | I believe pre-provisioned VMs are suspended? | 14:56 |
| Odd_Bloke | (Maybe I'm inventing that as an explanation for this behaviour though. :p) | 14:56 |
| smoser | well.. some time ago they told me that that was just not possible | 14:56 |
| smoser | as i suggested that is how they should do it. | 14:56 |
| AnhVoMSFT | whether it's suspended or not is an implementation detail that can change, I believe. I think today they're not suspended | 14:56 |
| smoser | i susggesetd, as all the cloud-init magic would seem generally unnecessary if it got suspended. | 14:57 |
| AnhVoMSFT | however they might be in the future as they figure out some way to do it (I'm not in that team but did hear some plan about it a while back) | 14:57 |
| smoser | the platform could just say "oh, this instance id isn't ready yet - stop its cpu" | 14:57 |
| Odd_Bloke | Yeah, I guess properly suspended wouldn't bump the timestamps the way we're seeing. | 14:58 |
| smoser | well, it would | 14:58 |
| AnhVoMSFT | the system is indeed eerily quiet during that time. I think we ran tcpdump and saw almost no incoming/outgoing packet during that time | 14:58 |
| smoser | as your laptop counts uptime even when suspended | 14:58 |
| smoser | (i think... althoguh maybe i saw somethign about a kernel feature that did or did not do that) | 14:58 |
| AnhVoMSFT | i think the cloud-init "magic" was meant to save on kernel initialization and python module loading overhead | 15:00 |
| AnhVoMSFT | for smaller VM sizes a smaller number of IOPS is allocated to the VM and as such they would benefit more from pre-provisioning | 15:02 |
| AnhVoMSFT | @Odd_Bloke did you guys cut an SRU yet? | 15:08 |
| AnhVoMSFT | I saw a github release 6 days ago ubuntu/20.2-30-g8bcf1c06-0ubuntu1 - is this the SRU? | 15:09 |
| blackboxsw | AnhVoMSFT: we just landed branches last night to fixup a daily build recipe. and yes AnhVoMSFT that 'version' of cloud-init will be what we kick off the SRU for. | 15:12 |
| blackboxsw | AnhVoMSFT: I'll talk with the team today. I hope to queue the upload of the SRU this afternoon | 15:13 |
| blackboxsw | and will post to this channel and the mailinglist when ready for our verification. | 15:13 |
| Odd_Bloke | AnhVoMSFT: We expect to be doing another SRU in the next 3-6 weeks for https://github.com/canonical/cloud-init/pull/358 | 15:16 |
| falcojr | hrmph | 17:50 |
| falcojr | looks like pyflakes doesn't provide a way to ignore a line | 17:50 |
| falcojr | and it's complaining about my "from feature_overrides import *" line | 17:50 |
| falcojr | can we ditch pyflakes and pycodestyle in favor of flake8? | 17:50 |
| falcojr | doesn't make sense to use two different tools when one covers the same functionality | 17:51 |
| falcojr | and lets you ignore lines :D | 17:51 |
| falcojr | pyflakes github even says "If you like Pyflakes but also want stylistic checks, you want flake8, which combines Pyflakes with style checks against PEP 8 and adds per-project configuration ability." | 17:52 |
| meena | 19:50 <falcojr> can we ditch pyflakes and pycodestyle in favor of flake8? ← i thought i had recently seen a commit that got rid of flake8 xD | 17:58 |
| powersj | falcojr, the feature flags need to import *? | 18:00 |
| falcojr | yeah, as a means to provide a way to override them | 18:03 |
| falcojr | discussion on it starts here | 18:03 |
| falcojr | https://github.com/canonical/cloud-init/pull/367#issuecomment-630213500 | 18:03 |
| * powersj has some reading to do | 18:05 | |
| falcojr | huh, didn't know it was ever in the project, but it was removed here | 18:07 |
| falcojr | https://github.com/canonical/cloud-init/commit/80dfb3b023a268d6d6204220665c2cf43eac66df | 18:07 |
| falcojr | flake8 entails everything pyflakes and pycodestyle does | 18:07 |
| meena | i know, i use it… elsewhere. | 18:12 |
| rharper | Odd_Bloke: so meena 's https://github.com/canonical/cloud-init/pull/385 is good to go, except github shows it waiting on travis status, however, the two runs for the PR are green, https://travis-ci.org/github/canonical/cloud-init/builds/691054573 | 18:27 |
| rharper | meena: nice! close/open does the trick =) | 18:42 |
| rharper | new job pending | 18:42 |
| blackboxsw | paride: thanks for all the reviews on the ubuntu/* branches dropping pyflakes. I've checked our daily build CI runs for focal and we are all green on lxd/kvm using the newly built debs from yesterday. rharper Odd_Bloke paride I think we want to turn on our ec2-f CI integration test running and drop ec2-e https://jenkins.ubuntu.com/server/view/cloud-init,%20curtin,%20streams/job/cloud-init-integration-ec2-f/ | 18:49 |
| blackboxsw | ^ what do you folks think ^ on the jenkins running for ec2-f: on ec2-e: off? | 18:49 |
| blackboxsw | or both on. | 18:49 |
| blackboxsw | or even ec2-g ec2-f ec2-b would probably be more appropriate | 18:50 |
| rharper | blackboxsw: e is not yet EOL, no ? | 18:50 |
| rharper | so, +1 on ec2-g, f and b | 18:50 |
| rharper | and I suspect we leave e on until EoL | 18:50 |
| blackboxsw | rharper: nope you're right | 18:51 |
| blackboxsw | I'll enable them now. and put a PR up for server-jenkins-jobs to fix that (and the default ) | 18:51 |
| rharper | hrm, ubuntu-distro-info ; I thought used to show the EoL date (or days of support left etc) | 18:52 |
| rharper | is that gone now ? | 18:52 |
| rharper | or am I thinking of a different tool ? | 18:52 |
| blackboxsw | I aways looked at the raw /usr/share/distro-info/ubuntu.csv | 18:53 |
| blackboxsw | now there are two eols :) | 18:53 |
| blackboxsw | with esm around | 18:53 |
| blackboxsw | well 3 actually: eol,eol-server,eol-esm | 18:53 |
| Odd_Bloke | falcojr: Yep, I also want to switch back to flake8 for a task I have on my plate ATM. | 18:54 |
| Odd_Bloke | falcojr: But given my current state (back pain, for those following along at home), you should feel free to go ahead and DIY. | 18:54 |
| rharper | blackboxsw: heh | 18:55 |
| rharper | blackboxsw: so close, 2020-07-17 | 18:56 |
| blackboxsw | ok so we'll turn it on and add a timebomb for 2020-07-17 | 18:56 |
| blackboxsw | like if date >= 2020-07-17 raise RuntimeError :) | 18:57 |
| blackboxsw | RuntimeError('remember to turn this job off after 2020-07-17') | 18:57 |
| rharper | hehe, no need for timebombs; I don't think | 19:00 |
| rharper | maybe a trello card | 19:00 |
| Odd_Bloke | rharper: I think you were thinking of `ubuntu-distro-info --days=eol --series eoan` | 19:06 |
| rharper | yes | 19:06 |
| rharper | the no-input help message not the default --help message | 19:06 |
| rharper | =/ | 19:06 |
| * rharper needs more breadcumbs | 19:07 | |
| Odd_Bloke | rharper: I only know the minutiae of u-d-i because, well, lol: https://github.com/OddBloke/distro-info-rs | 19:10 |
| rharper | nice! | 19:11 |
| Odd_Bloke | I think that needs updating for ESM, actuallly. | 19:11 |
| kamoser | Hi all, when I have a package list like "packages: | 19:38 |
| kamoser | how can I ensure 2 and 3 are installed even if 1 fails? | 19:39 |
| kamoser | Reason for this is to simplify installs across distros. So in case the first package works on Amazon Linux but fails on Ubuntu, I want the rest of the packages to install | 19:40 |
| meena | kamoser: you could use a jinja2 template for your cloud-config | 19:40 |
| kamoser | meena Thanks. I assume you mean write statements based on the distro like "install package if Ubuntu" | 19:47 |
| kamoser | I will look into that, I have use Jinja with Salt previously | 19:47 |
| meena | is everyone fleeing Salt rn? | 19:47 |
| rharper | kamoser: what meena means is your #cloud-config file can *be* a jinja2 template directly, cloud-init itself can fill out the template with cloud-init provided values, like the distro name; ##template: jinja\n#cloud-config\npackages:\n{{ jinja logic here}} | 19:48 |
| rharper | kamoser: it's also a reasonable feature request, if you'd like, file an issue at https://bugs.launchpad.net/cloud-init/+filebug | 19:49 |
| kamoser | rharper Thanks! I will do that as well. It would be hard to guarantee the package list doesn't change in the future, and our AMIs build automatically. Appreciate the help from both of you. | 19:53 |
| rharper | kamoser: so, you might look at using a #include URL to your list | 19:54 |
| rharper | so you can independently modify the "packages" cloud-config from the AMI itself | 19:54 |
| rharper | cloud-init will fetch cloud-config from a URL | 19:54 |
| rharper | you can use object store as well, but only via http (not sure if you need it to be pulled via https); | 19:54 |
| kamoser | rharper awesome options. thanks :) | 19:56 |
| rharper | yw | 19:56 |
| meena | rharper: i… disagree :P failures should fail… although, installing packages does tend to be very transitively… <word that explains things> | 20:18 |
| rharper | meena: I think we're on the same page ... I would not change the *default* behavior; rather, if folks wanted to ignore install failures, that seems like a reasonable option for users to control; | 20:18 |
| rharper | meena: another though would be to extend the config space, to a dictionary, which could specify package lists under distro keys; | 20:19 |
| rharper | it's worth having a bug to discuss options | 20:19 |
| meena | rharper: *nod *nod *nod | 20:20 |
| blackboxsw | rharper: Odd_Bloke game of would you rather: our new-upstream-snapshot automatically updates to latest commit on master, and we released to groovy a few days ago so other commits have landed. Do we want to extend new-upstream-snapshot to take a commitish from which to 'snapshot' or do we want to propose a new groovy upload right now and queue an upload for SRU with what has currently landed since 8bcf1c06 | 20:31 |
| kamoser | I changed my template to say "## template: jinja | 20:31 |
| kamoser | #cloud-config". Should the header remain as Content-Type: text/cloud-config or should it be changed to text/jinja2? I don't see examples in the docs that clarify this point, but I am probably being dumb. | 20:31 |
| blackboxsw | rharper: Odd_Bloke queuing a groovy upload should be 5 mins of work | 20:31 |
| blackboxsw | +1 review time. | 20:31 |
| blackboxsw | an extension to new-upstream-snapshot to take an optional commitish could be helpful for future SRUs if we know that reviewers will trail authors when reviewing the upload PRs | 20:32 |
| blackboxsw | here's the list of extra commits in tip of master at the moment https://paste.ubuntu.com/p/jScGYQWVFY/ | 20:34 |
| blackboxsw | one functional change for ubuntu, couple unit tests/CI, contributors signed up and bsd enablement for cfg mgmt (doesn't affect ubuntu-proper) | 20:35 |
| rharper | blackboxsw: ok ... I need to do a PR on new-upstream-snapshot too to help for the "first-time-sru" scenario ; maybe you ran into it already ; I currently used: new-upstream-snapshot --no-bugs --sur-bug XXXX which mostly DTRT (with exception of not appending ~XX.YY.1 to the release version string | 20:42 |
| blackboxsw | right rharper https://trello.com/c/YZue6zO9/35-new-upstream-snapshot-needs-to-dtrt-and-append-proper-prefix-when-a-release-is-stable-on-first-sru | 20:44 |
| rharper | blackboxsw: heheh | 20:44 |
| blackboxsw | we have a coupe of onetime items that we need to fixup for SRU | 20:44 |
| rharper | \o/ | 20:44 |
| blackboxsw | I added your mention to that item, as I was going to manually 'handle' that in the ubuntu/focal SRU for cloud-init | 20:45 |
| blackboxsw | I think it'd be better to sort the new-upstream-snapshot for that | 20:45 |
| blackboxsw | then future us can forget all about that pain | 20:45 |
| blackboxsw | ok I'll put up a minor new-upstream-snapshot to take an optional commitish param | 20:46 |
| blackboxsw | rharper: want me to add functionality to inspect ubuntu.csv or are you already working that | 20:46 |
| blackboxsw | hrm hold the phone I think new-upstream-snapshot takes a positional commitish | 20:49 |
| rharper | blackboxsw: not working anything at the moment; fixing up some curtin vmtest issues pre-sru | 20:57 |
| blackboxsw | rharper: want to weigh in on this level of manual for ubuntu/xenial ? | 21:20 |
| rharper | ? | 21:20 |
| blackboxsw | rharper: https://github.com/canonical/cloud-init/pull/393 | 21:20 |
| rharper | looking | 21:21 |
| blackboxsw | since we did a one-off new-upstream-snapshot that we didn't release, the tool doesn't handle consolidating multiple UNRELEASED changelog entries into one from two separate new-upstream-snapshots | 21:21 |
| blackboxsw | so I had to manually move around a couple of entries | 21:21 |
| blackboxsw | per steps 1-3 | 21:21 |
| rharper | ah | 21:21 |
| rharper | yes, I've always worried about these | 21:22 |
| rharper | I think instead of writing "steps", it'd be worth showing a diff between what the new-upstream-snapshot does; and what we actually want to commit ? | 21:22 |
| blackboxsw | yeah I didn't really want to add specific debian/changelog content parsing logic in new-upstream-snapshot if we can avoid it | 21:22 |
| blackboxsw | sure that works | 21:22 |
| rharper | so do it once with no changes, then again with your manual steps and show the changes ... folks can look at the delta and see what they need to change during changelog edit I think | 21:23 |
| rharper | and I suspect, if we can automate that combintation of un-released snapshots, it would be a good backlog item to do (it's not often but nice for tooling to handle this stuff) | 21:23 |
| Odd_Bloke | blackboxsw: +1 on cutting a new groovy release (unless I'm too late and you've already committed to not doing that, then don't sweat it :p) | 21:25 |
| blackboxsw | Odd_Bloke: cutting groovy is also easy and I'll push that now. since I have to rework ubuntu/xenial anyway a bit to show a manual diff let's do it | 21:26 |
| blackboxsw | PR coming Odd_Bloke | 21:26 |
| Odd_Bloke | falcojr: https://github.com/canonical/cloud-init/pull/392 <-- we're unblocked on the flake8 issue, thanks to powersj | 21:28 |
| blackboxsw | Odd_Bloke: https://github.com/canonical/cloud-init/pull/394 | 21:29 |
| falcojr | lol, I was about to push basically the same thing | 21:30 |
| Odd_Bloke | blackboxsw: Approved. | 21:31 |
| blackboxsw | Odd_Bloke: thanks uploading | 21:32 |
| blackboxsw | at least our frequency of groovy uploads is healthy | 21:32 |
| blackboxsw | now to get SRUs healthy | 21:32 |
| powersj | falcojr, sorry :P | 21:34 |
| falcojr | haha, np | 21:34 |
| blackboxsw | community notice: ubuntu/groovy-proposed] cloud-init 20.2-38-g8377897b-0ubuntu1 (Accepted) | 21:35 |
| falcojr | over time we should fix some of those. E.g., whitespace around operators or the hanging indents | 21:35 |
| blackboxsw | community notice: ubuntu Groovy 20.10 has an upload representing tip of master that will show up in cloudimages in the coming days | 21:35 |
| powersj | agreed | 21:35 |
| falcojr | flake8 is a little more picky than the other tools were | 21:35 |
| falcojr | also, extend the max line length | 21:36 |
| * falcojr runs away | 21:36 | |
| Odd_Bloke | I would like to keep to 80 lines, but I would not object to figuring out a way to blacken the codebase over time. | 21:38 |
| Odd_Bloke | Upstream have been a little reticent to engage in partial formatting. It's a reasonable position to take, but it does make it hard for projects with a lot of history to move, because I don't really want `git blame` to be forever ruined by a formatting commit. | 21:40 |
| Odd_Bloke | Well, I guess more than a little reticent: https://github.com/psf/black/issues/134 | 21:42 |
| blackboxsw | rharper: here's the ubuntu/xenial SRU branch https://github.com/canonical/cloud-init/pull/395 | 21:50 |
| blackboxsw | if that approach makes sense I'll have something comparable for bionic eoan and focal I believe | 21:51 |
| rharper | blackboxsw: lemme look | 22:00 |
| kamoser | Some lines omitted but why doesn't this work on EC2 user data? "Content-Type: text/jinja2; | 22:16 |
| kamoser | ## template: jinja | 22:16 |
| kamoser | #cloud-config | 22:16 |
| kamoser | packages: | 22:16 |
| kamoser | - {{ v1.distro }}" | 22:16 |
| kamoser | Just says CI_MISSING_JINJA_VAR/distro | 22:16 |
| rharper | kamoser: one sec | 22:18 |
| rharper | kamoser: cloud-init --version ? what are you currently running there? and Ubuntu image or Amazon Linux ? | 22:19 |
| kamoser | ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20191002 (ami-04b9e92b5572fa0d1) | 22:20 |
| kamoser | and /usr/bin/cloud-init 19.2-36-g059d049c-0ubuntu1~18.04.1 | 22:20 |
| kamoser | To test it, I'm running /usr/bin/cloud-init -d init then I'm looking at the file /var/lib/cloud/instance/cloud-config.txt which is where I see the statement CI_MISSING_JINJA_VAR/distro | 22:21 |
| rharper | y | 22:21 |
| kamoser | keeps kicking me. reason is, I am just trying to debug why my user data won't work on ec2. | 22:26 |
| rharper | kamoser: ok, I can reproduce, that *seems* like a bug, you should be able to use v1.distro or distro, the original jinja commit mentions | 22:26 |
| rharper | Additionally, any standardized instance-data.json keys scoped below a | 22:26 |
| rharper | '<v#>' key will be promoted as a top-level key for ease of reference in | 22:26 |
| rharper | templates. This means that '{{ local_hostname }}' is the same as using the | 22:26 |
| rharper | latest '{{ v#.local_hostname }}'. | 22:26 |
| rharper | try with just {{ distro }} | 22:27 |
| rharper | for now | 22:27 |
| rharper | hrm, nope; thats broken too | 22:27 |
| kamoser | Bummer. Should I just be using bash script to workaround? Basically what I'm trying to achieve is Ubuntu doesn't have package "A" so I was writing Jinja that said "if package is not Ubuntu" to go with my cloud-config packages list. | 22:29 |
| kamoser | I was hoping the package list would come out as e.g. RHEL has packages A, B but Ubuntu only has packages B. So I inserted a Jinja if statement into the middle of my cloud-config "packages" list. | 22:30 |
| rharper | kamoser: well, we need to fix our bug, but that won't help you for now; you can in your shell script use fetch the distro value via /run/cloud-init/instance-data.json ;;; use jq on it or whatever else you need | 22:31 |
| kamoser | No worries, will do. | 22:33 |
| rharper | kamoser: ah, I think on bionic, we don't have the newer keys available; | 22:36 |
| rharper | kamoser: this will work on a focal image, but not bionic *yet*; blackboxsw is starting our SRU process where we bring back the newer features into the older releases, so in a few weeks this should work on bionic; | 22:36 |
| rharper | kamoser: and you can see the current keys available with : cloud-init query --all , and test them out with cloud-init query -f "{{ variable }}" ; putting in "{{ distro }}" returns exactly what you saw on bionic; here's it working on focal, for reference https://paste.ubuntu.com/p/Db8djv3nNc/ | 22:42 |
| kamoser | ok awesome. great tip | 22:44 |
| rharper | kamoser: also, cloud-init devel render --user-data </path to your template file> to render the whole thing | 22:48 |
| blackboxsw | yeah +1 kamoser SRU will get you that 'distro' jinja variable :/ sry 'bout that | 22:49 |
| blackboxsw | a bit delayed | 22:49 |
| kamoser | rharper excellent, now I can test a lot faster without startup/shutdown of the instance. No problem about the variable... only thing is, our customer specifically requested Ubuntu 18.04. But I am going to check if we can get the newer version | 22:50 |
| rharper | kamoser: it should be available in a few weeks; if you *cant* wait till the SRU is done, you can try using our daily-ppa for bionic which runs cloud-init master built for bionic, https://launchpad.net/~cloud-init-dev/+archive/ubuntu/daily | 22:51 |
| kamoser | Thanks! | 22:52 |
| blackboxsw | and kamoser you could also test using ubuntu groovy 20.10 which already has the fix | 22:53 |
| blackboxsw | if that ubuntu release is available for you :/ | 22:53 |
| blackboxsw | thanks for the review rharper I've pushed ubuntu/bionic in the same light https://github.com/canonical/cloud-init/pull/396 | 23:45 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!