| chillysurfer | good day, all! could i get a maintainer to review this pr? https://github.com/canonical/cloud-init/pull/509 thanks so much! | 12:39 |
|---|---|---|
| blackboxsw_ | apoligies I've got something funky going on with my irc proxy. So, I'll have to kick off this meeting as dunder bbsw | 16:54 |
| blackboxsw_ | #startmeeting cloud-init status meeting | 16:55 |
| meetingology | Meeting started Tue Jul 28 16:55:17 2020 UTC. The chair is blackboxsw_. Information about MeetBot at http://wiki.ubuntu.com/meetingology. | 16:55 |
| meetingology | Available commands: action commands idea info link nick | 16:55 |
| blackboxsw_ | community notice: time for another bi-weekly (or semi-monthly if you prefer) cloud-init community status meeting | 16:56 |
| blackboxsw_ | #chair Odd_Bloke smoser rharper | 16:56 |
| meetingology | Current chairs: Odd_Bloke blackboxsw_ rharper smoser | 16:56 |
| blackboxsw_ | Hello folks, cloud-init community status roundup. We gather here in this IRC channel every 2 weeks to discuss current development tasks and progress on cloud-init. | 16:57 |
| blackboxsw_ | All questions. side-conversations and interruptions are welcome | 16:57 |
| blackboxsw_ | Last meeting minutes live here | 16:58 |
| blackboxsw_ | #link https://cloud-init.github.io/status-2020-07-14.html#status-2020-07-14 | 16:58 |
| blackboxsw_ | he topics we'll cover today: Previous Actions, Recent Changes, In-progress Development, Community Charter, Office Hours (~30 mins). | 16:58 |
| blackboxsw_ | #topic Previous Actions | 16:58 |
| blackboxsw_ | None found in meeting minutes from last session. | 16:58 |
| * blackboxsw_ sets the topic for next meeting. | 16:59 | |
| blackboxsw_ | +2 weeks from now | 16:59 |
| === blackboxsw_ changed the topic of #cloud-init to: pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting Aug 11 16:15 UTC | 20.2 (Apr 26) | https://bugs.launchpad.net/cloud-init/+filebug | ||
| blackboxsw_ | August 11th 16:15 UTC | 16:59 |
| blackboxsw_ | #topic Recent Changes | 17:00 |
| blackboxsw_ | The following commits have been landed on master of upstream branch since last meeting: found via git log --since 2020-07-14 | 17:00 |
| blackboxsw_ | #link https://paste.ubuntu.com/p/RjZcwtk6Hd/ | 17:01 |
| blackboxsw_ | features of note: | 17:02 |
| blackboxsw_ | - azure: avoid bouncing hostname if set hostname fails | 17:02 |
| blackboxsw_ | - vmware: new defaults for post customization script overrides on vCloud | 17:03 |
| blackboxsw_ | - azure ValueError raised if JsonDecodeErrors is not available when parsing metadata | 17:04 |
| blackboxsw_ | Thanks Goneri otubo anhVo and dermotbradley for community contributions this round | 17:05 |
| blackboxsw_ | #topic In-progress Development | 17:05 |
| blackboxsw_ | Current projects for cloud-init are leading us to additional features: | 17:09 |
| blackboxsw_ | - network device hot plug support for cloud-init post-boot | 17:10 |
| blackboxsw_ | - better integration testing on other clouds, Oracle support | 17:10 |
| blackboxsw_ | - extended json schema validation and publishing full static schema versions for external tools | 17:11 |
| blackboxsw_ | #topic Community Charter | 17:11 |
| blackboxsw_ | The following topics are still topics for ongoing community development: | 17:12 |
| blackboxsw_ | - JSON schema extensions to validate user-data before instance launch: https://bugs.launchpad.net/cloud-init/?field.tag=bitesize | 17:13 |
| blackboxsw_ | - Datasource documentation and updates | 17:13 |
| blackboxsw_ | - cloudinit.net refactor into distro-specific networking subclasses cloudinit.distros.networking: https://bugs.launchpad.net/cloud-init/+bugs?field.tag=net-refactor | 17:14 |
| blackboxsw_ | As always: thank you all for bug contributions, PR submissions, triage and discussion participation. | 17:15 |
| blackboxsw_ | If anyone would like to be involved more than they currently are, please feel free to contact us here in IRC #cloud-init on Freenode or on the mailing list cloud-init@lists.launchpad.net and we can see how best we can get you "set up" | 17:15 |
| blackboxsw_ | #topic Office Hours (next ~30 mins) | 17:15 |
| blackboxsw_ | This time of the meeting is really just an open door for any discussions, concerns, bugs, questions or general prodding of upstream devs to make sure existing development work is unblocked where possible. | 17:17 |
| blackboxsw_ | In the absence of discussions, review of existing PRs is addressed. | 17:17 |
| blackboxsw_ | Thanks for tuning in folks. have a good one! | 17:56 |
| blackboxsw_ | #endmeeting | 17:56 |
| meetingology | Meeting ended Tue Jul 28 17:56:15 2020 UTC. | 17:56 |
| meetingology | Minutes: http://ubottu.com/meetingology/logs/cloud-init/2020/cloud-init.2020-07-28-16.55.moin.txt | 17:56 |
| Odd_Bloke | rharper: If you have a minute, could you take a look at https://bugs.launchpad.net/cloud-init/+bug/1888822 for me? The submitter has done a good job of triaging the issue themselves, and the commit "to blame" mentions you and a bug, but doesn't include enough info for me to fully understand the root of the change (including a lack of link to the mentioned bug). | 17:56 |
| ubot5 | Ubuntu bug 1888822 in cloud-init "cloud-init caches files and never checks again" [Undecided,New] | 17:56 |
| rharper | Odd_Bloke: yeah, I looked at it the other day ... and I got as far as not understanding how it worked in the first place; let me look at if there are any updates | 17:58 |
| Odd_Bloke | rharper: There was a comment an hour ago with the triage I mentioned. | 17:58 |
| rharper | yeah, I see it | 17:58 |
| rharper | I had suspected the change but haven't worked out what's happening yet | 17:59 |
| Odd_Bloke | So we've expanded the set of Content-Types that we consider "in need of determination of Content-Type" from TYPE_NEEDED to also include all Content-Types that are used for including too. | 18:00 |
| rharper | right | 18:00 |
| rharper | the issue is that their second item in their payload is not *present* during boot | 18:00 |
| rharper | so when we go to read the content type of the payload; it returns empty/none; so that modifies the mime-type | 18:01 |
| Odd_Bloke | But this means that if an include sets its Content-Type but _does not_ include the corresponding first line, it will be detected as x-shellscript. | 18:01 |
| rharper | then they *rerun* cloud-init.service *inside* their boot hook | 18:01 |
| rharper | it is not | 18:01 |
| rharper | it is set as an unmapped type | 18:01 |
| Odd_Bloke | I think this is the line that indicates the problem: 2020-07-24 09:26:04,614 - __init__.py[DEBUG]: {'Content-Type': 'text/x-shellscript', 'Content-Disposition': 'attachment; filename="part-001"'} | 18:11 |
| rharper | the logs are not clear; there are two days, 2020-07-23 using 19.4 ... and it emits an interesting like the 'Empty payload' which is when it tries to read /etc/secret-userdata.txt (and the file exists but is empty); | 18:12 |
| Odd_Bloke | I've just tested locally, and using user-data generated by `./tools/make-mime.py -a boothook:cloud-boothook` where `boothook` has the header and doesn't have the header behaves differently: namely, it is interpreted as x-shellscript without the header. | 18:12 |
| rharper | this is not present at all in the 07-24 run ... so I'm not sure we're getting clean logs rom them | 18:12 |
| rharper | that's the one of the points; if you provide a payload, we'll look at the content | 18:12 |
| rharper | an set the mime type by the content | 18:13 |
| rharper | their issue is that the content is not yet present when we look now | 18:13 |
| Odd_Bloke | Well, our docs say "Begins with: #cloud-boothook or Content-Type: text/cloud-boothook when using a MIME archive." | 18:13 |
| Odd_Bloke | Not "and". | 18:13 |
| rharper | they have two payloads | 18:13 |
| rharper | one is a boot hook script, which runs, to fetch the second contents for the second payload | 18:13 |
| Odd_Bloke | Right, and the boothook doesn't get run at boothook time any longer, it gets run as a user-data shellscript. | 18:13 |
| rharper | it does | 18:13 |
| rharper | the issue is the resulting cloud-config.txt does not include the secret-userdata | 18:13 |
| rharper | so their restart of cloud-init.service does not have any new user-data (From the secret they downloaded) to consume | 18:14 |
| rharper | the first payload very much does run | 18:14 |
| Odd_Bloke | I never said it didn't run. | 18:14 |
| rharper | why do you think it runs as a user-data script instead of the boot hook ? | 18:15 |
| Odd_Bloke | rharper: The output comes after this line: Cloud-init v. 20.2-45-g5f7825e2-0ubuntu1~18.04.1 running 'modules:config' at Fri, 24 Jul 2020 09:26:07 +0000. Up 112.48 seconds. | 18:19 |
| Odd_Bloke | Whereas I believe a boothook would have its output come substantially earlier than that? | 18:19 |
| Odd_Bloke | (In cloud-init-output.log.) | 18:19 |
| rharper | yeah, I think I can see that; I'm not sure that having the script run as boot hook would fix the issue; but maybe it would mean that the consume-data semaphor won't have been touched yet ? | 18:20 |
| rharper | I'm going to ask for a collect-logs from a successful run; | 18:20 |
| rharper | the logs provided are somewhat confusing as mentioned | 18:20 |
| rharper | Odd_Bloke: do we trust the payload or the mime-types ? previously, if you said it was a boot-hook and payload was #!/bin/sh it ran at boot-hook time; now we look at the payload and if we don't see the #! then it's a shell script; the payload would need to have #cloud-boothook ; | 18:31 |
| Odd_Bloke | rharper: I believe the linked commit moved us from "trust the mime-type, infer from the content if either we don't have a mime-type or the mime-type is in cloudinit.user_data.TYPE_NEEDED" to the same but add cloudinit.handlers.INCLUSION_TYPES_MAP.values() to the set of mime-types we won't trust. | 18:39 |
| Odd_Bloke | And I think that means that we will now not trust the mime-types for ~all mime-types people might pass to us. | 18:39 |
| Odd_Bloke | (So effectively we've moved to "always infer from the content, even if we've already been told what's in there".) | 18:40 |
| rharper | Yeah; I agree. And if we revert that; then the older scenario which always sets a mime-type to x-shell-script but provides #cloud-config instead stays broken; | 18:40 |
| Odd_Bloke | And boothooks have to start with a shebang (or perhaps be a binary executable? I haven't thought that through too much), which we detect as x-shellscript. | 18:40 |
| rharper | alternatively, the newer bits which want boot-hook can also provide payload with the right header for payload detection | 18:41 |
| Odd_Bloke | (I agree that I don't fully understand why their config was working before, to be clear. :p) | 18:41 |
| rharper | I see some clear paths with (if the mime-type and content-type match; continue on) if there's a mismatch; what do we want to do? I guess reverting makes the most sense as this regresses existing payloads which were tagged correctly (and mapped their payload to a reasonable handler) | 18:42 |
| Odd_Bloke | Yeah, I think we should revert; in the older scenario, is the first line of the content a reliable indicator of what's in there? (i.e. Is it true that they want us to treat text/x-shellscript as text/plain? Or do they want us to treat it as text/cloud-config?) | 18:46 |
| Odd_Bloke | So I just tested and if an x-shellscript part does not start with a shebang then we fail to execute it. | 18:49 |
| Odd_Bloke | So I think we can safely check the content on x-shellscript parts, because we will either (a) redetect it as x-shellscript because it starts with #!, or (b) were going to fail to run it later anyway because it didn't. | 18:50 |
| Odd_Bloke | (By "fail to execute it", I mean: 2020-07-28 18:48:36,307 - subp.py[DEBUG]: Exec format error. Missing #! in script? | 18:50 |
| rharper | that failed to execute happens when the first pass of boot attempts to run the empty file (secret-userdata.txt) file is present but empty (no content); | 18:52 |
| Odd_Bloke | Nope, I'm talking about a local run in a lxd container. | 18:54 |
| Odd_Bloke | Not their logs. | 18:54 |
| rharper | oh, I see, yes | 18:54 |
| rharper | that at least plugs the old hole, without harming this new breakage ; no need to check content of boot-hook; the only "known" case of mislabeled content is x-shell-script | 18:54 |
| Odd_Bloke | Right, I think the previous fix just spreads its net too wide. | 18:55 |
| rharper | ack | 18:57 |
| rharper | Odd_Bloke: https://paste.ubuntu.com/p/ZPydQdt8kY/ I think that should cover what we wanted; | 19:23 |
| rharper | I tried to recreate the bug's scenario in a unittest but I've not had any luck getting it to work; AFAICT, if the /etc/secret-userdata.txt file isn't present during boot; cloud-init will fail saying missing file import or something like that; if the file is present but empty (touch /etc/secret-userdata.txt) then we take a different path where the payload has it's mime-type changed to x-not-multipart due to not being able to | 19:25 |
| rharper | read the content; and subsequent consume_userdata() calls don't then view the payload as a #include url anymore; | 19:25 |
| Odd_Bloke | rharper: I've just posted a comment to the bug, could you quickly review it before I ping other folks to let them know about the regression? | 19:30 |
| rharper | yeah | 19:33 |
| Odd_Bloke | Thx | 19:34 |
| rharper | Odd_Bloke: yeah, that looks good. | 19:35 |
| Odd_Bloke | rick_h: blackboxsw_: falcojr: lucasmoura: FYI, we've had a regression from the last SRU reported as https://bugs.launchpad.net/cloud-init/+bug/1888822; I've summarised the discussion that Ryan and I have been having into my most recent comment. | 19:35 |
| ubot5 | Ubuntu bug 1888822 in cloud-init "cloud-init caches files and never checks again" [Critical,Triaged] | 19:35 |
| rharper | I guess we will also need a boothook/shellscript multi-part mime test | 19:36 |
| Odd_Bloke | The title of that bug is misleading, let me update that. | 19:37 |
| rharper | yeah; though ... I'm super interested in seeing what they describe actually work (and understanding why) ... it really boggles my mind on including a non-existing file and restarting cloud-init ... | 19:38 |
| rharper | what they want is something like we discussed about having cloud-init fetch secrets from aws/ebs encrypted store a while back | 19:39 |
| rharper | no restarting of cloud-init.service needed | 19:39 |
| Odd_Bloke | We vaguely discussed previously having some sort of x-config-fetcher part which would be a Python script that would emit config on stdout. | 19:39 |
| Odd_Bloke | Yeah, I'm thinking of the same convo. | 19:39 |
| rharper | yeah | 19:39 |
| Odd_Bloke | Well, I guess it need not be Python if the interface is stdout. | 19:40 |
| Odd_Bloke | rharper: I would expand that comment to mention that we can safely round-trip true x-shellscript parts through find_ctype, that's why it's safe to carve that exception out; other than that, I agree that's the change we want/need. | 19:51 |
| Odd_Bloke | Are you working on this/tests for it ATM, or shall I pick it up? | 19:51 |
| Odd_Bloke | smoser: BTW, I believe https://github.com/canonical/cloud-init/pull/493 is now ready for review; as you implemented the Oracle DS originally, I wanted to give you a chance to review it. | 20:13 |
| rharper | Odd_Bloke: there's an existing test in-tree that tests the case where content is not shellscript; and we consume it correctly ( content is cloud-config) ; what test remains that we ned ? | 20:19 |
| rharper | Odd_Bloke: I'm not actively working on a new unittest; I was attempting to test that bookhook + x-include-url file://path scenario ... but I can't make it work; | 20:20 |
| Odd_Bloke | rharper: I wasn't sure, just knew you were poking at some testing. :) | 20:21 |
| rharper | fruitlessly; so thanks for spotting the initial issue | 20:23 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!