[07:27]  * meena doesn't understand people who just log off from irc
[14:44] <Odd_Bloke> cpaelzer: Sounds like you just had a conversation about the ifupdown cloud-init thing?  What was the resolution?
[14:45] <Odd_Bloke> meena: Same!  (Though we use IRC internal to Canonical, so I suppose for some people it might be part of how they stop working at the end of the day. :p)
[14:45] <cpaelzer> yes, give me a sec Odd_Bloke
[14:46] <cpaelzer> back with you
[14:46] <cpaelzer> the solution is that I clearly showed that it has avalid alternative dependencies
[14:46] <cpaelzer> so it isn't really a problem to get into the archive
[14:46] <cpaelzer> there are two solutions discussed atm
[14:47] <cpaelzer> 1. add extra-exclude in the seeds to get these packages out (I don't think it fits the case, but if it works ...)
[14:47] <cpaelzer> 2. the tooling generating component mismatches might be confused by "Provides"
[14:47] <cpaelzer> doko will talk to colin what the right path out of it will be
[14:47] <cpaelzer> he will ping me with the outcome
[14:48] <cpaelzer> until then we wait
[14:48] <cpaelzer> Odd_Bloke: anything to add?
[14:48] <cpaelzer> the recently synced logcheck has the same problem btw and also is a server package
[14:48] <Odd_Bloke> Nope, that sounds good to me, thanks!
[14:48] <meena> Odd_Bloke: i just close my laptop
[14:49] <meena> (and then open my phone)
[14:49] <Odd_Bloke> I don't normally check IRC outside of work hours, but it's still sitting there in a tmux waiting for me. :p
[15:45] <Odd_Bloke> meena: Are you able to attach the logs requested to https://bugs.launchpad.net/cloud-init/+bug/1855170 ?
[15:45] <ubot5> Ubuntu bug 1855170 in cloud-init "system_info can change the distro" [Undecided,Incomplete]
[15:46] <Odd_Bloke> (I see you replied, so wasn't sure if you were going to follow up with them. :)
[15:48] <Odd_Bloke> cpaelzer: Could you add a few words to https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1855557, please?
[15:48] <ubot5> Ubuntu bug 1855557 in cloud-init (Ubuntu) "please drop ifupdown depedency" [High,Triaged]
[15:59] <cpaelzer> Odd_Bloke: done
[16:01] <Odd_Bloke> Thanks!
[16:15] <ahosmanMSFT> Hi blackboxsw, what's needed to get idchange PR merged. I've tested and validated the bug fix.
[16:20] <blackboxsw> hi ahosmanMSFT I think your branch still has travis CI test errors that need resolution. The branch is checked red https://github.com/canonical/cloud-init/pull/84
[16:21] <blackboxsw> ahosmanMSFT: and also a unit test for dsaz.check_instance_id in test_azure.py too as we mentioned in discussion on the PR
[16:22] <ahosmanMSFT> There is a unit test for instance_id already, right
[16:23] <ahosmanMSFT> Or do you mean to add a specific one for this functionality
[16:27] <ahosmanMSFT> blackboxsw looking at the Travis CI tracebacks:
[16:27] <ahosmanMSFT> FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ci-TestAzureBounce.88hts2am/data/instance-id'
[16:28] <ahosmanMSFT> is that file suppose to be generated by the CI
[16:29] <ahosmanMSFT> All the errors are of that type
[16:29] <ahosmanMSFT> FileNotFoundError: [Errno 2] No such file or directory: '/tmp/ci-TestAzureBounce.~~~/data/instance-id'
[16:51] <blackboxsw> ahosmanMSFT: so each test sets up a tmpdir if using self.tmp_dir()  that gets removed during tearDown. it's likely that directory is being torn down before your test completes maybe? I think we'll need a pdb in the setup as I mentioned on the PR to walk through a given failing test.
[16:51] <blackboxsw> then you can run .tox/py27/bin/nosetests tests.unittests.test_datasource.test_azure:TestAzureDataSource.test_waagent_d_has_0700_perms -s
[16:52] <blackboxsw> and it'll let you step through the test and see why there is a race that the unit test loses.
[16:52] <blackboxsw> I'll peek now for a few mins.
[16:54] <blackboxsw> loooks like something is failing in get_boot_telemetry
[16:56] <blackboxsw> ahosmanMSFT: ahh your branch is reading config data fromprevious = util.load_file(os.path.join(
[16:56] <blackboxsw> +            self.paths.get_cpath('data'),
[16:56] <blackboxsw> +            'instance-id')).strip()
[16:56] <blackboxsw>   via dsaz._iid  (which is why we also need a unit test that exercises the datasource properly for the changeset you introduced).
[16:57] <blackboxsw> this is what's causing the test failures, because those failing unit tests  need to create the instance-id data file  in self.paths.get_cpath
[17:05] <ahosmanMSFT> blackboxsw instead of adding that file to all the unit-tests is there anything else we can do, like mock the file to return a fixed iid
[17:07] <blackboxsw> ahosmanMSFT: here's a patch that fixes one test, since iid is now comparing previous iid to next, those tests need to actually see a viable instance id instead of test-instance-id https://paste.ubuntu.com/p/Gxc3K2KH4t/
[17:07] <blackboxsw> you'll need to move that write_file up into either setUp or _get_ds
[17:08] <blackboxsw> so that all unit tests (not just the perms test) get viable previous_iid vs iid.
[17:08] <blackboxsw> and you'll need an extra test in that class to show a byte_swapped failure to match to make sure behavior of __iid works as intended given a check_instance_id
[17:09]  * blackboxsw jumps back on SRU validation for a bit. cloud-init status meeting in a few mins
[17:09] <blackboxsw> ahosmanMSFT: on your PR too, we need a sensible PR summary and description that can be used for the commit message if possible (because we'll squash merge it)
[17:16] <blackboxsw> https://github.com/cloud-init/ubuntu-sru/pull/71 is up for review for SRU verification on softlayer
[17:16] <blackboxsw> ok time for meeting start :)
[17:17] <blackboxsw> #startmeeting Cloud-init bi-weekly status
[17:17] <meetingology> Meeting started Tue Dec 10 17:17:14 2019 UTC.  The chair is blackboxsw. Information about MeetBot at http://wiki.ubuntu.com/meetingology.
[17:17] <meetingology> Available commands: action commands idea info link nick
[17:18] <blackboxsw> Welcome folks to another cloud-init status meeting. Probably the last one of the year I presume due to upcoming Holidays in two weeks
[17:18] <blackboxsw> #chair rharper
[17:18] <meetingology> Current chairs: blackboxsw rharper
[17:18] <blackboxsw> #chair Odd_Bloke
[17:18] <meetingology> Current chairs: Odd_Bloke blackboxsw rharper
[17:18] <rharper> o/
[17:18] <blackboxsw>  cloud-init upstream uses this meeting as a platform for community updates, feature/bug discussions, and an opportunity to get some extra input on current development.
[17:19] <blackboxsw> Let's start the meeting with setting the next meeting time
[17:19] <blackboxsw> day/time
[17:20] <blackboxsw> I think most of upstream is out the last two weeks of December. Shall we try January 7th?
[17:20] <blackboxsw> Anyone opposed can voice their discontent as I remember the keystrokes to set the topic of the channel ;)_
=== blackboxsw changed the topic of #cloud-init to: cloud-init pull-requests https://git.io/JeVed | Meeting minutes: https://goo.gl/mrHdaj | Next status meeting January 7 17:15 UTC | 19.4 (Dec 17) | 20.1 DROP py2.7 (Jan 2020) | https://bugs.launchpad.net/cloud-init/+filebug
[17:21] <blackboxsw> I also dropped 19.3 upstream release date from the channel topic as "that's soooo November"
[17:22] <blackboxsw> topics for this round: Feel free to interject/suggest other topics at any time.  Our typical format is the following: Previous Actions, Recent Changes, In-progress Development, Community Charter, Upcoming Meetings, Office Hours (~30 mins).
[17:22] <blackboxsw>  #topic Previous Actions
[17:22] <blackboxsw> #topic Previous Actions
[17:23] <blackboxsw> and oops, forgot to publish meeting minutes from last session.
[17:23] <blackboxsw> doing that now.
[17:24] <meena> is it meeting time or did I miss it?
[17:25] <Odd_Bloke> meena: It's happening as we speak.
[17:27] <blackboxsw> ok sorry for the delay. just pushed published meeting minutes
[17:27] <blackboxsw> #link https://cloud-init.github.io/status-2019-11-26.html#status-2019-11-26
[17:27] <blackboxsw> ok so previous actions from last meeting:
[17:28] <blackboxsw> no unresolved actions listed
[17:28] <blackboxsw> #topic Recent Changes
[17:29] <blackboxsw> found via git log --since 11.26.2019
[17:29] <blackboxsw>     - dhcp: Support RedHat dhcp rfc3442 lease format for option 121 (#76)
[17:29] <blackboxsw>       [Eric Lafontaine] (LP: #1850642)
[17:29] <blackboxsw>     - network_state: handle empty v1 config (#45) (LP: #1852496)
[17:29] <blackboxsw>     - Merge pull request #94 from gaughen/patch-1 [Joshua Powers]
[17:29] <blackboxsw>     - removed a couple of "the"s [gaughen]
[17:29] <blackboxsw>     - docs: fix line length and remove highlighting [Joshua Powers]
[17:29] <blackboxsw>     - docs: Add security.md to readthedocs [Joshua Powers]
[17:29] <ubot5> Launchpad bug 1850642 in cloud-init "No support for classless-static-routes on centos 7" [Medium,Triaged] https://launchpad.net/bugs/1850642
[17:29] <blackboxsw>     - Multiple file fix for AuthorizedKeysFile config (#60) [Eduardo Otubo]
[17:29] <blackboxsw>     - Merge pull request #88 from OddBloke/travis [Joshua Powers]
[17:29] <blackboxsw>     - Revert "travis: only run CI on pull requests"
[17:29] <blackboxsw>     - doc: update links on README.md [Joshua Powers]
[17:29] <ubot5> Launchpad bug 1852496 in cloud-init (Ubuntu) "nocloud network-config mishandles empty config" [Low,In progress] https://launchpad.net/bugs/1852496
[17:29] <blackboxsw>     - doc: Updates to wording of README.md [Joshua Powers]
[17:29] <blackboxsw>     - Add security.md [Joshua Powers]
[17:29] <blackboxsw>     - setup.py: Amazon Linux sets libexec to /usr/libexec (#52)
[17:29] <blackboxsw>       [Frederick Lefebvre]
[17:29] <blackboxsw>     - Fix linting failure in test_url_helper (#83) [Eric Lafontaine]
[17:29] <blackboxsw>     - url_helper: read_file_or_url should pass headers param into readurl
[17:29] <blackboxsw>       (#66) (LP: #1854084)
[17:29] <blackboxsw>     - dmidecode: log result *after* stripping n [Igor Galić]
[17:29] <blackboxsw>     - cloud_tests: add azure platform support to integration tests
[17:29] <blackboxsw>       [ahosmanmsft]
[17:29] <ubot5> Launchpad bug 1854084 in cloud-init "Headers no longer passed through read_file_or_url" [Undecided,Fix committed] https://launchpad.net/bugs/1854084
[17:31] <blackboxsw> thanks for all the FreeBSD work landing, utility improvements, caching and doc changes. + the dhcp lease format updates
[17:31] <blackboxsw> #topic In-progress Development
[17:32] <blackboxsw> Generally upstream is doing a fair job of getting reviews to the community for PRs, though lately we've been spending a few cycles on SRU validation for cloud-init 19.4.31 into Ubuntu Xenial, Bionic, Disco and Eoan.
[17:32] <blackboxsw> expectation is that we should be able to clear this SRU validation today/tomorrow and get back onto the review queue in github for cloud-init
[17:33] <blackboxsw> most major cloud-platforms have passed validation with no regressions, so risk is low with this release
[17:34] <blackboxsw>  oops cloud-init 19.3.41  not 19.4.31
[17:35] <blackboxsw> there is plenty of work in flight by meena (FreeBSD improvements) and ahosmanMSFT (Azure instance-id work). that we hope to get reviewed and landed.
[17:36] <blackboxsw> community notice: Reminder we are working toward a 19.4 upstream release by end of year. So if there are bits/features that you hope to make the cut. please get those branches in shape by next tuesday Decemeber 17th
[17:36] <blackboxsw> #link https://lists.launchpad.net/cloud-init/msg00236.html
[17:37] <blackboxsw> community notice: Also a reminder that cloud-inig 19.4 will be the last release that claims official support for py2.7
[17:37] <blackboxsw> in January, tip of cloud-init will be allowed to drift from python2.7 support and tox -e py27 will no longer be exercised by upstream CI
[17:38] <blackboxsw> Odd_Bloke: or rharper anyything else in-progress at the moment?
[17:39] <blackboxsw> also, anyone interested in cloud-init development, please run tools/migrate-lp-user-to-github as mentioned in the hacking guide to make sure we can account for the CLA (contributor license agreement) for cloud-init
[17:39] <blackboxsw> #link https://cloudinit.readthedocs.io/en/latest/topics/hacking.html
[17:40] <blackboxsw> #topic Community Charter
[17:40] <blackboxsw> reminder on 19.4 release covered above, and getting your github account authorized for cloud-init conributions....
[17:41] <Odd_Bloke> Nothing from me!
[17:41] <meena> goneri and i have been working on / testing his freebsd render
[17:42] <blackboxsw> for folks with time to burn and bite-sized branches/fixes. we've got a lane on our trello board that gives a variety of fixes that the community can grab if they are looking for quick suggestions. "Community low-hanging fruit"
[17:42] <blackboxsw> #link https://trello.com/b/hFtWKUn3/daily-cloud-init-curtin
[17:42] <blackboxsw> meena: Goneri yes and thanks for the continued efforts there. A lot of good is coming out for cloud-init upstream as a result.
[17:42] <meena> Goneri removed a lot of code, but from what i gather, that code is still used elsewhere
[17:43] <meena> could someone explain why our where our how
[17:43] <blackboxsw> as always, any community member is encouraged to review other PRs from devs. All reviews and input welcome.
[17:44] <blackboxsw> meena: I'm not sure I follow, are you referring to a specific branch?
[17:44] <blackboxsw> or just where stale snapshots or cloud-init code lives on certain distros?
[17:45] <blackboxsw> #topic Office Hours (next ~30 mins)
[17:45] <blackboxsw> might as well kick the topic and open office hours for general cloud-init discussions, questions, bug/feature work etc.
[17:46] <blackboxsw> some upstream devs will have eyes/time available for discussion around anything cloud-init. This time will also be spent grooming the active review queue @ https://git.io/JeVed
[17:46] <blackboxsw> #link https://git.io/JeVed
[17:48] <meena> blackboxsw: https://github.com/canonical/cloud-init/pull/61#pullrequestreview-329139891
[17:48] <blackboxsw> checking
[17:52] <blackboxsw> meena: so Distro._bring_up_interface() is called by Distro._bring_up_interfaces() base class via Distro.apply_network
[17:59] <blackboxsw> which is called from Init.apply_network_config which gets run during "cloud-init init"  when network is brought up
=== ahosmanMSFT62 is now known as ahosmanMSFT
[18:06] <blackboxsw> I think https://github.com/canonical/cloud-init/pull/42 looks ready to merge. rharper has an outstanding "changes requested" but I believe those have been fixed. rharper I'll defer to you on this one
[18:08] <blackboxsw> and policy question on reviews: if we get one upstream core-dev +1 can we proceed to land the branch as long as the "changes requested" from other upstream core seem to be resolved?
[18:09] <blackboxsw> I'd vote that most recent core-dev on the PR that +1's can squash merge if they see that any prior core-dev's concerns seem to be addressed
[18:12] <blackboxsw> I think I'll add an action for next meeting to make sure we iron out review/merge policy so PRs don't sit stale
[18:13] <blackboxsw> #action rharper confirm no concerns on https://github.com/canonical/cloud-init/pull/42 and that PR can land.
[18:13] <meetingology> ACTION: rharper confirm no concerns on https://github.com/canonical/cloud-init/pull/42 and that PR can land.
[18:14] <blackboxsw> #action upstream core-devs to decide about whether a PR can land if any upstream dev still has 'requested changes'
[18:14] <meetingology> ACTION: upstream core-devs to decide about whether a PR can land if any upstream dev still has 'requested changes'
[18:16] <blackboxsw> ok I *think* that about wraps the meeting. Merry Christmas, Happy Hanukkah, Happy New Year and all that good stuff. See you all online.
[18:16] <blackboxsw> #endmeeting
[18:16] <meetingology> Meeting ended Tue Dec 10 18:16:39 2019 UTC.
[18:16] <meetingology> Minutes:        http://ubottu.com/meetingology/logs/cloud-init/2019/cloud-init.2019-12-10-17.17.moin.txt
[18:21] <blackboxsw> Meeting minutes published: https://cloud-init.github.io/status-2019-12-10.html#status-2019-12-10
[18:35] <meena> does anyone have any opinions on vendor-data being able to override "distro"?
[18:36] <meena> is rharper not here?
[19:18] <blackboxsw> it feels like vendor data shouldn't need to override distro, and frankly it feels a bit strange for cloud-init to be determining distro based on system_info. I'm not sure for which environment we wouldn't want cloud-init to just use util.get_linux_distro() nowadays. I'm thinking there used to be a couple of legacy cases where we'd need to set that distro as cloud-init couldn't detect it, but I'm wondering if can
[19:18] <blackboxsw> now be made unnecessary.   The reason I could see sourcing distro: X from cloud.cfg would be for folks rolling their own package and/or images, wanting to override the detected underlying distro because they know of preferred distro-class behavior that is applicable to their images. I can't really think of a compelling other reason off the top of my head. Maybe other folks have ideas there.
[19:19] <blackboxsw> I'd rather do-away with the cloud-init package (rpm/deb/etc) providing an opinionated system_info: distro: X and rely on util.get_linux_distro() if  system_info.get('distro') is None.
[19:21] <blackboxsw> but maybe that dynamic parsing of /etc/os-release or /bin/freebsd-version is considered too costly or brittle?  I *think* we may be at a place where we can make that shift now to a smarted cloud-init instead of forcing a package to strictly define the distro value
[19:22] <blackboxsw> meena, I presume you were referring to https://bugs.launchpad.net/cloud-init/+bug/1855170
[19:22] <ubot5> Ubuntu bug 1855170 in cloud-init "system_info can change the distro" [Undecided,Incomplete]
[19:25] <rharper> meena: my general opinion is that vendor-data is useful for setting *vendor* defaults; for example, a number of platforms provide a ntp endpoint, which is cloud specific; that's a perfectly good example where by default, if you enable ntp in your cloud-config, you'll get the vendor supplied ntp configuration; though if a user provides their own config, then that overrides it.
[19:26] <rharper> I've seen in some other cases where vendor-data is used to override default behavior in an image; which I think is less great;  I think some of the baked-in vendor-data in Digital Ocean, for example, is a bit over-the-top;
[19:27] <rharper> the bug specified Hetzner vendor-data as problematic; I've not seen what's included in that; but generally 1)  users can complain to vendor  2) let them know they're not using the vendor and why  3)  cloud-init should provide a way for you to override any vendor-data provided.  In that bug, I'd like to see if we can't collect enough information so we can provide an  docs page on disabling/overriding vendor-data ;
[19:30] <blackboxsw> +1 on that rharper I added that sudo cloud-init query vendordata could easily give us a gimpse of that data on a hetzner instance if someone has access
[19:42] <blackboxsw> rharper: initial thoughts on https://github.com/canonical/cloud-init/pull/48#pullrequestreview-328538167
[19:43] <blackboxsw> the one-shot daemon
[19:43] <rharper> thanks
[19:43] <blackboxsw> we should probably have a discussion about the cimain entry points and whether it's worth cleaning up cmd/main to use a better entry point for the daemon
[19:44] <blackboxsw> maybe tomorrow we can setup a mtg
[20:07] <Odd_Bloke> rharper: blackboxsw: I'm seeing disco and eoan lxd images built using lxc-proposed-snapshot fail to launch properly (AFAICT not due to cloud-init issues).  Have either of you seen such a problem?
[20:08] <blackboxsw> Odd_Bloke: I'm about to file a bug on that now. disco & eoan snapd.seeded.service is busted
[20:08] <blackboxsw> and blocking systemd-analyze
[20:08] <blackboxsw> systemctl list-jobs shows it for me
[20:09] <blackboxsw> is that what you are seeing?
[20:10] <blackboxsw> Odd_Bloke: looked a bit like this https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1806070
[20:10] <ubot5> Ubuntu bug 1806070 in snapd "snapd.seeded.service never completes preventing full boot to default target" [High,Fix released]
[20:10] <Odd_Bloke> Yep, that's it.
[20:10] <Odd_Bloke> Good find.
[20:10] <Odd_Bloke> Sorry, I'm seeing the same boot issue.
[20:10] <Odd_Bloke> Not 100% sure it's that bug. :)
[20:12] <rharper> yeah, has anyone tried looking at the snap changes output ?
[20:12] <Odd_Bloke> There are multiple failed changes.
[20:12] <rharper> bleh
[20:13] <Odd_Bloke> I don't think it's that bug, it looks like a mount is failing.
[20:13] <rharper> what about launching daily eoan or disco images ? without doing our upgrade
[20:13] <rharper> does it work ?
[20:13] <Odd_Bloke> Dec 10 20:10:26 eoan-lp1852100 mount[1237]: mount: only root can use "--options" option (effective UID is 1000000)
[20:13] <Odd_Bloke> Yep, it works.
[20:13] <Odd_Bloke> I think the capture is messing up UIDs somehow.
[20:13] <rharper> hrm
[20:14] <Odd_Bloke> # ls -lah /root/.bashrc
[20:14] <rharper> yeah, my manual launch + add proposed upgrade works
[20:14] <Odd_Bloke> -rw-r--r-- 1 1000000 1000000 3.1K Aug 27 18:31 /root/.bashrc
[20:14] <rharper> I think we've seen the snapshot stuff break before
[20:14] <rharper> there was an upstream issue filed, and I think we were told to deal with it ...
[20:15] <blackboxsw> rharper: snap changes 1    Error   today at 20:00 UTC  today at 20:00 UTC  Initialize system state
[20:15] <blackboxsw> 2    Error   today at 20:00 UTC  today at 20:00 UTC  Initialize device
[20:16] <Odd_Bloke> Yep, that's what I'm seeing.
[20:16] <blackboxsw> rharper: without upgrade it also breaks for me
[20:16] <Odd_Bloke> Yep, it's definitely the capture.
[20:20] <rharper> check in #snappy on freenode I guess
[20:20] <rharper> ah, blackboxsw you're already there =) +1
[20:20] <blackboxsw> yeah me also tries the internal channel as I'm getting no response there
[20:21] <blackboxsw> strange Odd_Bloke rharper is that I have 2 focal machines that appear to be on the same versions of snap-related pkgs. and one succeeds, the other doesn't
[20:22] <blackboxsw> kernel diffs though 5.3.0-19-generic (failure box) 5.3.0-23-generic (success box)
[20:22] <blackboxsw> I'll try a kernel upgrade on the host system and see if that fixes the broken
[20:25] <blackboxsw> Odd_Bloke: rharper I have  success after a kernel upgrade
[20:26] <blackboxsw> testing the proposed lxc launch to be sure
[20:26] <blackboxsw> well, to be specific, apt-get dist-upgrade on focal
[20:26] <rharper> are you using lxd from snap ?
[20:26] <rharper> I wonder if the shiftfs feature is at play
[20:26] <blackboxsw> rharper: yes on the 'formerly broken' box lxd         3.0.4                11348  3.0/stable/…  canonical✓     -
[20:27] <blackboxsw> rharper: and yes on the formerly-working box lxd                   3.18                        12631  stable    canonical✓        -
[20:27] <blackboxsw> so laptop (working) was on 3.18. desktop on 3.0.4 (broken originally)
[20:27] <blackboxsw> I'll refresh the snaps for both
[20:33] <blackboxsw> hrm ok -proposed image creation is still busted for me on lxc via using our lxc-setup-proposed tool.
[20:33] <blackboxsw> but bare lxc launch ubuntu-daily:disco work
[20:34] <blackboxsw> lxc-proposed-shapshot rather
[20:39] <blackboxsw> hrm on the broken disco.  curl https://api.snapcraft.io/api/v1/snaps/sections
[20:39] <blackboxsw> curl: (6) Could not resolve host: api.snapcraft.io.   (errors I saw related to snapd service in /var/log/syslog about dns errors)
[20:40] <blackboxsw> on bionic/xenial I can curl that hostname
[20:40] <blackboxsw> something funky w/ networking disco/eoan
[20:44] <rharper> that should be solvable
[20:44] <Odd_Bloke> blackboxsw: So which lxd version is working for you?
[20:44] <Odd_Bloke> s/working/working better/
[20:45] <rharper> networkctl , systemd-resolve --status
[20:46] <blackboxsw> yeah, about that. lxd                   3.18                        12631  stable    canonical✓        -   as a snap
[20:46] <blackboxsw> I'm testing snap refresh lxd --beta right now
[20:46] <blackboxsw> though I think --beta dropped the ubuntu-daily: image list (I am using ubuntu:XXX instead )
[20:47] <blackboxsw> awaiting an lxc spin up now
[20:47] <blackboxsw> bah, have to run an errand. more on this later if I get it working on my failed box
[20:49] <blackboxsw> btw, I'm also seeing tracebacks in cloud-init.log regarding ssh-import-id: [me] on bionic/xenial.
[20:50] <blackboxsw> I'm wondering what our lxc-proposed-snapshot is twiddling that might be botching initial network setup.
[20:58] <Odd_Bloke> blackboxsw: It might just be an issue with image capture in lxd?
[20:58] <Odd_Bloke> Rather than something specific to our script.
[21:03] <Odd_Bloke> powersj: FYI, we're running into some tooling issues with lxd that are hindering SRU work ATM.
[21:03] <powersj> that is what I have gathered :P
[21:04] <powersj> Odd_Bloke, you all don't know if it is lxd or the script yet though?
[21:04] <Odd_Bloke> I just launch ubuntu:d and a `lxc publish` didn't produce a broken image.
[21:04] <Odd_Bloke> *launched
[21:05] <Odd_Bloke> So it's some interaction between our tooling and lxd that's causing the problem.
[21:16] <Odd_Bloke> blackboxsw: rharper: I don't really know where to start on this.  Any thoughts?
[21:17] <rharper> Odd_Bloke: is this the snap failing one?
[21:17] <Odd_Bloke> I don't follow.
[21:18] <Odd_Bloke> lxc-proposed-snapshot produces broken images on disco and eoan.
[21:18] <rharper> ah, ok
[21:18] <Odd_Bloke> I _think_ everything else we've seen is a symptom of that.
[21:18] <rharper> can you reproduce on diglett? and I can just in and look at it ?
[21:18] <Odd_Bloke> I can try!
[21:19] <rharper> blackboxsw: it sounds like a general DNS failure; which could be an issue with networkd/resolved or our network config
[21:19] <rharper> if we can't hit the snap api end point nor launchpad (ssh import id failing) then we have a networking issue
[21:22] <Odd_Bloke> rharper: The image is oddbloke-disco-test and there's a launched container named neat-gopher.
[21:22] <rharper> thx
[21:22] <Odd_Bloke> (Note that `lxc shell ...` fails but `lxc exec ... /bin/bash` works.)
[21:23] <rharper> # networkctl status
[21:23] <rharper> WARNING: systemd-networkd is not running,
[21:23] <rharper> that's not good
[21:24] <rharper> Dec 10 21:22:00.423425 neat-gopher systemd-networkd[260]: Directory "/run/systemd/netif" already exists, but is owned by 0:0 (101:103 was requested), refusing.
[21:24] <rharper> looks like somethings up with the snapshots and permissions
[21:24] <rharper> I suspect shiftfs related
[21:40] <Odd_Bloke> rharper: Yeah, if you look at /root, the files are owned by 100000.
[21:40] <Odd_Bloke> Or 1000000.
[21:40] <rharper> right
[21:40] <Odd_Bloke> r"10+" is my point. :p
[21:41] <rharper> I don't know why that's the case with starting from snapshots
[21:41] <Odd_Bloke> The confusing thing is that it doesn't happen if you just do `lxc publish` of a running container.
[21:41] <rharper> and I've not found the google-fu to find the lxd issue; I'm sure there's one for this
[21:41] <Odd_Bloke> s/running/manually launched/
[21:41] <rharper> what commands do we run ?
[21:43] <rharper> ahh, the pstart
[21:43] <rharper> I wonder if there's something going on with init + modifications
[21:54] <Odd_Bloke> Wait, we inject a fake init?
[21:54] <blackboxsw> hrm is it possible we are constructing he network profile wrong in pstart?
[21:54] <blackboxsw> csmith@downtown:~/src/ubuntu-sru (sru/19.3.41-softlayer)$ lxc profile show pstart0
[21:54] <blackboxsw> config:
[21:54] <blackboxsw>   raw.lxc: lxc.init.cmd=/sbin/psuedo-init --network=10.3.23.1/24
[21:54] <blackboxsw> description: Profile for psuedo-start.
[21:54] <blackboxsw> devices:
[21:54] <blackboxsw>   eth0:
[21:54] <blackboxsw>     nictype: bridged
[21:54] <blackboxsw>     parent: pstart0
[21:54] <blackboxsw>     type: nic
[21:54] <blackboxsw> name: pstart0
[21:54] <blackboxsw> used_by:
[21:54] <blackboxsw> - /1.0/containers/disco-proposed-259097524
[21:54] <Odd_Bloke> Pastebin!
[21:54] <blackboxsw> bah
[21:54] <blackboxsw> wrong copy after pastebin
[21:55] <Odd_Bloke> blackboxsw: Are the UIDs in the broken-network container correct?
[21:55] <Odd_Bloke> (i.e. what are the files in /root owned by?)
[21:55] <Odd_Bloke> (If not, then I suspect the networking failure is just a symptom of root not owning any files on the system. :p)
[21:56] <blackboxsw> 10000... 1000 as you mentioned earlier
[21:57] <blackboxsw> 1000000 rather. yeah that could be
[21:57] <rharper> yes, this is related to non-root files
[21:57] <rharper> so, the question is what 3.18 did to break lxc-pstart
[21:57] <rharper> I think if you run this on older lxd, we should be fine
[21:57] <rharper> so a Xenial VM with lxd 2.0 will surely work fine
[21:59] <ahosmanMSFT> blackboxsw if I write to a file to test do I have to rewrite to that file... https://paste.ubuntu.com/p/7QrcSCP4wM/
[22:00] <Odd_Bloke> rharper: Right, good point about being able to just use a VM.
[22:08] <Odd_Bloke> lxc-proposed-snapshot requires lxd 2.1 or later, so a xenial VM doesn't do the trick.
[22:09] <rharper> oh, huh
[22:10] <rharper> I swear I ran it before on xenial
[22:10] <rharper> 3.0.3 in bionic debs
[22:10] <rharper> so maybe that's unbroken
[22:10] <Odd_Bloke> Yeah, I'm trying bionic next.
[22:36] <rharper> blackboxsw: huh, I didn't realize bionic didn't do network_data.json complete rendering ...
[22:36] <rharper> just spent 30 minutues trying to sort out why the dhcpv6-stateful config didn't render the right network-config during boot; but net-convert dumps it correctly  ...
[22:42] <Odd_Bloke> OK, a bionic VM appears to work fine.
[22:43] <Odd_Bloke> I'll finish this task up tomorrow morning.
[22:50] <blackboxsw> ahosmanMSFT: /7QrcSCP4wM/
[22:50] <blackboxsw> ahosmanMSFT: https://paste.ubuntu.com/p/hkgBtD5fN7/
[22:50] <blackboxsw> want to assert that a byteswapped instance-id results in emitting the previous iid. non-byteswapped results in emitting the current iid
[22:57] <ahosmanMSFT> blackboxsw thx, you think we'll have time to get this merged today once travis is looking good?
[22:59] <blackboxsw> ahosmanMSFT: will probably get through the final review tomorrow, as soon as the SRU publishes. need to fix the lxc issues we are seeing on the existing SRU so we can release that queued upload
[22:59] <blackboxsw> then we can also publish your changeset to ubuntu focal as soon as it lands
[22:59] <blackboxsw> as that publishing to Ubuntu Focal only takes about 5 mins to queue (no SRU testing required)
[23:00] <blackboxsw> so that changeset will be testable as soon as Azure cloud images get rebuilt
[23:01] <ahosmanMSFT> blackboxsw Great
[23:05] <blackboxsw> rharper: smoser Odd_Bloke what do you think about us dropping the pstart lxc-proposed-snapshot and instead trying to boot lxc with cloud-init disabled out of the gate. upgrading to -proposed and then rebooting with cloud-init enabled
[23:06] <blackboxsw> maybe that is a little 'dirtier' but we don't get messed up by lxc internal config changing w.r.t. snapshots and hijacking init
[23:22] <ahosmanMSFT> blackboxsw fixed idchange squash commit and travis run
[23:24] <blackboxsw> ok ahosmanMSFT and I'm changing your PR title to azure: avoid re-running cloud-init when instance-id is byte-swapped
[23:24] <ahosmanMSFT> (y)
[23:25] <blackboxsw> because we like to make the commit title to represent something descriptive about the area of code that is being touched
[23:53] <ahosmanMSFT> blackboxsw Looks all good from my side, let me know if I should add anything else https://github.com/canonical/cloud-init/pull/84