[07:38] <nomad_fr> Hi all
[08:14] <Laney> rbalint: I don't understand what you mean with your "re: mozjs", I thought you were interested in gcc-10 crashing on armhf so told you about it
[08:14] <Laney> is that "no, I am not interested in that"?
[08:38] <slyon> Hello! I recently fixed the autopkgtests of 'mailman-hyperkitty'. Could someone please trigger this test for me, to make britney happy and unlock the 'zope.interface' proposed-migration? https://autopkgtest.ubuntu.com/request.cgi?release=groovy&arch=amd64&package=mailman-hyperkitty&trigger=zope.interface/4.7.1-2ubuntu1&trigger=setuptools/49.3.1-2 (needed for amd64, arm64, armhf, ppc64el, s390x)
[08:54] <Laney> slyon: done, thanks for fixing
[09:14] <rbalint> Laney, yes not interested at the moment, but interested in general. i'm wonderig it the armhf builders/testbeds have some hw fault that causes the reliability issues and this could be tested by running the builds/tests elsewhere
[09:45] <slyon> Thank you Laney!
[10:44] <LocutusOfBorg> tdaitx, hello, FYI in syncd src:groovy into groovy, and the build didn't fail anymore
[10:44] <LocutusOfBorg> so the patch is now dropped
[10:44] <LocutusOfBorg> having groovy in sync with groovy is... desiderable
[16:26] <Odd_Bloke> slyon: xnox: o/ I'm reading through https://bugs.launchpad.net/cloud-init/+bug/1892851 and I have one initial question: what would run `systemctl isolate cloud-stage0.target` in this design?
[16:29] <xnox> Odd_Bloke:  cloud-init's systemd system generator would divert default.target to point at cloud-stage0.target, meaning that systemd starts booting not to sysinit.target/multi-user.target, but to cloud-stage0.target.
[16:31] <Odd_Bloke> xnox: OK, that makes sense; what would handle setting it back to its previous value?
[16:32] <Odd_Bloke> (Well, what would handle that, the daemon-reload, and kicking off the second systemd run?)
[17:46] <xnox> Odd_Bloke:  any unit, that is wanted/after cloud-stage0.target, which is started whould have to kick the next stage to i.e. start/isolate to multi-user.target (the real default-target).
[17:47] <xnox> Odd_Bloke:  i.e. systemd in initrd does that, with e.g. https://github.com/systemd/systemd/blob/master/units/initrd-parse-etc.service which is wanted by initrd.target, and starts to a new-target.
[17:50] <xnox> Odd_Bloke:  my vision, for example, is that a gold image may have none of networkd/NetworkManager/ifupdown enabled at all. Cloud init it starts, discovers local datasource or that weird "private" network that it does to fetch data source, reads things there, and then like. Oh i should use netplan with netowrk-manager, generate that, reload, continue booting. And thus this time around on first boot we
[17:50] <xnox> enabled NetworkManager.service / NetworkManager-wait-online.
[17:50] <xnox> but if datasource has netplan config with networkd, instead netplan generate hooks up systemd-networkd.service / systemd-networkd-wait-online.service and boots with that.
[17:51] <xnox> today above is not possible, and we require the image to have hard-coded enabled the "default" network provider, and it's impossible to switch them like that, upon first boot, if both of them are available, but neither are configured or enabled.
[17:53] <Odd_Bloke> xnox: I don't think I understand enough about how initrd systemd is configured to understand how https://github.com/systemd/systemd/blob/master/units/initrd-parse-etc.service maps onto this discussion, could you break that down a little further?
[17:54] <xnox> Odd_Bloke:  https://www.freedesktop.org/software/systemd/man/bootup.html#Bootup%20in%20the%20Initial%20RAM%20Disk%20(initrd)
[17:55] <xnox> From there, you can see that systemd in initrd has "default.target -> initrd.target"
[17:55] <xnox> when it reaches that, a service is started that does daemon-reload, starts a new target and isolates to a new one to pivot root.
[17:56] <xnox> effectively completing the boot, and starting a different target right after.
[17:56] <xnox> we can talk about https://www.freedesktop.org/software/systemd/man/bootup.html#System%20Manager%20Bootup too and how i want to subvert that. and include such reload/re-start-a-new-target.
[17:58] <Odd_Bloke> Aha, OK, that makes a lot more sense, thank you!
[18:01] <Odd_Bloke> xnox: And are you thinking that cloud-init will ship the systemd configuration which will create this two-phase boot?  Or are we proposing that Ubuntu more generally switches to this two-phase boot, and cloud-init will integrate appropriately?
[18:01] <Odd_Bloke> (And if you're thinking that cloud-init should ship it, are you concerned about the major divergence in boot graphs between different Ubuntu systems that this will cause?)
[18:03] <xnox> Odd_Bloke:  there is no point for two-phase boot, in general. it is only needed for cloud-init case, and dynamically, when we are doing first-boot with cloud-init only.
[18:03] <Odd_Bloke> (Where by "different systems" I think I just mean desktop vs server (vs d-i server, though I suppose that's only a concern if we're backporting), but I may be missing some other place where cloud-init wouldn't be installed.)
[18:03] <xnox> Odd_Bloke:  because during normal boots, we have systemd system-generators that are executed before anything of this starts. Thus for example, netplan system generator runs, creates all units / deps / ordering, then systemd calculates the first transaction and boots it.
[18:04] <xnox> Odd_Bloke:  the issue is that with cloud-init first boot, when the first transaction is calculated we don't know if we will or will not need (netplan, networkmanager, wpa_supplicant, ovs, etc.....) because once the transaction has started, one cannot change what it will do with daemon-reload/adding/removing any units | before-after | wants | requires | etc.
[18:05] <xnox> one has to complete a transaction (meaning all units are started). Call daemon-reload, pick up the new units, calculate a new transaction and execute that.
[18:06] <xnox> Thus in general case, without cloud-init or "second boot, after cloud-init completed" there will be just regular single transaction boot.
[18:07]  * xnox is not sure how much sense i make
[18:09] <Odd_Bloke> xnox: cloud-init has to dynamically determine if this is a first boot or not during execution, so I think we would need to be two-phase on every boot (but I haven't thought that through very much at all, so I may be completely wrong :p).
[18:10] <xnox> Odd_Bloke:  no, because you have system generator already, before the first transaction is calculated which decides whether to enable cloud-init or not.
[18:11] <xnox> Odd_Bloke:  /lib/systemd/system-generators/cloud-init-generator the ds-identify thing.
[18:11] <xnox> that is executed on every boot, before systemd calculates the boot transaction and starts any units.
[18:12] <xnox> instead of just enabling/disabling cloud-init.target, i want that to subvert default.target; and then later start cloud-init-target/default-target, after initial stage is done.
[18:13] <xnox> Odd_Bloke:  on those graphs, before anything is started/attempted to be reached, systemd mounts /proc; /sys; /dev; cgroups, executes environment generators, and system-generators. Which should decide what we shall boot today.
[18:14] <xnox> with cloud-init  installed, most of the time, it's like "ds-identify says no idea, do not start any cloud-init" =)
[18:14] <Odd_Bloke> xnox: Two-phase on every boot of a cloud-init-enabled system, I mean, yeah.
[18:16] <xnox> Odd_Bloke:  we could optimize slightly if need be. I.e. if at generator time, you can see that /var/lib/cloud/..... exists, we might not need to do two-stage boot. Cause networking was seeded already.
[18:17] <xnox> Odd_Bloke:  cause any dynamic networking changes, are expected to rerun daemon-reload, netplan apply things, rather than just do "netplan generate" just-in-time and hope that that didn't change any runtime deps.
[18:18] <Odd_Bloke> Nope, that's what I'm saying: cloud-init cannot determine without querying the cloud fabric (or _sometimes_ examining DMI data) whether this a second/subsequent boot of an instance or the first boot of an instance launched from an image captured from an instance which booted.
[18:18] <xnox> Odd_Bloke:  cause for most clouds, on second-boot, /etc/netplan/* is populated during system-generator time, and netplan system generator hooks up all the units to ensure that ovs/wpa_supplicant/networkmanager/etc are all will be started as part of the first (only) boot transaction.
[18:18] <xnox> oh, i see.
[18:18] <xnox> as in the /etc/netplan/ could be stale captured stuff.
[18:19] <xnox> i didn't know you support such stuff.
[18:19] <xnox> very interesting.
[18:19] <Odd_Bloke> No-one does, but a lot of people rely on it. :p
[18:19] <Odd_Bloke> (I'm, in fact, writing documentation for it at the moment. :)
[18:19] <xnox> har har har
[18:20] <xnox> to be on the save side, yes, you want two stage boot, whenever cloud-init is active then. (regardless if it is firstboot or reboot)
[18:21] <xnox> cause if one captures image, boots new instance, with new instance id metadata saying ovs, whilst previous one didn't have ovs, one will want to do things properly.
[18:21] <xnox> Odd_Bloke:  there is a different place we can cut things at.
[18:22] <xnox> Odd_Bloke:  for example, we could cut cloud-final into a separate transaction. Such that we complete boot, then do daemon-reload, then do netplan apply, then start cloud-final.
[18:22] <xnox> Odd_Bloke:  because the issue is that when cloud-final runs, ovs/wpa_supplicant must all be up already.
[18:22] <xnox> but i guess it depends on which cloud-init modules one is runnning when.
[18:22] <xnox> and when one expects to be able to access wifi/ovs
[18:51] <ahasenack> kanashiro: Odd_Bloke: is MM down for you too?
[18:52] <ahasenack> back now
[18:53] <Odd_Bloke> xnox: OK, and this is what I was thinking of earlier when I said desktop vs server: subiquity-installed systems have cloud-init active, so they would all also end up with {two-phase boot, isolated cloud-final, etc.}.
[19:00] <xnox> Odd_Bloke:  hm, true. In subiquity case, it was expected to be one time only config however.
[19:00] <xnox> Odd_Bloke:  so even in case of nocloud ds-identify, we still enable cloud-init.target on everyboot?
[19:01] <xnox> Odd_Bloke:  i thought, our expectation, was that cloud-init is a one-time only thing on subiquity server.
[19:01] <xnox> mwhudson:  ^^^^^
[19:01] <xnox> (see odd_block & I conversing, on doing a ground breaking different way to boot by default)
[19:23] <Odd_Bloke> "Odd_Block"?? smh my head
[19:25] <Odd_Bloke> xnox: If we detect a NoCloud seed directory then cloud-init runs (if not otherwise disabled), the assumption being someone wouldn't have put one there if they didn't want cloud-init to run.
[19:25] <Odd_Bloke> subiquity may currently be breaking this assumption, of course. :)
[19:39] <mwhudson> subiquity doesn't make a nocloud seed any more
[19:39] <mwhudson> it configures the "none" datasource
[19:50] <rafaeldtinoco> ddstreet: completed the nvme-cli bug review.. I think we should generate needed scrips in Bionic as well. Dropped you a bunch of comments. Would love to have lab to give it all a try (NVMEoF FTW)
[20:01] <Odd_Bloke> Oh, hmm, guess I should have updated the installer before this test then!
[20:11] <Odd_Bloke> xnox: So I'm reading through the conversation at https://github.com/CanonicalLtd/netplan/pull/157/ and an important point is raised: how will we handle fixing this on stable releases?  This two-phase boot feels like a _major_ change which wouldn't be appropriate for backport.  (Certainly I wouldn't feel comfortable backporting it as part of cloud-init, even with our exception.)
[20:12] <Odd_Bloke> xnox: So do we need to be (in parallel) pursuing a more targetted fix (perhaps that PR?) which we _can_ backport?
[20:22] <Odd_Bloke> Aha, but it still leaves the None seed in place, so the same issue persists.
[20:22] <ahasenack> vorlon: hi, I think the motd-news sru (base-files, ubuntu-meta) has been verified for a week now, any objections in releasing it you think?
[20:22] <ahasenack> (not that I can do it)
[20:23] <vorlon> ahasenack: I haven't looked; do you want to check with the SRU vanguard?
[20:23] <ahasenack> who is it today? Robie?
[20:23] <ahasenack> if it's him, he is eod
[20:24] <vorlon> ahasenack: seems so
[20:24] <Odd_Bloke> subiquity also configures a single datasource, which IIRC cloud-init also takes as an indication that it is intended to be run.
[20:25] <vorlon> ahasenack: and tomorrow is sil2100 so he's also not around; I'll try to take a look after my current call
[20:26] <ahasenack> vorlon: thank you
[20:34] <vorlon> ahasenack: done
[20:40] <ahasenack> vorlon: \o/
[21:00] <xnox> Odd_Bloke:  the multi-transaction boot, is the only way to fix all of the crippling issues around dynamically created units/dependencies. All the pre-existing ones, current or future ones.
[21:01] <xnox> Odd_Bloke:  backports => i'm not sure what we can do for backports as a more targeted fix. We can reboot upon observing netplan that will not correctly apply prior to cloud-final in a single-transaction boot.
[21:02] <xnox> Odd_Bloke: or we build custom images, that somehow have units pre-enabled. Aka the stuff that was missnamed as a "workaround" and dismissed by smoser.
[21:23] <Odd_Bloke> xnox: I mean, there's a PR with a better solution than that sitting right there, no?
[21:23] <Odd_Bloke> (For backport, I mean.)
[21:24] <Odd_Bloke> xnox: s/smoser/slyon/ ?
[21:24] <mwhudson> Odd_Bloke: fwiw i had been thinking that subiquity's cloud init config should do something to disable it after
[21:25] <mwhudson> Odd_Bloke: like writing something to ds-identify.cfg or whatever
[21:25]  * mwhudson has some backlog to read apparently
[21:33] <smoser> multi-transactional boot seems overly complicated, and yes, a workaround.
[21:34] <smoser> but /me going afk now
[21:44] <xnox> smoser:  no, multi-transactional is the better alternative to what was proposed before - stub units that are started half way during boot.
[21:44] <xnox> smoser:  neither of these two designs i would call "workaround" and i hate that such term was used in pr or bug reports.
[21:45] <xnox> smoser:  only upstart had events, which allow to dynamically add/remove/replace units as part of the boot. No other init system offers that.
[21:46] <xnox> Odd_Bloke:  so the other alternative, is to have cloud-init start a partially loaded netplan.target after cloud-init has loaded up datasource and wrote out netplan yaml.
[21:47] <xnox> Odd_Bloke: either cloud-init starting netplan-hook.target, or cloud-init providing cloud-init-hook.target which netplan generate can add things to and start.
[21:47] <xnox> Odd_Bloke:  there was disagreement from smoser for neither cloud-init nor netplan providing such targets, and neither of them calling them.
[21:48] <xnox> smoser: Odd_Bloke: one way or the other, we do need to do something. Apart from the above two designs, I'm out of ideas.
[21:48] <xnox> smoser:  what is your expectation? for us to switch to upstart? =)