=== SuperKaramba is now known as BenderRodriguez === franksmcb_ is now known as franksmcb === ricotz_ is now known as ricotz [07:38] Hi all [08:14] rbalint: I don't understand what you mean with your "re: mozjs", I thought you were interested in gcc-10 crashing on armhf so told you about it [08:14] is that "no, I am not interested in that"? [08:38] Hello! I recently fixed the autopkgtests of 'mailman-hyperkitty'. Could someone please trigger this test for me, to make britney happy and unlock the 'zope.interface' proposed-migration? https://autopkgtest.ubuntu.com/request.cgi?release=groovy&arch=amd64&package=mailman-hyperkitty&trigger=zope.interface/4.7.1-2ubuntu1&trigger=setuptools/49.3.1-2 (needed for amd64, arm64, armhf, ppc64el, s390x) [08:54] slyon: done, thanks for fixing [09:14] Laney, yes not interested at the moment, but interested in general. i'm wonderig it the armhf builders/testbeds have some hw fault that causes the reliability issues and this could be tested by running the builds/tests elsewhere [09:45] Thank you Laney! [10:44] tdaitx, hello, FYI in syncd src:groovy into groovy, and the build didn't fail anymore [10:44] so the patch is now dropped [10:44] having groovy in sync with groovy is... desiderable === didrocks999 is now known as didrocks [16:26] slyon: xnox: o/ I'm reading through https://bugs.launchpad.net/cloud-init/+bug/1892851 and I have one initial question: what would run `systemctl isolate cloud-stage0.target` in this design? [16:26] Launchpad bug 1892851 in cloud-init "Staged boot, to fix integration of systemd generators" [Undecided,Confirmed] [16:29] Odd_Bloke: cloud-init's systemd system generator would divert default.target to point at cloud-stage0.target, meaning that systemd starts booting not to sysinit.target/multi-user.target, but to cloud-stage0.target. [16:31] xnox: OK, that makes sense; what would handle setting it back to its previous value? [16:32] (Well, what would handle that, the daemon-reload, and kicking off the second systemd run?) [17:46] Odd_Bloke: any unit, that is wanted/after cloud-stage0.target, which is started whould have to kick the next stage to i.e. start/isolate to multi-user.target (the real default-target). [17:47] Odd_Bloke: i.e. systemd in initrd does that, with e.g. https://github.com/systemd/systemd/blob/master/units/initrd-parse-etc.service which is wanted by initrd.target, and starts to a new-target. [17:50] Odd_Bloke: my vision, for example, is that a gold image may have none of networkd/NetworkManager/ifupdown enabled at all. Cloud init it starts, discovers local datasource or that weird "private" network that it does to fetch data source, reads things there, and then like. Oh i should use netplan with netowrk-manager, generate that, reload, continue booting. And thus this time around on first boot we [17:50] enabled NetworkManager.service / NetworkManager-wait-online. [17:50] but if datasource has netplan config with networkd, instead netplan generate hooks up systemd-networkd.service / systemd-networkd-wait-online.service and boots with that. [17:51] today above is not possible, and we require the image to have hard-coded enabled the "default" network provider, and it's impossible to switch them like that, upon first boot, if both of them are available, but neither are configured or enabled. [17:53] xnox: I don't think I understand enough about how initrd systemd is configured to understand how https://github.com/systemd/systemd/blob/master/units/initrd-parse-etc.service maps onto this discussion, could you break that down a little further? [17:54] Odd_Bloke: https://www.freedesktop.org/software/systemd/man/bootup.html#Bootup%20in%20the%20Initial%20RAM%20Disk%20(initrd) [17:55] From there, you can see that systemd in initrd has "default.target -> initrd.target" [17:55] when it reaches that, a service is started that does daemon-reload, starts a new target and isolates to a new one to pivot root. [17:56] effectively completing the boot, and starting a different target right after. [17:56] we can talk about https://www.freedesktop.org/software/systemd/man/bootup.html#System%20Manager%20Bootup too and how i want to subvert that. and include such reload/re-start-a-new-target. [17:58] Aha, OK, that makes a lot more sense, thank you! [18:01] xnox: And are you thinking that cloud-init will ship the systemd configuration which will create this two-phase boot? Or are we proposing that Ubuntu more generally switches to this two-phase boot, and cloud-init will integrate appropriately? [18:01] (And if you're thinking that cloud-init should ship it, are you concerned about the major divergence in boot graphs between different Ubuntu systems that this will cause?) [18:03] Odd_Bloke: there is no point for two-phase boot, in general. it is only needed for cloud-init case, and dynamically, when we are doing first-boot with cloud-init only. [18:03] (Where by "different systems" I think I just mean desktop vs server (vs d-i server, though I suppose that's only a concern if we're backporting), but I may be missing some other place where cloud-init wouldn't be installed.) [18:03] Odd_Bloke: because during normal boots, we have systemd system-generators that are executed before anything of this starts. Thus for example, netplan system generator runs, creates all units / deps / ordering, then systemd calculates the first transaction and boots it. [18:04] Odd_Bloke: the issue is that with cloud-init first boot, when the first transaction is calculated we don't know if we will or will not need (netplan, networkmanager, wpa_supplicant, ovs, etc.....) because once the transaction has started, one cannot change what it will do with daemon-reload/adding/removing any units | before-after | wants | requires | etc. [18:05] one has to complete a transaction (meaning all units are started). Call daemon-reload, pick up the new units, calculate a new transaction and execute that. [18:06] Thus in general case, without cloud-init or "second boot, after cloud-init completed" there will be just regular single transaction boot. [18:07] * xnox is not sure how much sense i make [18:09] xnox: cloud-init has to dynamically determine if this is a first boot or not during execution, so I think we would need to be two-phase on every boot (but I haven't thought that through very much at all, so I may be completely wrong :p). [18:10] Odd_Bloke: no, because you have system generator already, before the first transaction is calculated which decides whether to enable cloud-init or not. [18:11] Odd_Bloke: /lib/systemd/system-generators/cloud-init-generator the ds-identify thing. [18:11] that is executed on every boot, before systemd calculates the boot transaction and starts any units. [18:12] instead of just enabling/disabling cloud-init.target, i want that to subvert default.target; and then later start cloud-init-target/default-target, after initial stage is done. [18:13] Odd_Bloke: on those graphs, before anything is started/attempted to be reached, systemd mounts /proc; /sys; /dev; cgroups, executes environment generators, and system-generators. Which should decide what we shall boot today. [18:14] with cloud-init installed, most of the time, it's like "ds-identify says no idea, do not start any cloud-init" =) [18:14] xnox: Two-phase on every boot of a cloud-init-enabled system, I mean, yeah. [18:16] Odd_Bloke: we could optimize slightly if need be. I.e. if at generator time, you can see that /var/lib/cloud/..... exists, we might not need to do two-stage boot. Cause networking was seeded already. [18:17] Odd_Bloke: cause any dynamic networking changes, are expected to rerun daemon-reload, netplan apply things, rather than just do "netplan generate" just-in-time and hope that that didn't change any runtime deps. [18:18] Nope, that's what I'm saying: cloud-init cannot determine without querying the cloud fabric (or _sometimes_ examining DMI data) whether this a second/subsequent boot of an instance or the first boot of an instance launched from an image captured from an instance which booted. [18:18] Odd_Bloke: cause for most clouds, on second-boot, /etc/netplan/* is populated during system-generator time, and netplan system generator hooks up all the units to ensure that ovs/wpa_supplicant/networkmanager/etc are all will be started as part of the first (only) boot transaction. [18:18] oh, i see. [18:18] as in the /etc/netplan/ could be stale captured stuff. [18:19] i didn't know you support such stuff. [18:19] very interesting. [18:19] No-one does, but a lot of people rely on it. :p [18:19] (I'm, in fact, writing documentation for it at the moment. :) [18:19] har har har [18:20] to be on the save side, yes, you want two stage boot, whenever cloud-init is active then. (regardless if it is firstboot or reboot) [18:21] cause if one captures image, boots new instance, with new instance id metadata saying ovs, whilst previous one didn't have ovs, one will want to do things properly. [18:21] Odd_Bloke: there is a different place we can cut things at. [18:22] Odd_Bloke: for example, we could cut cloud-final into a separate transaction. Such that we complete boot, then do daemon-reload, then do netplan apply, then start cloud-final. [18:22] Odd_Bloke: because the issue is that when cloud-final runs, ovs/wpa_supplicant must all be up already. [18:22] but i guess it depends on which cloud-init modules one is runnning when. [18:22] and when one expects to be able to access wifi/ovs [18:51] kanashiro: Odd_Bloke: is MM down for you too? [18:52] back now [18:53] xnox: OK, and this is what I was thinking of earlier when I said desktop vs server: subiquity-installed systems have cloud-init active, so they would all also end up with {two-phase boot, isolated cloud-final, etc.}. [19:00] Odd_Bloke: hm, true. In subiquity case, it was expected to be one time only config however. [19:00] Odd_Bloke: so even in case of nocloud ds-identify, we still enable cloud-init.target on everyboot? [19:01] Odd_Bloke: i thought, our expectation, was that cloud-init is a one-time only thing on subiquity server. [19:01] mwhudson: ^^^^^ [19:01] (see odd_block & I conversing, on doing a ground breaking different way to boot by default) [19:23] "Odd_Block"?? smh my head [19:25] xnox: If we detect a NoCloud seed directory then cloud-init runs (if not otherwise disabled), the assumption being someone wouldn't have put one there if they didn't want cloud-init to run. [19:25] subiquity may currently be breaking this assumption, of course. :) [19:39] subiquity doesn't make a nocloud seed any more [19:39] it configures the "none" datasource [19:50] ddstreet: completed the nvme-cli bug review.. I think we should generate needed scrips in Bionic as well. Dropped you a bunch of comments. Would love to have lab to give it all a try (NVMEoF FTW) [20:01] Oh, hmm, guess I should have updated the installer before this test then! [20:11] xnox: So I'm reading through the conversation at https://github.com/CanonicalLtd/netplan/pull/157/ and an important point is raised: how will we handle fixing this on stable releases? This two-phase boot feels like a _major_ change which wouldn't be appropriate for backport. (Certainly I wouldn't feel comfortable backporting it as part of cloud-init, even with our exception.) [20:12] xnox: So do we need to be (in parallel) pursuing a more targetted fix (perhaps that PR?) which we _can_ backport? [20:22] Aha, but it still leaves the None seed in place, so the same issue persists. [20:22] vorlon: hi, I think the motd-news sru (base-files, ubuntu-meta) has been verified for a week now, any objections in releasing it you think? [20:22] (not that I can do it) [20:23] ahasenack: I haven't looked; do you want to check with the SRU vanguard? [20:23] who is it today? Robie? [20:23] if it's him, he is eod [20:24] ahasenack: seems so [20:24] subiquity also configures a single datasource, which IIRC cloud-init also takes as an indication that it is intended to be run. [20:25] ahasenack: and tomorrow is sil2100 so he's also not around; I'll try to take a look after my current call [20:26] vorlon: thank you [20:34] ahasenack: done [20:40] vorlon: \o/ [21:00] Odd_Bloke: the multi-transaction boot, is the only way to fix all of the crippling issues around dynamically created units/dependencies. All the pre-existing ones, current or future ones. [21:01] Odd_Bloke: backports => i'm not sure what we can do for backports as a more targeted fix. We can reboot upon observing netplan that will not correctly apply prior to cloud-final in a single-transaction boot. [21:02] Odd_Bloke: or we build custom images, that somehow have units pre-enabled. Aka the stuff that was missnamed as a "workaround" and dismissed by smoser. [21:23] xnox: I mean, there's a PR with a better solution than that sitting right there, no? [21:23] (For backport, I mean.) [21:24] xnox: s/smoser/slyon/ ? [21:24] Odd_Bloke: fwiw i had been thinking that subiquity's cloud init config should do something to disable it after [21:25] Odd_Bloke: like writing something to ds-identify.cfg or whatever [21:25] * mwhudson has some backlog to read apparently [21:33] multi-transactional boot seems overly complicated, and yes, a workaround. [21:34] but /me going afk now [21:44] smoser: no, multi-transactional is the better alternative to what was proposed before - stub units that are started half way during boot. [21:44] smoser: neither of these two designs i would call "workaround" and i hate that such term was used in pr or bug reports. [21:45] smoser: only upstart had events, which allow to dynamically add/remove/replace units as part of the boot. No other init system offers that. [21:46] Odd_Bloke: so the other alternative, is to have cloud-init start a partially loaded netplan.target after cloud-init has loaded up datasource and wrote out netplan yaml. [21:47] Odd_Bloke: either cloud-init starting netplan-hook.target, or cloud-init providing cloud-init-hook.target which netplan generate can add things to and start. [21:47] Odd_Bloke: there was disagreement from smoser for neither cloud-init nor netplan providing such targets, and neither of them calling them. [21:48] smoser: Odd_Bloke: one way or the other, we do need to do something. Apart from the above two designs, I'm out of ideas. [21:48] smoser: what is your expectation? for us to switch to upstart? =)