[09:36] that seems like an interesting omission… [13:42] meena: What does? [13:43] LongLiveCHIEF: Can you confirm what version of netplan you had in that image (`dpkg -l netplan.io`)? [13:44] sure, one moment [13:46] 0.99-0ubuntu3 for arm64 [13:46] q [13:46] q! [13:46] whoops [13:48] :) [13:48] OK, bummer, that's the latest version. [13:48] (So my hopes that this problem would be solved by someone else have been dashed. ;) [13:48] yeah... :sad: [13:49] there's still many hopes in that regard. That particular bucket of hope is bottomless [13:50] something interesting came up in #raspberrypi over the weekend about rfkill disabling wifi early on when wireless region wasn't set [13:50] in wpa [13:51] ...and it did get me thinking [13:52] is there a way to halt cloud-init and drop into a shell before a specific stage? or to log output of a specific command after/before a stage? [14:03] LongLiveCHIEF: OK, I've confirmed that waveform, the Ubuntu Pi person, is still looking into the wifi issue. [14:03] I don't think we have a good way of doing something like that, no. [14:04] no worries, i have my ways [14:07] forward my info to the ubuntu pi person if he wants someone in the field to test and report. my gh is longlivechief [14:50] blackboxsw: I think we can improve your force-push method to not need a tracking file: we already have all that information in git history. I'm putting something together now. [15:29] hi guys. i am building a custom image with packer. this gets build on digitalocean, and then reused on digitalocean. what this means is the first timeit runs it gets information from cloud-init regarding networking information, but when i instantiate the image on a new virutal server that original network information is no longer correct [15:29] the problem is that, when i create a new server with this image, the FIRST time it boots, it has the incorrect information for networking [15:30] however, i think this is because the networking information is inputted into /etc/network/interfaces.d/* AFTER network services are already running. So the first time, it'll not have a net conncetion, but correct values, and then subsequently it will work just fine [15:30] so i am looking for an idea on how to solve this. My one thought was to move /etc/init.d/network to a later sysvinit stage [15:39] +1 Odd_Bloke [15:40] mutantturkey: What distribution are these images for? And how are you installing cloud-init? [15:40] mutantturkey: I'm not super familiar with packer builds. but do you have the opportunity to run 'cloud-init clean --logs' before shutting down that image and redeploying it? [15:43] blackboxsw: `clean` wouldn't remove network configuration, right? [15:44] Odd_Bloke: no but it would tell cloud-init to re-run network config on next boot (the first redeploy of that created image) [15:44] I was making an assumption that mutantturkey's guess on /etc/network/interfaces.d/* being written AFTER networking services are running is probably unlikely [15:44] cloud-init should detect that its a new instance without a `clean`, though, so doing that could mask a more general issue on DO. [15:45] If cloud-init isn't plumbed into the boot sequence correctly, it's plausible (and perhaps likely) that it would run after networking services. [15:45] good point. it *could* be a general issue with DO, but I was presuming it was an issue with how the image is being created with packer [15:46] (And if we're talking about SysV init, then this presumably isn't an Ubuntu image, where I would have higher confidence that cloud-init will get plumbed in correctly simply via installation.) [15:46] Odd_Bloke: that's a good though too. yeah I guess if the before=networking-target.online (or whatever it's called) wasn't present then right, we could be in that camp [15:47] or sysV init too right [15:47] Right, except we get no Before= in a SysV world. :p [15:54] blackboxsw: https://paste.ubuntu.com/p/kbC4XkHq3H/ is my iteration of the force-push proposal. [16:37] Odd_Bloke: yes excellent [16:38] Odd_Bloke: though we need to limit that globbing to just debian/patches/*cpick* [16:38] blackboxsw: When would we have any cherry-picks in the daily branch that we want to retain? [16:38] Oh, I suppose for non-devel releases? [16:43] Odd_Bloke: any release can have release-specific packaging, config disabling changs [16:43] and those debian/patches would not contain a *cpick* or fox-cpick prefix [16:43] blackboxsw: i do have that opportunity [16:44] would that fofce on next boot [16:44] thats perfect [16:44] this is sysvinit, this is debian buster with a bunch of custom stuff on top [16:44] like https://github.com/canonical/cloud-init/tree/ubuntu/xenial/debian/patches [16:44] i can do a rebuild will take an hour or two, dwill that clean thing work? [16:44] Odd_Bloke: ^ [16:45] mutantturkey: the clean thing removes all semaphores that marked that networking was created in the first place.... but as Odd_Bloke mentoined, it might be masking some other pre-instance change that cloud-init tries to rely on when you are re-launching your created image [16:46] okay, then let me try that [16:46] again, sysvinit is more opaque so i dont really want to muck with it if possible :-) [16:47] mutantturkey: and ok on sysvinit, and Debian, so as Odd_Bloke mentioned the sysvinit dependency ordering may not have been created in your debian image properly to ensure cloud-init writes network before network is brough up by the system, without a strict startup dependency added, cloud-init *could* be writing the network config too late as you guessed [16:47] what does --logs do [16:47] wouldn't i want t o keep my logs [16:48] mutantturkey: --logs rms /var/log/cloud-init.log /var/log/cloud-init-output.log [16:48] alright im giving it a swing [16:49] which should allow you to distill what cloud-init does on 'first boot' during your first relaunch [16:49] blackboxsw: The presence of the semaphores isn't relevant if cloud-init isn't running at the right point in boot; we would expect the semaphore to be ignored for a new instance boot anyway. [16:49] mutantturkey: Installing cloud-init from the Debian archive? [16:49] Odd_Bloke: agreed, but it is useful to compare the last boot stage timestamp against the dmesg or journalctl logs [16:50] I was only thinking to avoid any previous boot logs from during the image creation process [16:55] Odd_Bloke: yes [16:55] okay, blackboxsw i will look at the logs. Clearing out the old config seems like it would be useful to force cloud init to redo its business. [17:07] Odd_Bloke: also I don't think we can individually just git log --online through all commits as we have had times where we needed to fixup interrelated cherrypicks and lack of a ordered set of commits across all cpick* patches would mean we can't back them out without conflicts partway through [17:11] actually oops. right git log debian/patche/*cpick* will order that for us properly anyway [17:19] that did not appear t owork, rebooting now [17:26] well now it's online but refusing ssh, i think something stuck let me try to reboot again [17:37] blackboxsw: Yep, that should take care of it. [17:37] Odd_Bloke: good, tested and looks good. I'm resolving your review comments now [17:37] Cool, thanks! [17:51] Odd_Bloke: I think I'm done https://github.com/canonical/uss-tableflip/pull/45 [17:56] blackboxsw: Did you push? I'm seeing a lot of unresolved comments. [18:03] hrm, maybe not. checking again and going through comments [18:43] I had missed about 5 comments Odd_Bloke pushed the changes === tds4 is now known as tds [20:27] Goneri: +1 on your branch https://github.com/canonical/cloud-init/pull/305/files with a minor nit. Just needs that and a rebase against master and we are good to go [20:29] blackboxsw, done [20:30] * blackboxsw just twiddles thumbs waiting on Travis thanks Goneri will land today [20:41] falcojr: Thanks for the CLA PR, as you can see, I've landed it. :) [20:41] cool, thanks [20:49] lucasmoura: Thanks for the CLA PR! [21:07] alright got sidetracked today but [21:07] first boot with cloud-init clean doesn't fix the problem [21:07] first boot i can verify has the correct *updated* configs [21:07] but it's not applied (i can see that via ifconfig) [21:08] Goneri: merged thanks ! [21:09] restarting networking does fix theproblem [21:09] mutantturkey: so Odd_Bloke's suggestion that maybe the deb didn't wire the proper SysV init script dependencies in your image might be the issue there [21:09] yep [21:09] theoretically i think i understand how sysviinit works but [21:09] the goal is, networking should run after cloud init [21:10] we expect to have a 4 stages wired in the right order during startup something akin to this https://github.com/canonical/cloud-init/tree/master/sysvinit/debian [21:12] so those are all different init scripts in /etc/init.d right? [21:15] okay I have those files cool [21:15] mutantturkey: yep. [21:15] so during boot this should all go to syslog? [21:15] I was just checking an older release of cloud-init on trusty to confirm separate config files in /etc/init [21:16] yes i do have those [21:19] thanks blackboxsw :-) [21:24] okay blackboxsw we have a lead! [21:24] it appears none of these are configred [21:24] mutantturkey: guaranteed to go to /var/log/cloud-init.log not sure where debian typically logs cloud-init emitted logfiles besides the default [21:24] if i read rc*.d right, anything that starts with S means 'start" [21:24] s/besides/in addition to/ [21:24] k means, kill at tha trun level [21:24] networking is in the rc.S runlevel [21:24] so looks like cloud-init's 4 things aren't gettng run at startp [21:24] that sounds like reasonable failure path ;) [21:25] though clearly something is running because cloud-init does LOG [21:26] ooff [21:26] mutantturkey: cat /run/cloud-init/status.json that'll tell you what stages did run (on newer cloud-init you can run "cloud-init analyze show") [21:28] blackboxsw: Note that trusty is upstart, not SysV. [21:29] oopsie, right-o though our collection of init scripts mentioned here https://github.com/canonical/cloud-init/tree/master/sysvinit/debian should be all-inclusive of what we expect to see on debian right [21:29] S12networking [21:29] that's so whack [21:30] blackboxsw: I wouldn't assume so (particularly as those haven't been touched in 7 years). [21:37] well that makes non sense [21:37] let me jump into #debian and seewhatup [21:41] blackboxsw: alright,here we go. it's at runelevel S, which is like single user mode, happens before evreything else [21:42] that's before _all_ the cloud service bits run [21:42] which well makes since to me because other services might try to bind to a specifc IP etc and fail if networking is not up [21:45] so, givne that, is there a move to make here [21:48] so i think "cloud-init writes config files, but doesnt run ifup or the like to configure networking after then config is changed' [21:52] bso my simple thought is another init file called /restart-networking-hack that runs on runelevel 6 that reruns ifup