| meena | that seems like an interesting omission… | 09:36 |
|---|---|---|
| Odd_Bloke | meena: What does? | 13:42 |
| Odd_Bloke | LongLiveCHIEF: Can you confirm what version of netplan you had in that image (`dpkg -l netplan.io`)? | 13:43 |
| LongLiveCHIEF | sure, one moment | 13:44 |
| LongLiveCHIEF | 0.99-0ubuntu3 for arm64 | 13:46 |
| LongLiveCHIEF | q | 13:46 |
| LongLiveCHIEF | q! | 13:46 |
| LongLiveCHIEF | whoops | 13:46 |
| Odd_Bloke | :) | 13:48 |
| Odd_Bloke | OK, bummer, that's the latest version. | 13:48 |
| Odd_Bloke | (So my hopes that this problem would be solved by someone else have been dashed. ;) | 13:48 |
| LongLiveCHIEF | yeah... :sad: | 13:48 |
| LongLiveCHIEF | there's still many hopes in that regard. That particular bucket of hope is bottomless | 13:49 |
| LongLiveCHIEF | something interesting came up in #raspberrypi over the weekend about rfkill disabling wifi early on when wireless region wasn't set | 13:50 |
| LongLiveCHIEF | in wpa | 13:50 |
| LongLiveCHIEF | ...and it did get me thinking | 13:51 |
| LongLiveCHIEF | is there a way to halt cloud-init and drop into a shell before a specific stage? or to log output of a specific command after/before a stage? | 13:52 |
| Odd_Bloke | LongLiveCHIEF: OK, I've confirmed that waveform, the Ubuntu Pi person, is still looking into the wifi issue. | 14:03 |
| Odd_Bloke | I don't think we have a good way of doing something like that, no. | 14:03 |
| LongLiveCHIEF | no worries, i have my ways | 14:04 |
| LongLiveCHIEF | forward my info to the ubuntu pi person if he wants someone in the field to test and report. my gh is longlivechief | 14:07 |
| Odd_Bloke | blackboxsw: I think we can improve your force-push method to not need a tracking file: we already have all that information in git history. I'm putting something together now. | 14:50 |
| mutantturkey | hi guys. i am building a custom image with packer. this gets build on digitalocean, and then reused on digitalocean. what this means is the first timeit runs it gets information from cloud-init regarding networking information, but when i instantiate the image on a new virutal server that original network information is no longer correct | 15:29 |
| mutantturkey | the problem is that, when i create a new server with this image, the FIRST time it boots, it has the incorrect information for networking | 15:29 |
| mutantturkey | however, i think this is because the networking information is inputted into /etc/network/interfaces.d/* AFTER network services are already running. So the first time, it'll not have a net conncetion, but correct values, and then subsequently it will work just fine | 15:30 |
| mutantturkey | so i am looking for an idea on how to solve this. My one thought was to move /etc/init.d/network to a later sysvinit stage | 15:30 |
| blackboxsw | +1 Odd_Bloke | 15:39 |
| Odd_Bloke | mutantturkey: What distribution are these images for? And how are you installing cloud-init? | 15:40 |
| blackboxsw | mutantturkey: I'm not super familiar with packer builds. but do you have the opportunity to run 'cloud-init clean --logs' before shutting down that image and redeploying it? | 15:40 |
| Odd_Bloke | blackboxsw: `clean` wouldn't remove network configuration, right? | 15:43 |
| blackboxsw | Odd_Bloke: no but it would tell cloud-init to re-run network config on next boot (the first redeploy of that created image) | 15:44 |
| blackboxsw | I was making an assumption that mutantturkey's guess on /etc/network/interfaces.d/* being written AFTER networking services are running is probably unlikely | 15:44 |
| Odd_Bloke | cloud-init should detect that its a new instance without a `clean`, though, so doing that could mask a more general issue on DO. | 15:44 |
| Odd_Bloke | If cloud-init isn't plumbed into the boot sequence correctly, it's plausible (and perhaps likely) that it would run after networking services. | 15:45 |
| blackboxsw | good point. it *could* be a general issue with DO, but I was presuming it was an issue with how the image is being created with packer | 15:45 |
| Odd_Bloke | (And if we're talking about SysV init, then this presumably isn't an Ubuntu image, where I would have higher confidence that cloud-init will get plumbed in correctly simply via installation.) | 15:46 |
| blackboxsw | Odd_Bloke: that's a good though too. yeah I guess if the before=networking-target.online (or whatever it's called) wasn't present then right, we could be in that camp | 15:46 |
| blackboxsw | or sysV init too right | 15:47 |
| Odd_Bloke | Right, except we get no Before= in a SysV world. :p | 15:47 |
| Odd_Bloke | blackboxsw: https://paste.ubuntu.com/p/kbC4XkHq3H/ is my iteration of the force-push proposal. | 15:54 |
| blackboxsw | Odd_Bloke: yes excellent | 16:37 |
| blackboxsw | Odd_Bloke: though we need to limit that globbing to just debian/patches/*cpick* | 16:38 |
| Odd_Bloke | blackboxsw: When would we have any cherry-picks in the daily branch that we want to retain? | 16:38 |
| Odd_Bloke | Oh, I suppose for non-devel releases? | 16:38 |
| blackboxsw | Odd_Bloke: any release can have release-specific packaging, config disabling changs | 16:43 |
| blackboxsw | and those debian/patches would not contain a *cpick* or fox-cpick prefix | 16:43 |
| mutantturkey | blackboxsw: i do have that opportunity | 16:43 |
| mutantturkey | would that fofce on next boot | 16:44 |
| mutantturkey | thats perfect | 16:44 |
| mutantturkey | this is sysvinit, this is debian buster with a bunch of custom stuff on top | 16:44 |
| blackboxsw | like https://github.com/canonical/cloud-init/tree/ubuntu/xenial/debian/patches | 16:44 |
| mutantturkey | i can do a rebuild will take an hour or two, dwill that clean thing work? | 16:44 |
| blackboxsw | Odd_Bloke: ^ | 16:44 |
| blackboxsw | mutantturkey: the clean thing removes all semaphores that marked that networking was created in the first place.... but as Odd_Bloke mentoined, it might be masking some other pre-instance change that cloud-init tries to rely on when you are re-launching your created image | 16:45 |
| mutantturkey | okay, then let me try that | 16:46 |
| mutantturkey | again, sysvinit is more opaque so i dont really want to muck with it if possible :-) | 16:46 |
| blackboxsw | mutantturkey: and ok on sysvinit, and Debian, so as Odd_Bloke mentioned the sysvinit dependency ordering may not have been created in your debian image properly to ensure cloud-init writes network before network is brough up by the system, without a strict startup dependency added, cloud-init *could* be writing the network config too late as you guessed | 16:47 |
| mutantturkey | what does --logs do | 16:47 |
| mutantturkey | wouldn't i want t o keep my logs | 16:47 |
| blackboxsw | mutantturkey: --logs rms /var/log/cloud-init.log /var/log/cloud-init-output.log | 16:48 |
| mutantturkey | alright im giving it a swing | 16:48 |
| blackboxsw | which should allow you to distill what cloud-init does on 'first boot' during your first relaunch | 16:49 |
| Odd_Bloke | blackboxsw: The presence of the semaphores isn't relevant if cloud-init isn't running at the right point in boot; we would expect the semaphore to be ignored for a new instance boot anyway. | 16:49 |
| Odd_Bloke | mutantturkey: Installing cloud-init from the Debian archive? | 16:49 |
| blackboxsw | Odd_Bloke: agreed, but it is useful to compare the last boot stage timestamp against the dmesg or journalctl logs | 16:49 |
| blackboxsw | I was only thinking to avoid any previous boot logs from during the image creation process | 16:50 |
| mutantturkey | Odd_Bloke: yes | 16:55 |
| mutantturkey | okay, blackboxsw i will look at the logs. Clearing out the old config seems like it would be useful to force cloud init to redo its business. | 16:55 |
| blackboxsw | Odd_Bloke: also I don't think we can individually just git log --online through all commits as we have had times where we needed to fixup interrelated cherrypicks and lack of a ordered set of commits across all cpick* patches would mean we can't back them out without conflicts partway through | 17:07 |
| blackboxsw | actually oops. right git log debian/patche/*cpick* will order that for us properly anyway | 17:11 |
| mutantturkey | that did not appear t owork, rebooting now | 17:19 |
| mutantturkey | well now it's online but refusing ssh, i think something stuck let me try to reboot again | 17:26 |
| Odd_Bloke | blackboxsw: Yep, that should take care of it. | 17:37 |
| blackboxsw | Odd_Bloke: good, tested and looks good. I'm resolving your review comments now | 17:37 |
| Odd_Bloke | Cool, thanks! | 17:37 |
| blackboxsw | Odd_Bloke: I think I'm done https://github.com/canonical/uss-tableflip/pull/45 | 17:51 |
| Odd_Bloke | blackboxsw: Did you push? I'm seeing a lot of unresolved comments. | 17:56 |
| blackboxsw | hrm, maybe not. checking again and going through comments | 18:03 |
| blackboxsw | I had missed about 5 comments Odd_Bloke pushed the changes | 18:43 |
| === tds4 is now known as tds | ||
| blackboxsw | Goneri: +1 on your branch https://github.com/canonical/cloud-init/pull/305/files with a minor nit. Just needs that and a rebase against master and we are good to go | 20:27 |
| Goneri | blackboxsw, done | 20:29 |
| * blackboxsw just twiddles thumbs waiting on Travis thanks Goneri will land today | 20:30 | |
| Odd_Bloke | falcojr: Thanks for the CLA PR, as you can see, I've landed it. :) | 20:41 |
| falcojr | cool, thanks | 20:41 |
| Odd_Bloke | lucasmoura: Thanks for the CLA PR! | 20:49 |
| mutantturkey | alright got sidetracked today but | 21:07 |
| mutantturkey | first boot with cloud-init clean doesn't fix the problem | 21:07 |
| mutantturkey | first boot i can verify has the correct *updated* configs | 21:07 |
| mutantturkey | but it's not applied (i can see that via ifconfig) | 21:07 |
| blackboxsw | Goneri: merged thanks ! | 21:08 |
| mutantturkey | restarting networking does fix theproblem | 21:09 |
| blackboxsw | mutantturkey: so Odd_Bloke's suggestion that maybe the deb didn't wire the proper SysV init script dependencies in your image might be the issue there | 21:09 |
| mutantturkey | yep | 21:09 |
| mutantturkey | theoretically i think i understand how sysviinit works but | 21:09 |
| mutantturkey | the goal is, networking should run after cloud init | 21:09 |
| blackboxsw | we expect to have a 4 stages wired in the right order during startup something akin to this https://github.com/canonical/cloud-init/tree/master/sysvinit/debian | 21:10 |
| mutantturkey | so those are all different init scripts in /etc/init.d right? | 21:12 |
| mutantturkey | okay I have those files cool | 21:15 |
| blackboxsw | mutantturkey: yep. | 21:15 |
| mutantturkey | so during boot this should all go to syslog? | 21:15 |
| blackboxsw | I was just checking an older release of cloud-init on trusty to confirm separate config files in /etc/init | 21:15 |
| mutantturkey | yes i do have those | 21:16 |
| Goneri | thanks blackboxsw :-) | 21:19 |
| mutantturkey | okay blackboxsw we have a lead! | 21:24 |
| mutantturkey | it appears none of these are configred | 21:24 |
| blackboxsw | mutantturkey: guaranteed to go to /var/log/cloud-init.log not sure where debian typically logs cloud-init emitted logfiles besides the default | 21:24 |
| mutantturkey | if i read rc*.d right, anything that starts with S means 'start" | 21:24 |
| blackboxsw | s/besides/in addition to/ | 21:24 |
| mutantturkey | k means, kill at tha trun level | 21:24 |
| mutantturkey | networking is in the rc.S runlevel | 21:24 |
| mutantturkey | so looks like cloud-init's 4 things aren't gettng run at startp | 21:24 |
| blackboxsw | that sounds like reasonable failure path ;) | 21:24 |
| mutantturkey | though clearly something is running because cloud-init does LOG | 21:25 |
| mutantturkey | ooff | 21:26 |
| blackboxsw | mutantturkey: cat /run/cloud-init/status.json that'll tell you what stages did run (on newer cloud-init you can run "cloud-init analyze show") | 21:26 |
| Odd_Bloke | blackboxsw: Note that trusty is upstart, not SysV. | 21:28 |
| blackboxsw | oopsie, right-o though our collection of init scripts mentioned here https://github.com/canonical/cloud-init/tree/master/sysvinit/debian should be all-inclusive of what we expect to see on debian right | 21:29 |
| mutantturkey | S12networking | 21:29 |
| mutantturkey | that's so whack | 21:29 |
| Odd_Bloke | blackboxsw: I wouldn't assume so (particularly as those haven't been touched in 7 years). | 21:30 |
| mutantturkey | well that makes non sense | 21:37 |
| mutantturkey | let me jump into #debian and seewhatup | 21:37 |
| mutantturkey | blackboxsw: alright,here we go. it's at runelevel S, which is like single user mode, happens before evreything else | 21:41 |
| mutantturkey | that's before _all_ the cloud service bits run | 21:42 |
| mutantturkey | which well makes since to me because other services might try to bind to a specifc IP etc and fail if networking is not up | 21:42 |
| mutantturkey | so, givne that, is there a move to make here | 21:45 |
| mutantturkey | so i think "cloud-init writes config files, but doesnt run ifup or the like to configure networking after then config is changed' | 21:48 |
| mutantturkey | bso my simple thought is another init file called /restart-networking-hack that runs on runelevel 6 that reruns ifup | 21:52 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!