| === aaa is now known as Guest6095 | ||
| meena | maybe this should've been a bug report https://github.com/canonical/cloud-init/commit/0755cff078d5931e1d8e151bdcb84afb92bc0f02#commitcomment-121577820 | 09:00 |
|---|---|---|
| -ubottu:#cloud-init- Commit 0755cff in canonical/cloud-init "cc_final_message: don't create directories when writing boot-finished (#445)" | 09:00 | |
| === lesbraz is now known as sbtaz | ||
| === sbtaz is now known as sbraz | ||
| esv | hey folks, I am a bit puzzled by the behavior of my azure instance, I push this cloud-init file to the machine: https://paste.centos.org/view/f640db8d before I encrypt the box, so far so good, the machine is deployed, the cloud-init config happens and the encryption kicks in, up until this point things work the way they are supposed to | 19:21 |
| === Sammitch_ is now known as Sammitch | ||
| esv | however, if I fully stop the machine (deallocate), when the machine starts again, it recreates the testvg-testlv entry in /etc/fstab. isn't this part of the config supposed to happen only once? or the mount part is per-boot......mmmmm, I can check that. | 19:22 |
| esv | well, documentation says once per instance for the mounts section | 19:25 |
| esv | .... interesting | 19:25 |
| minimal | esv: what do you mean *recreates*? does the cloud-init.log show it running twice? | 19:26 |
| minimal | esv: also when you restart the machine does it get the *sam* instance id or a new instance id? | 19:28 |
| minimal | s/*sam*/*same*/ | 19:28 |
| esv | azure disk encryption replaces the /dev/mapper/testvg-testlv /data with the /dev/mapper entry of the entry in /etc/crypttab, but after the machine comes up from unallocated state, the testvg-testlv entry is back in fstab | 19:29 |
| esv | I am pretty sure is the same id, as I deployed another image w/o encryption and has the same number of entries in /var/lib/cloud/instances and the 1st one is the same id and timestamp. | 19:31 |
| esv | I am reproducing the issue again | 19:31 |
| minimal | so what does /var/log/cloud-init.log show? (assuming debugging is enabled) | 19:31 |
| minimal | it will show what each cloud-init module is doing | 19:32 |
| esv | guess I need to check that too, let me kick another run and will dive there. | 19:32 |
| blackboxsw | esv: cloud-init analyze show # will also show what modules ran in a slightly easier format to read. If a config module is skipped you'll see "->config-mounts previously run" or something | 19:33 |
| esv | will check. | 19:33 |
| blackboxsw | analyze show shows you per-boot whether modules are activated on that particular boot | 19:33 |
| esv | thnx | 19:34 |
| minimal | I've got to get used to those commands, I'm old fashioned and used to trawling through logfiles lol | 19:34 |
| blackboxsw | no prob: that said Azure shows a couple of extra boot "stages" in analyze output if I recall due to some of the early boot preprovisioning system environments. | 19:34 |
| esv | minimal, me too. | 19:35 |
| blackboxsw | minimal: old-fashined gets us to the details anyway... we need to improve that tool so it's formatted in a simple machine readable format too | 19:35 |
| esv | how does "analyze show" determine a new boot, that's always puzzled me (always ~ 10-18 months maybe) | 19:37 |
| minimal | perhaps we need c-iGPT to tell us what's wrong? ;-) | 19:37 |
| blackboxsw | heh, esv : cloud-init analyze show just parses /var/log/cloud-init.log (or any cloud-init log provided to it) looking for the anchor messages for known cloud-init boot stages of the format: Cloud-init v. <VERSION> running 'init-local' at XXX. | 19:44 |
| esv | so, I ran : cloud-init analyze show | egrep -n "Boot|mount" | 19:44 |
| esv | here are the outputs | 19:44 |
| blackboxsw | it's a simple tool that collects start/end timing logs from within our very noisy log to capture time offsets for various modules and boot stages | 19:44 |
| esv | this is the encrypted machine: https://paste.centos.org/view/20383656 | 19:44 |
| blackboxsw | c-GPT can now mine those answers for the future :) | 19:45 |
| esv | here is the non-encrypted: https://paste.centos.org/view/e5d61867 | 19:46 |
| esv | as you can tell, upto Boot #4 they're the same | 19:46 |
| blackboxsw | esv: so encrypted machine: something changed on boot record 16 that re-allowed cc_mounts to run. Likely an instance-id change. `egrep -i 'Cloud-init|previous iid' /var/log/cloud-init.log` to see when instance id's changed | 19:47 |
| blackboxsw | if any cloud changes instance-id in metadata cloud-init re-runs all modules as if it's a fresh (green-field) deployment | 19:48 |
| esv | but on the non-encrypted one I see mount module running twice, on the last boot, I stopped the machine completely | 19:48 |
| esv | blackboxsw, i get it... I love c-i, wish I knew of it much earlier. I did a lousy job trying to implement something a similar concept @ job-2 (much more limited version) | 19:50 |
| blackboxsw | we get a lot of help making cloud-init better from the broad community adoption here from different distros and clouds... so many hands make light work (and give us a better perspective on what lousy stuff just doesn't fly in other distros/clouds etc) | 19:51 |
| blackboxsw | </end_opensource_party_line> :) | 19:52 |
| esv | I was getting too much info, I switched the command to this: egrep "Cloud-init.*running 'init-local|previous iid" /var/log/cloud-init.log | 20:02 |
| esv | here the output: https://paste.centos.org/view/147521b5 | 20:02 |
| esv | I've deployed a new server from the same image, it has the same cloud-init beginning as the previous 2. | 20:08 |
| blackboxsw | esv: not sure from the looks of things. 2 instance id changes which accounts for re-running mount again in this cases. probably needs a bit more debugging/triage to see what cloud-init is doing across that specific reboot if instance-id is staying the same | 20:14 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!