=== aaa is now known as Guest6095 [09:00] maybe this should've been a bug report https://github.com/canonical/cloud-init/commit/0755cff078d5931e1d8e151bdcb84afb92bc0f02#commitcomment-121577820 [09:00] -ubottu:#cloud-init- Commit 0755cff in canonical/cloud-init "cc_final_message: don't create directories when writing boot-finished (#445)" === lesbraz is now known as sbtaz === sbtaz is now known as sbraz [19:21] hey folks, I am a bit puzzled by the behavior of my azure instance, I push this cloud-init file to the machine: https://paste.centos.org/view/f640db8d before I encrypt the box, so far so good, the machine is deployed, the cloud-init config happens and the encryption kicks in, up until this point things work the way they are supposed to === Sammitch_ is now known as Sammitch [19:22] however, if I fully stop the machine (deallocate), when the machine starts again, it recreates the testvg-testlv entry in /etc/fstab. isn't this part of the config supposed to happen only once? or the mount part is per-boot......mmmmm, I can check that. [19:25] well, documentation says once per instance for the mounts section [19:25] .... interesting [19:26] esv: what do you mean *recreates*? does the cloud-init.log show it running twice? [19:28] esv: also when you restart the machine does it get the *sam* instance id or a new instance id? [19:28] s/*sam*/*same*/ [19:29] azure disk encryption replaces the /dev/mapper/testvg-testlv /data with the /dev/mapper entry of the entry in /etc/crypttab, but after the machine comes up from unallocated state, the testvg-testlv entry is back in fstab [19:31] I am pretty sure is the same id, as I deployed another image w/o encryption and has the same number of entries in /var/lib/cloud/instances and the 1st one is the same id and timestamp. [19:31] I am reproducing the issue again [19:31] so what does /var/log/cloud-init.log show? (assuming debugging is enabled) [19:32] it will show what each cloud-init module is doing [19:32] guess I need to check that too, let me kick another run and will dive there. [19:33] esv: cloud-init analyze show # will also show what modules ran in a slightly easier format to read. If a config module is skipped you'll see "->config-mounts previously run" or something [19:33] will check. [19:33] analyze show shows you per-boot whether modules are activated on that particular boot [19:34] thnx [19:34] I've got to get used to those commands, I'm old fashioned and used to trawling through logfiles lol [19:34] no prob: that said Azure shows a couple of extra boot "stages" in analyze output if I recall due to some of the early boot preprovisioning system environments. [19:35] minimal, me too. [19:35] minimal: old-fashined gets us to the details anyway... we need to improve that tool so it's formatted in a simple machine readable format too [19:37] how does "analyze show" determine a new boot, that's always puzzled me (always ~ 10-18 months maybe) [19:37] perhaps we need c-iGPT to tell us what's wrong? ;-) [19:44] heh, esv : cloud-init analyze show just parses /var/log/cloud-init.log (or any cloud-init log provided to it) looking for the anchor messages for known cloud-init boot stages of the format: Cloud-init v. running 'init-local' at XXX. [19:44] so, I ran : cloud-init analyze show | egrep -n "Boot|mount" [19:44] here are the outputs [19:44] it's a simple tool that collects start/end timing logs from within our very noisy log to capture time offsets for various modules and boot stages [19:44] this is the encrypted machine: https://paste.centos.org/view/20383656 [19:45] c-GPT can now mine those answers for the future :) [19:46] here is the non-encrypted: https://paste.centos.org/view/e5d61867 [19:46] as you can tell, upto Boot #4 they're the same [19:47] esv: so encrypted machine: something changed on boot record 16 that re-allowed cc_mounts to run. Likely an instance-id change. `egrep -i 'Cloud-init|previous iid' /var/log/cloud-init.log` to see when instance id's changed [19:48] if any cloud changes instance-id in metadata cloud-init re-runs all modules as if it's a fresh (green-field) deployment [19:48] but on the non-encrypted one I see mount module running twice, on the last boot, I stopped the machine completely [19:50] blackboxsw, i get it... I love c-i, wish I knew of it much earlier. I did a lousy job trying to implement something a similar concept @ job-2 (much more limited version) [19:51] we get a lot of help making cloud-init better from the broad community adoption here from different distros and clouds... so many hands make light work (and give us a better perspective on what lousy stuff just doesn't fly in other distros/clouds etc) [19:52] :) [20:02] I was getting too much info, I switched the command to this: egrep "Cloud-init.*running 'init-local|previous iid" /var/log/cloud-init.log [20:02] here the output: https://paste.centos.org/view/147521b5 [20:08] I've deployed a new server from the same image, it has the same cloud-init beginning as the previous 2. [20:14] esv: not sure from the looks of things. 2 instance id changes which accounts for re-running mount again in this cases. probably needs a bit more debugging/triage to see what cloud-init is doing across that specific reboot if instance-id is staying the same