=== aaa is now known as Guest6095
[09:00] <meena> maybe this should've been a bug report https://github.com/canonical/cloud-init/commit/0755cff078d5931e1d8e151bdcb84afb92bc0f02#commitcomment-121577820
[09:00] -ubottu:#cloud-init- Commit 0755cff in canonical/cloud-init "cc_final_message: don't create directories when writing boot-finished (#445)"
=== lesbraz is now known as sbtaz
=== sbtaz is now known as sbraz
[19:21] <esv> hey folks, I am a bit puzzled by the behavior of my azure instance, I push this cloud-init file to the machine: https://paste.centos.org/view/f640db8d before I encrypt the box, so far so good, the machine is deployed, the cloud-init config happens and the encryption kicks in, up until this point things work the way they are supposed to
=== Sammitch_ is now known as Sammitch
[19:22] <esv> however, if I fully stop the machine (deallocate), when the machine starts again, it recreates the testvg-testlv entry in /etc/fstab. isn't this part of the config supposed to happen only once? or the mount part is per-boot......mmmmm, I can check that.
[19:25] <esv> well, documentation says once per instance for the mounts section
[19:25] <esv> .... interesting
[19:26] <minimal> esv: what do you mean *recreates*? does the cloud-init.log show it running twice?
[19:28] <minimal> esv: also when you restart the machine does it get the *sam* instance id or a new instance id?
[19:28] <minimal> s/*sam*/*same*/
[19:29] <esv> azure disk encryption replaces the /dev/mapper/testvg-testlv /data with the /dev/mapper entry of the entry in /etc/crypttab, but after the machine comes up from unallocated state, the testvg-testlv entry is back in fstab
[19:31] <esv> I am pretty sure is the same id, as I deployed another image w/o encryption and has the same number of entries in /var/lib/cloud/instances and the 1st one is the same id and timestamp.
[19:31] <esv> I am reproducing the issue again
[19:31] <minimal> so what does /var/log/cloud-init.log show? (assuming debugging is enabled)
[19:32] <minimal> it will show what each cloud-init module is doing
[19:32] <esv> guess I need to check that too, let me kick another run and will dive there. 
[19:33] <blackboxsw> esv: cloud-init analyze show  # will also show what modules ran in a slightly easier format to read. If a config module is skipped you'll see "->config-mounts previously run" or something
[19:33] <esv> will check.
[19:33] <blackboxsw> analyze show shows you per-boot whether modules are activated  on that particular boot
[19:34] <esv> thnx
[19:34] <minimal> I've got to get used to those commands, I'm old fashioned and used to trawling through logfiles lol
[19:34] <blackboxsw> no prob: that said Azure shows a couple of extra boot "stages" in analyze output if I recall due to some of the early boot preprovisioning system environments.
[19:35] <esv> minimal, me too. 
[19:35] <blackboxsw> minimal: old-fashined gets us to the details anyway... we need to improve that tool so it's formatted in a simple machine readable format too
[19:37] <esv> how does "analyze show" determine a new boot, that's always puzzled me (always ~ 10-18 months maybe)
[19:37] <minimal> perhaps we need c-iGPT to tell us what's wrong? ;-)
[19:44] <blackboxsw> heh, esv : cloud-init analyze show just parses /var/log/cloud-init.log (or any cloud-init log provided to it) looking for the anchor messages for known cloud-init boot stages of the format:  Cloud-init v. <VERSION> running 'init-local' at XXX.
[19:44] <esv> so, I ran : cloud-init analyze show | egrep -n "Boot|mount"
[19:44] <esv> here are the outputs
[19:44] <blackboxsw> it's a simple tool that collects start/end timing logs from within our very noisy log to capture time offsets for various modules and boot stages
[19:44] <esv> this is the encrypted machine:  https://paste.centos.org/view/20383656
[19:45] <blackboxsw> c-GPT can now mine those answers for the future :)
[19:46] <esv> here is the non-encrypted: https://paste.centos.org/view/e5d61867
[19:46] <esv> as you can tell, upto Boot #4 they're the same
[19:47] <blackboxsw> esv: so encrypted machine: something changed on boot record 16 that re-allowed cc_mounts to run. Likely an instance-id change.  `egrep -i 'Cloud-init|previous iid' /var/log/cloud-init.log` to see when instance id's changed
[19:48] <blackboxsw> if any cloud changes instance-id in metadata cloud-init re-runs all modules as if it's a fresh (green-field) deployment
[19:48] <esv> but on the non-encrypted one I see mount module running twice, on the last boot, I stopped the machine completely 
[19:50] <esv> blackboxsw, i get it... I love c-i, wish I knew of it much earlier. I did a lousy job trying to implement something a similar concept @ job-2 (much more limited version) 
[19:51] <blackboxsw> we get a lot of help making cloud-init better from the broad community adoption here from different distros and clouds... so many hands make light work (and give us a better perspective on what lousy stuff just doesn't fly in other distros/clouds etc)
[19:52] <blackboxsw> </end_opensource_party_line> :)
[20:02] <esv> I was getting too much info, I switched the command to this: egrep "Cloud-init.*running 'init-local|previous iid" /var/log/cloud-init.log
[20:02] <esv> here the output: https://paste.centos.org/view/147521b5
[20:08] <esv> I've deployed a new server from the same image, it has the same cloud-init beginning as the previous 2.
[20:14] <blackboxsw> esv: not sure from the looks of things. 2 instance id changes which accounts for re-running mount  again in this cases. probably needs a bit more debugging/triage to see what cloud-init is doing across that specific reboot if instance-id is staying the same