[03:26] <blackboxsw> rharper: volley for tomorrow https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/371895
[08:58] <matthiasbaur> Hey, I try to contribute to cloud-init and been told that I have to sign a CLA. As I'm not sure if the company I working for signed one, may I talk to you in private smoser?
[12:20] <smoser> matthiasbaur: talk to powersj .
[12:42] <matthiasbaur> powersj may I talk to you in private?
[18:04] <iliv> hey guys
[18:06] <powersj> o/
[18:08] <iliv> I have run into what probably is a well-known issue by know but new to me, which is if I use root EBS volume that is greater than 250G (e.g. >= 300G) for a Debian Jessie AMI based EC2 instance, that root EBS volume is partitioned with what seems like a default fallback option. Namely, 2 partitions: swap and 8GB root partition. As I undersatnd cloud-init is responsible for doing the partitioning. Is that correct?
[18:19] <rharper> iliv: cloud-init can do partitioning, and I believe certain platforms have some defaults; let me look at the ec2 datasource;   do you know what version of cloud-init is in your image? do you have a /var/log/cloud-init.log from the instance ?
[18:20] <iliv> Cloud-init v. 0.7.7
[18:20] <iliv> this is Debian Jessie AMI
[18:22] <rharper> iliv: look at our code; couldn't init doesn't do any partitioning of the root device (by default) ; it may grow or resize a root partition and it can enable/create swap;
[18:23] <rharper> iliv: typically images will have the root partition be the last partition on the disk, so that cloud-init can grow the last partition size to end of disk adn resize the rootfs as well
[18:23] <rharper> so I wonder if the AMI partition layout is the source of the issue;
[18:25] <iliv> it's kinda weird that it grows it only if ebs root volume size is <=250G
[18:26] <rharper> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=784004
[18:26] <ubot5> Debian bug 784004 in cloud-init "cloud-init growpart module fails to execute due to different growpart version" [Normal,Open]
[18:26] <rharper> could be related
[18:26] <iliv> just to be clear, when using 500G root EBS volume I get 2 partitions: 1 swap and 1 8GB root partition. the remaining disk space is simply unallocated.
[18:27] <rharper> yes, this sounds like the above bug
[18:28] <rharper> well, maybe something else but related, so ebs size of 200G and you get the right size ?
[18:28] <rharper> in any case; the bug mentions a downlevel version of growpart in Debian Jessie which was the source of the root partition not being resized correctly
[18:29] <rharper> there are a couple workaround mentions w.r.t including an updated version of growpart in the image
[18:32] <iliv> > so ebs size of 200G and you get the right size ? // up to 250G
[18:32] <iliv> I tested 500G, 400G, 300G, 250G and 200G
[18:33] <iliv> if it's <=250G I get a 250G ext4 root partition
[18:33] <iliv> if it's >= 300G, I get only 8GB ext4 root partition
[18:34] <rharper> could be a new bug then; including the /var/log/cloud-init.log will be help, whether you file a downstream or an upstream bug;  if you use newer debian AMI (with newer cloud-inti) do you see the same issue ? or is it jessie only ?
[18:35] <iliv> https://dpaste.de/Zvbt/raw
[18:36] <iliv> I tested this with the Jessie AMI only
[18:37] <rharper> https://github.com/andsens/bootstrap-vz/issues/359
[18:37] <rharper> looks like that issue
[18:38] <rharper> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868875
[18:38] <ubot5> Debian bug 868875 in cloud-utils "cloud-init growpart fails to grow root volumes larger than 256GiB" [Normal,Open]
[18:38] <rharper> already filed in debian
[18:38] <rharper> and it looks related to the util-linux version in Jessie;
[18:39] <iliv> you do all the dirty work for me :D
[18:39] <iliv> thanks, appreciate it!
[18:39] <rharper> np
[18:42] <iliv> "No longer marked as found in versions cloud-init/0.7.7~bzr1156-1~bpo8+1."
[18:43] <iliv> does that mean this bug was fixed in that version of cloud-init?
[18:49] <rharper> doubtful
[18:49] <rharper> it was a growpart issue (which cloud-init executes) and is part of the cloud-guest-utils package
[18:53] <iliv> so, nothing cloud-init can do here as long as we're stuck with this particular AMI?
[18:53] <iliv> I saw a reference to cloud-initramfs-growroot in one of the bug reports
[18:53] <iliv> not sure if that could help
[18:54] <iliv> but the theory goes that the growing would happen very early in the process before the filesystem is mounted
[18:54] <iliv> so on the surface sounds like an option to explore
[18:54] <rharper> you can disable growpart via cloud-config and then perform it manually
[18:54] <rharper> however, if the issue is related to the tools in the OS (util-linux package) being downlevel, then you'd somehow need to get the newer version of the tools and use them instead
[18:54] <iliv> oh, I tried that but it failed
[18:58] <rharper> https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1244662
[18:58] <ubot5> Launchpad bug 1244662 in util-linux (Ubuntu) "growpart reverses changes because of partx failure" [Medium,Fix released]
[18:58] <rharper> this was fixed in newer util-linux
[18:58] <rharper> so, likely need t pull in a newer util-linux into the image and then try the growpart again manually
[19:05] <iliv> Installed: 2.25.2-6
[19:05] <iliv> "This bug was fixed in the package util-linux - 2.20.1-5.1ubuntu12"
[19:06] <iliv> however, that is ubuntu package
[19:07] <rharper> yes, but 2.25 is > 2.20
[19:08] <rharper> it was forwarded to upstream, I can see if we still have delta, but I suspect it was accepted upstream
[19:10] <iliv> well, if that's the case I saw exactly the same error message as shown in that Launchpad bug report when I tried to run growpart /dev/xvda 2
[19:11] <iliv> https://dpaste.de/NiFr/raw
[19:17] <rharper> https://github.com/karelzak/util-linux/commit/84ac311212c06963d616b1b9a40644842f9adabd ; landed upstream and released in 2.24,  https://lwn.net/Articles/581276/
[19:50] <blackboxsw> tribaal: if you have time next week, cloud-init SRU v 19.2-24-ge7881d5c-0ubuntu1 got re-queued for Xenial/Bionic and Disco with the Exoscale fixes for cloud_config_modules which we published in tip this week to Eoan.   If you get a chance to enable <series>-proposed on an Xenial or Bionic instance you have, then you can sudo cloud-init clean --logs;  apt-get remove --purge cloud-init; apt-get install cloud-init
[19:51] <blackboxsw> #version 19.2-24 sudo reboot   to test that Exoscale comes up properly
[19:51] <blackboxsw> I'll send an email about that too.
[20:11] <blackboxsw> rharper: thanks on SRU, minor comments and please land https://github.com/cloud-init/ubuntu-sru/pull/36/files and https://github.com/cloud-init/ubuntu-sru/pull/35/files .   Or your could file a 3rd PR that just updates all sru-templates with the new "local" template variables
[20:12] <blackboxsw> I'm adding it the the ec2 manual run I've kicked off now.
[20:25] <rharper> blackboxsw: ill check, I'll likely put up a separate PR for the template file
[20:33] <blackboxsw> +1 rharper