| blackboxsw | rharper: volley for tomorrow https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/371895 | 03:26 |
|---|---|---|
| matthiasbaur | Hey, I try to contribute to cloud-init and been told that I have to sign a CLA. As I'm not sure if the company I working for signed one, may I talk to you in private smoser? | 08:58 |
| smoser | matthiasbaur: talk to powersj . | 12:20 |
| matthiasbaur | powersj may I talk to you in private? | 12:42 |
| iliv | hey guys | 18:04 |
| powersj | o/ | 18:06 |
| iliv | I have run into what probably is a well-known issue by know but new to me, which is if I use root EBS volume that is greater than 250G (e.g. >= 300G) for a Debian Jessie AMI based EC2 instance, that root EBS volume is partitioned with what seems like a default fallback option. Namely, 2 partitions: swap and 8GB root partition. As I undersatnd cloud-init is responsible for doing the partitioning. Is that correct? | 18:08 |
| rharper | iliv: cloud-init can do partitioning, and I believe certain platforms have some defaults; let me look at the ec2 datasource; do you know what version of cloud-init is in your image? do you have a /var/log/cloud-init.log from the instance ? | 18:19 |
| iliv | Cloud-init v. 0.7.7 | 18:20 |
| iliv | this is Debian Jessie AMI | 18:20 |
| rharper | iliv: look at our code; couldn't init doesn't do any partitioning of the root device (by default) ; it may grow or resize a root partition and it can enable/create swap; | 18:22 |
| rharper | iliv: typically images will have the root partition be the last partition on the disk, so that cloud-init can grow the last partition size to end of disk adn resize the rootfs as well | 18:23 |
| rharper | so I wonder if the AMI partition layout is the source of the issue; | 18:23 |
| iliv | it's kinda weird that it grows it only if ebs root volume size is <=250G | 18:25 |
| rharper | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=784004 | 18:26 |
| ubot5 | Debian bug 784004 in cloud-init "cloud-init growpart module fails to execute due to different growpart version" [Normal,Open] | 18:26 |
| rharper | could be related | 18:26 |
| iliv | just to be clear, when using 500G root EBS volume I get 2 partitions: 1 swap and 1 8GB root partition. the remaining disk space is simply unallocated. | 18:26 |
| rharper | yes, this sounds like the above bug | 18:27 |
| rharper | well, maybe something else but related, so ebs size of 200G and you get the right size ? | 18:28 |
| rharper | in any case; the bug mentions a downlevel version of growpart in Debian Jessie which was the source of the root partition not being resized correctly | 18:28 |
| rharper | there are a couple workaround mentions w.r.t including an updated version of growpart in the image | 18:29 |
| iliv | > so ebs size of 200G and you get the right size ? // up to 250G | 18:32 |
| iliv | I tested 500G, 400G, 300G, 250G and 200G | 18:32 |
| iliv | if it's <=250G I get a 250G ext4 root partition | 18:33 |
| iliv | if it's >= 300G, I get only 8GB ext4 root partition | 18:33 |
| rharper | could be a new bug then; including the /var/log/cloud-init.log will be help, whether you file a downstream or an upstream bug; if you use newer debian AMI (with newer cloud-inti) do you see the same issue ? or is it jessie only ? | 18:34 |
| iliv | https://dpaste.de/Zvbt/raw | 18:35 |
| iliv | I tested this with the Jessie AMI only | 18:36 |
| rharper | https://github.com/andsens/bootstrap-vz/issues/359 | 18:37 |
| rharper | looks like that issue | 18:37 |
| rharper | https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=868875 | 18:38 |
| ubot5 | Debian bug 868875 in cloud-utils "cloud-init growpart fails to grow root volumes larger than 256GiB" [Normal,Open] | 18:38 |
| rharper | already filed in debian | 18:38 |
| rharper | and it looks related to the util-linux version in Jessie; | 18:38 |
| iliv | you do all the dirty work for me :D | 18:39 |
| iliv | thanks, appreciate it! | 18:39 |
| rharper | np | 18:39 |
| iliv | "No longer marked as found in versions cloud-init/0.7.7~bzr1156-1~bpo8+1." | 18:42 |
| iliv | does that mean this bug was fixed in that version of cloud-init? | 18:43 |
| rharper | doubtful | 18:49 |
| rharper | it was a growpart issue (which cloud-init executes) and is part of the cloud-guest-utils package | 18:49 |
| iliv | so, nothing cloud-init can do here as long as we're stuck with this particular AMI? | 18:53 |
| iliv | I saw a reference to cloud-initramfs-growroot in one of the bug reports | 18:53 |
| iliv | not sure if that could help | 18:53 |
| iliv | but the theory goes that the growing would happen very early in the process before the filesystem is mounted | 18:54 |
| iliv | so on the surface sounds like an option to explore | 18:54 |
| rharper | you can disable growpart via cloud-config and then perform it manually | 18:54 |
| rharper | however, if the issue is related to the tools in the OS (util-linux package) being downlevel, then you'd somehow need to get the newer version of the tools and use them instead | 18:54 |
| iliv | oh, I tried that but it failed | 18:54 |
| rharper | https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1244662 | 18:58 |
| ubot5 | Launchpad bug 1244662 in util-linux (Ubuntu) "growpart reverses changes because of partx failure" [Medium,Fix released] | 18:58 |
| rharper | this was fixed in newer util-linux | 18:58 |
| rharper | so, likely need t pull in a newer util-linux into the image and then try the growpart again manually | 18:58 |
| iliv | Installed: 2.25.2-6 | 19:05 |
| iliv | "This bug was fixed in the package util-linux - 2.20.1-5.1ubuntu12" | 19:05 |
| iliv | however, that is ubuntu package | 19:06 |
| rharper | yes, but 2.25 is > 2.20 | 19:07 |
| rharper | it was forwarded to upstream, I can see if we still have delta, but I suspect it was accepted upstream | 19:08 |
| iliv | well, if that's the case I saw exactly the same error message as shown in that Launchpad bug report when I tried to run growpart /dev/xvda 2 | 19:10 |
| iliv | https://dpaste.de/NiFr/raw | 19:11 |
| rharper | https://github.com/karelzak/util-linux/commit/84ac311212c06963d616b1b9a40644842f9adabd ; landed upstream and released in 2.24, https://lwn.net/Articles/581276/ | 19:17 |
| blackboxsw | tribaal: if you have time next week, cloud-init SRU v 19.2-24-ge7881d5c-0ubuntu1 got re-queued for Xenial/Bionic and Disco with the Exoscale fixes for cloud_config_modules which we published in tip this week to Eoan. If you get a chance to enable <series>-proposed on an Xenial or Bionic instance you have, then you can sudo cloud-init clean --logs; apt-get remove --purge cloud-init; apt-get install cloud-init | 19:50 |
| blackboxsw | #version 19.2-24 sudo reboot to test that Exoscale comes up properly | 19:51 |
| blackboxsw | I'll send an email about that too. | 19:51 |
| blackboxsw | rharper: thanks on SRU, minor comments and please land https://github.com/cloud-init/ubuntu-sru/pull/36/files and https://github.com/cloud-init/ubuntu-sru/pull/35/files . Or your could file a 3rd PR that just updates all sru-templates with the new "local" template variables | 20:11 |
| blackboxsw | I'm adding it the the ec2 manual run I've kicked off now. | 20:12 |
| rharper | blackboxsw: ill check, I'll likely put up a separate PR for the template file | 20:25 |
| blackboxsw | +1 rharper | 20:33 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!