| === shardy_afk is now known as shardy | ||
| === evilissimo|afk is now known as evilissimo | ||
| === hatchetation_ is now known as hatchetation | ||
| smoser | mnaser, mgagne a couple things: | 14:45 |
|---|---|---|
| smoser | a.) kernel resize of online disk is orders of magnitude faster with newer kernels . that got magically fast ~ 3.5 i think. | 14:47 |
| smoser | b.) backgrounding the resize ends up not being all that useful on the older kernels. It should background, but it is extremely IO intensive and halts most other things. it should work correclty, but everything will be slow. | 14:47 |
| mnaser | smoser: i notied that, also it doesnt seem like it background cloud-init .. if nothing else in cloud-init is setup to run, it won't really be backgrounded | 14:48 |
| smoser | it shoudl bacground | 14:50 |
| smoser | mnaser, is the call to util.fork_cb just broken ? | 14:51 |
| smoser | it looks right | 14:52 |
| mnaser | smoser: so what was happening is that it was saying "Backgrounding resize" but it was still blocking | 14:52 |
| smoser | hm.. i really doubt its broken. | 14:53 |
| mnaser | and the proof to that even more is that the "part" that does resizing would say "took 180 seonds" let's say and it was blocked there | 14:53 |
| smoser | hm.. | 14:53 |
| mnaser | but this is cloud-init of CentOS 6 | 14:53 |
| mnaser | so maybe that's not been tested | 14:53 |
| mnaser | also i recall | 14:53 |
| smoser | i dont know what is in centos 6. i'm just looking at trunk | 14:53 |
| smoser | but it *is* crazy io intensive | 14:53 |
| mnaser | there was an error | 14:53 |
| smoser | and so is boot | 14:53 |
| smoser | so originally when i did this, i found it to be almost useless. | 14:53 |
| mnaser | with the callback being a NoneType or something | 14:53 |
| smoser | hm.. | 14:54 |
| smoser | let me look at that again | 14:54 |
| mnaser | also this is 10x ssd drives in raid-10 so it's pretty fast | 14:55 |
| smoser | you're right | 14:56 |
| smoser | its busted | 14:56 |
| mnaser | smoser: Jul 7 12:34:20 localhost [CLOUDINIT] util.py[DEBUG]: backgrounded Resizing took 41.487 seconds | 14:56 |
| mnaser | Jul 7 12:34:20 localhost [CLOUDINIT] util.py[DEBUG]: cloud-init mode 'init' took 41.799 seconds (41.80) | 14:56 |
| mnaser | and then it continues afterwards | 14:56 |
| smoser | :-( | 14:56 |
| smoser | and i broke it when i added the util.log_ | 14:56 |
| smoser | open a bug ? | 14:56 |
| mnaser | perhaps | 14:56 |
| mnaser | sure | 14:56 |
| mnaser | smoser: https://bugs.launchpad.net/cloud-init/+bug/1338614 | 15:01 |
| smoser | mnaser, try: http://paste.ubuntu.com/7760542/ ? | 15:06 |
| smoser | are you able to easily try that ? | 15:06 |
| mnaser | hm | 15:06 |
| mnaser | i think i can spin up a new vm with cloud config disabling resize | 15:07 |
| mnaser | then patch it and run the module manually | 15:07 |
| mnaser | i assume if i run it manually it should background as well | 15:07 |
| mnaser | heck, let me try to patch one that is already running because it's failing to fork resizefs anyways | 15:08 |
| smoser | it should work, yea. | 15:10 |
| smoser | mnaser, thank you. | 15:10 |
| === evilissimo is now known as evilissimo|afk | ||
| mnaser | smoser: fyi in the cc_resizefs.py file | 15:11 |
| mnaser | 'args': (resize_cmd, log)) should be 'args': (resize_cmd, log)} | 15:11 |
| smoser | yeah | 15:12 |
| mnaser | doesnt give that warning anymore... now to test if it's actually properly backgrounding will take a bit longer | 15:12 |
| mnaser | but let me do it, solving this will be very usefu | 15:12 |
| smoser | mnaser, it was calling fork_cb completely wrong. | 15:14 |
| smoser | so i'm trying to just fix how it calls it, by adding a fork_cb_kwargs and calling that one, and making fork_cb backwards compat and call the fork_cb_kwargs. | 15:14 |
| smoser | my concern is that fork_cb might not be completley backwards compat | 15:14 |
| mnaser | and maybe other parts thats rely on it might be broken now | 15:16 |
| mnaser | right so just made a new vm that was not resized automatically | 15:16 |
| mnaser | now i'll do the patch for it, delete all the cloud init stuff, reboot and *hopefully* it'll background | 15:17 |
| mnaser | or should i just run it manually in the shell after patching and it'll be okay, smoser ? | 15:17 |
| smoser | rm -Rf /var/lib/cloud | 15:18 |
| smoser | and reboot | 15:18 |
| smoser | and see | 15:18 |
| smoser | that'll be more complete at least | 15:18 |
| smoser | there is only one other caller though | 15:19 |
| smoser | so i could just patch it too | 15:19 |
| mnaser | alright so its patched up now | 15:19 |
| mnaser | rebooting | 15:19 |
| mnaser | its only going to do a resize from 20gb to 40gb but i think it'll still show | 15:20 |
| mnaser | oh, it didnt do the resize again beacuse it read the user data (configdrive) and that says not too | 15:21 |
| mnaser | hm | 15:21 |
| smoser | ah. rihgt. | 15:22 |
| smoser | and you can't easily override that. | 15:22 |
| mnaser | Jul 7 15:22:03 resize-test [CLOUDINIT] cc_resizefs.py[DEBUG]: Skipping module named resizefs, resizing disabled | 15:22 |
| mnaser | even if i force it to run | 15:22 |
| mnaser | from the shell | 15:23 |
| smoser | well if you tell it to run it still reads config | 15:23 |
| mnaser | i guess what ill do is | 15:23 |
| mnaser | take a snapshot of this server | 15:23 |
| mnaser | and boot another one | 15:23 |
| smoser | mnaser, just put 'resize_root = NOBLOCK' | 15:24 |
| smoser | right before that | 15:24 |
| smoser | ie, force it on | 15:24 |
| smoser | actually.. | 15:24 |
| mnaser | before where exactly? | 15:24 |
| smoser | if you run it manually you *should* be able to pass it in | 15:24 |
| mnaser | cloud-init --debug single -n resizefs | 15:24 |
| mnaser | this didnt run it manually | 15:24 |
| smoser | cloud-init --debug single -n resizefs noblock | 15:25 |
| smoser | try that | 15:25 |
| mnaser | i'll take a snapshot and boot from it a much larger machien so we have a feel of what it's like | 15:25 |
| mnaser | so if anything else is disturbing, we'll see it | 15:25 |
| smoser | mnaser, it is crazy slow. | 15:26 |
| smoser | i have done resize on spinning disk from 2G to 2TB | 15:26 |
| mnaser | oh ill make 20G to 80G resize | 15:26 |
| smoser | and it took > 1 our i think | 15:26 |
| mnaser | yeah i know that pain :P | 15:26 |
| smoser | 1 hour | 15:26 |
| mnaser | ssd, 20G to 1.2TB was 10 minutes | 15:26 |
| mnaser | still al ot | 15:26 |
| smoser | yeah. | 15:26 |
| smoser | and it is crazy magic fast in 3.5 | 15:26 |
| smoser | its 1000x faster | 15:26 |
| mnaser | oh well, centos :-( | 15:27 |
| smoser | yeah | 15:27 |
| mnaser | alright, got snapshot, booting it now | 15:27 |
| mnaser | smoser | 15:34 |
| mnaser | Jul 7 15:31:57 resize-test [CLOUDINIT] util.py[WARNING]: Failed forking and calling callback log_time | 15:34 |
| mnaser | Jul 7 15:31:57 resize-test [CLOUDINIT] util.py[DEBUG]: Failed forking and calling callback log_time#012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 224, in fork_cb_kwargs#012 child_cb(*args, **kwargs)#012TypeError: log_time() argument after * must be a sequence, not NoneType | 15:34 |
| smoser | boo. | 15:37 |
| smoser | mnaser, http://paste.ubuntu.com/7760686/ | 15:38 |
| smoser | try that | 15:38 |
| smoser | and i apologize | 15:38 |
| smoser | i wondered about that | 15:38 |
| mnaser | no worries | 15:38 |
| mnaser | now i can easily rename and reboot :p | 15:38 |
| mnaser | btw smoser | 15:40 |
| mnaser | backgrounded Resizing = Backgrounded resizing | 15:41 |
| mnaser | cannot imagine how much that bothered me in logs :-P | 15:41 |
| smoser | ? | 15:41 |
| smoser | oh. i see. capitalize :) | 15:42 |
| mnaser | yes :p | 15:42 |
| smoser | well, the 'R' makes sense. | 15:42 |
| smoser | sort of | 15:42 |
| smoser | in my world | 15:42 |
| smoser | as the 'Resizing' in the non-backgrounded path. | 15:42 |
| mnaser | smoser: really silly | 15:42 |
| mnaser | but | 15:42 |
| smoser | (so grepping for 'Resizing' would get either) | 15:42 |
| mnaser | why not just do this | 15:42 |
| mnaser | def fork_cb_kwargs(child_cb, args=[], kwargs={}): | 15:42 |
| smoser | mnaser, generally, http://satyajit.ranjeev.in/2012/01/12/python--dangerous-default-value-as-argument.html | 15:46 |
| mnaser | aah | 15:47 |
| mnaser | that's good to know | 15:47 |
| smoser | https://docs.python.org/2/tutorial/controlflow.html#default-argument-values | 15:47 |
| mnaser | rebooting | 15:47 |
| smoser | see the 'important warning' there | 15:47 |
| mnaser | always learning something new | 15:48 |
| mnaser | Jul 7 15:48:46 resize-test2 [CLOUDINIT] util.py[WARNING]: Failed forking and calling callback log_time | 15:49 |
| mnaser | Jul 7 15:48:46 resize-test2 [CLOUDINIT] util.py[DEBUG]: Failed forking and calling callback log_time#012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 229, in fork_cb_kwargs#012 child_cb(*args, **kwargs)#012 File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1830, in log_time#012 ret = func(*args, **kwargs)#012TypeError: 'str' object is not callable | 15:49 |
| mnaser | smoser ^ | 15:49 |
| mnaser | tho | 15:49 |
| mnaser | i think i should ahv esaid this earlier | 15:49 |
| mnaser | running cloud-init 0.7.4 not trunk | 15:50 |
| mnaser | so maybe this has to do with it | 15:50 |
| smoser | mnaser, i'll stop bothering you and i'll just make sure it works on trunk before asking again. | 16:01 |
| smoser | k? | 16:01 |
| mnaser | smoser: no problem, thank you so much :) | 16:34 |
| === harlowja_away is now known as harlowja | ||
| harlowja | smoser u might want to respond to http://lists.openstack.org/pipermail/openstack-dev/2014-July/039514.html :) | 19:04 |
| harlowja | see last line ;) | 19:04 |
| cloudnoob | harlowja: I emailed you about cloud-init-dev over the weekend? | 19:11 |
| cloudnoob | I will paste my email into irc :) | 19:11 |
| harlowja | cloudnoob howdy | 19:11 |
| cloudnoob | It looks like the default configuration of cloud-init on Ubuntu 12.04 LTS causes a new RSA key & fingerprint to be generated when an existing (root) EBS volume is attached to a new instance[1]. I'd like to avoid this behavior on some of my instances. After some poking around, I think removing 'ssh' from cloud_init_modules list in /etc/cloud/cloud.cfg will prevent this from happening[2]. Can a cloud-init-dev@'er confirm this is t | 19:12 |
| cloudnoob | oh hmmm | 19:12 |
| cloudnoob | looks like I hit a limit | 19:12 |
| cloudnoob | 1/ It looks like the default configuration of cloud-init on Ubuntu 12.04 LTS causes a new RSA key & fingerprint to be generated when an existing (root) EBS volume is attached to a new instance | 19:12 |
| harlowja | that would seem fine to do | 19:12 |
| cloudnoob | 2/ I'd like to avoid this behavior on some of my instances. | 19:12 |
| harlowja | take ssh module out | 19:12 |
| cloudnoob | 3/ After some poking around, I think removing 'ssh' from cloud_init_modules list in /etc/cloud/cloud.cfg will prevent this from happening | 19:12 |
| cloudnoob | 4/ Can a cloud-init-dev@'er confirm this is the case, and that this is a safe/secure thing to do for existing systems which have already been through once-per-instance setup? I'm also open to advice or suggestions on better approaches. | 19:12 |
| cloudnoob | see http://serverfault.com/questions/294039/rsa-fingerprint-changed-after-moving-ebs-and-eip-to-new-instance for a similar writeup | 19:13 |
| cloudnoob | also my understanding is based on the assumption that 'ssh' in this list corresponds to executing the handler defined in 'cc_ssh.py'. | 19:13 |
| harlowja | cloudnoob i think u can also place ssh_genkeytypes: [] in userdata | 19:13 |
| harlowja | to avoid it doing any extra key creation | 19:13 |
| cloudnoob | ok | 19:14 |
| harlowja | or adjust it in /etc/cloud/cloud.cfg | 19:14 |
| cloudnoob | is that better than removing from cloud_init_modules? | 19:14 |
| harlowja | disabling ssh makes it impossible for any user keys to come in, if they ever want | 19:15 |
| harlowja | i think u are hitting http://bazaar.launchpad.net/~cloud-init-dev/cloud-init/trunk/view/head:/cloudinit/config/cc_ssh.py#L91 which the ssh_genkeytypes: [] should just stop | 19:15 |
| cloudnoob | ah | 19:15 |
| harlowja | aka, no more generation, but if people provide them that code path will work | 19:15 |
| cloudnoob | okay cool | 19:15 |
| cloudnoob | I'll try it | 19:15 |
| cloudnoob | there are existing keys so yeah, just don't want to overwrite them | 19:16 |
| cloudnoob | thanks for the suggestions | 19:16 |
| harlowja | np | 19:16 |
| cloudnoob | and thanks for the software of course | 19:16 |
| cloudnoob | teh useful | 19:16 |
| harlowja | np | 19:16 |
| harlowja | thx for using it :-P | 19:17 |
| smoser | harlowja, thanks for the heads up. responded | 19:29 |
| harlowja | smoser cool | 19:29 |
| harlowja | i'm not touching that question ;) | 19:30 |
| harlowja | u can, haha | 19:30 |
| harlowja | *not the ipv6 question, the lets get rid of metadata service | 19:36 |
| harmw | can't we just ditch ip6? | 19:39 |
| harmw | far easier | 19:39 |
| harmw | everything just works with ip4 | 19:39 |
| harmw | lets keep it that way | 19:39 |
| ijw | yo | 19:44 |
| ijw | Since I'm here, I wanted to ask: has anyone thought about a generic format for network information (rather than the /etc/network/interfaces file that Openstack likes to template, badly, and then inject)? | 19:45 |
| harlowja | ijw yes | 19:52 |
| ijw | How much thinking so far? | 19:52 |
| harlowja | ijw http://bazaar.launchpad.net/~harlowja/cloud-init/better-todo/view/head:/TODO.rst (this is updated with this, although its been thought of for a long time, ha) | 19:53 |
| harlowja | smoser u should merge that branch :) | 19:53 |
| ijw | I have an MTU problem that is really not going to fit in with the current mechanisms | 19:53 |
| harlowja | ijw i don't think its cloud-init that would have the biggest problem with changing, its the other things that write the format :) | 19:53 |
| ijw | Win7 doesn't advertise for MTU on DHCP, and Linux has Opinions on how big the MTU can be | 19:53 |
| harlowja | ijw https://fedorahosted.org/netcf/ has been discussed (although its xml, eck, haha) | 19:54 |
| ijw | Given that the in-page example is a bridge it's more complex than we strictly need anyway | 19:55 |
| harlowja | sure | 19:55 |
| ijw | (also, you can't specify data by eth0/1/2 since they're assigned by the OS) | 19:55 |
| harlowja | ijw the issue is that other surronding projects need to get away from writing this format also, this imho is the harder part :) | 19:55 |
| harlowja | *neutron/nova in openstack | 19:55 |
| harlowja | ijw ideally someone really needs to own that format change and push it through the various openstack projects | 19:56 |
| harlowja | the cloud-init work i don't think would be that much | 19:56 |
| harlowja | *at least to do the basics | 19:57 |
| smoser | harlowja, we have a fail safe datasource. | 19:57 |
| smoser | ijw, yeah, mtu sucks. | 19:57 |
| harlowja | smoser damn gotta change that in the todo, no longer a todo then, ha | 19:57 |
| smoser | ijw, from my perspective, it needs to be specified by the "provider" in one way or another. | 19:58 |
| ijw | It's not even a format change - it's an additional file on the web server or config drive that could happily be ignored by anything that didn't know about it | 19:58 |
| smoser | dhcp is reasonable for dhcp . but ideally it'd be available if not dhcp. | 19:58 |
| harlowja | ijw sure, just write "network.xml" (or something) | 19:58 |
| ijw | And I think if it had interface, optionally with VLAN/v4(s)/v6(es)/mtu that would be sufficient | 19:58 |
| smoser | this advertising needs to be more part of the network imo | 19:58 |
| harlowja | and cloud-init will use it if it exists (or if not look for the old way) | 19:59 |
| smoser | but maybe not. | 19:59 |
| smoser | it is very problematic | 19:59 |
| smoser | and we're hitting it here too. | 19:59 |
| ijw | smoser: DHCP in either the v4 or v6 forms has various issues advertising MTU. The client needs to know, and v6's spec is written with a pedantic respect to the fact that ethernet doesn't standardise packet sizes >1500 bytes (so making bigger MTUs is hard) | 20:00 |
| ijw | Assuming YAML wouldn't make people cry, the spec could be very straightforward, in fact | 20:01 |
| * ijw finds an etherpad | 20:01 | |
| harlowja | :), bb, meeting | 20:02 |
| smoser | it just seems really odd that this isn't sovled | 20:02 |
| smoser | harlowja, merged. dont say i never did anything for you :) | 20:03 |
| === mnaser is now known as Perk | ||
| === Perk is now known as mnaser | ||
| === harlowja is now known as harlowja_away | ||
| === gondoi is now known as zz_gondoi | ||
| === harlowja_away is now known as harlowja | ||
| harlowja | smoser woot :-P | 22:08 |
| ijw | By sheer coincidence someone's just talked to be about bonding, too... | 23:06 |
Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!