=== harlowja is now known as harlowja_away === praneshp_ is now known as praneshp [06:56] smoser: I'm to sucky a python programmer to fix the test on my own, so a little help or example would be helpfull yes :) === praneshp_ is now known as praneshp === smoser` is now known as smoser [12:51] JayF, so you're the only one using vendordata on openstack to my knowledge. at least with cloud-init. [13:10] JayF, https://etherpad.openstack.org/p/cloud-init-vendor-data [13:11] harmw, i'll see if i can't help ith that test today. harlowja wrote the other ones there. cloud-init suffers badly from having too many test case mechanisms. [13:55] smoser: I did an update on #1246485 - not sure if the suggested solution from alexius is perfect, but it works in my situation. also affects latest version... [13:57] siert, just fyi, man page for 'hostname' recommends against using 'hostname -f' [13:57] :) [14:04] smoser: hm, ok. but the RHEL install guide is slightly more ambiguous - "either as a fqdn or as a short host name" [14:10] siert, thank you for loking at this. [14:11] while it seems straight forward, i think its probably as much as you'd hope. [14:11] my experience is that any time i touch 'hostname' or '/etc/hosts', something breaks. [14:15] https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1052664 [14:21] the /etc/hostname should always have the short name only. on rhel, when there is no hostname defined in /etc/sysconfig/network, then /etc/hostname is used as a fallback, to set the systems hostname. [14:27] debian/ubuntu have the short name in /etc/hostname and the rest is read from /etc/hosts. [14:31] hm. [14:37] having 'hostname' return a fqdn seems broken to me. [14:39] ubuntu 14.04 returns hostname version 3.15. rhel based systems: net-tools 1.60 and hostname 1.100 (2001-04-14) [14:40] maybe the difference is in the version of 'hostname', let me see... :s [14:40] well, on ubuntu, it returns whatever you set it to. [14:41] hostname [14:41] brickies [14:41] er.. [14:41] $ hostname [14:41] brickies [14:41] $ sudo hostname brickies.foobar.example.com [14:41] $ sudo hostname [14:41] sudo: unable to resolve host brickies.foobar.example.com [14:41] brickies.foobar.example.com [14:42] $ sudo hostname brickies [14:42] sudo: unable to resolve host brickies.foobar.example.com [14:42] $ hostname [14:42] brickies [14:42] and that initially gets set from the content of /etc/hostname [14:43] yeah, by /etc/init/hostname.conf (hostname -b -F) [14:45] right. [14:45] http://manpages.ubuntu.com/manpages/trusty/en/man1/hostname.1.html [14:45] "historically this file was supposed to only contain the hostname and not the full canonical FQDN. ..." [14:46] siert, i know it sounds like i'm paying too much attention here, but i've just been bit many times. [14:46] basically my experience is that anything you do breaks something. [14:51] thanks for you time, I gtg. will look into it tomorrow, maybe there's another way of getting rhel to behave the same as ubuntu when it comes to hostnames. === zz_gondoi is now known as gondoi [16:24] smoser: Is that a bad thing? Good thing? Heh [16:24] We've used vendor_data.json internally for stuff for a while, just now integrating it with cloud-init [16:26] JayF, well, i would not want to break anyone's usage [16:26] and i was proposing changes in consumpiton of it [16:26] even though that consumption i spretty broken at the moment. [16:30] smoser: not sure I grok what you're trying to suggest in that etherpad [16:30] Have you had feature requests along those lines from other folks using vendor_data? [16:31] that is how vendor data works [16:31] in other datasources [16:31] and actually , if there is 'cloud-init' in the network metadata service on openstack, then it probably works like that now. [16:32] (well, for the 'dict' case) [16:32] ie: [16:32] {'cloud-init': 'this-is-processed-same-as-user-data'} [16:33] Hmm. I honestly don't like that format, but you say that's what's already being used for other data sources (including, I presume, metadata service vendor_data?) [16:37] what dont you like about it? [16:37] let me make sure I fully understand [16:37] so I get what you're saying for the dict case [16:39] the cloud-config-archive stuff looks strange, and I'd be curious as to how that fits into a 'list' as far as the json is concerned? [16:39] and for the string case I'm just not sure what source data would lead json.loads() to simply return a string [16:41] you know; here's actually the primary reason I'd want vendor_data treated somewhat differently [16:41] with regards to what we're using it for; network configs; I'd not want that to be something a user could override or influence with user data [16:42] makes me wonder if today a user can inject a /etc/network/interfaces in order to give themselves broken networking in the existing status quo [16:44] as far as what would give a string: [16:44] $ python -c 'import json, pprint; pprint.pprint(json.dumps("hi mom"))' [16:44] '"hi mom"' [16:44] gotcha [16:44] in Ruby that would throw an exception :) [16:44] ...I think? [16:45] yeah for sure --> JSON::ParserError: 746: unexpected token at 'hello world' [16:45] i thought it would in python earlier this morning [16:45] so that makes a little more sense [16:45] i dotn knwo that it does really. [16:45] why would you not be able to represent a string as json [16:45] I'm saying your etherpad makes more sense :) [16:45] knowing it doesn't bail in python [16:46] ah. [16:46] so anyway. [16:47] So for my use case, I think with what you've proposed [16:47] hmm [16:47] so vd['cloud_init'] could /not/ be a dict? [16:48] if that's the case, it'd 100% not work for what I'd like to do [16:48] well, I say 'like to do' but am-actually-doing in a branch [16:49] I think there are cases for sure where I'd want to send multiple data formats into my vendor_data [16:49] like where we want to, I guess in a sense introduce the network json format proposed for nova into vendor_data [16:49] in the format you lay out I don't think that's doable [16:50] hm.. [16:50] no i think it is [16:51] howso? [16:51] cloud-init is very forgiving on usre-data. it pays attention to a fairly small set of explicit things, and ignores all the others. [16:51] and this would be the same. [16:51] but you said in the etherpad [16:51] vd['cloud_init'] could be a list or a string [16:51] if the goal is to have json encoded network information [16:51] and cloud_init only reads from that key [16:52] then the only way to get json into that key would be to serialize it into a string... which seems obtuse [16:52] ah. i se. so you want to put that into 'network-config' and have cloud-init read it from ther.e [16:52] i am ok with that. [16:52] or into vd['cloud_init'][network_config] [16:52] wlel, i dont know. [16:53] so 2 ways we could do that [16:53] a.) 'network_config' is not explicitly for cloud-init. so it would be at [16:53] vendor_data['network-config' [16:53] as it will eventually (we hope) be a sibling of meta-data.json (or possibly inside meta-data.json) [16:54] ie, its not explicitly cloud-init. [16:54] yes exactly [16:54] that makes sense [16:54] so today have cloud-init look for vd['network-config'] and md['network-config'] and do things with them [16:54] b.) cloud-init handle a part of type 'network-config' [16:54] the way it handles cloud-config now. [16:54] probably have some versioning concept so whatever differences come out between our 'prerelease' vendor-data version of it and the final openstack release will be able to be smoothly migrated [16:54] ie, either explicit mime type, or '#network-config' (startswith) [16:55] yeah. i agree on vendor. [16:55] where would the #network-config or mimetype come from if it's just a json dict? [16:55] I'd say just look for a key, that key may change in the future, and also have it specify a version [16:55] well, in cloud-config-archive, yo ucan supply mimetype [16:56] hell, have the initial vendor_data key be something like rackspace_network_config to completely ensure we don't eat the namespace [16:56] and in just a string, if it startswith('#network-config'), then it gets handled as that. [16:56] I'd strongly prefer not having a metaformat in this case, given it's json data I don't want to have #network-config followed by a json doc [16:56] hm.. [16:57] because all we've done is add a magic string at top that breaks all other json parsing tools [16:57] yeah, i think that is fine. [16:57] oh. to address one other thing above. [16:57] yes, the usre can break things. [16:57] rackspace seems very much to want to magically stop users from being able to break things. [16:57] but i dont aare. [16:57] don't care. [16:58] from my perspective, the user is in charge. [16:59] I'd rather software be in charge and prevent the user from doing bad, broken things they probably didn't intend [17:00] but it's fine, I get why you think about it that way :) [17:00] if they say "put this file in /etc/network/interfaces" i'm not going to say "i'm smarter than you". [17:00] same as 'rm -rf /etc' [17:01] I get it, I don't agree but that's a-okay [17:01] i understand why you all want to protect users. but there is only so much you can do. [17:01] fwiw, vendor data is disable-able by user entirely. [17:01] haha [17:01] by design. if they want to feed user-data to turn it off, that is fine. [17:01] if someone does that on an onmetal server [17:02] they're going to be paying a lot of money for a server that doesn't have internet [17:02] hah [17:02] So path forward: [17:02] cloud-init stuff handled the way you specified [17:03] look for a separate key in vd for network configs [17:03] i put a bit more in that pad [17:03] yeah [17:03] interesting. [17:03] right now that key is 'network_info' [17:03] fwiw [17:04] thats fine. [17:04] so i think we should stuff YYYYMMDD in there just right away [17:05] when / if it gets into openstack, it will live at metadata-url/YYYYMMDD/network_info.json [17:05] and that YYYYMMDD would fall out there. [17:05] not injected into meta_data.json? [17:05] i dont have strong feelings there. [17:05] * JayF needs to read that spec again [17:05] it certainly could live in meta-data.json [17:06] in which case the YYYYMMDD would certainly fall out (as it would be provided by the openstack version) [17:06] do you think we need to build the version info into network_config [17:06] that was the real question [17:06] when it appears in vendordata [17:06] I think versioning the key is 'fine' [17:06] I dislike all options I can think of [17:06] and that seems the easiest to implement :) [17:07] gimme a sec and I'll give you a good example of a network json format [17:07] fresh off a configdrive [17:07] versioning the key. [17:07] as in network_config_YYYYMMDD ? [17:07] aye [17:07] i don tknow. i tihnk you're optimizing because you dont think it iwll live there long [17:07] :) [17:08] I mean, the reason I'm working with you now is so we don't hold these patches for long [17:08] yeah. [17:08] would be great if cloud-init support was mostly there when it goes into openstack master [17:08] and as a bonus we don't have to hold a patch [17:08] although the hard part is building custom cloud-inits for a ton of operating systems :) [17:09] so i just think this is cleaner... its more explicit: [17:09] for version in vendor_data_json.get('network_config', {}): [17:09] versus: [17:10] for ver in [k.split("_")[2] for k in vendor_data_json if k.startswith("network_config_"): [17:11] so you'd do [17:11] (like in the pad) [17:11] network-info: { "20140101": {network info here } } [17:11] right. [17:12] smoser: so this is what we're putting down looks like right now --> https://gist.github.com/jayofdoom/b035067523defec7fb53 [17:12] smoser: so the patch I posted earlier that pquerna did is currently, on ubuntu, fedora, and centos, already parsing and generating configs against that [17:12] which is pretty nice :) [17:13] ? [17:13] note that I redacted specifics of the server I'm using and knocked out other vendor_data stuff we use [17:13] really? where is that work by pquerna ? [17:13] https://github.com/pquerna/cloud-init/pull/1 [17:13] that's what we're running right now, that patch on top of latest release (.7.5 iirc?) [17:14] i'd not seen that. [17:15] I thought I linked it in here earlier [17:15] my apologies :) [17:15] you probably did. but if i remembered or correctly processed every thing people say to me [17:15] then i'd be pretty awesome [17:15] :) [17:15] heh, that's what logs and git/bzr are for, right? [17:16] so this supports bonds [17:16] and vlans [17:16] and mtu [17:16] Which is why we had to do that :) [17:16] the interfaces file parsing didn't handle it [17:16] and we wanted to use the network json instead [17:16] this is awesome. [17:17] if you spin up a rackspace onmetal server, you get network like that (10gbit bonded, vlans handed down over that bond) [17:17] i was disapointed by onmental minimum $500/month [17:17] I actually think there's a bug, because ubuntu trusty and debian anything can't both use the same /etc/network/interfaces file to achieve that setup [17:17] kind of turns me off from trying it :) [17:17] we charge by the hour [17:17] but monthly minimum. [17:17] by the minute [17:18] no? [17:18] there is a $50 minimum for support fee [17:18] (unforunately) [17:19] dwonder why i thought it was $500. [17:19] well [17:19] maybe that was something else. [17:19] there is a $500 option [17:19] JayF so we can get free onmetal [17:19] thx much, lol [17:20] then i don't need to request it here @ yahoo, lol [17:20] * harlowja getting vms is easier than getting hardware :-P [17:20] managed infrastructure -> $50, managed operations -> $500 [17:20] managed ops = rackers will configure / operate software, etc, etc. [17:20] managed infra is what most... of 'us' types want. [17:21] def [17:21] none of that stuff that makes people do things [17:21] heh [17:21] well http://www.rackspace.com/cloud/servers/onmetal/ [17:21] still confuses me [17:21] so i dont think i simply mis-remembered. [17:22] * JayF brb shortly [17:22] raw infrastructure: $500/server/mo. [17:22] but i guess you're saying really that is : [17:22] so, thats billed per-minute [17:22] those are just monthly prices to be 'nice' [17:22] (le sigh) [17:22] $50 + minutes * (500/30/24/60) [17:22] right [17:23] $0.68493/hr or $0.0114155/min [17:23] pquerna, ipv6 support in that patch ? [17:24] no :( [17:24] ok. so see, you're not perfect [17:24] :) [17:24] this is really neat though. [17:24] we don't do ipv6 right now for.... sigh. switches. [17:24] one day i'll be able to run ubuntu on my switches. it'll be great :) [17:25] JoshNang: actually, i guess we should make the nova/neutron changes to support v6 [17:25] ok. so i'm out for thursday -> tuesday. [17:25] JoshNang: even if our networks don't support it right now [17:25] yeah, there is work to do on that front too [17:25] JoshNang: and do the cloud-init work [17:25] dreamhost has it [17:25] but your network config format *has* to support it [17:26] pquerna: oh definitely. [17:26] +1 [17:26] and cloud-init should generically be able to support it too. [17:26] JoshNang: then at least we can write test cases on both sides [17:26] when we originally drew up the json format, we had ipv6 in there. i just don't think it made it into the spec explicitly [17:26] on dreamcompute you only get a public ipv6 by default. [17:26] nice [17:26] the future [17:26] awesome [17:26] yeah, but i had to find somewhere to ssh hop through to get to my instances there :) [17:27] ok. [17:27] JayF u see my comment on https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 ? [17:27] i can't quite remember if the new inline comment feature sends emails or not, lol [17:27] i'm out the rest of the week. after US EOD. and out monday too. [17:27] but i really would lik to get this stuff in quick [17:27] harlowja, you able to patch harmw's test case for him ? [17:28] https://code.launchpad.net/~harmw/cloud-init/freebsd/+merge/231024 [17:28] hmmm, lets see [17:28] pquerna, or JayF can you get me as much data as possible on what you have for network config ? [17:28] (i'm also out starting Friday until... September 24th, wedding/honeymoon, so JayF/Josh will be on this) [17:28] wtf, why u guys going on vacation [17:28] no good, lol [17:29] pquerna, well congratulations on that. sir. [17:29] u found a female, impassible [17:29] lol [17:29] harlowja al rescate! [17:31] smoser do u have someone looking into https://bugs.launchpad.net/nova/+bug/1361357 or tbd? [17:32] i've got someone trying to verify if it regressed between 2014.1.1 and 2014.1.2 [17:32] but thats been slow going. [17:32] i *think* it is neutron related. [17:32] kk [17:32] hmmm [17:32] i as happy to see the ML post [17:32] that seemingly its not just us [17:32] ya, not just u [17:32] thats always good [17:33] we really need to do something there. [17:33] even the 3 seconds that it took before it regressed is laughable [17:34] on amazon, the ~ 20 requests it takes to crawl their MD service takes ~ 0.07 seconds [17:35] :-/ [17:35] ya, it makes no sense [17:36] the time it takes for such a simple service should be like < 0.01 seconds [17:36] no reason for it not to me [17:36] *not to be [17:39] weird. i thought the MD service put it all in memcache anyways. hrm. [17:39] harlowja: it didn't email me [17:39] pquerna, yeah, its definitely supposed to be. [17:40] we don't yet deploy MD service, but its on the short list of things i want. [17:40] its just silly that we don't [17:40] so... lets stop doing silly things. [17:42] less silly is good [17:42] but u don't want to remove all the silly [17:42] thats no fun [17:42] smoser: whre should I run that script of yours, inside cirros? (regarding the nova bug) [17:43] or would just any vm suffice [17:43] since I'm running openstack-nova-api-2014.1.1-2.el6.noarch [17:44] old comparison code: 0.103947877884 [17:44] new comparision code: 4.36138916016 [17:45] :-/ [17:45] harmw, what script did you run ? [17:45] the wget ? [17:45] i dont recal. [17:45] how is 4.2 more seconds even possible [17:45] thats insane [17:45] ec2metadata is a command on ubuntu or cirros [17:46] harmw, so what are your bencharms there ? [17:46] hhe [17:46] bencharms. [17:46] this is from a centos7 instance in my cloud :P [17:46] bench marks [17:46] but "old" and "new" [17:46] I took the script at https://bugs.launchpad.net/nova/+bug/1361357 [17:47] and it printed that [17:47] oh. nah. [17:47] that was red-herring. [17:47] i should just change the description of that bug. [17:47] i'll do that now [17:47] hhehe [17:48] well, if you have some benchmark I could run [17:48] the string comparison code is massively slower than it was [17:48] just let me know [17:48] but that is actually by design [17:48] harmw, [17:48] time ec2metadata [17:48] just run that. [17:49] ah, and in which package should I find that binary? [17:49] smoser i don't get how even crappy string comparison can slow down by that much [17:50] its not like massive strings are being compared right [17:50] where massive is in the MB range [17:50] its gotta be something else than that [17:52] right. [17:52] it was redherring [17:53] ya, does the metadata server now use the conductor and object model [17:54] https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L130 [17:54] that thing adds on alot of new RPC requests and such [17:54] harlowja: https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 I don't see any comments on this at all [17:55] JayF ah, it saves unsaved comment [17:55] thats not cool [17:55] odd [17:55] oh, u have to press the save comment button [17:55] weird [17:55] JayF ok try now [17:56] totally got an email this time [17:58] harlowja: I guess I don't see how that's cleaner in this case; it's more lines of code to do the same thing? [17:58] harlowja: if there's a difference in functionality help me understand please [17:58] smoser: you've got a direct link on ec2metadata binary? [17:58] or is it included with c-i? [17:58] JayF version can be passed in as a function argument, so the code i was doing was just ensuring that u aren't going to rescan the same version twice [17:58] aka someone passes in openstack.OS_GRIZZLY [17:59] see I almost was tempted [17:59] to not even want to use that kwarg anymore [17:59] because I didn't see any instances where it was being used [17:59] JayF kk, then lets drop it? [18:00] seems fair to me [18:01] harlowja: is there a better way to get the merge req to update other than deleting and making a new one (in lp?) [18:01] afaik u can just do bzr ci, and push and it the merge request will update [18:01] bzr ci? [18:01] bzr commit i guess [18:02] guess thats my shorthand [18:02] harmw, sorry. not in cloud-init. sorry behind on reading. [18:02] here.... [18:02] I've been doing bzr commit -> bzr push lp:~jason-oldos/cloud-init/upgrade-configdrive [18:02] this will work if you have sane cloud-init [18:02] http://paste.ubuntu.com/8151758/ [18:02] and either I'm not patient enough to see the merge req update or it never does [18:02] JayF ya, that should be fine [18:02] me being impatient very likely [18:02] JayF it should show 'commit updating' or somethign usually [18:02] http://paste.ubuntu.com/8161416/ is what i sent to a co-worker who i think will get involved with cloud-init [18:03] the basics [18:03] harlowja, well, the string comparison is really by design. like explicitly complex to avoid magic optimizaiton that would give informatin away. [18:03] s/ci/commit/ [18:03] and that matches what Iw as doing [18:03] smoser ya, i've put something similar in another part of code that uses hmac stuff [18:04] smoser: I think my cloud sux :P [18:04] crawl of openstack took 17.666 seconds [18:04] crawl of ec2 took 14.452 seconds [18:04] harmw lol [18:04] super-speed [18:05] https://github.com/stackforge/osprofiler/commit/034987f083cfa1aff2347a4f (for the other place i've used that hmac comparison func) [18:06] * https://docs.python.org/2/library/hmac.html#hmac.compare_digest [18:06] anyways [18:13] hehe, neutron-server taking 40% cpu :p I should realy upgrade this Atom server [18:14] :( [18:41] harlowja, so something is odd in your 'smart-read' branch [18:41] could be [18:41] lol [18:41] time nosetests tests/unittests/test_datasource/test_openstack.py [18:41] not good. it takes like ... (still going) [18:41]