=== harlowja is now known as harlowja_away
=== praneshp_ is now known as praneshp
harmwsmoser: I'm to sucky a python programmer to fix the test on my own, so a little help or example would be helpfull yes :)06:56
=== praneshp_ is now known as praneshp
=== smoser` is now known as smoser
smoserJayF, so you're the only one using vendordata on openstack to my knowledge. at least with cloud-init.12:51
smoserJayF, https://etherpad.openstack.org/p/cloud-init-vendor-data13:10
smoserharmw, i'll see if i can't help ith that test today.  harlowja wrote the other ones there.  cloud-init suffers badly from having too many test case mechanisms.13:11
siertsmoser: I did an update on #1246485 - not sure if the suggested solution from alexius is perfect, but it works in my situation. also affects latest version...13:55
smosersiert, just fyi, man page for 'hostname' recommends against using 'hostname -f'13:57
siertsmoser: hm, ok. but the RHEL install guide is slightly more ambiguous - "either as a fqdn or as a short host name"14:04
smosersiert,  thank you for loking at this.14:10
smoserwhile it seems straight forward, i think its probably as much as you'd hope.14:11
smosermy experience is that any time i touch 'hostname' or '/etc/hosts', something breaks.14:11
smoser https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/105266414:15
siertthe /etc/hostname should always have the short name only. on rhel, when there is no hostname defined in /etc/sysconfig/network, then /etc/hostname is used as a fallback, to set the systems hostname.14:21
siertdebian/ubuntu have the short name in /etc/hostname and the rest is read from /etc/hosts.14:27
smoserhaving 'hostname' return a fqdn seems broken to me.14:37
siertubuntu 14.04 returns hostname version 3.15. rhel based systems: net-tools 1.60 and hostname 1.100 (2001-04-14)14:39
siertmaybe the difference is in the version of 'hostname', let me see... :s14:40
smoserwell, on ubuntu, it returns whatever you set it to.14:40
smoser hostname14:41
smoser$ hostname14:41
smoser$ sudo hostname brickies.foobar.example.com14:41
smoser$ sudo hostname14:41
smosersudo: unable to resolve host brickies.foobar.example.com14:41
smoser$ sudo hostname brickies14:42
smosersudo: unable to resolve host brickies.foobar.example.com14:42
smoser$ hostname14:42
smoserand that initially gets set from the content of /etc/hostname14:42
siertyeah, by /etc/init/hostname.conf (hostname -b -F)14:43
smoser"historically this file was supposed to only contain the hostname and not the full canonical FQDN. ..."14:45
smosersiert, i know it sounds like i'm paying too much attention here, but i've just been bit many times.14:46
smoserbasically my experience is that anything you do breaks something.14:46
siertthanks for you time, I gtg. will look into it tomorrow, maybe there's another way of getting rhel to behave the same as ubuntu when it comes to hostnames.14:51
=== zz_gondoi is now known as gondoi
JayFsmoser: Is that a bad thing? Good thing? Heh16:24
JayFWe've used vendor_data.json internally for stuff for a while, just now integrating it with cloud-init16:24
smoserJayF, well, i would not want to break anyone's usage16:26
smoserand i was proposing changes in consumpiton of it16:26
smosereven though that consumption i spretty broken at the moment.16:26
JayFsmoser: not sure I grok what you're trying to suggest in that etherpad16:30
JayFHave you had feature requests along those lines from other folks using vendor_data?16:30
smoserthat is how vendor data works16:31
smoserin other datasources16:31
smoserand actually , if there is 'cloud-init' in the network metadata service on openstack, then it probably works like that now.16:31
smoser(well, for the 'dict' case)16:32
smoserie: 16:32
smoser {'cloud-init': 'this-is-processed-same-as-user-data'}16:32
JayFHmm. I honestly don't like that format, but you say that's what's already being used for other data sources (including, I presume, metadata service vendor_data?)16:33
smoserwhat dont you like about it?16:37
JayFlet me make sure I fully understand16:37
JayFso I get what you're saying for the dict case16:37
JayFthe cloud-config-archive stuff looks strange, and I'd be curious as to how that fits into a 'list' as far as the json is concerned?16:39
JayFand for the string case I'm just not sure what source data would lead json.loads() to simply return a string16:39
JayFyou know; here's actually the primary reason I'd want vendor_data treated somewhat differently16:41
JayFwith regards to what we're using it for; network configs; I'd not want that to be something a user could override or influence with user data16:41
JayFmakes me wonder if today a user can inject a /etc/network/interfaces in order to give themselves broken networking in the existing status quo 16:42
smoseras far as what would give a string:16:44
smoser$ python -c 'import json, pprint; pprint.pprint(json.dumps("hi mom"))'16:44
smoser'"hi mom"'16:44
JayFin Ruby that would throw an exception :)16:44
JayF...I think?16:44
JayFyeah for sure --> JSON::ParserError: 746: unexpected token at 'hello world'16:45
smoseri thought it would in python earlier this morning16:45
JayFso that makes a little more sense16:45
smoseri dotn knwo that it does really.16:45
smoserwhy would you not be able to represent a string as json16:45
JayFI'm saying your etherpad makes more sense :)16:45
JayFknowing it doesn't bail in python16:45
smoserso anyway.16:46
JayFSo for my use case, I think with what you've proposed16:47
JayFso vd['cloud_init'] could /not/ be a dict?16:47
JayFif that's the case, it'd 100% not work for what I'd like to do16:48
JayFwell, I say 'like to do' but am-actually-doing in a branch16:48
JayFI think there are cases for sure where I'd want to send multiple data formats into my vendor_data16:49
JayFlike where we want to, I guess in a sense introduce the network json format proposed for nova into vendor_data16:49
JayFin the format you lay out I don't think that's doable16:49
smoserno i think it is 16:50
smosercloud-init is very forgiving on usre-data. it pays attention to a fairly small set of explicit things, and ignores all the others.16:51
smoserand this would be the same.16:51
JayFbut you said in the etherpad16:51
JayFvd['cloud_init'] could be a list or a string16:51
JayFif the goal is to have json encoded network information16:51
JayFand cloud_init only reads from that key16:51
JayFthen the only way to get json into that key would be to serialize it into a string... which seems obtuse16:52
smoserah. i se. so you want to put that into 'network-config' and have cloud-init read it from ther.e16:52
smoseri am ok with that. 16:52
JayFor into vd['cloud_init'][network_config]16:52
smoserwlel, i dont know.16:52
smoserso 2 ways we could do that16:53
smosera.) 'network_config' is not explicitly for cloud-init. so it would be at16:53
smoser   vendor_data['network-config'16:53
smoser  as it will eventually (we hope) be a sibling of meta-data.json (or possibly inside meta-data.json)16:53
smoserie, its not explicitly cloud-init.16:54
JayFyes exactly16:54
JayFthat makes sense16:54
JayFso today have cloud-init look for vd['network-config'] and md['network-config'] and do things with them16:54
smoserb.) cloud-init handle a part  of type 'network-config'16:54
smoser the way it handles cloud-config now.16:54
JayFprobably have some versioning concept so whatever differences come out between our 'prerelease' vendor-data version of it and the final openstack release will be able to be smoothly migrated16:54
smoser ie, either explicit mime type, or '#network-config' (startswith)16:54
smoseryeah. i agree on vendor.16:55
JayFwhere would the #network-config or mimetype come from if it's just a json dict?16:55
JayFI'd say just look for a key, that key may change in the future, and also have it specify a version16:55
smoserwell, in cloud-config-archive, yo ucan supply mimetype16:55
JayFhell, have the initial vendor_data key be something like rackspace_network_config to completely ensure we don't eat the namespace16:56
smoserand in just a string, if it startswith('#network-config'), then it gets handled as that.16:56
JayFI'd strongly prefer not having a metaformat in this case, given it's json data I don't want to have #network-config followed by a json doc16:56
JayFbecause all we've done is add a magic string at top that breaks all other json parsing tools16:57
smoseryeah, i think that is fine.16:57
smoseroh. to address one other thing above.16:57
smoseryes, the usre can break things.16:57
smoserrackspace seems very much to want to magically stop users from being able to break things.16:57
smoserbut i dont aare.16:57
smoserdon't care.16:57
smoserfrom my perspective, the user is in charge.16:58
JayFI'd rather software be in charge and prevent the user from doing bad, broken things they probably didn't intend16:59
JayFbut it's fine, I get why you think about it that way :)17:00
smoserif they say "put this file in /etc/network/interfaces" i'm not going to say "i'm smarter than you".17:00
smosersame as 'rm -rf /etc'17:00
JayFI get it, I don't agree but that's a-okay17:01
smoseri understand why you all want to protect users. but there is only so much you can do.17:01
smoserfwiw, vendor data is disable-able by user entirely.17:01
smoserby design. if they want to feed user-data to turn it off, that is fine.17:01
JayFif someone does that on an onmetal server17:01
JayFthey're going to be paying a lot of money for a server that doesn't have internet17:02
JayFSo path forward:17:02
JayFcloud-init stuff handled the way you specified17:02
JayFlook for a separate key in vd for network configs17:03
smoseri put a bit more in that pad17:03
JayFright now that key is 'network_info'17:03
smoserthats fine.17:04
smoserso i think we should stuff YYYYMMDD in there just right away17:04
smoserwhen / if it gets into openstack, it will live at metadata-url/YYYYMMDD/network_info.json17:05
smoserand that YYYYMMDD would fall out there.17:05
JayFnot injected into meta_data.json?17:05
smoseri dont have strong feelings there.17:05
* JayF needs to read that spec again17:05
smoserit certainly could live in meta-data.json17:05
smoserin which case the YYYYMMDD would certainly fall out (as it would be provided by the openstack version)17:06
smoserdo you think we need to build the version info into network_config17:06
smoserthat was the real question17:06
smoserwhen it appears in vendordata17:06
JayFI think versioning the key is 'fine'17:06
JayFI dislike all options I can think of17:06
JayFand that seems the easiest to implement :)17:06
JayFgimme a sec and I'll give you a good example of a network json format17:07
JayFfresh off a configdrive17:07
smoserversioning the key.17:07
smoseras in network_config_YYYYMMDD ?17:07
smoseri don tknow. i tihnk you're optimizing because you dont think it iwll live there long17:07
JayFI mean, the reason I'm working with you now is so we don't hold these patches for long17:08
smoseryeah. 17:08
JayFwould be great if cloud-init support was mostly there when it goes into openstack master17:08
JayFand as a bonus we don't have to hold a patch17:08
JayFalthough the hard part is building custom cloud-inits for a ton of operating systems :)17:08
smoserso i just think this is cleaner... its more explicit:17:09
smoserfor version in vendor_data_json.get('network_config', {}):17:09
smoserfor ver in [k.split("_")[2] for k in vendor_data_json if k.startswith("network_config_"):17:10
JayFso you'd do17:11
smoser(like in the pad)17:11
JayFnetwork-info: { "20140101": {network info here } }17:11
JayFsmoser: so this is what we're putting down looks like right now --> https://gist.github.com/jayofdoom/b035067523defec7fb5317:12
JayFsmoser: so the patch I posted earlier that pquerna did is currently, on ubuntu, fedora, and centos, already parsing and generating configs against that17:12
JayFwhich is pretty nice :)17:12
JayFnote that I redacted specifics of the server I'm using and knocked out other vendor_data stuff we use17:13
smoserreally? where is that work by pquerna ?17:13
JayFthat's what we're running right now, that patch on top of latest release (.7.5 iirc?)17:13
smoseri'd not seen that. 17:14
JayFI thought I linked it in here earlier17:15
JayFmy apologies :)17:15
smoseryou probably did. but if i remembered or correctly processed every thing people say to me17:15
smoserthen i'd be pretty awesome17:15
JayFheh, that's what logs and git/bzr are for, right?17:15
smoserso this supports bonds17:16
smoserand vlans17:16
smoserand mtu17:16
JayFWhich is why we had to do that :)17:16
JayFthe interfaces file parsing didn't handle it17:16
JayFand we wanted to use the network json instead17:16
smoserthis is awesome.17:16
JayFif you spin up a rackspace onmetal server, you get network like that (10gbit bonded, vlans handed down over that bond)17:17
smoseri was disapointed by onmental minimum $500/month17:17
JayFI actually think there's a bug, because ubuntu trusty and debian anything can't both use the same /etc/network/interfaces file to achieve that setup17:17
smoserkind of turns me off from trying it :)17:17
JayFwe charge by the hour17:17
smoserbut monthly minimum.17:17
pquernaby the minute17:17
pquernathere is a $50 minimum for support fee17:18
smoserdwonder why i thought it was $500.17:19
smosermaybe that was something else.17:19
pquernathere is a $500 option17:19
harlowjaJayF so we can get free onmetal17:19
harlowjathx much, lol17:19
harlowjathen i don't need to request it here @ yahoo, lol17:20
* harlowja getting vms is easier than getting hardware :-P17:20
pquernamanaged infrastructure -> $50, managed operations -> $50017:20
pquernamanaged ops = rackers will configure / operate software, etc, etc.17:20
pquernamanaged infra is what most... of 'us' types want.17:20
harlowjanone of that stuff that makes people do things17:21
smoserwell http://www.rackspace.com/cloud/servers/onmetal/17:21
smoserstill confuses me17:21
smoserso i dont think i simply mis-remembered.17:21
* JayF brb shortly17:22
smoserraw infrastructure: $500/server/mo.17:22
smoserbut i guess you're saying really that is :17:22
pquernaso, thats billed per-minute17:22
pquernathose are just monthly prices to be 'nice'17:22
pquerna(le sigh)17:22
smoser $50 + minutes * (500/30/24/60)17:22
pquerna$0.68493/hr or $0.0114155/min17:23
smoserpquerna, ipv6 support in that patch ?17:23
pquernano :(17:24
smoserok. so see, you're not perfect17:24
smoserthis is really neat though.17:24
pquernawe don't do ipv6 right now for.... sigh. switches.17:24
pquernaone day i'll be able to run ubuntu on my switches. it'll be great :)17:24
pquernaJoshNang: actually, i guess we should make the nova/neutron changes to support v617:25
smoserok. so i'm out for thursday -> tuesday.17:25
pquernaJoshNang: even if our networks don't support it right now17:25
smoseryeah, there is work to do on that front too17:25
pquernaJoshNang: and do the cloud-init work17:25
smoserdreamhost has it 17:25
smoserbut your network config format *has* to support it17:25
JoshNangpquerna: oh definitely.17:26
smoserand cloud-init should generically be able to support it too.17:26
pquernaJoshNang: then at least we can write test cases on both sides17:26
JoshNangwhen we originally drew up the json format, we had ipv6 in there. i just don't think it made it into the spec explicitly17:26
smoseron dreamcompute you only get a public ipv6 by default.17:26
pquernathe future17:26
smoseryeah, but i had to find somewhere to ssh hop through to get to my instances there :)17:26
harlowjaJayF u see my comment on https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 ?17:27
harlowjai can't quite remember if the new inline comment feature sends emails or not, lol17:27
smoseri'm out the rest of the week. after US EOD. and out monday too. 17:27
smoserbut i really would lik to get this stuff in quick17:27
smoserharlowja, you able to patch harmw's test case for him ?17:27
harlowjahmmm, lets see17:28
smoserpquerna, or JayF can you get me as much data as possible on what you have for network config ?17:28
pquerna(i'm also out starting Friday until... September 24th, wedding/honeymoon, so JayF/Josh will be on this)17:28
harlowjawtf, why u guys going on vacation17:28
harlowjano good, lol17:28
smoserpquerna, well congratulations on that. sir.17:29
harlowjau found a female, impassible17:29
harmwharlowja al rescate!17:29
harlowjasmoser do u have someone looking into https://bugs.launchpad.net/nova/+bug/1361357 or tbd?17:31
smoseri've got someone trying to verify if it regressed between 2014.1.1 and 2014.1.217:32
smoserbut thats been slow going.17:32
smoseri *think* it is neutron related.17:32
smoseri as happy to see the ML post17:32
smoserthat seemingly its not just us17:32
harlowjaya, not just u 17:32
harlowjathats always good17:32
smoserwe really need to do something there.17:33
smosereven the 3 seconds that it took before it regressed is laughable17:33
smoseron amazon, the ~ 20 requests it takes to crawl their MD service takes ~ 0.07 seconds 17:34
harlowjaya, it makes no sense17:35
harlowjathe time it takes for such a simple service should be like < 0.01 seconds17:36
harlowjano reason for it not to me17:36
harlowja*not to be17:36
pquernaweird. i thought the MD service put it all in memcache anyways. hrm.17:39
JayFharlowja: it didn't email me17:39
smoserpquerna, yeah, its definitely supposed to be.17:39
pquernawe don't yet deploy MD service, but its on the short list of things i want.17:40
pquernaits just silly that we don't17:40
pquernaso... lets stop doing silly things.17:40
harlowjaless silly is good17:42
harlowjabut u don't want to remove all the silly17:42
harlowjathats no fun17:42
harmwsmoser: whre should I run that script of yours, inside cirros? (regarding the nova bug)17:42
harmwor would just any vm suffice17:43
harmwsince I'm running openstack-nova-api-2014.1.1-2.el6.noarch17:43
harmwold comparison code:  0.10394787788417:44
harmwnew comparision code:  4.3613891601617:44
smoserharmw, what script did you run ? 17:45
smoserthe wget ?17:45
smoseri dont recal.17:45
harlowjahow is 4.2 more seconds even possible17:45
harlowjathats insane17:45
smoserec2metadata is a command on ubuntu or cirros17:45
smoserharmw, so what are your bencharms there ?17:46
harmwthis is from a centos7 instance in my cloud :P17:46
smoserbench marks17:46
smoserbut "old" and "new"17:46
harmwI took the script at https://bugs.launchpad.net/nova/+bug/136135717:46
harmwand it printed that17:47
smoseroh. nah. 17:47
smoserthat was red-herring.17:47
smoseri should just change the description of that bug.17:47
smoseri'll do that now17:47
harmwwell, if you have some benchmark I could run17:48
smoserthe string comparison code is massively slower than it was 17:48
harmwjust let me know17:48
smoserbut that is actually by design17:48
smoserharmw, 17:48
smosertime ec2metadata17:48
smoserjust run that. 17:48
harmwah, and in which package should I find that binary?17:49
harlowjasmoser i don't get how even crappy string comparison can slow down by that much17:49
harlowjaits not like massive strings are being compared right17:50
harlowjawhere massive is in the MB range17:50
harlowjaits gotta be something else than that17:50
smoserit was redherring17:52
harlowjaya, does the metadata server now use the conductor and object model17:53
harlowjahttps://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L130 17:54
harlowjathat thing adds on alot of new RPC requests and such17:54
JayFharlowja: https://code.launchpad.net/~jason-oldos/cloud-init/upgrade-configdrive/+merge/232312 I don't see any comments on this at all17:54
harlowjaJayF ah, it saves unsaved comment17:55
harlowjathats not cool17:55
harlowjaoh, u have to press the save comment button17:55
harlowjaJayF ok try now17:55
JayFtotally got an email this time17:56
JayFharlowja: I guess I don't see how that's cleaner in this case; it's more lines of code to do the same thing?17:58
JayFharlowja: if there's a difference in functionality help me understand please 17:58
harmwsmoser: you've got a direct link on ec2metadata binary?17:58
harmwor is it included with c-i?17:58
harlowjaJayF version can be passed in as a function argument, so the code i was doing was just ensuring that u aren't going to rescan the same version twice17:58
harlowjaaka someone passes in openstack.OS_GRIZZLY 17:58
JayFsee I almost was tempted17:59
JayFto not even want to use that kwarg anymore17:59
JayFbecause I didn't see any instances where it was being used17:59
harlowjaJayF kk, then lets drop it?17:59
harlowjaseems fair to me 18:00
JayFharlowja: is there a better way to get the merge req to update other than deleting and making a new one (in lp?)18:01
harlowjaafaik u can just do bzr ci, and push and it the merge request will update18:01
JayFbzr ci?18:01
harlowjabzr commit i guess18:01
harlowjaguess thats my shorthand18:02
smoserharmw, sorry. not in cloud-init. sorry behind on reading.18:02
JayFI've been doing bzr commit -> bzr push lp:~jason-oldos/cloud-init/upgrade-configdrive18:02
smoserthis will work if you have sane cloud-init18:02
JayFand either I'm not patient enough to see the merge req update or it never does18:02
harlowjaJayF ya, that should be fine18:02
JayFme being impatient very likely18:02
harlowjaJayF it should show 'commit updating' or somethign usually18:02
harlowjahttp://paste.ubuntu.com/8161416/ is what i sent to a co-worker who i think will get involved with cloud-init18:02
harlowjathe basics18:03
smoserharlowja, well, the string comparison is really by design. like explicitly complex to avoid magic optimizaiton that would give informatin away.18:03
JayFand that matches what Iw as doing18:03
harlowjasmoser ya, i've put something similar in another part of code that uses hmac stuff18:03
harmwsmoser: I think my cloud sux :P18:04
harmwcrawl of openstack took 17.666 seconds18:04
harmwcrawl of ec2 took 14.452 seconds18:04
harlowjaharmw lol18:04
harlowjahttps://github.com/stackforge/osprofiler/commit/034987f083cfa1aff2347a4f (for the other place i've used that hmac comparison func)18:05
harlowja* https://docs.python.org/2/library/hmac.html#hmac.compare_digest18:06
harmwhehe, neutron-server taking 40% cpu :p I should realy upgrade this Atom server18:13
smoserharlowja, so something is odd in your 'smart-read' branch18:41
harlowjacould be18:41
smosertime nosetests tests/unittests/test_datasource/test_openstack.py18:41
smosernot good. it takes like ... (still going)18:41
harlowjakk, let me see what i can find out18:41
smoser65 seconds18:41
smoserbut returns success (which is worse)18:41
harlowjaprobably retrying i think18:42
smoserharlowja, i was going to do this:18:42
smoser http://paste.ubuntu.com/8161712/18:42
smoserto remove the duplication in find_working_version... put the differencein _avialable_versions18:42
harlowjathat seems ok18:43
smoserharlowja, wrt the openstakc metadata service and conductor...19:00
smoserhopefully we cache the InstanceMetaData()19:01
smoserand reuse it.19:01
smoserso yeah, you'd hit that once, and that would suck, but it is supposed to be good for 15 seconds after tat.19:01
harlowjasure, even without caching  there is no way it should take that long :-P19:02
harlowjai mean i can render most of yahoo in +10 seconds19:02
harlowjaya, i'd almost want to disable that cache for figuring out the root cause19:06
smoserhm.. wonder if that is in 2014.1.219:07
harlowjasmoser alright, https://code.launchpad.net/~harlowja/cloud-init/smart-read/+merge/232316 updated19:34
smoserharlowja, hey. ok. i just merged that.21:08
smoserharlowja, if you could fix (or help harm) fix his bsd bug21:08
smoserthat'd be good21:08
smoserone of his tests fails on linux21:08
smoserand then you are welcome to push that to trunk21:08
* smoser feels really bad for always taking so long to do things for harm.21:09
harlowjasmoser checking21:21
harlowjawill help boss21:21
harlowjaharmw alright, let me see if i can get your thingy adjusted, will get u a patch i guess21:27
=== gondoi is now known as zz_gondoi
harlowja_harmw if u get a chance, http://paste.ubuntu.com/8163345/22:25
harlowja_applying that to your freebsd makes it work for me22:25
harlowja_hopefully fine with u, should just be apply to apply it and resubmit your branch22:26

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!