smosermgagne, hm..11:50
smoseryour 'is_bonding_slave' idea is fine, but i really dontunderstand why it shoudl be needed.11:50
smoseryou mentioned upstart, are you using upstart somewhere ?11:51
smoseryou dont want to mess with dsmode really.11:56
smoserin current trunk, dsmode=local would allow you to make init_modules run earlier (without access to network)11:57
=== rangerpbzzzz is now known as rangerpb
smosermgagne, around ?15:44
mgagnesmoser: I am now15:56
smoserso, are you using upstart ?15:58
mgagnesmoser: I'm not using upstart, I was reading source code and commenting about it. cloud-init is running twice as per bug description. Running it a second time fails because cloud-init doesn't expect bonding to be configured at this point or in fact, all code and tests were done without bonding support so a lot of assumption were made which aren't true anymore.15:58
mgagneI'm booting on ubuntu 16.04, it's systemd afaik15:59
smosercloud-init does run twice for sure, but only the first time should set the networking up.15:59
smoseroh. but we rename on every boot, so maybe we're doign that twice.15:59
mgagneok, well that's not the case on my side, bug 3.1) and 3.2) were caused by this double network config run15:59
smoseri'll have a look in a bit. your is_bonding_slave change seems to make sense.16:00
mgagneno no, I boot ONCE and it fails, I'm not even testing reboot at this point16:00
mgagnebecause of this mac/link/device mapping, 2nd run fails because of how bonding behaves, it changes the mac of the bonding slaves, hence the added logic for is_bonding_slave.16:01
mgagneI didn't do extensive tests, just boot, ping, ssh (with sshkey) and check hostname16:02
harlowjasmoser u ever get a chance to look over https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/30260918:02
harlowjaits the future!18:02
smoserharlowja, i've not looked at it now.18:02
smoserso ..18:04
smoserInstead of looking in a very specific location for18:04
smosercloudinit config modules; which for those adding there18:04
smoserown modules makes it hard to do without patching that18:04
smoserlocation instead use entrypoints and register all18:04
smosercurrent cloudinit config modules by default with that18:04
smosernew entrypoint (and use that same entrypoint namespace18:04
smoserfor later finding needed modules).18:04
smoserhow does registering the entry points help "those adding their own modules"18:05
smoserrharper, what shall i do for mgagne's auto-bringup of bond.18:06
smoserdid you have work on that that i didnt' see ?18:06
rharpersmoser: the fix is what i had18:06
rharperbut in general, we need to think about v4 vs v618:07
smoserwhat fix ?18:07
rharperin eni.py18:07
smoseri didnts ee, sorry.18:07
smoserdidnt see18:07
rharperhe posted patches, basically adds the if 'bond-master' or 'bond-slaves' in iface, then emit auto18:07
harlowjasmoser  so they still need to add a entry to cloud.cfg (either at packaging time, or at userdata/runtime)18:07
rharpersmoser: <mgagne> rharper: all patches: http://paste.ubuntu.com/23059836/18:07
harlowjai didn't go into the path of discovering and creating a cloud[init,config,final] sections of that config18:07
harlowjabecause though i could, its umm, non trival :-P18:08
harlowjaand likely requires more metadata on modules to define there ordering (not via cloud.cfg at that point)18:08
rharperwe probably should instead check if iface['type'] in ['bond', 'vlan'] and possibly 'bridge' ;18:08
smoserrharper, so you're just assuming all bonds (or vlans or bridges) then are 'auto'18:09
harlowjaso that kind of stuff seems like a larger change, vs just attempting to find modules that are already defined in cloud.cfg via entrypoints (leaving the change to be just a different way to find modules)18:09
rharpersmoser: we default to auto if an interface has a subnet18:09
rharperin this case, it's a bond with no subnets18:09
rharperas it's being assembled but not with a subnet;18:10
smoserie, those default to 'auto' while others (even with 'subnets') default to non-auto18:10
smoserwe do default to auto if a subnet ?18:10
rharperno we always default to auto unless 'control' is set in subnet18:10
smoserhm.. you're saying that is true after your change or before18:11
rharperthere are a few known cases where config explicitly wants subnet + control=manual (aka iscsiroot)18:11
rharperif iface has subnet, control=auto for the iface/index pair18:11
rharperif you do not include any subnet, then no auto (except for bond-slaves)18:11
rharperthat really should be any interface with a nested config (master/slave); I'm pretty sure18:12
smoserif iface has subnet and no control=18:12
rharperthen control is set to auto18:12
rharperfor iscsiroot, we specify control: manual18:12
rharperoverride the default;18:12
smoserright, so you're not actually checking for bond-master.18:15
smoseryou're just trunign oauto on18:15
rharperno, we check for bond-master in the case if iface with no subnets18:16
rharperand then auto it, *if* it's a slave (slaves point to their master with bond-master key)18:16
rharperbut, if the bond master itself (bond0) doesn't configure a subnet, it doesn't get an auto18:17
rharperI suspect the code in ifupdown/if-pre-up.d/ifenslave could be fixed to raise the bond master independent of whether it's marked auto or not; but it currently does *not* bring up the master unless listed in allow-auto (or marked auto)18:18
rharperif bond0 doesn't come up then the rest of the config won't succeed (we timeout waiting on bond0 to be created via slave ifup hook)18:18
rharpera bond-specific solution/workaround is to also include the bond master (indicated by key bond-slaves in iface) to be marked auto;18:19
rharperthat might be enough, but I'd like to test/check bridges without subnets and vlans without subnets to see if we generally need to mark non-subnet interfaces with auto by default;  that is, I don't yet know of a config where we want a manual bond/vlan/bridge18:20
smoserok. for now i'm good with the fix as you all had.18:56
smoserit is kind of wierd and possibly wrong that we are renaming devices in the 'init' stage (in addition to init-local).18:58
smoserharlowja, what i dont understand is how you are making the problem of adding a config thing any easier.18:59
smoserthe cc_foo.py can now be placed in some additional directory ?18:59
harlowjasmoser i can put the config modules in my own library, expose a named entrypoint, then just update cloud.cfg to reference that module18:59
harlowjaso cc_blahblah no longer needs to be patched into cloud-init19:00
smoserhow do you expose a named entry point ?19:00
harlowjasame way the modification there to cloud-init setup.py is19:00
harlowjaso library would just need to add a entrypoints entry (like in that setup.py) in there own module19:01
harlowjaso in said libraries setup.py there would be an entry like19:03
harlowja    'cloud.config': [19:03
harlowja        'my_thing = my_thing.my_cloud_handler',19:03
harlowja    ],19:03
harlowjaso when cloudinit looks for a way to call 'my_thing' (assuming its in a cloud.cfg listing somewhere) then it can go out and try to find it (and load this library to get at it)19:03
harlowja(or if nobody registered that module, then die as usual)19:04
smoserharlowja, so..19:16
smoser 19:16
smoserthat is what i dont like about entry poitns19:16
smosertakes ~ 0.01 to bring up python , 0.03 to bring up python3 on a reasonably current SSD19:16
smoser(with '0' as first arg)19:17
smoserimporting the pkg_resources takes 0.3 seconds on python, and 0.25-ish on python319:17
smoserit does look like it caches stuff as 10 runs take about the same as 119:17
smoseri'm guessing that python3 is faster in my test only because i have fewer entry points or packages installed on the system in python3 compared to python219:18
smoserso its doing less work.19:18
smoserthis is also embarrasing:19:20
smoser http://paste.ubuntu.com/23062545/19:20
smoserand it needs fixing19:20
smoserbut i'm somewhat hesitant to add something like that.19:20
harlowjaso thats just because u imported 'pkg_resources' ?19:31
smoserthe pkg resources import takes quite some time (~.1 seconds)19:32
smoserthe enumerating of some non-existant namespace takes .2 seconds19:32
smoserobviously very scientific data there.19:32
smoseri should have done a -119:33
smoserlets re-do that paste19:33
smoser http://paste.ubuntu.com/23062572/19:36
smoserthere. -1 is just cost of bringing up python19:36
smoserthere ^19:37
smoser-1 is cost of python19:37
smoser 0 is cost of import pkg_resoruces19:38
smoser 1 is cost of one call to 'iter_entry_points'19:38
smoser10 is cost of 10 calls19:38
smoserwith revised my.py at http://paste.ubuntu.com/23062581/19:38
harlowjaseems like they need to better optimize that entrypoint 'catalog' lol19:38
smoseryeah, it is stat crazy19:38
smoserthose openstack cli programs do taht.19:39
smoserthey do cache well19:39
smosersince 10 runs takes basically nothing more than 119:39
harlowjabut assuming a entrypoint catalog existed, in the core python, then i'd assume that stuff wouldn't take forever19:39
harlowjaaka a tiny sqllite db19:39
harlowjawonder why such a thing doesn't exist19:40
smoseryeah, but i think the entry points are stuff in taht egg.info right ?19:40
smoserthats how those are loaded ?19:40
smoserso python goes looking in any possible directory in sys.path for a file egg.info or something and then goes reading it and such.19:40
harlowjathats one location of it, but u'd think that pip could update a sqllite db or something19:40
harlowjai wonder if the python community is working on anything like that19:41
harlowjaseems pretty obvious to do that19:41
harlowjathen X people wouldn't be making there own entrypoint-thing due to this19:42
smosernot too long ago i had a spinning disk19:43
smoser(more embarrasment)19:43
harlowjawhats that crap19:43
smoserand running 'nova' on it took like 3 seconds to load.19:44
smosernova as in the cli tool, not the service :)19:44
harlowjaso ya, the other option is that we make our own loader slightly more advanced19:48
harlowjaso that say in cloud.cfg u could have fully specified modules + functions19:48
harlowjathen i could have a entry like19:48
harlowjathough that starts to just make our own entrypoint like thing :-/19:49
smoserok. one more thing.. http://paste.ubuntu.com/23062607/19:54
smoserfor my reference mostly.19:54
smoserthat just runs it with strace too, and counts stats or opens19:54
harlowjastats or opens: 256119:54
harlowjaya, idk why they aren't backing that crap via sqllite19:54
harlowjaafaik entrypoints are all 'static'19:54
harlowjain that they are all defined by packing (in setup.py or other)19:55
harlowjaseems dumb to rescan the filesystem to find them19:55
harlowjait'd seem like a win for most of python if it wasn't so scan happy19:56
harlowjathough of course any change to do that would probably hit the people that will say its all in cache and such and blah blah19:56
harlowjaand gets into the question, of make our own thing, or just work with the python stuffs19:59
smoserso i had a start of my own thing20:06
smoserthat took a list of directories20:06
smoserand would look in those.20:06
smosercloud-init needs lots of performance improvements for sure20:06
harlowjawhy not at that point just explicitly name full modules in cloud.cfg ?20:08
harlowjai'd rather not make our own full entrypoint thing :(20:08
harlowjaor just at least, try to talk to python-devs, asking what's a solution (is there any, is sqlite db possible, or a static file that everyone updates or ...)20:09
smoserok... i'm just saying this out loud for my own logs and such.20:22
smoserthat is a bzr revno to git hash mapping that seems correct for right now.20:22
mgagneso a coworker tested the "fixed" cloud-init and had some form of race condition, the gateway and routes weren't properly configured. rebooting fixed the issue.20:54
=== rangerpb is now known as rangerpbzzzz
mgagnewill do more tests tomorrow21:17

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!