/srv/irclogs.ubuntu.com/2015/09/04/#cloud-init.txt

=== natorious is now known as zz_natorious
=== natorious is now known as zz_natorious
Odd_Blokeclaudiupopa: harlowja: I like the look of taskflow, but there are a couple of problems: (a) it isn't packaged for Python 3 in Debian/Ubuntu, and (b) it pulls in a lot of dependencies that I don't think we'll use (e.g. MySQL/PostgreSQL drivers, Zookeeper library).11:41
Odd_Blokeclaudiupopa: harlowja: smoser: So it looks like there are a few hurdles to getting a python3-taskflow building, but they probably aren't insurmountable for 16.04.12:39
Odd_BlokeBut I don't really want to try and get it sorted unless we are actually going to use taskflow. :p12:41
Odd_BlokeI expect that we could also convert quite a few of those dependencies to Recommends/Suggests.12:42
smoserOdd_Bloke, one thing to do is just file a bug13:19
smoserand tell openstack team.13:19
smoserthey have to solve it at some point.13:19
Odd_Blokesmoser: Ah, true.13:20
Odd_Blokesmoser: Will they have to solve it for/by 16.04, do you know?13:20
smosernot really, but i filed a ton of "no python3" and they did get fixed.13:21
Odd_Bloke(It turns out there's also some pain with building both python-taskflow and python3-taskflow at the same time, python{,3}-networkx conflict with one another)13:22
smoserhttps://bugs.launchpad.net/ubuntu/+source/python-novaclient/+bug/131914513:22
smoserOdd_Bloke,  you can even just file in debian :)13:23
Odd_Blokesmoser: I've filed https://bugs.launchpad.net/ubuntu/+source/python-taskflow/+bug/149226713:24
Odd_BlokeCan I file a Debian bug and link them somehow?13:25
smoseryou can, yeah.13:33
smoseruse 'reportbug' to file debian bug13:34
smoserand then you can link it in launchpad13:34
Odd_BlokeDone.13:41
smoserOdd_Bloke, question..13:45
smoserdo you knwo if i can specify a bzr repository to tox13:45
smoseri want to use simplestreams in a tox, but dont want to push to pypi13:46
Odd_Blokesmoser: You mean as a requirement to install?13:49
smoseras a 'deps', yeah13:49
Odd_BlokeYou can do it with pip via https://pip.pypa.io/en/latest/reference/pip_install.html#bazaar13:49
Odd_BlokeSo you should be able to get tox to do it.13:50
=== zz_natorious is now known as natorious
openstackgerritDaniel Watkins proposed stackforge/cloud-init: Configure basic logging, and make it possible to log to console.  https://review.openstack.org/22053614:42
openstackgerritDaniel Watkins proposed stackforge/cloud-init: Use a single source for version information.  https://review.openstack.org/22054314:53
smoserOdd_Bloke, you can. i did, its nice.14:54
smoserhttp://paste.ubuntu.com/12273678/14:54
Odd_Bloke:)14:55
=== natorious is now known as zz_natorious
Odd_Blokeharlowja: I'm playing around with TaskFlow; I was wondering if you have any recommendations for handing objects around as results/arguments.16:11
Odd_Blokeharlowja: I've just switched to the dir/files storage implementation, and it can't serialise a DataSource to disk (unsurprisingly).16:11
=== zz_natorious is now known as natorious
=== natorious is now known as zz_natorious
Odd_Blokeharlowja: OK, maybe a more general question: I want to be able to do things in discrete runs of cloud-init (e.g. run it once to configure networking, once to find a data source, and once to do configuration).16:36
Odd_Blokeharlowja: So I figured that I could do this by re-using a logbook (and maybe a flow, but I haven't managed to get that far yet).16:36
Odd_Blokeharlowja: But DirBackend.get_connection().get_logbooks() consistently returns no items.16:37
Odd_Blokeharlowja: There are loads of logbooks stored on disk, but it only looks for them if they are links.16:37
Odd_Blokeharlowja: Am I hitting a bug in my understanding, or a bug in taskflow? :p16:37
harlowjaOdd_Bloke yo yo16:43
Odd_Blokeharlowja: o/16:44
harlowja\o16:44
harlowjaha16:44
harlowjaso did u get the handing objects around working?16:44
Odd_Blokeharlowja: Nah, I moved on to trying to get all the different bits working together.16:45
Odd_BlokeBecause if I can't get the different runs to see the same data, then it doesn't matter what I can store. :p16:45
harlowjaso let's see here16:45
harlowjadr.josh on the case16:46
harlowjaha16:46
Odd_Blokeharlowja: I think this might be a legit bug; books are always written as directories, but _get_children will only ever return links.16:46
harlowjaya, it might be16:46
harlowjalegit bug ftw16:47
harlowjaha16:47
harlowjai don't think many people have been using the dir backend, most afaik have use the sql one16:47
harlowjaor the zookeeper one16:47
harlowjabtw, as for dependencies, most are actually optional, depending on what u use16:47
harlowjai should probably reorganize them now that better optional dependency support exists in pip16:48
Odd_Blokeharlowja: Yeah, I figured they would be.16:48
Odd_BlokeSo hopefully it's mostly just shifting them from Depends to Suggests.16:48
harlowjaya16:48
harlowjai think pbr only recently got support for this, so thats part of it16:48
harlowja* support for https://www.python.org/dev/peps/pep-0426/#extras-optional-dependencies16:48
harlowja*thats part of why16:48
Odd_BlokeWell, let me switch to using sqlite for now.16:49
harlowjak16:50
harlowjabut now u guys know my other project :-P16:50
harlowjaha16:50
harlowjaok, https://bugs.launchpad.net/taskflow/+bug/1492392 opened for optional deps16:53
harlowjahttp://paste.openstack.org/show/445602/ another example Odd_Bloke16:56
harlowjathat one dumps out the in-memory backend16:56
Odd_Blokeharlowja: So I'm trying to work out how I should do this resuming stuff.16:58
Odd_Blokeharlowja: Should I be building the entire cloud-init flow up front, and then different commands can do different parts of it?16:59
Odd_BlokeOr can the commands just build their own, smaller flows which can be extended/resumed by later commands.16:59
harlowjai've seen people do both17:00
harlowjabuilding up-front allows for more parallelsim17:00
harlowjabuilding smaller ones and executing them allows for less17:00
harlowjabuilding up front gets complicated if u need to do conditionals, this kind of programming model (the one taskflow has, typically called dataflow-like) does require some brain twisting, since its basically ahead-of-time defintion of all the things :-P17:02
harlowjaso i'd start out simple (which seems to be what most people do)17:02
harlowjaOdd_Bloke here is an example that uses a prior runs data17:04
harlowjahttp://paste.openstack.org/show/445605/17:04
harlowjaoutput @ http://paste.openstack.org/show/445608/17:05
harlowja^ will avoid recomputation17:05
Odd_BlokeAha, I think I'm missing the models.FlowDetail part.17:05
Odd_BlokeI was trying to do something messy with logbooks.17:06
harlowjaya, that might be part of it, the bug probably is still legit though17:06
Odd_Blokeharlowja: taskflow.exceptions.NotFound: No flow details found with uuid '6c1a3f09-8a58-426f-bffb-8262680fcb67'17:10
Odd_Blokeharlowja: Using sqlite on the first run with it added in.17:10
harlowjaya, k, the memory backend is sorta different in that case, so u have to do a little more for other backends17:10
harlowjabasically https://github.com/openstack/taskflow/blob/master/taskflow/examples/resume_from_backend.py#L106 (those 5 lines)17:11
harlowjathe in-memory stuff sorta automatically saves the provided flow detail when a backend is not provided (because its an in-memory one, and the default)17:11
Odd_Blokeharlowja: OK, so I'm persisting and fetching the same flow_detail now.17:20
harlowjak17:20
Odd_Blokeharlowja: What I have ATM is two Flows A and B.  A produces 'data_source' and B requires 'data_source'.  I'm running A on one invocation of cloud-init (i.e. e = engines.load(A, ...); ...; e.run()) and B on the second.17:21
harlowjak17:21
Odd_Blokeharlowja: But B always fails because 'no other entity produces said requirements'.17:21
roychriIm trying to find how to access my ec2 metadata (specifically the private_ipv4) so I can use it in bootcmd, runcmd or write_files. Any pointers?17:22
harlowjaOdd_Bloke can u pastebin the code u have?17:22
harlowjaOdd_Bloke https://bugs.launchpad.net/taskflow/+bug/1492403 also (for the dir stuff); fixing that right now17:23
openstackgerritDaniel Watkins proposed stackforge/cloud-init: [WIP] TaskFlow for running shell commands  https://review.openstack.org/22059317:23
harlowjais that the pastebin ;)17:24
Odd_BlokeYep. :p17:24
harlowja:)17:24
harlowjaOdd_Bloke can u also try turning log level 5 on, that will show the symbol lookup resolution17:25
* harlowja aka the 'trace' log level17:25
harlowjathe lookup loggin should show u why / what is being searched for and found17:25
Odd_Bloke"2015-09-04 18:26:14,869 [Level 5] taskflow.storage: Looking for 'data_source' <= 'data_source' for atom named: cloudinit.flows.get_config_flow.<locals>.PlaceHolderTask"17:26
harlowja(yes i know thats a non-standard log level, but the some openstack people complained about it being to noisy, ha)17:26
Odd_Blokeharlowja: So if you look at flows.py, all works fine but doing the two separately doesn't.17:27
Odd_BlokeWhich makes sense.17:27
Odd_BlokeBut I don't know how I should go about doing those two separately.17:27
harlowjacan u insert the search flow (Even if it already finished) into the config_flow17:28
harlowjai'm pretty sure that it needs to know the names of providers, so if u don't insert it, it doesn't know about the other prior tasks that already saved stuff17:28
harlowjaso u could either add dummy tasks, or just insert the search flow17:28
Odd_BlokeOK, that did work.17:29
Odd_BlokeBut that makes inserting earlier steps tricky.17:29
Odd_BlokeSo I'm wondering if I should actually just be building the whole flow up-front, and using a targeted graph thingie just to execute up to where I want?17:29
harlowjathat could work to17:29
harlowjamost of the advanced users use the graph stuff, its more powerful imho17:30
Odd_BlokeOr maybe it's just the way I'm defining things that makes it seem messy doing it this way..17:30
harlowjai think the graph stuff would make it better, then running up to a point17:31
Odd_BlokeCool, I'll try that.17:31
openstackgerritDaniel Watkins proposed stackforge/cloud-init: [WIP] TaskFlow for running shell commands  https://review.openstack.org/22059317:31
Odd_BlokeThat's the fixed version of what I have now.17:31
harlowjabasically the argument providers aren't persisted, aka, what each task provides, so without those being inserted, the lookup mechanism goes 'idk where that is' for later tasks that want to use it17:31
Odd_BlokeRight, that makes sense.17:32
harlowjanow maybe those should be saved17:32
harlowjanot especially hard, just isn't right now17:32
Odd_BlokeI'd assumed that they were shoved in to the same global dict as the stuff you give as store={...}.17:32
Odd_BlokeBut I think we'll want to transition to the graph stuff anyway.17:33
Odd_BlokeSo I'll try that.17:33
Odd_BlokeAnd if it's too hard then you can fix the library to do whatever I want it to do. ;)17:33
harlowjanah, during the compile() step the whole tasks are validated against who provides what (by name) so its a little more complicated :-P17:33
harlowja:(17:33
harlowjahaha17:33
harlowjahttp://docs.openstack.org/developer/taskflow/engines.html#scoping is basically the 'lookup algo'17:33
harlowjahttp://docs.openstack.org/developer/taskflow/engines.html#taskflow.engines.action_engine.scopes.ScopeWalker.__iter__ (the meat of it)17:34
harlowjaOdd_Bloke as long as other openstack projects aren't affected by those changes then i guess i can change it for u :-P17:34
harlowjaha17:34
Odd_BlokeNah, they can work around us.17:34
harlowjaopenstack/other projects (rackspace uses this for http://www.rackspace.com/cloud/big-data afaik)17:34
harlowjalol17:34
Odd_BlokeSee if their instances will boot if cloud-init breaks. ;)17:35
harlowja:-P17:35
harlowjahave u figured out the parallel stuff yet btw?17:35
Odd_Bloke"the parallel stuff"?17:35
harlowjahttps://github.com/openstack/taskflow/blob/master/taskflow/examples/hello_world.py#L8217:35
harlowjaexecuting non-dependent tasks at the same time17:35
Odd_BlokeOh, not yet.17:35
harlowjak17:36
harlowjanp17:36
harlowjathats step 1.5 of the 4 step taskflow program17:36
Odd_BlokeEverything's been one big linear flow up until now, so it wouldn't have made a difference anyway, right?17:36
harlowjalol17:36
harlowjaright, it wouldn't have17:36
Odd_BlokePhew.17:36
Odd_BlokeI understood something. :p17:36
harlowjaya, linear stuff not so paralleizable17:36
harlowja*where not so == not at all, lol17:37
Odd_Blokeharlowja: So how should I target this execution?17:39
Odd_BlokeI can't just say "until data_source is set", right?17:39
harlowjanah, u have to set the node to run 'up to'17:40
harlowjahttps://github.com/openstack/taskflow/blob/master/taskflow/patterns/graph_flow.py#L31417:40
harlowjaso set_target(config_task_obj)17:40
harlowjaor whatever17:40
Odd_Blokeharlowja: It works. \o/17:48
harlowjawoot17:48
harlowjadamn, amazing shit17:49
harlowjaha17:49
Odd_BlokeWho wrote this?17:49
Odd_BlokeThey must be amazing!17:49
harlowjamostly me :-P17:49
harlowjaha17:49
Odd_BlokeOh...17:49
Odd_BlokeNever mind.17:49
Odd_Bloke;)17:49
harlowja:-P17:49
harlowjaand thats just part 1 of its awesomeness17:50
harlowjalol17:50
openstackgerritDaniel Watkins proposed stackforge/cloud-init: [WIP] TaskFlow for running shell commands  https://review.openstack.org/22059317:50
harlowjastuff that won't be likely used by cloud-init, but is used by others http://docs.openstack.org/developer/taskflow/jobs.html17:50
harlowjaonce u have resumption, now imagine tying a job to a set of tasks, and having that job be placed somewhere for others to work on17:50
harlowjaand if those job 'workers' crash the job gets resumed by others that then try to finish it...17:51
harlowjaannnnd magic17:51
harlowjaha17:51
harlowjaOdd_Bloke https://docs.google.com/presentation/d/1EZoY4FE2SDjfCqMCgBRrwo7ovHF4vYZFGKjjvFvWHIE/17:51
harlowjathat might be useful now that u have seen some of it, ha17:51
harlowja(a talk i did with the hp folks)17:52
Odd_Blokeharlowja: That demo sucked.17:53
Odd_BlokeBut the rest of the slideshow was helpful.17:54
harlowjalol17:54
harlowjahttps://github.com/openstack/taskflow/blob/master/taskflow/examples/99_bottles.py#L39 :-P17:54
harlowja^ the demo, ha17:54
harlowjaDIY17:54
harlowjalol17:54
harlowjaDIY demo17:54
roychriIm trying to find how to access my ec2 metadata (specifically the private_ipv4) so I can use it in bootcmd, runcmd or write_files. Any pointers?17:58
Odd_Blokeroychri: You could hit the EC2 metadata server yourself.17:59
roychriusing curl?17:59
roychrithat will work in runcmd, but write_files ?18:00
Odd_Blokeroychri: I don't know that you can do it in write_files.18:00
Odd_Blokeroychri: Though you can, of course, write out files with your runcmd.18:00
roychriok, I thought the metadata could be available thru some variables of some kind...18:00
Odd_Blokeroychri: You're right, let me dig up the docs on what's available.18:00
roychriI skimmed thru the datasources docs, I couldn't find anything there...18:01
Odd_Blokeroychri: Actually, looking at it, you can't do substitution with write_files.18:03
Odd_Blokecc_final_message lets you do some substitution.18:03
roychriMaybe I should just use #!/bin/bash instead of #cloud-config18:03
Odd_Blokeroychri: You can send multi-part cloud-config over.18:04
Odd_Blokeroychri: Which would let you have the best of both worlds.18:04
roychriIm not there yet :)18:04
roychriI'll get my first instance to work, then I'll look at multipart.18:05
openstackgerritDaniel Watkins proposed stackforge/cloud-init: [WIP] TaskFlow for running shell commands  https://review.openstack.org/22059318:05
Odd_Blokeharlowja: Could you do a sanity check of ^ before I spend any time firming it up with tests/logging/etc.?18:09
harlowjaodd is 'PlaceHolderTask' going to stay?18:09
harlowjaOdd_Bloke ^18:09
harlowjaotherwise seems like a good start to me18:10
Odd_Blokeharlowja: That represents "all the configuration", so I'll probably move that out to a separate module to be a start for that.18:10
harlowjak18:10
Odd_Blokeharlowja: How do you feel about the 'select an arbitrary UUID' approach to identifying flows?18:11
harlowjaunsure, mixed feelings :-P18:11
harlowjai prefer useful names :-/18:12
Odd_Blokeharlowja: They require a UUID, though.18:12
harlowjafor the flow detail storage part, ya18:12
Odd_BlokeYeah, that's the bit I meant.18:13
harlowjaso i guess arbitrary UUID is fine then, its really used to reconnect with later runs (if the same names are used)18:13
Odd_Blokeharlowja: Actually, UUIDs aren't just random, I might be able to do something more sensible.18:14
Odd_Blokes/aren't just/don't have to be/18:14
Odd_BlokeBlargh, they always have a part which is a UUID.18:15
Odd_BlokeIn [37]: uuid.UUID(bytes=b'cloudinitiscool!')18:17
Odd_BlokeOut[37]: UUID('636c6f75-6469-6e69-7469-73636f6f6c21')18:17
harlowjalol18:17
harlowjaso there is really no check that u are providing a uuid, btw, u can probably  just provide 'cloudinitiscool!' as flow detail uuid arg18:18
Odd_BlokeOh, cool.18:18
harlowjanot saying thats the best, but perhaps this could be tweaked in taskflow, lol18:19
harlowjato be named 'ident' or something instead18:19
harlowjavs uuid, lol18:19
harlowjabut bygones be bygones, ha18:19
Odd_BlokeUntil that happens, I'll probably keep it as a UUID.18:20
Odd_BlokeBecause who knows if you'll end up going in the other direction and enforcing UUIDs? :p18:20
harlowja:-/18:20
harlowjaha18:20
harlowjaok, https://review.openstack.org/#/c/220607/ fixes the get_logbooks for dir(s)18:20
harlowjathx for finding that Odd_Bloke18:22
Odd_Blokeharlowja: No worries, thanks for fixing it. :)18:25
harlowjasureee18:25
harlowjabtw18:26
harlowjauuid.uuid5(uuid.NAMESPACE_URL, "https://launchpad.net/cloud-init")18:26
harlowjathat seems to always create18:26
harlowjaUUID('bbd9656b-aed9-5912-9702-7ddde940f8f6')18:26
harlowjaso that could be your uuid :-P18:26
Odd_BlokeOh, nice.18:26
harlowja(pick other url as u want, ha)18:26
Odd_BlokeI hadn't seen that there were hardcoded namespaces you could use.18:26
roychriWhy does runcmd have two formats (string and array)? In what use case I should use one over the other?18:32
harlowjaOdd_Bloke and someone from yahoo is taking over https://bugs.launchpad.net/taskflow/+bug/1492392 so that hopefully will get done soon to18:34
harlowja(the splitting the optional/non-optional taskflow deps up)18:34
Odd_BlokeAh, yeah, that would make sorting it out in packaging downstream a lot easier.18:35
Odd_Blokeharlowja: Thanks!18:35
harlowjaya18:35
harlowjanp18:35
harlowjasmoser u been keeping track of all of this?? ;)18:35
harlowjashit about to get cray cray18:36
harlowjalol18:36
harlowjaha18:36
smoserbeen keeping track of nothing18:36
harlowjalol18:36
harlowjawoot18:36
harlowjaOdd_Bloke also #openstack-state-management channel if u ever don't find me for questions :-P18:46
harlowjasomeone else in there might know to, ha18:46
=== zz_natorious is now known as natorious
=== natorious is now known as zz_natorious

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!