=== cortana [n=sam@62-31-146-25.cable.ubr12.azte.blueyonder.co.uk] has joined #upstart
=== wasabi_ [n=wasabi@cpe-76-184-122-13.tx.res.rr.com] has joined #upstart
=== MasseR [n=masse@dsl-tkubrasgw1-fe11fa00-85.dhcp.inet.fi] has joined #upstart
=== Keybuk [n=scott@quest.netsplit.com] has joined #upstart
AlexExtremetrying to code shortly after you wake up never works too well10:01
Keybukyeah I find that10:09
AlexExtremei'm trying to work on this profiles code and my brain just says "uhh?" :p10:10
Keybukhave a bananananana :p10:12
AlexExtremeoh well, bbl10:19
=== Md [i=md@freenode/staff/md] has joined #upstart
=== Keybuk tackles the problem of namespace collision
Keybuktoo many structs :-/12:08
Keybukconf_sources -(hash)-> ConfSource -(hash)-> ConfFile -(list)-> ConfItem -> job/state/etc.12:08
Keybukto track names, I'd need12:08
Keybuknamespace -(hash)-> Name -(list-> job/state/etc.12:08
=== Keybuk wonders whether objects of different types should share namespaces
Keybuke.g. should states and jobs share a namespace?12:39
=== Md [i=md@freenode/staff/md] has joined #upstart
=== sadleder [n=sadleder@p50813612.dip0.t-ipconnect.de] has joined #upstart
=== sadleder [n=sadleder@p50813612.dip0.t-ipconnect.de] has left #upstart []
=== sadleder [n=sadleder@p50813612.dip0.t-ipconnect.de] has joined #upstart
=== sadleder [n=sadleder@p50813612.dip0.t-ipconnect.de] has left #upstart []
=== asdaf [n=Ack@213-140-11-128.fastres.net] has joined #upstart
=== tale [n=tale@] has joined #upstart
=== ion_ [i=ion@heh.fi] has joined #upstart
wasabistates and jobs?04:10
wasabistates are jobs, no?04:10
wasabiyou mean something other than what I was thinking you did, then04:11
Keybukan example of a state is the period between tty-added and tty-removed for the same $TTY04:12
Keybukthis state can have multiple concurrent instances, since you can have multiple $TTYs04:12
wasabiHmm. This is all getting very confusing. I like dit when you needed one file per tty. :004:12
wasabiIt made the idea of a state easy: while job-name04:13
KeybukI like the idea of one file for all ttys04:20
Keybuksince they're identical04:20
=== Amaranth [n=travis@ubuntu/member/Amaranth] has joined #upstart
=== phoenix24 [i=shp@ns38066.ovh.net] has joined #upstart
wasabiJust being identically doesn't make it automatically a case to combine.05:43
wasabiWhat was wrong with our initial idea of jobs themselves defining named states?05:43
Keybukdoesn't work for the tty case05:44
wasabiActually, I guess I don't really even know what I'm talking about anymore. Ya'll have probably done a lot of work since I was last in on it.05:44
Keybukor the network interface case05:44
Keybukit works on paper, but not for the use cases it's actually needed for05:45
wasabiExplain the tty case?05:45
Keybuknetwork interface is less controversial, so let's use that as an example05:45
wasabiOkay, that. ;005:45
Keybukwe have a pair of events with a common variable05:45
Keybukinterface-up eth005:45
Keybukinterface-down eth005:45
Keybukso we can define the pairing and name that, say, interface-is-up05:45
Keybuk  interface-up ... interface-down $IFACE05:46
Keybukso when any interface comes up, the state is true05:46
Keybukand when that same interfaces goes down, the state becomes false05:46
Keybukcomputers have multiple interfaces05:46
Keybukwe don't just want to track the first one that we see, we want to track them all05:47
Keybukso when we see "interface-up lo", the state is true "for lo"05:47
Keybukwe might next see "interface-up eth0", now the state is true "for lo" and "for eth0"05:47
Keybuknext we might see "interface-down eth0"05:47
Keybukthis only matches the second half of the "for eth0" true state, so that state becomes false05:47
Keybuknow the state is only true "for lo"05:47
Keybukby thinking in this way, we can answer the questions05:48
Keybukis the state true for any interface (any network interface is up!)05:48
wasabiI'd start with a network job, which only started when Any interfaces were up, and stopped itself when the last interface went down. You can then depend on the start/stopped of that job to define a state where any interaface is up.05:48
Keybukis the state true for a specific interface (or any non-lo interface)05:48
Keybukwasabi: but that involves defining a job that tracks the up/down events it receives, no?05:49
wasabiYes, it does.05:49
Keybukhow would you define a job that was running while any interface, apart from lo, was up?05:49
wasabiIt would start on any interface up, and stop on any interface down. And the pre-stop handler would check if ALL interfaces were down, if not, it wouldn't really stop.05:49
wasabiIn code.05:50
Keybukthe nice things about having first-class states is that you don't need to do that05:50
Keybukupstart can track that for you05:50
Keybukin fact, we then get to do things like05:50
Keybuk"is the set of paths that are mounted a subset of the following list?"05:50
Keybukso upstart itself can track the problem of the writable filesystem05:50
wasabiNot sure if that's completely beneficial to implement that way or not. The script thing, as far as I can tell, works.05:51
Keybukjobs implicitly have a state coupled to them05:52
Keybukso by defining a job, you are also defining the state which defines when they are running05:52
Keybukbut I figured that since the machine is sufficiently useful, one should be able to define states in their own right, for use in the definitions of other jobs05:52
Keybukthe /etc/init/conf.d/udev example holds here; where udev ships a rule that calls initctl for appropriate events, and ships upstart states for each of the event pairs05:53
Keybukso a job doesn't have to worry about module-loaded ... module-removed05:53
Keybukyou could do this simply by defining jobs for them all05:54
Keybukbut they would have to be all instance jobs05:54
wasabiAll the modules? Not worth it. They are very situation specific.05:54
Keybuk(an alternative viewpoint is to make all jobs instance jobs by default :p)05:54
wasabiWhy is from module-loaded foo to module-removed foo so bad?05:55
wasabiYa know, another thing I'm concerned about in all of this is starting a job midstream.  What if you install the job file while the foo module is loaded, does it sit there waiting for foo to show up?05:56
wasabiOr are you going to always model all system state in upstart at all times?05:56
wasabiI don't think so. I think you'd install the job, and ask it to start, right now. And it would check if the module was loaded, in pre-start.05:56
wasabiAnd this is all weird anyways because it's inherently race. You can never guarentee that the module didn't remove itself after the job was started.05:57
wasabiSo every job still has to verify that the system is proper, it has to check if the module is loaded.05:57
wasabiSo inevitably each script will have sanity checks in pre-start. Regardless how much help upstart gives it.05:58
wasabiAll of this together makes me wonder if it's not getting just too complicated.06:03
wasabiMaybe I'm just being negative today.06:06
wasabiYou could do network interfaces like that today.06:08
wasabiinterface-up job which fires when udev tells it an interface is up. It can itself main some state files in /var/run or something.06:08
Keybukbelieve it or not, this way is simpler to implement than having raw jobs as states06:08
wasabiAnd it can keep track itself what interfaces are or are not up... and emit events for specific interfaces.06:09
KeybukI think that services will almost always describe the states in which they should be running, rather than being directly event based06:09
Keybukie. "while there is a network interface up, and the filesystem is writable, and dbus is running"06:09
Keybukif true, the state graph can be evaluated when the job is created, so yes, it would start automatically06:10
Keybukit is inherently racey, so the service should fail normally if the resources it expects are not available06:10
wasabiIt can only be evaluated if the state which is defined by events is installed before those events happen.06:10
Keybuk(this is not unreasonable)06:10
Keybukright, that is true in the current upstart model06:11
Keybukupstart would need to record all events to avoid that06:11
wasabiYeah, and that's probably unreasonable.06:11
wasabiSo, somebody is still going to have to, after installing a new job with a new state, give it a push.06:11
wasabiWrite some code which checks if the state is true by evaluating the system.06:11
KeybukI think it will be rare that this is true, no?06:12
=== mbiebl [n=michael@e180071063.adsl.alicedsl.de] has joined #upstart
wasabiI'm not sure.06:12
Keybukjobs that should be started in postinst will rarely need to define a state based on events06:12
Keybukbut yes, that is a concern06:12
Keybukthere's more interesting examples06:12
wasabiBut if those states were maintained by stateless jobs, it's not a concern.06:12
wasabiSo you have a network job, which fires anytime a network interface, any interface, comes up or down. It keeps a count in /var someplace about the total number of active interfaces, by actually checking the interfaces, not reading the events.06:13
mbieblKeybuk: hi06:14
wasabiAnd a job which cares about the network would need to start on any network event also, and check that file. Or something.06:14
wasabiI don't know.06:14
wasabiWhich is still racey.06:14
Keybukthe same network-monitoring job could register the states in upstart's memory directly06:14
wasabiIt would have to check itself.06:14
Keybukavoiding the use of filesystems06:14
wasabiBut then we're just talking about reusable scripts.06:14
Keybukfiddly scripts :)06:15
wasabiExplain how states in upstart are maintained?06:15
wasabiIt makes me wonder if what is being built doesn't actually solve any issue, is all.06:16
wasabiSince hte issue is still there.06:16
Keybukwhat issue do you think we're attempting to solve?06:16
wasabiEvery job that cares about network has to actually check the network and exit gracefully in pre-start.06:16
Keybukwhy pre-start?06:16
wasabiOr start.06:17
Keybukit can exit ungracefully in main06:17
wasabiTrue, you are correct, but still, every job has to do that.06:17
Keybukand log in syslog that it was unable to bind to the interface06:17
Keybuk*shrug* every job does that already if it's checking the return codes of its syscalls like a good daemon06:17
mbieblKeybuk: I've got two questions. 06:17
wasabiSo, by recording states, at all, in upstart or otherwise, what are you solving? They may be recorded in upstart, but jobs still have to check on their own properly.06:17
wasabiSo why not just let jobs do that?06:18
Keybukwasabi: solving the reattempt to start the job issue06:18
mbieblFirst: Is the Ubuntu udev patched to create /dev/console,null and the std* symlinks?06:18
Keybukthe job doesn't just have to check it, it has to accept all possible states can fail intermittently, and fallback to some kind of "waiting for appropriate state" inner loop06:19
Keybukif it cannot bind() to the interface, it has to loop until it can06:19
wasabiBut doesn't the job have to reeveluate that in that loop on EVERY event that might contribute to the state?06:19
Keybukperhaps with some kind of asynchronous notification from an interface daemon that a new interface is up, to reattempt the bind06:19
Keybukthis is the launchd model, btw06:20
wasabiWell, no. If it can't bind, it dies. And starts again next time something that might make it work appears.06:20
wasabiYeah, I know.06:20
Keybukit doesn't start again06:20
Keybukbecause nothing will restart it06:20
Keybukit failed, bad bad job06:20
Keybukwhat upstart provides is that loop06:20
Keybukthe job defines what state it likes06:20
wasabiEh? If it's waiting for both the file system and network, if any of those happen, it will restart.06:20
Keybukand upstart guarantees that it will attempt to start the job every time the system is in that state06:20
Keybukand that it will kill the job when the system goes out of that state06:20
wasabiAnd check to see if both conditions are acceptable. So, when the file system comes up, it will start and look for the network. No network? Okay die. Network comes up a few minutes later and it starts again. Checks again and starts successfully.06:21
=== phoenix24 [i=doy@ns38066.ovh.net] has joined #upstart
Keybukit cannot guarantee that the state will remain true for any period after the initial "this is true"06:21
Keybukbut it can guarantee that the job will be killed again if it hasn't noticed06:21
Keybuk*and* it guarantees that the job will be restarted if the state should become true again06:21
Keybukwasabi: sorry, I appear to have confused you06:21
wasabiI confuse easily these days.06:21
Keybukwasabi: my initial description of the looping application is the upstart-less world06:21
wasabiI think my brain has been leaking a lot lately.06:21
Keybukthe launchd model:06:22
Keybuk - all jobs are started immediately06:22
Keybuk - if a resource the job needs is not available, the job should sleep until it is available06:22
Keybuk - if a resource the job needs becomes unavailable, the job should sleep until it is available again06:22
ion_As far as ive understood what Keybuk has been describing, it sounds good.06:23
Keybukie. monitor your syscalls, if any fail due to an error (bind fails, write fails, etc.) you should fall into a kind of slumber loop06:23
Keybukhow you wake yourself up from that slumber loop is anyone's guess06:23
Keybukasynchronous notification of the potential availability of resources?06:23
Keybukor maybe you just use blocking writes and blocking binds? :p06:23
wasabiYa know, I don't really mind that model, except for the wake up part.06:23
wasabiBut upstart has the wakeup part, in events.06:23
wasabiSo slumber, but get poked when an event happens.06:24
Keybukso what upstart provides is the acceptance of the reality that states come and go06:24
wasabiWhere slumber == "just exit and let upstart start you again later"06:24
Keybukit guarantees that you'll be started when the state is true, and stopped when it becomes false06:24
Keybukso you can assume that any failure to obtain resources is bad, and just exit(1)06:24
Keybukbecause you'll get restarted again next time you can have a go06:24
wasabiExcept that those states are inherently hard to monitor, and can't be relied upon anyways.06:24
Keybukstates are easy to monitor06:25
wasabiSince they might be untrue by the time you get around to actually running.06:25
mbieblKeybuk: What for ressources that upstart can't easily monitor?06:25
Keybukand can be relied on to be true06:25
Keybuksure, they can become false again06:25
Keybukupstart says it will stop you if that happens06:25
Keybukok, you might hit the failure first, but the worst thing there is a syslog entry06:25
mbieblE.g. remote services that are required, e.g. tomcat requiring a remote sql service.06:25
Keybukbut you *will* get restarted next time the state is true for a while06:26
wasabimbiebl: I really don't think upstart offers anything there. You'd want tomcat and the database to be started independently of ech other... as there might be tomcat services that don't need the database.06:26
Keybukapps become "just assume that syscalls should work, check the return value, and bail out if they don't"06:26
Keybukupstart takes care of restarting you when the state is true again06:26
wasabiWell, okay. Yeah. I like that... but that's how it is without upstart monitoring states too.06:26
mbieblWell, how is upstart supposed to know the state "remote sql service available"06:27
mbieblMy point is, there are states, that upstart can't provide.06:27
wasabimbiebl: It's not. You'd have to implement something which feeds that state to upstart.06:27
wasabiKeybuk: apps can function in the maner you describe whether upstart watches states or not. Since upstart *will* start it again when any event happens that might make it runnable.06:28
Keybukupstart doesn't *monitor* states06:28
Keybukupstart just provides a state whiteboard for everything else06:28
Keybuke.g. heartbeat could do it for the remove case06:29
Keybukremote case06:29
Keybukor monit06:29
Keybukor whatever06:29
Keybukthey just emit events which upstart can combine into states06:29
Keybukor can set states true/false directly06:29
Keybuk(registered through the usual initctl/libupstart layer)06:29
wasabiHmm. You can set states.06:29
KeybukI don't see why not06:29
wasabiOkay, so a postinst script should set appropiate states.06:29
wasabiThat solves that.06:29
wasabiA new state, for instance.06:30
Keybukpostinsts for new things might end up having some kind of udevtrigger-a-like06:30
Keybukin fact, since many deviceish states will come with udev and HAL, udevtrigger is all you'd run :p06:30
wasabipostinst: "hi upstart, I know this state is valid from x to y and n to m, but I just checked x and n, and it's good. So set it right now."06:30
Keybukinitctl set wibble true06:31
wasabiSo actually, a postinst for a new job might in fact set all states for that job to true without checking.06:31
Keybukmbiebl: remote services should be easy with heartbeat or monit06:31
wasabiAnd the job might fail. But that's okay.06:31
Keybukwasabi: or it could just "start" the job *shrug*06:31
wasabiWell, if it starts the job, and the job exits, the might might want to be started again properly.06:32
wasabiEven though events that contribute to the state the job cares about aren't set.06:32
Keybukmbiebl: A to question 1 -- ubuntu's udev copies /lib/udev/devices into /dev before starting, that directory contains the usual console, null, etc. devices06:32
wasabiI am jumbling all my words. I have no idea why I do that.06:32
wasabiLet me try again.06:32
mbieblKeybuk: Well, you'd still have to patch heartbeat to emit upstart events.06:32
wasabijob: from X to Y and N to M.   The postinst runs. Currently X is true and N is false, but upstart doens't yet know. So the postinst starts the job. The job exits because N is false (syscall fails). N becomes true, but since X isn't yet known, the job doesn't start again.06:34
wasabiHence the postinst has to set X and N to the proper values at the time of being installed, and let new events from that point on alter them.06:34
Keybukpostinsts for packages registering new states, or jobs that use unique states, should make an effort to check whether the requisites are true and set the state accordingly06:36
wasabiAnd what sets the state?06:36
Keybuke.g. a package installing a state that says whether users are logged in should perhaps look at utmp06:36
wasabiIn the normal case?06:36
Keybukin the normal case, the state would be set by the daemon or by events06:36
wasabiWhere are teh events for the state defined?06:37
wasabiAnd are states binary?06:37
Keybukby some daemon or other?06:37
wasabiHmm. I mean for, like the network interface case.06:37
wasabiWho watches the interfaces and sets the states?06:37
Keybuknot quite following06:37
Keybukwhoever installs the postinst06:38
Keybukupstart does no watching06:38
Keybuknetwork interface case => udev06:38
Keybukor maybe Network Manager06:38
KeybukI can't remember whether that one comes via udev, HAL or NM06:38
Keybukbut it does come from one of them :p06:38
wasabiSo udev essentially runs initctl set network-up $IFACE true/false?06:38
wasabiOr are the states defined in a file which contains `from X to Y`?06:39
Keybukeither is valid06:39
Keybukin the udev case, I would have it emit events06:39
Keybuk(in fact, I think events are generally preferred)06:40
wasabiSo oddly enough, states are back to being exactly what a job is: started or stopped, true or false. :006:40
wasabiJust without any executable.06:40
Keybukthe implementation is very closely coupled internally06:41
Keybukin fact, all jobs have a state associated with them06:41
Keybuksince it's that state that causes them to be started or stopped06:41
Keybukthe difference in definition is simply that one has more options than the other06:41
wasabiHmm. I see.06:41
wasabiSo, you might in fact have a state and a job file both in /etc/event.d. Both files would look about the same, except the state one wouldn't have any exec lines.06:42
wasabiAnd why the syntax difference between 'set' and 'start/stop' in initctl?06:42
Keybukthat's the bit I'm trying to work out now :)06:42
Keybukwhether it is worth exposing the internal difference externally06:43
ion_Are states going to go to a separate directory than jobs according to the current plan?06:43
wasabiWell, if jobs are internally states...06:43
wasabiThen they belong in the same dir.06:43
Keybukand if the difference isn't exposed, how do we avoid the bloat of every state carrying the entire Job structure with it?06:43
wasabiAre we talking about /etc/state.d? :006:43
wasabiKeybuk: Well, you have a set of states, and a set of jobs.06:44
wasabiJobs depend on states. There is no external visibility of a job, except for the various running executables.06:44
Keybukjobs show up in initctl list06:44
wasabiinitctl stop foo actually means "set state `foo` to false", which internally results in the job structure going through the lifecycle for termination.06:44
Keybukshould states?06:44
ion_Im in favour of putting jobs and states to separate directories. When there are going to be a lot of files in the directories, it will be helpful.06:45
Keybukion_: then you have namespace collision issues06:45
wasabiI'd not put them in seperate directories because it will intrduce some confusion. If a job is a state... then a job can depend on another job.06:45
wasabiBut also on a state.06:46
wasabiBecause there is no difference.06:46
wasabiSo if a job depends on `foo`, go find foo.06:46
Keybukthere is a difference at the moment06:46
Keybukstates are instantiable by default06:46
Keybukjobs aren't06:46
wasabiAnyways, we've come full circle again. Back to jobs being exactly the same as states. There being no real internal difference except one has a structure for process lifecycle management.06:47
Keybukwhich is where it becomes interesting06:47
Keybukbecause if the lifecycle management can be separated, then we get to interesting ideas06:47
Keybukfor example, imagine you have a state for the existance of a particular file06:48
Keybuktied into inotify maybe06:48
Keybukjobs could be run while and for /etc/site/*/apache.conf06:48
Keybukin other words, one copy of the job is run for each of the files that exists06:48
wasabiThat is interesting.06:49
wasabiOr, not really.06:49
wasabiIs that one state or many states? heh06:49
wasabiWell, it's one state... effected by many files.06:49
wasabiwhere each file defines a job structure hanging off the state.06:49
wasabiWhen using the inotify/glob thing you just said, what is the value of the state at any time?06:50
Keybuktrue for a given filename06:51
wasabiBut when does it become false?06:51
Keybukstates can have one false value, or one or more truths06:51
Keybukwhen there are no trues06:51
wasabiWell, inotify is an event that says the file was altered. There is no correspondening point in time that the file "is not altered".06:51
Keybuknot 100% sure about this bit yet06:52
wasabiSo it doesn't form a timeline of any sort.06:52
Keybukwith inotify, we know when the file exists, when it is deleted and when it is modified06:52
Keybukso you put create on the left of the state06:52
Keybukdelete on the right of the state06:52
Keybukand put modify on both sides, so the job is restarted06:52
wasabiSo while the file exists?06:52
Keybukthe part of upstart that supplies this notification would set the states on startup using stat()06:53
wasabiAnd now you have a true/false state that actually means file existance, but it toggles when it's modified.06:53
wasabiBut you can't detect the toggle, you can just see the fall out from it.06:53
Keybukyou may be more discreet, and define a file-exists state, and a file-same state or something06:53
Keybukso the job can choose06:53
wasabiThis could be implemented outside of upstart as well, in a way.06:55
wasabiWhich might be ... better?06:55
wasabiUpstart only knows of the state. A seperate runnable daemon watches the files and toggles the states.06:55
wasabiAn extremely simple daemon.06:56
Keybukmy theory is that upstart would be able to supply the answer to a request of "what arguments to the file-created event are states expecting?"07:00
Keybukso the daemon would register that it supplies that event07:01
Keybukand upstart would respond with the arguments that it knows, and with new ones as jobs are created07:01
Keybukso the daemon knows what to watch07:01
Keybukthis is all my next thing to tackle, anyway07:10
Keybuknow that the config code is better07:10
=== phoenix24_ [i=hemvy@ns38066.ovh.net] has joined #upstart
AlexExtremeporting the profile code to the new config stuff is harder than I would have liked ;)08:20
wasabiNow, what do profiles do?08:20
wasabiI think I missed that conversation.08:20
AlexExtreme(this code isn't in main, it's in my branch)08:21
=== phoenix24 [i=pgsgrhq@ns38066.ovh.net] has joined #upstart
wasabiSeems simple, reasonable, and sane.08:23
wasabiie not a huge subsystem, just a simple filter of state names.08:23
wasabiat uds-mtv we talked about some sort of flag thing which you could set, and instead of using profiles (something outside of the job) to determine whether the job would run, the job itself would check the flag.08:23
=== wasabi shrugs.
wasabiWhich could interestingly enough be done with states.08:24
AlexExtremei didn't particularly like that idea for some reason which i can't remember08:24
wasabiwhere states were settable from the loader prompt.08:24
=== Md [i=md@freenode/staff/md] has joined #upstart
=== Keybuk [n=scott@quest.netsplit.com] has joined #upstart
Keybukan interesting thought has occurred09:21
Keybukthe state in which the apache job can be run09:21
Keybukis not the same as09:21
Keybukthe state in which the apache job is running09:23
=== juergbi [n=juerg@80-219-16-162.dclient.hispeed.ch] has joined #upstart
wasabiHmm. I barely understand that.09:44
=== Amaranth [n=travis@ubuntu/member/Amaranth] has joined #upstart
=== cortana [n=sam@62-31-146-25.cable.ubr12.azte.blueyonder.co.uk] has joined #upstart

Generated by irclog2html.py 2.7 by Marius Gedminas - find it at mg.pov.lt!