yet another Python state machine (and why you might care)

TL;DR: I wrote a minimalistic state machine implementation in Python. You can find the code on GitHub. The rest of this post explains what a state machine is and why you might (or might not) care. The post is slanted towards scientists who are technically inclined but lack formal training in computer science or software development. If you just want some documentation or examples, see the README.

A common problem that arises in many software applications is the need to manage an application’s trajectory through a state of discrete states. This problem will be familiar, for instance, to almost every researcher who has ever had to program an experiment for a study involving human subjects: there are typically a number of different states your study can be in (informed consent, demographic information, stimulus presentation, response collection, etc.), and these states are governed by a set of rules that determine the valid progression of your participants from one state to another. For example, a participant can proceed from informed consent to a cognitive task, but never the reverse (on pain of entering IRB hell!).

In the best possible case, the transition rules are straightforward. For example, given states [A, B, C, D], life would be simple if the the only valid transitions were A –> B, B –> C, and C –> D. Unfortunately, the real world is more complicated, and state transitions are rarely completely sequential. More commonly, at least some states have multiple potential destinations. Sometimes the identity of the next state depends on meeting certain conditions while in the current state (e.g., if the subject responded incorrectly, the study may transition to a different state than if they had responded correctly); other times the rules may be probabilistic, or depend on the recent trajectory through state space (e.g., a slot machine transitions to a winning or losing state with some fixed probability that may also depend on its current position, recent history, etc.).

In software development, a standard method for dealing with this kind of problem is to use something called a finite-state machine (FSM). FSMs have been around a relatively long time (at least since Mealy and Moore’s work in the 1950s), and have all kinds of useful applications. In a nutshell, what a good state machine implementation does is represent much of the messy logic governing state transitions in a more abstract, formal and clean way. Rather than having to write a lot of complicated nested logic to direct the flow of the application through state space, one can usually get away with a terse description of (a) the possible states of the machine and (b) a list of possible transitions, including a specification of the source and destination states for each transition, what conditions must be met in order for the transition to execute, etc.

For example, suppose you need to write some code to transition between different phases in an online experiment. Your naive implementation might look vaguely like this (leaving out a lot of supporting code and focusing just on the core logic):

if state == 'consent' and user_response == 'Agree':
    state = 'demographics'
elif state == 'demographics' and validate_demographics(data):
    save_demographics()
    state = 'personality'
elif state == 'personality':
    save_personality_responses(data)
    if not has_more_questions():
        state = 'task'
elif state == 'task':
...

This is a minimalistic example, but already, it illustrates several common scenarios–e.g., that the transition from one state to another often depends on meeting some specified condition (we don’t advance beyond the informed consent stage until the user signs the document), and that there may be some actions we want to issue immediately before or after a particular kind of transition (e.g., we save survey responses before we move onto the next phase).

The above code is still quite manageable, so if things never get any more complex than this, there may be no reason to abandon a (potentially lengthy) chain of conditionals in favor of a fundamentally different approach. But trouble tends to arises when the complexity does increase–e.g., you need to throw a few more states into the mix later on–or when you need to move stuff around (e.g., you decide to administer the task before the demographic survey). If you’ve ever had the frustrating experience of tracing the flow of your app through convoluted logic scattered across several files, and being unable to figure out why your code is entering the wrong state in response to some triggered event, the state machine pattern may be right for you.

I’ve made extensive use of state machines in the past when building online studies, and finding a suitable implementation has never been a problem. For example, in Rails–which is what most of my apps have been built in–there are a number of excellent options, including the state_machine plugin and (more recently) Statesman. In the last year or two, though, I’ve begun to transition all of my web development to Python (if you want to know why, read this). Python is a very common language, and the basic FSM pattern is very simple, so there are dozens of Python FSM implementations out there. But for some reason, very few of the Python implementations are as elegant and usable as their Ruby analogs. This isn’t to say there aren’t some nice ones (I’m partial to Fysom, for instance)–just that none of them quite meet my needs (in particular, there are very few fully object-oriented implementations, and I like to have my state machine tightly coupled with the model it’s managing). So I decided to write one. It’s called Transitions, and you can find the code on GitHub, or install it directly from the command prompt (“pip install transitions”, assuming you have pip installed). It’s very lightweight–fewer than 200 lines of code (the documentation is about 10 times as long!)–but still turns out to be quite functional.

For example, here’s some code that does almost exactly the same thing as what we saw above (there are much more extensive examples and documentation in the GitHub README):

from transitions import Machine

# define our states and transitions
states = ['consent', 'demographics', 'personality', 'task']
transitions = [
    {   
        'trigger': 'advance',
        'source': 'consent',
        'dest': 'demographics',
        'conditions': 'user_agrees'
    },
    { 
        'trigger': 'advance', 
        'source': 'demographics', 
        'dest': 'personality', 
        'conditions': 'validate_demographics', 
        'before': 'save_demographics'
    },
    { 
        'trigger': 'advance',
        'source': 'personality',
        'dest': 'task',
        'conditions': 'no_more_items',
        'before': 'save_items'
    }
]

# Initialize the state machine with the above states and transitions, and start out life in the solid state.
machine = Machine(states=states, transitions=transitions, initial='consent')

# Let's see how it works...
machine.state
> 'consent'
machine.advance() # Trigger methods are magically added for us!
machine.state
> 'demographics'
...

That’s it! And now we have a nice object-oriented state machine that elegantly transitions between phases of matter, triggers callback functions as needed, and supports conditional transitions, branching, and various other nice features, all without ever having to write a single explicit conditional or for-loop. Understanding what’s going on is as simple as looking at the specification of the states and transitions. For example, we can tell at a glance from the second transition that if the model is currently in the ‘demographics’ state, calling advance() will effect a transition to the ‘personality’ state–conditional on the validate_demographics() function returns True. Also, right before the transition executes, the save_demographics() callback will be called.

As I noted above, given the simplicity of the example, this may not seem like a huge win. If anything, the second snippet is slightly longer than the first. But it’s also much clearer (once you’re familiar with the semantics of Transitions), scales much better as complexity increases, and will be vastly easier to modify when you need to change anything.

Anyway, I mention all of this here for two reasons. First, as small and simple a project as this is, I think it ended up being one of the more elegant and functional minimalistic Python FSMs–so I imagine a few other people might find it useful (yes, I’m basically just exploiting my PageRank on Google to drive traffic to GitHub). And second, I know many people who read this blog are researchers who regularly program experiments, but probably haven’t encountered state machines before. So, Python implementation aside, the general idea that there’s a better way to manage complex state transitions than writing a lot of ugly logic seems worth spreading.

14 thoughts on “yet another Python state machine (and why you might care)”

  1. Is there any reason why you didn’t go the method chaining route?
    There doesn’t seem to be any state machine implementation out there that uses some kind of builder pattern.

    The example at least reminds me a lot of fysom (I’m maintaining it).

    Specifying the transitions with dictionary is IHMO not very object oriented.
    I’d rather configure my state machine with something like this for example :

    “`
    machine = Machine()
    machine.when(“any-event-name”).and_condition(any_callback).then_transition_to(“next-state”)
    “`

  2. Is there any reason why you didn’t go the method chaining route?

    To be honest, it never occurred to me. But I like the idea a lot, and it wouldn’t take much work to ad d.

    The example at least reminds me a lot of fysom (I’m maintaining it).

    Yes, I modeled transitions in large part on fysom (with some additional features borrowed from Ruby/Rails plugins). As I say in the post, I like fysom a lot in terms of functionality–I just wanted to have a more object-oriented module that could be easily integrated directly into other classes.

    Specifying the transitions with dictionary is IHMO not very object oriented.

    The initialization interface may not look very object-oriented, but the underlying code is. The dictionary format is actually just a convenient shorthand for something like:

    “`
    transitions = [Transition(trigger=’move’, source=’old’, dest=’new’, before=…), Transition(…)]
    “`

    The events named in the Transition constructor are in turn each instantiated as their own Event object, etc.

    I’d rather configure my state machine with something like this for example:

    I agree, this would be really nice! It would be pretty easy to add that in. I’ll give it a pass when I next work on this (which might not be for a while–this was just a small side project incidental to a larger one). But feel free to submit a PR if you like–I think it would just amount to a a few extra methods in the Machine, Event, and Transition classes.

  3. My one trouble is combining your nice FSM implementation with so-called main loop of messaging. I need my FSM to transition conditionally to other state, in a perpetual fashion. I may utilize “before”, “after”, “on_enter”, “on_exit” callbacks. That’s fine. But I would end up with infinite call stack. Since all of above four are callables, and I might have to call next state’s callable before ending the current callable.

    Can you advise a proper design instead?

    1. Hi Porce,

      I’m not sure I understand you completely… conditional transitions are currently implemented; you can make transitions conditional on passing one or more conditions. Is your point that you would like transitions to be chained automatically, and they currently aren’t?

      I’m happy to discuss this in more detail, but it’s probably best if you open an issue on the github repo. Thanks!

  4. @mriehl that looks like a good idea – I haven’t found a state machine implementation in python that I like yet, but I might give this a go – we could do with more APIs like this in python.

  5. Great work. Between this post and your documentation on GitHub, it was simple to implement too. Thanks for sharing!!

  6. I found it very simple yet powerful, thanks !

    My question is: is there any way to run a generic callback function, no matter what transition is called ? Or should I declare a callback function for each transition ?

    Thanks in advance !

    Alejandro

  7. Stumbled upon your repo and this blog, while trying to find ways to build FSM using python. Not only is this one great library for FSM, but a very elegant piece of code. Thanks for all the work.

  8. Totally agree with Shashank: useful tool, great code. Thanks for the work! Are there any code examples / experiences of using this for an API (which would need state persistence etc)?

  9. Hi Tried running your example code and I encountered an error: AttributeError: Callable with name 'user_agrees' could neither be retrieved from the passed model nor imported from a module.

Leave a Reply