Welcome to the documentation for Vivarium Core!¶
The Vivarium Core library provides the Vivarium interface and engine for composing and simulating integrative, multiscale models.
Getting Started¶
Download and Installation¶
Getting Organized¶
Creating Enclosing Directory¶
Create a vivarium_work folder anywhere you like. But for installing
some third-party software, everything we do will occur inside this
folder.
Setting PYTHONPATH¶
Vivarium Core needs the root of the repository to be in your
PYTHONPATH environment variable so that Python can find Vivarium
Core. To make this easy to set, we suggest adding this line to your
shell startup file:
alias pycd='export PYTHONPATH="$PWD"'
Now when you are about to work on Vivarium Core, navigate to the root of
the Vivarium Core repository (vivarium_work/vivarium-template) and run
pycd in your terminal. You will need to do this for each terminal
window you use.
Installing Dependencies¶
Below, we list the dependencies Vivarium Core requires, how to check whether you have them, how to install them, and in some cases, how to set them up for Vivarium Core. Make sure you have each of them installed.
Python 3¶
Vivarium Core requires Python 3.8, 3.9, or 3.10.
Check Installation
$ python --version
Python <version>
Make sure you see a version starting with 3.8, 3.9, or 3.10.
Install
Download the latest installer from the Python download page
MongoDB¶
We use a MongoDB database to store the data collected from running simulations. This can be a remote server, but for this guide we will run a MongoDB server locally.
Note: MongoDB is only required if you want to store data in MongoDB or want to run experiments that do so. You don’t need MongoDB to work through this guide.
Check Installation
$ mongod --version
db version v4.2.3
...
Make sure you see a version at least 3.2.
Install
If you are on macOS, you can install MongoDB using Homebrew. You will need to add the MongoDB tap following the instructions here.
If you are on Linux, see the MongoDB documentation’s instructions.
Setup
You can get a MongoDB server up and running locally any number of ways. Here is one:
Create a folder
vivarium_work/mongodb. This is where MongoDB will store the database We store the database here instead of at the default location in/usr/local/var/mongodbto avoid permissions issues if you are not running as an administrator.Make a copy of the
mongodconfiguration file so we can make changes:$ cp /usr/local/etc/mongod.conf vivarium_work/mongod.confNote that your configuration file may be somewhere slightly different. Check the MongoDB documentation for your system.
In
vivarium_work/mongod.confchange the path afterdbPath:to point tovivarium_work/mongodb.Create a shell script
vivarium_work/mongo.shwith the following content:#!/bin/bash mongod --config mongod.confMake the script executable:
$ chmod 700 vivarium_work/mongo.sh
Now you can launch MongoDB by running this script:
$ vivarium_work/mongo.sh
Download and Setup Template Project¶
Download the Code¶
The template code is available on GitHub. Move into
your vivarium_work directory and clone the repository to download
the code
$ cd vivarium_work
$ git clone https://github.com/vivarium-collective/vivarium-template.git
This will create a vivarium-template folder inside vivarium_work.
All the code for your model will be inside this vivarium-template
folder.
Repository Structure¶
The repository you downloaded should look like this:
.
├── README.md
├── pytest.ini
├── release.sh
├── requirements.txt
├── setup.py
└── template
├── __init__.py
├── compartments
├── composites
│ ├── __init__.py
│ └── injected_glc_phosphorylation.py
├── experiments
│ ├── __init__.py
│ └── glucose_phosphorylation.py
├── library
│ └── __init__.py
└── processes
├── __init__.py
├── glucose_phosphorylation.py
└── template_process.py
We suggest you use the structure laid out here, but you don’t have to.
The template repository has TODO notes where you’ll need to make
changes. Before you publish your project, make sure you have removed all
the TODO notes!
The template directory is where your package will live. Under it, we
have the following sub-folders:
library: This is for utility functions like those shared across processes, composers, and/or experiments.processes: This is where you’ll write your processes. We’ve provided atemplate_process.pyfile to get you started. Generally you’ll want to have one process per file.composites: This folder will hold your composers, which generate composites of multiple processes.experiments: This folder will hold your experiments. These are the files you’ll probably be executing to run your simulations.
To show how you can build models with Vivarium, we’ve included some examples around modeling glucose phosphorylation.
Installing Python Packages¶
Above we installed all the non-Python dependencies, but we still have to install the Python packages Vivarium Core uses.
Move into the
vivarium-templatefolder created when you cloned the repository.(optional) Create and activate a virtual environment using
venvorpyenv virtualenv, e.g.:$ python3 -m venv venv --prompt "vivarium-template" ... $ source venv/bin/activate
Install packages
$ pip install -r requirements.txt
Now you are all set to create models and run simulations with Vivarium Core!
Run Simulations¶
Some Terminology: Processes and Composites¶
We break our cell models into processes. Each process models part of the cell’s function. For example, you might have processes for metabolism, transcription, and translation. We can combine these processes into composites that model a system with all the functionality modeled by the included processes. For example, we could compose transcription and translation to create a fuller gene expression model.
We store individual processes in vivarium-template/template/processes
and the composers that generate composites of processes in
vivarium-template/template/composites. We recommend you use a similar
structure when creating your own processes and composers.
Running Experiments¶
Running experiments is as easy as executing their files. For example,
this repository comes with an example experiment in
vivarium-template/template/experiments/glucose_phosphorylation.py. Try
running it like this:
$ python template/experiments/glucose_phosphorylation.py
In out/experiments/glucose_phosphorylation you should see a file
simulation.png that looks like this:
Run Tests¶
We strongly encourage you to write tests for your code. It will make
development much easier for you. The template repository comes with some
tests already. To run them, just execute pytest.
Getting Started for Developers¶
Note
This guide is for developers who want to contribute to Vivarium Core. If you just want to use Vivarium to build models, see the getting started guide.
Download and Installation¶
This guide assumes that you have access to an administrator or sudoer account on a macOS or Linux system.
Getting Organized¶
Creating Enclosing Directory¶
Create a vivarium_work folder anywhere you like. But for installing
some third-party software, everything we do will occur inside this
folder.
Setting PYTHONPATH¶
Vivarium Core needs the root of the repository to be in your
PYTHONPATH environment variable so that Python can find Vivarium
Core. To make this easy to set, we suggest adding this line to your
shell startup file:
alias pycd='export PYTHONPATH="$PWD"'
Now when you are about to work on Vivarium Core, navigate to the root of
the Vivarium Core repository (vivarium_work/vivarium-core) and run
pycd in your terminal. You will need to do this for each terminal
window you use.
Installing Dependencies¶
Below, we list the dependencies Vivarium Core requires, how to check whether you have them, how to install them, and in some cases, how to set them up for Vivarium Core. Make sure you have each of them installed.
Python 3¶
Vivarium Core requires Python 3.8, 3.9, or 3.10.
Check Installation
$ python --version
Python <version>
Make sure you see a version beginning with 3.8, 3.9, or 3.10.
Install
Download the latest installer from the Python download page
MongoDB¶
We use a MongoDB database to store the data collected from running simulations. This can be a remote server, but for this guide we will run a MongoDB server locally.
Note: MongoDB is only required if you want to store data in MongoDB or want to run experiments that do so. You don’t need MongoDB to work through this guide.
Check Installation
$ mongod --version
db version v4.2.3
...
Make sure you see a version at least 4.2.
Install
If you are on macOS, you can install MongoDB using Homebrew. You will need to add the MongoDB tap following the instructions here.
If you are on Linux, see the MongoDB documentation’s instructions.
Setup
You can get a MongoDB server up and running locally any number of ways. Here is one:
Create a folder
vivarium_work/mongodb. This is where MongoDB will store the database We store the database here instead of at the default location in/usr/local/var/mongodbto avoid permissions issues if you are not running as an administrator.Make a copy of the
mongodconfiguration file so we can make changes:$ cp /usr/local/etc/mongod.conf vivarium_work/mongod.confNote that your configuration file may be somewhere slightly different. Check the MongoDB documentation for your system.
In
vivarium_work/mongod.confchange the path afterdbPath:to point tovivarium_work/mongodb.Create a shell script
vivarium_work/mongo.shwith the following content:#!/bin/bash mongod --config mongodb.confMake the script executable:
$ chmod 700 vivarium_work/mongo.sh
Now you can launch MongoDB by running this script:
$ vivarium_work/mongo.sh
Download and Setup Vivarium Core¶
Download the Code¶
Note: These instructions give you the latest development version of
Vivarium Core. If you want to use the most recent release, which is more
stable, you can instead run pip install vivarium-core in the
Installing Python Packages section below.
The Vivarium Core code is available on GitHub. Move into your
vivarium_work directory and clone the repository to
download the code
$ cd vivarium_work
$ git clone https://github.com/vivarium-collective/vivarium-core.git
This will create a vivarium-core folder inside vivarium_work.
All the code for Vivarium Core is inside this vivarium-core folder.
Installing Python Packages¶
Above we installed all the non-Python dependencies, but we still have to install the Python packages Vivarium Core uses.
Move into the
vivarium-corefolder created when you cloned the repository.(optional) Create and activate a virtual environment using
venvorpyenv virtualenv, e.g.:$ python3 -m venv venv --prompt "vivarium-core" ... $ source venv/bin/activate
Install packages
$ pip install -r requirements.txt
Now you are all set to begin developing! Be sure to review our contribution instructions before you get started.
Topic Guides¶
Each of these guides dives into the technical details of an important concept in Vivarium. If you’re looking to use Vivarium, you should work through the getting started guide first.
Processes¶
You should interpret words and phrases that appear fully capitalized in this document as described in RFC 2119. Here is a brief summary of the RFC:
“MUST” indicates absolute requirements. Vivarium may not work correctly if you don’t follow these.
“SHOULD” indicates strong suggestions. You might have a valid reason for deviating from them, but be careful that you understand the ramifications.
“MAY” indicates truly optional features that you can include or exclude as you wish.
Models in Vivarium are built by combining processes, each of
which models a mechanism in the system being studied. These processes
can be combined in a composite to build more complicated
models. Process models are defined in classes that inherit from
vivarium.core.process.Process, and these process
classes can be instantiated to create individual processes. During
instantiation, the process class may accept configuration options.
Note
Processes are the foundational building blocks of models in Vivarium, and they should be as simple to define and compose as possible.
Process Interface Protocol¶
Each process class MUST implement the application programming interface (API) that we describe below.
Class Variables¶
Each process class SHOULD define default configurations in a
defaults class variable. The constructor SHOULD read these defaults.
For example:
class MyProcess:
defaults = {
'growth_rate': 0.0006,
}
Constructor¶
The constructor of a process class MUST accept as its first positional argument an optional dictionary of configurations. If the process class is configurable, it SHOULD accept configuration options through this dictionary.
In the constructor, the process class MUST call its superclass constructor with a dictionary of parameters.
Passing Parameters to Superclass Constructor¶
The dictionary of parameters SHOULD include any configuration options
not used by the process class. Any information needed by the process
class MAY also be included in these parameters. Once the object has
been instantiated, these parameters are available as
self.parameters, where they have been stored by the
vivarium.core.process.Process constructor.
Example Constructor¶
Let’s examine an example constructor from a growth process class.
def __init__(self, initial_parameters=None):
if initial_parameters == None:
initial_parameters = {}
parameters = {'growth_rate': self.defaults['growth_rate']}
parameters.update(initial_parameters)
super().__init__(parameters)
Note that Vivarium Core actually handles combining the provided parameters with the default parameters, so a constructor as simple as the one above can actually be dropped. The superclass constructor makes it redundant, but we show it here for clarity.
Warning
Python creates only one instance of both class variables
and function argument defaults. This means that you MUST not change
the default parameters object. Make a copy instead. This also means
that you SHOULD avoid using a mutable object as a default argument.
This is why we use None as the default for initial_parameters
instead of {}.
While the default growth rate is 0.0006, this can be overridden by
including a growth_rate key in the configuration dictionary passed
to initial_parameters.
These special parameters get handled by the superclass constructor:
name: The value of thenameparameter gets assigned to the process’snameattribute (e.g.my_process.name). If no name is specified in the parameters or as a class variable, we useself.__class__.__name__as the name.time_step: If not specified, thetime_stepparameter is set to 1. This parameter determines how frequently the simulation engine runs this process’snext_updatefunction._condition: The value of this parameter should be a path in thestatesdictionary passed tonext_update()to a variable. The variable should hold a boolean specifying whether the process’snext_updatefunction should run.
Ports Schema¶
Each process declares what stores it expects by specifying a port for each store it accepts. Note that if two processes are to be combined in a model and share variables through a shared store, the processes MUST use the same variable names for the shared variables.
The process class MUST implement a ports_schema method with no
required arguments. This method MUST return nested dictionaries of the
following form:
{
'port_name': {
'variable_name': {
'schema_key': 'schema_value',
...
},
...
},
...
}
Schema keys¶
schema_key MUST be a schema key and have an appropriate
value. Any applicable and omitted schema keys will take on their default
values. Note that every variable SHOULD specify _default. If the
cell will be dividing, every variable also MUST specify _divider.
Variables in the ports schema SHOULD NOT specify _value.
Available schema keys include:
_default: The default value of the state variable if no initial value is provided. This also sets the data type of the variable, including units._updater: How to apply state variable updates. Available updaters are listed in below_divider: How to divide the state variable’s values between daughter cells. Available dividers are listed below._emit: A Boolean value that sets whether to log this variable to the simulation database for later analysis._properties: User-defined properties such as molecular weight. These can be used for calculating variables such as total system mass.
Updaters¶
Updaters are methods by which an update from a process is applied to a variable’s value.
Updaters provided by vivarium-core include:
accumulate: The default updater. Add the update value to the current value.set: The update value becomes the new current value.merge: Update an existing dictionary with new values, and add any newly declared keys.null: Do not apply the update.nonnegative_accumulate: Add the update value to the current value, and set to 0 if the result is negative.dict_value: translates_addand_delete-style updates to operations on a dictionary.
New updaters can be easily defined and passed into a port schema:
# updater that returns a random value
def random_updater(current_value, update_value):
return random.random()
def port_schema(self):
ports = {
'port1': {
'variable1': {
'_default': 1.0
'_updater': {
'updater': random_updater
}
}
}
}
return ports
Dividers¶
Dividers are methods by which a variable’s value is divided when division is triggered.
Dividers available in vivarium-core include:
set: The default divider. Daughters get the same value as the mother.binomial: Sample the first daughter’s value from a binomial distribution of the mother’s value, and the second daughter gets the remainder.split: Divide the mother’s value in two. Odd integers will make one daughter receive 1 more than the other daughter.split_dict: Splits a dictionary of{key: value}pairs, with each daughter receiving a dictionary with the same keys, but with each value split.zero: Daughter values are both set to 0.no_divide: Asserts that this value should not be divided.
New dividers can be easily defined and passed into a port schema:
# divider that returns a random value for each daughter
def random_divider(mother_value, state):
return [
random.random(),
random.random()]
def port_schema(self):
ports = {
'port1': {
'variable1': {
'_default': 1.0
'_divider': {
'divider': random_divider
}
}
}
}
return ports
Example Ports Schema¶
def ports_schema(self):
return {
'global': {
'mass': {
'_emit': True,
'_default': 1339 * units.fg,
'_updater': 'set',
'_divider': 'split'},
'volume': {
'_updater': 'set',
'_divider': 'split'},
'divide': {
'_default': False,
'_updater': 'set'
}
}
}
Here we specify that only mass should be emitted. We assign a
default value of 1339 fg to mass, and we declare that the mass
and volume variables should be split in half on division. Further,
we specify that all the three variables should have their updates set,
not accumulated.
Views¶
When the process is asked to provide an update to the model state, it is only provided the variables it specifies. For example, it might get a model state like this:
{
'global': {
'mass': 1339 <Unit('femtogram')>,
'volume': 1.2,
'divide': False,
},
}
This would happen even if the store linked to the global port
contained more variables. We call this stripping-out of variables the
process doesn’t need masking.
Advanced Ports Schema¶
Use the glob * schema to declare expected sub-store structure,
and view all child values of the store:
schema = {
'port1': {
'*': {
'_default': 1.0
}
}
}
Use the glob ** schema to connect to an entire sub-branch, including
child nodes, grandchild nodes, etc:
schema = {
'port1': '**'
}
Ports flagged as output-only won’t be viewed through the next_update’s states, which can save some overhead time:
schema = {
'port1': {
'_output': True,
'A': {'_default': 1.0},
}
}
Next Updates¶
Each process class MUST implement a next_update method that accepts
two positional arguments: the timestep and the current state of
the model. The timestep describes, in units of seconds, the length of
time for which the update should be computed.
State Format¶
The next_update method MUST accept the simulation state as a
dictionary of the same form as the ports schema dictionary, but with the dictionary of schema keys
replaced with the current (i.e. pre-update) value of the variable.
Note
In the code, you may see the simulation state referred to as
states. This is left over from when stores were called states,
and so the simulation state was a collection of these states. As you
may already notice, this naming was confusing, which is why we now
use the name “stores.”
Because of masking, each port will contain only the variables specified in the ports schema, even if the linked store contains more variables.
Warning
The next_update method MUST NOT modify the states it is
passed in any way. The state’s variables are not copied before they
are passed to next_update, so changes to any objects in the state
will affect the simulation state before the update is applied.
Update Format¶
next_update MUST return a single dictionary, the update that
describes how the modeled mechanism would change the simulation state
over the specified time. The update dictionary MUST be of the same form
as the ports schema dictionary, though
with the dictionaries of schema keys replaced with update values. Also,
variables that do not need to be updated can be excluded.
Example Next Update Method¶
Here is an example next_update method for our growth process:
def next_update(self, timestep, states):
mass = states['global']['mass']
new_mass = mass * np.exp(self.parameters['growth_rate'] * timestep)
return {'global': {'mass': new_mass}}
Recall from our example schema that we use
the set updater for the mass variable. Thus, we compute the new
mass of the cell and include it in our update. Notice that we access the
growth rate specified in the constructor by using the
self.parameters attribute.
Note
Notice that this function works regardless of what timestep we use. This is important because different simulations may need different timesteps based on what they are modeling.
Process Class Examples¶
Many of our process classes have examples in the form of test functions at the bottom. These are great resources if you are trying to figure out how to use a process.
If you are writing your own process, please include these examples!
Also, executing the process class Python file should execute one of
these examples and save the output as demonstrated in
vivarium.processes.glucose_phosphorylation. Lastly, any
top-level functions you include that are prefixed with test_ will be
executed by pytest. Please add these tests to help future developers
make sure they haven’t broken your process!
Steps¶
Processes have one major drawback: you cannot specify when or in what order they run. Processes can request timesteps, but the Vivarium engine may not honor that request. This behavior can be problematic when you have operations that need to run in a particular order. For example, imagine that you want to model transcription and chromosome replication in a bacterium. It seems natural to have a transcription process and another replication process, but then how do you handle collisions between the replisome and the RNA Polymerase (RNAP)? You might want to say something like “If a replisome and RNAP collide, remove the RNAP from the chromosome.” To support this kind of statement, you can create a step.
vivarium.core.process.Step is a subclass of
vivarium.core.process.Process that is not time-dependent.
Steps run before the first timestep and after the dynamic processes
during simulation. They run according to a dependency graph called a
flow (like a workflow) – see our guide to flows. These can serve many different roles, including
translating states between different modeling formats, implementing lift
or restriction operators to translate states between scales, and as
auxiliary processes that offload complexity. As an example of offloading
complexity, a step might recalculate concentrations after counts have
been updated.
To create a step, you follow the same steps as you would to create a
process except that your class should inherit from
vivarium.core.process.Step. For example, we could create a
replisome-RNAP collision reconciler like this:
class CollisionReconciler(Step):
def ports_schema(self):
return {
'replisomes': {
'*': {
'position': {'_default': 0},
},
},
'RNAPs': {
'*': {
'position': {'_default': 0},
},
},
}
def next_update(self, timestep, states):
# We can ignore the timestep since it will always be 0.
replisome_positions
replisome['position']
for replisome in states['replisomes'].values()
])
rnap_positions = np.array([
rnap['position']
for rnap in states['RNAPs'].values()
])
# Assume that our timestep is small enough that we can
# ignore RNAPs and replisomes that move past each other
# (instead of to the same position) in one timestep.
collision_mask = replisome_positions == rnap_positions
rnap_keys = np.array(list(states['RNAPs'].keys()))
to_remove = rnap_keys[collision_mask]
return {
'RNAPs': {
'_delete': to_remove.tolist(),
},
}
Note
Steps are always given a timestep of 0 by the simulation engine.
Step Implementation Details¶
Steps are technically identified by whether their
vivarium.core.process.Process.is_step() methods return
True. This means that you can make a process that determines whether
it should be a Step based on its configuration. Note however that we do
not support changing whether a process is a step mid-simulation.
Advanced Features¶
Adaptive Timesteps¶
You can set process timesteps for the duration of a simulation using the
time_step parameter, but you can also override the
vivarium.core.process.Process.calculate_timestep() method to
compute timesteps dynamically based on the same view into the simulation
state that next_update() sees.
Conditional Updates¶
Sometimes you might want the simulation engine to skip a process when
generating updates. You can implement this by overriding
vivarium.core.process.Process.update_condition() to return
False whenever you don’t want the process to run. This method takes
as a parameter the same view into the simulation state that
next_update() sees.
Using Process Objects¶
Your use of process objects will likely be limited to instantiating them and passing them to other functions in Vivarium that handle running the simulation. Still, you may find that in some instances, using process objects directly is helpful. For example, for simple processes, the clearest way to write a test may be to run your own simulation loop.
Simulating a process can be sketched by the following pseudocode:
# Create the process
configuration = {...}
process = ProcessClass(configuration)
# Get the initial state from the process's schema
# This means the stores and ports are the same
state = {}
schema = process.ports_schema()
for port, port_dict in schema.items():
for variable, variable_schema in port_dict.items():
state[port][variable] = variable_schema["_default"]
# Run the simulation in a loop for 10 seconds
time = 0
while time < 10:
# We are using a timestep of 1 second
update = process.next_update(1, state)
# This is a simplified way to apply the update that assumes all
# all variables are numbers and all updaters are "accumulate"
for port in update:
for variable_name, value in port.items():
state[port][variable_name] += value
# Now that the loop is finished, the predicted state after 10
# seconds is in "state"
The above pseudocode is simplified, and for all but the most simple processes you will be better off using Vivarium’s built-in simulation capabilities. We hope though that this helps you understand how processes are simulated and the purpose of the API we defined.
Parallel Processing¶
Process Commands¶
When a process is run in parallel, we can’t interact with it in the normal Python way. Instead, we can only exchange messages with it through a pipe. Vivarium structures these exchanges using process commands.
Vivarium provides some built-in commands, which are documented in
vivarium.core.process.Process.send_command(). Also see that
method’s documentation for instructions on how to add support for your
own commands.
Process commands are designed to be used asynchronously, so to retrieve
the result of running a command, you need to call
vivarium.core.process.Process.get_command_result(). As a
convenience, you can also call
vivarium.core.process.Process.run_command() to send a command
and get its result as a return value in one function call.
Running Processes in Parallel¶
In normal situations though, you shouldn’t have to worry about process
commands. Instead, just pass '_parallel': True in a process’s
configuration dictionary, and the Vivarium Engine will handle the
parallelization for you. Just remember that parallelization requires
that processes be serialized and deserialized at the start of the
simulation, and this serialization only preserves the process
parameters. This means that if you instantiate a process and then change
its instance variables, those changes won’t be preserved when the
process gets parallelized.
Composites¶
Once you start building models with Vivarium, you will probably discover groups of processes that you want to reuse. For example, if you have a group of processes that model a cell, you might want to create many instances of those processes to a collection of cells. In Vivarium, we support grouping processes using composites.
Note
The terminology here can be tricky because “process” can refer both to the process object and to the process class. However, for composites, the analogous ideas have different names: “composer” and “composite.” You can use a composer object to generate different composite objects depending on what configuration you provide. Therefore, composers are like process classes while composites are like process objects.
Processes and Stores¶
A model in Vivarium consists of state variables, which are grouped into collections called stores, and processes which mutate those variables at each timestep. A topology defines which processes operate on which stores.
For example, consider a variable tracking ATP concentrations. We might assign a “cytoplasm” store to two processes: one for metabolism and one for sodium pumps. Now when we run a simulation the metabolism and sodium pump processes will be changing the same variable, the ATP concentration in the cytoplasm store. This means that if the rate of metabolism decreases, the cytoplasmic ATP concentration variable will drop, so the sodium pump will export less sodium. Thus, shared stores in composites let us simulate interacting concurrent processes.
Process and Store Implementation¶
Processes¶
We write processes as classes that inherit from
vivarium.core.process.Process. To create a composite, we
create instances of these classes to make the processes we want to
compose. If a process is configurable, we might provide a dictionary of
configuration options. For information on configuring a process, see the
process’s documentation, e.g.
vivarium.processes.tree_mass.TreeMass.
We uniquely name each process in the composite. This lets us include instances of the same process class.
Stores¶
We represent stores with the vivarium.core.store.Store
class; see its documentation for further details. Note that when
constructing a composite, you don’t need to create the stores. Instead,
the Vivarium engine automatically creates the stores based on the
topology you specify.
Tip
To see the data held by a store, you can use the
vivarium.core.store.Store.get_config() function. This
returns a dictionary representation of the store’s data. To show this
dictionary more readably, use
vivarium.library.pretty.format_dict().
Topologies¶
Each process has a list of named ports, one for each store it expects. The process can perform all its computations in terms of these ports, and the process also provides its update using port names. This means that a composite can apply each process to any collection of stores, making processes modular.
This modularity is analogous to the modularity of Python functions. Think of each process as a function like this:
def sodium_pump(cytoplasm, extracellularSpace):
...
return "Update: Decrease ATP concentration in cytoplasm by x mM"
A function’s modularity comes from the fact that we can pass in different
objects for the cytoplasm parameter, even objects the function
authors hadn’t thought of. cytoplasm is like the port, to which we
can provide any store we like.
How do we specify which store goes with which port? To continue the function analogy from above, we need something analogous to this:
cell = Cell()
bloodVessel = BloodVessel()
# We need something like the line below
update = sodium_pump(cytoplasm=cell, extracellularSpace=bloodVessel)
When we call sodium_pump, we specify which objects go with which
parameters. Analogously, we specify the mapping between ports and stores
using a topology.
Defining Topologies¶
We define topologies as dictionaries with process names as keys and dictionaries (termed “sub-dictionaries”) as values. These sub-dictionaries have port names as keys and paths to stores as values. For example, the topology for the ATP example we have been considering might look like this:
{
'sodium_pump': {
'cytoplasm': ('cell',),
'extracellularSpace': ('bloodVessel',),
},
'metabolism': {
'cytoplasm': ('cell',),
},
}
Advanced Topologies¶
The syntax used for declaring paths is a Unix-style tuple, with every
element in the tuple going further down the path from the root compartment,
and .. moving up a level to an outer compartment.
topology = {
'process': {
'port1': ('path','to','store'), # connect port1 to inner compartment
'port2': ('..','outer_store') # connect port2 to outer compartment
}
}
You can splitting a port into multiple stores. Variables read through the same
port can come from different stores. To do this, the port is mapped to a
dictionary with a _path key that specifies the path to the default store.
Variables that need to be read from different stores each get their own path in
that same dictionary. This same approach can be used to remap variable names, so
different processes can use the same variable but see it with different names.
topology = {
# split a port into multiple stores
'process1': {
'port': {
'_path': ('path_to','default_store'),
'rewired_variable': ('path_to','alternate_store')
}
}
# mapping variable names in process to different name in store
'process2': {
'port': {
'_path': ('path to','default_store'),
'variable_name': 'new_variable_name'
}
}
}
Flows for Ordered Step Operations¶
When constructing a composite of many steps, you may find that some steps depend on other steps. For example, you might have one step that calculates the cell’s mass and another step that calculates the cell’s volume based on that mass. Vivarium supports these dependencies, which you can specify in a flow. Flows have the same structure as topologies, but instead of their leaf values being paths, they are lists of paths where each path specifies a dependency step. For example, this flow would represent our mass-volume dependency:
{
'mass_calculator': [],
'volume_calculator': [('mass_calculator',)],
}
The simulation engine will automatically figure out what order to run the steps in such that the dependencies in the flow are respected. Note that if two orderings both respect the flow, you should not assume that the engine will pick one of the two orderings.
Note
Step updates are applied immediately after the step executes, which is unlike process updates.
Composers¶
Most of the time, you won’t need to create composites directly. Instead,
you’ll create composers that know how to generate composites. To create
a composer, you need to define a composer class that inherits from
vivarium.core.composer.Composer and implements the
vivarium.core.composer.Composer.generate_processes() and
vivarium.core.composer.Composer.generate_topology() methods.
generate_processes should return a mapping from process names to
instantiated process objects, while generate_topology should return
a topology.
Example Composer¶
To put all this information together, let’s take a look at an example composer that combines the glucose phosphorylation process from the process-writing tutorial with an injector, which lets us “inject” molecules into a store.
class InjectedGlcPhosphorylation(Composer):
defaults = {
'glucose_phosphorylation': {
'k_cat': 1e-2,
},
'injector': {
'substrate_rate_map': {
'GLC': 1e-4,
'ATP': 1e-3,
},
},
}
def generate_processes(self, config):
injector = Injector(self.config['injector'])
glucose_phosphorylation = GlucosePhosphorylation(
self.config['glucose_phosphorylation'])
return {
'injector': injector,
'glucose_phosphorylation': glucose_phosphorylation,
}
def generate_topology(self, config):
return {
'injector': {
'internal': ('internal', ),
},
'glucose_phosphorylation': {
'cytoplasm': ('cell', ),
'nucleoside_phosphates': ('cell', ),
'global': ('global', ),
},
}
Notice how we use the generate_processes function to create a
dictionary that maps process names to instantiated and configured
process objects. Similarly, we use generate_topology to create a
dictionary that maps port names to stores. To create steps and flows,
use the generate_steps and generate_flow methods.
You may wonder why we identify stores with tuples. In more complex compartments, these tuples could contain many elements that specify a kind of file path. We represent the total model state as a tree, and we can create a store at any node to represent the sub-tree rooted at that node. This tree is analogous to directory trees on a filesystem, and we use tuples of store names to specify a path through this tree. We call this tree the hierarchy, and we discuss it in more detail in the hierarchy guide.
Experiments¶
Once you have created a model using processes and compartments, you might want to run simulations of your model and save simulation parameters so you and others can reproduce your results. Vivarium uses experiments to define these simulations, which can contain arbitrarily nested compartments. For example, you could create an experiment with an environment compartment that contains several cell compartments. Then you could run the experiment to see how these cells might interact in a shared environment.
Defining Experiments¶
To create an experiment, you need only instantiate the
vivarium.core.engine.Engine class. To help others
reproduce your experiment, create a file in the
vivarium-core/experiments directory that defines a function that
generates your experiment. For example, here is the function from
vivarium.experiments.glucose_phosphorylation to create a toy
experiment that simulates the phosphorylation of injected glucose:
def glucose_phosphorylation_experiment(config=None):
if config is None:
config = {}
default_config = {
'injected_glc_phosphorylation': {},
'emitter': {
'type': 'timeseries',
},
'initial_state': {},
}
default_config.update(config)
config = default_config
compartment = InjectedGlcPhosphorylation(
config['injected_glc_phosphorylation'])
compartment_dict = compartment.generate()
experiment = Engine(
processes=compartment_dict['processes'],
topology=compartment_dict['topology'],
emitter=config['emitter'],
initial_state=config['initial_state'],
)
return experiment
Notice that most of the function just sets up configurations. The main steps are:
Instantiate the composite that your experiment will simulate.
Generate the processes and topology dictionaries that describe the composite using
vivarium.core.composer.Composer.generate().Instantiate the experiment, passing along the processes and topology dictionaries. We also specify the emitter the experiment should send data to and the initial state of the model. If we don’t specify an initial state, it will be constructed based on the defaults we specified in the composite’s processes.
Emitters¶
We provide a variety of emitter classes to save simulation output,
which can be specified with arguments to engine.
vivarium.core.emitter.RAMEmitter saves the simulation output to
RAM – this is useful for small simulations that you might not need to return to.
vivarium.core.emitter.DatabaseEmitter saves simulation output to MongoDB.
vivarium.core.emitter.NullEmitter is an easy way to ignore all emits.
Other emitters can be developed by subclassing the vivarium.core.emitter.Emitter class.
Data Representations¶
We represent simulation data in three ways:
Raw Data
Embedded Timeseries, which are sometimes called just “timeseries”
Path Timeseries
We describe each in more detail below.
Raw Data¶
The raw data is a dictionary with times as keys and the simulation state at that time as a dictionary. For example:
{
0: {
'agents': {
'agent1': {
'boundary': {
'volume': 20,
},
},
'agent2': {
'boundary': {
'volume': 10,
},
},
},
},
1: {
'agents': {
'agent1': {
'boundary': {
'volume': 30,
},
},
'agent2': {
'boundary': {
'volume': 20,
},
},
},
},
}
You can get this data from an emitter using its
vivarium.core.emitter.Emitter.get_data() method. We recommend
keeping the main copy of your data in this form throughout your code
because it can be transformed into either of the other two forms using
the functions:
vivarium.core.emitter.path_timeseries_from_data()to get a path timeseriesvivarium.core.emitter.timeseries_from_data()to get an embedded timeseries
Embedded Timeseries¶
Note
Embedded timeseries are sometimes called just “timeseries.”
An embedded timeseries is a dictionary with the same form as the
simulation state dictionary, only with an additional time key. Each
variable in the dictionary is a key nested arbitrarily deep
within the state dictionary. Each of these keys has as its value a list
of the variable’s values at each time in the list of timepoints
associated with the time key. For example:
{
'agents': {
'agent1': {
'boundary': {
'volume': [20, 30],
},
},
'agent2': {
'boundary': {
'volume': [10, 20],
},
},
},
'time': [0, 1],
}
You can get data in this format from an emitter using its
vivarium.core.emitter.Emitter.get_timeseries() function.
Path Timeseries¶
A path timeseries is a flattened form of an embedded timeseries. We take
each variable and its list of timepoints from an embedded timeseries and
make each its own entry in the dictionary. The keys are tuples
specifying the paths to each variable, and the values are the lists of
timepoints. Like in embedded timeseries, we also have a time key
with the time values for each timepoint. For example:
{
('agents', 'agent1', 'boundary', 'volume'): [20, 30],
('agents', 'agent2', 'boundary', 'volume'): [10, 20],
'time': [0, 1],
}
Hierarchy¶
In Vivarium, we represent the simulation as a tree of processes and stores (panel A). The processes and stores are wired together by topologies to form compartments (panel B), which can then be nested to form a tree called the hierarchy (panel C).
The relationships between stores, processes (panel A), and compartments (panel B) in the hierarchy (panel C).¶
Note that in panel C, only the compartments and boundary stores are shown. The full hierarchy also contains the stores and processes within each compartment.
Note
Here we have shown a simplified picture where processes are only wired to stores in their own compartment or to boundary stores. In reality, processes can be wired to any store in the hierarchy, but keeping cross-compartment wiring to a minimum can help simplify your models.
We recommend using Vivarium compartments to represent spatial regions of your modeled system that are conceptually distinct. For example, you might model a cell as a compartment and its environment as another compartment. This is not a technical requirement though, so you can use compartments to represent whatever makes sense for your circumstances.
Compartment Interactions¶
We model cross-compartment interactions using boundary stores between compartments. For example, the boundary store between a cell and its environment might track the flux of metabolites between the cell and environment compartments.
When compartments are nested, these boundary stores also exist between the inner and the outer compartment. Thus nested compartments form a tree whose nodes are compartments and whose edges are boundary stores. A node’s parent is its outer compartment, while its children are the compartments within it.
Since boundary stores can also exist between compartments who share a parent, you may find it useful to think of compartments and their boundary stores as a bigraph (not a bipartite graph) where the tree denotes nesting and all the edges (including those in the tree) represent boundary stores.
Hierarchy Structure¶
In the example below, we print out the full hierarchy as a dictionary.
>>> from vivarium.experiments.glucose_phosphorylation import (
... glucose_phosphorylation_experiment,
... )
>>> from vivarium.core.engine import Engine
>>> from vivarium.core.composer import Composer
>>> from vivarium.library.pretty import format_dict
>>>
>>>
>>> experiment = glucose_phosphorylation_experiment()
>>> print(format_dict(experiment.state.get_config()))
{
"cell": {
"ADP": {
"_default": 0.0,
"_emit": true,
"_updater": "<function update_accumulate>",
"_value": 0.0
},
"ATP": {
"_default": 2.0,
"_emit": true,
"_updater": "<function update_accumulate>",
"_value": 2.0
},
"G6P": {
"_default": 0.0,
"_emit": true,
"_properties": {
"mw": "1.0 gram / mole"
},
"_updater": "<function update_accumulate>",
"_value": 0.0
},
"GLC": {
"_default": 1.0,
"_emit": true,
"_properties": {
"mw": "1.0 gram / mole"
},
"_updater": "<function update_accumulate>",
"_value": 1.0
},
"HK": {
"_default": 0.1,
"_properties": {
"mw": "1.0 gram / mole"
},
"_updater": "<function update_accumulate>",
"_value": 0.1
}
},
"global": {
"initial_mass": {
"_default": "0.0 femtogram",
"_divider": "<function divide_split>",
"_units": "<Unit('femtogram')>",
"_updater": "<function update_set>",
"_value": "0.0 femtogram"
},
"mass": {
"_default": null,
"_emit": true,
"_updater": "<function update_set>",
"_value": "1.826592973891231e-09 femtogram"
}
},
"glucose_phosphorylation": {
"_default": null,
"_updater": "<function update_set>",
"_value": "<vivarium.processes.glucose_phosphorylation.GlucosePhosphorylation object>"
},
"injector": {
"_default": null,
"_updater": "<function update_set>",
"_value": "<vivarium.processes.injector.Injector object>"
},
"my_deriver": {
"_default": null,
"_updater": "<function update_set>",
"_value": "<vivarium.processes.tree_mass.TreeMass object>"
}
}
We can represent this hierarchy graphically like this:
Notice that in the dictionary above, each leaf node in the tree is a key with a value that is a dictionary of schema keys.
Hierarchy Paths¶
A hierarchy in Vivarium is like a directory tree on a filesystem. In
line with this analogy, we specify nodes in the hierarchy with paths.
Each path is a tuple of node names (variable names or store names)
relative to some other node. For example, in the topology from the
example above, we used the path ('cell', ) to say that the cell
store maps to the injector’s internal port. This path was
relative to the compartment root (root in our diagram) as is the
case for all topologies. Thus the path is analogous to ./cell in a
directory.
Special Symbols¶
Continuing our analogy between hierarchy paths and file paths, the following symbols have special meanings in hierarchy paths:
..refers to a parent node. One example use for this is a division process that needs to access the parent (environment) compartment to create the daughter cells. In fact, this is what we do in the growth and division compartment.
Versioning and API Stability¶
We want to keep our API stable so that you can rely on it. To help clearly communicate how our API is changing, we strive to following the guidelines below when releasing new versions of Vivarium.
We follow semantic versioning, which in brief means the following:
Our version numbers have the form
major.minor.patch(e.g.1.2.3).We increment
majorfor breaking changes to Vivarium’s supported application programming interface (API). Upgrading to a new major release may break your code.We increment
minorwhen we release new functionality. New minor releases should be backwards-compatible with the previous version of Vivarium.We increment
patchfor bug fixes that neither release new functionality nor introduce breaking changes.
Note that the above rules only apply to Vivarium’s supported API. If you
use unsupported features, they may break with a new minor or patch
release. Our supported API consists of all public, documented interfaces
that are not marked as being experimental in their documentation. Note
that attributes beginning with an underscore (_) are private. The
following are not considered breaking API changes:
Adding to a function a parameter with a default value (i.e. an optional parameter).
Adding a new key to a dictionary. For example, expanding the
Compositedictionary to include an extra key.When a function accepts a dictionary as an argument, adding more, optional keys to that dictionary. For example, letting the user specify a new key in a configuration dictionary.
Changes to the supported API will be reflected in Vivarium’s versioning. We will also try to mark interfaces as deprecated in their documentation and raise warnings at least one release before actually removing them.
Working with Documentation¶
We write Vivarium’s documentation in plain text that utilizes the reStructured Text markup language. You can compile it to HTML with Sphinx, and you can also read it as plain text.
Reading Documentation¶
You’re welcome to read the plain text documentation in this folder, but you’ll probably enjoy the pretty HTML version more. Read the Docs hosts the compiled HTML here: https://wc-vivarium.rtfd.io/
If you want to generate the HTML documentation yourself, check out the instructions on building documentation below.
Writing Documentation¶
Where to Write¶
We write four kinds of documentation for Vivarium:
Tutorials: These are step-by-step instructions that walk the reader through doing something with Vivarium. We store these in
doc/tutorials. Make sure to list all tutorials indoc/tutorials/index.rstso that they appear in the sidebar and the list of tutorials.Guides: These dive into the details of Vivarium and should be comprehensive. We store guides in
doc/guidesand list them indoc/guides/index.rst. Guides should focus on the conceptual aspects of Vivarium, leaving technical details to the API reference.References: Reference material should cater to users who already know what they’re looking for and just need to find it. For example, a user looking up a particular process or term. Our reference material consists of a glossary and an API reference. The glossary is stored in
doc/glossary.rst, while the API reference is auto-generated from docstrings in the code. These docstrings can take advantage of all the reStructuredText syntax we use elsewhere in Vivarium. Eventually, we will remove from the reference material the stubs for functions that aren’t user-facing and the auto-generated titles on each page.For an example of reference documentation that defines an API, see the
deathprocess. For an example of documentation that explains how to use a process, look at themetabolismprocess (both are invivarium-cell).Note
From the compiled HTML reference documentation, you can click on
[source]to see the source code, including the docstrings. This can be helpful for looking up reStructuredText syntax.Warning
For each class, include at most one of the class and constructor docstrings. They are concatenated when the HTML is compiled, so you can provide either one.
class MyClass: '''This is the class docstring''' def __init__(self): '''This is the constructor docstring'''
Glossary vs Guide vs API Reference¶
In the guide, describe the concept and perhaps our rationale behind any design choices we made. Link terms to the glossary, which succinctly describes the term and links to relevant API reference pages and guides. In the API reference, describe the technical details.
We try to keep technical details in the API reference because the API reference is built from docstrings. Since these docstrings live alongside the code, they are more likely to be kept up-to-date than a separate guide.
Pointers for Technical Writing¶
Here are resources for writing good documentation and technical writing in general:
Style Guide¶
Here we document the stylistic decisions we have made for this documentation:
We use first-person plural pronouns to refer to ourselves (e.g. “We decided”).
We write tutorials in the second-person, future tense, for example “First, you’ll need to install”. We also frequently use the imperative (“Install this”).
We use the following admonitions. We don’t want to overload our users with admonitions, so we don’t use any others.
We warn users about potential problems with warning admonitions. These often describe important steps that we think users might forget.
Warning
.. WARNING::These are also appropriate for warning users about experimental APIs.
We use notes to highlight important points. These should not be used for asides that aren’t important enough to integrate directly into the text.
Note
.. note::We give users helpful tips using the tip admonition. These help highlight tips that some users might not use but that will help users who are debugging problems.
Tip
.. tip::We use danger admonitions for the most critical warnings. Use these sparingly.
Danger
.. DANGER::
We use Vale to lint our documentation. You can run the linter by executing
doc/test.sh. This linter checks some Vivarium-specific naming and capitalization conventions. It also runs theproselintandwrite-goodlinters, which check for generally good style.
What APIs to Document¶
A standard convention in software development is that documented APIs (e.g. functions, methods, classes, and constants) are supported, so other users can count on them to work. Any changes to the inputs these APIs accept or their behavior constitutes an API change, and any changes that are not backward-compatible are breaking API changes that require a new major version release according to Semantic Versioning.
For Vivarium, being “documented” means appearing in the compiled API reference. Sphinx includes docstrings in this reference as follows:
By default, everything appears in the reference, even if it doesn’t have a docstring.
Tests (any members whose names start with
test_) are not included, even if they have docstrings.Private members (members that start with
_) are not included, even if they have docstrings.
Building the Documentation¶
To build the documentation, we will use Sphinx to generate HTML files from plain text. Here are stepwise instructions:
(optional) Create a virtual environment for the documentation-building packages. You might want this to be separate from the environment you use for the rest of Vivarium.
Install dependencies:
$ pip install -r doc/requirements.txtBuild the HTML!
$ cd doc $ make html
Your HTML will now be in
doc/_build/html. To view it, opendoc/_build/html/index.htmlin a web browser.
Tutorials¶
These Jupyter Notebooks illustrate how to use Vivarium. You can click on a notebook below to view a static rendering of the notebook’s code and output, or you can download and execute the notebook yourself. These notebooks can be found under notebooks/ on GitHub.
Vivarium interface basics¶
Overview¶
This notebook introduces the Vivarium interface protocol by working through a simple, qualitative example of transcription/translation, and iterating on model design to add more complexity.
Note: The included examples often skirt best coding practice in favor of simplicity. See Vivarium templates on github for better starting examples: https://github.com/vivarium-collective/vivarium-template
[1]:
#uncomment to install vivarium-core
#!pip install vivarium-core
[2]:
# Imports and Notebook Utilities
import os
import copy
import pylab as plt
import numpy as np
from scipy import constants
import matplotlib.pyplot as plt
import matplotlib
# Process, Deriver, and Composer base classes
from vivarium.core.process import Process, Deriver
from vivarium.core.composer import Composer
from vivarium.core.registry import process_registry
# other vivarium imports
from vivarium.core.engine import Engine, pp
from vivarium.library.units import units
# plotting functions
from vivarium.plots.simulation_output import (
plot_simulation_output, plot_variables)
from vivarium.plots.simulation_output import _save_fig_to_dir as save_fig_to_dir
from vivarium.plots.agents_multigen import plot_agents_multigen
from vivarium.plots.topology import plot_topology
# supress warnings in notebook
import warnings
warnings.filterwarnings('ignore')
AVOGADRO = constants.N_A * 1 / units.mol
# helper functions for composition
def process_in_experiment(
process, settings, initial_state):
composite = process.generate()
return Engine(
composite=composite,
initial_state=initial_state,
**settings)
def composite_in_experiment(
composite, settings, initial_state):
return Engine(
composite=composite,
initial_state=initial_state,
**settings)
store_cmap = matplotlib.cm.get_cmap('Dark2')
dna_color = matplotlib.colors.to_rgba(store_cmap(0))
rna_color = matplotlib.colors.to_rgba(store_cmap(1))
protein_color = matplotlib.colors.to_rgba(store_cmap(2))
global_color = matplotlib.colors.to_rgba(store_cmap(7))
store_colors = {
'DNA': dna_color,
'DNA\n(counts)': dna_color,
'DNA\n(mg/mL)': dna_color,
'mRNA': rna_color,
'mRNA\n(counts)': rna_color,
'mRNA\n(mg/mL)': rna_color,
'Protein': protein_color,
'Protein\n(mg/mL)': protein_color,
'global': global_color}
# plotting configurations
topology_plot_config = {
'settings': {
'coordinates': {
'Tl': (-1,0),
'Tx': (-1,-1),
'Protein': (1,0),
'mRNA': (1,-1),
'DNA': (1,-2),
},
'node_distance': 3.0,
'process_color': 'k',
'store_colors': store_colors,
'dashed_edges': True,
'graph_format': 'vertical',
'color_edges': False},
'out_dir': 'out/'}
plot_var_config = {
'row_height': 2,
'row_padding': 0.2,
'column_width': 10,
'out_dir': 'out'}
Make a Process: minimal transcription¶
Transcription is the biological process by which RNA is synthesized from a DNA template. Here, we define a model with a single mRNA species, \(C\), transcribed from a single gene, \(G\), at transcription rate \(k_{tsc}\). RNA also degrades at rate \(k_{deg}\).
This can written as the difference equation \(\Delta RNA_{C} = (k_{tsc}[Gene_{G}] - k_{deg}[RNA_{C}]) \Delta t\)
Vivarium’s basic elements¶
Process interface protocol¶
If standard modeling formats are an “HTML” for systems biology, we need an “interface protocol” such as TCP/IP serves for the internet – a protocol for connecting separate systems into a complex and open-ended network that anyone can contribute to.
Making a dynamical model into a Vivarium Process requires the following protocol: 1. A constructor that accepts parameters and configures the model. 2. A ports_schema that declares the ports and their schema. 3. A next_update that runs the model and returns an update.
Constructor¶
defaultparameters are used in absense of an other provided parameters.The constructor’s
parametersarguments overrides thedefaultparameters.
class Tx(Process):
defaults = {
'ktsc': 1e-2,
'kdeg': 1e-3}
def __init__(self, parameters=None):
super().__init__(parameters)
Ports Schema¶
Ports are the connections by which Process are wired to Stores.
ports_schemadeclares the ports, the variables that go through them, and how those variables operate.Here,
Txdeclares a port formRNAwith variableC, and a port forDNAwith variableG.
def ports_schema(self):
return {
'mRNA': {
'C': {
'_default': 0.0,
'_updater': 'accumulate',
'_divider': 'set',
'_properties': {
'mw': 111.1 units.g / units.mol}},
'DNA': {
'G': {
'_default': 1.0}}
dictionary comprehensions are useful for declaring schema for configured variables.
def ports_schema(self):
molecule_schema = {
'_default': 0.0,
'_emit': True}
return {
'molecules': {
mol_id: molecule_schema
for mol_id in self.parameters['molecules']}}
Use the glob
'*'schema to declare expected sub-store structure, and view all child values of the store:
def ports_schema(self):
schema = {
'port1': {
'*': {
'_default': 1.0
}
}
}
Use the glob
'**'schema to connect to an entire sub-branch, including child nodes, grandchild nodes, etc:
def ports_schema(self):
schema = {
'port1': {
'*': {
'_default': 1.0
}
}
}
Schema methods can also be declared by passing in functions.
The asymmetric_division divider makes molecules in the ‘front’ go to one daughter cell upon division, and those in the ‘back’ go to the other daughter.
def asymmetric_division(value, topology):
if 'front' in topology:
return [value, 0.0]
elif 'back' in topology:
return [0.0, value]
def ports_schema(self):
return {
'front': {
'molecule': {
'_divider': {
'divider': asymmetric_division,
'topology': {'front': ('molecule',)},
}}},
'back': {
'molecule': {
'_divider': {
'divider': asymmetric_division,
'topology': {'back': ('molecule',)},
}}}}
Initial State¶
Each Process MAY provide an
initial_statemethod. This can be retrieved, reconfigured, and passed into a simulation.If left empty, a simulation initializes at the
'_default'values.
def initial_state(self, config):
return {
'DNA': {'G': 1.0},
'mRNA': {'C': 0.0}}
Update Method¶
Retrieve the state variables through the ports.
Run the model for the timestep’s duration.
Return an update to the state variable through the ports.
def next_update(self, states, timestep):
# Retrieve
G = states['DNA']['G']
C = states['mRNA']['C']
# Run
dC = (self.ktsc * G - self.kdeg * C) * timestep
# Return
return {
'mRNA': {
'C': dC}}
Tx: a deterministic transcription process¶
According to BioNumbers, the concentration of DNA in an E. coli cell is on the order of 11-18 mg/mL. The concentration of RNA is 75-120 mg/ml.
[3]:
class Tx(Process):
defaults = {
'ktsc': 1e-2,
'kdeg': 1e-3}
def __init__(self, parameters=None):
super().__init__(parameters)
def ports_schema(self):
return {
'DNA': {
'G': {
'_default': 10 * units.mg / units.mL,
'_updater': 'accumulate',
'_emit': True}},
'mRNA': {
'C': {
'_default': 100 * units.mg / units.mL,
'_updater': 'accumulate',
'_emit': True}}}
def next_update(self, timestep, states):
G = states['DNA']['G']
C = states['mRNA']['C']
dC = (self.parameters['ktsc'] * G - self.parameters['kdeg'] * C) * timestep
return {
'mRNA': {
'C': dC}}
plot Tx topology¶
[4]:
fig = plot_topology(Tx(), filename='tx_topology.pdf', **topology_plot_config)
Writing out/tx_topology.pdf
run Tx¶
[5]:
# tsc configuration
tx_config = {'time_step': 10}
tx_sim_settings = {
'experiment_id': 'TX'}
tx_initial_state = {
'mRNA': {'C': 0.0 * units.mg/units.mL}}
tx_plot_config = {
'variables': [
{
'variable': ('mRNA', ('C', 'milligram / milliliter')),
'color': store_colors['mRNA']
},
{
'variable': ('DNA', ('G', 'milligram / milliliter')),
'color': store_colors['DNA']
}],
'filename': 'tx_output.pdf',
**plot_var_config}
[6]:
# initialize
tx_process = Tx(tx_config)
# make the experiment
tx_exp = process_in_experiment(
tx_process, tx_sim_settings, tx_initial_state)
# run
tx_exp.update(10000)
# retrieve the data as a timeseries
tx_output = tx_exp.emitter.get_timeseries()
# plot
fig = plot_variables(tx_output, **tx_plot_config)
Simulation ID: TX
Created: 05/18/2022 at 09:43:24
Completed in 0.283171 seconds
Writing out/tx_output.pdf
Tl: a deterministic translation process¶
Translation is the biological process by which protein is synthesized with an mRNA template. Here, we define a model with a single protein species, \(Protein_{X}\), transcribed from a single gene, \(RNA_{C}\), at translation rate \(k_{trl}\). Protein also degrades at rate \(k_{deg}\).
This can be represented by a chemical reaction network with the form: * $RNA_{C} \xrightarrow[]{k_{trl}} RNA_{C} + Protein_{X} $ * \(Protein_{X} \xrightarrow[]{k_{deg}} \emptyset\)
According to BioNumbers, the concentration of RNA in an E. coli cell is on the order of 75-120 mg/ml. The concentration of protein is 200-320 mg/ml.
[7]:
class Tl(Process):
defaults = {
'ktrl': 5e-4,
'kdeg': 5e-5}
def ports_schema(self):
return {
'mRNA': {
'C': {
'_default': 100 * units.mg / units.mL,
'_divider': 'split',
'_emit': True}},
'Protein': {
'X': {
'_default': 200 * units.mg / units.mL,
'_divider': 'split',
'_emit': True}}}
def next_update(self, timestep, states):
C = states['mRNA']['C']
X = states['Protein']['X']
dX = (self.parameters['ktrl'] * C - self.parameters['kdeg'] * X) * timestep
return {
'Protein': {
'X': dX}}
run Tl¶
[8]:
# trl configuration
tl_config = {'time_step': 10}
tl_sim_settings = {'experiment_id': 'TL'}
tl_initial_state = {
'Protein': {'X': 0.0 * units.mg / units.mL}}
tl_plot_config = {
'variables': [
{
'variable': ('Protein', ('X', 'milligram / milliliter')),
'color': store_colors['Protein']
},
{
'variable': ('mRNA', ('C', 'milligram / milliliter')),
'color': store_colors['mRNA']
},
],
'filename': 'tl_output.pdf',
**plot_var_config}
[9]:
# initialize
tl_process = Tl(tl_config)
# make the experiment
tl_exp = process_in_experiment(
tl_process, tl_sim_settings, tl_initial_state)
# run
tl_exp.update(10000)
# retrieve the data as a timeseries
tl_output = tl_exp.emitter.get_timeseries()
# plot
fig = plot_variables(tl_output, **tl_plot_config)
Simulation ID: TL
Created: 05/18/2022 at 09:43:26
Completed in 0.297745 seconds
Writing out/tl_output.pdf
Make a Composite¶
A Composite is a set of Processes and Stores. Vivarium constructs the Stores from the Processes’s port_schema methods and wires them up as instructed by a Topology. The only communication between Processes is through variables in shared Stores.
TxTl: a transcription/translation composite¶
We demonstrate composition by combining the Tx and Tl processes.
Composition protocol¶
Composers, which combine processes into composites are implemented with the protocol: 1. A constructor that accepts configuration data, which can override the consituent Processes’ default parameters. 1. A generate_processes method that constructs the Processes, passing model parameters as needed. 2. A generate_topology method that returns the Topology definition which tells Vivarium how to wire up the Processes to Stores.
class TxTl(Composer):
defaults = {
'Tx': {
'ktsc': 1e-2},
'Tl': {
'ktrl': 1e-3}}
def __init__(self, config=None):
super().__init__(config)
Here,
generate_topology()returns the Topology definition that wires these Processes together with 3 Stores, one of them shared.
def generate_topology(self, config):
return {
'Tx': {
'DNA': ('DNA',), # connect TSC's 'DNA' Port to a 'DNA' Store
'mRNA': ('mRNA',)}, # connect TSC's 'mRNA' Port to a 'mRNA' Store
'Tl': {
'mRNA': ('mRNA',), # connect TRL's 'mRNA' Port to the same 'mRNA' Store
'Protein': ('Protein',)}}
embedding in a hierarchy: to connect to sub-stores in a hierarchy, declare the path through each substore, as done to ‘lipids’.
To connect to supra-stores use
'..'for each level up, as done to'external'.
splitting ports: One port can connect to multiple stores by specifying the path for each variable, as is done to
'transport'.This can be used to re-map variable names, for integration of different models.
def generate_topology(config):
return {
'process_1': {
'lipids': ('organelle', 'membrane', 'lipid'),
'external': ('..', 'environment'),
'transport': {
'glucose_external': ('external', 'glucose'),
'glucose_internal': ('internal', 'glucose'),
}
}}
TxTl Composer¶
[10]:
class TxTl(Composer):
defaults = {
'Tx': {'time_step': 10},
'Tl': {'time_step': 10}}
def generate_processes(self, config):
return {
'Tx': Tx(config['Tx']),
'Tl': Tl(config['Tl'])}
def generate_topology(self, config):
return {
'Tx': {
'DNA': ('DNA',),
'mRNA': ('mRNA',)},
'Tl': {
'mRNA': ('mRNA',),
'Protein': ('Protein',)}}
plot TxTl topology¶
[11]:
txtl_topology_plot_config = copy.deepcopy(topology_plot_config)
txtl_topology_plot_config['settings']['node_distance'] = 2
fig = plot_topology(TxTl(), filename='txtl_topology.pdf', **topology_plot_config)
Writing out/txtl_topology.pdf
run TxTl¶
[12]:
# tsc_trl configuration
txtl_config = {}
txtl_exp_settings = {'experiment_id': 'TXTL'}
txtl_plot_config = {
'variables':[
{
'variable': ('Protein', ('X', 'milligram / milliliter')),
'color': store_colors['Protein']
},
{
'variable': ('mRNA', ('C', 'milligram / milliliter')),
'color': store_colors['mRNA']
},
{
'variable': ('DNA', ('G', 'milligram / milliliter')),
'color': store_colors['DNA']
},
],
'filename': 'txtl_output.pdf',
**plot_var_config}
tl_initial_state = {
'mRNA': {'C': 0.0 * units.mg / units.mL},
'Protein': {'X': 0.0 * units.mg / units.mL}}
[13]:
# construct TxTl
txtl_composite = TxTl(txtl_config).generate()
# make the experiment
txtl_experiment = composite_in_experiment(
txtl_composite, txtl_exp_settings, tl_initial_state)
# run it and retrieve the data that was emitted to the simulation log
txtl_experiment.update(20000)
txtl_output = txtl_experiment.emitter.get_timeseries()
# plot the output
fig = plot_variables(txtl_output, **txtl_plot_config)
Simulation ID: TXTL
Created: 05/18/2022 at 09:43:27
Completed in 0.862072 seconds
Writing out/txtl_output.pdf
Adding Complexity¶
Process modularity allows modelers to iterate on model design by swapping out different models. We demonstrated this by replacing the deterministic Transcription Process with a Stochastic Transcription Process.
Stochastic transcription requires variable timesteps, which Vivarium accomodates with multi-timestepping.
StochasticTx: a stochastic transcription process¶
This process uses the Gillespie algorithm in its next_update() method.
[14]:
stoch_exp_settings = {
'settings': {
'experiment_id': 'stochastic_txtl'},
'initial_state': {
'DNA\n(counts)': {
'G': 1.0
},
'mRNA\n(counts)': {
'C': 0.0
},
'Protein\n(mg/mL)': {
'X': 0.0 * units.mg / units.mL
}}}
stoch_plot_config = {
'variables':[
{
'variable': ('Protein\n(mg/mL)', ('X', 'milligram / milliliter')),
'color': store_colors['Protein'],
'display': 'Protein: X (mg/mL)'},
{
'variable': ('mRNA\n(mg/mL)', ('C', 'milligram / milliliter')),
'color': store_colors['mRNA'],
'display': 'mRNA: C (mg/mL)'},
{
'variable': ('DNA\n(counts)', 'G'),
'color': store_colors['DNA'],
'display': 'DNA: G (counts)'},
],
'filename': 'stochastic_txtl_output.pdf',
**plot_var_config}
[15]:
class StochasticTx(Process):
defaults = {
'ktsc': 1e0,
'kdeg': 1e-3}
def __init__(self, parameters=None):
super().__init__(parameters)
self.ktsc = self.parameters['ktsc']
self.kdeg = self.parameters['kdeg']
self.stoichiometry = np.array([[0, 1], [0, -1]])
self.time_left = None
self.event = None
# initialize the next timestep
initial_state = self.initial_state()
self.calculate_timestep(initial_state)
def initial_state(self, config=None):
return {
'DNA': {
'G': 1.0
},
'mRNA': {
'C': 1.0
}
}
def ports_schema(self):
return {
'DNA': {
'G': {
'_default': 1.0,
'_emit': True}},
'mRNA': {
'C': {
'_default': 1.0,
'_emit': True}}}
def calculate_timestep(self, states):
# retrieve the state values
g = states['DNA']['G']
c = states['mRNA']['C']
array_state = np.array([g, c])
# Calculate propensities
propensities = [
self.ktsc * array_state[0], self.kdeg * array_state[1]]
prop_sum = sum(propensities)
# The wait time is distributed exponentially
self.calculated_timestep = np.random.exponential(scale=prop_sum)
return self.calculated_timestep
def next_reaction(self, x):
"""get the next reaction and return a new state"""
propensities = [self.ktsc * x[0], self.kdeg * x[1]]
prop_sum = sum(propensities)
# Choose the next reaction
r_rxn = np.random.uniform()
i = 0
for i, _ in enumerate(propensities):
if r_rxn < propensities[i] / prop_sum:
# This means propensity i fires
break
x += self.stoichiometry[i]
return x
def next_update(self, timestep, states):
if self.time_left is not None:
if timestep >= self.time_left:
event = self.event
self.event = None
self.time_left = None
return event
self.time_left -= timestep
return {}
# retrieve the state values, put them in array
g = states['DNA']['G']
c = states['mRNA']['C']
array_state = np.array([g, c])
# calculate the next reaction
new_state = self.next_reaction(array_state)
# get delta mRNA
c1 = new_state[1]
d_c = c1 - c
update = {
'mRNA': {
'C': d_c}}
if self.calculated_timestep > timestep:
# didn't get all of our time, store the event for later
self.time_left = self.calculated_timestep - timestep
self.event = update
return {}
# return an update
return {
'mRNA': {
'C': d_c}}
plot variable timesteps¶
[16]:
stoch_tx_process = StochasticTx(tx_config)
stoch_tx_exp = process_in_experiment(stoch_tx_process, **stoch_exp_settings)
stoch_tx_exp.update(10000)
stoch_tx_output = stoch_tx_exp.emitter.get_timeseries()
# calculate the timesteps
times = stoch_tx_output['time']
timesteps = []
for x, y in zip(times[0::], times[1::]):
# if y-x != 1.0:
timesteps.append(y-x)
fig = plt.hist(timesteps, 50, color='tab:gray')
plt.xlabel('timestep (seconds)')
plt.savefig('out/stochastic_timesteps.pdf')
Simulation ID: stochastic_txtl
Created: 05/18/2022 at 09:43:29
Completed in 0.473557 seconds
Auxiliary Processes¶
Connecting different Processes may require addition ‘helper’ Processes to make conversions and adapt their unique requirements different values.
Derivers are a subclass of Process that runs after the other dynamic Processes and derives some states from others.
A concentration deriver convert the counts of the stochastic process to concentrations. This is available in the process_registry
concentrations_deriver = process_registry.access('concentrations_deriver')
Combining stochastic Tx with deterministic Tl¶
[17]:
# configuration data
mw_config = {'C': 1e8 * units.g / units.mol}
[18]:
class StochasticTxTl(Composer):
defaults = {
'stochastic_Tx': {},
'Tl': {'time_step': 1},
'concs': {
'molecular_weights': mw_config}}
def generate_processes(self, config):
counts_to_concentration = process_registry.access('counts_to_concentration')
return {
'stochastic\nTx': StochasticTx(config['stochastic_Tx']),
'Tl': Tl(config['Tl']),
'counts\nto\nmg/mL': counts_to_concentration(config['concs'])}
def generate_topology(self, config):
return {
'stochastic\nTx': {
'DNA': ('DNA\n(counts)',),
'mRNA': ('mRNA\n(counts)',)
},
'Tl': {
'mRNA': ('mRNA\n(mg/mL)',),
'Protein': ('Protein\n(mg/mL)',)
},
'counts\nto\nmg/mL': {
'global': ('global',),
'input': ('mRNA\n(counts)',),
'output': ('mRNA\n(mg/mL)',)
}}
plot StochasticTxTl topology¶
[19]:
# plot topology after merge
stochastic_topology_plot_config = copy.deepcopy(topology_plot_config)
stochastic_topology_plot_config['settings']['graph_format'] = 'horizontal'
stochastic_topology_plot_config['settings']['coordinates'] = {
'stochastic\nTx': (2, 1),
'counts\nto\nmg/mL': (3, 1),
'Tl': (4, 1),
'DNA\n(counts)': (1,-1),
'mRNA\n(counts)': (2,-1),
'mRNA\n(mg/mL)': (3,-1),
'Protein\n(mg/mL)': (4,-1)}
stochastic_topology_plot_config['settings']['dashed_edges'] = True
stochastic_topology_plot_config['settings']['show_ports'] = False
stochastic_topology_plot_config['settings']['node_distance'] = 2.2
stochastic_txtl = StochasticTxTl()
fig = plot_topology(
stochastic_txtl,
filename='stochastic_txtl_topology.pdf',
**stochastic_topology_plot_config)
Writing out/stochastic_txtl_topology.pdf
run StochasticTxTl¶
[20]:
# make the experiment
stoch_experiment = composite_in_experiment(stochastic_txtl.generate(), **stoch_exp_settings)
# simulate and retrieve the data
stoch_experiment.update(1000)
stochastic_txtl_output = stoch_experiment.emitter.get_timeseries()
# plot output
fig = plot_variables(stochastic_txtl_output, **stoch_plot_config)
Simulation ID: stochastic_txtl
Created: 05/18/2022 at 09:43:30
Completed in 0.983180 seconds
Writing out/stochastic_txtl_output.pdf
Growth and Division¶
We here extend the Transcription/Translation model with division. This require many instances of the processes to run simultaneously in a single simulation. To support such phenomena, Vivarium adopts an agent-based modeling bigraphical formalism, with embedded compartments that can spawn new compartments during runtime.
Hierarchical Embedding¶
To support this requirement, Processes can be embedded in a hierarchical representation of embedded compartments. Vivarium uses a bigraph formalism – a graph with embeddable nodes that can be placed within other nodes.
Hierarchy updates¶
The structure of a hierarchy has its own type of constructive dynamics with formation/destruction, merging/division, engulfing/expelling of compartments
[21]:
# add imported division processes
from vivarium.processes.divide_condition import DivideCondition
from vivarium.processes.meta_division import MetaDivision
from vivarium.processes.growth_rate import GrowthRate
TIMESTEP = 10
class TxTlDivision(Composer):
defaults = {
'stochastic_Tx': {'time_step': TIMESTEP},
'Tl': {'time_step': TIMESTEP},
'concs': {
'molecular_weights': mw_config},
'growth': {
'time_step': 1,
'default_growth_rate': 0.0005,
'default_growth_noise': 0.001,
'variables': ['volume']},
'agent_id': np.random.randint(0, 100),
'divide_condition': {
'threshold': 2.5 * units.fL},
'agents_path': ('..', '..', 'agents',),
'daughter_path': tuple(),
'_schema': {
'concs': {
'input': {'C': {'_divider': 'binomial'}},
'output': {'C': {'_divider': 'set'}},
}
}
}
def generate_processes(self, config):
counts_to_concentration = process_registry.access('counts_to_concentration')
division_config = dict(
daughter_path=config['daughter_path'],
agent_id=config['agent_id'],
composer=self)
return {
'stochastic_Tx': StochasticTx(config['stochastic_Tx']),
'Tl': Tl(config['Tl']),
'concs': counts_to_concentration(config['concs']),
'growth': GrowthRate(config['growth']),
'divide_condition': DivideCondition(config['divide_condition']),
'division': MetaDivision(division_config),
}
def generate_topology(self, config):
return {
'stochastic_Tx': {
'DNA': ('DNA',),
'mRNA': ('RNA_counts',)
},
'Tl': {
'mRNA': ('RNA',),
'Protein': ('Protein',)
},
'concs': {
'global': ('global',),
'input': ('RNA_counts',),
'output': ('RNA',)
},
'growth': {
'variables': ('global',),
'rates': ('rates',)
},
'divide_condition': {
'variable': ('global', 'volume',),
'divide': ('global', 'divide',)},
'division': {
'global': ('global',),
'agents': config['agents_path']}
}
Colony-level processes¶
[22]:
from vivarium.library.units import Quantity
def calculate_volume(value, path, node):
if isinstance(node.value, Quantity) and node.units == units.fL:
return value + node.value
else:
return value
class ColonyVolume(Deriver):
defaults = {
'colony_path': ('..', '..', 'agents')}
def ports_schema(self):
return {
'colony': {
'volume': {
'_default': 1.0 * units.fL,
'_updater': 'set',
'_emit': True}}}
def next_update(self, timestep, states):
return {
'colony': {
'volume': {
'_reduce': {
'reducer': calculate_volume,
'from': self.parameters['colony_path'],
'initial': 0.0 * units.fL}}}}
[23]:
# configure hierarchy
# agent config
agent_id = '0'
agent_config = {'agent_id': agent_id}
# environment config
env_config = {}
# initial state
hierarchy_initial_state = {
'agents': {
agent_id: {
'global': {'volume': 1.2 * units.fL},
'DNA': {'G': 1},
'RNA': {'C': 5 * units.mg / units.mL},
'Protein': {'X': 50 * units.mg / units.mL}}}}
# experiment settings
exp_settings = {
'experiment_id': 'hierarchy_experiment',
'initial_state': hierarchy_initial_state,
'emit_step': 100.0}
# plot config
hierarchy_plot_settings = {
'include_paths': [
('RNA_counts', 'C'),
('RNA', 'C'),
('DNA', 'G'),
('Protein', 'X'),
],
'store_order': ('Protein', 'RNA_counts', 'DNA', 'RNA'),
'titles_map': {
('Protein', 'X',): 'Protein',
('RNA_counts', 'C'): 'RNA',
('DNA', 'G',): 'DNA',
'RNA': 'RNA',
},
'column_width': 6,
'row_height': 1.0,
'stack_column': True,
'tick_label_size': 10,
'linewidth': 1,
'title_size': 10}
colony_plot_config = {
'variables': [('global', ('volume', 'femtoliter'))],
'filename': 'colony_growth.pdf',
**plot_var_config}
# hierarchy topology plot
agent_0_string = 'agents\n0'
agent_1_string = 'agents\n00'
agent_2_string = 'agents\n01'
row_1 = 0
row_2 = -1
row_3 = -2
row_4 = -3
node_space = 0.75
vertical_space=0.9
bump = 0.1
process_column = -0.2
agent_row = -3.2
agent_column = bump/2 #0.5
hierarchy_topology_plot_config = {
'settings': {
'graph_format': 'hierarchy',
'node_size': 6000,
'process_color': 'k',
'store_color': global_color,
'store_colors': {
f'{agent_0_string}\nDNA': dna_color,
f'{agent_0_string}\nRNA': rna_color,
f'{agent_0_string}\nRNA_counts': rna_color,
f'{agent_0_string}\nProtein': protein_color,
},
'dashed_edges': True,
'show_ports': False,
'coordinates': {
# Processes
'ColonyVolume': (2.5, 0),
'agents\n0\nstochastic_Tx': (agent_column, agent_row*vertical_space),
'agents\n0\nTl': (agent_column+node_space, agent_row*vertical_space),
'agents\n0\nconcs': (agent_column+2*node_space, agent_row*vertical_space),
'agents\n0\ndivision': (agent_column+3*node_space, agent_row*vertical_space),
# Stores
'agents': (1.5*node_space, row_1*vertical_space),
'agents\n0': (1.5*node_space, row_2*vertical_space),
'agents\n0\nagents': (1.5*node_space, row_1*vertical_space),
'agents\n0\nDNA': (0, row_3*vertical_space),
'agents\n0\nRNA_counts': (node_space+bump, row_3*vertical_space),
'agents\n0\nRNA': (node_space, (row_3-bump)*vertical_space),
'agents\n0\nProtein': (2*node_space+bump, row_3*vertical_space),
'agents\n0\nglobal': (3*node_space+bump, row_3*vertical_space),
},
'node_labels': {
# Processes
'ColonyVolume': 'Colony\nVolume',
'agents\n0\nstochastic_Tx': 'stochastic\nTx',
'agents\n0\nTl': 'Tl',
'agents\n0\nconcs': 'counts\nto\nmg/mL',
'agents\n0\ngrowth': 'growth',
'agents\n0\ndivision': 'division',
# Stores
# third
'agents\n0': '0',
'agents\n0\nDNA': 'DNA',
'agents\n0\nRNA': 'RNA',
'agents\n0\nrates': 'rates',
'agents\n0\nRNA_counts': '',
'agents\n0\nglobal': 'global',
'agents\n0\nProtein': 'Protein',
# fourth
'agents\n0\nrates\ngrowth_rate': 'growth_rate',
'agents\n0\nrates\ngrowth_noise': 'growth_noise',
},
'remove_nodes': [
'agents\n0\nrates\ngrowth_rate',
'agents\n0\nrates\ngrowth_noise',
'agents\n0\nrates',
'agents\n0\ngrowth',
'agents\n0\ndivide_condition',
'agents\n0\nglobal\ndivide',
'agents\n0\nglobal\nvolume',
]
},
'out_dir': 'out/'
}
# topology plot config for after division
agent_2_dist = 3.5
hierarchy_topology_plot_config2 = copy.deepcopy(hierarchy_topology_plot_config)
# redo coordinates, labels, store_colors, and removal
hierarchy_topology_plot_config2['settings']['node_distance'] = 2.5
hierarchy_topology_plot_config2['settings']['coordinates'] = {}
hierarchy_topology_plot_config2['settings']['node_labels'] = {}
hierarchy_topology_plot_config2['settings']['store_colors'] = {}
# hierarchy_topology_plot_config2['settings']['remove_nodes'] = []
for node_id, coord in hierarchy_topology_plot_config['settings']['coordinates'].items():
if agent_0_string in node_id:
new_id1 = node_id.replace(agent_0_string, agent_1_string)
new_id2 = node_id.replace(agent_0_string, agent_2_string)
hierarchy_topology_plot_config2['settings']['coordinates'][new_id1] = coord
hierarchy_topology_plot_config2['settings']['coordinates'][new_id2] = (coord[0]+agent_2_dist, coord[1])
else:
hierarchy_topology_plot_config2['settings']['coordinates'][node_id] = (coord[0]+agent_2_dist/2, coord[1])
hierarchy_topology_plot_config2['settings']['coordinates']['ColonyVolume'] = (5.5, 0)
for node_id, label in hierarchy_topology_plot_config['settings']['node_labels'].items():
if agent_0_string in node_id:
new_id1 = node_id.replace(agent_0_string, agent_1_string)
new_id2 = node_id.replace(agent_0_string, agent_2_string)
hierarchy_topology_plot_config2['settings']['node_labels'][new_id1] = label
hierarchy_topology_plot_config2['settings']['node_labels'][new_id2] = label
else:
hierarchy_topology_plot_config2['settings']['node_labels'][node_id] = label
hierarchy_topology_plot_config2['settings']['node_labels']['agents\n00'] = '1'
hierarchy_topology_plot_config2['settings']['node_labels']['agents\n01'] = '2'
for node_id, color in hierarchy_topology_plot_config['settings']['store_colors'].items():
if agent_0_string in node_id:
new_id1 = node_id.replace(agent_0_string, agent_1_string)
new_id2 = node_id.replace(agent_0_string, agent_2_string)
hierarchy_topology_plot_config2['settings']['store_colors'][new_id1] = color
hierarchy_topology_plot_config2['settings']['store_colors'][new_id2] = color
else:
hierarchy_topology_plot_config2['settings']['store_colors'][node_id] = color
for node_id in hierarchy_topology_plot_config['settings']['remove_nodes']:
if agent_0_string in node_id:
new_id1 = node_id.replace(agent_0_string, agent_1_string)
new_id2 = node_id.replace(agent_0_string, agent_2_string)
hierarchy_topology_plot_config2['settings']['remove_nodes'].extend([new_id1, new_id2])
use composite.merge to combine colony processes with agents¶
[24]:
# make a txtl composite, embedded under an agents store
txtl_composer = TxTlDivision(agent_config)
hierarchy_composite = txtl_composer.generate(path=('agents', agent_id))
# make a colony composite, and a topology that connects its colony port to agents store
colony_composer = ColonyVolume(env_config)
colony_composite = colony_composer.generate()
colony_topology = {'ColonyVolume': {'colony': ('agents',)}}
# perform merge
hierarchy_composite.merge(composite=colony_composite, topology=colony_topology)
plot hierarchy topology with before division¶
[25]:
fig = plot_topology(
hierarchy_composite,
filename='hierarchy_topology.pdf',
**hierarchy_topology_plot_config)
Writing out/hierarchy_topology.pdf
plot hierarchy topology after division¶
[26]:
# initial state
initial_state = {
'agents': {
agent_id: {
'global': {'volume': 1.2 * units.fL},
'DNA': {'G': 1},
'RNA': {'C': 5 * units.mg / units.mL},
'Protein': {'X': 50 * units.mg / units.mL}}}}
# make a copy of the composite
txtl_composite1 = copy.deepcopy(hierarchy_composite)
# make the experiment
hierarchy_experiment1 = composite_in_experiment(
composite=txtl_composite1,
settings={},
initial_state=initial_state)
# run the experiment long enough to divide
hierarchy_experiment1.update(2000)
Simulation ID: a7e6aa2e-d6c9-11ec-8a5b-8c85908ac627
Created: 05/18/2022 at 09:43:33
Completed in 5.25 seconds
[27]:
fig = plot_topology(
txtl_composite1,
filename='hierarchy_topology_2.pdf',
**hierarchy_topology_plot_config2)
Writing out/hierarchy_topology_2.pdf
run hierarchy experiment¶
[28]:
# initial state
initial_state = {
'agents': {
agent_id: {
'global': {'volume': 1.2 * units.fL},
'DNA': {'G': 1},
'RNA': {'C': 5 * units.mg / units.mL},
'Protein': {'X': 50 * units.mg / units.mL}}}}
# make the experiment
hierarchy_experiment = composite_in_experiment(
composite=hierarchy_composite,
settings={},
initial_state=initial_state)
# run the experiment
hierarchy_experiment.update(6000)
Simulation ID: ab2e8210-d6c9-11ec-8a5b-8c85908ac627
Created: 05/18/2022 at 09:43:39
Completed in 117.13 seconds
[29]:
# retrieve the data
hierarchy_data = hierarchy_experiment.emitter.get_data_unitless()
path_ts = hierarchy_experiment.emitter.get_path_timeseries()
# add agent colors
dimgray = (0.4,0.4,0.4)
paths = list(path_ts.keys())
agent_ids = set([path[1] for path in paths if '0' in path[1]])
agent_colors = {agent_id: dimgray for agent_id in agent_ids}
hierarchy_plot_settings.update({'agent_colors': agent_colors})
[30]:
# initialize the plot
multigen_fig = plot_agents_multigen(hierarchy_data, hierarchy_plot_settings)
plt.close()
Colony-level metrics¶
[31]:
gd_timeseries = hierarchy_experiment.emitter.get_timeseries()
colony_series = gd_timeseries['agents'][('volume', 'femtoliter')]
time_vec = gd_timeseries['time']
[32]:
# get the RNA_counts axis, to replace with colony volume
allaxes = multigen_fig.get_axes()
ax = None
for axis in allaxes:
if axis.get_title() == 'RNA \nC':
ax = axis
[33]:
# multigen_fig
ax.clear()
ax.plot(time_vec, colony_series, linewidth=1.0, color='darkslategray')
ax.set_xlim([time_vec[0], time_vec[-1]])
ax.set_title('colony volume (fL)', rotation=0)
ax.set_xlabel('time (s)')
ax.spines['bottom'].set_position(('axes', -0.2))
save_fig_to_dir(multigen_fig, 'growth_division_output.pdf')
multigen_fig
Writing out/growth_division_output.pdf
[33]:
We also have a tutorial for writing a new process that doesn’t use Jupyter Notebooks.
How to Write a Process¶
Why Write Processes?¶
Vivarium comes with a number of processes ready for you to use, but combining processes to form composites will only take you so far. For many models, the existing processes will be insufficient, so you will need to create your own.
Note
Processes are the building blocks of Vivarium models, so creating a process to model a phenomenon you know well lets other modelers build on your expertise. Processes are a great way to share your knowledge with the modeling community!
Let’s Write a Process!¶
Suppose we want to model how hexokinase helps maintain intracellular glucose concentrations. Hexokinase catalyzes glucose phosphorylation, which we can describe by the following reaction:
In the reaction above and throughout this tutorial, we will use the following abbreviations:
GLC: D-Glucose
ATP: Adenosine triphosphate
G6P: α-D-Glucose-6-phosphate
ADP: Adenosine diphosphate
v: Rate of the forward reaction
HK: Hexokinase
Tip
Once you have worked through the example in this tutorial, try modeling a different reaction that you work with!
Conceptualize the Model¶
Make Assumptions¶
A critical component of modeling is deciding what parts to describe mechanistically and what to approximate. For our scenario, we could model this reaction using molecular dynamics, but this would be computationally intensive and would not scale to larger simulations. Instead, we will assume Michaelis-Menten, sequential bisubstrate kinetics:
Let’s also assume that diffusion is much faster than the reaction so that the concentrations of our enzyme and substrates are constant throughout the cell.
Note
If you actually wanted to model a reaction like this, the convenience kinetics process can be configured to model any Michaelis-Menten enzyme kinetics.
Translate Model into Updates¶
Processes in Vivarium work by repeatedly changing the state of the model. A process makes these changes by computing an update based on the model’s current state and a timestep \(t\).
For the current example, each update will include the following:
A decrease in GLC: \(-v t\)
A decrease in ATP: \(-v t\)
An increase in G6P: \(v t\)
An increase in ADP: \(v t\)
Note
In this example and most of the time in Vivarium, we work in terms of concentrations. We also normally use units of mM, but you can use different units so long as you are consistent.
Determine the Ports¶
We partition the overall simulation state into stores, which can be shared among processes. Each process declares ports, each of which will receive a store. When creating a process, you need to decide what ports to declare.
When someone else uses your process, they will create a composite of it and other processes. These processes will interact by sharing stores. While any number of your process’s ports may be linked to the same store, a port cannot be split between stores. This means that you should put in separate ports any variables that a user might want in separate stores.
For example, ATP and ADP are turned over rapidly in the cell, so a user might want to isolate those variables from others that get updated more slowly. We will therefore create two ports:
nucleoside_phosphates: This port will store theATPandADPvariables.cytoplasm: This port will store theGLC,G6P, andHKvariables.
Implement the Model¶
To implement the model, create a new Python file named
glucose_phosphorylation.py in the vivarium/processes/ directory.
Then we create a new class that inherits from
vivarium.core.process.Process:
from vivarium.core.process import Process
class GlucosePhosphorylation(Process):
pass
The Constructor¶
In the constructor we can configure the process. We accept
configurations as a dictionary called initial_parameters. For
example, we can let the user configure the kinetic parameters
\(k_{cat}\), \(K_{GLC}\), and \(K_{ATP}\). We can also
provide default values for these parameters.
The configurations (with any missing parameters filled in with defaults) and ports are passed to the superclass constructor to instantiate the process.
import copy
from vivarium.core.process import Process
class GlucosePhosphorylation(Process):
defaults = {
'k_cat': 2e-3,
'K_ATP': 5e-2,
'K_GLC': 4e-2,
}
def __init__(self, initial_parameters=None):
if initial_parameters == None:
initial_parameters = {}
parameters = copy.deepcopy(GlucosePhosphorylation.defaults)
parameters.update(initial_parameters)
super().__init__(parameters)
It turns out that the vivarium.core.process.Process
constructor handles merging initial_parameters with the defaults
automatically, so we don’t actually have to include a constructor at
all!
Even though we’re just getting started on our process, let’s try it out!
At the bottom of the glucose_phosphorylation.py file, instantiate
the process and take a look at some of its attributes:
if __name__ == '__main__':
parameters = {
'k_cat': 1.5,
}
my_process = GlucosePhosphorylation(parameters)
print(my_process.parameters['k_cat'])
print(my_process.parameters['K_ATP'])
Then run your code by executing the whole file:
$ python glucose_phosphorylation.py
1.5
0.05
Notice that the k_cat parameter updated to the value we supplied and
that k_ATP took on the default value.
Wait! Where did the parameters attribute come from? We never
created that attribute, but
vivarium.core.process.Process made it from the
parameters argument we passed to its constructor. We’ll take
advantage of this in the next step.
Generating Updates¶
Now we can write the next_update method, which generates updates for
each port based on a provided simulation state and timestep.
Warning
The states parameter passed into the update function is
a view of the overall state, so it must not be changed.
def next_update(self, timestep, states):
# Get concentrations from state
cytoplasm = states['cytoplasm']
nucleoside_phosphates = states['nucleoside_phosphates']
hk = cytoplasm['HK']
glc = cytoplasm['GLC']
atp = nucleoside_phosphates['ATP']
# Get kinetic parameters
k_cat = self.parameters['k_cat']
k_atp = self.parameters['K_ATP']
k_glc = self.parameters['K_GLC']
# Compute reaction rate with michaelis-menten equation
rate = k_cat * hk * glc * atp / (
k_glc * k_atp + k_glc * atp + k_atp * glc + glc * atp)
# Compute concentration changes from rate and timestep
delta_glc = -rate * timestep
delta_atp = -rate * timestep
delta_g6p = rate * timestep
delta_adp = rate * timestep
# Compile changes into an update
update = {
'cytoplasm': {
'GLC': delta_glc,
'G6P': delta_g6p,
# We exclude HK because it doesn't change
},
'nucleoside_phosphates': {
'ATP': delta_atp,
'ADP': delta_adp,
},
}
return update
Now let’s test this update function by seeing how it changes a state we provide. Replace the testing code we added to the bottom of the file with this:
if __name__ == '__main__':
parameters = {
'k_cat': 1.5,
}
my_process = GlucosePhosphorylation(parameters)
state = {
'cytoplasm': {
'GLC': 1.0,
'G6P': 0.0,
'HK': 0.1,
},
'nucleoside_phosphates': {
'ATP': 2.0,
'ADP': 0.0,
},
}
update = my_process.next_update(3.0, state)
print(update['cytoplasm']['G6P'])
With these parameters, we can calculate the reaction rate:
Therefore, we expect the change in concentration of G6P to be:
Let’s see if our process models this reaction as we expect:
$ python glucose_phosphorylation.py
0.4221388367729832
Hooray! This is what we expected.
Ports Schema¶
Our process works, but we had to manually set the state. We also haven’t
shown yet how to apply the update we generate to the simulation state.
Luckily for us, these steps will be handled automatically by Vivarium.
We just need to create a ports_schema method that provides a
schema. A schema is a nested dictionary that describes each
variable the process will interact with. Each variable is defined by a
dictionary of schema keys that specify its default value, how it
should be updated, and other properties.
For this example, our updates are expressed as deltas that should be added to the old value of the variable. This is the default, so the schema can leave out the updater specification. Still, we’ll specify one of the updaters for demonstration.
def ports_schema(self):
return {
'cytoplasm': {
'GLC': {
# accumulate means to add the updates
'_updater': 'accumulate',
'_default': 1.0,
'_properties': {
'mw': 1.0 * units.g / units.mol,
},
'_emit': True,
},
# accumulate is the default, so we don't need to specify
# updaters for the rest of the variables
'G6P': {
'_default': 0.0,
'_properties': {
'mw': 1.0 * units.g / units.mol,
},
'_emit': True,
},
'HK': {
'_default': 0.1,
'_properties': {
'mw': 1.0 * units.g / units.mol,
},
},
},
'nucleoside_phosphates': {
'ATP': {
'_default': 2.0,
'_emit': True,
},
'ADP': {
'_default': 0.0,
'_emit': True,
}
},
}
By convention, we usually put the ports_schema method before the
next_update method.
Now, we can run a simulation using Vivarium’s Engine
vivarium.core.engine.Engine()
like this:
from vivarium.core.engine import Engine
from vivarium.plots.simulation_output import plot_simulation_output
...
if __name__ == '__main__':
parameters = {
'k_cat': 1.5,
}
my_process = GlucosePhosphorylation(parameters)
# make a composite
my_composite = my_process.generate()
# run a simulation
sim = Engine(composite=my_composite)
total_time = 10
sim.update(total_time)
# get the timeseries
timeseries = sim.emitter.get_timeseries()
plot_simulation_output(timeseries, {}, './')
We use
vivarium.plots.simulation_output.plot_simulation_output to
plot the output from our simulation. In simulation.png you should
see an output plot like this:
Tip
If a process is erroneously reporting negative values, try decreasing the timestep.
Oops, it looks like the cytoplasmic GLC concentration dropped below zero
around time 8! This happens when the timestep is too long and so our
approximation doesn’t adjust fast enough to dropping concentrations. To
fix this, let’s change the timestep to 0.1.
Note
You may be wondering, “What unit is the timestep in?” The answer is that it doesn’t matter! We just need the parameters and timestep to use the same unit of time.
Here’s the parameters dictionary with the updated timestep:
parameters = {
'k_cat': 1.5,
'time_step': 0.1,
}
Now if we run the file again, we should get a simulation.png like
this:
You can download the completed process file here.
Great job; you’ve written a new process! Now consider writing one to model a mechanism you are familiar with.
Reference Material¶
This reference material is for users who need to look up a specific piece of information. If you want to understand a topic more comprehensively, try a topic guide instead. If you are new to Vivarium, check out our getting started guide.
The API Reference is a representation of the docstrings from Vivarium’s code. This is a great place to look up details on how to use one of our classes or functions.
In our glossary we explain how we use terms specific to Vivarium.
Vivarium¶
Core Vivarium Package¶
Experiment Control¶
Run experiments and analyses from the command line.
-
class
vivarium.core.control.Control(out_dir: Optional[str] = None, experiments: Optional[Dict[str, Any]] = None, composers: Optional[Dict[str, Any]] = None, plots: Optional[Dict[str, Any]] = None, workflows: Optional[Dict[str, Any]] = None, args: Optional[Sequence[str]] = None)[source]¶ Bases:
objectControl experiments from the command line
Load experiments, plots, and workflows in this Control class, and trigger them from the command line
-
parse_args(args: Optional[Sequence[str]] = None) → argparse.Namespace[source]¶
-
run_one_plot(plot_config: Union[str, dict], data: Dict[Union[Tuple, str], Any], out_dir: Optional[str] = None) → None[source]¶
-
-
vivarium.core.control.run_library_cli(library: dict, args: Optional[list] = None) → None[source]¶ Run experiments from the command line
- Parameters
library (dict) – maps experiment id to experiment function
-
vivarium.core.control.toy_control(args: Optional[Sequence[str]] = None) → vivarium.core.control.Control[source]¶ a toy example of control
To run: > python vivarium/core/control.py -w 1
Emitters¶
Emitters log configuration data and time-series data somewhere.
-
class
vivarium.core.emitter.DatabaseEmitter(config: Dict[str, Any])[source]¶ Bases:
vivarium.core.emitter.EmitterEmit data to a mongoDB database
Example:
>>> config = { ... 'host': 'localhost:27017', ... 'database': 'DB_NAME', ... } >>> # The line below works only if you have to have 27017 open locally >>> # emitter = DatabaseEmitter(config)
config may have ‘host’ and ‘database’ items.
-
client_dict: Dict[int, pymongo.mongo_client.MongoClient] = {}¶
-
classmethod
create_indexes(table: Any, columns: List[Any]) → None[source]¶ Create the listed column indexes for the given DB table.
-
default_host= 'localhost:27017'¶
-
-
class
vivarium.core.emitter.Emitter(config: Dict[str, str])[source]¶ Bases:
objectBase class for emitters.
This emitter simply emits to STDOUT.
- Parameters
config – Emitter configuration.
-
emit(data: Dict[str, Any]) → None[source]¶ Emit data.
- Parameters
data – The data to emit. This gets called by the Vivarium engine with a snapshot of the simulation state.
-
get_data(query: Optional[list] = None) → dict[source]¶ Get the emitted data.
- Returns
The data that has been emitted to the database in the raw data format. For this particular class, an empty dictionary is returned.
-
get_data_deserialized(query: Optional[list] = None) → Any[source]¶ Get the emitted data with variable values deserialized.
- Returns
The data that has been emitted to the database in the raw data format. Before being returned, serialized values in the data are deserialized.
-
get_data_unitless(query: Optional[list] = None) → Any[source]¶ Get the emitted data with units stripped from variable values.
- Returns
The data that has been emitted to the database in the raw data format. Before being returned, units are stripped from values.
-
get_path_timeseries(query: Optional[list] = None) → dict[source]¶ Get the deserialized data as a path timeseries.
- Returns
The deserialized emitted data, formatted as a path timeseries.
-
get_timeseries(query: Optional[list] = None) → dict[source]¶ Get the deserialized data as an embedded timeseries.
- Returns
The deserialized emitted data, formatted as an embedded timeseries.
-
class
vivarium.core.emitter.NullEmitter(config: Dict[str, str])[source]¶ Bases:
vivarium.core.emitter.EmitterDon’t emit anything
Base class for emitters.
This emitter simply emits to STDOUT.
- Parameters
config – Emitter configuration.
-
class
vivarium.core.emitter.RAMEmitter(config: Dict[str, Any])[source]¶ Bases:
vivarium.core.emitter.EmitterAccumulate the timeseries history portion of the “emitted” data to a table in RAM.
Bases:
vivarium.core.emitter.RAMEmitterAccumulate the timeseries history portion of the “emitted” data to a table in RAM that is shared across all instances of the emitter.
-
vivarium.core.emitter.apply_func(document: Any, field: Tuple, f: Optional[Callable[[…], Any]] = None) → Any[source]¶
-
vivarium.core.emitter.breakdown_data(limit: float, data: Any, path: Tuple = (), size: Optional[float] = None) → list[source]¶
-
vivarium.core.emitter.data_from_database(experiment_id: str, client: Any, query: Optional[list] = None, func_dict: Optional[dict] = None, f: Optional[Callable[[…], Any]] = None, filters: Optional[dict] = None, start_time: Union[int, bson.min_key.MinKey] = MinKey(), end_time: Union[int, bson.max_key.MaxKey] = MaxKey(), cpus: int = 1) → Tuple[dict, Any][source]¶ Fetch something from a MongoDB.
- Parameters
experiment_id – the experiment id which is being retrieved
client – a MongoClient instance connected to the DB
query – a list of tuples pointing to fields within the experiment data. In the format: [(‘path’, ‘to’, ‘field1’), (‘path’, ‘to’, ‘field2’)]
func_dict – a dict which maps the given query paths to a function that operates on the retrieved values and returns the results. If None then the raw values are returned. In the format: {(‘path’, ‘to’, ‘field1’): function}
f – a function that applies equally to all fields in query. func_dict is the recommended approach and takes priority over f.
filters – MongoDB query arguments to further filter results beyond matching the experiment ID.
start_time – first and last simulation time to query
end_time – first and last simulation time to query
cpus – splits query into this many chunks to run in parallel
- Returns
data (dict)
-
vivarium.core.emitter.data_to_database(data: Dict[float, dict], environment_config: Any, client: Any) → Any[source]¶ Insert something into a MongoDB.
-
vivarium.core.emitter.delete_experiment(host: str = 'localhost', port: Any = 27017, query: Optional[dict] = None) → None[source]¶ Helper function to delete experiment data in parallel
- Parameters
host – Host name of database. This can usually be left as the default.
port – Port number of database. This can usually be left as the default.
query – Filter for documents to delete.
-
vivarium.core.emitter.delete_experiment_from_database(experiment_id: str, host: str = 'localhost', port: Any = 27017, cpus: int = 1) → None[source]¶ Delete an experiment’s data from a database.
- Parameters
experiment_id – Identifier of experiment.
host – Host name of database. This can usually be left as the default.
port – Port number of database. This can usually be left as the default.
cpus – Number of chunks to split delete operation into to be run in parallel. Useful if single-threaded delete does not saturate I/O.
-
vivarium.core.emitter.get_atlas_client(secrets_path: str) → Any[source]¶ Open a MongoDB client using the named secrets config JSON file.
-
vivarium.core.emitter.get_atlas_database_emitter_config(username: str, password: str, cluster_subdomain: Any, database: str) → Dict[str, Any][source]¶ Construct an Emitter config for a MongoDB on the Atlas service.
-
vivarium.core.emitter.get_data_chunks(history_collection: Any, experiment_id: str, start_time: Union[int, bson.min_key.MinKey] = MinKey(), end_time: Union[int, bson.max_key.MaxKey] = MaxKey(), cpus: int = 8) → list[source]¶ Helper function to get chunks for parallel queries
- Parameters
history_collection – the MongoDB history collection to query
experiment_id – the experiment id which is being retrieved
start_time – first and last simulation time to query
end_time – first and last simulation time to query
cpus – number of chunks to create
- Returns
List of ObjectId tuples that represent chunk boundaries. For each tuple, include
{'_id': {$gte: tuple[0], $lt: tuple[1]}}in the query to search its corresponding chunk.
-
vivarium.core.emitter.get_emitter(config: Optional[Dict[str, str]]) → vivarium.core.emitter.Emitter[source]¶ Construct an Emitter using the provided config.
The available Emitter type names and their classes are:
database:DatabaseEmitternull:NullEmitterprint:Emitter, prints to stdouttimeseries:RAMEmitter
- Parameters
config – Must comtain the
typekey, which specifies the emitter type name (e.g.database).- Returns
A new Emitter instance.
-
vivarium.core.emitter.get_experiment_database(port: Any = 27017, database_name: str = 'simulations') → Any[source]¶ Get a database object.
- Parameters
port – Port number of database. This can usually be left as the default.
database_name – Name of the database table. This can usually be left as the default.
- Returns
The database object.
-
vivarium.core.emitter.get_history_data_db(history_collection: Any, experiment_id: Any, query: Optional[list] = None, func_dict: Optional[dict] = None, f: Optional[Callable[[…], Any]] = None, filters: Optional[dict] = None, start_time: Union[int, bson.min_key.MinKey] = MinKey(), end_time: Union[int, bson.max_key.MaxKey] = MaxKey(), cpus: int = 1, host: str = 'localhost', port: Any = '27017') → Dict[float, dict][source]¶ Query MongoDB for history data.
- Parameters
history_collection – a MongoDB collection
experiment_id – the experiment id which is being retrieved
query – a list of tuples pointing to fields within the experiment data. In the format: [(‘path’, ‘to’, ‘field1’), (‘path’, ‘to’, ‘field2’)]
func_dict – a dict which maps the given query paths to a function that operates on the retrieved values and returns the results. If None then the raw values are returned. In the format: {(‘path’, ‘to’, ‘field1’): function}
f – a function that applies equally to all fields in query. func_dict is the recommended approach and takes priority over f.
filters – MongoDB query arguments to further filter results beyond matching the experiment ID.
start_time – first and last simulation time to query
end_time – first and last simulation time to query
cpus – splits query into this many chunks to run in parallel, useful if single-threaded query does not saturate I/O (e.g. on Google Cloud)
host – used if cpus>1 to create MongoClient in parallel processes
port – used if cpus>1 to create MongoClient in parallel processes
- Returns
data (dict)
-
vivarium.core.emitter.get_local_client(host: str, port: Any, database_name: str) → Any[source]¶ Open a MongoDB client onto the given host, port, and DB.
-
vivarium.core.emitter.get_query(projection: dict, host: str, port: Any, query: dict) → list[source]¶ Helper function for parallel queries
- Parameters
projection – a MongoDB projection in dictionary form
host – used to create new MongoClient for each parallel process
port – used to create new MongoClient for each parallel process
query – a MongoDB query in dictionary form
- Returns
List of projected documents for given query
-
vivarium.core.emitter.path_timeseries_from_data(data: dict) → dict[source]¶ Convert from raw data to a path timeseries.
-
vivarium.core.emitter.path_timeseries_from_embedded_timeseries(embedded_timeseries: dict) → dict[source]¶ Convert an embedded timeseries to a path timeseries.
Engine¶
Engine runs the simulation.
-
class
vivarium.core.engine.Defer(defer: Any, f: Callable, args: Tuple)[source]¶ Bases:
objectAllows for delayed application of a function to an update.
The object simply holds the provided arguments until it’s time for the computation to be performed. Then, the function is called.
- Parameters
defer – An object with a
.get_command_result()method whose output will be passed to the function. For example, the object could be anvivarium.core.process.Processobject whose.get_command_result()method will return the process update.function – The function. For example,
invert_topology()to transform the returned update.args – Passed as the second argument to the function.
-
class
vivarium.core.engine.EmptyDefer[source]¶ Bases:
vivarium.core.engine.Defer
-
class
vivarium.core.engine.Engine(composite: Optional[vivarium.core.composer.Composite] = None, processes: Optional[Dict[str, Any]] = None, steps: Optional[Dict[str, Any]] = None, flow: Optional[Dict[str, Sequence[Tuple[str, …]]]] = None, topology: Optional[Dict[str, Union[Tuple[str, …], dict, object]]] = None, store: Optional[vivarium.core.store.Store] = None, initial_state: Optional[Dict[str, Any]] = None, experiment_id: Optional[str] = None, experiment_name: Optional[str] = None, metadata: Optional[dict] = None, description: str = '', emitter: Union[str, dict] = 'timeseries', store_schema: Optional[dict] = None, emit_topology: bool = True, emit_processes: bool = False, emit_config: bool = False, emit_step: float = 1, display_info: bool = True, progress_bar: bool = False, global_time_precision: Optional[int] = None, profile: bool = False, initial_global_time: float = 0)[source]¶ Bases:
objectDefines simulations
- Parameters
composite – A Composite, which specifies the processes, steps, flow, and topology. This is an alternative to passing in processes and topology dict, which can not be loaded at the same time.
processes – A dictionary that maps process names to process objects. You will usually get this from the
processeskey of the dictionary fromvivarium.core.composer.Composer.generate().steps – A dictionary that maps step names to step objects. You will usually get this from the
stepskey of the dictionary fromvivarium.core.composer.Composer.generate().flow – A dictionary that maps step names to sequences of paths to the steps that the step depends on. You will usually get this from the
flowkey of the dictionary fromvivarium.core.composer.Composer.generate().topology – A dictionary that maps process names to sub-dictionaries. These sub-dictionaries map the process’s port names to tuples that specify a path through the tree from the compartment root to the store that will be passed to the process for that port.
store – A pre-loaded Store. This is an alternative to passing in processes and topology dict, which can not be loaded at the same time. Note that if you provide this argument, you must ensure that all parallel processes (i.e.
vivarium.core.process.Processobjects with theparallelattribute set toTrue) are instances ofvivarium.core.process.ParallelProcess. This constructor converts parallel processes toParallelProcessobjects automatically if you do not provide thisstoreargument.initial_state – By default an empty dictionary, this is the initial state of the simulation.
experiment_id – A unique identifier for the experiment. A UUID will be generated if none is provided.
metadata – A dictionary with additional data about the experiment, which is saved by the emitter with the configuration.
description – A description of the experiment. A blank string by default.
emitter – An emitter configuration which must conform to the specification in the documentation for
vivarium.core.emitter.get_emitter(). The experiment ID will be added to the dictionary you provide as the value for the keyexperiment_id.display_info – prints experiment info
progress_bar – shows a progress bar
global_time_precision – an optional int that sets the decimal precision of global_time. This is useful for remove floating- point rounding errors for the time keys of saved states.
store_schema – An optional dictionary to expand the store hierarchy configuration, and also to turn emits on or off. The dictionary needs to be structured as a hierarchy, which will expand the existing store hierarchy. Setting an emit value for a branch node will set the emits of all the leaves to that value.
emit_topology – If True, this will emit the topology with the configuration data.
emit_processes – If True, this will emit the serialized processes with the configuration data.
emit_config – If True, this will emit the serialized initial state with the configuration data.
profile – Whether to profile the simulation with cProfile.
initial_global_time – The initial time for the simulation. Useful when this Engine is part of a larger, older simulation.
-
apply_update(update: Dict[str, Any], state: vivarium.core.store.Store) → bool[source]¶ Apply an update to the simulation state.
- Parameters
update – The update to apply. Must be relative to
state.state – The store to which the update is relative (usually root of simulation state. We need this so to preserve the “perspective” from which the update was generated.
- Returns
a bool indicating whether the topology_views expired.
-
end() → None[source]¶ Terminate all processes running in parallel.
This MUST be called at the end of any simulation with parallel processes. This function also ends profiling and computes profiling stats, including stats from parallel sub-processes. These stats are stored in
self.stats.
-
run_for(interval: float, force_complete: bool = False) → None[source]¶ Run each process within the given interval and update their states.
run_for()gives the caller more control over the simulation loop thanupdate(). In particular, it may be called repeatedly within a caller-managed simulation loop without forcing processes to complete after each call (as would be the case withupdate()). It is the responsibility of the caller to ensure that whenrun_for()is called for the last time in the caller-managed simulation loop, force_complete=True so that all the processes finish at the end of the simulation.- Parameters
interval – the amount of time to simulate the composite.
force_complete – a bool indicating whether to force processes to complete at the end of the interval.
-
vivarium.core.engine.invert_topology(update: Dict[str, Any], args: Tuple[Tuple[str, …], Dict[str, Union[Tuple[str, …], dict, object]]]) → Dict[str, Any][source]¶ Wrapper function around
inverse_topology.Wraps
vivarium.library.topology.inverse_topology().Updates are produced relative to the process that produced them. To transform them such that they are relative to the root of the simulation hierarchy, this function “inverts” a topology.
- Parameters
update – The update.
args – Tuple of the path to which the update is relative and the topology.
- Returns
The update, relative to the root of
path.
-
vivarium.core.engine.print_progress_bar(iteration: float, total: float, decimals: float = 1, length: int = 50) → None[source]¶ Create terminal progress bar
- Parameters
iteration – Current iteration
total – Total iterations
decimals – Positive number of decimals in percent complete
length – Character length of bar
Process Classes¶
-
vivarium.core.process.Deriver¶ Deriver is just an alias for
Stepnow that Derivers have been deprecated.alias of
vivarium.core.process.Step
-
class
vivarium.core.process.ParallelProcess(process: vivarium.core.process.Process, profile: bool = False, stats_objs: Optional[List[pstats.Stats]] = None)[source]¶ Bases:
vivarium.core.process.ProcessWraps a
Processfor multiprocessing.To run a simulation distributed across multiple processors, we use Python’s multiprocessing tools. This object runs in the main process and manages communication between the main (parent) process and the child process with the
Processthat this object manages.Most methods pass their name and arguments to
Process.run_command.- Parameters
process – The Process to manage.
profile – Whether to use cProfile to profile the subprocess.
stats_objs – List to add cProfile stats objs to when process is deleted. Only used if
profileis true.
-
property
condition_path¶
-
end() → None[source]¶ End the child process.
If profiling was enabled, then when the child process ends, it will compile its profiling stats and send those to the parent. The parent then saves those stats in
self.stats.
-
generate_topology(config: Optional[dict] = None) → Dict[str, Union[Tuple[str, …], dict, object]][source]¶
-
get_command_result() → Dict[str, Any][source]¶ Get the result of a command sent to the parallel process.
Commands and their results work like a queue, so unlike
Process, you can technically call this method multiple times and get different return values each time. This behavior is subject to change, so you should not rely on it.- Returns
The command result.
-
property
parameters¶
-
property
schema¶
-
property
schema_override¶
-
class
vivarium.core.process.Process(parameters: Optional[dict] = None)[source]¶ Bases:
objectProcess parent class.
All process classes must inherit from this class. Each class can provide a
defaultsclass variable to specify the process defaults as a dictionary.Note that subclasses should call the superclass init function first. This allows the superclass to correctly save the initial parameters before they are mutated by subclass constructor code. We need access to the original parameters for serialization to work properly.
- Parameters
parameters –
Override the class defaults. This dictionary may also contain the following special keys:
name: Saved toself.name._schema: Overrides the schema._parallel: Indicates that the process should be parallelized.self.parallelwill be set to True._condition: Path to a variable whose value will be returned byvivarium.core.process.Process.update_condition().time_step: Returned byvivarium.core.process.Process.calculate_timestep().
-
ATTRIBUTE_READ_COMMANDS= ('schema_override', 'parameters', 'condition_path', 'schema')¶
-
ATTRIBUTE_WRITE_COMMANDS= ('set_schema',)¶
-
METHOD_COMMANDS= ('initial_state', 'generate_processes', 'generate_steps', 'generate_topology', 'generate_flow', 'merge_overrides', 'calculate_timestep', 'is_step', 'get_private_state', 'ports_schema', 'next_update', 'update_condition')¶
-
calculate_timestep(states: Optional[Dict[str, Any]]) → Union[float, int][source]¶ Return the next process time step
A process subclass may override this method to implement adaptive timesteps. By default it returns self.parameters[‘timestep’].
-
property
condition_path¶
-
default_state() → Dict[str, Any][source]¶ Get the default values of the variables in each port.
The default values are computed based on the schema.
- Returns
A state dictionary that assigns each variable’s default value to that variable.
-
generate_processes(config: Optional[dict] = None) → Dict[str, Any][source]¶ Do not override this method.
-
generate_steps(config: Optional[dict] = None) → Dict[str, Any][source]¶ Do not override this method.
-
generate_topology(config: Optional[dict] = None) → Dict[str, Union[Tuple[str, …], dict, object]][source]¶ Do not override this method.
-
get_command_result() → Any[source]¶ Retrieve the result from the last-run command.
- Returns
The result of the last command run. Note that this method should only be called once immediately after each call to
send_command().- Raises
RuntimeError – When there is no command pending. This can happen when this method is called twice without an intervening call to
send_command().
-
get_private_state() → Dict[str, Any][source]¶ Get the process’s private state.
Processes can store state in instance variables instead of in the stores that hold the simulation-wide state. These instance variables hold a private state that is not shared with other processes. You can override this function to let experiments emit the process’s private state.
- Returns
An empty dictionary. You may override this behavior to return your process’s private state.
-
get_schema(override: Optional[Dict[str, Any]] = None) → dict[source]¶ Get the process’s schema, optionally with a schema override.
- Parameters
override – Override schema
- Returns
The combined schema.
-
initial_state(config: Optional[dict] = None) → Dict[str, Any][source]¶ Get initial state in embedded path dictionary.
Every subclass may override this method.
-
is_deriver() → bool[source]¶ Check whether this process is a deriver.
Deprecated since version 0.3.14: Derivers have been deprecated in favor of steps, so please override
is_stepinstead ofis_deriver. Support for Derivers may be removed in a future release.- Returns
Whether this process is a deriver. This class always returns
False, but subclasses may change this.
-
is_step() → bool[source]¶ Check whether this process is a step.
- Returns
Whether this process is a step. This class always returns
False, but subclasses may change this behavior.
-
merge_overrides(override: Dict[str, Any]) → None[source]¶ Add a schema override to the process’s schema overrides.
- Parameters
override – The schema override to add.
-
abstract
next_update(timestep: Union[float, int], states: Dict[str, Any]) → Dict[str, Any][source]¶ Compute the next update to the simulation state.
- Parameters
timestep – The duration for which the update should be computed.
states – The pre-update simulation state. This will take the same form as the process’s schema, except with a value for each variable.
- Returns
An empty dictionary for now. This should be overridden by each subclass to return an update.
-
property
parallel¶
-
property
parameters¶
-
ports() → Dict[str, List[str]][source]¶ Get ports and each port’s variables.
- Returns
A map from port names to lists of the variables that go into that port.
-
abstract
ports_schema() → Dict[str, Any][source]¶ Get the schemas for each port.
This must be overridden by any subclasses.
Use the glob ‘*’ schema to declare expected sub-store structure, and view all child values of the store:
schema = { 'port1': { '*': { '_default': 1.0 } } }
Use the glob ‘**’ schema to connect to an entire sub-branch, including child nodes, grandchild nodes, etc:
schema = { 'port1': '**' }
Ports flagged as output-only won’t be viewed through the next_update’s states, which can save some overhead time:
schema = { 'port1': { '_output': True, 'A': {'_default': 1.0}, } }
- Returns
A dictionary that declares which states are expected by the processes, and how each state will behave. State keys can be assigned properties through schema_keys declared in
vivarium.core.store.Store.
-
pre_send_command(command: str, args: Optional[tuple], kwargs: Optional[dict]) → None[source]¶ Run pre-checks before starting a command.
This method should be called at the start of every implementation of
send_command().- Parameters
command – The name of the command to run.
args – A tuple of positional arguments for the command.
kwargs – A dictionary of keyword arguments for the command.
- Raises
RuntimeError – Raised when a user tries to send a command while a previous command is still pending (i.e. the user hasn’t called
get_command_result()yet for the previous command).
-
run_command(command: str, args: Optional[tuple] = None, kwargs: Optional[dict] = None) → Any[source]¶ Helper function that sends a command and returns result.
-
run_command_method(command: str, args: tuple, kwargs: dict) → Any[source]¶ Run a command whose name and interface match a method.
- Parameters
command – The command name, which must equal to a method of
self.args – The positional arguments to pass to the method.
kwargs – The keywords arguments for the method.
- Returns
The result of calling
self.command(*args, **kwargs)is returned for commandcommand.
-
property
schema¶
-
property
schema_override¶
-
send_command(command: str, args: Optional[tuple] = None, kwargs: Optional[dict] = None, run_pre_check: bool = True) → None[source]¶ Handle process commands.
This method handles the commands listed in
METHOD_COMMANDSby passingargsandkwargsto the method ofselfwith the name of the command and saving the return value as the result.This method handles the commands listed in
ATTRIBUTE_READ_COMMANDSby returning the attribute ofselfwith the name matching the command, and it handles the commands listed inATTRIBUTE_WRITE_COMMANDSby setting the attribute in the command to the first argument inargs. The command must be namedset_attrfor attributeattr.To add support for a custom command, override this function in your subclass. Each command is defined by a name (a string) and accepts both positional and keyword arguments. Any custom commands you add should have associated methods such that:
The command name matches the method name.
The command and method accept the same positional and keyword arguments.
The command and method return the same values.
If all of the above are satisfied, you can use
Process.run_command_method()to handle the command.Your implementation of this function needs to handle all the commands you want to support. When presented with an unknown command, you should call the superclass method, which will either handle the command or call its superclass method. At the top of this recursive chain, this
Process.send_command()method handles some built-in commands and will raise an error for unknown commands.Any overrides of this method must also call
pre_send_command()at the start of the method. This call will check that no command is currently pending to avoid confusing behavior when multiple commands are started without intervening retrievals of command results. Since your overriding method will have already performed the pre-check, it should passrun_pre_check=Falsewhen calling the superclass method.- Parameters
command – The name of the command to run.
args – A tuple of positional arguments for the command.
kwargs – A dictionary of keyword arguments for the command.
run_pre_check – Whether to run the pre-checks implemented in
pre_send_command(). This should be left at its default value unless the pre-checks have already been performed (e.g. if this method is being called by a subclass’s overriding method.)
- Returns
None. This method just starts the command running.
- Raises
ValueError – For unknown commands.
-
class
vivarium.core.process.Step(parameters: Optional[dict] = None)[source]¶ Bases:
vivarium.core.process.ProcessBase class for steps.
-
class
vivarium.core.process.ToySerializedProcessInheritance(parameters: Optional[dict] = None)[source]¶
-
vivarium.core.process._handle_parallel_process(connection: multiprocessing.connection.Connection, process: vivarium.core.process.Process, profile: bool) → None[source]¶ Handle a parallel Vivarium process.
This function is designed to be passed as
targettoMultiprocess(). In a loop, it receives process commands from a pipe, passes those commands to the parallel process, and passes the result back along the pipe.The special command
endis handled directly by this function. This command causes the function to exit and therefore shut down the OS process created by multiprocessing.- Parameters
connection – The child end of a multiprocessing pipe. All communications received from the pipe should be a 3-tuple of the form
(command, args, kwargs), and the tuple contents will be passed toProcess.run_command(). The result, which may be of any type, will be sent back through the pipe.process – The process running in parallel.
profile – Whether to profile the process.
-
vivarium.core.process.assoc_in(d: dict, path: Tuple[str, …], value: Any) → dict[source]¶ Insert a value into a dictionary at an arbitrary depth.
Empty dictionaries will be created as needed to insert the value at the specified depth.
>>> d = {'a': {'b': 1}} >>> assoc_in(d, ('a', 'c', 'd'), 2) {'a': {'b': 1, 'c': {'d': 2}}}
- Parameters
d – Dictionary to insert into.
path – Path in the dictionary where the value will be inserted. Each element of the path is dictionary key, which will be added if not already present. Any given element (except the first) refers to a key in the dictionary that is the value associated with the immediately preceding path element.
value – The value to insert.
- Returns
Dictionary with the value inserted.
Composite, Composer, and MetaComposer Classes¶
-
class
vivarium.core.composer.Composer(config: Optional[dict] = None)[source]¶ Bases:
objectBase class for composer classes.
Composers generate composites.
All composer classes must inherit from this class.
- Parameters
config – Dictionary of configuration options that can override the class defaults.
-
generate(config: Optional[dict] = None, path: Tuple[str, …] = ()) → vivarium.core.composer.Composite[source]¶ Generate processes and topology dictionaries.
- Parameters
config – Updates values in the configuration declared in the constructor.
path – Tuple with (‘path’, ‘to’, ‘level’) associates the processes and topology at this level.
- Returns
Dictionary with the following keys
processes: Generated bygenerate_processes().steps: Generated bygenerate_steps().flow: Generated bygenerate_flow().topology: Generated bygenerate_topology().
The values of these keys are all dictionaries suitable to be passed to the constructor for
vivarium.core.engine.Engine.
-
generate_flow(config: Optional[dict]) → Dict[str, Sequence[Tuple[str, …]]][source]¶ Generate the flow of step dependencies.
- Parameters
config – A dictionary of configuration options. All subclass implementation must accept this parameter, but some may ignore it.
- Returns
Subclass implementations should return a dictionary mapping step names to sequences (e.g. lists or tuples) of paths. Steps with no dependencies must be included, but they should be mapped to an empty sequence. Any steps returned by
generate_steps()orgenerate_processes()that are not included in the flow will be treated as if they depend on every step previously added to the engine.
-
abstract
generate_processes(config: Optional[dict]) → Dict[str, Any][source]¶ Generate processes dictionary.
Every subclass must override this method. For backwards compatibility,
vivarium.core.process.Stepobjects may be included in the returned dictionary, but this practice is discouraged and may be disallowed in a future release.- Parameters
config – A dictionary of configuration options. All subclass implementation must accept this parameter, but some may ignore it.
- Returns
Subclass implementations must return a dictionary mapping process names to instantiated and configured
vivarium.core.process.Processobjects.
-
generate_steps(config: Optional[dict]) → Dict[str, Any][source]¶ Generate the steps dictionary.
Subclasses that want to include steps should override this method. This method is the preferred way to specify steps, though they may also be returned by
generate_processes().- Parameters
config – A dictionary of configuration options. All subclass implementation must accept this parameter, but some may ignore it.
- Returns
Subclass implementations should return a dictionary mapping step names to instantiated and configured
vivarium.core.process.Stepobjects.
-
generate_store(config: Optional[dict] = None) → vivarium.core.store.Store[source]¶
-
abstract
generate_topology(config: Optional[dict]) → Dict[str, Union[Tuple[str, …], dict, object]][source]¶ Generate topology dictionary.
Every subclass must override this method.
- Parameters
config – A dictionary of configuration options. All subclass implementation must accept this parameter, but some may ignore it.
- Returns
Subclass implementations must return a topology dictionary.
-
get_parameters() → dict[source]¶ Get the parameters for all processes.
- Returns
A map from process names to dictionaries of those processes’ parameters.
-
class
vivarium.core.composer.Composite(config: Optional[Dict[str, Any]] = None, store: Optional[vivarium.core.store.Store] = None, processes: Optional[Dict[str, Any]] = None, steps: Optional[Dict[str, Any]] = None, flow: Optional[Dict[str, Sequence[Tuple[str, …]]]] = None, topology: Optional[Dict[str, Union[Tuple[str, …], dict, object]]] = None, state: Optional[Dict[str, Any]] = None)[source]¶ Bases:
vivarium.library.datum.DatumComposite parent class.
Contains keys for processes and topology
-
default_state(config: Optional[dict] = None) → Optional[Dict[str, Any]][source]¶ Merge all processes’ default states :param config: A dictionary of configuration options. All :type config: dict :param subclass implementation must accept this parameter: :param but: :param some may ignore it.:
- Returns
Subclass implementations must return a dictionary mapping state paths to default values.
- Return type
(dict)
-
generate_store(config: Optional[dict] = None) → vivarium.core.store.Store[source]¶
-
get_parameters() → Dict[source]¶ Get the parameters for all processes. :returns: A map from process names to parameters.
-
initial_state(config: Optional[dict] = None) → Optional[Dict[str, Any]][source]¶ Merge all processes’ initial states :param config: A dictionary of configuration options. All :type config: dict :param subclass implementation must accept this parameter: :param but: :param some may ignore it.:
- Returns
Subclass implementations must return a dictionary mapping state paths to initial values.
- Return type
(dict)
-
merge(composite: Optional[vivarium.core.composer.Composite] = None, processes: Optional[Dict[str, vivarium.core.process.Process]] = None, topology: Optional[Dict[str, Union[Tuple[str, …], dict, object]]] = None, steps: Optional[Dict[str, Any]] = None, flow: Optional[Dict[str, Sequence[Tuple[str, …]]]] = None, state: Optional[Dict[str, Any]] = None, path: Optional[Tuple[str, …]] = None, schema_override: Optional[Dict[str, Any]] = None) → None[source]¶
-
-
class
vivarium.core.composer.MetaComposer(composers: Iterable[Any] = (), config: Optional[dict] = None)[source]¶ Bases:
vivarium.core.composer.ComposerA collection of
Composerobjects.The MetaComposer can be used to create composites that combine all the composers in the collection.
- Parameters
composers – Initial collection of composers.
config – Initial configuration.
-
add_composer(composer: vivarium.core.composer.Composer, config: Optional[Dict] = None) → None[source]¶ Add a composer to the collection of stored composers.
- Parameters
composer – The composer to add.
config – The composer’s configuration, which will be merged with the stored config.
-
vivarium.core.composer.get_composite_from_store(store: vivarium.core.store.Store) → vivarium.core.composer.Composite[source]¶
Registry of Updaters, Dividers, and Serializers¶
You should interpret words and phrases that appear fully capitalized in this document as described in RFC 2119. Here is a brief summary of the RFC:
“MUST” indicates absolute requirements. Vivarium may not work correctly if you don’t follow these.
“SHOULD” indicates strong suggestions. You might have a valid reason for deviating from them, but be careful that you understand the ramifications.
“MAY” indicates truly optional features that you can include or exclude as you wish.
Updaters¶
Each updater is defined as a function whose name begins with
update_. Vivarium uses these functions to apply updates to
variables. Updater names are registered in
updater_registry, which maps these names to updater functions.
An updater function SHOULD have a name that begins with update_. The
function MUST accept exactly three positional arguments: the first MUST
be the current value of the variable (i.e. before applying the update),
the second MUST be the value associated with the variable in the update,
and the third MUST be either a dictionary of states from the simulation
hierarchy or None if no port_mapping key was specified in the
updater definition. The function SHOULD not accept any other parameters.
The function MUST return the updated value of the variable only.
Dividers¶
Each divider is defined by a function that follows the API we
describe below. Vivarium uses these dividers to generate daughter cell
states from the mother cell’s state. Divider names are registered in
divider_registry, which maps these names to divider functions.
Each divider function SHOULD have a name prefixed with divide_. The
function MUST accept a single positional argument, the value of the
variable in the mother cell. It SHOULD accept no other arguments. The
function MUST return either:
A
listwith two elements: the values of the variables in each of the daughter cells.None, in which case division will be skipped for that variable.
Note
Dividers MAY not be deterministic and MAY not be symmetric. For example, a divider splitting an odd, integer-valued value may randomly decide which daughter cell receives the remainder.
Serializers¶
Each serializer is defined as a class that follows the API we
describe below. Vivarium uses these serializers to convert emitted data
into a BSON-compatible format for database storage. Serializer names are
registered in serializer_registry, which maps these names to
serializer subclasses. For maximum performance, register serializers
using a key equal to the string representation of its designated type
(e.g. str(Serializer.python_type)).
Serializers MUST define the following:
The
python_typeclass attributes that determines what types are handled by the serializerThe
vivarium.core.registry.Serializer.serialize()method which is called on all objects of typepython_type
Avoid defining custom serializers for built-in or Numpy types as these are
automatically handled by orjson, the package used to serialize data.
Note
All dictionary keys MUST be Python strings for orjson to work.
Numpy strings (np.str_) are not allowed.
If it is necessary to redefine the how objects are serialized by orjson,
assign custom serializers to the stores containing objects of the affected
type(s) using the _serializer ports schema key. This can also be used
to serialize objects of the same Python type differently. To ensure that
objects serialized this way are deserialized correctly, you SHOULD consider
implementing the following as well:
vivarium.core.registry.Serializer.can_deserialize()to determine whether to calldeserializeon datavivarium.core.registry.Serializer.deserialize()to deserialize data
If it is necessary to deserialize objects of the same BSON type differently, the corresponding serializer(s) MUST implement these 2 methods.
-
class
vivarium.core.registry.Registry[source]¶ Bases:
objectA Registry holds a collection of functions or objects.
-
register(key, item, alternate_keys=())[source]¶ Add an item to the registry.
- Parameters
key – Item key.
item – The item to add.
alternate_keys –
Additional keys under which to register the item. These keys will not be included in the list returned by
Registry.list().This may be useful if you want to be able to look up an item in the registry under multiple keys.
-
-
class
vivarium.core.registry.Serializer[source]¶ Bases:
objectBase serializer class.
Serializers work together to convert Python objects, which may be collections of many different kinds of objects, into BSON-compatible representations. Those representations can then be deserialized to recover the original object.
Serialization of Python’s built-in datatypes and most Numpy types is handled directly by the
orjson.dumps()method.The serialization routines in Serializers are compiled into a fallback function that is called on objects not handled by
orjson.If one wishes to modify how built-in/Numpy objects are serialized, the relevant stores can be assigned a custom serializer using the
_serializerkey. Thevivarium.core.registry.Serializer.serialize()method of that serializer will be called on the values in these stores before they are passed toorjson.During deserialization, the
can_deserializemethod of each Serializer is used to determine whether to call thedeserializemethod.- Parameters
name – Name of the serializer. Defaults to the class name.
-
can_deserialize(data)[source]¶ This tells
vivarium.core.serialize.deserialize_value()whether to calldeserializeon data.
-
deserialize(data)[source]¶ This allows for data of the same BSON type to be deserialized differently (see regex matching of strings in
vivarium.core.serialize.UnitsSerializer.deserialize()for an example).
-
python_type: Any = None¶ Type matching is NOT exact (subclasses included)
-
vivarium.core.registry.assert_no_divide(state)[source]¶ Assert that the variable is never divided
- Raises
AssertionError – If the variable is divided
-
vivarium.core.registry.divide_null(state)[source]¶ Divider that causes the variable to be skipped during division.
- Returns
Noneso that no divided values are provided to the daughter cells. This is useful for process objects, which are handled separately during division.
-
vivarium.core.registry.divide_set(state)[source]¶ Set Divider
- Returns
A list
[state, state]. No copying is performed.
-
vivarium.core.registry.divide_set_value(state, config)[source]¶ Set Value Divider :param ‘state’: value
- Returns
A list
[value, value]. No copying is performed.
-
vivarium.core.registry.divide_split(state)[source]¶ Split Divider
- Parameters
state – Must be an
int, afloat, or astrof valueInfinity.- Returns
A list, each of whose elements contains half of
state. Ifstateis anint, the remainder is placed at random in one of the two elements. Ifstateis infinite, the return value is[state, state](no copying is done).- Raises
Exception – if
stateis of an unrecognized type.
-
vivarium.core.registry.divide_split_dict(state)[source]¶ Split-Dictionary Divider
- Returns
A list of two dictionaries. The first dictionary stores the first half of the key-value pairs in
state, and the second dictionary stores the rest of the key-value pairs.Note
Since dictionaries are unordered, you should avoid making any assumptions about which keys will be sent to which daughter cell.
-
vivarium.core.registry.divider_registry= <vivarium.core.registry.Registry object>¶ Map divider names to divider functions
-
vivarium.core.registry.emitter_registry= <vivarium.core.registry.Registry object>¶ Map serializer names to Emitter classes
-
vivarium.core.registry.process_registry= <vivarium.core.registry.Registry object>¶ Maps process names to process classes
-
vivarium.core.registry.serializer_registry= <vivarium.core.registry.Registry object>¶ Map serializer names to serializer classes
-
vivarium.core.registry.update_accumulate(current_value, new_value)[source]¶ Accumulate Updater
- Returns
The sum of
current_valueandnew_value.
-
vivarium.core.registry.update_dictionary(current, update)[source]¶ Dictionary Updater Updater that translates _add and _delete -style updates into operations on a dictionary.
Expects current to be a dictionary, with no restriction on the types of objects stored within it, and no defaults values.
-
vivarium.core.registry.update_merge(current_value, new_value)[source]¶ Merge Updater
- Returns
The merger of
current_valueandnew_value. For any shared keys, the value innew_valueis used.- Return type
-
vivarium.core.registry.update_nonnegative_accumulate(current_value, new_value)[source]¶ Non-negative Accumulate Updater
- Returns
The sum of
current_valueandnew_valueif positive, 0 if negative.
-
vivarium.core.registry.update_null(current_value, new_value)[source]¶ Null Updater
- Returns
The value provided in
current_value.
Serialize¶
Collection of serializers that transform Python data into a BSON-compatible form.
-
class
vivarium.core.serialize.DictDeserializer[source]¶ Bases:
vivarium.core.registry.SerializerIterates through dictionaries and applies deserializers.
-
class
vivarium.core.serialize.FunctionSerializer[source]¶ Bases:
vivarium.core.registry.SerializerSerializer for function objects.
-
serialize(data: collections.abc.Callable) → str[source]¶
-
-
class
vivarium.core.serialize.NumpyFallbackSerializer[source]¶ Bases:
vivarium.core.registry.SerializerOrjson does not handle Numpy arrays with strings
-
class
vivarium.core.serialize.ProcessSerializer[source]¶ Bases:
vivarium.core.registry.SerializerSerializer for processes if
emit_processis enabled.-
serialize(data: vivarium.core.process.Process) → str[source]¶
-
-
class
vivarium.core.serialize.QuantitySerializer[source]¶ Bases:
vivarium.core.registry.SerializerSerializes data with units into strings of the form
!units[...], where...is the result of callingstr(data). Deserializes strings of this form back into data with units.
-
class
vivarium.core.serialize.SequenceDeserializer[source]¶ Bases:
vivarium.core.registry.SerializerIterates through lists and applies deserializers.
-
class
vivarium.core.serialize.SetSerializer[source]¶ Bases:
vivarium.core.registry.SerializerSerializer for set objects.
-
class
vivarium.core.serialize.UnitsSerializer[source]¶ Bases:
vivarium.core.registry.SerializerSerializes data with units into strings of the form
!units[...], where...is the result of callingstr(data). Deserializes strings of this form back into data with units.-
deserialize(data: str, unit: Optional[pint.unit.Unit] = None) → pint.quantity.Quantity[source]¶ Deserialize data with units from a human-readable string.
- Parameters
data – The data to deserialize. Providing a list here is deprecated. You should use
deserialize_valueinstead, which uses a separate list deserializer.unit – The units to convert
datato after deserializing. If omitted, no conversion occurs. This option is deprecated.
- Returns
A single deserialized object or, if
datais a list, a list of deserialized objects.
-
-
vivarium.core.serialize.deserialize_value(value: Any) → Any[source]¶ Find and apply the correct serializer for a value by calling each registered serializer’s
vivarium.core.registry.Serializer.can_deserialize()method. Returns the value as is if no compatible serializer is found.- Parameters
value (Any) – Data to be deserialized
- Raises
ValueError – Only one serializer should apply for any given value
- Returns
Deserialized data
- Return type
Any
-
vivarium.core.serialize.find_numpy_and_non_strings(d: dict, curr_path: tuple = (), saved_paths: Optional[List[tuple]] = None) → List[tuple][source]¶ Return list of paths which terminate in a non-string or Numpy string dictionary key. Orjson does not handle these types of keys by default.
-
vivarium.core.serialize.make_fallback_serializer_function() → collections.abc.Callable[source]¶ Creates a fallback function that is called by orjson on data of types that are not natively supported. Define and register instances of
vivarium.core.registry.Serializer()with serialization routines for the types in question.
-
vivarium.core.serialize.serialize_value(value: Any, default: Optional[collections.abc.Callable] = None) → Any[source]¶ Apply orjson-based serialization routine on
value.- Parameters
value (Any) – Data to be serialized. All keys must be strings. Notably, Numpy strings (
np.str_) are not acceptable keys.default (Callable) – A function that is called on any data of a type that is not natively supported by orjson. Returns an object that can be handled by default up to 254 times before an exception is raised.
- Returns
Serialized data
- Return type
Any
Store¶
The file system for storing and updating state variables during an experiment.
-
class
vivarium.core.store.Store(config, outer=None, source=None)[source]¶ Bases:
objectHolds a subset of the overall model state
The total state of the model can be broken down into stores, each of which is represented by an instance of this Store class. The store’s state is a set of variables, each of which is defined by a set of schema key-value pairs. The valid schema keys are listed in
schema_keys, and they are:_default (Type should match the variable value): The default value of the variable.
_updater (
str): The name of the updater to use. By default this isaccumulate._divider (
str): The name of the divider to use. Note that_divideris not included in theschema_keysset because it can be applied to any node in the hierarchy, not just leaves (which represent variables)._value (Type should match the variable value): The current value of the variable. This is
Noneby default._properties (
dict): Extra properties of the variable that don’t have a specific schema key. This is an empty dictionary by default._emit (
bool): Whether to emit the variable to the emitter. This isFalseby default._serializer (
vivarium.core.registry.Serializerorstr): Serializer (or name of serializer) whoseserializemethod should be called on data in this store before emitting and whosedeserializemethod should be called when repopulating this store from serialized data. Only define if it is necessary to serialize and deserialize data in this store differently from other data of the same type.
-
apply_update(update, state=None)[source]¶ Given an arbitrary update, map all the values in that update to their positions in the tree where they apply, and update these values using each node’s _updater.
- Parameters
update – The update being applied.
state – The state at the start of the time step.
There are five topology update methods, which use the following special update keys:
- _add - Adds states into the subtree, given a list of dicts
containing:
path - Path to the added state key.
state - The value of the added state.
_move - Moves a node from a source to a target location in the tree. This uses an update to an outer port, which contains both the source and target node locations. Can move multiple nodes according to a list of dicts containing:
source - the source path from an outer process port
target - the location where the node will be placed.
_generate - The value has four keys, which are essentially the arguments to the generate() function:
path - Path into the tree to generate this subtree.
processes - Tree of processes to generate.
topology - Connections of all the process’s ports_schema().
initial_state - Initial state for this new subtree.
_divide - Performs cell division by constructing two new daughter cells and removing the mother. Takes a dict with two keys:
mother - The id of the mother (for removal)
- daughters - List of two new daughter generate directives, of the
same form as the _generate value above.
_delete - The value here is a list of paths (tuples) to delete from the tree.
Additional special update keys for different update operations:
_updater - Override the default updater with any updater you want.
_reduce - This allows a reduction over the entire subtree from some point downward. Its three keys are:
from - What point to start the reduction.
initial - The initial value of the reduction.
reducer - A function of three arguments, which is called on every node from the from point in the tree down:
value - The current accumulated value of the reduction.
path - The path to this point in the tree
node - The actual node being visited.
This function returns the next value for the reduction. The result of the reduction will be assigned to this point in the tree.
-
connect(path, value, absolute=False)[source]¶ Wire a store’s process to another store.
This function must not be used unless
selfholds a process.- Parameters
path – Path of the port to connect.
value – The store (or the path to the store) to connect to the port at
path.
- Raises
AssertionError – If
self.valueis not an instance ofvivarium.core.process.Process.Exception – If
valueis aStorethat is in a different tree thanself.
-
depth(path=(), filter_function=None)[source]¶ Create a mapping of every path in the tree to the node living at that path in the tree. An optional filter argument is a function that can declares the instances that will be returned, for example: * filter=lambda x: isinstance(x.value, Process)
-
divide_value()[source]¶ Apply the divider for each node to the value in that node to assemble two parallel divided states of this subtree.
-
emit_data()[source]¶ Emit the value at this Store.
Obeys the schema (namely emits only if
_emitis true). Also applies serializers and converts units as necessary.- Returns
The value to emit, or None if nothing should be emitted.
-
generate(path, processes, steps, flow, topology, initial_state)[source]¶ Generate a subtree of this store at the given path. The processes will be mapped into locations in the tree by the topology, and once everything is constructed the initial_state will be applied.
-
generate_value(value)[source]¶ generate the structure for this value that don’t exist, but don’t overwrite any existing values.
-
get_config(sources=False)[source]¶ Assemble a dictionary representation of the config for this node. A desired property is that the node can be exactly recreated by applying the resulting config to an empty node again.
-
get_flow()[source]¶ Get the flow for all steps under this node.
For example:
>>> from vivarium.core.store import Store >>> from vivarium.core.process import Step >>> class MyStep(Step): ... def ports_schema(self): ... return { ... 'port': ['variable'], ... } ... def next_update(self, timestep, states): ... return {} >>> schema = { ... 'agent1': { ... 'store': { ... 'variable': { ... '_default': 0, ... }, ... }, ... 'step1': { ... '_value': MyStep(), ... '_topology': { ... 'port': ('store',), ... }, ... '_flow': [], ... }, ... 'step2': { ... '_value': MyStep(), ... '_topology': { ... 'port': ('store',), ... }, ... '_flow': [('step1',)], ... }, ... }, ... } >>> store = Store(schema) >>> store.get_flow() {'agent1': {'step1': [], 'step2': [('step1',)]}}
-
get_paths(paths)[source]¶ Get the nodes at each of the specified paths.
- Parameters
paths – Map from keys to paths.
- Returns
A dictionary with the same keys as
paths. Each key is mapped to the Store object at the associated path.
-
get_template(template)[source]¶ Pass in a template dict with None for each value you want to retrieve from the tree!
-
get_value(condition=None, f=None)[source]¶ Pull the values out of the tree in a structure symmetrical to the tree.
-
get_values(paths)[source]¶ Get the values at each of the provided paths.
- Parameters
paths – Map from keys to paths.
- Returns
A dictionary with the same keys as
paths. Each key is mapped to the value at the associated path.
-
outer_path(path, source=None)[source]¶ Address a topology with the _path keyword if present, establishing a path to this node and using it as the starting point for future path operations.
-
schema_keys= {'_default', '_emit', '_properties', '_serializer', '_updater', '_value'}¶
-
schema_topology(schema, topology)[source]¶ Fill in the structure of the given schema with the connected stores according to the given topology.
-
set_emit_values(paths=None, emit=False)[source]¶ Turn on/off emits for all inner nodes of the list of paths.
-
set_path(path, value)[source]¶ Set a value at a path in the hierarchy.
- Parameters
path – The path relative to
selfwhere the value should be set.value – The value to set. The store node at
pathwill holdvaluewhen this function returns.
-
set_value(value)[source]¶ Set the value for the given tree elements directly instead of using the updaters from their nodes.
-
topology_state(topology)[source]¶ Fill in the structure of the given topology with the values at all the paths the topology points at. Essentially, anywhere in the topology that has a tuple path will be filled in with the value at that path.
This is the inverse function of the standalone inverse_topology.
-
vivarium.core.store.generate_state(processes: Dict[str, Any], topology: Dict[str, Union[Tuple[str, …], dict, object]], initial_state: Optional[Dict[str, Any]], steps: Optional[Dict[str, Any]] = None, flow: Optional[Dict[str, Sequence[Tuple[str, …]]]] = None) → vivarium.core.store.Store[source]¶ Initialize a simulation’s state.
- Parameters
processes – Simulation processes.
topology – Topology linking process ports to stores.
initial_state – Initial simulation state. Omitted variables will be assigned values based on schema defaults.
- Returns
Initialized state.
-
vivarium.core.store.hierarchy_depth(hierarchy, path=())[source]¶ Create a mapping of every path in the hierarchy to the node living at that path in the hierarchy.
-
vivarium.core.store.key_for_value(d, looking)[source]¶ Get the key associated with a value in a dictionary.
Only top-level keys are searched.
- Parameters
d – The dictionary.
looking – The value to look for.
- Returns
The associated key, or None if no key found.
Types¶
-
vivarium.core.types.CompositeDict¶ A dictionary that specifies the processes and topology of a composite.
alias of Dict[str, Any]
-
vivarium.core.types.Flow¶ Mapping from step names to sequences of HierarchyPaths that specify the step’s dependencies.
alias of Dict[str, Sequence[Tuple[str, …]]]
-
vivarium.core.types.HierarchyPath¶ Relative path between nodes in the hierarchy. Like Unix file paths, “..” refers to the parent directory.
alias of Tuple[str, …]
-
vivarium.core.types.OutputDict¶ A dictionary that contains the retrieved output of an experiment
alias of Dict[Union[Tuple, str], Any]
-
vivarium.core.types.Processes¶ Mapping from processes names to Processes, which can be embedded in a hierarchy.
alias of Dict[str, Any]
-
vivarium.core.types.State¶ A dictionary that has the form of a schema, except instead of specifying the properties of each variable, it specifies each variable’s value.
alias of Dict[str, Any]
-
vivarium.core.types.Steps¶ Mapping from step names to Steps, which can be embedded in a hierarchy.
alias of Dict[str, Any]
Processes Package¶
Tree Mass¶
-
class
vivarium.processes.tree_mass.TreeMass(parameters=None)[source]¶ Bases:
vivarium.core.process.StepDerive total mass from molecular counts and weights.
- Parameters
parameters (dict) –
Dictionary of parameters. The following keys are required:
from_path (
tuple): Path to the root of the subtree whose mass will be summed.
-
name= 'mass_deriver'¶
-
vivarium.processes.tree_mass.calculate_mass(value, path, node)[source]¶ Reducer for summing masses in hierarchy
- Parameters
value – The value to add mass to.
path – Unused.
node – The node whose mass will be added.
- Returns
The mass of the node (accounting for the node’s molecular weight, which should be stored in its
mwproperty) added tovalue.
Library Package¶
Utilities for Formatting Output¶
-
vivarium.library.pretty.format_dict(d, sort_keys=True)[source]¶ Format a dict as a pretty string
Aside from the normal JSON-serializable data types, data of type
numpy.int64are supported.For example:
>>> import numpy as np >>> d = { ... 'foo': { ... 'bar': 1, ... '3.0': np.int64(5), ... }, ... 'a': 'hi!', ... 'quantity': 1 * units.fg, ... 'unit': units.fg, ... } >>> print(format_dict(d)) { "a": "hi!", "foo": { "3.0": 5, "bar": 1 }, "quantity": "1 femtogram", "unit": "<Unit('femtogram')>" }
- Parameters
d – The dictionary to format
sort_keys – Whether to sort the dictionary keys. This is useful for reproducible output.
- Returns
A string of the prettily-formatted dictionary
Topology Utilities¶
-
class
vivarium.library.topology.TestUpdateIn[source]¶ Bases:
object-
d= {'bar': {'c': 'd'}, 'foo': {1: {'a': 'b'}}}¶
-
-
vivarium.library.topology.assoc_path(d, path, value)[source]¶ Insert
valueinto the dictionarydatpath.>>> d = {'a': {'b': 'c'}} >>> assoc_path(d, ('a', 'd'), 'e') {'a': {'b': 'c', 'd': 'e'}} >>> d {'a': {'b': 'c', 'd': 'e'}}
Create new dictionaries recursively as needed.
-
vivarium.library.topology.delete_in(d, path)[source]¶ Delete an item from a dictionary by its path.
>>> d = {'a': {'b': 'c', 'd': 'e'}} >>> delete_in(d, ('a', 'b')) >>> d {'a': {'d': 'e'}}
-
vivarium.library.topology.dict_to_paths(root, d)[source]¶ Get all the paths in a dictionary.
For example:
>>> root = ('root', 'subroot') >>> d = { ... 'a': { ... 'b': 'c', ... }, ... 'd': 'e', ... } >>> dict_to_paths(root, d) [(('root', 'subroot', 'a', 'b'), 'c'), (('root', 'subroot', 'd'), 'e')]
-
vivarium.library.topology.get_in(d, path, default=None)[source]¶ Get the value from a dictionary by its path.
>>> d = {'a': {'b': 'c', 'd': 'e'}} >>> get_in(d, ('a', 'b')) 'c' >>> get_in(d, ('a', 'z')) >>> get_in(d, ('a', 'z'), 'y') 'y'
-
vivarium.library.topology.inverse_topology(outer, update, topology, inverse=None, multi_updates=True)[source]¶ Transform an update from the form its process produced into one aligned to the given topology.
The inverse of this function (using a topology to construct a view for the perspective of a Process ports_schema()) lives in Store, called topology_state. This one stands alone as it does not require a store to calculate.
-
vivarium.library.topology.normalize_path(path)[source]¶ Make a path absolute by resolving
..elements.
-
vivarium.library.topology.paths_to_dict(path_list, f=<function <lambda>>)[source]¶ Create a new dictionary that has the paths in
path_list.- Parameters
path_list – A list of tuples
(path, value).f – A function to apply to each value before inserting it into the dictionary.
- Returns
A new dictionary with the specified values (after being passed through
f) at each associated path.
-
vivarium.library.topology.update_in(d, path, f)[source]¶ Update every value in a dictionary based on
f.- Parameters
d – The dictionary
pathapplies to. This object is not modified.path – Path to the sub-dictionary within
dthat should be updated.f – Function to call on every value in the dictionary to update. The updated dictionary’s values will be return values from
f.
- Returns
A copy of
dwith all the values underpathupdated to the value returned whenfis called on the original value.
Datum¶
-
class
vivarium.library.datum.Datum(config)[source]¶ Bases:
dictThe Datum class enables functions to be defined on dicts of a certain schema. It provides two class level variables:
defaults: a dictionary of keys to default values this Datum will have if none is provided to __init__
schema: a dictionary of keys to constructors which invoke subdata.
Once these are defined, a Datum subclass can be constructed with a dict that provides any values beyond the defaults, and then all of the defined methods for that Datum subclass are available to operate on its values. Once the modifications are complete, it can be rendered back into a dict using the to_dict() method.
Plotting Utilities Package¶
Simulation Output Utilities¶
-
vivarium.plots.simulation_output.get_variable_title(path)[source]¶ Get figure title from a variable path.
- Parameters
path – Path to the variable.
- Returns
String representation of the variable suitable for a figure title.
-
vivarium.plots.simulation_output.plot_simulation_output(timeseries_raw, settings: Optional[Dict[str, Any]] = None, out_dir=None, filename='simulation')[source]¶ Plot simulation output, with rows organized into separate columns.
- Arguments::
timeseries (dict): This can be obtained from simulation output with convert_to_timeseries() settings (dict): Accepts the following keys:
column_width (
int): the width (inches) of each column in the figuremax_rows (
int): ports with more states than this number of states get wrapped into a new columnremove_zeros (
bool): if True, timeseries with all zeros get removedremove_flat (
bool): if True, timeseries with all the same value get removedremove_first_timestep (
bool): if True, skips the first timestepskip_ports (
list): entire ports that won’t be plottedshow_state (
list): with[('port_id', 'state_id')]for all states that will be highlighted, even if they are otherwise to be removed TODO: Obsolete?
-
vivarium.plots.simulation_output.plot_variables(output, variables, column_width=8, row_height=1.2, row_padding=0.8, linewidth=3.0, sci_notation=False, default_color='tab:blue', out_dir=None, filename='variables')[source]¶ Create a simple figure with a timeseries for every variable.
- Parameters
output – Simulation output as a map from variable names or paths to timeseries data. Should contain a
timekey whose value is a list of time points.variables – The variables to plot. May be a list of variable names (if simulation output keys are just variable names) or a dictionary with keys
variable(for the variable path),color(for the color to use for the plot), anddisplay(the variable name to display). Ifdisplayis not provided, the result of callingget_variable_title()on the variable path is used.column_width – Figure width.
row_height – Height of each row. Each variable gets one row.
row_padding – Space between rows.
linewidth – Width of timeseries lines.
sci_notation – Either
Falsefor no scientific notation or an integer \(x\) such that scientific notation will be used for values outside the range \([10^{-x}, 10^x]\).default_color – Default timeseries color.
out_dir – Output directory.
filename – Output filename.
- Returns
The figure.
-
vivarium.plots.simulation_output.set_axes(ax, show_xaxis=False, sci_notation=False, y_offset=0.0)[source]¶ Set up plot axes.
- Parameters
ax – The axes to set up.
show_xaxis – Whether to show the x axis.
sci_notation – Either
Falseto not use scientific notation or an integer \(x\) such that scientific notation will be used outside the range \([10^{-x}, 10^x]\).y_offset – Horizontal distance between axis offset text (typically for scientific notation) and the y-axis.
Glossary¶
Note
Some fully-capitalized words and phrases have the meanings specified in RFC 2119.
- ABM
- Agent-Based Model
- Agent-based model
- agent-based model
- Agent-Based Models
- Agent-based models
- agent-based models
Agent-based modeling is a modeling paradigm in which population-level phenomena are emergent from interactions among simple agents. An agent-based model is a model constructed using this paradigm.
- Boundary Store
- boundary store
- Boundary Stores
- boundary stores
Compartments interact through boundary stores that represent how the compartments affect each other. For example, between an environment compartment and a cell compartment, there might be a boundary store to track the flux of metabolites from the cell to the environment and vice versa.
- Compartment
- compartment
- Compartments
- compartments
Compartments are composites at a single level of a hierarchy. Each compartment is like an agent in an agent-based model, which can be nested in an environment and interact with neighbors, parent, and child compartments. These interactions are possible through boundary stores that connect internal processes to states outside of the compartment. Thus, a model might contain a compartment for an environment which contains two child compartments for the two cells in the environment. For more details, see our guide to compartments.
- composer
- Composer
An object with a generate method, which returns a Composite with Processes and Topologies.
- Composite
- composite
- Composites
- composites
Composites are dictionaries with special keys for processes and topology. The processes key map to a dictionary with initialized processes, and the topology specifies how they are wire to stores.
- Deriver
- deriver
- Derivers
- derivers
Derivers have been deprecated in favor of steps.
- Divider
- divider
- Dividers
- dividers
When a cell divides, we have to decide how to generate the states of its daughter cells. Dividers specify how to generate these daughter cells, for example by assigning half of the value of the variable in the mother cell to each of the daughter cells. We assign a divider to each variable in the schema. For more details, see the documentation for
vivarium.core.registry.- Embedded Timeseries
- Embedded timeseries
- embedded timeseries
An embedded timeseries has nearly the same shape as a simulation state dictionary, only each variable’s value is a list of values over time, and there is an additional
timekey. For details, see the guide on emitters.- Emitter
- emitter
- Emitters
- emitters
While a simulation is running, the current state is stored in stores, but this information is overwritten at each timestep with an updated state. When we want to save off variable values for later analysis, we send these data to one of our emitters, each of which formats the data for a storage medium, for example a database or a Kafka message. We then query the emitter to get the formatted data.
- Exchange
- exchange
- Exchanges
- exchanges
The flux between a cell and its environment. This is stored in a boundary store.
- Engine
- engine
- Experiment
- experiment
- Experiments
- experiments
Vivarium defines simulations using
vivarium.core.engine.Engineobjects. These simulations can contain arbitrarily nested compartments, and you can run them to simulate your model over time. See the documentation for theEngineclass and our guide to experiments for more details.- Flow
- flow
- Flows
- flows
Flows specify dependency graphs for running steps. They have the same structure as topologies, but instead of their leaf values being paths, they are lists of paths where each path specifies a dependency step.
- Inner
- inner
A once-removed internal node position relative to a given node in the tree. Nodes can have multiple inners connected to them. The reciprocal relationship is an outer, but a node can have at most one outer.
- Masking
- masking
When Vivarium passes stores to processes, it includes only the variables the process has requested. We call this filtering masking.
- MSM
- Multiscale Model
- Multiscale model
- multiscale model
- Multiscale Models
- Multiscale models
- multiscale models
Multiscale models use different spatial and temporal scales for their component sub-models. For example, Vivarium models a cell’s internal processes and the interactions between cells and their environment at different temporal scales since these processes require different degrees of temporal precision.
- Outer
- outer
A once-removed external node position relative to a given node in the tree. Each node, except for the top-most node, has one outer node that it exists within. The reciprocal relationship is an inner, but a node can have many inners.
- Path
- path
- Paths
- paths
In its most general form, a path specifies an item in a nested dictionary as a tuple of dictionary keys. For example, consider the following nested dictionary:
{ 'a': { 'b': { 'c': None, }, 'd': {}, }, }
In this example, the path
('a', 'b', 'c')points toNone.In Vivarium, we often use paths to identify nodes within the hierarchy. This works because the hierarchy is just a collection of nested dictionaries.
- Path Timeseries
- Path timeseries
- path timeseries
A path timeseries is a flattened form of an embedded timeseries where keys are paths in the simulation state dictionary and values are lists of the variable value over time. We describe simulation data formats in more detail in our guide to emitters.
- Port
- port
- Ports
- ports
When a process needs access to part of the model state, it will be provided a store. The ports of a process are what the process calls those stores. When running a process, you provide a store to each of the process’s ports. Think of the ports as physical ports into which a cable to a store can be plugged.
- Process
- process
- Processes
- processes
A process in Vivarium models a cellular process by defining how the state of the model should change at each timepoint, given the current state of the model. During the simulation, each process is provided with the current state of the model and the timestep, and the process returns an update that changes the state of the model. Each process is an instance of a process class.
To learn how to write a process, check out our process-writing tutorial. For a detailed guide to processes, see our guide to processes.
- Process Class
- Process class
- process class
- Process Classes
- Process classes
- process classes
A process class is a Python class that defines a process’s model. These classes can be instantiated, and optionally configured, to create processes. Each process class must subclass either
vivarium.core.process.Processor another process class.- Process Command
- Process command
- process command
- Process Commands
- Process commands
- process commands
Instructions that let Vivarium communicate with parallel processes in a remote-procedure-call-like fashion. See the processes guide for details.
- Raw Data
- Raw data
- raw data
The primary format for simulation data is “raw data.” See the guide on emitters.
- Schema
- schema
- Schemas
- schemas
A schema defines the properties of a set of variables by associating with each variable a set of schema key-value pairs.
- Schema Key
- Schema key
- schema key
- Schema Keys
- Schema keys
- schema keys
- Schema Value
- Schema value
- schema value
- Schema Values
- Schema values
- schema values
- Schema Key-Value Pair
- Schema key-value pair
- schema key-value pair
- Schema Key-Value Pairs
- Schema key-value pairs
- schema key-value pairs
Each variable is defined by a set of schema key-value pairs. The available keys are defined in
vivarium.core.store.Store.schema_keys. These keys are described in more detail in the documentation forvivarium.core.store.Store.- Serializer
- serializer
- Serializers
- serializers
A serializer is an object that converts data of a certain type into a format that can transmitted and stored.
- Step
- step
- Steps
- steps
Steps run after all processes have run for a timepoint and compute values from the state of the model. These steps may have dependencies on each other, and these dependencies are specified in the
flowkey ofvivarium.core.composer.Compositedictionaries. For more information, see the documentation forvivarium.core.composer.Composer.- Store
- store
- Stores
- stores
The state of the model is broken down into stores, each of which represents the state of some physical or conceptual subset of the overall state. For example, a cell model might have a store for the proteins in the cytoplasm, another for the transcripts in the cytoplasm, and one for the transcripts in the nucleus. Each variable must belong to exactly one store.
- Store API
- store API
An experimental API that allows for simulations to be built up from stores.
- Template
- template
- Templates
- templates
A template describes a genetic element, its binding site, and the available downstream termination sites on genetic material. A chromosome has operons as its templates which include sites for RNA binding and release. An mRNA transcript also has templates which describe where a ribosome can bind and will subsequently release the transcript. Templates are defined in template specifications.
- Template Specification
- Template specification
- template specification
- Template Specifications
- Template specifications
- template specifications
Template specifications define templates as
dictobjects with the following keys:id (
str): The template name. You SHOULD use the name of the associated operon or transcript.position (
int): The index in the genetic sequence of the start of the genetic element being described. In a chromosome, for example, this would denote the start of the modeled operon’s promoter. On mRNA transcripts (where we are describing how ribosomes bind), this SHOULD be set to0.direction (
int):1if the template should be read in the forward direction,-1to proceed in the reverse direction. For mRNA transcripts, this SHOULD be1.sites (
list): A list of binding sites. Each binding site is specified as adictwith the following keys:position (
int): The offset in the sequence from the template position to the start of the binding site. This value is not currently used and MAY be set to 0.length (
int): The length, in base-pairs, of the binding site. This value is not currently used and MAY be set to 0.thresholds (
list): A list of tuples, each of which has a factor name as the first element and a concentration threshold as the second. When the concentration of the factor exceeds the threshold, the site will bind the factor. For example, in an operon the factor would be a transcription factor.
terminators (
list): A list of terminators, which halt reading of the template. As such, which genes are encoded on a template depends on which terminator halts transcription or translation. Each terminator is specified as adictwith the following keys:position (
int): The index in the genetic sequence of the terminator.strength (
int): The relative strength of the terminator. For example, if there remain two terminators ahead of RNA polymerase, the first of strength 3 and the second of strength 1, then there is a 75% chance that the polymerase will stop at the first terminator. If the polymerase does not stop, it is guaranteed to stop at the second terminator.products (
list): A list of the genes that will be transcribed or translated should transcription/translation halt at this terminator.
- Timepoint
- timepoint
- Timepoints
- timepoints
We discretize time into timepoints and update the model state at each timepoint. We collect data from the model at each timepoint. Note that each compartment may be running with different timesteps depending on how finely we need to discretize time.
- Timeseries
- timeseries
“Timeseries” can refer to the general way in whcih we store simulation data or to an embedded timeseries. See the guide on emitters for details.
- Timestep
- timestep
- Timesteps
- timesteps
The amount of time elapsed between two timepoints. This is the amount of time for which processes compute an update. For example, if we discretize time into two-second intervals, then each process will be asked to compute an update for how the state changes over the next two seconds. The timestep is two seconds.
- Topology
- topology
- Topologies
- topologies
A topology defines how stores are associated to ports. This tells Vivarium which store to pass to each port of each process during the simulation. See the constructor documentation for
vivarium.core.engine.Enginefor a more detailed specification of the form of a topology.- Hierarchy
- hierarchy
- Hierarchies
- hierarchies
- Compartment Hierarchy
- compartment hierarchy
- Tree
- tree
- Trees
- trees
We nest the stores of a model to form a tree called a hierarchy. Each internal node is a store and each leaf node is a variable. This tree can be traversed like a directory tree, and stores are identified by paths. For details see the hierarchy guide. Note that this used to be called a tree.
- Update
- update
- Updates
- updates
An update describes how the model state should change due to the influence of a process over some period of time (usually a timestep).
- Updater
- updater
- Updaters
- updaters
An updater describes how an update should be applied to the model state to produce the updated state. For example, the update could be added to the old value or replace it. Updaters are described in more detail in the documentation for
vivarium.core.registry.- Variable
- variable
- Variables
- variables
The state of the model is a collection of variables. Each variable stores a piece of information about the full model state. For example, the concentration of glucose in the cytoplasm might be a variable, while the concentration of glucose-6-phosphate in the cytoplasm is another variable. The extracellular concentration of glucose might be a third variable. As these examples illustrate, variables are often track the amount of a molecule in a physical region. Exceptions exist though, for instance whether a cell is dead could also be a variable.
Each variable is defined by a set of schema key-value pairs.
- WCM
- Whole-Cell Model
- Whole-cell model
- whole-cell model
- Whole-Cell Models
- Whole-cell models
- whole-cell models
Whole-cell models seek to simulate a cell by modeling the molecular mechanisms that occur within it. For example, a cell’s export of antibiotics might be modeled by the transcription of the appropriate genes, translation of the produced transcripts, and finally complexation of the translated subunits. Ideally the simulated phenotype is emergent from the modeled processes, though many such models also include assumptions that simplify the model.
Introduction¶
A vivarium, literally a “place of life,” is a controlled environment in which organisms can be studied. You might have encountered examples of vivaria like an aquarium or a terrarium. The Vivarium project provides a framework for building biological systems in-silico.
Vivarium does not include any specific modeling frameworks, but instead focuses on the interface between such frameworks, and provides a powerful multiscale simulation engine that combines and runs them. Users of Vivarium can therefore implement any type of model module they prefer – whether it is a custom module of a specific biophysical system, a configurable model with its own standard format, or a wrapper for an off-the-shelf library. The multiscale engine supports complex agents operating at multiple timescales, and facilitates parallel computation across multiple CPUs.
Using This Documentation¶
If you want to run Vivarium, start with our getting started guide, which will walk you through getting Vivarium up and running.
For step-by-step instructions, check out our tutorials.
For a technical deep-dive into the important concepts in Vivarium, check out our topic guides.
If you want to look something up like the configuration options for some process or the definition of a word, see our reference information.
Citing Vivarium¶
- Please use the following reference when citing Vivarium:
Agmon, E., Spangler, R.K., Skalnik, C.J., Poole, W., Morrison, J.H., Peirce, S.M., and Covert, M.W. (2022). Vivarium: an interface and engine for integrative multiscale modeling in computational biology. Bioinformatics, btac049. link to article