Discrete Time Series Benchmarking module

causalai.benchmark.time_series.discrete

This is the benchmarking module for discrete time_series data. This module supports methods that evaluates causal discovery algorithms against various challenges, such as their sample complexity, variable complexity, etc. Users can use either synthetically generated data, or provide their own data for benchmarking.

The default evaluation metrics supported by this module are Precision, Recall, F1 Score, and Time Taken by the algorithm. There is also an option for users to include their own custom metrics when calling the benchmarking module.

We provide support for a default set of causal discovery algorithms. Users also have the option to include their own algorithm when calling the benchmarking module.

Data:

1. Synthetic data: this module randomly generates both the causal graph (and the corresponding structural equation model) and the data associated with it. This module supports several benchmarking methods which evaluate causasl discovery algorithms on various aspects such as sample complexity, variable complexity, graph sparsity, etc. Depending on what is being evaluated, the corresponding method generates the graphs and data accordingly. Synthetic data evaluation serves two purposes:

compare the performance of each causal discovery algorithm across different values of a variant (e.g. increasing number of sample),

compare the performance of different causal discovery algorithms for any on given value of a variant.

2. User provided data: In this case, since the data is fixed, this module helps evaluate the performance of one or more causal discovery algorithms on the provided data. Since the data is not synthetically generated, in order to compute the evaluation metrics such as Precision/Recall, we need the ground truth causal graph. Therefore, the user provided data accepted by this module must contain this information. Specifically, the data must be a list of tuples, where each tuple contains the triplet (data_array, var_names, graph_gt), where data_array is a 2D Numpy data array of shape (samples x variables), var_names is a list of variable names, and graph_gt is the ground truth causal graph in the form of a Python dictionary, where keys are the variable names, and the corresponding values are a list of parent names.

class causalai.benchmark.time_series.discrete.BenchmarkDiscreteTimeSeries(algo_dict: Dict | None = None, kargs_dict: Dict | None = None, num_exp: int = 20, custom_metric_dict: Dict | None = {}, **kargs)

Discrete time_series data benchmarking module. This class inherits the methods and variables from BenchmarkTimeSeriesBase and BenchmarkDiscreteTimeSeriesBase, and defines benchmarking methods that evaluates causal discovery algorithms against various challenges, such as their sample complexity, variable complexity, etc.

__init__(algo_dict: Dict | None = None, kargs_dict: Dict | None = None, num_exp: int = 20, custom_metric_dict: Dict | None = {}, **kargs)

Discrete time_series data benchmarking module

Parameters:

algo_dict (Dict) --
A Python dictionary where keys are names of causal discovery algorithms, and values are the unistantiated class objects for the corresponding algorithm. Note that this class must be inherited from the BaseTimeSeriesAlgoFull class that can be found in causalai.models.time_series.base. Crucially, this class constructor must take a TimeSeriesData object (found in causalai.data.time_series) as input, and should have a run method which performs the causal discovery and returns a Python dictionary. The keys of this dictionary should be of the form:

{
var_name1: {'parents': [par(var_name1)]}, var_name2: {'parents': [par(var_name2)]}

}

where par(.) denotes the parent variable name of the argument variable name.
kargs_dict (Dict) -- A Python dictionary where keys are names of causal discovery algorithms (same as algo_dict), and the corresponding values contain any arguments to be passed to the run method of the class object specified in algo_dict.
num_exp (int) -- The number of independent runs to perform per experiment, each with a different random seed. A different random seed generates a different synthetic graph and data for any given configuration. Note that for use provided data, num_exp is not used.
custom_metric_dict (Dict) -- A Python dictionary for specifying custom metrics in addition to the default evaluation metrics calculated for each experiment (precision, recall, F1 score, and time taken). The keys of this dictionary are the names of the user specified metrics, and the corresponding values are callable functions that take as input (graph_est, graph_gt). Here graph_est and graph_gt are the estimated and ground truth causal graph. These graphs are specified as Python Dictionaries, where keys are the children names, and the corresponding values are lists of parent variable names.

benchmark_data_max_lag(data_max_lag_list: ~typing.List[int] = [1, 5, 10], num_vars: int = 20, graph_density: float = 0.1, T: int = 1000, fn: ~typing.Callable = <function BenchmarkDiscreteTimeSeries.<lambda>>, coef: float = 0.1, noise_fn: ~typing.Callable = <built-in method randn of numpy.random.mtrand.RandomState object>)

Graph density: Benchmark algorithms on synthetic data with different number of samples. The synthetic data for any variable is generated using a structural equation model (SEM) of the form:

child = sum_i coef* parent_i + noise

and then discretized by binning.

Parameters:

data_max_lag_list (List[int]) -- It contains list of max lag values to be used to generate the causal graph. Each value must be an integer greater than 0. For instance, a max lag of 2 implies that the parents of a node (the the generated causal graph) cannot be more than 2 time steps behind the time step of that node.
num_vars (int) -- Integer value specifying the number of variables in the generated data.
graph_density (float) -- Float value in (0,1] specifying the density of the causal graph. The value is used as the probability with which an edge between 2 nodes exists during the causal graph generation process.
T (int) -- Integer value specifying the number of samples in the generated data.
fn (Callable) -- Callable function that acts as the non-linearity on parent variable value in the structural equation model. This same function is applied on all parents.
coef (float) -- Coefficient for the parent variable value in the structural equation model. This same coefficient is used for all the parents.
noise_fn (Callable) -- Callable function from which noise is sampled in the structural equation model. This same function is used in all the equations.

benchmark_graph_density(graph_density_list: ~typing.List[float] = [0.05, 0.1, 0.2, 0.5], num_vars: int = 20, T: int = 1000, data_max_lag: int = 3, fn: ~typing.Callable = <function BenchmarkDiscreteTimeSeries.<lambda>>, coef: float = 0.1, noise_fn: ~typing.Callable = <built-in method randn of numpy.random.mtrand.RandomState object>)

Graph density: Benchmark algorithms on synthetic data with different number of samples. The synthetic data for any variable is generated using a structural equation model (SEM) of the form:

child = sum_i coef* parent_i + noise

and then discretized by binning.

Parameters:

graph_density_list (List[float]) -- It contains list of graph denity values to be used to generate the causal graph. Each value must be in (0,1].
num_vars (int) -- Integer value specifying the number of variables in the generated data.
T (int) -- Integer value specifying the number of samples in the generated data.
data_max_lag (int) -- max lag value to be used to generate the causal graph. This value must be an integer greater than 0. For instance, a max lag of 2 implies that the parents of a node (the the generated causal graph) cannot be more than 2 time steps behind the time step of that node.
fn (Callable) -- Callable function that acts as the non-linearity on parent variable value in the structural equation model. This same function is applied on all parents.
coef (float) -- Coefficient for the parent variable value in the structural equation model. This same coefficient is used for all the parents.
noise_fn (Callable) -- Callable function from which noise is sampled in the structural equation model. This same function is used in all the equations.

benchmark_sample_complexity(T_list: ~typing.List[int] = [100, 500, 1000, 5000], num_vars: int = 20, graph_density: float = 0.1, data_max_lag: int = 3, fn: ~typing.Callable = <function BenchmarkDiscreteTimeSeries.<lambda>>, coef: float = 0.1, noise_fn: ~typing.Callable = <built-in method randn of numpy.random.mtrand.RandomState object>)

Sample Complexity: Benchmark algorithms on synthetic data with different number of samples. The synthetic data for any variable is generated using a structural equation model (SEM) of the form:

child = sum_i coef* parent_i + noise

and then discretized by binning.

Parameters:

T_list (List[int]) -- It contains list of number of samples to be used to generate synthetic data.
num_vars (int) -- Integer value specifying the number of variables in the generated data.
graph_density (float) -- Float value in (0,1] specifying the density of the causal graph. The value is used as the probability with which an edge between 2 nodes exists during the causal graph generation process.
data_max_lag (int) -- max lag value to be used to generate the causal graph. This value must be an integer greater than 0. For instance, a max lag of 2 implies that the parents of a node (the the generated causal graph) cannot be more than 2 time steps behind the time step of that node.
fn (Callable) -- Callable function that acts as the non-linearity on parent variable value in the structural equation model. This same function is applied on all parents.
coef (float) -- Coefficient for the parent variable value in the structural equation model. This same coefficient is used for all the parents.
noise_fn (Callable) -- Callable function from which noise is sampled in the structural equation model. This same function is used in all the equations.

benchmark_variable_complexity(num_vars_list: ~typing.List[int] = [2, 10, 20, 40], graph_density: float = 0.1, T: int = 1000, data_max_lag: int = 3, fn: ~typing.Callable = <function BenchmarkDiscreteTimeSeries.<lambda>>, coef: float = 0.1, noise_fn: ~typing.Callable = <built-in method randn of numpy.random.mtrand.RandomState object>)

Variable Complexity: Benchmark algorithms on synthetic data with different number of variables. The synthetic data for any variable is generated using a structural equation model (SEM) of the form:

child = sum_i coef* parent_i + noise

and then discretized by binning.

Parameters:

num_vars_list (List[int]) -- It contains list of number of variables to be used to generate synthetic data.
graph_density (float) -- Float value in (0,1] specifying the density of the causal graph. The value is used as the probability with which an edge between 2 nodes exists during the causal graph generation process.
T (int) -- Integer value specifying the number of samples in the generated data.
data_max_lag (int) -- max lag value to be used to generate the causal graph. This value must be an integer greater than 0. For instance, a max lag of 2 implies that the parents of a node (the the generated causal graph) cannot be more than 2 time steps behind the time step of that node.
fn (Callable) -- Callable function that acts as the non-linearity on parent variable value in the structural equation model. This same function is applied on all parents.
coef (float) -- Coefficient for the parent variable value in the structural equation model. This same coefficient is used for all the parents.
noise_fn (Callable) -- Callable function from which noise is sampled in the structural equation model. This same function is used in all the equations.