logai.algorithms.nn_model package

Subpackages

Submodules

logai.algorithms.nn_model.transformers module

class logai.algorithms.nn_model.transformers.LogDataset(encodings, labels)

Bases: Dataset

Wrapper class for Log Dataset, to wrap over torch Dataset class.

class logai.algorithms.nn_model.transformers.TransformerAlgo(config: TransformerAlgoConfig)

Bases: object

HuggingFace Transformer based Pretrained Language model (e.g. “bert-base-cased”), with a sequence classifier head for any supervised log classification task. For e.g. log anomaly detection is one type of log classfication task where the labels are Normal (Label 0) or Anomalous (Label 1). Currently it supports only binary classification, to change this num_labels of AutoModelForSequenceClassification has to be changed accordingly along with the prediction logic in predict method.

predict(test_logs: Series, test_labels: Series) → Tuple[Series, ndarray, Dict[str, float]]

Predict method for running evaluation on test log data.

Parameters:

  • test_logs – The test log features data (output of LogVectorizer).

  • test_labels – The labels of test log data.

Returns:

  • res (pd.Series): Predicted test labels as pandas Series object.

  • label_ids (np.ndarray, optional): True test labels (if the dataset contained some).

  • metrics (Dict[str, float], optional): The potential dictionary of metrics.

save(output_dir: str)

Save model in given directory.

Parameters:

output_dir – The path to output directory where model should be dumped.

train(train_logs: Series, train_labels: Series)

Train method for Transformer based pretrained language model with a sequence classification head for supervised log classification task. Internally this method also splits the available training logs into train and dev data.

Parameters:

  • train_logs – The training log vectors data (after LogVectorizer).

  • train_labels – The training label data.

train_with_native_torch(train_logs: Series, train_labels: Series)

Train models in native torch way.

Parameters:

  • train_logs – The training log features data (after LogVectorizer).

  • train_labels – The label data for training logs.

class logai.algorithms.nn_model.transformers.TransformerAlgoConfig(tokenizer_config: dict = {'model': 'bert-base-cased', 'name': 'auto'}, trainer_config: dict = {})

Bases: Config

Config class for Transformer based model for log classification tasks.

tokenizer_config: dict
trainer_config: dict

Module contents