{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Integrated-gradient on IMDB dataset (Tensorflow)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This is an example of the integrated-gradient method on text classification with a Tensorflow model. If using this explainer, please cite the original work: https://github.com/ankurtaly/Integrated-Gradients." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "import unittest\n", "import numpy as np\n", "import pandas as pd\n", "import tensorflow as tf\n", "import sklearn\n", "from sklearn.datasets import fetch_20newsgroups\n", "from omnixai.data.text import Text\n", "from omnixai.preprocessing.text import Word2Id\n", "from omnixai.explainers.nlp.specific.ig import IntegratedGradientText" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We apply a simple CNN model for this text classification task. Note that the method `call` has two inputs `inputs` (token ids) and `masks` (the sentence masks). For `IntegratedGradientText`, the first input of the model must be the token ids." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "class TextModel(tf.keras.Model):\n", "\n", " def __init__(self, num_embeddings, num_classes, **kwargs):\n", " super().__init__()\n", " self.num_embeddings = num_embeddings\n", " self.embedding_size = kwargs.get(\"embedding_size\", 50)\n", " hidden_size = kwargs.get(\"hidden_size\", 100)\n", " kernel_sizes = kwargs.get(\"kernel_sizes\", [3, 4, 5])\n", "\n", " self.embedding = tf.keras.layers.Embedding(\n", " num_embeddings,\n", " self.embedding_size,\n", " embeddings_initializer=tf.keras.initializers.RandomUniform(minval=-0.1, maxval=0.1),\n", " name='embedding'\n", " )\n", " self.conv_layers = [\n", " tf.keras.layers.Conv1D(hidden_size, k, activation='relu', padding='same')\n", " for k in kernel_sizes\n", " ]\n", " self.dropout = tf.keras.layers.Dropout(0.2)\n", " self.output_layer = tf.keras.layers.Dense(num_classes)\n", "\n", " def call(self, inputs, masks, training=False):\n", " embeddings = self.embedding(inputs)\n", " x = embeddings * tf.expand_dims(masks, axis=-1)\n", " x = [tf.reduce_max(layer(x), axis=1) for layer in self.conv_layers]\n", " x = self.dropout(tf.concat(x, axis=1)) if training \\\n", " else tf.concat(x, axis=1)\n", " outputs = self.output_layer(x)\n", " return outputs" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We use a `Text` object to represent a batch of texts/sentences. The package `omnixai.preprocessing.text` provides some transforms related to text data such as `Tfidf` and `Word2Id`." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# Load the training and test datasets\n", "train_data = pd.read_csv('/home/ywz/data/imdb/labeledTrainData.tsv', sep='\\t')\n", "n = int(0.8 * len(train_data))\n", "x_train = Text(train_data[\"review\"].values[:n])\n", "y_train = train_data[\"sentiment\"].values[:n].astype(int)\n", "x_test = Text(train_data[\"review\"].values[n:])\n", "y_test = train_data[\"sentiment\"].values[n:].astype(int)\n", "class_names = [\"negative\", \"positive\"]\n", "# The transform for converting words/tokens to IDs\n", "transform = Word2Id().fit(x_train)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The preprocessing function converts a batch of texts into token IDs and the masks. The outputs of the preprocessing function must fit the inputs of the model." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "max_length = 256\n", "\n", "def preprocess(X: Text):\n", " samples = transform.transform(X)\n", " max_len = 0\n", " for i in range(len(samples)):\n", " max_len = max(max_len, len(samples[i]))\n", " max_len = min(max_len, max_length)\n", " inputs = np.zeros((len(samples), max_len), dtype=int)\n", " masks = np.zeros((len(samples), max_len), dtype=np.float32)\n", " for i in range(len(samples)):\n", " x = samples[i][:max_len]\n", " inputs[i, :len(x)] = x\n", " masks[i, :len(x)] = 1\n", " return inputs, masks" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We now train the CNN model and evaluate its performance." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Training loss at epoch 0, step 0: 0.6866752505302429\n", "Training loss at epoch 1, step 0: 0.4109169542789459\n", "Training loss at epoch 2, step 0: 0.21237820386886597\n", "Training loss at epoch 3, step 0: 0.1540527492761612\n", "Training loss at epoch 4, step 0: 0.08126655220985413\n", "Training loss at epoch 5, step 0: 0.02999718114733696\n", "Training loss at epoch 6, step 0: 0.008433952927589417\n", "Training loss at epoch 7, step 0: 0.009998280555009842\n", "Training loss at epoch 8, step 0: 0.0030068857595324516\n", "Training loss at epoch 9, step 0: 0.001554026734083891\n" ] } ], "source": [ "learning_rate=1e-3\n", "batch_size=128\n", "num_epochs=10\n", "\n", "model = TextModel(\n", " num_embeddings=transform.vocab_size,\n", " num_classes=len(class_names)\n", ")\n", "inputs, masks = preprocess(x_train)\n", "optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)\n", "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)\n", "train_dataset = tf.data.Dataset.from_tensor_slices((inputs, masks, y_train))\n", "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(batch_size)\n", "\n", "for epoch in range(num_epochs):\n", " for step, (ids, masks, labels) in enumerate(train_dataset):\n", " with tf.GradientTape() as tape:\n", " logits = model(ids, masks, training=True)\n", " loss = loss_fn(labels, logits)\n", " grads = tape.gradient(loss, model.trainable_weights)\n", " optimizer.apply_gradients(zip(grads, model.trainable_weights))\n", " if step % 200 == 0:\n", " print(f\"Training loss at epoch {epoch}, step {step}: {float(loss)}\")" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Test accuracy: 0.8560798903465829\n" ] } ], "source": [ "inputs, masks = preprocess(x_test)\n", "outputs = model(inputs, masks).numpy()\n", "predictions = np.argmax(outputs, axis=1)\n", "print('Test accuracy: {}'.format(\n", " sklearn.metrics.f1_score(y_test, predictions, average='binary')))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "To initialize `IntegratedGradientText`, we need to set the following parameters:\n", "\n", " - `model`: The model to explain, whose type is `tf.keras.Model` or `torch.nn.Module`.\n", " - `embedding_layer`: The embedding layer in the model, which can be `tf.keras.layers.Layer` or `torch.nn.Module`.\n", " - `preprocess_function`: The pre-processing function that converts the raw input data into the inputs of `model`. The first output of `preprocess_function` should be the token ids.\n", " - `mode`: The task type, e.g., `classification` or `regression`.\n", " - `id2token`: The mapping from token ids to tokens." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
Instance 0: Class positive
\n", "
what a great movie if you have no taste

\n", "
Instance 1: Class positive
\n", "
it was a fantastic performance

\n", "
Instance 2: Class positive
\n", "
best film ever

\n", "
Instance 3: Class positive
\n", "
such a great show

\n", "
Instance 4: Class negative
\n", "
it was a horrible movie

\n", "
Instance 5: Class negative
\n", "
i never watched something as bad

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "explainer = IntegratedGradientText(\n", " model=model,\n", " embedding_layer=model.embedding,\n", " preprocess_function=preprocess,\n", " id2token=transform.id_to_word\n", ")\n", "x = Text([\n", " \"What a great movie! if you have no taste.\",\n", " \"it was a fantastic performance!\",\n", " \"best film ever\",\n", " \"such a great show!\",\n", " \"it was a horrible movie\",\n", " \"i've never watched something as bad\"\n", "])\n", "explanations = explainer.explain(x)\n", "explanations.ipython_plot(class_names=class_names)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.5" } }, "nbformat": 4, "nbformat_minor": 2 }