{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "\n# Debugger script\n\nThis script defines the pipeline entry point, the parameter-space,\n and invokes one of BugDoc's algorithm to debug the pipeline.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Importing algorithm from BugDoc's API\nWe choose the Stacked Shortcut Algorithm to debug the pipeline.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "from bugdoc.algos.stacked_shortcut import StackedShortcut"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Parameter space definition\nThe parameter-values that BugDoc tries can be retrieved in two ways:\nFrom previous executions of the pipeline or specifying the all possible values each parameter can take.\nIn the following, we provide the entry point of the pipeline and a dictionary with the parameter names as keys\nand a list of parameter-values as the corresponding value.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "filename = 'dataflow_pipeline.json'\nparameter_space = {\n    'operator1_script': [\n        '/data/scripts/operator1/script1.py',\n        '/data/scripts/operator1/script2.py',\n        '/data/scripts/operator1/script3.py',\n        '/data/scripts/operator1/script4.py',\n        '/data/scripts/operator1/script5.py',\n        '/data/scripts/operator1/script6.py',\n        '/data/scripts/operator1/script7.py',\n        '/data/scripts/operator1/script8.py',\n        '/data/scripts/operator1/script9.py',\n    ],\n    'read_path': [\n        '/tmp/inputs_20200527.csv',\n        '/tmp/inputs_20200528.csv',\n        '/tmp/inputs_20200529.csv',\n        '/tmp/inputs_20200530.csv',\n        '/tmp/inputs_20200531.csv',\n    ],\n    'read_mode': ['once', 'repeat'],\n    'write_mode': ['append', 'concat', 'replace'],\n    'operator2_script': [\n        '/data/scripts/operator2/aggregation.py',\n        '/data/scripts/operator2/average.py',\n        '/data/scripts/operator2/reduce.py',\n    ],\n    'threshold': [0.1, 0.5],\n    'folds': [2, 5, 10]\n}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Pipeline Debugging\nWe initialize the Stacked Shortcut Algorithm object and run it passing the pipeline entry point and\nthe parameter space. The Algorithm will generate new pipeline instances and exchange messages with the *Worker*\nscript to execute and evaluate the instances. This process will be blocked if no *Worker* is running.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "debugger = StackedShortcut()\nresult = debugger.run(filename, parameter_space)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Revealing the root cause\nWhen the algorithm finishes we can display the root cause of error.\n\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "collapsed": false
      },
      "outputs": [],
      "source": [
        "root, _, _ = result\nparameters = list(parameter_space.keys())\nprint('Root Cause: \\n%s' % (\n    ' OR '.join(\n     [\n        ' AND '.join(\n         [parameters[pair[0]]+' = '+pair[1] for pair in r]\n                    )\n        for r in root])))"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.20"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}