{ "cells": [ { "cell_type": "markdown", "id": "322dd88a", "metadata": {}, "source": [ "***\n", "# Building an Image Classifier using AutoMLx\n", "

by the Oracle AutoMLx Team

\n", "\n", "***" ] }, { "cell_type": "markdown", "id": "6d8cbf4b", "metadata": {}, "source": [ "Image Classification Demo Notebook.\n", "\n", "Copyright © 2025, Oracle and/or its affiliates.\n", "\n", "Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/" ] }, { "cell_type": "markdown", "id": "faf204cc", "metadata": {}, "source": [ "## Overview of this Notebook\n", "\n", "In this notebook we will build a image classifier using the Oracle AutoMLx tool for the public PneumoniaMNIST dataset which is part of MedMNIST datasets. The dataset is a multi-label classification dataset, and more details about the dataset can be found at https://medmnist.com/.\n", "We explore the various options provided by the Oracle AutoMLx tool, allowing the user to exercise control over the AutoML training process. We then evaluate the different models trained by AutoML.\n", "\n", "---\n", "## Prerequisites\n", "\n", " - Experience level: Novice (Python and Machine Learning)\n", " - Professional experience: Some industry experience\n", "---\n", "\n", "## Business Use\n", "\n", "Data analytics and modeling problems using Machine Learning (ML) are becoming popular and often rely on data science expertise to build accurate ML models. Such modeling tasks primarily involve the following steps:\n", "- Pick an appropriate model for the given dataset and prediction task at hand.\n", "- Tune the chosen model’s hyperparameters for the given dataset.\n", "\n", "All of these steps are significantly time consuming and heavily rely on data scientist expertise. Unfortunately, to make this problem harder, the best model, and hyperparameter choice widely varies with the dataset and the prediction task. Hence, there is no one-size-fits-all solution to achieve reasonably good model performance. Using a simple Python API, AutoMLx can quickly jump-start the datascience process with an accurately-tuned model for a given prediction task.\n", "\n", "## Table of Contents\n", "\n", "- Setup\n", "- Load the PneumoniaMNIST dataset\n", "- AutoML\n", " - Setting the execution engine\n", " - Create an Instance of Oracle AutoMLx\n", " - Train a Model using AutoML\n", " - Analyze the AutoML optimization process \n", " - Algorithm Selection\n", " - Adaptive Sampling\n", " - Model Tuning\n", " - Confusion Matrix\n", " - Advanced AutoML Configuration \n", "- References" ] }, { "cell_type": "markdown", "id": "25354007", "metadata": {}, "source": [ "\n", "## Setup\n", "\n", "Basic setup for the Notebook." ] }, { "cell_type": "code", "execution_count": 1, "id": "24950533", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:17.928127Z", "iopub.status.busy": "2025-04-25T10:15:17.927799Z", "iopub.status.idle": "2025-04-25T10:15:18.665789Z", "shell.execute_reply": "2025-04-25T10:15:18.665103Z" } }, "outputs": [], "source": [ "\n", "%matplotlib inline\n", "%load_ext autoreload\n", "%autoreload 2" ] }, { "cell_type": "markdown", "id": "7c105ab0", "metadata": {}, "source": [ "Load the required modules." ] }, { "cell_type": "code", "execution_count": 2, "id": "7a3b26ba", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:18.667923Z", "iopub.status.busy": "2025-04-25T10:15:18.667629Z", "iopub.status.idle": "2025-04-25T10:15:21.742265Z", "shell.execute_reply": "2025-04-25T10:15:21.741594Z" }, "lines_to_next_cell": 0 }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import plotly.express as px\n", "from sklearn.metrics import balanced_accuracy_score, roc_auc_score\n", "from datasets import load_dataset\n", "from sklearn.model_selection import train_test_split\n", "\n", "# Settings for plots\n", "plt.rcParams['figure.figsize'] = [4, 3]\n", "plt.rcParams['font.size'] = 15\n", "\n", "import automlx" ] }, { "cell_type": "markdown", "id": "168a2cd3", "metadata": {}, "source": [ "\n", "## Load the PneumoniaMNIST dataset\n", "We start by reading in the dataset from Hugging Face." ] }, { "cell_type": "code", "execution_count": 3, "id": "b4da04c9", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:21.744596Z", "iopub.status.busy": "2025-04-25T10:15:21.744172Z", "iopub.status.idle": "2025-04-25T10:15:23.306705Z", "shell.execute_reply": "2025-04-25T10:15:23.306165Z" } }, "outputs": [], "source": [ "dataset = load_dataset(\"albertvillanova/medmnist-v2\", \"pneumoniamnist\")" ] }, { "cell_type": "markdown", "id": "fedcf8c9", "metadata": {}, "source": [ "Lets look at a few of the values in the data" ] }, { "cell_type": "code", "execution_count": 4, "id": "1bababf0", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:23.308953Z", "iopub.status.busy": "2025-04-25T10:15:23.308481Z", "iopub.status.idle": "2025-04-25T10:15:23.359988Z", "shell.execute_reply": "2025-04-25T10:15:23.359486Z" } }, "outputs": [ { "data": { "text/plain": [ "{'image': [,\n", " ,\n", " ,\n", " ,\n", " ],\n", " 'label': [1, 1, 1, 1, 1]}" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dataset[\"train\"][:5]" ] }, { "cell_type": "markdown", "id": "fe8c24e5", "metadata": {}, "source": [ "Plot one of the images as an example" ] }, { "cell_type": "code", "execution_count": 5, "id": "16fcf459", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:23.361844Z", "iopub.status.busy": "2025-04-25T10:15:23.361436Z", "iopub.status.idle": "2025-04-25T10:15:23.387475Z", "shell.execute_reply": "2025-04-25T10:15:23.387020Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Pneumonia\n" ] }, { "data": { "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/wAALCAAcABwBAREA/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/9oACAEBAAA/AH2On2utWoLxRrCFySBxitnRrOzsA39mMF2rtcR/dzjtVPxPo7a9E1tPOv3MBWPJHfAFeYSfDO4SRlF22M8fJ2ru9Nnaw8Iadb/P5ssSvJtHPPQf59a7Pw/ZpHBlUIbGWJ6k1k+IvMt7uG6RsfNjOe9RX8yySxSnOZIlaqI0DU72VZYUjWJAojDHnAGP6V3OjpKlsVuFCSAAEde1ZGsmKe2lWQjbzye1cZc3ZkittpPyxbT9dzV6npsa+QvHatFY02N8orIvII1jKqgA9BXFXdlBb3LrEmxSd2B61//Z", "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAACX0lEQVR4ARVSy3IbOQwEQZCch0YaS7uytcm6ktr1LYd8UP44ySHXTe0hhyQuyyVnNJoXSYChcGxUoRvdrT60pR+H5+Pxze2/D/eH5VKr/z7//PoRjBZ6KPV0qard+ybIpS/Pzdg9BqQ6iFe0MohYbGDd9h6Grntyw9JC/+6LB6CjLcmQcV27n0eIQGUwDO71dJoK6kooi7ou2VlCRrXWrFa6plfHC1FMScy6wbfwpJ1VEmbY7kx3XjWjopNV2lXwC7rj7haRmlj4OanZNI7puP6jqMvL83cvWwJdC4efj3/+fTelecCXpbSH8WWuyfnFFfudnaHonleHg2Wak74bzz+mYNpdRSoyp7jE7qk1SuFFTHs6vUR1c7ixRIrJaojDcUyEFN1N6IfS0HaNrG1ayBmERc2tU4g3tyc/msUih+xnfgxIS/C96ES0xsexS+NqcmVhFWsxTTOw7gIE3GxO5wk6KhAyGxImWjdaYBIVcd/2Clr3ZlsbgoRF9JJtmxZAFHxVfR+evhU+bR0aSUNVO3D3+879b2daBZ+KhLJE0ERWpiBQNnM/uRjozvdBi5+iaAQODMxo6rG+RPa0Ob/MKx+g25Q1CDNAEgZb/WIRWgZ0m4DmfE4tKUMd5NjYNLLMTIEe6BCU9I3JdwE8S0qAJH3PVBV/pX0kHssBmWMMXjDx1KXhoqiudoPSBdRqEZb8XGCS0D8r73MH17zJYQikHVGW2ltnkm8On05f7ynXKgenQSWlc5z6VqnEQvKPjSnXUufmUIauS8TtlZetem3OC0HGrjpVVnodBVetLLxR3W9GolWycYOSHQAAAABJRU5ErkJggg==", "text/plain": [ "" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "print(\"Pneumonia\" if dataset[\"train\"][0]['label'] == 1 else 'Normal')\n", "dataset[\"train\"][0]['image']" ] }, { "cell_type": "markdown", "id": "92c5c6cf", "metadata": {}, "source": [ "We visualize the distribution of the target variable in the training data." ] }, { "cell_type": "code", "execution_count": 6, "id": "fa9f441e", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:23.389288Z", "iopub.status.busy": "2025-04-25T10:15:23.388884Z", "iopub.status.idle": "2025-04-25T10:15:24.141994Z", "shell.execute_reply": "2025-04-25T10:15:24.141480Z" }, "lines_to_next_cell": 0 }, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "application/vnd.plotly.v1+json": { "config": { "plotlyServerURL": "https://plot.ly" }, "data": [ { "alignmentgroup": "True", "bingroup": "x", "hovertemplate": "label=%{x}
count=%{y}", "legendgroup": "", "marker": { "color": "#636efa", "pattern": { "shape": "" } }, "name": "", "offsetgroup": "", "orientation": "v", "showlegend": false, "type": "histogram", "x": [ "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Normal", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia", "Pneumonia", "Pneumonia", "Normal", "Pneumonia" ], "xaxis": "x", "yaxis": "y" } ], "layout": { "barmode": "group", "legend": { "tracegroupgap": 0 }, "margin": { "t": 60 }, "template": { "data": { "bar": [ { "error_x": { "color": "#2a3f5f" }, "error_y": { "color": "#2a3f5f" }, "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "bar" } ], "barpolar": [ { "marker": { "line": { "color": "#E5ECF6", "width": 0.5 }, "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "barpolar" } ], "carpet": [ { "aaxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "baxis": { "endlinecolor": "#2a3f5f", "gridcolor": "white", "linecolor": "white", "minorgridcolor": "white", "startlinecolor": "#2a3f5f" }, "type": "carpet" } ], "choropleth": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "choropleth" } ], "contour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "contour" } ], "contourcarpet": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "contourcarpet" } ], "heatmap": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmap" } ], "heatmapgl": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "heatmapgl" } ], "histogram": [ { "marker": { "pattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 } }, "type": "histogram" } ], "histogram2d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2d" } ], "histogram2dcontour": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "histogram2dcontour" } ], "mesh3d": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "type": "mesh3d" } ], "parcoords": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "parcoords" } ], "pie": [ { "automargin": true, "type": "pie" } ], "scatter": [ { "fillpattern": { "fillmode": "overlay", "size": 10, "solidity": 0.2 }, "type": "scatter" } ], "scatter3d": [ { "line": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatter3d" } ], "scattercarpet": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattercarpet" } ], "scattergeo": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergeo" } ], "scattergl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattergl" } ], "scattermapbox": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scattermapbox" } ], "scatterpolar": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolar" } ], "scatterpolargl": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterpolargl" } ], "scatterternary": [ { "marker": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "type": "scatterternary" } ], "surface": [ { "colorbar": { "outlinewidth": 0, "ticks": "" }, "colorscale": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "type": "surface" } ], "table": [ { "cells": { "fill": { "color": "#EBF0F8" }, "line": { "color": "white" } }, "header": { "fill": { "color": "#C8D4E3" }, "line": { "color": "white" } }, "type": "table" } ] }, "layout": { "annotationdefaults": { "arrowcolor": "#2a3f5f", "arrowhead": 0, "arrowwidth": 1 }, "autotypenumbers": "strict", "coloraxis": { "colorbar": { "outlinewidth": 0, "ticks": "" } }, "colorscale": { "diverging": [ [ 0, "#8e0152" ], [ 0.1, "#c51b7d" ], [ 0.2, "#de77ae" ], [ 0.3, "#f1b6da" ], [ 0.4, "#fde0ef" ], [ 0.5, "#f7f7f7" ], [ 0.6, "#e6f5d0" ], [ 0.7, "#b8e186" ], [ 0.8, "#7fbc41" ], [ 0.9, "#4d9221" ], [ 1, "#276419" ] ], "sequential": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ], "sequentialminus": [ [ 0.0, "#0d0887" ], [ 0.1111111111111111, "#46039f" ], [ 0.2222222222222222, "#7201a8" ], [ 0.3333333333333333, "#9c179e" ], [ 0.4444444444444444, "#bd3786" ], [ 0.5555555555555556, "#d8576b" ], [ 0.6666666666666666, "#ed7953" ], [ 0.7777777777777778, "#fb9f3a" ], [ 0.8888888888888888, "#fdca26" ], [ 1.0, "#f0f921" ] ] }, "colorway": [ "#636efa", "#EF553B", "#00cc96", "#ab63fa", "#FFA15A", "#19d3f3", "#FF6692", "#B6E880", "#FF97FF", "#FECB52" ], "font": { "color": "#2a3f5f" }, "geo": { "bgcolor": "white", "lakecolor": "white", "landcolor": "#E5ECF6", "showlakes": true, "showland": true, "subunitcolor": "white" }, "hoverlabel": { "align": "left" }, "hovermode": "closest", "mapbox": { "style": "light" }, "paper_bgcolor": "white", "plot_bgcolor": "#E5ECF6", "polar": { "angularaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "radialaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "scene": { "xaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "yaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" }, "zaxis": { "backgroundcolor": "#E5ECF6", "gridcolor": "white", "gridwidth": 2, "linecolor": "white", "showbackground": true, "ticks": "", "zerolinecolor": "white" } }, "shapedefaults": { "line": { "color": "#2a3f5f" } }, "ternary": { "aaxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "baxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" }, "bgcolor": "#E5ECF6", "caxis": { "gridcolor": "white", "linecolor": "white", "ticks": "" } }, "title": { "x": 0.05 }, "xaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 }, "yaxis": { "automargin": true, "gridcolor": "white", "linecolor": "white", "ticks": "", "title": { "standoff": 15 }, "zerolinecolor": "white", "zerolinewidth": 2 } } }, "xaxis": { "anchor": "y", "domain": [ 0.0, 1.0 ], "title": { "text": "label" } }, "yaxis": { "anchor": "x", "domain": [ 0.0, 1.0 ], "title": { "text": "count" } } } }, "text/html": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "y_df = pd.DataFrame(dataset[\"train\"][\"label\"])\n", "y_df.columns = ['label']\n", "\n", "fig = px.histogram(y_df[\"label\"].apply(lambda x: \"Normal\" if x == 0 else \"Pneumonia\"), x=\"label\", barmode=\"group\")\n", "fig.show()" ] }, { "cell_type": "markdown", "id": "49de4f9e", "metadata": { "jp-MarkdownHeadingCollapsed": true }, "source": [ "We now separate the predictions (`y`) from the training data (`X`) for both the training (70%) and test (30%) datasets. The training set will be used to create a Machine Learning model using AutoML, and the test set will be used to evaluate the model's performance on unseen data." ] }, { "cell_type": "code", "execution_count": 7, "id": "c811f583", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:24.176941Z", "iopub.status.busy": "2025-04-25T10:15:24.176415Z", "iopub.status.idle": "2025-04-25T10:15:24.505743Z", "shell.execute_reply": "2025-04-25T10:15:24.505155Z" } }, "outputs": [], "source": [ "X = pd.DataFrame(dataset[\"train\"][\"image\"], columns=['images'])\n", "y = pd.DataFrame(dataset[\"train\"][\"label\"])\n", "X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, random_state=0)\n", "\n", "# reducing the number of samples in training set to speed up the demo\n", "X_train = X_train[:1000]\n", "y_train = y_train[:1000]" ] }, { "cell_type": "markdown", "id": "2f98a92b", "metadata": {}, "source": [ "\n", "## AutoML" ] }, { "cell_type": "markdown", "id": "310a3cb4", "metadata": {}, "source": [ "\n", "### Create an instance of Oracle AutoMLx\n", "\n", "The Oracle AutoMLx solution provides a pipeline that automatically finds a tuned model given a prediction task and a training dataset.\n", "\n", "AutoML consists of three main steps for image classification:\n", "- **Algorithm Selection** : Identify the right classification algorithm for a given dataset.\n", "- **Adaptive Sampling** : Select a subset of the data samples for the model to be trained on.\n", "- **Model Tuning** : Find the right model parameters (including model size) that maximize score for the given dataset.\n", "\n", "All these pieces are readily combined into a simple AutoML pipeline which automates the entire Machine Learning process with minimal user input/interaction." ] }, { "cell_type": "markdown", "id": "a1e146e1", "metadata": {}, "source": [ "\n", "### Train a model using AutoML" ] }, { "cell_type": "markdown", "id": "8d2ed040", "metadata": {}, "source": [ "The AutoMLx API is quite simple to work with. We create an instance of the pipeline. Next, the training data is passed to the `fit()` function which executes the previously mentioned steps.\n", "Note that we decreased number of tuning trials in Model Tuning, to speed up the demo notebook." ] }, { "cell_type": "code", "execution_count": 8, "id": "6ff83718", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:15:24.508237Z", "iopub.status.busy": "2025-04-25T10:15:24.507748Z", "iopub.status.idle": "2025-04-25T10:33:56.789064Z", "shell.execute_reply": "2025-04-25T10:33:56.788491Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:24,710] [automlx.backend] Overwriting ray session directory to /tmp/j9903zbf/ray, which will be deleted at engine shutdown. If you wish to retain ray logs, provide _temp_dir in ray_setup dict of engine_opts when initializing the AutoMLx engine.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:29,829] [automlx.interface] Dataset shape: (1000,1)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:33,331] [sanerec.autotuning.parameter] Hyperparameter epsilon autotune range is set to its validation range. This could lead to long training times\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:33,859] [sanerec.autotuning.parameter] Hyperparameter repeat_quality_threshold autotune range is set to its validation range. This could lead to long training times\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:33,868] [sanerec.autotuning.parameter] Hyperparameter scope autotune range is set to its validation range. This could lead to long training times\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:33,941] [automlx.data_transform] Running preprocessing. Number of features: 2\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:34,824] [automlx.data_transform] Preprocessing completed. Took 0.883 secs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:34,870] [automlx.process] Running Model Generation\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:34,911] [automlx.process] Model Generation completed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:15:34,981] [automlx.model_selection] Running Model Selection\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,401] [automlx.model_selection] Model Selection completed - Took 157.420 sec - Selected models: [['ResNet']]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,429] [automlx.adaptive_sampling] Running Adaptive Sampling. Dataset shape: (1000,2).\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,479] [automlx.adaptive_sampling] Adaptive Sampling: top_limit: 583 < bottom_limit: 1000,\n", "sampling process will be skipped\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,489] [automlx.adaptive_sampling] Adaptive Sampling: top_limit: 146 < bottom_limit: 1000,\n", "sampling process will be skipped\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,584] [automlx.feature_selection] Starting feature ranking for ResNet\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,585] [automlx.feature_selection] min_features is larger than model input features for ResNet, skipping feature selection for it\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:18:12,620] [automlx.trials] Running Model Tuning for ['ResNet']\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:31:47,337] [automlx.trials] Best parameters for ResNet: {'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.0031630600334930504, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:31:47,338] [automlx.trials] Model Tuning completed. Took: 814.718 secs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:31:48,319] [automlx.interface] Re-fitting pipeline\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:31:48,337] [automlx.final_fit] Skipping updating parameter seed, already fixed by FinalFit_d9a60282-f\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:33:54,686] [automlx.interface] AutoMLx completed.\n" ] }, { "data": { "text/plain": [ "" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "est1 = automlx.Pipeline(task='classification', max_tuning_trials=10, score_metric=\"balanced_accuracy\")\n", "est1.fit(X_train, y_train)" ] }, { "cell_type": "markdown", "id": "892f72ed", "metadata": {}, "source": [ "A model is then generated (`est1`) and can be used for prediction tasks. We use the `balanced accuracy` scoring metric to evaluate the performance of this model on unseen data (`X_test`)." ] }, { "cell_type": "code", "execution_count": 9, "id": "bd059b93", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:56.791707Z", "iopub.status.busy": "2025-04-25T10:33:56.791117Z", "iopub.status.idle": "2025-04-25T10:33:59.365352Z", "shell.execute_reply": "2025-04-25T10:33:59.364801Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Score on test data : 0.8916\n" ] } ], "source": [ "y_pred = est1.predict(X_test)\n", "score_default = balanced_accuracy_score(y_test, y_pred)\n", "\n", "print(f'Score on test data : {score_default:.4f}')" ] }, { "cell_type": "markdown", "id": "73ccec9b", "metadata": {}, "source": [ "\n", "### Analyze the AutoML optimization process\n", "\n", "During the AutoML process, a summary of the optimization process is logged. It consists of:\n", "- Information about the training data .\n", "- Information about the AutoML, such as:\n", " - Selected algorithm that was the best choice for this data;\n", " - Selected hyperparameters for the selected algorithm." ] }, { "cell_type": "markdown", "id": "82f5595b", "metadata": {}, "source": [ "AutoMLx provides a `print_summary` API to output all the different trials performed." ] }, { "cell_type": "code", "execution_count": 10, "id": "ce80e60f", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:59.367479Z", "iopub.status.busy": "2025-04-25T10:33:59.366979Z", "iopub.status.idle": "2025-04-25T10:33:59.444125Z", "shell.execute_reply": "2025-04-25T10:33:59.443618Z" } }, "outputs": [ { "data": { "text/html": [ "
General Summary
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
(1000, 1)
None
KFoldSplit(Shuffle=True, Seed=7, folds=5, stratify by=target)
balanced_accuracy
ResNet
{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.0031630600334930504, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}
25.2.1
3.9.21 (main, Dec 11 2024, 16:24:11) \\n[GCC 11.2.0]
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "
Trials Summary
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Step# Samples# FeaturesAlgorithmHyperparametersScore (balanced_accuracy)All MetricsRuntime (Seconds)Memory Usage (GB)Finished
Model Selection{5: 800, 2: 800, 3: 800, 1: 800, 4: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.9429{'balanced_accuracy': 0.9429372134190274}566.48780.8595Fri Apr 25 03:17:38 2025
Model Selection{1: 800, 3: 800, 5: 800, 2: 800, 4: 800}1EfficientNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': 'b0'}0.8153{'balanced_accuracy': 0.8153082390588295}734.15410.8685Fri Apr 25 03:18:12 2025
Model Tuning{4: 800, 3: 800, 2: 800, 1: 800, 5: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.0031630600334930504, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.9421{'balanced_accuracy': 0.9420862061013219}1679.72990.9625Fri Apr 25 03:27:58 2025
Model Tuning{4: 800, 3: 800, 1: 800, 5: 800, 2: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.9409{'balanced_accuracy': 0.9409403911576797}1654.45500.8447Fri Apr 25 03:24:15 2025
Model Tuning{4: 800, 3: 800, 5: 800, 2: 800, 1: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001000099, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.94{'balanced_accuracy': 0.9399735188445627}1676.65480.8474Fri Apr 25 03:24:22 2025
Model Tuning{1: 800, 2: 800, 3: 800, 4: 800, 5: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.0031631590334930504, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.932{'balanced_accuracy': 0.9320400827675271}1606.37380.9188Fri Apr 25 03:28:28 2025
Model Tuning{1: 800, 2: 800, 3: 800, 4: 800, 5: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '34'}0.9319{'balanced_accuracy': 0.9318589293978995}1835.80361.0745Fri Apr 25 03:31:47 2025
Model Tuning{1: 800, 2: 800, 3: 800, 4: 800, 5: 800}1ResNet{'optimizer_class': 'Adagrad', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.9304{'balanced_accuracy': 0.9303550701472287}1599.99090.8875Fri Apr 25 03:28:47 2025
Model Tuning{1: 800, 3: 800, 2: 800, 5: 800, 4: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.000100198, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.9215{'balanced_accuracy': 0.9215387221291802}1473.41440.8585Fri Apr 25 03:23:27 2025
Model Tuning{1: 800, 2: 800, 3: 800, 4: 800, 5: 800}1ResNet{'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.000100099, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.921{'balanced_accuracy': 0.9210114051871254}1514.97270.9292Fri Apr 25 03:23:26 2025
Model Tuning{1: 800, 3: 800, 2: 800, 4: 800, 5: 800}1ResNet{'optimizer_class': 'RMSprop', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.8965{'balanced_accuracy': 0.8965285352342509}1535.37660.8253Fri Apr 25 03:28:50 2025
Model Tuning{1: 800, 2: 800, 3: 800, 4: 800, 5: 800}1ResNet{'optimizer_class': 'SGD', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 128, 'lr': 0.001, 'epochs': 18, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': '18'}0.8746{'balanced_accuracy': 0.874649502424656}1438.39490.7970Fri Apr 25 03:29:03 2025
" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "est1.print_summary()" ] }, { "cell_type": "markdown", "id": "4a92c0c5", "metadata": {}, "source": [ "We also provide the capability to visualize the results of each stage of the AutoML." ] }, { "cell_type": "markdown", "id": "47ddd5d7", "metadata": {}, "source": [ "\n", "#### Algorithm Selection\n", "\n", "The plot below shows the scores predicted by Algorithm Selection for each algorithm. The horizontal line shows the average score across all algorithms. Algorithms below the line are colored turquoise, whereas those with a score higher than the mean are colored teal." ] }, { "cell_type": "code", "execution_count": 11, "id": "8a8ef05a", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:59.446108Z", "iopub.status.busy": "2025-04-25T10:33:59.445592Z", "iopub.status.idle": "2025-04-25T10:33:59.625186Z", "shell.execute_reply": "2025-04-25T10:33:59.624597Z" }, "lines_to_next_cell": 0 }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGZCAYAAADPQ+U8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAABOgElEQVR4nO3deVxU1f8/8NcFBQTZIUUQRMWNAhK1hAQkRbOPKJpbWaKVWdkn10Q/ariluabZYqXgVmZquOWGipoLBeSWhguruYHKJoIC9/eHv7lfR2Zg8CIXZl7Px2MeOveee857Zi7znnvvuecIoiiKICIiqmFGSgdARESGiQmIiIgUwQRERESKYAIiIiJFMAEREZEimICIiEgRTEBERKQIJiAiIlIEExARESmCCaiOiYyMhCAICAoKUjqUJyYIAgRBQFxc3BNtHx0dDUEQ0KxZs2qNy1AEBQVBEARERkYqHUqVNGvWDIIgIDo6WulQaoXw8HAIgoDw8PBqrbcm/76YgBR0584dNGjQQPpCvnjxotIhKSonJweRkZGIjIxETk6O0uE8dVlZWZgzZw5eeuklODg4oH79+nBwcICnpyd69+6Nzz//HMeOHVM6zKcuOjoakZGRT/yDpDZKS0uT/q6f5KFP70VF6ikdgCFbv349ioqKpOerVq3C3LlzFYyoZrRu3RoAYG5urrY8JycHM2bMAPDw152NjU1Nh1Zj9u/fj4EDB+L27dvSMgsLCzx48ADnzp3DuXPnsGPHDgCAvg/XGB0djUOHDgFAhUf2LVq0gJmZGaytrWsosidnbGyMRo0aaVyXm5uLoqIiGBkZwdHRUWMZExOTSttwcnJC69at4eTkJCtWJTEBKWjlypUAgI8++ghffvklVq9ejdmzZ8PY2FjhyJ6uf/75R+kQFJWRkYG+ffuioKAAzZo1w/Tp0xEWFiYl3NzcXMTHx2Pbtm348ccflQ22Ftm/f7/SIeisadOmuH79usZ14eHhWL16NZo2bYq0tLQnbmPu3Ll1/gcrT8EpJCkpCSdPnoSNjQ3mz58Pd3d3XLt2Db/99pvSodFTtmLFChQUFMDExASHDh3C8OHD1Y72rK2tERISguXLl+Pff/9VLlCip4wJSCGqo59BgwbBzMwMb731FoCHp+Hk2rp1K4KDg2FjY4OGDRvC29sb8+fPx4MHD3TqxBAXF4cBAwbA2dkZpqamcHBwwMsvv4yoqCiUlpZq3Obxejdv3oyQkBA888wzMDIyUrvgrek8d1BQENzd3aXn7u7uaufEK4o3MTERAwcOhJOTE0xNTdG8eXOMGzcOd+7c0SnWbdu24eWXX4a9vT2srKzg5+eHmJgYtW3Wrl0Lf39/2NraomHDhggICHjiX+QnT54EAPj4+MDV1bXCsg0aNNC6rqysDOvXr0evXr3QqFEjmJiYwNHRESEhIfjpp59knbo7e/YsRo4cCQ8PD5ibm6Nhw4bw8vLC//73P2RnZ1e47d27d7F48WIEBgbCwcEBJiYmcHFxQWBgIBYtWoQbN24A+L+L3arTbzNmzCh3LeTRI4TKOiGUlpZi1apVCA4OhoODA0xNTeHs7IwBAwZUeE3l0U4Zoiji+++/xwsvvAArKytYWlqic+fOWLduXZXevyf16LWjtLQ0XL58GSNHjoS7uztMTU3VOgZU1Anhzp07WLlyJQYOHIjnnnsOdnZ2MDMzg5ubG15//XWcOHHiiWOMj4/HG2+8AXd3d5iZmcHCwgJubm4IDAzErFmzcOXKFd0rE6nG3bt3T7SxsREBiEePHhVFURQvX74sCoIg1qtXT7x+/brWbT/99FMRgBgYGKhx/fjx40UA0sPGxkasV6+eCEAMCAgQp0yZUuH2Y8eOlbYVBEG0sbERjY2NpWXBwcFiXl5ehXGNGzdO2t7W1lY0NjYWP/30U6msqq6DBw9Ky8LCwkQHBwdpnYODg9ioUSPpERYWJpWNiooSAYhubm7i+vXrxfr164sARGtra9HIyEiqw9PTU8zPz68w1unTp4sARCMjI9Ha2lrtvfv222/FsrIycdiwYSIAsV69eqKlpaW03tjYWNyxY4fWz0qbXr16iQBEFxcXsaysrMrbi6Io3rp1SwwICFCL9/H4Q0NDxeLi4nLbBgYGigDUPpNHff7552rvo7m5uWhiYiI9d3JyEpOSkjRum5iYKDZt2lQqa2RkJNrZ2YmmpqbSsiVLloiiKIobNmwQGzVqJH1+FhYWap95o0aNxIyMDKluNzc3EYAYFRVVrt2cnBwxKChI7bOxsbERBUGQlk2YMEFjzKr3Y+rUqWKfPn2kz9rKykrt/Zw+fXrFH4qOVPuTm5tbuXWpqalSe+vXrxcbNmwofQYWFhZq26jqGTZsWLl6VPu46r2wtbVV+wwEQRCXLl2qMb5H/74eFx0drfaempqalnufNH0+2jABKWDdunUiALFly5Zqy7t06SICEOfPn69124oS0E8//STtBK+//rp45coVURQfJrzvvvtONDMzE21tbbVu/+WXX0rbjxw5Urx27ZooiqJYUFAgLlmyREpkgwYN0hqX6g9m0qRJ4s2bN0VRFMWioiIxLS1NKqspAYmi+h9famqq1vdA9Qdibm4umpqaiu+88470RXX37l1x+fLl0pfatGnTtMZqbW0tGhsbi3PmzBFzcnJEURTFK1euiD169BABiJaWluL06dPFBg0aiN9++6149+5dURRF8cKFC2KHDh1EAKKrq6tYWlqqNVZNIiMjpdc5btw4saCgoErbl5SUSF+aPj4+4vbt26XYCgoKxNWrV4vPPPOMCEAcM2ZMue0rSkA//PCD9DnOmTNH2gdKSkrEhIQEMTg4WEqejyf3jIwM6UdE06ZNxQ0bNkhxlZWViX///bcYGRkprlu3Tud4HlVRAurfv78IQDQxMRGXLVsmtXvt2jVxxIgR0vv9zTffaH0/bG1tRWtrazE6OlosLCwURVEUMzMzxd69e0vJ9MKFCxXGqAtdE1DDhg3FF154Qfzzzz+l9cnJyeXq0ZSAVqxYIX766adiQkKC9COkrKxMTElJET/++GNREATR2NhY4w8JbQno7t270g+woUOHipcuXZLWFRQUiAkJCeLEiRPFnTt36vxeMAEpoGvXriIAcebMmWrLv//+exGA2KZNG63baktAZWVlYsuWLUUAYvfu3TX+slbtWJq2LywsFO3s7EQA4pAhQzS2vWzZMmn7hIQEjXGpvlQrUl0JSNsfnyiK0lHY40n+8Vhnz55dbn1ubq5oYWEhlXn8C1MURfHSpUvS+iNHjlT4eh+XlZUlNmnSRNrewsJC7Nmzpzht2jQxJiZGvHHjRoXbr1mzRtpPVInzcQkJCaIgCKKJiUm5+rR94efl5UlH5rt379ZY74MHD0RfX1+1IxmVoUOHigBEe3t7tSOXyshNQCdOnJDeyxUrVmjcVpWgHBwcxHv37mlsH4B44MCBctsWFRVJn5em/aWqdE1Abm5uGo/gH69H299ART788EMRgPj222+XW6ctAcXHx0v764MHD6rcpia8BlTDUlJSEBcXB0EQ8Oabb6qtGzhwIBo0aIB//vmnyvd/nDx5EpcuXQIATJkyBYIglCszbNgwrdcc9u3bJ3UJ1naD4gcffCB1+dTWO8vIyAiTJk2qUuxyTJ06VePyPn36AAAuXbqEwsJCjWXMzMwwZsyYcsutrKzQuXNnAICrqytef/31cmVatGiBli1bAgBOnz5dpZgdHBzw+++/o3v37gAeXjPZvXs3Zs2ahb59+6JRo0bo0KEDoqOjUVZWVm571fXD999/X2uXZF9fX3h6euL+/fs4ePCgTnFt3rwZOTk5eP7559GjRw+NZerVq4chQ4YAAPbs2SMtv3v3Ln7++WcAQEREBJo2bapTm9VB1a6LiwveeecdjWVmzZoFAMjOzsa+ffs0lvH390fXrl3LLTc1NZXej6p+1nKMHj0aDRs2fCp1v/rqqwCA33//XedtVB1l7t+/j1u3blVLHExANSwqKgqiKKJLly7l7jS2srJC3759Afzfl4yukpKSAAD169eHn5+fxjKCICAwMFDjuoSEBAAPu4+2atVKYxljY2MEBwerlX9cy5Yt8cwzz1Qp9idlZ2cnJYHHNWnSRPq/ts4I7dq1g4WFhcZ1qns4OnTooDGZP1pGW/0VcXd3x969e3Hu3DnMmzcPffr0UftxkJiYiOHDh+OVV15Ru1estLRUuoAcGRmJxo0ba30kJycDANLT03WK6ejRowCA8+fPV1jvzJkzy9WbkJCABw8eAAB69+5d5fdDDtW+2LVrVxgZaf5Ka9u2LZydndXKP+6FF17Q2oZqf3r0vq2nzd/fX9b2KSkpmDBhAnx9fWFjYwNjY2Opg0OvXr0AoEodBlq0aIE2bdrgwYMHeOGFF/D555/j5MmTWjsm6YL3AdWgsrIyqQePqtfb44YNG4affvoJGzduxNKlS3X+BZSVlQUAsLe3r/AmNtUf4eNu3rxZ4XoVFxcXtfKPq6nkAwCWlpZa19Wr93+7tuqL8Um216WMtvp10bZtW7Rt21Z6fv36dWzduhVz585Feno69u7di6lTp2LhwoUAHn4BFhcXA9A98Wk7Anzc1atXAQBFRUVqSU+Xeh+958XNzU2n9qpLVfbdf//9V+u++7Q/66qS87f066+/YsiQIdK+Ajz8gWtmZgZBEHD//n3cuXMHd+/e1blOY2NjbNiwAWFhYUhNTUVERAQiIiJgbm4OPz8/9OvXD8OGDSt3g3lFeARUg/bs2SP94njnnXc0DsHRs2dPAEBBQQE2btxY5Ta0/VqvKfp+E+3T1rhxY7z33nuIj4+XvoBWrVolnYp79Nfmrl27ID68jlvhQ9cx31R1Dxo0SKd6H+0irfR+p4+e9G/p1q1bCA8PR3FxMYKDgxEXF4fCwkLk5ubixo0buH79On755Zcnqtvb2xv//PMPNm/ejJEjR+LZZ5/FvXv3EBsbiw8++ABt2rTBmTNndK6PCagGVfW0WlXKq4b0yM7Oxv3797WW03Zjo+rLrrJDctX6mjzSMUSNGjWSrmPduXNH7QhX9Wtc11NrumrcuPET16va9km3l4P7rrrffvsNeXl5sLW1xfbt2xEYGFjufjJtozTowsTEBP369cOKFStw5swZZGVl4dtvv4WdnR0yMzMxbNgwnetiAqohWVlZ2LZtGwBg06ZNyM/P1/r4448/AADHjh2TzuNXpn379gAeniLQ1oFBFEUcPnxY47oOHToAePhHeuHCBY1lSktLpQvaHTt21Cmuqnj0/L2o5+Of6eLR06+mpqYAHl7j69SpEwBg+/bt1dqe6ppDYmIirl27VqVtO3ToIJ36rWpcqs/9ST9z1b578OBBjZ02gIfDP6l+fD2Nfbc2yczMBPBwzEVtp8NiY2OrrT17e3u89957+PzzzwEAf/31l86dFJiAasjatWvx4MEDWFtbo3fv3mjYsKHWR8eOHdGmTRsAuh8F+fj4SBfk582bp/GPed26dVp/nXbv3h329vYAtPeCW7FihXSdQNUTqjpZWVlJ/9fn0bCPHDlS6XWZgoICbNmyBcDDDguPDtUzcuRIAA9/6VY2dFNVLpoPGDAANjY2ePDgAcaNG1dhQigrK1P7jMzNzTF48GAAD/c/1ZegLlSf+5N+5qp2//33X/zwww8ay0yfPh3Awx6I3bp1e6J26gpVz8gLFy5ovJZ38uTJJxpj8NHrSZo8epSlrTPI45iAaogqkfTp00enkW4HDBgAAFizZg1KSkoqLS8IgjSS9J49ezBs2DC1i8orV67Ee++9B1tbW43bN2jQQEo8P/30E0aNGiUNmVJYWIhly5ZJXZYHDRoEX1/fSmOqKhsbG+lCclRUlE6vuy5aunQpXF1d8dFHHyE2NhZ5eXnSury8PGzcuBF+fn7Sj4Xx48erbT906FB069YNoigiLCwMs2fPlj5r4GGX6IMHD+LDDz9E8+bNdY7LxsYGX3zxBQBgw4YNePXVVxEfHy8dVZSVleH8+fNYtGgRPD09pdG6VebMmQMHBwfcunUL/v7+2LhxI+7duwfg4dHN2bNnMXHiRKxdu1Ztu2effRbAw4T6JGPfderUCf379wfwcGDf5cuXSwn++vXrePfdd6VrHrNmzYKZmVmV26hLQkJCYGRkhNu3b+ONN96Q3tP79+9j48aNCAkJqbDDhTYbNmyAv78/VqxYgZSUFGl5aWkp9uzZg4iICABA586dtX7PlFMtdxNRhY4fPy7dXLZ9+3adtjl9+rS0TUxMjLS8sqF4xowZozbchq2trTQqQHBwsDh58mQRgNijRw+N2z8+FI+tra00AgIAsWvXrpUOxVMZVV2P34gqiqI4a9YstWE+mjZtKrq5uamNvlDRUCEqFd3Uqkusutzkp+sNlI8bPHiwFJvqYWlpKY0ioXoYGRmJkyZN0lhHbm6u+J///EetvJWVVbnhZ+rVq1fluL/55hu1oXdMTU1Fe3t7aT9SPTTdoJuYmCg6OztLZYyNjUV7e3vRzMxMWvb4DawXLlyQ1hsZGYmNGjUS3dzcRDc3NzEzM1MqV9lQPI/eUFqvXj3R1ta2SkPxVPQ5VmX/royuN6JWdDP2o/Vo2kcnTZqk9llZW1tLn5+7u7u4fv16ad3jtP19PXoD+KP7xaPDNjVp0kQ8f/68zu8Fj4BqgOroRzXKsS6ee+45qXtuVTojLFmyBFu2bEFQUBAsLS1RXFyMtm3bYsGCBdizZ4/U7VLbXDuLFy/GgQMH0L9/fzRq1AgFBQWwtLRE165dsWrVKuzbt++Jfj3pasqUKVi6dCk6dOiA+vXr48qVK0hPT5d10bS2Wbt2LQ4cOIDJkyfj5ZdfhouLC+7fv4+ioiLY2tqiY8eOGDt2LP766y/MmzdPYx1WVlbYvn07fvvtNwwaNAiurq4oLi5GYWEhnJ2dERISgrlz5+p8DfFRo0aNQnJyMiZMmABvb2+YmpoiJycHDRs2RIcOHfDRRx9h3759Gk/Dtm/fHufPn8e8efPw4osvwtLSEvn5+XB0dERQUBAWL15c7sZeDw8PHDx4EKGhoXB0dMStW7eQnp6O9PR0nY+Cra2tsX//fqxcuVLa9wsKCtC4cWP0798fBw8exIIFC6r8XtRV8+bNw5o1a9CpUyc0aNAADx48QMuWLTFlyhT89ddfavfJ6So0NBRr1qzB8OHD4e3tDWtra+Tm5sLS0hKdOnXCrFmz8Pfff0uXD3QhiCKv9hoSf39/HDt2DDNnzsS0adOUDoeIDBiPgAzIoUOHpB5yqvuNiIiUwgSkZz788ENER0fj+vXrUi+mnJwcrFixQrqvJDg4WO+7ohJR7cdTcHrGx8cHp06dAvDw3hFzc3Pk5ORIyahdu3bYu3dvpcOWEBE9bUxAembbtm2IiYlBfHw8bty4gdzcXFhZWcHT0xP9+vXDyJEjqzRWExHR08IEREREiuA1ICIiUgSnY6hhZWVluHr1KiwtLTmCMBHpBVEUkZ+fjyZNmug8DA/ABFTjrl69WqOzRRIR1ZTMzExpzjBd1PoEdO/ePcydOxcbNmxARkYG7Ozs0LNnT8yaNavKPblOnDiBuXPn4ujRoygoKICrqysGDBiAKVOmaJwZMzIyUhpfTZNJkyZpvVNdG9UoApmZmWqDbxIR1VV5eXlo2rRplUdJqdUJqKioCMHBwThx4gScnJzQp08fpKWlISoqCjt27MCJEyd0Hmxx/fr1GDZsGEpLS9G+fXu4ubkhMTERn332GXbs2IEjR45oTQj+/v4ap35+kgE5VafdrKysmICISK9U+bKCzqPGKeB///ufCEDs3LmzmJ+fLy1ftGhRlQYGzMzMlAY7XLlypbS8uLhYHDJkiAhAHDlyZLntVAMQahr88Enl5uaKAMTc3Nxqq5OISElP+r1Wa3vB3b9/H8uXLwcAfPXVV2qTc40bNw5eXl44dOgQEhMTK60rOjoaRUVF6N69O0aMGCEtNzExwfLly2FpaYlVq1bpPIkSERHJV2sT0NGjR5Gbm4sWLVrg+eefL7f+tddeA6Db7IuqJBUUFFRunZ2dHby8vFBSUoKdO3fKC5qIiHRWa68BqYaTUU01/TjV8tOnT1dal2oKAm2TJKlmAlW1+bgDBw7g5MmTKCoqgouLC1555ZWnMiEbEZEhqbUJKCMjAwC0dulTLdc2xfSjHB0dKyybmppa4frHZ3CcNm0a+vfvj+joaLVTg5oUFxerTWX76OyXRESGrNaegisoKAAAreOWqbpN5+fnV1pXQEAAgIdTTd+/f19tXUJCAs6cOaOxrpYtW2LhwoX4+++/UVBQgMzMTKxfvx7Ozs7YvHkz3nzzzUrbnjt3LqytraUH7wEiInqo1iag6vTGG2/AxcUFGRkZCA0NxdmzZ5Gfn4+9e/eif//+qFfv4YHg43fwDh06FOPHj0e7du1gYWEBFxcXvP766/jzzz9hb2+PmJgYnDhxosK2J0+ejNzcXOmRmZn51F4nEVFdUmsTkOrUVmFhocb1qus6utz41LBhQ+zYsQMuLi7Ys2cPnnvuOVhZWaFHjx4wMTHB+PHjAWi/RvQ4JycnDB8+HACwe/fuCsuamppK9/zw3h8iov9Ta68Bubq6AgCuXLmicb1quZubm071eXt7Izk5GRs3bkRSUpJ0Q+rgwYMxd+5cAICnp6fO8Xl4eAAArl27pvM2RET0f2ptAvL29gYAJCUlaVyvWu7l5aVznebm5ggPD0d4eLjactU01Zq6aWtz584dANA4hA8REVWu1p6C8/f3h7W1NS5fvoyTJ0+WW79p0yYAQO/evWW1c/r0aRw6dAienp7w9/fXaRtRFPHrr78C0N5NnIiIKlZrE5CJiQlGjx4NAPjwww+laz4AsHjxYpw+fRqBgYFq9+MsX74cbdq0weTJk8vVd/LkSZSUlKgtO3/+PPr37w9RFPHll1+qrcvKysJXX31VrmdcQUEB3n//fcTHx6Nx48bo16+f7NdKRGSIau0pOACYOnUqYmNjcezYMXh4eKBLly5IT09HfHw8HB0dsWrVKrXy2dnZSE5O1nhdZsyYMTh37hy8vb3h6OiIzMxMHD9+HIIgYMWKFejatata+bt372L06NGIiIhAx44d4eTkhKysLCQlJeHWrVuwsbHBpk2bOL01EdETqrVHQABgZmaGgwcPYtq0aTA3N0dMTAzS09MRHh6OpKQknUfCBh52qW7Xrh1OnTqFTZs2ISUlBYMGDcKff/6Jd999t1x5e3t7TJo0Cb6+vrhw4QI2b96Mo0ePonHjxhg/fjzOnj2r8yk7IiIqTxBFUVQ6CEOSl5cHa2tr5Obmsks2EemFJ/1eq9Wn4KjmZWRkIDs7W+kw6hQHBwfptgEi0p2sBJSWloZmzZpVUyiktIyMDLRt0xqF94qUDqVOMW9ghvP/JDMJEVWRrATUsmVLdOvWDe+++y769OkjDWlDdVN2djYK7xVh3QdA2yZKR1M3nL8KDP26CNnZ2UxARFUkK2O0aNECe/fuxb59++Do6Ijw8HC8/fbb0igBVDe1bQK0d1c6CiLSd7J6wSUnJ+PgwYMYMmQI8vLyMH/+fLRp0wbBwcHYsGFDuZGniYiIVGR3ww4MDMS6detw9epVLF26FM8++yzi4uLwxhtvoEmTJhg3bhzOnz9fHbESEZEeqbb7gGxsbPDRRx/h1KlTOHHiBEaMGIH79+9LSalLly5Yu3at2uRsRERkuJ7KjaidOnXChAkTMHjwYIiiCFEUcfToUYSHh8PV1RXLly9/Gs0SEVEdUq0JqKioCGvXrkVAQADatWuHH374AY0bN8aUKVMQGxuLd955BwUFBfj4448xa9as6myaiIjqmGpJQKdPn8ZHH32EJk2aIDw8HEePHkXXrl2xceNGZGRkYPbs2QgODsaKFStw7tw5PPPMM/juu++qo2kiIqqjZHXD/uGHH/D9998jISEBoijC3t4eY8eOxXvvvae1K7abmxu6d++O9evXy2maiIjqOFkJaOTIkQAAPz8/jBo1CgMGDICpqWml23l5eSEzM1NO00REVMfJSkAffvgh3nvvPTz77LNV2m7ChAmYMGGCnKaJiKiOk5WAHp/EjYiISFeyOiHcuHED27ZtQ2pqqtYyqamp2LZtG27evCmnKSIi0jOyEtDixYsRFhaGoiLtoyffu3cPYWFhWLp0qZymiIhIz8hKQLt27YKnpyfatm2rtUy7du3g6emJnTt3ymmKiIj0jKwElJ6ejlatWlVazsPDAxkZGXKaIiIiPSMrAZWWlupUThAEjgFHRERqZCWg5s2b4/jx4ygpKdFapqSkBMePH+dkXUREpEZWAurduzeuX7+OiIgIiKKosczkyZNx/fp1hIaGymmKiIj0jKz7gMaPH481a9ZgyZIl2LdvH95++220aNECAHD58mWsXLkSZ8+eRePGjTFx4sRqCZiIiPSDrARkZ2eHvXv3IiwsDGfOnMHYsWPV1ouiiFatWmHz5s1wcHCQFSgREekXWQkIANq2bYu///4bW7ZsQWxsrDTGW9OmTdGtWzf069cPxsbGsgMlIiL9IjsBAYCxsTEGDBiAAQMGVEd1RERkAJ7KjKhERESVqZYjIADIz8/H5cuXkZ+fr7VHXEBAQHU1R0REdZzsBHT27FmMGTMGcXFxWhOPiq43rhIRkf6TlYAuXryIl156CXl5efD398e1a9eQmpqKwYMHIyUlBUlJSSgpKUFoaChsbGyqKWQiItIHsq4BzZ49G/n5+YiKisKRI0fQpUsXAMD69etx/Phx/P3333jppZdw7tw5LF68uFoCJiIi/SArAR04cABt27bFsGHDNK5v2bIltm7diqysLEybNk1OU0REpGdkJaCbN2+iXbt20vP69esDgNr8QDY2NggKCsKOHTvkNEVERHpGVgKys7NTG+Xazs4OwMNpGh7HGVGJiOhRshKQu7u7WrLx8fGBKIr4+eefpWXZ2dmIi4vjaNhERKRGVgIKCQnB2bNnpSTUu3dvODg4YObMmRg8eDDGjx+Pjh07Ijc3FwMHDqyWgImISD/I6ob95ptvori4GDdu3ICbmxssLCywYcMGDBw4EBs3bpTKde/eHf/73/9kB0tERPpDVgJq0aIF5s6dq7YsODgY6enpOHLkCO7cuYNWrVrB19dXVpBERKR/ZCWgbdu2oX79+njllVfUlltYWKBnz56yAiMiIv0m6xpQWFgYli1bVl2xEBGRAZGVgBwdHWFra1tdsRARkQGRlYCCgoLwxx9/VDoIKRER0eNkJaBZs2YhOzsbY8eOVRv9gIiIqDKyOiH89NNP6NWrF7788kts2LAB3bp1g6urK8zMzMqVFQSB48EREZFEVgKKjIyEIAgQRRE3b97Ejz/+qLUsExARET1KVgKKioqqrjiIiMjAyEpA2qZhICIiqoysTghERERPigmIiIgUIesUXPPmzXUuKwgCLl++LKc5IiLSI7ISUFpaWjWFQUREhkbWKbiysjKNj9LSUqSlpeG7776Dk5MTJk6ciLKysuqKmYiI9ICsIyBtBEGAq6sr3nnnHXTo0AGdO3dGy5Yt8e677z6N5oiIqA566p0QfHx80KlTJ3z55ZdPuykiIqpDaqQXnIODAy5dulQTTRERUR3x1BPQ7du3cfToUdjY2DztpoiIqA6RdQ3o8OHDWtcVFBTgwoUL+Oabb5CVlYVRo0bJaYqIiPSMrAQUFBQEQRAqLCOKIgIDAzFv3jw5TRERkZ6RlYDeeustrQnIxMQETk5OCAwMRNeuXeU0Q0REekhWAoqOjq6mMLS7d+8e5s6diw0bNiAjIwN2dnbo2bMnZs2aBWdn5yrVdeLECcydOxdHjx5FQUEBXF1dMWDAAEyZMgUWFhZat4uOjsbXX3+Nc+fOwcTEBC+++CKmTp0KPz8/uS+PiMhg1eqx4IqKihAcHIxZs2ahoKAAffr0QdOmTREVFYXnn38eKSkpOte1fv16vPTSS9i2bRvc3NzQq1cvFBcX47PPPoOfnx/y8vI0bjdmzBgMHz4cZ8+eRbdu3dCpUyfs27cPAQEBiImJqaZXSkRkeGQloBs3bmDbtm1ITU3VWiY1NRXbtm3DzZs3q1z/7NmzceLECXTu3BkXLlzAzz//jPj4eCxatAhZWVkYMWKETvVcuXIF77zzDkpLS7Fy5UokJiZiy5YtuHjxIoYMGYLTp09j4sSJ5baLjY3F0qVLYW9vj1OnTiEmJga7d+/G4cOHYWxsjOHDhyMnJ6fKr4uIiGQmoMWLFyMsLAxFRUVay9y7dw9hYWFYunRpleq+f/8+li9fDgD46quv0LBhQ2nduHHj4OXlhUOHDiExMbHSuqKjo1FUVITu3burJS0TExMsX74clpaWWLVqFW7dulXu9QHA1KlT4eHhIS3v3LkzRo0ahZycHKxcubJKr4uIiB6SlYB27doFT09PtG3bVmuZdu3awdPTEzt37qxS3UePHkVubi5atGiB559/vtz61157DQCwffv2SutSJamgoKBy6+zs7ODl5YWSkhK1GO/du4cDBw6otfWk7RMRUXmyElB6ejpatWpVaTkPDw9kZGRUqe5Tp04BANq3b69xvWr56dOnK63r7t27AABbW1uN6+3t7dXaBIDk5GQUFxfD0dERLi4ustonIqLyZCWg0tJSncoJgoDi4uIq1a1KWJq+/B9dnp6eXmldjo6OFZZVXcN6dH1l7VtYWMDGxgZ37txBfn5+pTEQEZE6WQmoefPmOH78OEpKSrSWKSkpwfHjx+Hq6lqlugsKCgAA5ubmGteruk3r8uUfEBAAAPjpp59w//59tXUJCQk4c+ZMuboqa1/XGIqLi5GXl6f2ICIimQmod+/euH79OiIiIiCKosYykydPxvXr1xEaGiqnKVneeOMNuLi4ICMjA6GhoTh79izy8/Oxd+9e9O/fH/XqPbwdysio+nulz507F9bW1tKjadOm1d4GEVFdJOsbd/z48WjSpAmWLFkCHx8fLFu2DDt37sTOnTuxbNkyeHt7Y/HixWjcuLHGbs4VUfV6Kyws1LhedV3H0tJSp7p27NgBFxcX7NmzB8899xysrKzQo0cPmJiYYPz48QDUrxFV1r6uMUyePBm5ubnSIzMzs9J4iYgMgayREOzs7LB3716EhYXhzJkzGDt2rNp6URTRqlUrbN68GQ4ODlWqW3XK7sqVKxrXq5a7ubnpVJ+3tzeSk5OxceNGJCUlobS0FO3bt8fgwYMxd+5cAICnp6fO7d+9exc5OTmwtbWtMAGZmprC1NRUpxiJiAyJ7BlR27Zti7///htbtmxBbGys9Au/adOm6NatG/r16wdjY+Mq1+vt7Q0ASEpK0rhetdzLy0vnOs3NzREeHo7w8HC15ceOHQOg3k27devWMDU1RVZWFv79999yw/48SftERPR/qmVKbmNjYwwYMAADBgyojuoAAP7+/rC2tsbly5dx8uRJ+Pj4qK3ftGkTgIfXoeQ4ffo0Dh06BE9PT/j7+0vLGzRogODgYOzatQu//PILxowZ81TaJyIyVLV2LDgTExOMHj0aAPDhhx9K11uAhyMUnD59GoGBgfD19ZWWL1++HG3atMHkyZPL1Xfy5MlyvfXOnz+P/v37QxRFjVOGjxs3DsDDIYEuXrwoLT9+/DhWrFgBGxsbvP322/JeKBGRgZKVgLZs2YL27dtj//79WsvExsaiffv22Lp1a5Xrnzp1Kl544QUcO3YMHh4eGDRoEF588UWMHz8ejo6OWLVqlVr57OxsJCcn49q1a+XqGjNmDJo0aYLu3bvj9ddfR5cuXfDcc88hLS0NK1as0DhlRLdu3fDxxx/j1q1b8PHxQd++fdGrVy8EBASgpKQEUVFRnOmViOgJyUpAUVFRSE9Px0svvaS1TJcuXZCWllYuWejCzMwMBw8exLRp02Bubo6YmBikp6cjPDwcSUlJaN68uc51DR06FO3atcOpU6ewadMmpKSkYNCgQfjzzz/x7rvvat3uiy++QFRUFNq2bYt9+/bh+PHj6NatGw4fPoy+fftW+TUREdFDgqjtBh4duLq6omXLltKYadoEBwfj8uXLOo1aoO/y8vJgbW2N3NxcWFlZKR2OmqSkJPj6+iJxNtDeXelo6oakVMB36sPxBrUNG0Wk7570e03WEdDNmzfRpEmTSss5OTk90XQMRESkv2QlIBsbG50GGc3MzFSbToGIiEhWAurUqROOHz8ujaWmyZkzZ3D8+HF07NhRTlNERKRnZCWgDz74AKWlpXj11Vel+2IetWnTJrz66qsoKyvDBx98IKcpIiLSM7JuRO3ZsyfGjh2LJUuWYNCgQbCxsZF6pqWkpCAnJweiKOK///0v/vOf/1RLwEREpB9k34i6aNEirFmzBq1bt8adO3eQmJiIxMRE3LlzB23atMHq1avxxRdfVEOoRESkT6plKJ6hQ4di6NChuHbtmtpYcE5OTtVRPRER6aFqSUAqTk5OTDpERKSTWjsWHBER6bdqOQL6/fffsXXrVly8eBH5+fkaZ0cVBKHCMeOIiMiwyEpAoiji7bffxurVq6WkIwiCWgJSPRcEQV6kRESkV2Sdgvv2228RHR0NX19f7Nu3D/369QMAJCcnY9euXQgPD4eRkREmTpyIlJSUagmYiIj0g6wjoOjoaFhYWGDXrl2wt7fHunXrAAAeHh7w8PBAjx490KtXLwwaNAh+fn46T59NRET6T9YR0Pnz5+Hn5wd7e3sAkE6zlZaWSmVee+01+Pr6YuHChXKaIiIiPSMrAZWVlUnJBwDMzc0BAHfu3FEr5+HhUeF4cUREZHhkJSBnZ2dcvXpVeq46xfbXX3+plbtw4QLq1avWW46IiKiOk5WA2rdvj3Pnzkmn3EJCQiCKIj755BP8888/yM/Px4IFC5CYmIjnn3++WgImIiL9ICsBhYaGIjs7Gzt37gQAeHt7Y/DgwTh16hQ8PT1hY2ODiIgI1KtXD3PmzKmWgImISD/IOi82ZMgQ9OvXT+302urVq+Hl5YWYmBjcuXMHrVq1wieffIJOnTrJDpaIiPSH7Aszpqamas/r16+PiIgIREREyK2aiIj0mGJjwQ0fPpwdE4iIDJiig5FqGjOOiIgMAw9BFDJo0CDUr19f6TDU5ObmAgDGrgOszRUOpo7ILXz479ixY2Ftba1sMEQKefDgwRNtxwSkkJ9//hlWVlZKh6EmKSkJvr6+WDIUaO+udDR1Q1Iq4DsVWLJkCdq3b690OESKyMvLe6IfYJwPiIiIFMEEREREimACIiIiRTABERGRIpiAiIhIEUxARESkCMUSkL29PVxdXZVqnoiIFKZYAlq4cCFSU1OVap6IiBRWpRtRZ86c+cQNCYKAadOmPfH2RESkX6qUgCIjIyEIQrkx3ARBkP6vWvf4MiYgIiJ6VJUSUFRUVLllx48fx3fffQcXFxe89tpraNasGQAgPT0dmzdvRkZGBkaOHInOnTtXS8BERKQfqpSAhg0bpvb8jz/+wKhRoxAREYGZM2eWm15h/vz5mD59OhYtWoThw4fLj5aIiPSGrE4I06dPR4sWLfDZZ59pnNvH2NgYc+bMQcuWLTF9+nQ5TRERkZ6RlYDi4+Ph7e1daTlvb2/88ccfcpoiIiI9IysBlZaWIiUlpdJyKSkpKC0tldMUERHpGVkJqFOnTvjjjz+wZs0arWXWrFmD+Ph4dOrUSU5TRESkZ2RNSDdjxgwcPnwYw4cPR3R0NAYNGgQ3NzcAD3vBbdy4EXFxcahXrx5mzJhRLQETEZF+kJWA/P39sWXLFowYMQJxcXE4dOiQ2npRFGFvb4+VK1fC399fVqBERKRfZE/J/Z///AcpKSnYtGkTjhw5gqtXrwIAnJyc0KVLFwwYMAANGzaUHSgREekX2QkIABo2bIjw8HCEh4dXR3VERGQAOB0DEREpoloS0N69exEWFgZnZ2eYmppixIgR0ro9e/Zg3Lhx0qk5IiIioBoS0Mcff4xXXnkFW7duRX5+Ph48eKC23snJCV988QV+/vlnuU0REZEekZWA1qxZgy+//BK+vr5ISkpCXl5euTJeXl5o2rQptm/fLqcpIiLSM7I6IXzzzTewsbHBzp074ejoqLWcl5cXzpw5I6cpIiLSM7KOgM6ePQs/P78Kkw8AWFtb48aNG3KaIiIiPSP7GtCjE89pc/XqVTRo0EBuU0REpEdkJSAPDw8kJSWV63jwqPz8fJw8eRKenp5ymiIiIj0jKwENGDAA165dQ0REhNYykydPRm5uLgYPHiynKSIi0jOyOiGMGTMGGzZswBdffIFjx46hT58+AIDLly9jyZIl+PXXX/H777+jffv2ePfdd6slYCIi0g+yElCDBg0QGxuL8PBw7Nq1S5p07siRIzhy5AgAoHv37li3bh1MTEzkR0tERHpD9lhwjo6O2LlzJ06dOoW9e/ciLS0NZWVlcHFxQffu3TkPEBERaVQtg5ECD6fd1mV6biIiIoCDkRIRkUJkJaAffvgBdnZ22L17t9Yyu3btgp2dHaKjo+U0RUREekZWAtqwYQNMTU0REhKitUxISAhMTEzw448/PlEb9+7dw/Tp09GqVSuYmZmhSZMmGDFiBP79998q17Vv3z68+uqrcHR0RP369WFvb4+QkBD8+uuvGssHBQVBEAStj4oSLxERVUzWNaBz587By8sLRkba85ixsTG8vb3x999/V7n+oqIiBAcH48SJE3ByckKfPn2QlpaGqKgo7NixAydOnEDz5s11quuLL77A2LFjIQgCOnfujKZNmyIzMxOxsbHYt28fpkyZgjlz5mjctn///hpndXV2dq7yayIioodkJaDbt2/DwcGh0nIODg7Izs6ucv2zZ8/GiRMn0LlzZ+zdu1dKAosXL8b48eMxYsQIxMXFVVpPVlYWIiIiUL9+fezbtw+BgYHSusOHDyMkJARz587F22+/rTGhLVy4EM2aNaty/EREpJ2sU3AODg64ePFipeUuXrwIW1vbKtV9//59LF++HADw1VdfqR2BjBs3Dl5eXjh06BASExMrrSs+Ph7FxcUIDg5WSz4AEBAQgB49ekAURSQkJFQpRiIienKyEtBLL72ExMTECo9C4uLikJCQAH9//yrVffToUeTm5qJFixZ4/vnny61/7bXXAECneYZMTU11atPe3r5KMRIR0ZOTlYDGjRsHQRDQt29fLFy4ELm5udK6vLw8LFy4EGFhYTAyMsLYsWOrVPepU6cAAO3bt9e4XrX89OnTldbVqVMn2NjY4MCBAzh06JDausOHD2PPnj3w8PBAly5dNG6/cuVKfPDBBxg9ejSWLVuGjIyMqrwUIiLSQNY1oE6dOmHRokUYN24cJk2ahEmTJsHOzg7Aw+tDKgsWLKjyEZDqS97FxUXjetXy9PT0SuuytrbGypUr8frrr6Nr167w8/ODi4sLrly5gmPHjsHf3x9r1qzROlzQ7Nmz1Z5PmDAB06ZNw7Rp0yptu7i4GMXFxdJzTbPGEhEZItk3on788cc4ePAgevToATMzM9y6dQu3bt2CmZkZevbsiYMHD2LcuHFVrregoAAAYG5urnG9hYUFgIfTPeiiX79+2LVrF+zt7XH06FH8/PPPOHr0KCwtLRESEqKxR1tAQADWrl2Ly5cvo7CwEMnJyZgzZw7q1auH6dOnY+nSpZW2O3fuXFhbW0uPpk2b6hQvEZG+q5aREAICAvDbb78hLy8P169fx/Xr15Gfn4+dO3ciICCgOpqQbdGiRejWrRsCAgJw+vRpFBQU4PTp0wgODsb06dPRr1+/ctvMnDkTQ4cORfPmzdGgQQO0atUKU6ZMQUxMDAAgMjIS9+7dq7Bd1XQUqkdmZubTeHlERHVOtQ7FY2xsjGeeeQbPPPNMhfcG6ULV662wsFDj+rt37wIALC0tK60rLi4OEyZMgI+PD3755Rc899xzsLCwwHPPPYdNmzbBx8cHO3fuxK5du3SKLSQkBB06dEBOTg7i4+MrLGtqagorKyu1BxER1eKx4FxdXQEAV65c0bhetdzNza3SutauXQsAUoeIRxkbG0tHP4cPH9Y5Pg8PDwDAtWvXdN6GiIj+j+wEdO7cOYSHh0unqYyNjTU+6tWrWn8H1cjaSUlJGterlnt5eVValypZWVtba1yvWn7nzh2d41OVVV2LIiKiqpHVC+748ePo1q2bdB3Ezs4OjRs3rpbA/P39YW1tjcuXL+PkyZPw8fFRW79p0yYAQO/evSutSxWTthtN//zzTwDQebSDrKwsacI9bd3EiYioYrKOgCZPnox79+5hzJgxyM7ORnZ2NlJTU7U+qsLExASjR48GAHz44YfSNR/g4VA8p0+fRmBgIHx9faXly5cvR5s2bTB58mS1uvr27QsAWL9+PXbs2KG2buvWrfjxxx9hZGSEsLAwafmxY8cQExOD0tJStfJpaWkICwvD3bt3ERoaqrWbOBERVUzWEVBCQgJ8fHywePHi6opHzdSpUxEbG4tjx45JN4qmp6cjPj4ejo6OWLVqlVr57OxsJCcnl7su07dvXwwYMAC//PILevfujQ4dOsDd3R2pqanSUdGcOXPQunVraZsLFy5g+PDhaNy4Mdq3bw8bGxukp6cjMTERRUVF8PT0xPfff/9UXjcRkSGQdQRkYmKCNm3aVFcs5ZiZmeHgwYOYNm0azM3NERMTg/T0dISHhyMpKUnnkbAFQcDPP/+MlStXIiAgAJcuXcKvv/6KtLQ09OrVC7t27cKUKVPUtnnhhRfw/vvvo0mTJvjzzz+xceNGnD17Fj4+Pli0aBH+/PNPPPPMM0/jZRMRGQRBFEXxSTcODQ1FRkYGTp48WY0h6be8vDxYW1sjNze31nXJTkpKgq+vLxJnA+3dlY6mbkhKBXynAomJibweSAbrSb/XZB0BffbZZ7h06RK++uorOdUQEZEBknUNKCkpCcOHD8d///tfbNy4Ed27d4eLi4vWm1DfeustOc0REZEekZWAwsPDIQgCRFHEkSNHcOTIEQiCUK6cKIoQBIEJiIiIJLIS0PTp0zUmHCIiosrISkCRkZHVFAYRERmaWjsWHBER6TcmICIiUoSsU3Aqv//+O7Zu3YqLFy8iPz8fmm4tEgQB+/fvr47miIhID8hKQKIo4u2338bq1aulpKPqFaeies7OCkRE9ChZp+C+/fZbREdHw9fXF/v27ZPm1UlOTsauXbsQHh4OIyMjTJw4ESkpKdUSMBER6QdZR0DR0dGwsLDArl27YG9vj3Xr1gF4OFmbh4cHevTogV69emHQoEHw8/PTafI4IiIyDLKOgM6fPw8/Pz/Y29sDgHSa7dEpDF577TX4+vpi4cKFcpoiIiI9IysBlZWVSckHAMzNzQGUn1nUw8MDZ86ckdMUERHpGVkJyNnZGVevXpWeq06x/fXXX2rlLly4UOUpuYmISL/JSkDt27fHuXPnpFNuISEhEEURn3zyCf755x/k5+djwYIFSExMxPPPP18tARMRkX6QlYBCQ0ORnZ2NnTt3AgC8vb0xePBgnDp1Cp6enrCxsUFERATq1auHOXPmVEvARESkH2SdFxsyZAj69eundnpt9erV8PLyQkxMDO7cuYNWrVrhk08+QadOnWQHS0RE+kP2hRlTU1O15/Xr10dERAQiIiLkVk1ERHqMY8EREZEimICIiEgRVToF17x58yduSBAEXL58+Ym3JyIi/VKlBJSWlvaUwiAiIkNTpQRUVlb2tOIgIiIDw2tARESkCCYgIiJSRLUO0JaTk6N1RlQAcHV1rc7miIioDpOdgK5fv46pU6di27ZtuHXrltZygiCgpKREbnNERKQnZCWga9euoWPHjrh69SqcnZ3h6OiImzdvonPnzkhJScGNGzcgCAI6d+6M+vXrV1fMRESkB2RdA5o9ezauXr2KmTNnIjMzE6+88goEQcDRo0dx7do1xMXFoU2bNhAEAbt27aqumImISA/ISkC7d++Gu7s7pk6dqnF9QEAA9u7di7/++guzZs2S0xQREekZWQno33//hY+Pj/Tc2NgYAFBcXCwtc3Z2RteuXbFx40Y5TRERkZ6RlYCsrKzUntvY2AB4mJgeZWZmVm4ZEREZNlkJyNXVFRkZGdLzZ599FgDw22+/ScsKCwtx9OhRODk5yWmKiIj0jKxecMHBwVi6dCmysrLg6OiI0NBQWFhYYOLEibhy5QqcnZ2xbt063LhxA++//351xUxERHpAVgJ64403kJmZiXPnziEwMBB2dnZYsWIFhg8fjvnz50MQBIiiCE9PT07JTUREamQlIG9vb/z0009qy4YMGQJ/f3/s3LkTOTk5aNWqFUJDQ3kfEBERqanWoXgA4M6dO7C0tMSoUaMgCEJ1V09ERHqiWgYj3bZtG0JCQtCwYUM4ODjAwcEBlpaWCAkJwdatW6ujCSIi0jOyEpAoihgxYgTCwsIQGxuLwsJCWFtbw9raGoWFhYiNjUW/fv0QHh6udYBSIiIyTLIS0NKlSxEdHQ0nJyd88803yMnJwe3bt3H79m3k5ubi22+/hZOTE9auXYulS5dWV8xERKQHZCWg7777Dubm5jhy5Ajee+89tRtTLS0tMXLkSBw5cgQNGjTAd999JztYIiLSH7I6IaSmpiIkJATu7u5ay7i7u+Pll1/G3r175TRFRHomIyMD2dnZSodRpzg4OOjVvGqyEpCjoyNMTEwqLVe/fn04ODjIaYqI9EhGRgZat22LosJCpUOpU8zMzZF8/rzeJCFZCSgsLAzr1q3DnTt3YGtrq7HM7du3ceDAAbzxxhtymiIiPZKdnY2iwkK0/WIOzFs2VzqcOqHwUgrOj/kfsrOzmYCAh/MBHTt2DMHBwVi0aBGCg4PV1h88eBATJkxAixYt8Nlnn8kKlIj0j3nL5rB8tq3SYZBCqpSAHk8wAGBiYoLExER0794ddnZ2cHNzA/DwEFs1RfeLL76Ivn37Yv/+/dUQMhER6YMqJaC4uDit60RRxK1bt6Sk86jjx49zVAQiIlJTpQSUmpr6tOIgIiIDU6UEpDq9RkREJFe1jAVHRERUVUxARESkCCYgIiJSBBMQEREpggmIiIgUwQRERESKYAIiIiJFMAEREZEimICIiEgRTEBERKQIJiAiIlIEExARESlC1oR0VHWiKAIA8vLyFI6kvIKCgof/FgF5nClZJwVF///fgoJa+ZnWVqp9rbSwECX5BQpHUzeU/v/py2vjvqaKR/X9pitBrOoWJMuVK1fQtGlTpcMgIqp2mZmZcHFx0bk8E1ANKysrw9WrV2FpaclJ+nSUl5eHpk2bIjMzE1ZWVkqHQ3qM+9qTEUUR+fn5aNKkCYyMdL+yw1NwNczIyKhKvxDo/1hZWfFLgWoE97Wqs7a2rvI27IRARESKYAIiIiJFMAFRrWdqaopPP/0UpqamSodCeo77Ws1iJwQiIlIEj4CIiEgRTEBERKQIJiAiIlIEExARESmCCYiIiBTBBERERIpgAqJa5/Dhw7hw4UKl5S5evIjDhw/XQESkr7ivKYsJiGqdoKAgfP7555WWmz9/Prp27VoDEZG+4r6mLCYgqpV0uT+a91BTdeC+phwmIKqzrl69ioYNGyodBhkA7mtPB6djoFphzZo1as8vXbpUbplKSUkJkpOTERsbixdffLEmwiM9wn2t9uBYcFQrGBkZVWmCPlEUYWZmhm3btqFbt25PMTLSN9zXag8eAVGtMH36dAiCAFEUMXPmTPj4+KBPnz4ay5qYmKBJkyYICQmBk5NTDUdKdR33tdqDR0BU6zRr1gwDBw7E/PnzlQ6F9Bz3NWUxARERkSJ4Co5qtVOnTuGPP/5AdnY2PD09ERoaCgAoLi5GcXExrKysFI6Q9AX3tZrHbthUKyUnJ8PPzw/t27fHqFGjMHXqVMTExEjrf/zxR9ja2mL37t3KBUl6gfuacpiAqNbJzMxEQEAATpw4gd69e2P+/PnlbgQcOHAgTExMsHnzZoWiJH3AfU1ZTEBU68ycORPZ2dn44YcfEBMTg/Hjx5crY2FhAR8fH8THxysQIekL7mvKYgKiWmf37t3w8vLCiBEjKizXrFkz/PvvvzUUFekj7mvKYgKiWufmzZto3bp1peUePHiAwsLCGoiI9BX3NWUxAVGtY29vj4yMjErLXbhwgTcHkizc15TFBES1jr+/P/7880+cPHlSa5lDhw7h7NmzCAoKqrG4SP9wX1MWExDVOhMmTIAoiujTpw927dqF0tJStfUHDhzAm2++iXr16mHMmDHKBEl6gfuasjgSAtVKX3/9NT7++GOUlZXB3NwchYWFaNiwIYyMjJCXlwdBEPD1119j5MiRSodKdRz3NeXwCIhqpQ8++ABHjhxB7969pYEj8/PzUVxcjB49euDQoUP8QqBqwX1NOTwColpPFEVkZ2ejrKwMDg4OMDY2Vjok0lPc12oWExARESmCp+CIiEgRHA2bFBccHPzE2wqCgP3791djNKTPuK/VLjwFR4ozMqr6gbjqYrEgCOW6zhJpw32tduERECnu+PHjOpcVRRGbN2/G119/jXv37j3FqEgfcV+rXZiASHEvvPCCTuW2bt2KGTNm4NSpUxBFEV26dMGMGTOecnSkT7iv1S5MQFTr7dy5E5GRkUhKSoIoivDz88OMGTPw8ssvKx0a6RnuazWLCYhqrd27d+PTTz9FQkICRFHEiy++iBkzZqB79+5Kh0Z6hvuaMpiAqNbZt28fIiMjceLECYiiiI4dO2LGjBno2bOn0qGRnuG+piwmIKo1Dh48iE8//RRHjx6FKIrw9fVFZGQkXn31VaVDIz3Dfa12YAIixR0+fBiffvopDh8+DFEU4ePjgxkzZqB3795Kh0Z6hvta7cL7gEhxRkZGEAQBDRo0wPvvv4++fftCEASdt/fz83uK0ZE+4b5WuzABkeJUXwqqm/2qQhAElJSUPKXISN9wX6tdeAqOFBcQEFDlLwOiJ8F9rXbhERARESmCo2ETEZEimICoTsnNzUVCQgKuX7+udCikB4KDgzF//vxKyy1cuFDWSNqkGRMQ1Tp79+7FiBEj8Ndff6kt//LLL9G4cWO88MILcHFxwdixYxWKkPRFXFwc/vnnn0rLJScn49ChQzUQkWFhAqJa54cffsAvv/wCDw8PadmZM2cwZswYlJaW4sUXX4SVlRWWLVuGrVu3KhgpGYqioiLUq8c+W9WNCYhqnaSkJPj4+KBhw4bSsujoaOnfo0ePIjExESYmJvj6668VipIMRV5eHo4dOwYnJyelQ9E7TOlU69y4cQMdO3ZUW7Z//37Y2Nhg8ODBAAB3d3cEBgbi3LlzSoRIdVjz5s3Vnm/atAlxcXEay5aUlODGjRsoKSnB6NGjayA6w8IERLWOsbExioqKpOe3b9/G2bNn8Z///EdtRktHR0dkZWUpESLVYWlpadL/BUFAQUEBCgoKNJatX78+mjRpgtDQUMydO7eGIjQcTEBU6zRr1gzHjh3DgwcPUL9+fWzZsgWiKJYbGv/WrVuwt7dXKEqqq8rKyqT/GxkZITw8HKtWrVIwIsPFBES1zqBBg/C///0PAQEB8PPzQ1RUFOrXr4++fftKZURRRGJiIlq3bq1coFTnRUVFoWXLlkqHYbCYgKjW+e9//4utW7ciPj4e8fHxMDIywoIFC+Ds7CyVOXDgALKysjBq1CgFI6W6btiwYUqHYNA4FA/VSmVlZTh8+DCysrLg4+Oj1iUbeHj/xqlTp9C7d+9yF5WJqqqwsBAJCQm4du0aiouLtZZ76623ajAq/ccEREQGbfr06ViyZAkKCwu1llGNnl1aWlqDkek/noKjWu/ixYvIzs6Gvb09WrVqpXQ4pEfmz5+P2bNnw9jYGK+++ipatWoFS0tLpcMyGDwColqpuLgYM2bMwHfffYc7d+4AeHi+XtVbad26dVi8eDFWrVoFHx8fBSOluszDwwNXr17FkSNH0L59e6XDMTgcCYFqnXv37iEoKAiff/45TExM0KtXLzz+Oyk4OBinTp3Cxo0bFYqS9EFmZiYCAwOZfBTCBES1zvz58xEfH48RI0YgJSUF27dvL1emSZMmaNeuHWJjYxWIkPRF48aNYWFhoXQYBosJiGqdn3/+Ga6urvjmm29gZmamtVzr1q2RmZlZg5GRvhk8eDDi4uJw9+5dpUMxSExAVOukpqaiQ4cOlY4+bGJiIl0fInoSkZGRaNu2LUJDQ3Hp0iWlwzE47AVHtU6DBg10SiypqamwtbWtgYhIX/Xq1QtlZWWIi4tD27Zt4ebmBhcXF7UxB1UEQcD+/fsViFJ/MQFRrePj44OEhARkZWXB0dFRY5nU1FT89ddfCAkJqeHoSJ88Ogp2aWkpUlJSkJKSorGsIAg1FJXhYAKiWufdd99FXFwchgwZgg0bNsDBwUFtfU5ODkaMGIEHDx5g5MiRCkVJ+iA1NVXpEAwa7wOiWun111/Hhg0b0LBhQ/j5+WHv3r3w8PBAmzZtcOjQIeTl5eGtt96SJqojorqHCYhqJVEUsXDhQixYsADZ2dlq66ytrfHJJ58gIiKCp0WI6jAmIKrVSktLkZSUhLS0NJSVlcHFxQUdO3aEiYmJ0qGRHjl37hy+//57/PHHH8jOzkafPn0wf/58AMCxY8eQkJCAoUOHws7OTuFI9QuvAVGtZmxsjI4dO5abohsAbt68icWLF2PevHkKREb6YvHixYiIiEBJSQmAh50NHj/qHjt2LExNTfHee+8pEaLe4n1AVOdkZmbio48+gru7OxYsWKB0OFSH7dy5ExMmTEDTpk2xZcsW3Lx5s9ywT35+fnB0dMTWrVsVilJ/8QiIaoWysjJs2LABe/bswc2bN/HMM8/glVdewcCBA6V7MjIzMzFjxgysXbtW+rUaFhamZNhUxy1evBgWFhbYt29fhfNK+fj4IDk5uQYjMwxMQKS4kpIS9OrVC/v371f79blu3Tr88ssv2Lx5M1avXo3Ro0ejsLAQoiiib9++iIyMhJeXl4KRU12XmJiIF198sdJJDR0cHHDkyJEaispwMAGR4r766ivExsbCzMwM4eHh8PT0RH5+Pnbt2oWYmBiMGjUK33//PURRREhICObNm8cpGKha3L9/X6f5f27evFnp0FBUdXxHSXEbNmyAsbExDh06pNbZICIiAu+//z5WrFgBQRCwYMECjB8/XsFISd+4u7vj1KlTFZa5f/8+Tp8+zckQnwJ2QiDFnT9/Hn5+fhp7uk2cOBEA0KZNGyYfqnahoaFIS0vD4sWLtZaZP38+srKy0K9fvxqMzDDwCIgUl5+fj2bNmmlc5+7uDgDw9vauwYjIUHzyySdYv349Jk6ciPj4eKlTy40bN/Drr7/i119/xfr16+Hu7o7Ro0crHK3+YQIixYmiCGNjY43rVCMdVDQvENGTsrW1RWxsLF577TX88ssv2LRpEwBg9+7d2L17N0RRRLt27RATE6PTtSKqGiYgIjJorVq1wsmTJ7F9+3bs3btXbdSN7t27o3///lp/IJE8HIqHFGdkZPTEY7oJgiDdE0REdQuPgKhWeNLfQfz9RFR38QiIiAxGRkYGAMDZ2RnGxsbSc125uro+jbAMFhMQERkMIyMjGBkZ4dy5c2jVqlWVTv/ydG/14yk4IjIYAQEBEAQB5ubmas9JGTwCIiIiRXAkBCIiUgQTEBERKYIJiIgM1g8//AA7Ozvs3r1ba5ldu3bBzs4O0dHRNReYgeA1ICIyWN26dcPff/+Nf//9V5r48HGlpaVwdnaGl5cX9u7dW8MR6jceARGRwTp37hy8vLy0Jh8AMDY2hre3N86dO1eDkRkGJiAiMli3b9+Gg4NDpeUcHByQnZ1dAxEZFiYgIjJYDg4OuHjxYqXlLl68CFtb2xqIyLAwARGRwXrppZeQmJiIuLg4rWXi4uKQkJAAf3//mgvMQDABEZHBGjduHARBQN++fbFw4ULk5uZK6/Ly8rBw4UKEhYXByMgIY8eOVTBS/cRecERk0JYuXYpx48ZJz+3s7AA8vD6ksmDBArUyVD2YgIjI4B0+fBjz5s3DoUOHcO/ePQBAgwYNEBQUhEmTJiEgIEDhCPUTExAR0f9XWlqKW7duAXjYQaGi7tkkHxMQEREpgumdiIgUwfmAiMhgBAcHQxAErF69Gi4uLggODtZ5W0EQsH///qcYneHhKTgiMhiqGVDPnz8vzYiqK0EQUFpa+hSjMzw8AiIig5GamgoAcHZ2VntOymACIiKDkZ6ejsaNG6NevYdffW5ubgpHZNjYCYGIDEZQUBDmzZsnPQ8ODsb8+fMVjMiw8QiIiAyGIAgoKyuTnsfFxaFZs2bKBWTgeARERAbDzs5Op9GvqWbwCIiIDMZLL72Ebdu2oWvXrnB3dwcA/P777xgxYkSl2wqCgJUrVz7tEA0Ku2ETkcFISUlB//79cerUqSpvy27Y1Y8JiIgMiiiKSEtLQ2ZmJoKCgtCzZ09MmjRJp20DAwOfcnSGhafgiMigCIIAd3d36RRc48aNmVgUwk4IRGQwmjdvrna0ExUVhXfeeUfBiAwbExARGYy0tDRkZWVJz0eMGMGOBQpiAiIig9GgQQPk5ORIz0VRBC+DK4fXgIjIYLRp0wb79u3D6tWrpWtA169fx+HDh3XanjOjVi/2giMig/HTTz/hzTfflI56RFGEIAg6b89u2NWLR0BEZDCGDBkCDw8PbN++HZmZmYiOjkaLFi3g7++vdGgGiUdARGSwjIyMEB4ejlWrVikdikHiERARGayoqCi0bNlS6TAMFo+AiIhIEeyGTUQGY8SIEVpPt23btg0nT57UuO7TTz+Fr6/vU4zMMDEBEZHBiI6Oxu+//65xXd++fbFs2TKN6zIyMrQmJ3pyTEBERKQIJiAiIlIEExARESmCCYiIiBTBBERERIpgAiIiIkXwRlQiMhhGRkZVGnz0cRyMtHpxKB4iMihP+ptbTuIizXgEREREiuA1ICIiUgQTEBERKYIJiIiIFMEEREREimACIiIiRTABERGRIpiAiIhIEf8P9g7tUp1vJLoAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Each trial is a row in a dataframe that contains\n", "# Algorithm, Number of Samples, Number of Features, Hyperparameters, Score, Runtime, Memory Usage, Step as features\n", "trials = est1.completed_trials_summary_[est1.completed_trials_summary_[\"Step\"].str.contains('Model Selection')]\n", "name_of_score_column = f\"Score ({est1._inferred_score_metric[0].name})\"\n", "trials.replace([np.inf, -np.inf], np.nan, inplace=True)\n", "trials.dropna(subset=[name_of_score_column],inplace=True)\n", "colors = []\n", "scores = trials[name_of_score_column].tolist()\n", "models = trials['Algorithm'].tolist()\n", "y_margin = 0.10 * (max(scores) - min(scores))\n", "s = pd.Series(scores, index=models).sort_values(ascending=False)\n", "s = s.dropna()\n", "for f in s.keys():\n", " if f.strip() == est1.selected_model_.strip():\n", " colors.append('orange')\n", " elif s[f] >= s.mean():\n", " colors.append('teal')\n", " else:\n", " colors.append('turquoise')\n", "\n", "fig, ax = plt.subplots(1)\n", "ax.set_title(\"Algorithm Selection Trials\")\n", "ax.set_ylim(min(scores) - y_margin, max(scores) + y_margin)\n", "ax.set_ylabel(est1._inferred_score_metric[0].name)\n", "s.plot.bar(ax=ax, color=colors, edgecolor='black')\n", "ax.axhline(y=s.mean(), color='black', linewidth=0.5)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "1c950833", "metadata": {}, "source": [ "\n", "#### Model Tuning\n", "\n", "Model Tuning is the last stage of AutoML, and focuses on improving the chosen algorithm's score on the reduced dataset (after Adaptive Sampling). AutoML uses a novel algorithm to search across many hyperparameters dimensions, and converge automatically when optimal hyperparameters are identified. Each trial represents a particular hyperparameters configuration for the selected model." ] }, { "cell_type": "code", "execution_count": 12, "id": "7236cdb7", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:59.627391Z", "iopub.status.busy": "2025-04-25T10:33:59.626981Z", "iopub.status.idle": "2025-04-25T10:33:59.833480Z", "shell.execute_reply": "2025-04-25T10:33:59.832905Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAFNCAYAAACgxXBBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8ekN5oAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXnElEQVR4nO3dd1hT1/8H8HdYCUtmEGQpCA4UFK11IWoFtY4q7rpw1g53tVqrbbXWljpK7XRrrevrHlXcC3EgyiiKk6G4WAHZJOf3hyY/IgkkucEE+Lyeh+cp9557zifH23y49557Do8xxkAIIYToAQNdB0AIIYRIUVIihBCiNygpEUII0RuUlAghhOgNSkqEEEL0BiUlQggheoOSEiGEEL1BSYkQQojeoKRECCFEb1BSIjVe165dwePx8M0332i13m+++QY8Hg9du3bVar36Jjk5GTweDzweD8nJyboORy+cPXtW1ifaJq337NmzWq+7NqCkRGSkX8LSnx07dlR5TJ8+feSOqetfag0bNpTrD3V+tJ1UazPpHyKa/ISGhuo6fFIJI10HQPTXxo0bMXz4cKX709PTERER8RYj0n9CoRBFRUUVthcWFiI3NxcAYG9vD0NDwwplLCwsqj0+RYyNjdGkSRPZf9cEtra2qF+/foXtJSUlyM7OBgDY2NjAxMSkQhkrK6sq6zczM5P1CXm7KCmRCuzt7VFYWIiTJ0/i0aNHcHFxUVhuy5YtEIvFaNiwYZ2/QpK6du2awu2bNm3CuHHjZGUaNmz4FqOqnLOzM27fvq3rMNSyd+9ehdvPnj2Lbt26ycpoeuu1Xbt2Na5Pagu6fUcqMDc3x+DBgyGRSLBp0yal5TZu3AgAdDuEEKI1lJSIQtK/6pUlpYsXL+LOnTvw8PBAly5dqqyvqKgIP//8Mzp27AgbGxsIBAK4u7tjzJgxuHnzZqXHisVirF69Gv7+/jA3N4etrS26du2K3bt3q/x5IiMjMWrUKLi7u0MgEMDKygrt2rXDjz/+iJcvX6pcT3VQdaCB9HnVm/8mbx7/7NkzTJ8+HY0aNYJAIED9+vUxfPhwpX/5V9b+mw/87927h/Hjx8PV1RV8Ph8uLi6YNGkSHj9+XOlnjI+Px7Bhw+Do6AiBQAAPDw9MnToVz58/r9ZBBW8qPyimtLQUK1asQNu2bWFtbS03+KCymCQSCU6dOoVp06ahffv2cHFxgYmJCezs7BAYGIg///wTpaWlGsWXnZ2NRYsWwd/fH/Xq1YOJiQkcHR3h6+uLKVOm4NSpU1w+fs3ACHnt66+/ZgCYu7s7k0gkzNPTkwFg586dq1B2/PjxDABbvHgxO3PmDAPAALCHDx9WKPvo0SPWokULWRljY2NmZWUl+93AwID98ssvCmMqKipiPXv2lCtrbW3NeDweA8C++OILFhgYyACwr7/+usLxYrGYTZs2TXY8AGZhYcEMDQ1lvzdp0oQlJycr7Y/AwEB1u7KCjRs3Ku2jhw8fVtp/Uu7u7gwA27hxo9LjDx8+zBwcHBgAZmZmxvh8vmxfvXr12M2bNyvUW1n75f9tT58+zSwsLBgAZmlpyYyMjGT7GjRowB49eqQw7r179zJjY2O5/hcIBAwAc3JykusbrsrHe+bMmQr7pefKF198wTp27MgAMCMjI2ZjY8N4PJ7smPL1vKl8f0k/T/nzGQALCAhgBQUFCmNUFl9aWhpzc3OTO9dtbGzkzlVtnIv6jq6UiELlRylt2LBBbl9+fj527doFAwODKm/dicViDBo0CAkJCbCyssLWrVvx8uVL5OTk4P79++jbty8kEgmmT5+Oo0ePVjh+/vz5iIiIAI/Hw3fffYfs7GxkZ2fj6dOn+Pjjj/Hjjz9WeqX19ddf45dffoGDgwN+++03ZGZmIi8vD4WFhThz5gxat26NpKQkhISEQCKRqNtNemf06NHw8vLCtWvXkJ+fj5cvX+LEiRNwcnJCbm4upk6dqnHdgwYNQvfu3XHr1i3k5uYiPz8fO3fuhKWlJdLT0zF//vwKxzx48ACjRo1CaWkp/P39ER0djby8PBQUFODEiRMwMTHBrFmzuHxkjfz222+Ii4vDxo0bkZubi6ysLLx48QK+vr5VHmtkZISRI0fi4MGDsvMpJycHeXl52LhxIxo0aIALFy5gwYIFasX0zTffIDU1FQ0bNsTJkydRUlKCrKwsFBcXIzk5GX/88Qfat2+v6UeuOXSdFYn+KH+lxBhjqampzMDAgJmbm7O8vDxZuQ0bNjAALCgoiDHGKr1S2rFjh2xfREREhTZLS0vZu+++ywCwFi1ayO17/Pix7K/xhQsXKox5xIgRsvrfvFJ6+PAhMzQ0ZKampgqvEBhjLDc3l7m4uDAAbN++fQr7oyZdKTVt2lThX+gHDx6UlUlLS1O5/fL/tt26dWNisbhC3b/88gsDwExNTVlpaancvgkTJjAAzMHBgWVmZlY49vbt23JXc1ypeqUEgB08eFCletR17do1BoCZm5uzwsLCCvuVxdesWTMGgG3btk3tNmsTulIiSrm6uqJHjx6yKyMp6QCH8ePHV1nHzp07AQAdOnRAcHBwhf1GRkb4+uuvAQAJCQmIj4+X7du9ezfKyspgamqKzz//XGH9lb3bs2nTJojFYvTq1Qt+fn4Ky1haWmLAgAEAUCuGt8+ePRumpqYVtvfu3Vs2PLp8H6vjyy+/hIFBxa+MDz74AMCrYe93796VbWeMYc+ePQCAjz/+GLa2thWObdKkCYYOHapRPFz4+PigX79+1VJ327Zt4eDggPz8/Cqfl5ZnbW0NAHjy5Em1xFVTUFIilZIOeJDewrt37x4uXLgAGxsb2Zd5ZaKjowEAPXr0UFqmW7dusvd2pOXL/3fbtm1Rr149hcd6e3vD2dlZ4b7IyEgAwPHjx+Ho6Kj0R5pkU1JSqvw8+u7dd99VuN3IyAhCoRAAkJWVpdW6GzRoIPvv8nU/ePAAOTk5AIDAwECl9epixoxOnTpxOr6kpAR//vkngoOD0aBBA/D5fLkXdJ8/fw4AePTokcp19u3bFwAwb948TJ48GceOHZO921aX0HtKpFIDBw6EjY0NIiMjcffuXdnIrxEjRkAgEFR5vPR/TmWJAwAEAgHs7e3x7NkzWXlVjwUAFxcXhaO/0tPTAbx6Bpafn19lrAUFBVWW0XeWlpZK9xkZvfrfXdORYcrqltb7Zt0vXryQ/Xf5xPWmqv59q4ODg4PGxz5//hw9evSQu+KUnsPSP65evHgBiUSi0nknNWfOHMTGxmLXrl1Yu3Yt1q5dCx6PBx8fH/Tq1QsTJ06sEy/00pUSqRSfz8eIESMAAOvWrcOWLVsA/P8VlD4Ti8UAgC+++AKMsSp/aC6y6vM2hnurQ9GMGqqaOXMm4uPjYWdnhw0bNuDJkycoLCzEixcv8PTpUzx9+lSWhBljKtdrbGyMnTt34ubNm1i0aBG6d+8OMzMzJCQkYPny5fDx8cGKFSs0jrumoKREqiRNQD///DMePXqEFi1aoG3btiodK/2LtLLbGEVFRcjMzJQrX/6/q3oHRtl+R0dHAPp/W678lYaiKYqkRCLR2whHa6S3C4H/v2pVpKp/X31SWloqm03i119/xbhx42TnmZRYLEZGRobGbfj5+eHbb7/FqVOnkJOTg5MnT6JLly4Qi8Wyq6najJISqVLbtm3RsmVLlJSUAFBtgEP5YwFU+tLf2bNnUVZWBgB45513KhwbHR2t9AXXu3fvKk140ucGJ0+erPTLXtdsbGxk/52WlqawzJ07d2TPZ2oKDw8P2cP7yq5Ca9IV6osXL2TnUuvWrRWWuXjxotbONyMjI7z33ns4cuQI+Hw+GGM4efKkVurWV5SUiEp+/PFHzJ49G7Nnz8aoUaNUPk46oWtUVBSOHz9eYX9ZWRkWL14MAGjRogVatGgh2zdo0CAYGhqisLAQy5cvV1i/9FhFxo8fDyMjI2RkZMhG+ClTUlKis5kdzM3N4enpCQCy0WpvWrp06dsMSSt4PB5CQkIAAH/++adsotTy7t69KzeyU9/Vq1dPditS0RVLWVmZ2u8nSRUXFyvdx+fzZbccFY2ArE1q96cjWtO7d28sX74cy5cvl7stU5VBgwbJRm0NHToU27Ztkz0Mf/jwIQYNGoSoqCgAQFhYmNyxzs7O+PTTTwEAS5YswbJly5CXlwfg1V+sn332GbZu3ap01mdPT08sXLhQVveYMWOQkJAg219WVoabN29i8eLFaNy4sVrDd7VN+txuw4YN+P3331FYWAjg1ZXTxIkTsXPnTpiZmeksPk3Nnz8fpqamePbsGYKDg3Hjxg0Ar561nD59Gj179qxRn8vCwkJ2BT5r1iycPn1a9tJ1QkIC3n//fURHR8Pc3Fztut3d3TF//nxcvnxZLkHdu3cPI0eOREFBAQwMDNCzZ0/tfBh9pZO3o4heevPlWVWpMs2Qj4+PrIyJiQmztraWm04lPDxcYd2FhYWsR48esrKGhoayKWGgwjRDEomELVy4UFYer1/ytLOzk5u+BQC7ePGiwv6o7pdnGWMsLy+PNW/evMJ0Sng9LdP27dtVenmW68u3lb08WxlpGUUvrP7vf/+Tm5LI0tKSmZmZMQDM2dlZ1jd8Pr/SNlSh6suzis4VZfW8KTo6mpmbm8v28/l8ZmlpKZuyaMuWLUr7mjHlfVX+XJROMSSdjgkA4/F4bNWqVap3Rg1FV0qk2jk7OyM6OhorV65E+/btYWpqioKCAri6umL06NG4fv06pk2bpvBYgUCAo0ePIjw8HK1atYKJiQkYYwgICMCuXbvwww8/VNo2j8fD4sWLERcXh08++QTNmjWDoaEhRCIRbGxs0LFjR8yZMweXLl3i/O4KFxYWFrh48SJmzZqFRo0awcjICMbGxrIrycrWtdJ3gwcPRnR0NIYMGQKhUIji4mLUr18f06dPx40bN2RXutLnT/quTZs2uHr1KoYOHQp7e3tIJBJYWlpi6NChuHTpEkaPHq1RvcePH8f8+fMREBAAV1dX2dVy48aNMW7cOFy7dg0zZszQ4ifRTzzG1BizSAghWrZgwQJ8//336N69e92YBZtUiq6UCCE68+LFC6xbtw4A0KtXLx1HQ/QBXSkRQqrVL7/8goKCAgwePBgNGzaEkZERiouLcerUKcyePRu3b9+GUCjErVu3YGdnp+twiY5RUiKEVKsZM2YgPDwcwKuZFKysrJCbmyt7N83Kygr79+/XyRx4RP/Q3HeEkGo1duxYGBoa4vz583j8+DEyMzNhamqKRo0aoWfPnpg+fbpO5r8j+omulAghhOgNGuhACCFEb9Dtu7dMIpEgPT0dlpaWejdzMiGEaIIxhry8PDRo0IDzNEiUlN6y9PR0uLq66joMQgjRurS0NLi4uHCqg5LSWyZdKC0tLU3paqq1TVbhq9VIbU0rLodNlKN+0wz1m+Y07bvc3Fy4urpWusikqigpvWXSW3b16tWrM0mpzPjV0N96pnXj82oL9ZtmqN80x7XvtPFIggY6EEII0RuUlAghhOgNSkqEEEL0BiUlQggheoOSEiGEEL1BSYkQQojeoKRECCFEb1BSIoQQojcoKRFCCNEblJQIIYToDUpKhBBC9AYlJUIIIXqDkhIhhBC9QbOEE6KHUkUi3M96DgCwEhTD3swMblZWb63tjIIC2e9vs21dt19X25a2r6tzrjxKSoTomVSRCE1+/RVFZWWybQJDQyRNnQo3KyuUiMV4WVICQx4PVgKBrExucTHKJBJYmJjAxNAQAFAqFiOvpAQGPB6sFZQ1NzYG3+jV10CZRILEFy/Qbu1aFIvFsrJ8Q0NcnTQJ3nZ2ELwuK5ZIICouBg+AjamprOzLkhKUiMUwMzauUBYAbMuVzS8pQbFYDFMjI5gaGwMAknNy0PTXX+XaFxgZ4cbkyXCwsIDAyAhmr8syxpBdVAQAsBEIZMsmFJSWorhMDL5hCcxNTGT1ZBUWAgCsBQIYvC5bWFqKwrIy8A0NkVlYWGm/F5WVoaC0FCaGhrAoV292YSEYgHp8Poxer7oqLWtsYABLPl9WNqeoCBLG5MoWl5UhKTNTab83s7eH8et/T+m/vZGBAeqVq1dUVAQxYwr/7ZWdJ+X/7e9nZcHn998r9HvSZ5+99cREt+8I0TMZBQVyX4wAUCQWy/6KPvngAezCwtDj77/lyvT+5x/YhYXhyJ07sm2X0tJgFxaGjuvXy5UdvGsX7MLCsDsxUbbtxpMn8PvzT7kvJgAoFovh9+ef2BIbK9uWlJkJu7AweP/6q1zZjw4fhl1YGP64dk227VFuLuzCwuCycqVc2ZkREbALC8PKqCjZtvtZWRXaLyorw5Lz52EXFobvzp+XbS8sK4NdWBjswsJQUFoq274yKhqNV6/Fl6dOydUjLVv+amTV5cuwCwvDjGPHquz3v6KjYRcWhsmHDsmVafLrr7ALC0NSRoZs29+xsbALC8Ooffvkyrb+6y/YhYUh5skT2bY9t25V2u8XU1Nl247evQu7sDD02rpVrmzQ33/DLiwMJ+7fl207m5wMu7AwdNm0Sa7sBzt2wC4sDAeSkmTbzqekKOz38n31tnBKSsnJyVoKg5C6LSUnB58cOYJhu3frOhRCdIrHGGOaHmxkZIQePXpg0qRJ+OCDD2BkRHcDq5KbmwsrKyuIRKI6s/IsLU9dkVgiwcuSEtltlZScHDQMDwcPwLFRo9Dzjb+EAeD65Mnwd3KChDFIXv9vK70FBLy6/QYABjye7PYUYwxiBWXFEgmYgrLX0tPx7rp1Fdq+MnEi2jZooHG9qpa9np6OtmvXVmj/6sSJaO3kBB4Aw9d1lK/XkMeT3b57kZ8BBsDO1FZWtnz/lC8r7UsegNhnz9BmzZoKbb/Z7+VjUKVeVcpGq9jv6vzbKyurTr9LP3tVtPm9xulKydPTE8ePH8fQoUPh4uKCefPm4e7du5wCIqS22xYfD+eVK/HFyZOybe7W1ljctSv+HTkSnjY2sucxUgIjI9ibmQF49WViZGAg90UDQLbNoNyS1DwlZQ2VlHV8/dzmzbYdLSw41atqWaG5ucL261tYwMjAQO4Lvny95ZfhltZrqKR/ypeV9qWhgQHszcxU6nd161WlrKr9rs6/vbKy6vS79LO/TZyulADg3LlzWLt2Lfbu3YuioiLweDwEBgZi8uTJCAkJgUm5B4KErpTqGrFEgsi0NDSxs0N9CwsAwPH799Fz61Y0sbPDrU8/lfuCkno1EurVcwcrgVWdGwXGpX0u5xuNvtPsnNPm9xrnpCSVk5ODv//+G+vWrUN8fDx4PB5sbGwwZswYTJo0Cc2aNdNGMzUeJaW6ZeDOndh/+zZWBgdjZocOAF6Nijr54AHe8/CQjZRSpC73GxfUb5rTtO/05vZdedbW1pg6dSpiY2Nx+fJljB8/HiUlJQgPD0eLFi0QEBCAv//+G8Wvh4YSUptIGMPF1FTMjohASblRTN0bNoS1QCA3qsvY0BC9vbwqTUiE1FXVMjKhXbt2sLKyAo/Hw7rXD+8iIyNx6dIlfP7551i4cCE+++yz6miaEJ1gjGHwrl14lp+PHh4e6O3lBQCY4O+Pj9q2pQREiIq0mpSKiorwv//9D2vXrkVkZCQYY3B0dMT48ePRvXt37Ny5E1u3bsX06dORnZ2NhQsXarN5Usvowz12Re0/ycvDj5GRuJ+djUMjRgB49fB4fOvWSM/Lg+PrZ0cAZC96EkJUo5VnSnFxcVi7di3++ecfiEQiAEC3bt0wZcoUDBgwQG6oeEpKCtq3bw8jIyOkpaVxbbrGoWdKqlE4q8FbfMO8svZNjYzgtGIFxIzh3tSp8LStnmcX9GxEM9RvmtOHZ0qcrpTWrVuHtWvXIjo6Gowx2NnZYebMmfjoo4/g9fr2xZvc3d0RFBSEf/75h0vTpJZT+Hb96zfM18fE4MnLl/i8Y0d429kBeDUbwR/R0fC0scEXnTvLjll6/jxSRCJMbdcOLevXBwD89/w5wq9cgWu9elgYGCgr+1NkJO5mZWFymzYw4PGUtu/v5ISl3bvDx8EBLnXkDwtC3hZOSWny5MkAgI4dO2LKlCkYMmQI+OXmY1LG19e3Tl4lEe34X2IibmVkYGTLlrKklJyTg7UxMejo6iqXlPYnJSE6PR0DmjaVJaXHeXlYGxODVo6Ocknp8N27OJ+SgiAPjyqvfsq3QQjRHk5J6dNPP8VHH32EFi1aqHXc559/js8//5xL06QOm/7uu8goKEBDa2vZtuZCIb7r1q3ClcvHbdviSV6eLHkBQGNbW3zXrZvsvSGpia1bI9jDAy0cHFD4xlUSIeTt4JSUVq9era04lCosLMSyZcuwY8cOpKamwtbWFr169cKSJUvg7OysVl2XL1/GsmXLEBkZiZcvX8LNzQ1DhgzBl19+CXNzc5Xq6NGjB069nugxLS0NLi4uan8mUjXp2/VvPtOxNzPDR23bVijfxN4eC7p0qbB9fOvWFbZ52NgoLDvaz0/236kikdL2CSHVh9NAh2fPnuHKlSto2bIlGjVqpLDMw4cPER8fj/bt28PBwUGt+ouKitCtWzdcvnwZTk5OCAgIQHJyMq5evQqhUIjLly/Dw8NDpbr++ecfjB07FmKxGP7+/nB3d8f169eRmpoKX19fXLhwocoHdJs2bcK4cePA4/HAGNMoKdFAB9Xp6+i7t4Ue2GuG+k1z+jDQAYyDuXPnMgMDA5aYmKi0zH///ccMDAzYl19+qXb9CxYsYABYhw4dWF5enmz7ihUrGAAWGBioUj1paWlMIBAwAGz9+vWy7cXFxWzEiBEMAJs8eXKldTx//pzZ2tqy4OBg5u7uzgCwtLQ0tT+TSCRiAJhIJFL72JoqsyCTZRZk6jqMGof6TTPUb5rTtO+0+b3GKSm1bNmStWzZUqVyfn5+atVdXFzMrKysGAAWExNTYb+vry8DwKKjo6usa8mSJQwACwoKqrAvMzOTWVpaMiMjI5aRkaG0jg8//JAJBAJ27949Skpq0vREP/PwIdt04wa7l1k3v2Doy1Uz1G+a04ekxGmaoZSUFHh7e1dZzsvLC6nlFqpSRWRkJEQiETw9PdFawXOBwYMHAwAOvbHgliLXr18HAHTt2rXCPltbW/j6+qKsrAxHjhxRePyxY8ewbds2LFiwAJ6enmp8CsLF+hs3EHrgAHYkJOg6FELIW8IpKYnfWKlQGR6Pp/acd7GvV7n09/dXuF+6PS4ursq68vPzAQA2NjYK99u9HpkVW25lzfLHfvzxx2jatCnmzp1bdeBEa3wdHBDo7o531BzQQgipuTiNvvPw8EBUVBTKysqULvBXVlaGqKgouLm5qVW39MpK2UAC6faUlJQq6xIKhZWWffjwodL9ixYtQnJyMs6ePavRMhzFxcVyCTk3NxfAqweKZcZ1Y9hxdmG2RsdN8G+GCf6vZpeXPoCtSzTtt7qO+k1zmvZdbmGu1mLgdKXUr18/PH36FPPmzQNTMohv/vz5ePr0Kfr3769W3S9fvgQAmCkZgisdwp2Xl1dlXV1eD//dvn07SkpK5PZFR0cjPj5eYV0xMTEIDw/H2LFjEVjuJUt1LFu2DFZWVrIfV1dXjeohhJC6gNOV0uzZs7FlyxasWrUKJ06cwIQJE2TPXO7fv4/169cjISEBjo6OmDNnjlYC1sTIkSPx3XffITU1Ff3798fy5cvh7u6OqKgoTJo0CUZGRigrK4NB+SWDxWJMnDgR1tbWWL58ucZtz58/H7NmzZL9npubC1dXV9ia2qKead0YEi6lzjDT4rIymBgaKlwAr66hoc2aoX7TnLp9Z1Sqvbm9OdVka2uL48ePY+DAgYiPj8fMmTPl9jPG4O3tjT179sDe3l6tui1ev21fUO49kfKkz4ksLS1Vquvw4cPo27cvIiIiEBERIdvXuHFjzJ49Gz/++KPcM6eff/4ZN27cwPr169WOvTw+n6/S1EtE3uJz5/DbtWv4MiAAczt10nU4hJC3hHN6a9asGf777z/s3bsXJ0+elM1p5+rqih49eiAkJASGGqwlI30G9ejRI4X7pdvd3d1Vqs/Pzw9JSUnYtWsXYmJiZC/RDh8+HMuWLQMA+Pj4yMofOnQIPB4PmzdvxpYtW+Tqevr0KQDI5vqbN28eevXqpd4HJJX678ULiIqLaekHQuoYrVxzGRoaYsiQIRgyZIg2qgPwKokAr57rKCLd7uvrq3KdZmZmCA0NRWhoqNz2S5cuAag4ZJwxhvPnzyut7/LlywBQoT7C3a4hQ3A7IwP1VZz+iRBSO1TLyrPa0KlTJ1hZWeH+/fu4efMmWrVqJbd/9+7dAF4NtuAiLi4O586dg4+PDzqVu0109uxZpcc0bNgQKSkpNPddNTIxNITv61m9CSF1B6fRd+Xl5eXh5s2buHDhAs6fP6/wRx0mJiayJdM//fRT2TMkAFi5ciXi4uIQGBiINm3ayLb/+uuvaNq0KebPn1+hvps3b6LsjZmfb926hUGDBoEx9lYmlyWEEFI5zldKCQkJmDFjBs6ePat0WLiUqi/bSn311Vc4efIkLl26BC8vLwQEBCAlJQVXrlyBUCjEhg0b5MpnZGQgKSkJT548qVDXjBkzkJiYCD8/PwiFQqSlpSEqKgo8Hg9//fUXunXrplZspPocv38fF1NTEeThgQAVnxkSQmoHTldKd+/eRefOnXH69Gl06NBBNlP48OHD0a5dO9kLtf3798eYMWPUrl8gEODMmTNYuHAhzMzMsH//fqSkpCA0NBQxMTEqzxAOAKNGjULz5s0RGxuL3bt348GDBxg2bBiuXbuGSZMmqR0bqT4Hk5Kw5Px5HLpzR9ehEELeMk5LV4wdOxZbt27Fhg0bMHbsWIwbNw5btmyRXRHdu3cPEyZMwNOnT3H58mWl0/zUJbR0RdX2JCbi6L17GNy8OXo1blydoek1WoJBM9RvmtOHpSs4XSmdPn0azZo1w9ixYxXub9y4MQ4cOIAXL15g4cKFXJoidcig5s2xrn//Op2QCKmrOCWl58+fo3nz5rLfjV+/U1JUVCTbZm1tja5du+Lw4cNcmiKEEFIHcEpKtra2cpON2tq+uuRTNLHp8+fPuTRF6oiswkK8KDfSkhBSt3BKSo0aNZJLQK1atQJjDDt37pRty8jIwNmzZ9WeJZzUTWuvX4fD8uWYrMI6WYSQ2odTUgoODkZCQoIsMfXr1w/29vZYvHgxhg8fjtmzZ+Odd96BSCTC0KFDtRIwqd2evp4dvqG1tW4DIYToBKf3lEaPHo3i4mI8e/YM7u7uMDc3x44dOzB06FDs2rVLVi4oKAgLFizgHCyp/Vb16oUl3bujTCLRdSiEEB3gNCRcmfz8fFy4cAHZ2dnw9vaWm3WhrqMh4URV1G+aoX7TnD4MCed0pXTw4EEYGxujd+/ectvNzc1p1mxCCCFq4/RMaeDAgfjll1+0FQup447du4cRe/Zga1ycrkMhhOgIp6QkFApplgaiNedTUrAjIQEXU1N1HQohREc43b7r2rUrrl69CsYYLVtNOBvYtCnq8flo4+Sk61AIITrC6UppyZIlyMjIwMyZM+VmcSBEE+84O2Ne584I8vTUdSiEEB3hdKW0fft2vP/++1i9ejV27NiBHj16wM3NDQKBoEJZHo9H898RQgipFKch4QYGBuDxeFWuowS8SkrqrqdUG9GQcMWe5OUhOScHLRwcYMnnv63Q9BoNbdYM9ZvmavyQ8I0bN3JqnBCpA0lJ+PjIEfRu3Bj/jhyp63AIITrCKSkpW7KCEHWVisVwsrCAb/36ug6FEKJDnJdDJ0Qbpr77Lqa++y7ENL0QIXUap9F3hGiboQGdkoTUZZyulDw8PFQuy+PxcP/+fS7NEUIIqeU4JaXk5GQthUHqshP372PB6dPo4+WFr7t21XU4hBAd4nSvRCKRKPwRi8VITk7GmjVr4OTkhDlz5kBCzwqIEtefPMG19HQkZWbqOhRCiI5Vy0AHHo8HNzc3TJw4EW3btkWHDh3QuHFjTJo0qTqaIzXcKF9fNLa1RX1zc12HQgjRsWp/qtyqVSu0a9cOq1evru6mSA3lUq8eBjdvjgB3d12HQgjRsbcy1Mne3h737t17G00RQgipwao9KWVlZSEyMhLW1tbV3RSpgR7n5mJrXBz+e/5c16EQQvQAp2dK58+fV7rv5cuXuHPnDv744w+8ePECU6ZM4dIUqaXOpaRg9L596OTqiovjx+s6HEKIjnFeT6mqdZQYYwgMDMQPP/zApSlSS5kbGyPAzQ0dXFx0HQohRA9wSkpjxoxRmpRMTEzg5OSEwMBAdOvWjUszpBb7oGlTfNC0qa7DIIToCU5JadOmTVoKgxBCCKG574gOMcZUWouLEFJ3cEpKz549w8GDB/Hw4UOlZR4+fIiDBw/iOY2uIm+ITEuDXVgYhu/eretQCCF6glNSWrlyJQYOHIiioiKlZQoLCzFw4ECEh4dzaYrUQnHPniG7qAh5JSW6DoUQoic4LYfu6+sLAIiLi6uynIGBAW7evKlpU7UGLYf+/4rLynArIwMSxuDv5KSL0PQaLeutGeo3zenDcuicrpRSUlLg7e1dZTkvLy+kpqZyaYrUQnwjI7RydKSERAiR4ZSUxGKxSuV4PB6Ki4u5NEUIIaQO4JSUPDw8EBUVhbKyMqVlysrKEBUVBTc3Ny5NkVrm6cuX+PrMGRy4fVvXoRBC9AinpNSvXz88ffoU8+bNUzq0d/78+Xj69Cn69+/PpSlSy0Snp2Px+fP46swZXYdCCNEjnF6enT17NrZs2YJVq1bhxIkTmDBhAjw9PQEA9+/fx/r165GQkABHR0fMmTNHKwGT2qG+uTnGt2oF5zoy2IMQohpOScnW1hbHjx/HwIEDER8fj5kzZ8rtZ4zB29sbe/bsgb29PadASe3yjrMz3nF21nUYhBA9w3nl2WbNmuG///7D3r17cfLkSaSlpQEAXF1d0aNHD4SEhMDQ0JBzoIQQQmo/rSyHbmhoiCFDhmDIkCHaqI7UcmUSCURFRbAzM9N1KIQQPUNz35G3LvHFC9j/9BOa//abrkMhhOgZTklp79698Pf3x6lTp5SWOXnyJPz9/XHgwAEuTZFa5F6W9K1xUx1HQgjRN5yS0saNG5GSkoLOnTsrLRMQEIDk5GRs2LCBS1OkFglp1gy58+bh74EDdR0KIUTPcHqmFBsbCz8/P/D5fKVl+Hw+WrVqRfPeETmWfD4sKzlvCCF1E6crpefPn6NBgwZVlnNycqKlKwghhFSJU1KytrZWaaLVtLQ0WFhYcGmK1BI5RUUYtXcvwiIjaYE/QkgFnJJSu3btEBUVhfj4eKVl4uPjERUVhXfeeYdLU6SWiH/2DP/Ex+PXq1fB4/F0HQ4hRM9wSkqffPIJxGIx+vTpg90KVg/dvXs3+vTpA4lEgk8++YRLU6SWcKlXD0u7d8dn7drpOhRCiB7iNNChV69emDlzJlatWoVhw4bB2toaHh4eAIAHDx4gJycHjDFMmzYNffv21UrApGZrZGODLwMCdB0GIURPcX55dsWKFdiyZQuaNGmC7OxsXL9+HdevX0d2djaaNm2KzZs34+eff9ZCqIQQQmo7rUwzNGrUKIwaNQpPnjyRm/vOiVYUJeUwxnAtPR3NhUJYmJjoOhxCiB7SSlKScnJyokRElEoRifDuunXgGxoib/58GNNEvYSQN2g1KRFSmacvX6K+uTkczM0pIRFCFNJKUrp48SIOHDiAu3fvIi8vT+H7Jzwer9I58kjt197FBU8//xz5JSW6DoUQoqc4JSXGGCZMmIDNmzfLEhGPx5NLStLf6Z0UImVOz5MIIUpwGn33559/YtOmTWjTpg1OnDiBkJAQAEBSUhKOHj2K0NBQGBgYYM6cOXjw4IFWAiaEEFJ7cbpS2rRpE8zNzXH06FHY2dlh69atAAAvLy94eXmhZ8+eeP/99zFs2DB07NgR7u7uWgma1DwlYjE6rl+P5kIhfundG2bGxroOiRCihzhdKd26dQsdO3aEnZ0dAMhu0YnFYlmZwYMHo02bNli+fDmXpkgNdzczG1GPHmF3YiJMjWh8DSFEMU7fDhKJRJaQAMDs9fLW2dnZsLe3l2338vLCkSNHuDRFajjnehbYOXgwREVF9HyREKIUpyslZ2dnpKeny36X3p67ceOGXLk7d+7AiP46rtOsBQIM9fHBpDZtdB0KIUSPcUpK/v7+SExMlN2uCw4OBmMMc+fOxe3bt5GXl4effvoJ169fR+vWrbUSMCGEkNqLU1Lq378/MjIyZLfm/Pz8MHz4cMTGxsLHxwfW1taYN28ejIyMsHTpUq0ETGqmg0n3EP/sGcQSia5DIYToMU731EaMGIGQkBC5W3ObN2+Gr68v9u/fj+zsbHh7e2Pu3LloR0sV1Fk5RUUYd+Doq//+4gtYCQQ6jogQoq84P+jh8/lyvxsbG2PevHmYN28e16pJLZFdWIz2Lk4oKJVQQiKEVIrz0hWaGjduHA1+qCMa2VjhyIeDETtliq5DIYToOZ0lJQAK58gjhBBSd+k0KRFCCCHlUVIi1YoxBv+/NuOD7Xvx7OVLXYdDCNFz9FCHVKsUkQgpolyk572EramprsMhhOg5SkqkWjlbWuL0mGF4lJdHC/sRQqpESYlUK2NDQ/g5OsDP0UHXoRBCagB6pkQIIURvUFIi1eq3q1dx+M59vKQl0AkhKtD7pFRYWIhFixbB29sbAoEADRo0wPjx4/H48WO167p8+TI++OAD2NvbQyAQwNvbGwsWLEB+fr7C8gcPHsTYsWPRsmVL2Nvbw9jYGA4ODnj//fdx+PBhrh+t1isuK8P0Y8cwdv+/EBVTUiKEVE1nScnOzg5ubm6VlikqKkL37t2xZMkSvHz5Eh988AFcXV2xceNGtG7dWq0l1v/55x907twZBw8ehLu7O95//30UFxfj+++/R8eOHZGbm1vhmC1btuDvv/8GALz77rsYNGgQPDw8cPToUfTr1w9ffvmleh+6jnlZUoJRvr7o5OqMBhbmug6HEFITMD22YMECBoB16NCB5eXlybavWLGCAWCBgYEq1ZOWlsYEAgEDwNavXy/bXlxczEaMGMEAsMmTJ1c4LiYmhmVkZFTYfvnyZWZhYcF4PB6Li4tT6zOJRCIGgIlEIrWOq8kyCzJZZkGmrsOocajfNEP9pjlN+06b32s8xlSf62fx4sUaJz8ej4eFCxeqXL6kpAQODg4QiUSIiYmpsB6Tn58f4uLiEB0djTZVLBz33XffYeHChQgKCsLx48fl9mVlZaFhw4YoLCzE06dP5VbSrczEiROxfv16hIeHY9q0aSp/rtzcXFhZWUEkEqFevXoqH1eTZRVmAQBsTW11HEnNQv2mGeo3zWnad9r8XlNrSPg333wDHo9XYc668stbS/e9uU3dpBQZGQmRSARPT0+FCwQOHjwYcXFxOHToUJVJ6fr16wCArl27Vthna2sLX19fREZG4siRIxgzZoxK8RkbGwMATExMVCpfFxWUlsLsdT8RQogq1EpKGzdurLAtKioKa9asgYuLCwYPHoyGDRsCAFJSUrBnzx6kpqZi8uTJ6NChg1qBxcbGAni1uq0i0u1xcXFV1iUdyGBjY6Nwv/TqSNpmVeLj47Fz504YGxsjKChIpWPqIq/Vq8ED8L+h/dHEjv5qJYRUTa2kNHbsWLnfr169iilTpmDevHlYvHhxhaUowsLCsGjRIqxYsQLjxo1TK7DU1FQAgIuLi8L90u0pKSlV1iUUCist+/Dhw0r3Hzp0CHv27EFpaSlSU1Nx6dIlGBsbY+3atfD09Ky07eLiYhQXF8t+lw6oyCrMQplxWZWx11SiomKk5+UBAMyMypBdmK3jiGoe6jPNUL9pTtO+yy2sOFBMU5xG3y1atAienp74/vvvFa6NZGhoiKVLl6Jx48ZYtGiRWnW/fD15p5mZmcL95uavRnPlvf7iq0yXLl0AANu3b0fJG+/LREdHIz4+vtK6YmNjsXnzZmzbtg0XL14En8/H6tWrMXr06CrbXrZsGaysrGQ/rq6uVR5TG1gJ+Hg4fTKOjx4CCxO6hUcIURGXURLW1tbsww8/rLLchx9+yKytrdWqe9KkSQwAW7BggcL9d+/eZQCYl5dXlXXl5eUxFxcXBoD17NmTxcfHs9zcXBYREcHc3NyYkZERA8B69epVaT2FhYUsPj6effLJJwwA69u3LysuLq70mKKiIiYSiWQ/aWlpNPqOqIT6TTPUb5rTh9F3nK6UxGKxSu8KPXjwAGKxWK26LSwsAAAFBQUK90ufE1laWqpU1+HDh+Hi4oKIiAi0bNkS9erVQ8+ePWFiYoLZs2cDUP7MSUogEKBFixb47bffMHXqVBw+fBirV6+u9Bg+n4969erJ/RBCCFGMU1Jq164drl69ii1btigts2XLFly5cgXt2rVTq27pi7WPHj1SuF+63d3dXaX6/Pz8kJSUhI0bN2Lq1Kn45JNPsG7dOty8eVN269HHx0fl+KS37g4cOKDyMXXJt2fP4qfISDxR4fYqIYRIcZol/Ntvv8X58+cxbtw4bNq0CcOGDZMliZSUFOzatQtnz56FkZERvv32W7Xq9vPzAwDExMQo3C/d7uvrq3KdZmZmCA0NRWhoqNz2S5cuAVA8ZFwZe3t7AMCLFy9UPqaukDCG5VFReFlSgj7e3nC0oCUrCCGq4ZSUOnXqhL1792L8+PE4e/Yszp07J7efMQY7OzusX78enTp1UrtuKysr3L9/Hzdv3kSrVq3k9u/evRsA0K9fPy4fAXFxcTh37hx8fHzUilH6WasafVcXlYjFmNepExJevICXrS3ySkS6DokQUkNwnvuub9++ePDgATZs2IDQ0FAEBwcjODgYY8eOxfr16/Hw4UP0799f7XpNTEzw2WefAQA+/fRTuUlTV65cibi4OAQGBsq9OPvrr7+iadOmmD9/foX6bt68ibIy+SHYt27dwqBBg8AYq/Bs6MWLF1i7dq3CZ1onTpzA3LlzAUDtoe51gcDICAu6dMH2QYNoYT9CiFq0ssifhYWFwttiXH311Vc4efIkLl26BC8vLwQEBCAlJQVXrlyBUCjEhg0b5MpnZGQgKSkJT548qVDXjBkzkJiYCD8/PwiFQqSlpSEqKgo8Hg9//fUXunXrJlc+Pz8fkydPxowZM9CmTRu4uLggPz8fd+7cwe3btwEAM2fOxKBBg7T6mQkhpC7T66UrBAIBzpw5g4ULF8LMzAz79+9HSkoKQkNDERMTAw8PD5XrGjVqFJo3b47Y2Fjs3r0bDx48wLBhw3Dt2jVMmjSpQnkHBweEhYWha9euSE1Nxf79+xEREYGioiIMHz4cZ86cwcqVK7X5cWuNB9nZKCgt1XUYhJAaSK0JWZU5fvw4/vjjD1y9ehUZGRkYOXKk7ComIiICERER+Pzzz9GgQQPOAdd0dWFC1ua//YbbGRk4PXYsujZsSBNkaoj6TTPUb5rThwlZOV8pTZ8+Hb1798aBAweQl5eH0jf+QnZycsLPP/+MnTt3cm2K1ABlEglyiorAAHirOOM6IYRIcUpKW7ZswerVq9GmTRvExMQoXCjP19cXrq6uOHToEJemSA1hZGCA9Nmz8XT2bDi9fgGaEEJUxWmgwx9//AFra2scOXJENumpIr6+vrL55UjdUJ8SEiFEA5yulBISEtCxY8dKExIAWFlZ4dmzZ1yaIoQQUgdwHhJefjE/ZdLT02Fqasq1KVIDzI6IwMuSEkx79134ODjoOhxCSA3D6UrJy8sLMTExFQY3lJeXl4ebN2+qNa8cqbm2JyRgTUwMcsutIUUIIarilJSGDBmCJ0+eYN68eUrLzJ8/HyKRCMOHD+fSFKkBGGNY2bMnvuzcGS3oKokQogFO7ykVFhaiffv2SEhIQLt27fDBBx/gyy+/REBAAAYMGIB9+/bh4sWL8Pf3x6VLl2BiYqLN2GukuvCe0pvovRHNUL9phvpNc/rwnhKnZ0qmpqY4efIkQkNDcfToUVy9ehUAcOHCBVy4cAEAEBQUhK1bt1JCIoQQUiXOAx2EQiGOHDmC2NhYHD9+HMnJyZBIJHBxcUFQUJDa6yiRmuva48cwMzaGt50dTcRKCNGIViZkBV6tfyRdA4nUTVOPHsWVx4+xc/BgDKWBLYQQDej1hKykZqnH58PCxIQGORBCNMYpKa1btw62trY4duyY0jJHjx6Fra0tNm3axKUpUgMcHz0aonnz0Oz1qryEEKIuTklpx44d4PP5CA4OVlomODgYJiYm2LZtG5emSA1hwOOp9EI1IYQowikpJSYmwtfXFwYGyqsxNDSEn58fEhMTuTRFCCGkDuCUlLKysmCvwq0ae3t7ZGRkcGmK6Lkphw+j++bNOP3woa5DIYTUYJySkr29Pe7evVtlubt378LGxoZLU0TPnUtJwZnkZJSKxboOhRBSg3FKSp07d8b169dx9uxZpWXOnj2L6OhodOrUiUtTRM/9ExKCDf374x1nZ12HQgipwTglpVmzZoHH42HAgAFYvnw5RCKRbF9ubi6WL1+OgQMHwsDAADNnzuQcLNFf/k5OGNe6NWxpNnhCCAecklK7du2wYsUK5OXl4YsvvoCtrS2EQiGEQiFsbGzwxRdfIDc3F2FhYXSlRAghpEqcX56dPn06zpw5g549e0IgECAzMxOZmZkQCATo1asXzpw5g1mzZmkjVqKnTty/jwO3b+PZy5e6DoUQUsNpZZqhLl26oEuXLhCLxcjMzATwahBEZUPFSe2xPCoKx+/fx5q+fTGpTRtdh0MIqcG0Nvcd8OqdJAeaYqbO8REK8Tw/H60cHXUdCiGkhtNqUiJ108qePXUdAiGkluB8fy0xMRGhoaHw8PCAqakpDA0NFf4YGVH+I4QQUjlOmSIqKgo9evRAYWEhAMDW1haOdAunThFLJDCkZ4eEEC3hlJTmz5+PwsJCzJgxA1999RVsbWn54bpmyuHDOP7gAb7r1g2jaT0tQghHnJJSdHQ0WrVqhZUrV2orHlLDxD1/jlSRCHy6PUsI0QJO3yQmJiZo2rSptmIhNdCxkSOR8Pw5mguFug6FEFILcEpKnTt3piUp6jgbU1MEuLvrOgxCSC3B6Qn1999/j3v37uG3337TVjyEEELqME5XSjExMRg3bhymTZuGXbt2ISgoCC4uLkpnchgzZgyX5oieOZiUhDuZmQj29IRv/fq6DocQUgtwSkqhoaHg8XhgjOHChQu4cOGCwqWwGWPg8XiUlGqZrXFx+N/r27eUlAgh2sApKS1atEhhEiJ1Q/dGjcAAdHBx0XUohJBagscYY7oOoi7Jzc2FlZUVRCIR6tWrp+tw3oqswiwAgK0pvcemDuo3zVC/aU7TvtPm9xq9ik8IIURvUFIiGskuLERhaamuwyCE1DJaeQ3/4sWLOHDgAO7evYu8vDwouiPI4/Fw6tQpbTRH9MB358/j5ytX8E1gIBYGBuo6HEJILcEpKTHGMGHCBGzevFmWiKSj8aSkv9OAiNolWSSChDE4WVrqOhRCSC3C6fbdn3/+iU2bNqFNmzY4ceIEQkJCAABJSUk4evQoQkNDYWBggDlz5uDBgwdaCZjoh91DhiB91iwMbt5c16EQQmoRTldKmzZtgrm5OY4ePQo7Ozts3boVAODl5QUvLy/07NkT77//PoYNG4aOHTvCnaajqTV4PB5dJRFCtI7TldKtW7fQsWNH2NnZAYDsFp1YLJaVGTx4MNq0aYPly5dzaYoQQkgdwCkpSSQSWUICADMzMwBAdna2XDkvLy/Ex8dzaYrokd2JiZhy+DCO3bun61AIIbUMp6Tk7OyM9PR02e/S23M3btyQK3fnzh1aDr0WOXbvHv66fh2X0tJ0HQohpJbhlJT8/f2RmJgou10XHBwMxhjmzp2L27dvIy8vDz/99BOuX7+O1q1bayVgontDfXwwr1MnBHt66joUQkgtw+nypX///ti5cyeOHDmC/v37w8/PD8OHD8eOHTvg4+Pz/40YGWHp0qWcgyX6IdjTkxISIaRacEpKI0aMQEhIiNytuc2bN8PX1xf79+9HdnY2vL29MXfuXLRr145zsIQQQmo3mpD1LavpE7I+ys1FXnExvOzsYKRk3aw30QSZmqF+0wz1m+ZoQlZS42y4cQPNf/8dkw8d0nUohJBaiJISUUthaSnMjY3RwsFB16EQQmohtZ4peXh4aNwQj8fD/fv3NT6e6IdlPXpg6XvvobTcC9KEEKItaiWl5OTkagqD1CQGPB749N4ZIaQaqPXNIpFIqisOQgghhJ4pEdX977//EPT33/gzOlrXoRBCailKSkRlUY8e4eSDB7idkaHrUAghtZRWHwzk5OQoXXkWANzc3LTZHHnLJvn7w0cohA+NvCOEVBPOSenp06f46quvcPDgQWRmZiotx+PxUFZWxrU5okPNhEI0Ewp1HQYhpBbjlJSePHmCd955B+np6XB2doZQKMTz58/RoUMHPHjwAM+ePQOPx0OHDh1gbGysrZgJIYTUUpyeKX333XdIT0/H4sWLkZaWht69e4PH4yEyMhJPnjzB2bNn0bRpU/B4PBw9elRbMRMdeJidjUNJSUgTiXQdCiGkFuOUlI4dO4ZGjRrhq6++Uri/S5cuOH78OG7cuIElS5ZwaYro2IGkJPTfsQPTjx3TdSiEkFqMU1J6/PgxWrVqJfvd0NAQAFBcXCzb5uzsjG7dumHXrl1cmiI6Zm5sDN/69dHGyUnXoRBCajFOz5TenA3W2toawKtkVX5KIoFAgMePH3NpiujYpDZtMKlNG12HQQip5ThdKbm5uSE1NVX2e4sWLQAA//77r2xbQUEBIiMj4UR/YRNCCKkCpyul7t27Izw8HC9evIBQKET//v1hbm6OOXPm4NGjR3B2dsbWrVvx7NkzfPzxx9qKmRBCSC3FKSmNHDkSaWlpSExMRGBgIGxtbfHXX39h3LhxCAsLA4/HA2MMPj4+tBx6Dbbv1i3MjIjAoGbNsKJnT12HQwipxTglJT8/P2zfvl1u24gRI9CpUyccOXIEOTk58Pb2Rv/+/ek9pRoqVSTCsXv3kCIS4W5WFlJFIrhZWek6LEJILaX19Qeys7NhaWmJKVOmgMfjabt68hY9yM6Gz++/o+j1TByH7tzBiQcPkPTZZ5SYCCHVQisTsh48eBDBwcGwsLCAvb097O3tYWlpieDgYBw4cEAbTZBqlF1YiNRyL8UyxtBmzRo0Wb1alpCkisrKkFFQ8LZDJITUEZySEmMM48ePx8CBA3Hy5EkUFBTAysoKVlZWKCgowMmTJxESEoLQ0FClk7SSt+d+VhYO37mD7MJC2baNN27ANiwMUw4flm3j8Xgok0hQRv9mhJC3jFNSCg8Px6ZNm+Dk5IQ//vgDOTk5yMrKQlZWFkQiEf788084OTnh77//Rnh4uLZirnNSRSLEPHki+0mtYqqfJ3l52HzzJrbExspt779jB/pt346r5d4Z87CxAQCIyr3wDABbBgzA4REjtPQJCCFENZyeKa1ZswZmZma4cOECGjVqJLfP0tISkydPRlBQEFq2bIk1a9ZgxowZXJqrk1JFIjT59Ve522gCIyPZc51NN28iMjUVU9q2RZsGDQAA/714gdADB9DEzg5j/Pxkx7V2dISxgQHE5a6AOri6InfePFjy+XLt+jk6wsbUFAIjowpt25uZVdfHJYTUcZyS0sOHDxEcHFwhIZXXqFEjvPfeezh+/DiXpuqsjIICpc913KyssOfWLRy+cwetnZxkSam5UIjujRqhxRvLTGwNCalQv4mhIUxeTw/1JjcrKyR99pncMyR7MzMa5EAIqTackpJQKISJiUmV5YyNjWFvb8+lKaLEMB8ftHZ0RDtnZ9m2BpaWODVmjFbqd7OyoiRECHlrOCWlgQMHYuvWrcjOzobN62cTb8rKysLp06cxcuRILk0RJUb5+uo6BEII0RrO6yl5eHige/fuOH36dIX9Z86cQVBQEDw9PfH9999r1EZhYSEWLVoEb29vCAQCNGjQAOPHj9dogtfLly/jgw8+gL29PQQCAby9vbFgwQLk5+dXKCuRSHDhwgXMnTsXbdq0gaWlJfh8Pjw9PTFlyhQ8fPhQo8+jLnszMwiM5P92oOc6hJDaisfUGKvdvXv3CtuKi4sRFRUFHo8HW1tbuLu7AwBSU1Nly6O3b98eAoEAp06dUiu4oqIidOvWDZcvX4aTkxMCAgKQnJyMq1evQigU4vLly3KzkVfmn3/+wdixYyEWi+Hv7w93d3dcv34dqamp8PX1xYULF+RmPb937x68vLwAAI6OjmjXrh0MDQ1x9epVPH78GJaWlvj333/RuXNntT5Tbm4urKysIBKJKsyyrkyqSFSjn+tkFWYBAGxNbXUcSc1C/aYZ6jfNadp3mnyvKcXUwOPxNP4xMDBQpynGGGMLFixgAFiHDh1YXl6ebPuKFSsYABYYGKhSPWlpaUwgEDAAbP369bLtxcXFbMSIEQwAmzx5stwx9+7dY0FBQezUqVNMIpHIthcVFbHQ0FAGgLm5ubGSkhK1PpNIJGIAmEgkUuu4miyzIJNlFmTqOowah/pNM9RvmtO077T5vabWlVJKSgqnBCi9ilJFSUkJHBwcIBKJEBMTg9atW8vt9/PzQ1xcHKKjo9GminV+vvvuOyxcuBBBQUEVRgFmZWWhYcOGKCwsxNOnT2FnZ1dlbIWFhXBycoJIJMLZs2cRGBio8ufS6l8UNQT95aoZ6jfNUL9pTh+ulNQa6KBOUuEqMjISIpEInp6eFRISAAwePBhxcXE4dOhQlUnp+vXrAICuXbtW2GdrawtfX19ERkbiyJEjGKPCqDVTU1N4e3vj2rVrSE9PV+0DEUIIqZJW5r6rDrGvZyPw9/dXuF+6PS4ursq6pAMZlI0QlF4dxb4xA4IyEolEdtXo6Oio0jGEEEKqprdJSbqirYuLi8L90u2q3FIUvn6JVFlZ6Ug6VW9Pbt++Hc+fP4dQKETHjh1VOoYQQkjVtL50hba8fPkSAGCmZOizubk5ACAvL6/Kurp06YJt27Zh+/btWLx4sdwLv9HR0YiPj1e5rrS0NNl0SYsXLwb/jel53lRcXIzicvPK5ebmAnh177bMuEzZYbVKdmG2rkOokajfNEP9pjlN+y63MFdrMejtlZI2jRw5Ei4uLkhNTUX//v2RkJCAvLw8HD9+HIMGDYLR6/eADAwq7478/HyEhIQgIyMDAwYMwJQpU6pse9myZbKZ062srODq6qqVz0QIIbWR3l4pWVhYAAAKlKzdI31OZGlpqVJdhw8fRt++fREREYGIiAjZvsaNG2P27Nn48ccflT5zAoDS0lIMGTIE0dHR6Ny5M7Zt26bS55g/fz5mzZol+z03Nxeurq6wNbVFPdO6MfpOikZDaYb6TTPUb5pTt++MSrWXSvQ2Kbm5uQEAHj16pHC/dLuqIwL9/PyQlJSEXbt2ISYmRvYS7fDhw7Fs2TIAgI+Pj8JjJRIJxo4di6NHj6JVq1Y4dOgQTE1NVWqXz+dXeYuPEELIK3qblPxeL7kQExOjcL90u68ac7+ZmZkhNDQUoaGhctsvXboEQPGQcQCYOnUqtm/fDm9vb0RERMDa2lrlNgkhhKhOb58pderUCVZWVrh//z5u3rxZYf/u3bsBAP369ePUTlxcHM6dOwcfHx906tSpwv6vvvoKv//+O9zc3HDixAk4ODhwao8QQohyepuUTExM8NlnnwEAPv30U7lJU1euXIm4uDgEBgbKvTj766+/omnTppg/f36F+m7evImyN9YlunXrFgYNGgTGGFavXl3hmFWrVmHp0qVwdHTEyZMnZbcUCSGEVA+9vX0HvLpKOXnyJC5dugQvLy8EBAQgJSUFV65cgVAoxIYNG+TKZ2RkICkpCU+ePKlQ14wZM5CYmAg/Pz8IhUKkpaXJJpL966+/0K1bN7nyN2/exOzZswG8Wqhw6dKlCmOcOHGi2pOyEkIIUUyvk5JAIMCZM2ewbNkybNu2Dfv374etrS1CQ0OxZMkSpS/WKjJq1Chs3boVsbGxyMnJgVAoxLBhwzBnzhy0atWqQvmcnBxIpwWMiopCVFSUwnq7du1KSYkQQrRErQlZCXc0IStRFfWbZqjfNFfjJmQl3En/BpDO7FAXSN/21ua7DHUB9ZtmqN80p2nfSb/PtHGNQ/9qb5l0KiOa2YEQUtvk5eXBiuMCpHT77i2TSCRIT0+HpaUleDyersN5K6SzWKSlpdWZW5baQP2mGeo3zWnad4wx5OXloUGDBlVO11YVulJ6ywwMDNQaoFGb1KtXj74kNED9phnqN81p0ndcr5Ck9PY9JUIIIXUPJSVCCCF6g5ISqXZ8Ph9ff/01TUyrJuo3zVC/aU4f+o4GOhBCCNEbdKVECCFEb1BSIoQQojcoKRFCCNEblJSIygoKCrB//35MmDABTZo0gUAggLm5Ofz8/LB48WK8fPlSrfoaNmwIHo+n9Of27dvV9Enevq5du1b6WY8dO6ZWfdnZ2Zg+fTrc3d3B5/Ph7u6OGTNmICcnp3o+gI6cPXu20n6T/ixevFil+mrbOXf9+nX88MMPCAkJgYuLi+xzVGXTpk1o164dLCwsYGtri/fff1+22Km6xGIxVq1ahZYtW8LU1BRCoRBDhw7FrVu3NKqPXp4lKtu2bRsmTZoEAGjWrBn69++P3NxcXLp0CV9//TW2b9+Oc+fOqb0Q4tixYxVu19bLePpk0KBBsLCwqLDd2dlZ5ToyMjLQoUMH3Lt3Dx4eHhgwYAD+++8/hIeH4+jRo4iKioKtbe2YjNTR0VHp+SEWi7F161YAQEBAgFr11pZzbsmSJThw4IBax8yYMQPh4eEwNTVFcHAwioqKcOLECRw/fhy7d+/GgAEDVK5LIpFgyJAh2LdvH6ytrdGnTx9kZGRg9+7dOHLkCM6cOYN27dqp96EYISratGkTmzx5MktMTJTbnp6ezlq3bs0AsBEjRqhcn7u7O6srp2BgYCADwB4+fMi5rpEjRzIALCQkhJWWlsq2T506lQFgY8eO5dxGTfDvv/8yAMzV1ZVJJBKVjqlt59wPP/zAFi5cyA4ePMiePHnC+Hx+pZ/vxIkTDACzs7Njd+7ckW2/dOkSMzExYdbW1iw7O1vl9teuXcsAMC8vL/b06VPZ9t27dzMArHHjxnLnqCpqz78O0alLly4xAIzP57Pi4mKVjqltXxCV0VZSSk9PZwYGBszExETuS4AxxoqKiphQKGSGhobs2bNnnNqpCT788EMGgM2bN0/lY2r7OVdVUurduzcDwFatWlVh37Rp0xgAtnz5cpXba9asGQPA9u3bV2Ff//79GQC2e/duletjjDF6pkS0ws/PDwBQXFyMzMxMHUdTex07dgwSiQQBAQGoX7++3D4+n49+/fpBLBbj33//1VGEb0d+fr7sttXo0aN1HE3NUFhYiNOnTwMABg8eXGG/dNuhQ4dUqu/hw4e4desWTE1N0adPH871SdEzJaIVDx48AAAYGxur/Tzjp59+wv3798Hn8+Hj44OBAwdCKBRWR5g6t379emRmZsLAwADe3t4YMGAA3NzcVD4+NjYWAODv769wv7+/PzZs2IC4uDitxKuv9u7di/z8fLRu3RrNmzdX+/i6dM5JJSUlobi4GEKhUOGk0NJzStVzR3outmjRAsbGxpzrk1HruooQJSZOnMgAsH79+ql8jPRWyps/ZmZmbP369dUY7dsnvX335o+xsTFbvHixyvUMHDiQAWDh4eEK9+/fv1/2vKk2Cw4OZgDYypUr1Tqutp9zld2+O3DgAAPAWrdurfR4a2trBoDl5uZW2VZ4eDgDwAYOHKhwf05ODgPAbG1tVQv+Nbp9Rzj7999/sX79ehgbG2PJkiUqH9e/f3/s3bsXKSkpKCgoQEJCAmbNmoXi4mJMnDhR7VFF+qxLly74+++/cf/+fRQUFCApKQlLly6FkZERFi1ahPDwcJXqkQ67NzMzU7jf3NwcwP8vJlkbPXnyBKdOnYKhoSFGjBih1rF16Zx7U1XnDqDe+VNt56JaKYyQN9y6dYvZ2NgwAOznn3/WSp1r1qxhAFiTJk20Up8+i4iIYACYtbU1KygoqLJ8UFAQA8DWrl2rcL90dFVQUJC2Q9UbK1asYABYr169tFZnbTnnKrtS+ueffxgA1qlTJ6XHOzs7MwDs8ePHVba1dOlSBoCNHDlS4f7S0lLZ3QB10JUS0djjx4/Rq1cvZGdnY9asWZg+fbpW6p0wYQIcHByQlJSE5ORkrdSpr4KDg9G2bVvk5OTgypUrVZaXvuNUUFCgcH9+fj4AwNLSUntB6hnpu0naHOBQF865qs4dQL3zp7rORUpKRCNZWVkIDg5GSkoKxo0bh+XLl2utbgMDA3h6egJ4daumtvPy8gKg2meVDop49OiRwv3S7e7u7lqKTr/cunULN27cgIWFhVoveValLpxzVZ07+fn5yMnJgY2NjUqJpLrORUpKRG0vX75E7969kZiYiJCQEKxdu1alqU3UkZ2dDeD/70vXZup8VunQ+5iYGIX7pdt9fX21FJ1++fvvvwEAISEhlT4b0URtP+eaNGkCPp+PFy9e4PHjxxX2q3vuSM/FhIQElJaWcq5PRq2bfaTOKyoqYt27d2cAWM+ePVV+UVYdCQkJjMfjMTMzs2qpX588f/6cmZubMwAsLS2tyvLlX5598wXZ2v7yrEQikY2eO3HihFbrri3nXG14eZaSElFZWVmZbEhyQEAAy8/Pr/KY1atXsyZNmlR46/7IkSPs1KlTFcrHxsbKTvRp06ZpLXZdioyMZPv27WNlZWVy2x8+fMg6derEALD+/fvL7VPWb4z9/zRDgwYNkpvCRfqlUlunGTp37hwDwJydnZlYLFZari6fc1ymGeLz+QqnGbpy5Qpr0qQJ6969e4X6yk8zVP4PoT179mg8zRC9PEtU9uuvv2Lfvn0AAHt7e3zyyScKyy1fvhz29vYAXk0empSUVOE+/dWrV/Htt9/C3d0dfn5+MDMzw4MHDxATE4OysjJ07doVP/zwQ/V+oLfkzp07GDduHBwdHeHv7w9ra2ukpKTg+vXrKCoqgo+PD9auXSt3jLJ+A4Cff/4Zly9fxp49e9C0aVO0bdsW//33HxISEuDl5YWVK1e+rY/2VkkHOHz44YcwMFD+5KEunXNHjhyRew2jpKQEANC+fXvZtoULF8pmXOjRowemT5+O8PBwtGrVCkFBQSgpKcGJEyfAGMPGjRthbW0t14b0FYaioqIK7Y8fPx7//vsv9u3bh6ZNm+K9995DRkYGzp07B1NTU2zduhVGRmqmGbVSGKnTvv76a4UvHr75U35+N+kxb/71funSJTZ+/HjWsmVLZmdnx4yMjJitrS3r2rUrW7t2bYWriposMTGRffzxx8zf358JhUJmZGTErKysWPv27dmKFSsUDgVX1m9SmZmZbOrUqczV1ZWZmJgwV1dXNm3aNLUm06xJioqKZK8exMbGVlq2Lp1zGzdurPL/x40bNyo8rk2bNszMzIxZW1uzXr16scjISIVtnDlzhgFg7u7uCveXlZWxFStWMB8fHyYQCJidnR0bPHgw+++//zT6TDzGGFMvjRFCCCHVg0bfEUII0RuUlAghhOgNSkqEEEL0BiUlQggheoOSEiGEEL1BSYkQQojeoKRECCFEb1BSIoQQojcoKRFCCNEblJQIKYfH42l9GQ59wuPx0LBhQ12HQYhSlJQIqUJycjJ4PB66du2q61AqVVPiJKQyNEs4IXXIrVu3YGxsrOswCFGKkhIhdUjTpk11HQIhlaLbd4RU4ptvvkGjRo0AAOfOnZM9c+LxeAgNDZUrm5aWhs8++wyenp4QCASwtbVF3759cenSpQr1lr/Vlpubi1mzZqFRo0YwNjbGjBkzALxaK2f8+PFo1qwZ6tWrB3Nzc/j5+eH7779HcXGxRnFW9kwpKioKH3zwAYRCIfh8Pho2bIhPPvkE6enplcZfWFiIefPmwd3dHXw+H40bN8aPP/4IdRcguHfvHng8HoKCgpCfn49vv/0WzZo1g5mZGTw9PREWFqZ2naTmoSslQirRqlUrDBo0CHv27EH9+vXRq1cv2b7OnTvL/jsqKgp9+vRBdnY2mjRpgj59+uDFixeIiIjAsWPH8M8//2DYsGEV6i8sLERgYCBSUlIQGBgIf39/2NjYAAAmTJiAwsJCtGjRAr6+vhCJRLh69SoWLFiAU6dO4fjx4zA0NFQrTmW2bt2K0NBQiMVidOrUCa6uroiJicEff/yBvXv34uzZswqvskpKShAcHIzExER07doV+fn5OHfuHObNm4e8vDx89913Kvd1bGwsAMDW1hZ+fn4oLi5Ghw4d4OrqilOnTuGLL76Avb09xo8fr3KdpAbSaBUmQmopvF4YrbyHDx8yACwwMFDhMSKRiDk5OTFDQ0O2detWuX3Xrl1jNjY2zMLCgj1//rxCnQBYhw4dFC7Ot3///goLAObm5rK+ffsyAGzz5s1qxSn9fG8u1paamspMTU2ZoaEhO3DggGy7WCxmM2bMYABY27ZtFbYlbU8kEsl9ZkNDQ2ZmZsby8vKUxvKmhQsXyur88ssv5ZbRli5m16dPH5XrIzUTJSVCytEkKa1atYoBYLNnz1a4f+XKlQwAW7lyZYU6AbBr166pFePdu3cZABYSEqJWnIwpTkqLFi1iANiIESMqlC8qKmINGjRgANjFixcrtGVgYMBu375d4Thp4jxz5ozKn6tfv34MABs0aFCFfU+ePGEA2DvvvKNyfaRmott3hHB0/PhxAEBISIjC/QEBAQCAq1evVtjn5OSEtm3bKq377t27+Pfff3Hv3j3k5+dDIpHInqvcvXuXa+gAgAsXLgAARo4cWWEfn8/HkCFDEB4ejgsXLqBTp05y+93d3dGkSZMKx3l7ewMAnjx5onIc0tt3ixYtqrAvIyMDANCgQQOV6yM1EyUlQjhKTk4GgApf2G+SfrGW5+bmprAsYwyff/45Vq1apfThfl5ennqBKiEdyKBsAIR0++PHjyvsc3FxUXiMpaUlAFQYkKFMTk4OUlNT0ahRI/j6+lbYHxcXBwBo0aKFSvWRmouSEiEcSSQSAMDgwYNhbm6utJyigQICgUBh2Z07d2LlypVwdXXFqlWr0KFDBwiFQhgbG6OkpAR8Pv+tjUSrbIYLAwPtDOC9efMmAKBdu3YK99+4cQPAqwEdpHajpEQIRy4uLkhKSsK8efPQpk0brdS5b98+AMAff/yBPn36yO178OCBVtqQatCgAZKSkpCSkgIfH58K+6VXgs7OzlpttzzprbvWrVsr3E9Jqe6g95QIqYKJiQkAoKysTOH+oKAgAP+fSLQhOzsbgOLbY7t27VJ4TFVxKiN95rV9+/YK+0pKSvC///1Prlx1UCUpWVpawtPTs9piIPqBkhIhVbC3t4exsTHu378PsVhcYf9HH30EBwcHhIWFYc2aNbLbeVJlZWWIiIhAQkKCym1KBwqsWbNG7jbdhQsX8NNPP2kUpzITJkyAqakpduzYgSNHjsi2SyQSfPnll3j8+DHatGlT5TMzLqS37xQlpdTUVGRlZcHX17dWT5ZLXqGkREgVTExM0KtXLzx9+hR+fn4YM2YMJk6ciI0bNwIArK2tceDAAVhZWeGjjz5Cw4YN8f7772PkyJF47733IBQK0atXL9y7d0/lNqdNmwZzc3P8/vvvaNGiBUaMGIEuXbogMDAQU6ZM0ShOZdzc3PDXX39BIpGgX79+CAgIwIcffojmzZtjxYoVqF+/PrZu3ap6h6mprKwMiYmJcHZ2hlAorLCfbt3VLZSUCFHBunXrMHr0aGRmZmLbtm1Yv349zp07J9vfvn17xMfHY+7cuahXrx7OnTuH/fv3y2Zq2LRpE3r06KFye97e3oiOjka/fv2QkZGBgwcP4uXLl/jrr7+UXimpEqcyo0ePxoULF9C3b1/cunULu3fvRmFhIT7++GNcv369WufMu337NoqLi5UmHUpKdQuPva0hPIQQQkgV6EqJEEKI3qCkRAghRG9QUiKEEKI3KCkRQgjRG5SUCCGE6A1KSoQQQvQGJSVCCCF6g5ISIYQQvUFJiRBCiN6gpEQIIURvUFIihBCiNygpEUII0Rv/BwivZ2n1muy1AAAAAElFTkSuQmCC", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "# Each trial is a row in a dataframe that contains\n", "# Algorithm, Number of Samples, Number of Features, Hyperparameters, Score, Runtime, Memory Usage, Step as features\n", "trials = est1.completed_trials_summary_[est1.completed_trials_summary_[\"Step\"].str.contains('Model Tuning')]\n", "trials.replace([np.inf, -np.inf], np.nan, inplace=True)\n", "trials.dropna(subset=[name_of_score_column], inplace=True)\n", "trials = trials.sort_values(by=['Finished'],ascending=True)\n", "scores = trials[name_of_score_column].tolist()\n", "score = []\n", "score.append(scores[0])\n", "for i in range(1,len(scores)):\n", " if scores[i]>= score[i-1]:\n", " score.append(scores[i])\n", " else:\n", " score.append(score[i-1])\n", "y_margin = 0.10 * (max(score) - min(score))\n", "\n", "fig, ax = plt.subplots(1)\n", "ax.set_title(\"Model Tuning Trials\")\n", "ax.set_xlabel(\"Iteration $n$\")\n", "ax.set_ylabel(est1._inferred_score_metric[0].name)\n", "ax.grid(color='g', linestyle='-', linewidth=0.1)\n", "ax.set_ylim(min(score) - y_margin, max(score) + y_margin)\n", "ax.plot(range(1, len(trials) + 1), score, 'k:', marker=\"s\", color='teal', markersize=3)\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "1199dc89", "metadata": {}, "source": [ "\n", "### Advanced AutoML Configuration\n", "\n", "For customizing the model tuning step, the range of the hyperparameters of each of the models can be specified and passed to the pipeline." ] }, { "cell_type": "code", "execution_count": 13, "id": "69f4fed8", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:59.835207Z", "iopub.status.busy": "2025-04-25T10:33:59.834876Z", "iopub.status.idle": "2025-04-25T10:33:59.896817Z", "shell.execute_reply": "2025-04-25T10:33:59.896321Z" } }, "outputs": [], "source": [ "shared_hyperparameters = {\n", " \"epochs\": {'range': [1,5],\n", " 'type': 'discrete'\n", " },\n", " \"batch_size\": {'range': [16, 32],\n", " 'type': 'discrete'\n", " }\n", "}\n", "search_space = {\n", " \"EfficientNet\" : {\n", " \"size\": {'range': [\"b2\",\"b6\"],\n", " 'type': 'categorical'\n", " },\n", " **shared_hyperparameters\n", " }\n", "}" ] }, { "cell_type": "markdown", "id": "021f5459", "metadata": {}, "source": [ "You can also configure the pipeline with suitable parameters according to your needs." ] }, { "cell_type": "code", "execution_count": 14, "id": "bbc9bc67", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:59.898782Z", "iopub.status.busy": "2025-04-25T10:33:59.898355Z", "iopub.status.idle": "2025-04-25T10:33:59.959143Z", "shell.execute_reply": "2025-04-25T10:33:59.958659Z" } }, "outputs": [], "source": [ "custom_pipeline = automlx.Pipeline(\n", " task='classification',\n", " model_list=[ # Specify the models you want the AutoML to consider\n", " 'EfficientNet',\n", " ],\n", " n_algos_tuned=2, # Choose how many models to tune\n", " adaptive_sampling=False, # Disable or enable Adaptive Sampling step. Default to `True`\n", " search_space=search_space, # You can specify the hyper-parameters and ranges AutoML searches\n", " max_tuning_trials=2, # The maximum number of tuning trials. Can be integer or Dict (max number for each model)\n", " score_metric='f1_macro', # Any scikit-learn metric or a custom function\n", ")" ] }, { "cell_type": "markdown", "id": "070f7f6c", "metadata": {}, "source": [ "A few of the advanced settings can be passed directly to the pipeline's fit method, instead of its constructor." ] }, { "cell_type": "code", "execution_count": 15, "id": "cc152bb0", "metadata": { "execution": { "iopub.execute_input": "2025-04-25T10:33:59.960968Z", "iopub.status.busy": "2025-04-25T10:33:59.960565Z", "iopub.status.idle": "2025-04-25T10:34:41.395271Z", "shell.execute_reply": "2025-04-25T10:34:41.394722Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:00,140] [automlx.interface] Dataset shape: (1000,1)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:00,188] [automlx.interface] Adaptive Sampling disabled.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:00,225] [automlx.data_transform] Running preprocessing. Number of features: 2\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,091] [automlx.data_transform] Preprocessing completed. Took 0.866 secs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,123] [automlx.process] Running Model Generation\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,170] [automlx.process] Model Generation completed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,196] [automlx.model_selection] Running Model Selection\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,197] [automlx.trials] Fewer models (1) than top_k (2) provided, skipping...\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,305] [automlx.feature_selection] Starting feature ranking for EfficientNet\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,305] [automlx.feature_selection] min_features is larger than model input features for EfficientNet, skipping feature selection for it\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:01,340] [automlx.trials] Running Model Tuning for ['EfficientNet']\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:16,157] [automlx.backend] \u001b[91mJob cda0b026-21c0-11f0-b4ee-00001701c949 <> EfficientNet - Trial: 2 - Split: 3 of 5\n", " failed\n", "Stack trace:\n", "Traceback (most recent call last):\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_backend/utils.py\", line 65, in _handle_job_exception\n", " complete_results = profile.function(\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_utils/profile.py\", line 133, in profile_wrapper\n", " ) = _profile_resources(\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_utils/profile.py\", line 655, in _profile_resources\n", " function_output = func(*args, **kwargs, **pf)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_trials/strategies/base.py\", line 112, in run\n", " return self._run(reporter, resources, pipeline, train_dataset, callback_manager, pf=pf)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_trials/strategies/standard.py\", line 45, in _run\n", " pipeline.fit(train_dataset, {model_idx: callback_manager}, pf=pf)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_pipeline/pipeline.py\", line 282, in fit\n", " self.stages[i] = stage.fit(dataset, callback=callback)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_model/distributed/distributed_model.py\", line 69, in fit\n", " self._fit(dataset, callback)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_model/pytorch/base.py\", line 161, in _fit\n", " output = self._wrapped_model(batch)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torchvision/models/efficientnet.py\", line 343, in forward\n", " return self._forward_impl(x)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torchvision/models/efficientnet.py\", line 333, in _forward_impl\n", " x = self.features(x)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torchvision/models/efficientnet.py\", line 164, in forward\n", " result = self.block(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py\", line 171, in forward\n", " return F.batch_norm(\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/functional.py\", line 2448, in batch_norm\n", " _verify_batch_size(input.size())\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/functional.py\", line 2416, in _verify_batch_size\n", " raise ValueError(\"Expected more than 1 value per channel when training, got input size {}\".format(size))\n", "ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 720, 1, 1])\n", "\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:16,206] [automlx.backend] \u001b[91mJob cda0b026-21c0-11f0-b4ee-00001701c949 <> EfficientNet - Trial: 2 - Split: 2 of 5\n", " failed\n", "Stack trace:\n", "Traceback (most recent call last):\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_backend/utils.py\", line 65, in _handle_job_exception\n", " complete_results = profile.function(\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_utils/profile.py\", line 133, in profile_wrapper\n", " ) = _profile_resources(\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_utils/profile.py\", line 655, in _profile_resources\n", " function_output = func(*args, **kwargs, **pf)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_trials/strategies/base.py\", line 112, in run\n", " return self._run(reporter, resources, pipeline, train_dataset, callback_manager, pf=pf)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_trials/strategies/standard.py\", line 45, in _run\n", " pipeline.fit(train_dataset, {model_idx: callback_manager}, pf=pf)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_pipeline/pipeline.py\", line 282, in fit\n", " self.stages[i] = stage.fit(dataset, callback=callback)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_model/distributed/distributed_model.py\", line 69, in fit\n", " self._fit(dataset, callback)\n", " File \"/scratch_user/jenkins/workspace/v2_long_regress/src/automlx/_model/pytorch/base.py\", line 161, in _fit\n", " output = self._wrapped_model(batch)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torchvision/models/efficientnet.py\", line 343, in forward\n", " return self._forward_impl(x)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torchvision/models/efficientnet.py\", line 333, in _forward_impl\n", " x = self.features(x)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torchvision/models/efficientnet.py\", line 164, in forward\n", " result = self.block(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/container.py\", line 217, in forward\n", " input = module(input)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/module.py\", line 1501, in _call_impl\n", " return forward_call(*args, **kwargs)\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py\", line 171, in forward\n", " return F.batch_norm(\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/functional.py\", line 2448, in batch_norm\n", " _verify_batch_size(input.size())\n", " File \"/scratch_user/olautoml/.conda/envs/pipeline-run-3.9.19-releasev252/lib/python3.9/site-packages/torch/nn/functional.py\", line 2416, in _verify_batch_size\n", " raise ValueError(\"Expected more than 1 value per channel when training, got input size {}\".format(size))\n", "ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 720, 1, 1])\n", "\u001b[0m\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:17,092] [automlx.trials] Best parameters for EfficientNet: {'optimizer_class': 'Adam', 'shuffle_dataset_each_epoch': True, 'optimizer_params': {}, 'criterion_class': None, 'criterion_params': {}, 'scheduler_class': None, 'scheduler_params': {}, 'batch_size': 16, 'lr': 0.001, 'epochs': 1, 'input_transform': 'auto', 'tensorboard_dir': None, 'use_tqdm': None, 'prediction_batch_size': 128, 'prediction_input_transform': 'auto', 'shuffling_buffer_size': None, 'freeze.encoder': False, 'load.encoder': None, 'size': 'b2'}\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:17,094] [automlx.trials] Model Tuning completed. Took: 15.754 secs\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:17,268] [automlx.interface] Re-fitting pipeline\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:17,284] [automlx.final_fit] Skipping updating parameter seed, already fixed by FinalFit_8072097a-a\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "2025-04-25 03:34:21,520\tERROR worker.py:409 -- Unhandled error (suppress with 'RAY_IGNORE_UNHANDLED_ERRORS=1'): The worker died unexpectedly while executing this task. Check python-core-worker-*.log files for more information.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "[2025-04-25 03:34:37,909] [automlx.interface] AutoMLx completed.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Score on test data : 0.5000\n" ] } ], "source": [ "custom_pipeline.fit(\n", " X_train,\n", " y_train,\n", " time_budget= 20, # Specify time budget in seconds\n", " cv='auto' # Automatically pick a good cross-validation (cv) strategy for the user's dataset.\n", " # Ignored if X_valid and y_valid are provided.\n", " # Can also be:\n", " # - An integer (for example, to use 5-fold cross validation)\n", " # - A list of data indices to use as splits (for advanced, such as time-based splitting)\n", ")\n", "y_pred = custom_pipeline.predict(X_test)\n", "score_default = balanced_accuracy_score(y_test, y_pred)\n", "\n", "print(f'Score on test data : {score_default:.4f}')" ] }, { "cell_type": "markdown", "id": "29eaed82", "metadata": {}, "source": [ "\n", "## References\n", "* Oracle AutoML http://www.vldb.org/pvldb/vol13/p3166-yakovlev.pdf\n", "* scikit-learn https://scikit-learn.org/stable/\n", "* Hugging Face https://huggingface.co/\n", "* MedMNIST Dataset https://medmnist.com/" ] } ], "metadata": { "jupytext": { "formats": "ipynb,md,py:percent" }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.21" } }, "nbformat": 4, "nbformat_minor": 5 }