{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6K72VWPHwgvf"
   },
   "source": [
    "# Baseline model for urban sound tagging\n",
    "\n",
    "In this notebook, you will build and train a baseline model to perform urban sound tagging with [PyTorch](https://pytorch.org/).\n",
    "\n",
    "Similar to the original [baseline of the DCASE 2019 Challenge - Task 5](https://github.com/sonyc-project/urban-sound-tagging-baseline) presented in ([Cartwright et al., 2019)](https://dcase.community/documents/workshop2019/proceedings/DCASE2019Workshop_Cartwright_4.pdf), this baseline is a simple multi-label logistic regression model, i.e., a separate binary logistic regression model for each of the 8 classes in the SONYC-UST dataset.\n",
    "\n",
    "The model takes [VGGish](https://github.com/tensorflow/models/tree/master/research/audioset/vggish) embeddings as input, which originally returns a 128-dimensional vector given an audio signal of 0.96 seconds. The SONYC-UST audio data samples being 10-second long, we simply compute VGGish embeddings on short nonoverlapping frames and pool them temporally before feeding the resulting representation to the multi-label logistic regression model.\n",
    "\n",
    "VGGish was trained on [AudioSet](https://github.com/tensorflow/models/tree/master/research/audioset), a dataset of over 2 million human-labeled 10-second YouTube video soundtracks, with labels taken from an ontology of more than 600 audio event classes. This represents more than 5 thousand hours of audio.\n",
    "\n",
    "The baseline model is trained to minimize the binary cross-entropy loss, using the Adam optimizer. Early stopping on the validation set is used to mitigate overfitting.\n",
    "\n",
    "**Before working on the rest of this notebook, take some time to browse the above references about VGGish, Audioset, and the DCASE 2019 Challenge - Task 5 baseline.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CjQBtWoyvKJA"
   },
   "source": [
    "## Environment setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "id": "9BfpoUwfxE0-",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "SEED = 0\n",
    "import numpy as np\n",
    "np.random.seed(SEED)\n",
    "import torch\n",
    "torch.manual_seed(SEED)\n",
    "import torch.nn as nn\n",
    "from torch.utils.data.dataset import Dataset\n",
    "\n",
    "import os\n",
    "import shutil\n",
    "\n",
    "import pandas as pd\n",
    "import oyaml as yaml\n",
    "import pytz\n",
    "import datetime\n",
    "import json\n",
    "from tqdm import tqdm\n",
    "\n",
    "from utils import get_file_targets, get_subset_split, generate_output_file, predict\n",
    "from metrics import evaluate, micro_averaged_auprc, macro_averaged_auprc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 9,
     "status": "ok",
     "timestamp": 1708885950128,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "HwMtwyCmMlOz",
    "outputId": "d40c9238-8ef4-48c7-8b2a-e850b65f5923"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "baseline_2024-09-03-12h38m17s\n"
     ]
    }
   ],
   "source": [
    "from paths import (root_path, ust_data_dir, dataset_dir, annotation_file, \n",
    "                   taxonomy_file, log_mel_spec_dir)\n",
    "\n",
    "os.chdir(root_path)\n",
    "\n",
    "model_name = 'baseline'\n",
    "\n",
    "tz_Paris = pytz.timezone('Europe/Paris')\n",
    "datetime_Paris = datetime.datetime.now(tz_Paris)\n",
    "timestamp = datetime_Paris.strftime(\"%Y-%m-%d-%Hh%Mm%Ss\")\n",
    "\n",
    "exp_id = model_name + '_' + timestamp\n",
    "\n",
    "output_dir = os.path.join(root_path, 'data/output', exp_id)\n",
    "os.makedirs(output_dir)\n",
    "\n",
    "print(exp_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If a GPU is available we will use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda\n"
     ]
    }
   ],
   "source": [
    "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
    "print(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "dPakCQ0mwqRH"
   },
   "source": [
    "### Define parameters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "WUpxDiEB-Cn1"
   },
   "source": [
    "In the following cell, you have to set several hyperparameters of the learning algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "id": "jTSLmdEEwuNY"
   },
   "outputs": [],
   "source": [
    "batch_size = 128\n",
    "num_epochs = 100\n",
    "learning_rate = 1e-2\n",
    "patience = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "JSomLphkY8xT"
   },
   "outputs": [],
   "source": [
    "# save parameters to disk\n",
    "params = {'annotation_file': annotation_file,\n",
    "          'taxonomy_file': taxonomy_file,\n",
    "          'exp_id': exp_id,\n",
    "          'log_mel_spec_dir': log_mel_spec_dir,\n",
    "          'learning_rate': learning_rate,\n",
    "          'batch_size': batch_size,\n",
    "          'batch_size': batch_size,\n",
    "          'num_epochs': num_epochs,\n",
    "          'patience': patience}\n",
    "\n",
    "kwarg_file = os.path.join(output_dir, \"hyper_params.json\")\n",
    "with open(kwarg_file, 'w') as f:\n",
    "    json.dump(params, f, indent=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZeyPBjMHM3AP"
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MImVKG8Cw6ae"
   },
   "source": [
    "### Annotations and taxonomy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "id": "4eK-7Mwrw-AW"
   },
   "outputs": [],
   "source": [
    "# Create a Pandas DataFrame from the annotation CSV file\n",
    "annotation_data = pd.read_csv(annotation_file).sort_values('audio_filename')\n",
    "\n",
    "# List of all audio files\n",
    "file_list = annotation_data['audio_filename'].unique().tolist()\n",
    "\n",
    "# Load taxonomy\n",
    "with open(taxonomy_file, 'r') as f:\n",
    "    taxonomy = yaml.load(f, Loader=yaml.Loader)\n",
    "\n",
    "# get list of labels from taxonomy\n",
    "labels = [\"_\".join([str(k), v]) for k,v in taxonomy['coarse'].items()]\n",
    "\n",
    "# list of one-hot encoded labels for all audio files\n",
    "target_list = get_file_targets(annotation_data, labels)\n",
    "\n",
    "# list of idices for the training, validation and test subsets\n",
    "train_file_idxs, val_file_idxs, test_file_idxs = get_subset_split(annotation_data)\n",
    "\n",
    "# number of classes\n",
    "n_classes = len(labels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I0PMPGJQx8FM"
   },
   "source": [
    "### Log-Mel spectrograms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "bAtHMZuxfxAI"
   },
   "outputs": [],
   "source": [
    "how_saved = 'global' # 'individual' or 'global'\n",
    "\n",
    "if how_saved == 'global':\n",
    "  log_mel_spec_list = list(np.load(os.path.join(log_mel_spec_dir, 'data.npy')))\n",
    "\n",
    "elif how_saved == 'individual':\n",
    "  # Create a list of log-Mel spectrograms of size 998 frames × 64 Mel-frequency\n",
    "  log_mel_spec_list = []\n",
    "  for idx, filename in enumerate(file_list):\n",
    "      clear_output(wait=True)\n",
    "\n",
    "      log_mel_path = os.path.join(log_mel_spec_dir, os.path.splitext(filename)[0] + '.npy')\n",
    "      log_mel_spec = np.load(log_mel_path)\n",
    "      log_mel_spec_list.append(log_mel_spec)\n",
    "\n",
    "      print('({}/{})'.format(idx+1, len(file_list)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "Fy6Eo8KgRV0q"
   },
   "outputs": [],
   "source": [
    "# Create training and validation data loaders\n",
    "\n",
    "class MyDataset(Dataset):\n",
    "\tdef __init__(self, x, y):\n",
    "\t\tself.x = x\n",
    "\t\tself.y = y\n",
    "\n",
    "\tdef __getitem__(self, index):\n",
    "\t\tx = torch.Tensor(self.x[index]).to(device)\n",
    "\t\ty = torch.Tensor(self.y[index]).to(device)\n",
    "\t\treturn (x, y)\n",
    "\n",
    "\tdef __len__(self):\n",
    "\t\tcount = self.x.shape[0]\n",
    "\t\treturn count\n",
    "\n",
    "train_x = []\n",
    "train_y = []\n",
    "for idx in train_file_idxs:\n",
    "    train_x.append(log_mel_spec_list[idx])\n",
    "    train_y.append(target_list[idx])\n",
    "\n",
    "perm_train_idxs = np.random.permutation(len(train_x))\n",
    "\n",
    "train_x = np.array(train_x)[perm_train_idxs]\n",
    "train_y = np.array(train_y)[perm_train_idxs]\n",
    "\n",
    "val_x = []\n",
    "val_y = []\n",
    "for idx in val_file_idxs:\n",
    "    val_x.append(log_mel_spec_list[idx])\n",
    "    val_y.append(target_list[idx])\n",
    "\n",
    "perm_val_idxs = np.random.permutation(len(val_x))\n",
    "\n",
    "val_x = np.array(val_x)[perm_val_idxs]\n",
    "val_y = np.array(val_y)[perm_val_idxs]\n",
    "\n",
    "# reshape by adding a channel dimension of size 1\n",
    "# new shape : (num of examples, channel, frames, frequency bands)\n",
    "train_x = np.reshape(train_x,(-1,1,train_x.shape[1],train_x.shape[2]))\n",
    "val_x = np.reshape(val_x,(-1,1,val_x.shape[1],val_x.shape[2]))\n",
    "\n",
    "train_dataset = MyDataset(train_x, train_y)\n",
    "val_dataset = MyDataset(val_x, val_y)\n",
    "\n",
    "train_loader = torch.utils.data.DataLoader(dataset=train_dataset,\n",
    "                                           batch_size=batch_size,\n",
    "                                           shuffle=True)\n",
    "\n",
    "val_loader = torch.utils.data.DataLoader(dataset=val_dataset,\n",
    "                                         batch_size=batch_size,\n",
    "                                         shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda:0 torch.Size([128, 1, 998, 64])\n",
      "cuda:0 torch.Size([128, 8])\n"
     ]
    }
   ],
   "source": [
    "data, target = next(iter(train_loader))\n",
    "print(data.device, data.shape)\n",
    "print(target.device, target.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d7KWIZMtybx0"
   },
   "source": [
    "## UST baseline model\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZSmswK2TMuJg"
   },
   "source": [
    "\n",
    "### VGGish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BJiBq47C7H-Z"
   },
   "source": [
    "[VGGish](https://github.com/tensorflow/models/tree/master/research/audioset/vggish) is a variant of the [VGG](https://arxiv.org/abs/1409.1556) model, in\n",
    "particular Configuration A with 11 weight layers. Specifically, here are the\n",
    "changes that were made:\n",
    "\n",
    "* The input size was changed to 96x64 for log mel spectrogram audio inputs.\n",
    "\n",
    "* The last group of convolutional and maxpool layers was dropped, so we now have\n",
    "  only four groups of convolution/maxpool layers instead of five.\n",
    "\n",
    "* Instead of a 1000-wide fully connected layer at the end, 128-wide\n",
    "  fully connected layer was used. This acts as a compact embedding layer.\n",
    "  \n",
    "In the following cell, we define VGGish model in PyTorch. Take time to read the code, and compare it to the structure of the VGG and VGGish models as described in the above links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "AWUeUvMpyfSM"
   },
   "outputs": [],
   "source": [
    "class VGGish(nn.Module):\n",
    "  def __init__(self):\n",
    "    super(VGGish, self).__init__()\n",
    "\n",
    "    self.layer1_conv1 = nn.Sequential(\n",
    "        nn.Conv2d(1, 64, kernel_size=3, stride=1, padding=1),\n",
    "        nn.ReLU())\n",
    "    self.layer2_pool1 = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "\n",
    "    self.layer3_conv2 = nn.Sequential(\n",
    "        nn.Conv2d(64, 128,kernel_size=3, stride=1, padding=1),\n",
    "        nn.ReLU())\n",
    "    self.layer4_pool2 = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "\n",
    "    self.layer5_conv3_1 = nn.Sequential(\n",
    "        nn.Conv2d(128, 256,kernel_size=3, stride=1,padding=1),\n",
    "        nn.ReLU())\n",
    "    self.layer6_conv3_2 = nn.Sequential(\n",
    "        nn.Conv2d(256, 256,kernel_size=3, stride=1,padding=1),\n",
    "        nn.ReLU())\n",
    "    self.layer7_pool3 = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "\n",
    "    self.layer8_conv4_1 = nn.Sequential(\n",
    "        nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1),\n",
    "        nn.ReLU())\n",
    "    self.layer9_conv4_2 = nn.Sequential(\n",
    "        nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),\n",
    "        nn.ReLU())\n",
    "    self.layer10_pool4 = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "\n",
    "    self.fc1 = nn.Sequential(nn.Linear(512*24, 4096), nn.ReLU())\n",
    "    self.fc2 = nn.Sequential(nn.Linear(4096, 4096), nn.ReLU())\n",
    "    self.fc3 = nn.Sequential(nn.Linear(4096, 128), nn.ReLU())\n",
    "\n",
    "  def extract_features(self, x, print_shape=False):\n",
    "\n",
    "    if print_shape:\n",
    "      print('\\n' + self.__class__.__name__ + ' features extraction')\n",
    "      print(x.shape)\n",
    "\n",
    "    # Block 1\n",
    "    out = self.layer1_conv1(x)\n",
    "    out = self.layer2_pool1(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    # Block 2\n",
    "    out = self.layer3_conv2(out)\n",
    "    out = self.layer4_pool2(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    # Block 3\n",
    "    out = self.layer5_conv3_1(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "    out = self.layer6_conv3_2(out)\n",
    "    out = self.layer7_pool3(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    # Block 4\n",
    "    out = self.layer8_conv4_1(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "    out = self.layer9_conv4_2(out)\n",
    "    out = self.layer10_pool4(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    return out\n",
    "\n",
    "  def extract_embedding(self, x, print_shape=False):\n",
    "\n",
    "    if print_shape:\n",
    "      print('\\n' + self.__class__.__name__ + ' embedding extraction')\n",
    "      print(x.shape)\n",
    "\n",
    "    out = self.fc1(x)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    out = self.fc2(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    out = self.fc3(out)\n",
    "    if print_shape:\n",
    "      print(out.shape)\n",
    "\n",
    "    return out\n",
    "\n",
    "  def forward(self, x, print_shape=False):\n",
    "    '''\n",
    "    If the signal is longer than 96 frames (0.96 s, the default input\n",
    "    length of VGGish), we will reshape the input tensor to process it as\n",
    "    individual chunks of 96 frames.\n",
    "\n",
    "    This is done by expanding the batch_size dimension.\n",
    "\n",
    "    We discard the last frames in the input tensor if its length is not\n",
    "    an integer multiple of 96.\n",
    "\n",
    "    We output (n_frames//96)*96 VGGish embeddings of dim. 128.\n",
    "    '''\n",
    "\n",
    "    reshape = False\n",
    "\n",
    "    (batch_size, n_channels, n_frames, n_feat) = x.shape\n",
    "\n",
    "    if n_frames > 96:\n",
    "\n",
    "        if print_shape:\n",
    "            print('Input')\n",
    "            print(x.shape)\n",
    "            print('Reshape')\n",
    "\n",
    "        x = x[:,:,:96*(n_frames//96),:]\n",
    "        x = x.reshape(-1,1,96,x.shape[-1])\n",
    "        reshape = True\n",
    "\n",
    "    features = self.extract_features(x, print_shape)\n",
    "\n",
    "    features = features.view(features.shape[0],-1)\n",
    "\n",
    "    embedding = self.extract_embedding(features, print_shape)\n",
    "\n",
    "    if reshape:\n",
    "        embedding = embedding.reshape(batch_size, -1, 128)\n",
    "\n",
    "        if print_shape:\n",
    "            print('\\nReshape')\n",
    "            print(embedding.shape)\n",
    "            print('Output')\n",
    "\n",
    "    return embedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "id": "o-nRAMdHKpMe"
   },
   "outputs": [],
   "source": [
    "# Instantiate the model\n",
    "vggish = VGGish().to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4TwsNF8RUDSE"
   },
   "source": [
    "In the following cells, we download and load the weights of the model pretrained on Audioset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "id": "pLwfnP0HUDSF"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/tmp/ipykernel_5569/1253985691.py:5: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
      "  pretrained_state_dict = torch.load(vggish_weights_file)\n"
     ]
    }
   ],
   "source": [
    "# Weights file\n",
    "vggish_weights_file = os.path.join(root_path, 'data/vggish.pth')\n",
    "\n",
    "# Load pretrained weights in the instantiated model\n",
    "pretrained_state_dict = torch.load(vggish_weights_file)\n",
    "vggish.load_state_dict(pretrained_state_dict)\n",
    "\n",
    "# Freeze all parameters\n",
    "for param in vggish.parameters():\n",
    "    param.requires_grad = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TSeajVXlLm6C"
   },
   "source": [
    "Let's test the model on a batch."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 57176,
     "status": "ok",
     "timestamp": 1708886045324,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "KaSf8Rx50JCN",
    "outputId": "1f9960ee-4d31-4112-ca5f-26d3859097c2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input\n",
      "torch.Size([128, 1, 998, 64])\n",
      "Reshape\n",
      "\n",
      "VGGish features extraction\n",
      "torch.Size([1280, 1, 96, 64])\n",
      "torch.Size([1280, 64, 48, 32])\n",
      "torch.Size([1280, 128, 24, 16])\n",
      "torch.Size([1280, 256, 24, 16])\n",
      "torch.Size([1280, 256, 12, 8])\n",
      "torch.Size([1280, 512, 12, 8])\n",
      "torch.Size([1280, 512, 6, 4])\n",
      "\n",
      "VGGish embedding extraction\n",
      "torch.Size([1280, 12288])\n",
      "torch.Size([1280, 4096])\n",
      "torch.Size([1280, 4096])\n",
      "torch.Size([1280, 128])\n",
      "\n",
      "Reshape\n",
      "torch.Size([128, 10, 128])\n",
      "Output\n"
     ]
    }
   ],
   "source": [
    "data, target = next(iter(train_loader))\n",
    "vggish_embedding = vggish(data, print_shape=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mFGHel-TLvcp"
   },
   "source": [
    "### Multi-label logistic regression\n",
    "\n",
    "In the following cell, we define the multi-label logistic regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 19,
     "status": "ok",
     "timestamp": 1708886045324,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "kUZ2XdfPMC4v",
    "outputId": "4374a773-0445-4a18-8963-d6f24ccc26d9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "LogisticRegression\n",
      "torch.Size([128, 128])\n",
      "torch.Size([128, 8])\n"
     ]
    }
   ],
   "source": [
    "class LogisticRegression(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(LogisticRegression, self).__init__()\n",
    "\n",
    "        self.fc = nn.Sequential(\n",
    "            nn.Linear(128, 8),\n",
    "            nn.Sigmoid())\n",
    "\n",
    "    def forward(self, x, print_shape=False):\n",
    "\n",
    "        if print_shape:\n",
    "            print('\\n' + self.__class__.__name__)\n",
    "            print(x.shape)\n",
    "\n",
    "        out = self.fc(x)\n",
    "        if print_shape:\n",
    "            print(out.shape)\n",
    "\n",
    "        return out\n",
    "\n",
    "# Instantiate\n",
    "logreg = LogisticRegression().to(device)\n",
    "\n",
    "# Test\n",
    "with torch.no_grad():\n",
    "    vggish_embedding = torch.max(vggish_embedding, dim=1)[0]\n",
    "    prediction = logreg(vggish_embedding, print_shape=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vhRZNidoMQ4C"
   },
   "source": [
    "### Complete model\n",
    "\n",
    "We finally define the complete model, that combines VGGish embeddings extraction, temporal pooling, and multi-label logisitic regression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 58050,
     "status": "ok",
     "timestamp": 1708886103356,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "87wiAJBTMSpw",
    "outputId": "10808da7-b28b-4dd0-ae55-b3c4d21b246c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([128, 1, 998, 64])\n",
      "torch.Size([128, 8])\n",
      "torch.Size([128, 8])\n"
     ]
    }
   ],
   "source": [
    "# Define\n",
    "class UST(nn.Module):\n",
    "    def __init__(self, vggish, logreg):\n",
    "        super(UST, self).__init__()\n",
    "\n",
    "        self.vggish = vggish\n",
    "        self.logreg = logreg\n",
    "\n",
    "    def forward(self, x):\n",
    "\n",
    "        # extract VGGish embeddings\n",
    "        x = self.vggish(x)\n",
    "        # temporal max pooling\n",
    "        x = torch.max(x, dim=1)[0]\n",
    "        # prediction\n",
    "        x = self.logreg(x)\n",
    "\n",
    "        return x\n",
    "\n",
    "# Instantiate\n",
    "model = UST(vggish, logreg).to(device)\n",
    "\n",
    "# Test\n",
    "data, target = next(iter(train_loader))\n",
    "prediction = model(data)\n",
    "\n",
    "print(data.shape)\n",
    "print(target.shape)\n",
    "print(prediction.shape)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lanOLQv7N6A8"
   },
   "source": [
    "We verify that only the multi-label logistic regression model contains trainable parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 9,
     "status": "ok",
     "timestamp": 1708886103357,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "u5f7EkT5NaeE",
    "outputId": "a5db1ce4-8ffc-4372-cd46-3851220c925e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "vggish.layer1_conv1.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer1_conv1.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer3_conv2.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer3_conv2.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer5_conv3_1.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer5_conv3_1.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer6_conv3_2.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer6_conv3_2.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer8_conv4_1.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer8_conv4_1.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer9_conv4_2.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.layer9_conv4_2.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.fc1.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.fc1.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.fc2.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.fc2.0.bias\n",
      "False\n",
      "cuda:0\n",
      "vggish.fc3.0.weight\n",
      "False\n",
      "cuda:0\n",
      "vggish.fc3.0.bias\n",
      "False\n",
      "cuda:0\n",
      "logreg.fc.0.weight\n",
      "True\n",
      "cuda:0\n",
      "logreg.fc.0.bias\n",
      "True\n",
      "cuda:0\n"
     ]
    }
   ],
   "source": [
    "for name, param in model.named_parameters():\n",
    "    print(name)\n",
    "    print(param.requires_grad)\n",
    "    print(param.device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YiTTtyUdRdGz"
   },
   "source": [
    "## Training\n",
    "\n",
    "We are now going to train the above-defined model. Note that it is very inefficient to pass the complete dataset through the frozen VGGish model at each epoch. A much more efficient solution would be to extract and store the VGGish embeddings for the whole SONYC UST dataset and then use these embeddings as input data to the multi-label logistic regression model. However, we chose the above inefficient option to make it easier for you to modify and build upon the baseline model.\n",
    "\n",
    "Before training the model, check that you are using a GPU or a TPU as hardware accelerator.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 453524,
     "status": "ok",
     "timestamp": 1708886556874,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "-imciSMAMakA",
    "outputId": "5f9fc511-bc69-41fe-8707-cb43af33fa2a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.06it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.5116179110850744\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.44it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.43115173095235976\n",
      "\n",
      "Epoch 2/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.18it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4549653892160121\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3879189138756918\n",
      "\n",
      "Epoch 3/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.44149337766627866\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3982476512008004\n",
      "\n",
      "Epoch 4/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.17it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4375808220602309\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3893466655745194\n",
      "\n",
      "Epoch 5/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4336092874426783\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.48it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.38681010340191174\n",
      "\n",
      "Epoch 6/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.18it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.42993238319847643\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3777852108866044\n",
      "\n",
      "Epoch 7/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.42765030432183504\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.42it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.38583240626089726\n",
      "\n",
      "Epoch 8/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.18it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4253459416111192\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3898058629870146\n",
      "\n",
      "Epoch 9/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.19it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4240761897833689\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3810898496792494\n",
      "\n",
      "Epoch 10/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4219647049523576\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.36943493921385395\n",
      "\n",
      "Epoch 11/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.00it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4196228338068772\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3632496683511454\n",
      "\n",
      "Epoch 12/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  1.95it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.41839490296586124\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.02it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.36096879624620787\n",
      "\n",
      "Epoch 13/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:10<00:00,  1.78it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.417352295439074\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:02<00:00,  1.65it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3615072171790336\n",
      "\n",
      "Epoch 14/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:12<00:00,  1.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4171233050674137\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:02<00:00,  1.75it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.36007951499377094\n",
      "\n",
      "Epoch 15/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.41568095691150325\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3597040529579395\n",
      "\n",
      "Epoch 16/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.17it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.41340483365693736\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.41it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.36264628757203404\n",
      "\n",
      "Epoch 17/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.41357709552977046\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3659792911118363\n",
      "\n",
      "Epoch 18/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4133159603300829\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3544525173797564\n",
      "\n",
      "Epoch 19/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4116452308042868\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3651303077793552\n",
      "\n",
      "Epoch 20/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4106754020319143\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3604016483637334\n",
      "\n",
      "Epoch 21/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40971104145962756\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.36152440417162723\n",
      "\n",
      "Epoch 22/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.13it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40820785634368595\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35410889317165917\n",
      "\n",
      "Epoch 23/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40931316435666654\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3689355082624922\n",
      "\n",
      "Epoch 24/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4093705119350422\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.31it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3653377590545413\n",
      "\n",
      "Epoch 25/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4073535799473106\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3659973564992909\n",
      "\n",
      "Epoch 26/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40783119361383363\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.38it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3543065335626796\n",
      "\n",
      "Epoch 27/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4070389539396342\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35335856655919523\n",
      "\n",
      "Epoch 28/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4051198484871247\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3471278150652117\n",
      "\n",
      "Epoch 29/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.16it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4047151429001091\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3541979596657893\n",
      "\n",
      "Epoch 30/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.18it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4040214652738283\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3597200516922479\n",
      "\n",
      "Epoch 31/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4040892327886498\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3627333724471839\n",
      "\n",
      "Epoch 32/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.14it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40386267995793784\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35807823408268913\n",
      "\n",
      "Epoch 33/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4034483810654603\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3481229167625005\n",
      "\n",
      "Epoch 34/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4037856414996528\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.356013410516571\n",
      "\n",
      "Epoch 35/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40282806446074626\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.37it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35196270388351336\n",
      "\n",
      "Epoch 36/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.05it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40271469889829026\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3501295138562506\n",
      "\n",
      "Epoch 37/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.40106454533447566\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.27it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3617598793172944\n",
      "\n",
      "Epoch 38/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4010404799781622\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.39it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.34903007342099607\n",
      "\n",
      "Epoch 39/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4019123530296608\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.34it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3502418740608353\n",
      "\n",
      "Epoch 40/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:10<00:00,  1.86it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4018245420269335\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:02<00:00,  1.97it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35784488075473775\n",
      "\n",
      "Epoch 41/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.02it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4005504657947373\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.33it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3680538980471092\n",
      "\n",
      "Epoch 42/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4016602750485727\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.36it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3618762901485908\n",
      "\n",
      "Epoch 43/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.28it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4004241489111642\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.28it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3617446670144729\n",
      "\n",
      "Epoch 44/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.10it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4005723379485813\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.35it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.3552824944997749\n",
      "\n",
      "Epoch 45/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.12it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.3997899769900049\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.26it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.34941496240904313\n",
      "\n",
      "Epoch 46/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.25it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.3983064211210967\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.43it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35374657173996316\n",
      "\n",
      "Epoch 47/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:08<00:00,  2.15it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4008916879602819\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35690008571400866\n",
      "\n",
      "Epoch 48/100\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 19/19 [00:09<00:00,  2.09it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training loss: 0.4001141216135289\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4/4 [00:01<00:00,  2.33it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Validation loss: 0.35569876096049496\n",
      "\n",
      "Training done!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "train_loss_history = []\n",
    "val_loss_history = []\n",
    "min_loss = np.inf\n",
    "min_epoch = -1\n",
    "patience_counter = 0\n",
    "\n",
    "loss_function = nn.BCELoss()\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "\n",
    "  print('Epoch %d/%d\\n' % (epoch+1, num_epochs))\n",
    "\n",
    "  # Training\n",
    "  model.train()\n",
    "  train_loss = []\n",
    "\n",
    "  for batch_idx, (data, target) in tqdm(enumerate(train_loader),\n",
    "                                        total=len(train_loader)):\n",
    "\n",
    "    optimizer.zero_grad()\n",
    "    model_output = model(data)\n",
    "    loss = loss_function(model_output, target)\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "    train_loss.append(loss.item()*len(data))\n",
    "\n",
    "  train_loss_history.append(np.sum(train_loss)/len(train_dataset))\n",
    "  print('Training loss: {}'.format(train_loss_history[-1]))\n",
    "\n",
    "  # Validation\n",
    "  model.eval()\n",
    "  val_loss = []\n",
    "\n",
    "  with torch.no_grad():\n",
    "    for batch_idx, (data, target) in tqdm(enumerate(val_loader),\n",
    "                                        total=len(val_loader)):\n",
    "\n",
    "      model_output = model(data)\n",
    "      loss = loss_function(model_output, target)\n",
    "      val_loss.append(loss.item()*len(data))\n",
    "\n",
    "    val_loss_history.append(np.sum(val_loss)/len(val_dataset))\n",
    "    print('Validation loss: {}\\n'.format(val_loss_history[-1]))\n",
    "\n",
    "  with open(os.path.join(output_dir, 'log.txt'), 'a') as f:\n",
    "    f.write(\"Epoch: {}/{}\\n\".format(epoch+1, num_epochs))\n",
    "    f.write(\"      Training loss: {}\\n\".format(train_loss_history[-1]))\n",
    "    f.write(\"      Validation loss: {}\\n\".format(val_loss_history[-1]))\n",
    "    f.write(\"      Best epoch / loss: {} / {}\\n\".format(min_epoch+1, min_loss))\n",
    "\n",
    "  # model saving\n",
    "  if min_loss > val_loss_history[-1]:\n",
    "\n",
    "    # update best loss\n",
    "    min_epoch = epoch\n",
    "    min_loss = val_loss_history[-1]\n",
    "\n",
    "    # save model\n",
    "    model_path = os.path.join(output_dir, 'best_model.pth')\n",
    "    best_state_dict = model.state_dict()\n",
    "    torch.save(best_state_dict, model_path)\n",
    "\n",
    "  # early stopping\n",
    "  if len(val_loss_history) >= 2:\n",
    "\n",
    "    if val_loss_history[-1] > min_loss:\n",
    "      patience_counter+=1\n",
    "    else:\n",
    "      patience_counter = 0\n",
    "\n",
    "  if patience_counter >= patience:\n",
    "    print('Training done!')\n",
    "    break\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 26,
     "status": "ok",
     "timestamp": 1708886556874,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "A2FDUAgvFSjT",
    "outputId": "a9deb310-f8ca-4667-8b1e-bcebefe7b1c2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<All keys matched successfully>"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# restore best model\n",
    "model.load_state_dict(best_state_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 430
    },
    "executionInfo": {
     "elapsed": 396,
     "status": "ok",
     "timestamp": 1708886557263,
     "user": {
      "displayName": "Simon Leglaive",
      "userId": "11077923176141204628"
     },
     "user_tz": -60
    },
    "id": "Y6Gh4UOUdAJm",
    "outputId": "2128dfc7-b049-4d0b-f99b-02b5de248776"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjUAAAGdCAYAAADqsoKGAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABq5ElEQVR4nO3dd3zV1eH/8de92TskIQlhhSVDRiCByHAgqeBAcbRgURAVrYJVU39WWhfab+OqxVW1KnUL2roHikFRIAKCQWQJSJgZhJAJWffe3x8nuRBZuZB7b8b7+XjcRz753M849wZy3znT4nA4HIiIiIi0cFZvF0BERESkKSjUiIiISKugUCMiIiKtgkKNiIiItAoKNSIiItIqKNSIiIhIq6BQIyIiIq2CQo2IiIi0Cr7eLoCn2O129uzZQ1hYGBaLxdvFERERkUZwOByUlZWRkJCA1Xr8upg2E2r27NlD586dvV0MEREROQk7d+6kU6dOxz2mzYSasLAwwLwp4eHhXi6NiIiINEZpaSmdO3d2fo4fT5sJNfVNTuHh4Qo1IiIiLUxjuo6oo7CIiIi0Cgo1IiIi0ioo1IiIiEir0Gb61IiIiLiLzWajpqbG28VokXx8fPD19W2S6VYUakRERE5BeXk5u3btwuFweLsoLVZwcDAdOnTA39//lK6jUCMiInKSbDYbu3btIjg4mPbt22tyVxc5HA6qq6vZu3cv27Zto1evXiecYO94FGpEREROUk1NDQ6Hg/bt2xMUFOTt4rRIQUFB+Pn5sX37dqqrqwkMDDzpa6mjsIiIyClSDc2pOZXamQbXaZKriIiIiHiZQo2IiIi0Cgo1IiIictISExOZM2eOt4sBqKOwiIhIm3POOeeQlJTUJGFk5cqVhISEnHqhmoBCzSn6PqeIT9fm0Ts+lIlDu3i7OCIiIqfM4XBgs9nw9T1xTGjfvr0HStQ4an46RZvyy5i7dBtfbijwdlFERMTLHA4HB6prvfJo7OR/11xzDYsXL+aJJ57AYrFgsVh4+eWXsVgsfPbZZyQnJxMQEMCSJUvYunUrl1xyCXFxcYSGhjJ06FC+/PLLBtf7dfOTxWLhxRdf5NJLLyU4OJhevXrx4YcfNuXbfEyqqTlF0SFm9sOiimovl0RERLztYI2Nfvd+7pV7r39gLMH+J/5Yf+KJJ/j555/p378/DzzwAADr1q0D4K677uKxxx6je/futGvXjp07d3LBBRfwf//3fwQEBPDqq68yfvx4Nm3aRJcux26dmD17No888giPPvooTz31FJMnT2b79u1ERUU1zYs9BtXUnKKokAAA9pVXebkkIiIiJxYREYG/vz/BwcHEx8cTHx+Pj48PAA888AC/+c1v6NGjB1FRUQwaNIgbb7yR/v3706tXLx588EF69OhxwpqXa665hiuvvJKePXvy97//nfLyclasWOH216aamlMUVVdTs081NSIibV6Qnw/rHxjrtXufqpSUlAbfl5eXc//99/PJJ5+Qm5tLbW0tBw8eZMeOHce9zsCBA53bISEhhIeHU1Dg/m4aCjWnKCbUhJqyylqqa+34+6ryS0SkrbJYLI1qAmqufj2K6Y477mDhwoU89thj9OzZk6CgIK644gqqq4//h7yfn1+D7y0WC3a7vcnL+2st951vJsID/fCxWrDZHew/UE1c+MmvWSEiIuIJ/v7+2Gy2Ex63dOlSrrnmGi699FLA1Nzk5OS4uXQnT9UKp8hqtdAu2NTWFKpfjYiItACJiYksX76cnJwcCgsLj1mL0qtXL959912ys7NZs2YNv//97z1S43KyFGqagEZAiYhIS3LHHXfg4+NDv379aN++/TH7yDz++OO0a9eOESNGMH78eMaOHcuQIUM8XNrGU/NTE4gO9Yd8hRoREWkZTjvtNLKyshrsu+aaa444LjExkUWLFjXYN2PGjAbf/7o56mjz5RQXF59UOV2lmpom4BwBVa5QIyIi4i0KNU1AzU8iIiLep1DTBJwT8FWoo7CIiIi3KNQ0gahQNT+JiIh4m0JNE4hR85OIiIjXKdQ0gSiFGhEREa9TqGkC0aFa/0lERMTbFGqaQH1H4ZKDNdTYmu9MiyIiIq2ZQk0TiAzyw2ox2/tVWyMiIm1AYmIic+bM8XYxGlCoaQJWq+XQBHwKNSIiIl6hUNNE1FlYRETEu04q1DzzzDMkJiYSGBhIamoqK1asOOaxL7/8MhaLpcEjMDCwwTEOh4N7772XDh06EBQURFpaGps3b25wTFFREZMnTyY8PJzIyEiuu+46ysvLT6b4blEfarRSt4iINHf//ve/SUhIOGLF7UsuuYRrr72WrVu3cskllxAXF0doaChDhw7lyy+/9FJpG8/lUDN//nzS09O57777WL16NYMGDWLs2LEUFBQc85zw8HByc3Odj+3btzd4/pFHHuHJJ5/kueeeY/ny5YSEhDB27FgqKyudx0yePJl169axcOFCPv74Y7755htuuOEGV4vvNtF1nYVVUyMi0oY5HFBd4Z3HURaSPJbf/va37Nu3j6+++sq5r6ioiAULFjB58mTKy8u54IILyMzM5IcffmDcuHGMHz/+mKt5Nxcur9L9+OOPM336dKZNmwbAc889xyeffMLcuXO56667jnqOxWIhPj7+qM85HA7mzJnD3XffzSWXXALAq6++SlxcHO+//z6TJk1iw4YNLFiwgJUrV5KSkgLAU089xQUXXMBjjz1GQkKCqy+jydUP61aoERFpw2oOwN+99Jn0lz3gH9KoQ9u1a8f555/Pm2++yZgxYwD473//S0xMDKNHj8ZqtTJo0CDn8Q8++CDvvfceH374ITNnznRL8ZuCSzU11dXVrFq1irS0tEMXsFpJS0s7Ygnzw5WXl9O1a1c6d+7MJZdcwrp165zPbdu2jby8vAbXjIiIIDU11XnNrKwsIiMjnYEGIC0tDavVyvLly496z6qqKkpLSxs83EkdhUVEpCWZPHky//vf/6iqMt0m3njjDSZNmoTVaqW8vJw77riDvn37EhkZSWhoKBs2bGhdNTWFhYXYbDbi4uIa7I+Li2Pjxo1HPad3797MnTuXgQMHUlJSwmOPPcaIESNYt24dnTp1Ii8vz3mNX1+z/rm8vDxiY2MbFtzXl6ioKOcxv5aRkcHs2bNdeXmnxLlSt9Z/EhFpu/yCTY2Jt+7tgvHjx+NwOPjkk08YOnQo3377Lf/85z8BuOOOO1i4cCGPPfYYPXv2JCgoiCuuuILq6ub9Gedy85Orhg8fzvDhw53fjxgxgr59+/L888/z4IMPuu2+s2bNIj093fl9aWkpnTt3dtv9tFK3iIhgsTS6CcjbAgMDueyyy3jjjTfYsmULvXv3ZsiQIQAsXbqUa665hksvvRQwLS45OTleLG3juBRqYmJi8PHxIT8/v8H+/Pz8Y/aZ+TU/Pz8GDx7Mli1bAJzn5efn06FDhwbXTEpKch7z647ItbW1FBUVHfO+AQEBBAQENKpMTUFLJYiISEszefJkLrroItatW8dVV13l3N+rVy/effddxo8fj8Vi4Z577jlipFRz5FKfGn9/f5KTk8nMzHTus9vtZGZmNqiNOR6bzcbatWudAaZbt27Ex8c3uGZpaSnLly93XnP48OEUFxezatUq5zGLFi3CbreTmprqyktwm2jNUyMiIi3MueeeS1RUFJs2beL3v/+9c//jjz9Ou3btGDFiBOPHj2fs2LHOWpzmzOXmp/T0dKZOnUpKSgrDhg1jzpw5VFRUOEdDTZkyhY4dO5KRkQHAAw88wBlnnEHPnj0pLi7m0UcfZfv27Vx//fWAGRl122238be//Y1evXrRrVs37rnnHhISEpgwYQIAffv2Zdy4cUyfPp3nnnuOmpoaZs6cyaRJk5rFyCc41FG4+EANtTY7vj6a11BERJo3q9XKnj1H9gFKTExk0aJFDfbNmDGjwffNsTnK5VAzceJE9u7dy7333kteXh5JSUksWLDA2dF3x44dWK2HPtD379/P9OnTycvLo127diQnJ7Ns2TL69evnPObOO++koqKCG264geLiYkaNGsWCBQsaTNL3xhtvMHPmTMaMGYPVauXyyy/nySefPJXX3qQig/2xWMw0AUUHqokNCzzxSSIiItJkLA6HC7P1tGClpaVERERQUlJCeHi4W+4x5MGFFFVUs+C2M+kT7557iIhI81FZWcm2bdvo1q3bEbPlS+Md73105fNbbSRNSMO6RUREvEehpglpAj4RERHvUahpQloqQURExHsUapqQs6ZGK3WLiLQpbaR7qts01funUNOEop2zCqumRkSkLfDx8QFo9ssHNHcHDhwAzAS9p8LtyyS0JWp+EhFpW3x9fQkODmbv3r34+fk1mNJETszhcHDgwAEKCgqIjIx0hsSTpVDThNRRWESkbbFYLHTo0IFt27axfft2bxenxYqMjGz0ckvHo1DThNSnRkSk7fH396dXr15qgjpJfn5+p1xDU0+hpgnV96lR85OISNtitVo1+V4zoMa/JlTfp6b4YA02u3rCi4iIeJJCTRNqd9j6T/sPqLZGRETEkxRqmpCP1UJkkBmOpiYoERERz1KoaWL1nYUL1VlYRETEoxRqmlh0qDoLi4iIeINCTRNzrtStUCMiIuJRCjVN7NBcNQo1IiIinqRQ08SinbMKq0+NiIiIJynUNLEoNT+JiIh4hUJNE6vvKKzmJxEREc9SqGli6igsIiLiHQo1TSwqVKFGRETEGxRqmpizT82Baq3/JCIi4kEKNU0sKtiEGocDirX+k4iIiMco1DQxXx8rkcFa/0lERMTTFGrcwDkBn0KNiIiIxyjUuEG0ZhUWERHxOIUaNzg0AZ9mFRYREfEUhRo3cE7Ap+YnERERj1GocQNNwCciIuJ5CjVuoI7CIiIinqdQ4wbOUFOuPjUiIiKeolDjBjF1fWrU/CQiIuI5CjVuEKU+NSIiIh6nUOMGh3cUtmv9JxEREY9QqHGDdnWhxu6A4oM1Xi6NiIhI26BQ4wZ+PlYigurXf1JnYREREU9QqHETLZUgIiLiWQo1bqLOwiIiIp6lUOMmmoBPRETEsxRq3CQ6VM1PIiIinnRSoeaZZ54hMTGRwMBAUlNTWbFiRaPOmzdvHhaLhQkTJjTYb7FYjvp49NFHncckJiYe8fxDDz10MsX3iOiQ+gn41FFYRETEE1wONfPnzyc9PZ377ruP1atXM2jQIMaOHUtBQcFxz8vJyeGOO+7gzDPPPOK53NzcBo+5c+disVi4/PLLGxz3wAMPNDjulltucbX4HqPmJxEREc9yOdQ8/vjjTJ8+nWnTptGvXz+ee+45goODmTt37jHPsdlsTJ48mdmzZ9O9e/cjno+Pj2/w+OCDDxg9evQRx4aFhTU4LiQkxNXie0x985M6CouIiHiGS6GmurqaVatWkZaWdugCVitpaWlkZWUd87wHHniA2NhYrrvuuhPeIz8/n08++eSoxz700ENER0czePBgHn30UWpra495naqqKkpLSxs8PClKQ7pFREQ8yteVgwsLC7HZbMTFxTXYHxcXx8aNG496zpIlS3jppZfIzs5u1D1eeeUVwsLCuOyyyxrs/+Mf/8iQIUOIiopi2bJlzJo1i9zcXB5//PGjXicjI4PZs2c36p7uUN+nRs1PIiIinuFSqHFVWVkZV199NS+88AIxMTGNOmfu3LlMnjyZwMDABvvT09Od2wMHDsTf358bb7yRjIwMAgICjrjOrFmzGpxTWlpK586dT/KVuK6++Wn/AbP+k9Vq8di9RURE2iKXQk1MTAw+Pj7k5+c32J+fn098fPwRx2/dupWcnBzGjx/v3Ge3282NfX3ZtGkTPXr0cD737bffsmnTJubPn3/CsqSmplJbW0tOTg69e/c+4vmAgICjhh1PaRdsQo3N7qC0sobIuu9FRETEPVzqU+Pv709ycjKZmZnOfXa7nczMTIYPH37E8X369GHt2rVkZ2c7HxdffDGjR48mOzv7iJqTl156ieTkZAYNGnTCsmRnZ2O1WomNjXXlJXiMv6+VsECTGdUEJSIi4n4uNz+lp6czdepUUlJSGDZsGHPmzKGiooJp06YBMGXKFDp27EhGRgaBgYH079+/wfmRkZEAR+wvLS3lnXfe4R//+McR98zKymL58uWMHj2asLAwsrKyuP3227nqqqto166dqy/BY6JD/CmrrGVfeTU92nu7NCIiIq2by6Fm4sSJ7N27l3vvvZe8vDySkpJYsGCBs/Pwjh07sFpdn9Nv3rx5OBwOrrzyyiOeCwgIYN68edx///1UVVXRrVs3br/99gZ9Zpqj6NAAcvYd0AR8IiIiHmBxOBwObxfCE0pLS4mIiKCkpITw8HCP3HP6q9+zcH0+/3dpfyandvXIPUVERFoTVz6/tfaTG0XXr9StuWpERETcTqHGjbRUgoiIiOco1LhRdKgm4BMREfEUhRo3cjY/qaOwiIiI2ynUuJHWfxIREfEchRo3igrRSt0iIiKeolDjRvXrPxVVVNNGRs6LiIh4jUKNG9XX1NTaHZQerPVyaURERFo3hRo3CvD1ISygfv0ndRYWERFxJ4UaN4sKVb8aERERT1CocbP6JqhCjYASERFxK4UaN4sOMRPwqaZGRETEvRRq3EwT8ImIiHiGQo2b1fep0VIJIiIi7qVQ42bRmoBPRETEIxRq3ExLJYiIiHiGQo2baaVuERERz1CocTN1FBYREfEMhRo3O3xRS63/JCIi4j4KNW5WH2pqbA5KK7X+k4iIiLso1LhZoJ8PoXXrP2kElIiIiPso1HhAlPrViIiIuJ1CjQdoWLeIiIj7KdR4gCbgExERcT+FGg9w1tQo1IiIiLiNQo0HOCfgU/OTiIiI2yjUeIAm4BMREXE/hRoPUPOTiIiI+ynUeEBUqEY/iYiIuJtCjQfEhJg+NRr9JCIi4j4KNR5QX1Oj9Z9ERETcR6HGA+o7Clfb7JRXaf0nERERd1Co8YBAPx+C/X0ANUGJiIi4i0KNh9SPgCpUZ2ERERG3UKjxkC5RwQD8sGO/l0siIiLSOinUeMj5/eMB+GjNHi+XREREpHVSqPGQCwZ0wMdqYc2uEnIKK7xdHBERkVZHocZDokMDGNkzBoAPVVsjIiLS5BRqPOjiQQmACTWar0ZERKRpKdR40NjT4/D3tbKloJwNuWXeLo6IiEirclKh5plnniExMZHAwEBSU1NZsWJFo86bN28eFouFCRMmNNh/zTXXYLFYGjzGjRvX4JiioiImT55MeHg4kZGRXHfddZSXl59M8b0mLNCPMX1iATVBiYiINDWXQ838+fNJT0/nvvvuY/Xq1QwaNIixY8dSUFBw3PNycnK44447OPPMM4/6/Lhx48jNzXU+3nrrrQbPT548mXXr1rFw4UI+/vhjvvnmG2644QZXi+919U1QH63Zg92uJigREZGm4nKoefzxx5k+fTrTpk2jX79+PPfccwQHBzN37txjnmOz2Zg8eTKzZ8+me/fuRz0mICCA+Ph456Ndu3bO5zZs2MCCBQt48cUXSU1NZdSoUTz11FPMmzePPXtaVo3H6D6xhAb4srv4IKs1Z42IiEiTcSnUVFdXs2rVKtLS0g5dwGolLS2NrKysY573wAMPEBsby3XXXXfMY77++mtiY2Pp3bs3N910E/v27XM+l5WVRWRkJCkpKc59aWlpWK1Wli9fftTrVVVVUVpa2uDRHAT6+XDe6XGAmqBERESakkuhprCwEJvNRlxcXIP9cXFx5OXlHfWcJUuW8NJLL/HCCy8c87rjxo3j1VdfJTMzk4cffpjFixdz/vnnY7PZAMjLyyM2NrbBOb6+vkRFRR3zvhkZGURERDgfnTt3duWlutUlSR0B+OTHXGptdi+XRkREpHXwdefFy8rKuPrqq3nhhReIiYk55nGTJk1ybg8YMICBAwfSo0cPvv76a8aMGXNS9541axbp6enO70tLS5tNsBnZI5roEH/2VVSzdOs+zj6tvbeLJCIi0uK5FGpiYmLw8fEhPz+/wf78/Hzi4+OPOH7r1q3k5OQwfvx45z673dRM+Pr6smnTJnr06HHEed27dycmJoYtW7YwZswY4uPjj+iIXFtbS1FR0VHvC6aPTkBAgCsvz2N8faxcMKADr323nQ+z9yjUiIiINAGXmp/8/f1JTk4mMzPTuc9ut5OZmcnw4cOPOL5Pnz6sXbuW7Oxs5+Piiy9m9OjRZGdnH7PmZNeuXezbt48OHToAMHz4cIqLi1m1apXzmEWLFmG320lNTXXlJTQbFyeZUVCfr8ujssbm5dKIiIi0fC43P6WnpzN16lRSUlIYNmwYc+bMoaKigmnTpgEwZcoUOnbsSEZGBoGBgfTv37/B+ZGRkQDO/eXl5cyePZvLL7+c+Ph4tm7dyp133knPnj0ZO3YsAH379mXcuHFMnz6d5557jpqaGmbOnMmkSZNISEg4ldfvNcld2tExMojdxQf5amMB5w/o4O0iiYiItGguD+meOHEijz32GPfeey9JSUlkZ2ezYMECZ+fhHTt2kJub2+jr+fj48OOPP3LxxRdz2mmncd1115GcnMy3337boPnojTfeoE+fPowZM4YLLriAUaNG8e9//9vV4jcbVquFiwaZIKNRUCIiIqfO4mgjixCVlpYSERFBSUkJ4eHh3i4OAOv2lHDhk0vw97Wy6u40wgL9vF0kERGRZsWVz2+t/eRF/TqE06N9CNW1dr5Yl3/iE0REROSYFGq8yGKxOOes+UBNUCIiIqdEocbL6teCWrqlkH3lVV4ujYiISMulUONliTEhDOwUgc3u4NO1je9gLSIiIg0p1DQD9bU1GgUlIiJy8hRqmoGLBiZgscDKnP3sLj7o7eKIiIi0SAo1zUB8RCCp3aIA+Ei1NSIiIidFoaaZuHiQGQX1YbZCjYiIyMlQqGkmzu8fj6/VwvrcUrYUlHm7OCIiIi2OQk0z0S7E37lat2prREREXKdQ04zUr9z9n2U5LN1S6OXSiIiItCwKNc3IuP7xpHRtR1llLVPmruC1rBxvF0lERKTFUKhpRgJ8fXj9+lQuG9wRm93BPR+s494PfqLWZvd20URERJo9hZpmJtDPh3/8bhB3juuNxQKvZm3nmv+spORAjbeLJiIi0qwp1DRDFouFm8/pyfNXJRPs78OSLYVc+q+l/LK33NtFExERabYUapqx806P579/GEHHyCB+KaxgwjNLWbJZHYhFRESORqGmmeuXEM77M0YypEskpZW1TP2POhCLiIgcjUJNC9A+LIA3p5/RoAPxPe//RI06EIuIiDgp1LQQ9R2I/zyuDxYLvPbddq59eSUlB9WBWEREBBRqWhSLxcJN5/RwdiD+dnMhlz+7jB37Dni7aCIiIl6nUNMCnXd6PG/fOJz48EC2FJRzyTNLWJlT5O1iiYiIeJVCTQvVv2MEH8wcycBOEew/UMPkF5bzv1W7vF0sERERr1GoacHiwgOZf8Nwxp0eT7XNzp/eWcNjn2/Cbnd4u2giIiIep1DTwgX5+/CvyUO4+ZweADz91RZueesHDlbbvFwyERERz1KoaQWsVgt3juvDY78dhJ+PhU/W5jLp31kUlFZ6u2giIiIeo1DTilyR3InXr0slMtiPNbtKmPDMUtbvKfV2sURERDxCoaaVSe0ezfs3j6R7+xD2lFRy+bPL+GjNHm8XS0RExO0UalqhxJgQ3rtpJKN6xnCwxsYtb/3AAx+t1wzEIiLSqinUtFIRwX68cu0wZwfiuUu3MfmF5RSUqZ+NiIi0Tgo1rZhPXQfi565KJjTAlxU5RVz05BK+10R9IiLSCinUnKrKUshdAwUbvV2SYxrXP54PZo6kV2woBWVVTPr3d/xn6TYcDs1nIyIirYdCzala/So8fxZ884i3S3JcPdqH8v6MkVw4sAO1dgezP1rPbfOzOVBd6+2iiYiINAmFmlMVnmC+ljb/EUYhAb48feVg7r6wLz5WCx9k7+Gyfy0jp7DC20UTERE5ZQo1pyq8o/lautu75Wgki8XC9Wd2543rU4kJ9WdjXhnjn17CG8u3U1WrWYhFRKTlUqg5VfU1NWV5YG85Q6bP6B7Nx7ecyZAukZRV1vLX937inEe/5uWl26isUbgREZGWR6HmVIXFAxawVcOBfd4ujUviIwKZd8Nw7r6wL7FhAeSWVHL/R+sZ9fBX/PubrVRUqb+NiIi0HAo1p8rHD0JjzXYLaYI6nL+vlevP7M43d47mwUtOp2NkEIXlVfz9042MengRTy/aTGlljbeLKSIickIKNU2hBXUWPpZAPx+uHp7IV3ecwyOXD6RrdDD7D9Tw2Bc/M/KhRTz+xSb2V1R7u5giIiLHpFDTFFpYZ+Hj8fe18ruhnclMP5s5E5PoGRtKWWUtTy7awqiHF5Hx2QYKy6u8XUwREZEjKNQ0BWdn4VzvlqMJ+fpYmTC4I1/cdhbPTh5Cvw7hVFTbeH7xL4x6eBF/+3g9BaVackFERJqPkwo1zzzzDImJiQQGBpKamsqKFSsadd68efOwWCxMmDDBua+mpoY///nPDBgwgJCQEBISEpgyZQp79jRsyklMTMRisTR4PPTQQydT/KbXCpqfjsVqtXD+gA588sdRvDglhYGdIqissfPikm2c+chX3P/hOnJLDnq7mCIiIq6Hmvnz55Oens59993H6tWrGTRoEGPHjqWgoOC45+Xk5HDHHXdw5plnNth/4MABVq9ezT333MPq1at599132bRpExdffPER13jggQfIzc11Pm655RZXi+8eraj56VgsFgtp/eL4YMZIXp42lCFdIqmqtfPyshzOfuRr/vreWnbtP+DtYoqISBtmcbi4AFBqaipDhw7l6aefBsBut9O5c2duueUW7rrrrqOeY7PZOOuss7j22mv59ttvKS4u5v333z/mPVauXMmwYcPYvn07Xbp0AUxNzW233cZtt93mSnGdSktLiYiIoKSkhPDw8JO6xjFt+xZeuQiie8Itq5r22s2Uw+Fg2dZ9PJG5mRXbzAKZvlYLlw/pxNXDu9KvQzhWq8XLpRQRkZbOlc9vl2pqqqurWbVqFWlpaYcuYLWSlpZGVlbWMc974IEHiI2N5brrrmvUfUpKSrBYLERGRjbY/9BDDxEdHc3gwYN59NFHqa099jwqVVVVlJaWNni4zeHNT21kkUiLxcLInjG8feNw5t1wBiN7RlNrdzD/+51c9NQShv7fl/zxrR94+/udap4SERGP8HXl4MLCQmw2G3FxcQ32x8XFsXHj0VepXrJkCS+99BLZ2dmNukdlZSV//vOfufLKKxsksj/+8Y8MGTKEqKgoli1bxqxZs8jNzeXxxx8/6nUyMjKYPXt2417YqaoPNTUHoLIYgtp55r7NxBndozmjezSrthfx/OJfWLKlkH0V1Xy4Zg8frjH9jHrGhjKqZwxn9orhjO7RhAS49E9PRETkhNz6yVJWVsbVV1/NCy+8QExMzAmPr6mp4Xe/+x0Oh4Nnn322wXPp6enO7YEDB+Lv78+NN95IRkYGAQEBR1xr1qxZDc4pLS2lc+fOp/BqjsMvCIKi4GARlOa2uVBTL7lrFP+eEkV1rZ3VO/azZHMh324pZO2uYrYUlLOloJyXl+Xg52MhuWs7rhzWhQsGdMDPR4PwRETk1LkUamJiYvDx8SE/P7/B/vz8fOLj4484fuvWreTk5DB+/HjnPnvd+ki+vr5s2rSJHj16AIcCzfbt21m0aNEJ281SU1Opra0lJyeH3r17H/F8QEDAUcOO24R3rAs1eyCun+fu2wz5+1qdtTd3jO1NyYEalm01AWfJ5kJ2FB3gu1+K+O6XIh7+bCPXjExk0rAuhAf6ebvoIiLSgrkUavz9/UlOTiYzM9M5LNtut5OZmcnMmTOPOL5Pnz6sXbu2wb67776bsrIynnjiCWfNSX2g2bx5M1999RXR0dEnLEt2djZWq5XY2FhXXoL7hHeA/LWtegTUyYoI9uP8AR04f0AHALbvq+CD7D28mpXDnpJK/v7pRp7M3MKkoZ2ZNqobHSODvFxiERFpiVxufkpPT2fq1KmkpKQwbNgw5syZQ0VFBdOmTQNgypQpdOzYkYyMDAIDA+nfv3+D8+s7/9bvr6mp4YorrmD16tV8/PHH2Gw28vLyAIiKisLf35+srCyWL1/O6NGjCQsLIysri9tvv52rrrqKdu2aSVNPK56rpql1jQ7hj2N6ccNZ3fkgezcvfruNzQXlvLhkG/9ZlsMFAzow/cxuDOwU6e2iiohIC+JyqJk4cSJ79+7l3nvvJS8vj6SkJBYsWODsPLxjxw6s1sb3kdi9ezcffvghAElJSQ2e++qrrzjnnHMICAhg3rx53H///VRVVdGtWzduv/32Bn1mvK4NzFXT1AL9fJg4tAu/Te7M4s17efHbX1i6ZR8frdnDR2v2MKxbFDee1Z1z+8RisWh4uIiIHJ/L89S0VG6dpwbgh9fhgxnQMw2u+l/TX7+NWLenhJe+3caHa/ZQazf/NPt3DOe2Macxpq/CjYhIW+O2eWrkONT81CROT4jg8YlJfPvn0dx4VneC/X34aXcp17/6PeOfXsLC9fm0kRwuIiIuUqhpKmH1oUbNT02hQ0QQsy7oy5I/n8tN5/Rwhpvpr37PRU8t4Yt1eQo3IiLSgEJNU6mvqaksgapy75alFYkK8efP4/qw5M/ncvM5PQjx92HdnlJueG2Vwo2IiDSgPjVN6e+doLoMZn4PMb3cc482bn9FNS8u+YWXl+ZQUW0DoF+HcC4b0pERPWLoEx+mNadERFoRVz6/NVd9UwpPgMJNpglKocYt2oX48//G9uH6Ud2d4WZ9binrPzFre0WF+DO8RzQje8Qwsmc0XaKC1blYRKSNUKhpSs5Qk+vtkrR6h4eb/63exZIthazYVkRRRTWf/JjLJz+an0HHyCBG9oxmZM8YhnePJjY80MslFxERd1GoaUrh6izsae1C/Ln+zO5cf2Z3qmvtrNlVzNIthSzbso8fdu5nd/FB3v5+F29/vwuATu2CGNylHUO6RDKkSzv6JYRr7SkRkVZCoaYpaVi3V/n7WhmaGMXQxChuS4MD1bWszNnPsi2FLN1ayLo9pezaf5Bd+w/yUd3q4QG+VgZ2imBIl3YMrgs6qs0REWmZFGqakkJNsxLs78vZp7Xn7NPaA1BeVcuancWs3r6f1Tv288POYooP1LAyZz8rc/Y7z/PzsRDs70togC/B/j6EBPgSEuBDiL+vczsq2J+RPWNI7toOX9X0iIg0Cwo1TUlLJTRroQG+jOwZw8ieMQA4HA62FVawekcxq3fsZ/X2/fycX0aNzUHJwRpKDtYc93pPLtpCu2A/RveJ5bx+cZzZqz0hAfovJSLiLfoN3JTqa2rK1FG4JbBYLHRvH0r39qFckdwJgIPVNooPVlNRVUtFlc18ra7/Wuvcv6PoAF9tKmD/gRreXb2bd1fvxt/Xysge0aT1iyOtbxxxasYSEfEohZqmVD+rcMVeqK0C3wDvlkdcFuTvQ5B/UKOOrbXZ+X77fhauz2fh+vy6oLOXrzbt5a/v/cSgThGMH5TAlcO6qAZHRMQDNPleU3I44G9xYKuCW9dAu0T33EeaHYfDweaCcmfAyd5Z7HyuXbAf143qxpQRiYQH+nmvkCIiLZArn98KNU3tiSTYvw2mfQZdR7jvPtKsFZRW8vn6fF769hdy9h0AICzQl2tGJDJtZDeiQvy9XEIRkZZBq3R7k7OzsEZAtWWx4YFcfUZXvkw/mycmJdErNpSyylqeWrSFUQ8v4u+fbqCgrNLbxRQRaVXU0N/UNKxbDuPrY+WSpI6MH5jAF+vzeGrRFtbtKeXf3/zCK8tymDS0Mzee3YMOEYHU2BxU1tqorLZRWWOnstbGwWoblTU2KmvtOBwOwgJ9CQ3wIzTQl7BAX0L8ffHRWlciIoBCTdML72C+KtTIYaxWC+P6d2Ds6fF8vWkvTy7azA87inklazuvfrcdC2A/yYbgEH8fwgIPBZ3BndsxYXACAzpGaN0rEWlTFGqamuaqkeOwWCyM7hPLOb3bk7V1H08t2kLWL/twNDgGgvx8CPTzIdDXSqC/D4G+PgBUVNdSXllLWWUt1TZ73T6bWbHcrOnJDzuKmbt0G91jQrgkqSMTBifQNTrEw69URMTzFGqampqfpBEsFgsjesYwomcMBaWVOIBAXx8C/a34+1gbVcNSVWujvLKW8ioTcsoqayksr+KL9fksXJ/HL4UV/PPLn/nnlz+T1DmSCUkJXDQogZhQTTUgIq2TQk1TU6gRF53sWlMBvj4EhPoQ/auQMn5QAuVVtXz+Ux7vZ+9m6ZZCsncWk72zmAc/2cConjGMH5TA0MR2dIkKVhOViLQaCjVNrb75qTwPbLXgo7dYPC80wJfLkztxeXInCsoq+XhNLh9k72bNrhIW/7yXxT/vBSAy2I8BHSMY2CmCgZ0iGdQpkvgIzYQsIi2T5qlpanYbPNgeHDZI33Co5kakGfhlbzkfZO/h600FbMgtc/bLOVxsWAADO0XWBZ0IBnWKpJ3m1RERL9Hke0fhsVAD8PjpULoLrs+ETinuvZfISaqqtbEpr4w1u0pYu6uYH3eV8HN+2VFHYXWOCmJgx0hnjU7/juGEaXZkEfEAVz6/1TbiDuEJJtSU7gYUaqR5CvD1qauRiQS6AnCgupZ1e0pZs9OEnLW7S9hWWMHOooPsLDrIJ2vNYq0WC3SPCWFQXY3OOb1jSYzRCCsR8S6FGndQZ2FpoYL9fRmaGMXQxCjnvpIDNfy0p4Q1u4pZu6uEH3eVsLv4IFv3VrB1bwXv/rAbPlpPn/gwxp4ez7j+8fSJD3OpA3J+aSUrc4rI3lFMRXUttTYHNocDu92BzYH5aj+0LyTAl0uHdOTsXu2xavJBEamjUOMOmqtGWpGIYD9G9oxhZM8Y577C8ipnwFmZU8R3v+xjY14ZG/PKeCJzM12jgxl3ejxj+8eT1CmyQfBwOBxs3VvOypz9rMwpYmVOETuLDrpcrg/X7KFbTAhTh3fl8uROag4TEfWpcYtlT8EXd0P/K+CKl9x7L5FmoPhANZkbCliwLo9vft5LVe2hDshx4QGMPT2ejpFBfL99P9/nFLH/QE2D860W6NshnJSu7YgJDcBqteBjteBrtWC1mG2r1YKPxYKPFX7OL+ft73dSVlkLmNFeVyR3YuqIRLqpGUykVVFH4aPwaKj56X/w32uhywi49jP33kukmamoqmXxz3tZ8FMeizYWUF5Ve8QxAb5WBneJZGhiFCmJUQzpEulyTUtFVS3v/rCbl5duY+veCuf+0b3bc83IbpzZM+aIGqLSyloKSivJL60iv7SS/LJKSg7W0KldML3jwjgtLpTIYI30EmlOFGqOwqOhZsd3MHcsRHaF2350771EmrGqWhvLtu7ji3V57K+oYUjXSFISo+ifEIG/r7VJ7mG3O1iypZBXluWwaFMB9b/RuseE0C8hnILSKvLLKskvraSy5sgh7L/WPiyA0+JC6RUbxml1QadXXBgRQWreEvEGhZqj8GioKd4BcwaAjz/8NR+sTfPLW0SOL6ewgleztvPO9zspO0oNEUBEkB9x4QHEhQcSGxZIWKAv2/dV8HN+ObuLj923p3v7ECYN7cxvkztr3h4RD1KoOQqPhpraavhbLOCAO7ZAaHv33k9EGiivquWTH/dQVllLfEQgceGBxIUFEhseQKCfz3HP25xfxub8cn7OL+PngnI255eRW1LpPMbf18pFAzow+YyuDOkSqWUmRNxMoeYoPBpqAB7tBRUFcMNiSEhy//1ExG1KDtTw6U+5vP7ddtbtKXXu79shnMmpXZgwuCOhARpMKuIOCjVH4fFQ8/zZkJsNV86D3ue7/34i4nYOh4M1u0p4/bvtfLRmj3OUV4i/DxMGd+SqM7rSt4MHfr+ItCGaUbg5CO9oQo3mqhFpNSwWC0mdI0nqHMndF/blf6t388by7fyyt4I3lu/gjeU7CPLzITrUn+jQAGJC/J3b0SH+xIQGEB3qT2xYIF2jg4/bFCYirlOocRfNKizSqkUG+3PdqG5cOzKRrK37eGP5Dj5fl8fBGhu79h9k1/4TTyiYEBFIt/YhdIsJoVtMKN1jQkiMCaFTuyD8fDTAQMRVCjXuolAj0iZYLBZG9IxhRM8YDlbbKCirpLC8mn3lVRRVVLOvoprC8ir2lVezr8J83VN8kNLKWvaUVLKnpJKlW/Y1uKav1UKXqGBCA33N8hB2B7V2s0REbf2SEXXbDoeDIH8fQvx9CQnwISTAl2B/8zXE35fgAB9C/X0JD/IjOtTUFsXUfY0I8mtUR+fKGhvFB2ooPlhN8YEa7A4H/TqEa04faXYUatzFGWrU/CTSVgT5+9A1OoSu0cef1djhcLD/QA3bCsv5ZW8FOfsq2FZY4dyurLHzS2HFca/RgAuHHs7XanEGnejQAKKC/aissTvDS32QOdb8Pl2jgxnQMYKBnSIY0FGrt4v3KdS4izPU5Hq3HCLS7FgsFqJC/IkKiSK5a1SD5+x2B3mlleQUVlBZa8NqseBrteJTt3RE/fIR9dsWCxyotnGgykZFdS0Hqmspr7JxoKqWiur6r7UUH6hhX7mpNdpbXkVZZS21dkfd7MpVJyyzj9VCu2A/IoL8qLbZ2Vl0kO37DrB93wE+/rHh6u0DO0UyoGME8RGB+PlY8fOx4O9jxd/XWve9FX9fC34+VkIDfIkODXDL+yxtj0KNuzgXtdwDDof53y4icgJWq4WEyCASIoPcep+qWpsz5Owrr2ZveRX7K6oJ9vchItifdsF+RAb5ExnsR2SwH6EBvg2aqooPVPPT7lLn6u1rdzdcvf29HxpfSz2gYwSXJCVw0cAE4iMC3fFypY3QkG53qT4Af+9gtv+8HYIi3X9PEREvOnz19rW7Syg9WEO1zU6N8+GgutZ+aF+tnQM1NufSFhYLnNEtmkuSEji/fwcigpuuKavGZmfr3nI25JaSW1KJr9XUgPn5WPD1MTVhfj6H7bNa6RkbStfoYE2w6GVun6fmmWee4dFHHyUvL49Bgwbx1FNPMWzYsBOeN2/ePK688kouueQS3n//fed+h8PBfffdxwsvvEBxcTEjR47k2WefpVevXs5jioqKuOWWW/joo4+wWq1cfvnlPPHEE4SGhjaqzB4PNQAPJ8LB/XBTFsT188w9RURakH3lVXy6NpcPsvfw/fb9zv1+PhbO6R3LJUkJjOkTR5B/44e/lxysYUNuKev3lJqvuaVszi+n2nbitb9+rVO7IM7s1Z4ze8Uwokd0ozpH2+0Otu4tZ0VOEd/n7Of77UX4Wq2c0T2aUT1jGN4jmigttdFobg018+fPZ8qUKTz33HOkpqYyZ84c3nnnHTZt2kRsbOwxz8vJyWHUqFF0796dqKioBqHm4YcfJiMjg1deeYVu3bpxzz33sHbtWtavX09goKmKPP/888nNzeX555+npqaGadOmMXToUN58881GldsroeZfI6BgHUz+H/RK88w9RURaqF37D/DRmlw+yN7Nxrwy5/4Qfx8GdIowfYgw/YgsFgsWTO2OtW671u5gS8Gx1/AKC/Clb4dwOkcFY3c4qLHZsdkd1Ngc1Nrt1NoO7austbEpr4wa26GPSKsFBnSK5MyeMYzqFcOQLu3w97VSXWvnpz0lrNxWxMqc/azaXsT+AzXHfa2nJ4QzsqcJSsO6RRHs3zx7gzgcDlbv2M++8uq6mqsQfKyerblya6hJTU1l6NChPP300wDY7XY6d+7MLbfcwl133XXUc2w2G2eddRbXXnst3377LcXFxc5Q43A4SEhI4E9/+hN33HEHACUlJcTFxfHyyy8zadIkNmzYQL9+/Vi5ciUpKSkALFiwgAsuuIBdu3aRkJBwwnJ7JdS8fgVsWQgXPwVDpnjmniIircCmvDI+XLObD7L3NGrOn1/rGBlEv4Rw+nUIp2+HcE5PCKdTuyCXmpIqqmpZsa2IbzbvZcnmQjYXlDd4Ptjfh16xoWzKLztihFign5XBndsxNLEdKYlR1NjsLN2yj6VbCtmUX9bgWD8fC4O7tOOM7tEE+/tQVWOnqtZGZd3Xqlq7edSYbT8f0zR2Wlwop8WF0aN9qEs1WY310+4SMj7b0GDKAX9fK91jQjgtLoxesaH0qlvFvmtUML5umlvJbTMKV1dXs2rVKmbNmuXcZ7VaSUtLIysr65jnPfDAA8TGxnLdddfx7bffNnhu27Zt5OXlkZZ2qCYjIiKC1NRUsrKymDRpEllZWURGRjoDDUBaWhpWq5Xly5dz6aWXHnHPqqoqqqoO9egvLS094hi301w1IiInpXd8GP8vvg93nNeb7J3F7Cg6AJhxF3aHA4cDHJhtHODA/H3eNTqEvvHhTdIfJyTAl9F9Yhndx7RC5JYcZMnmQr7dXMjSLYXsq6hmza4SANoF+5GSGMXQxHYMTYzi9IQI/H0bfsiP6RsHQEFZJVlbTcBZumUfu4sPsmJbESu2FTW6bF9uyHduWyzQuV0wp8WF0jM2jNPiQukVG0bv+LAjytAYu4sP8o/PN/Fe9m4cDvCvC1G/FJZTWWNnY15Zg5o0MMd0iwnh4qQEZozu6fI9m4pLoaawsBCbzUZcXFyD/XFxcWzcuPGo5yxZsoSXXnqJ7Ozsoz6fl5fnvMavr1n/XF5e3hFNW76+vkRFRTmP+bWMjAxmz559wtfkVs4RUJqrRkTkZFgsphZjcJd23i4KHSKC+G1KZ36b0hm73cH63FK27i3n9IRwuseEYm1ks0xsWCCXJHXkkqSOOBwOtu87wNKthfywoxiAAF8rAb4+BPhZD237WgnwsxLo60NFda1zJfnNBeUUVVSzo+gAO4oO8OWGAud9wgJ9GdMnlvNOj+fs09oTcoJFV0sra/jXV1uZu3Qb1XXrml2SlMAd5/U2TXZ2B7v2H2Rzgbnvz/llbCkoZ3N+OQdrbGzKL2Nv2YmnB3AntzbilZWVcfXVV/PCCy8QExPjzlsdYdasWaSnpzu/Ly0tpXPnzh4tg2pqRERaJ6vVQv+OEfTvGHFK17FYLCTWLY8xObXrSV1jX3kVP+eXm7BRF3Y25ZdRfKCG97P38H72HgJ8rZzZK4bzTo8nrW9cg47K1bV2Xv9uO08t2uzsC3RG9yj+ckFfBnaKbPCau0QH0yU62FnrBKZj9O7ig2wpKCcu3LtD8l0KNTExMfj4+JCfn99gf35+PvHx8Uccv3XrVnJychg/frxzn91u0p+vry+bNm1ynpefn0+HDh0aXDMpKQmA+Ph4CgoOpU+A2tpaioqKjnpfgICAAAICvDyhU3jd61GoERERN4kODWB4aADDe0Q799nsDn7YsZ/P1+Xx+bp8Zy3OlxsKsFpgaGIUY0+Pp12IH3O+3Mz2faZ5r2dsKLPO78O5fWIb3f/IarXQOSqYzlHBbnl9rnAp1Pj7+5OcnExmZiYTJkwATEjJzMxk5syZRxzfp08f1q5d22Df3XffTVlZGU888QSdO3fGz8+P+Ph4MjMznSGmtLSU5cuXc9NNNwEwfPhwiouLWbVqFcnJyQAsWrQIu91Oamqqq6/Zc1xtfirfC69eDJ2GwsVPuq9cIiLSqvlYLaQkRpGSaGpcNuWX8flP+Xy+Lo/1uaUs31bE8sP68MSEBpD+m9P4XUont3X49QSXm5/S09OZOnUqKSkpDBs2jDlz5lBRUcG0adMAmDJlCh07diQjI4PAwED69+/f4PzIyEiABvtvu+02/va3v9GrVy/nkO6EhARncOrbty/jxo1j+vTpPPfcc9TU1DBz5kwmTZrUqJFPXlPf/FRZAtUV4H/89WBY/iwUrDePgRMhcaT7yygiIq2axWKhT3w4feLDuTWtFzuLDvD5ujy+WJ/PrqID/DalMzec1f2EfW5aApdfwcSJE9m7dy/33nsveXl5JCUlsWDBAmdH3x07dmC1upby7rzzTioqKrjhhhsoLi5m1KhRLFiwwDlHDcAbb7zBzJkzGTNmjHPyvSefbOa1GQHh4B8K1eVmDaiY4/QIryqHlS8d+j5zNlz7uZZXEBGRJtU5Kpjrz+zO9Wd293ZRmpyWSXC3p4dC4c8w5UPofvaxj1v+PHx2J0R0gYq9UHsQrpwPvcd5rqwiIiLNjCuf3y234aylCGtEZ2FbLWSZyQwZdRuk3mC2Fz0Idten9RYREWmLFGrcrTGdhTd8CMU7IDgakn4PI2+DgAjI/wl++p9HiikiItLSKdS4W31n4bLcoz/vcMCyur5Bw24AvyAIjoKRt5h9X/0f2I6/hoiIiIgo1LjfiSbg274U9vwAvoEw9PpD+1NvgpD2sH8brH7V/eUUERFp4RRq3O1EzU/LnjJfk34PIYfNuhwQCmf9P7O9+BGoPuC+MoqIiLQCCjXudrxZhQs2ws8LAAsMP3LyQpKvgcguUJ4HK553ZylFRERaPIUad6uvqanYC7W/WuirfsRTnwshuseR5/oGwDl/MdtL5sDBYneVUkREpMVTqHG34GjwqVs47PDOwmV58ON8sz3y1mOfP/B30L4vVBYf6lAsIiIiR1CocTeL5bDOwoeFmhX/Bls1dE6FzsOOfb7VB86922x/9yyU5R/7WBERkTZMocYTft1Z+PAlEUb88cTn97kQOqZAzQH49jH3lFFERKSFU6jxhF/PKvzD66Y5KaoH9D7/xOdbLDDmXrP9/X9gf447SikiItKiKdR4wuFz1dhq4btnzPfDZ5jmpcbofjZ0PwfsNfD1Q24ppoiISEumUOMJhzc/bfig4ZIIrqivrVkzD/LXN20ZRUREWjiFGk84vKZm6a+WRHBFx2ToezHggEV/a9IiNomyfPj4dq1XJSIiXqFQ4wn1NTV7foDc7LolEaaf3LXOvRssVtj0Cax6Gey2pirlqcn7CV4cA9/Phff+YGqjREREPEihxhPqa2ocdQEkaTKERJ/ctdr3hiFTzPZHt8LzZ8HmhWZhTG/5+XOYOxZKdprvbdXwVYb3yiMiIm2SQo0nhMaCpb5DsMV0ED4V5z8CabMhIALyf4I3roBXxsPuVadcVJc4HGbunLcmQXU5JJ4JV71rnlvzFuSv82x5RESkTVOo8QSrD4TFm+2+Fx19SQRX+AbAqNvg1mwYcQv4BEDOt/DCufD2VNi39VRLfGK2GvjkT7DgLnDYYfDVJtD0HAOnXwo44MvZ7i+HiIhIHYUaT+k6wvSlGZXedNcMjoLz/ga3rDJNWlhg/fvwzDD4ON19sw8fLIY3fgvfv2Tu+ZsH4eKnwLduOYhz7wGrL2z+HHKWuKcMIiIiv2JxOLzZGcNzSktLiYiIoKSkhPDwcM8XwFYDVWUmiLhL/jpTO7L5c/O9XwiMnmVqc5pK0TZ4cyIUbgK/YLjsBVP79Guf/AlWvmhmQr7+SzOBoIiIiItc+fxWTY2n+Pi5N9AAxJ0Ok9+Gaz6pW1ahAr64G7ZnNc31t2eZEU6Fm8wsydM+O3qgATjrThOqdn8PGz5qmvuLiIgch0JNa5Q4ytSOJF1lvl/04KmPjlr3Hrx6MRzYBx0GwfRFkJB07OPD4g51iM6cbWZSFhERcSOFmtbKYoHRfzGdiLcvha2LTv5apXvg/RlmqHafi0wNTf0w9eMZcYuZOXnfFvjhtZO/v4iISCMo1LRmER1h6PVm+1Rqa7683zRldRoKv3sN/EMad15guGmGArNeVXXFyd1fRESkERRqWrtRt5u+LXt+gI0fu37+juXw43zAAuc/DFYX/8mkTIPIrlCeZ+a0ERERcROFmtYutD2ccZPZXvR/ri2rYLfDZ3U1LYMnm7WnXOUbYIZ4Ayx9Air2uX4NERGRRlCoaQtG3AKBEbB3g2uLTWa/btaqCgiHMfed/P37Xw7xA6CqFL79x8lfR0RE5DgUatqCoEgY8Uez/dXfzZw5J3Kw+NCMwGf/2Sz1cLKsVrOsA8DKF2D/9pO/loiIyDEo1LQVqX+AkPawfxv88PqJj1/8CBwohJjTYNgNp37/HudCt7PrFrv8+6lfT0RE5FcUatqKgFA4809m+5tHoaby2Mfu3QQrnjfb4zIOLX9wKiwWSLvfbP84H/LWnvo1RUREDqNQ05YkT4PwjlC6G76fe/RjHA6zSKW9Fk47H3qmNd39Ow6B0y9Di12KiIg7KNS0JX6BcHbdaKZv/wFV5Uces+kzM1Gfjz+M/b+mL8O5d5vFLrcshJylTX99ERFpsxRq2pqkyRDV3fSXWf5cw+dqKuHzWWZ7+AyI7tH094/uAYOvNtvLNW+NiIg0HYWatsbHD875i9le+iQc3H/oue+egf05ZrHKM+9wXxnqOx5v/BTK8tx3HxERaVMUatqi/pdDbD+oKoFlT5l9pXvgm7o5ZNJmm47F7hLXDzqfAQ6b1oQSEZEmo1DTFlmtMPqvZvu756B8Lyy8r259p2Ew8HfuL0PKNPN11auuzXIsIiJyDAo1bVWfCyFhiAky/7sW1r4NWOCCR8zwa3frdwkERkLJDtiS6f77iYhIq6dQ01ZZLDCmbk2mbd+Yr4OvgoTBnrm/XxAk/d5sr/qPZ+4pIiKtmkJNW9Z9NHQdZbZPdX2nk5F8jfn68wIo2e3Ze4uISKujUNOWWSxw/sMQPxAu+qdZ0duT2veGriPBYVeHYREROWUnFWqeeeYZEhMTCQwMJDU1lRUrVhzz2HfffZeUlBQiIyMJCQkhKSmJ115r+AFmsViO+nj00UedxyQmJh7x/EMPPXQyxZfDxfeHP3wLA67wzv2T6zoMr34VbLXeKYOIiLQKLoea+fPnk56ezn333cfq1asZNGgQY8eOpaCg4KjHR0VF8de//pWsrCx+/PFHpk2bxrRp0/j888+dx+Tm5jZ4zJ07F4vFwuWXX97gWg888ECD42655RZXiy/NTb+LITjaLN2wZaG3SyMiIi2Yy6Hm8ccfZ/r06UybNo1+/frx3HPPERwczNy5R19L6JxzzuHSSy+lb9++9OjRg1tvvZWBAweyZMkS5zHx8fENHh988AGjR4+me/fuDa4VFhbW4LiQkBBXiy/NjW/AoQ7Dx1qPSkREpBFcCjXV1dWsWrWKtLRDixxarVbS0tLIyso64fkOh4PMzEw2bdrEWWedddRj8vPz+eSTT7juuuuOeO6hhx4iOjqawYMH8+ijj1Jbq+aKVqG+CWrzQije4d2yiIhIi+XrysGFhYXYbDbi4uIa7I+Li2Pjxo3HPK+kpISOHTtSVVWFj48P//rXv/jNb35z1GNfeeUVwsLCuOyyyxrs/+Mf/8iQIUOIiopi2bJlzJo1i9zcXB5//PGjXqeqqoqqqirn96WlpY19meJp0T2g21lmaPnqV82ilyIiIi5yKdScrLCwMLKzsykvLyczM5P09HS6d+/OOeecc8Sxc+fOZfLkyQQGBjbYn56e7tweOHAg/v7+3HjjjWRkZBAQEHDEdTIyMpg9e3aTvxZxk+RpdaHmNTj7z2aNKhERERe41PwUExODj48P+fn5Dfbn5+cTHx9/7JtYrfTs2ZOkpCT+9Kc/ccUVV5CRkXHEcd9++y2bNm3i+uuvP2FZUlNTqa2tJScn56jPz5o1i5KSEudj586dJ7ymeFGfiyCkPZTnmXlrREREXORSqPH39yc5OZnMzEPT2tvtdjIzMxk+fHijr2O32xs0DdV76aWXSE5OZtCgQSe8RnZ2NlarldjY2KM+HxAQQHh4eIOHNGO+/pA02Wx/rxmGRUTEdS43P6WnpzN16lRSUlIYNmwYc+bMoaKigmnTTGfPKVOm0LFjR2dNTEZGBikpKfTo0YOqqio+/fRTXnvtNZ599tkG1y0tLeWdd97hH//4xxH3zMrKYvny5YwePZqwsDCysrK4/fbbueqqq2jXrt3JvG5pjpKnwtI5sDUTirZBVDdvl0hERFoQl0PNxIkT2bt3L/feey95eXkkJSWxYMECZ+fhHTt2YLUeqgCqqKjg5ptvZteuXQQFBdGnTx9ef/11Jk6c2OC68+bNw+FwcOWVVx5xz4CAAObNm8f9999PVVUV3bp14/bbb2/Qz0Zagaju0ONc2LoIVr8Cafd7u0QiItKCWBwOh8PbhfCE0tJSIiIiKCkpUVNUc7b+Q3j7atO/5vb1pllKRETaLFc+v7X2kzQvvc+H0Dio2AubPvF2aUREpAVRqJHmxccPBl9tttVhWEREXKBQI81P8lTAAtsWw76t3i6NiIi0EAo10vxEdoGedUtxrFJtjYiINI5CjTRPKXXrQf3wOpTv9W5ZxHOqD0DbGLsgIm6gUCPNU6+x0L4PHNwP/7sO7DZvl0jcLXcNPNQFXr4ISnO9XRoRaYEUaqR58vGF374CfsGmb83iR7xdInG3H94Aew1sXwLP1y1wKtKc1RyEnStVu9iMKNRI8xXbBy6aY7YXP2wm5ZPWyeGAjXVD+EPaQ0UBvHoJfPMY2O3eLZvIsXycDi+lQeYD3i6J1FGokeZt0EQYMhVwwP+mQ+keb5dI3GHPD1C6C/xCYMYKsw6Yww6LHoS3JsGBIm+XUKSh/dvhx/lme8nj8PPn3i2PAAo10hKc/wjED4ADhfDfa8FW4+0SSVPb+LH52isNgqNgwr/g4qfBNxA2fw7Pnw27V3u3jCKH++5f4LCBb5D5/t0boHiHd8skCjXSAvgFmv41/mGwI6vtVvW25nb7DXWhps/4Q/uGXA3XLYR23aBkB8wdCytfbPz7UFmqpitxjwNFsPpVs/3blyFhCFQWwzvXQG21FwsmCjXSMkT3gAnPmO1lT8LGT71bHk/KX2dqKp4YCOUF3i5N0yvcDIWbwOoHp53X8LkOA+HGxdDnIrBVwyd/gnenQ1W5CTcVhbBzBayZB1/9Hf53PbxwLjycCA91hv+Ma91hULxj5YtQc8DUIJ821gSbwEjYvQoW3uPt0rVpLq/SLeI1/S6B1Jtg+bPw/h/gxm+gXaK3S+U+djuseB4W3ge2KrNv1Stw9v87teseLDZBqesIsFhOuZinbMNH5mu3syAw4sjnAyNg4uuQ9bR5L9a+A1u/MiGnqvT41965HLYvg8SRTV9uaZtqDsLy5832yNvM/6F2XeHS5+GtibD8OehyBpx+qVeL2VappkZalt88AB1ToLKkrqq3ytslco+yPHjjclhwlwk00b3M/lUvn/qcPe/dCC9fAB/f3jyaZ+r70/Qdf+xjLBYYcQtc8wmExpv+VfWBJrwTJJ5pOpSnzYbfvQZ/WAqDfm+e16zU0pSy3zT//iK6QL8Jh/b3HmdCDsAHt2iJFy+xOBxto27WlaXLpZkr3gnPn2km5ht2A1zwqLdL1LQ2fgIfzISDRaaj7Hl/g8FXweN9zWu+cr75BXoyCjbCv1IPfZ9yLVz4uPdqbEr3mNeFBe74GUJjT3xOZYnpNBzWwfyF7Bd09ON2r4YXRoOPP6RvhJDoJi26tEF2GzyVDPu3wbiH4Yw/NHzeVguvjIcdyyCuP1z/5bH/fUqjufL5rZoaaXkiO5uqXoAV/4af/ufd8jSV6gr46FaY93sTaOIHwA2LYdh084sxabI57vuXTv4ey581X6N7Ahb4fi58eof3+p3Uz03TObVxgQZMc1SP0WYeo+N9YHQcAh0GmWaqNW+eellFNnxkAk1QO9OR/dd8fOGKuWaupfyf4LM7PV/GNk6hRlqm08bCqNvN9od/bPnDfff8YGbRXfWy+X7ELXB9pvngrpdyrfm6eaGZI8NVFftMh1qAi58yw6axmE6Pn93pnWBT35+m70XuuX5y3Rpiq15Wh2E5NQ4HLH3CbA+dDv4hRz8uvANc/iJgMSOkst/yWBFFoUZastF3Q9eRUF1umhnmTTbhoCWx22DJP+HFNNi3BcISYMoHpsnJN6DhsdE9oPs5gANWv+L6vVbNhdpK6JAEXYZD0u/hkqcBi6nxWnCXZz/4DxRBzhKz3cdNoWbAFeAfat7b+nu1JmX58OPbZjSYuNf2pbBntWkSHnbD8Y/tfg6M/ovZ/vh2yF/v9uKJoVAjLZePL/zu1brOehbT4fTf58DrV8CO5V4uXCPk/mjCzJf3g70W+l4MNy2tCy7HUF9bs/pV1+bDqK2GFS+a7TNuPtSHZvBVptYGzKiNBbM8F2x+XmAmL4vrD1Hd3HOPgDAY8Fuz3Zo6DDsc5t/AM0PNEPdXLzF9jcR96mtpkiZDaPsTH3/mHdDjXKg9CG9PUfD0EA3plpYtJAZ+94rpALvkcTPcd8tC8+h2Fpz1/8zImOYwdLledQV8nQFZdTOSBoTD2L+bgHGicva+wIz+Kc8zIa7/ZY2757r3zDmh8UcONR1ytVmS4KM/mj43FiuM/T/3v2fOCffcVEtTL/kaE2jWf2jmtQmJce/93G3fVtP3KufbQ/t2fw+vXQpXvQtBkV4rmtvsWmWCf5fUEx/rDvnrYfMXgAWGz2jcOVYrXPYCPHcm7NtsRhxG9wSfAPD1NzU+Pv6mRrZ+X2Ak9L8cAkLd+WpaNYUaaR1i+8Bl/4az/2yac9a8ZVZ53vYNdD7DhJueY0xzT3kelOyGkp1Qurtue5dZe6gsD6K6Q+/zTYCI6dW05fz5CzOBXEnddOr9JsD5D0NYfOPO9/GDIVPgm0dMJ9/GhBqHA76rm7hw2HTzy/PXkuvW1/roVnOsxWKawI4WbCpLzYR3O5bB9iwo/BkueMT8Mm6s6grYmmm23dWfpl5CEiQMNk2T2W/AyFvdez93sdWauXq+zjDNiL5BcO5fIXGUCTS7V8FrE+Dq90xH1taiLB/+c74J3reugYiOni/DsrrazH4Xm2bgxgqJMRPzvXwB5K4xjxPJfgOu+dTURIvLNKRbWqfinaa6ePWrhyauC442E885XJjnJarHoYDTOfXkf9GU5po+K+vfN99HdIELHzMdnl1VsgvmDDC/5GeshPanHf/47cvMh4JvINy+/vhDm7//D3x8m9kecQv85kGo2GuusSPLfM3/ydz7cIERpixhcY17Des/hLevhsiu5oPK3bVCq14xNVFR3WHmKvNXdEuyJxs+vAXyfjTfdz/HrGBf32yXtxZeudiMmuuQZIJNcJR3ytrUvv3HoaVR0u4/NEDAU0p2m9m87bUwfRF0THb9GgUbTJ+c2mrz+8j5te5Rv2/Tp2b+pbPuNIFVANc+vxVqpHUryzN/ZX0/10xrDmD1hfAEiOgM4R3NX37hHc33oe3NSKpNn5laHvthi2cGtYNe55mQ02MMBDbi35HdboZgZz5gfllZfGD4zXDOrGOPnmiMt640vwDPuBnGZRz/2HmTTVNV8jUw/okTX3vlS/BJutkO72hqs36tXSJ0GQFdh5vRU7lr4PTL4LeN7Lfy7g1mhePhM01Tl7tVlcM/+kB1memIfbx+S81J9QFY/BAse9qE8cBI01SZ9Psjg2DeT/DqxXBgH8QPNK+zpQcbux2eHHRoocj2feDm7zzbnPz5X00NWeKZcM3H7r3XT/8zi/ZigakfQbcz3Xu/FkKh5igUatq4A0WwP8eEmZDYxv2lXlkKWxeZgLP5czPx3eECI0ztT3A0BMeYryGHfe8fYn4Z7lppjk8YYkJFh4Gn/no2L4Q3rjBl+NOmY8/XUrQNnhwMOODm5Q2HiB/PihfM/DUAWCDudDNiqutwE2bCOxw6NncN/Hu0+dD9/dsnrn2y1cCjPUzH1mkLzDU94eN0EzD7TTD9sJq7bd+Y6Qr2bzPfn36Zaao83nw++evN5G8HCs08R1M+bNnBZvOXZmbtgIi62oxKmP6VmYPIEw4Wwz9PNyMsJ/8Xev3G/ff8YAb88LoZCXnT0pb982sirnx+q9FO2obgKNd/OQSGw+kTzMNWC7tWmNqRjZ9C0VbzoVxZAkW/HP86/mEw5l4Yeh1YfU72FTTU41yI7GL+gl33nvnL/WhW/BtwmJqlxgYaMH1vYvuavi+dhx2/j0aHQab2adlTpr9Q15HH7+iY861530Lam2t7Sso0E2o2fmwWBm3sZH/esOkzMwmjw24+3C78B/S54MTnxfUztQmvjD/UJDXlg5Y7m/L3c83XpN+bZtCf/mvmWvJUqFn1HxNoYvtBzzTP3PP8R8zozX2bzczik95oXgMdmrkW1rAs4iU+vmYByPP+Bn9cDXduM31Ipi0wiy2OfwLOvcc0Bw2caEJEhyQYdCXMXAGpNzRdoAFzreRrzPbKY8wwXFkKq18z22fc7Po9EkeZWpfGdDo9Z5YJWSU74asTNCc5Rz1d2LTvyYnEDzD9Iey15i/h5ir3R/jvdSbQ9L8cZixvXKCpF9vXrJEVEgv5a03AqSh0X3ndpXSPGfYPJpAOutJsr33HtekMTlZtFXxXNwP3iD96Llj4h8AVL5mRUZs+Mc270miqqRE5GSdT89PUBl8NX2WY4by5a0yNyeF+eN30IYnpbUZ+uZN/CFz0T3j9cjPfzYArjt6h0m4/tDRCn+MsYOkuydPMKKHVr5jFB5tbh+HSXHhzItRUmH4/lz5vRry5qn1vE2xeuQgK1plgM+XDxs2v0lysfs00aXYZYV5PVA8IjYPyfDNlQ58L3Xv/H+ebe4V3dG1kX1PoMMgs3rvgLtOnp+sI0wQsJ9TM/keLSKOFxh5a2fr7X3XQtdtMuAA44ybP/JXZMw0G/M7UMHx4q+k782u7vzdD6gPCzTxCntb/MnPv/Tmw7WvP3/94qivgrYlQtscE0d++cnKBpl770w6tal6w3kzQ11JWtbfVHpo1u37CSR9fGPg7s53t5rW87HZY+qTZPuPmo0+D4G6pf4BeY01fov9eazqNywkp1Ii0ZPW/8H982zQ31dv4CRRvN01HAyd6rjxj/27umb8WvvvXkc/Xr/XU6zzvfFD4hxx6P34dBL3JboP/TTc1bsExMPntpplEL6YXTPvUXLNgnal9aAm2LDSj7oKizNww9eqboH7+3HT+d5cf55k+LQERdXM4eYHFYtZnC42DvRvh8794pxwtjEKNSEuWOApiTjPNFWvfPrS/vi9AyrXgH+y58oS2h/Pq+tR8lWFGX9VzOEwnXXD/hHvHk1K3yOWmT83Ebs3Bl/eZ/hM+ATDpTTNkvqlE94BRt5ntJXNMgGru6jsID57ccA20uNPNcHV7jRn+7A4HiuCLu832melmqQ1vCYkxk4piqZsV+wPPl8Fuh5/eNSMiCzY2+4VhFWpEWjKL5VBtzcq55hfOnh/MbL9WXxh6vefLlPR7M6dH7UEz3039L8GCDWakmE8A9PTA0NhjiTsdOg2r6zD8mvfKUW/Vy4dmrJ3wL/csBZB8jZnjpmjrodqy5qp4h5myAA6tsn64+tqaNW5a/frL+8xcP7H9Gr8kgjt1P+dQKP3wlkNz9nhC0S+mX9Z/p5kpHv6VCo+dBu9MMzWd+7Y2u5CjUCPS0g2aZKbML1hn5sSpr6U5/TIzL4+nWSxmNJhPgJnn58e6GqT6Wpoeo72/tk39yLHVr5i/RL1l61dmGDzAOX8xHazdISAMUm8020seP7UPok0L4NP/Z4bFu8PqVwGH6XN1tCUJBvzWTGK5exXs/blp773ju7r7Yzq+n0qfpqY0+q+m431liWmmtNW69352u6mZeXakmQnZL8T8oeIbBBUFsO5dM/P4U0PMPD7v3gg/vOHZwHUMCjUiLV1Qu0OjM77OMFXFYOaO8ZboHnD2nWb781lQse9QDYG7F7BsjNMvNf0lineY4OUNezfB21NNjdHAiYfeL3dJ/QP4BZt+Oyf7mot3wDvXmPmPXhxjmiOakq3mUKior4H8tdD2hybBa8raGlsNfFy3BMOQKdDljKa79qny8YPLXzJzXu38zqz95i77t5uZqT+9w8zCnngm3LzMzH9013azLtU5s8x8VD7+pu/Tj/Pgg5vN8i3/vc59ZWsEhRqR1qD+A2DrItPfoMtws4ijN4281VThH9gH715v1i2yWM0yE97mH2xquMD0VfC0ikJ447dQVWIWXL34KfePUAuOOlRDteSfJ3eNz/9imhXBBJyXzoNfFjdJ8QAz6WB5vpmYsfdxhmzXN0H9OL/patqynjGjxIKjIW1201yzKUV1g/FzzPY3jzZ9GHc4TF+mZ0eYCTL9guGCx8xUAPV9vHwDIHEknHOX6YD+5+1w9ftw5p+g01BTgxZzgrXo3EyhRqQ16DjEdKCsdzKT7TU1Hz8Y/yRgOfQLuMsI0/mxOXB2GP7M/HXqKTWVZj2u4u3mw2LSGw07w7rT8Blg9TMfWjtXunbulkxT22bxgakfm+BcVQKvX2aaHpqCs4Pw1ccfHXfaOLNESOluyPnm1O+7fzt8/ZDZPu9v3p+D6lgGXGHeG4fdDPPen9M01y3eaVZ4//h2M4NylxFmiYZh048/l5N/sGlOHnMvXP+lqckZNr1pynSSFGpEWgOLxSzDAGZmX3dPTNZYnYc2/CXnzVFPvxbb19SSOGxmFeankuF/15u/2LcvM4tgusOnd5gmhIAI+P07ng15EZ0ODWl3pbamtho++7PZHnaDWWjx6veh/xWm+eyDm2HR306tr07RL/DLV4DlxMOo/QIPNbmumXfy9wRT5s/uNDVQXUcdqgVqri54zPSvObjfhOPqipO/lsNhVrD/13D45WvTZ2bcQ2Z+o6jurl8vIMzrgVChRqS1GHw1nP8oTHrLs8sPnMi595ig5RcCfS8+8fGedO7dpmwA+7aYKfg//wv853zI6ARPDzOdIL97zoz0OFU/f25GXFmsZlHN9l6oqh91G2AxQ8gLNjTunO/+ZeZtCWkPo2eZfX6BcNkLcGbdwqffPArvTj/5Cf5WvWy+9ji3cUPa68PH+g9PLYBu/MQsx2D1g4seb/7rLPkFwu9eMz+L/J/M+lAnEyZrq+F/18FHfzQzj3dONbUzZ9zU/GbadoFW6RYR9ztQZDodRnTydkmOrqIQ9mSb4fC5dV9Ldzc8xsffdJLsPPTk7nGwGP51BpTlwohbTDOHt8y/GjZ8CAMnwWXPH//Y0j3wVIqZC2nCs0dfPHX1a2Y0jL3WNEtNetO1v9hrq+HxvmZ18YlvNK5Gz+EwtWtFW2HCc5B0EjUsVeXwzDDzsz7zDhhzj+vX8Jbty8zyF/Zas6TCyFsbf25VObx9tWkWtvpB2n2mybo5/TF0GFc+v1tuHBORliM4qvkGGjBNQL3S4Oz/Z/q4pK+HOzab5qFz/gIJQ8BWbebrONmZbD//iwk00T3NEF1vGlU3ymftOycehvvF3SbQdBpmQtDRDLkaJv/XLEGxIwteTHOtZmvjRybQhHUw/WUaw2I5bM6ak1w24esME2jaJcJZd5zcNbyl6wjTVATw5f2N7zh8oMgsmbF1kekM/Pv5JmQ300DjKoUaEZGjCY2F086Dc/4MUz4wCyqW7IT3b3J9xM3PX0D2G4AFLvkX+AW5pciN1nGImdTNYTs08d/RbPu2buZeC1z42PGbJXqMhuu+gIgupvbkxTTT3NaY96p+yYohU8waT41VvxbUtm9NZ1dX5K09NKfTBf/w/s/kZAy9HpKuMh2H35nWcAbvoyndY5pWd39vpoKY8qH7F7v1sJMKNc888wyJiYkEBgaSmprKihUrjnnsu+++S0pKCpGRkYSEhJCUlMRrrzWcxfOaa67BYrE0eIwb1zCtFxUVMXnyZMLDw4mMjOS6666jvNxNHflERA4XGA6/fdlMKPjzAsg6ThD4tYPF8FFd08DwGe6ZMfhkjEo3X1e/CuV7j3zeVmM60IKZMuDXq8AfTWxfMwomYQgcLII3fwdPDYZvHjMrkB/N3p/NaCyL1YQaV7TrauZRweHaulZ2O3x0mwl1/SaYWrqWyGKBC/9RNzFf8fE7Du/bCi+NNetIhXWAaZ+dfFNqM+ZyqJk/fz7p6encd999rF69mkGDBjF27FgKCo4+u2RUVBR//etfycrK4scff2TatGlMmzaNzz//vMFx48aNIzc31/l4662GkypNnjyZdevWsXDhQj7++GO++eYbbrjhBleLLyJycjoMhPMfNttfzjazzzbGF381K29H9fB+s9Phup1lPgxrK2H5s0c+v+IFM29LUJTpUN1YYXFm9EzqHw6tiL7oQfhnP3hzkumYe/gK7vUdhHudd3JNlPXzDa2Z1/gOs6tfNrUV/mEwLsP1ezYnh3ccLlgHH8w48n3IXWPmFCrZYf4dXvu5CaCtkMsdhVNTUxk6dChPP/00AHa7nc6dO3PLLbdw1113NeoaQ4YM4cILL+TBBx8ETE1NcXEx77///lGP37BhA/369WPlypWkpKQAsGDBAi644AJ27dpFQsKJp4JXR2EROWUOhxnhs/YdCEuAPyyBkOhjH7/5S3jjcsBi/jLuOtxjRW2UDR/B/KvM8PLbfzI1UmAW+nw6BapKzZIX9ZP2uar6gFmEcfWrZj2yeqFxpsPxgN/ByxeY4cm/fxtOG+v6ParK4NFeZkj29ZnQKeX4x5cXmNdWWQLjHoYz/uD6PZujwzsOp80+tF5UzhJ460rzs4wfCFe9a2ZlbkHc1lG4urqaVatWkZZ2qKrOarWSlpZGVlbWCc93OBxkZmayadMmzjrrrAbPff3118TGxtK7d29uuukm9u3b53wuKyuLyMhIZ6ABSEtLw2q1snz5cldegojIybNY4KI5EN3L1L68d8Ox+4xUlpjhsmCGyTa3QANm1t6Y3mYSvfqJ78As6lhVapqRBrvYJHQ4/2AzKunaz2DGShjxRwiOMbMGL/knPDvcBJqIztDzJJuAAsKg73izfaxlE6oPmHW2vpwNL19kfjYdBnlnwVd3ObzjcOZsM1nixk/htcvMz7LrKLPUQQsLNK5yKdQUFhZis9mIi4trsD8uLo68vLxjnldSUkJoaCj+/v5ceOGFPPXUU/zmN4dW6R03bhyvvvoqmZmZPPzwwyxevJjzzz8fm80GQF5eHrGxsQ2u6evrS1RU1DHvW1VVRWlpaYOHiMgpCwg1c8z4BsGWL2HpMSax++LuupE13cxcPc2R1XroL/rv/mVmO97x3aFwcMEJOge7ov1pcN6DkL7BNJf0/A1QNydMyrWnNvqmfjj32v+aeXJqq83rWPyICTEPdzUz5i55HAo3mTmTLprjWqfklmDo9TC4ruPw21NNLZytyoTXq/5nZmFu5TzyEw0LCyM7O5vy8nIyMzNJT0+ne/funHPOOQBMmnRomOCAAQMYOHAgPXr04Ouvv2bMmJPrmZ2RkcHs2c1w/Q4RafniTocLHoUPZ5qZdDufYdbEqbcls25hRgtM+JepsWiuBvwWFv0flO4yEwOufsXsH3w1dEpu+vv5+kO/i82jeCfkrzu0QOXJ6na26fxalgtzx5rOxzW/6jAblmD6EXU7y4z4CYs/tXs2RxaLGclVsMGsYg6QNNksV9LaAtwxuPQqY2Ji8PHxIT8/v8H+/Px84uOP/Q/EarXSs2dPAJKSktiwYQMZGRnOUPNr3bt3JyYmhi1btjBmzBji4+OP6IhcW1tLUVHRMe87a9Ys0tPTnd+XlpbSuXPnxrxMEZETG3wVbF9qajX+e63pXxPaHipL4cO6ZqfUG02zQHPm42fmKVnwZzOXjq3a/EWfdr/77x3Z2TxOldXHDO9e+oSZOBHMwpSJZ9YFmbPNyvHNfbbgpuAXCBNfN6O7Og01i0224BmCXeVSqPH39yc5OZnMzEwmTJgAmI7CmZmZzJw5s9HXsdvtVFUdeyrtXbt2sW/fPjp06ADA8OHDKS4uZtWqVSQnm78cFi1ahN1uJzX16MMjAwICCAjw0CJxItL21A+n3b3aNGm8O91U8X9xt6n1aJdoFvprCYZMgW8eMSuqg2kuay4LjzbWqHSw2yC8owkysf3a1Id5A+EJMPltb5fCK1yuj0pPT2fq1KmkpKQwbNgw5syZQ0VFBdOmmRVvp0yZQseOHcnIMMPkMjIySElJoUePHlRVVfHpp5/y2muv8eyzZghheXk5s2fP5vLLLyc+Pp6tW7dy55130rNnT8aONT3h+/bty7hx45g+fTrPPfccNTU1zJw5k0mTJjVq5JOIiFv4h5j+NS+caxZjfHsKbPzYPHfJM+b5lsA/2HRmXvQ3iBtg+ri0NEGRMPb/vF0K8TKXQ83EiRPZu3cv9957L3l5eSQlJbFgwQJn5+EdO3ZgPSwdV1RUcPPNN7Nr1y6CgoLo06cPr7/+OhMnmpVifXx8+PHHH3nllVcoLi4mISGB8847jwcffLBBTcsbb7zBzJkzGTNmDFarlcsvv5wnn3zyVF+/iMipie1ramzev+lQoBl2AySO8m65XDXyNgiJNf1bWsmU+dL2aEFLEZGm8P4MyH4dIrvCTcvMKCkROWWufH63je7QIiLuduE/zNwnPc5VoBHxEoUaEZGm4BcIqVq6RcSb2mjXcBEREWltFGpERESkVVCoERERkVZBoUZERERaBYUaERERaRUUakRERKRVUKgRERGRVkGhRkRERFoFhRoRERFpFRRqREREpFVQqBEREZFWQaFGREREWgWFGhEREWkV2swq3Q6HA4DS0lIvl0REREQaq/5zu/5z/HjaTKgpKysDoHPnzl4uiYiIiLiqrKyMiIiI4x5jcTQm+rQCdrudPXv2EBYWhsViadJrl5aW0rlzZ3bu3El4eHiTXltOTO+/d+n99x69996l998zHA4HZWVlJCQkYLUev9dMm6mpsVqtdOrUya33CA8P1z9sL9L77116/71H77136f13vxPV0NRTR2ERERFpFRRqREREpFVQqGkCAQEB3HfffQQEBHi7KG2S3n/v0vvvPXrvvUvvf/PTZjoKi4iISOummhoRERFpFRRqREREpFVQqBEREZFWQaFGREREWgWFmlP0zDPPkJiYSGBgIKmpqaxYscLbRWqVvvnmG8aPH09CQgIWi4X333+/wfMOh4N7772XDh06EBQURFpaGps3b/ZOYVuhjIwMhg4dSlhYGLGxsUyYMIFNmzY1OKayspIZM2YQHR1NaGgol19+Ofn5+V4qcevy7LPPMnDgQOckb8OHD+ezzz5zPq/33nMeeughLBYLt912m3Of3v/mQ6HmFMyfP5/09HTuu+8+Vq9ezaBBgxg7diwFBQXeLlqrU1FRwaBBg3jmmWeO+vwjjzzCk08+yXPPPcfy5csJCQlh7NixVFZWerikrdPixYuZMWMG3333HQsXLqSmpobzzjuPiooK5zG33347H330Ee+88w6LFy9mz549XHbZZV4sdevRqVMnHnroIVatWsX333/PueeeyyWXXMK6desAvfeesnLlSp5//nkGDhzYYL/e/2bEISdt2LBhjhkzZji/t9lsjoSEBEdGRoYXS9X6AY733nvP+b3dbnfEx8c7Hn30Uee+4uJiR0BAgOOtt97yQglbv4KCAgfgWLx4scPhMO+3n5+f45133nEes2HDBgfgyMrK8lYxW7V27do5XnzxRb33HlJWVubo1auXY+HChY6zzz7bceuttzocDv3bb25UU3OSqqurWbVqFWlpac59VquVtLQ0srKyvFiytmfbtm3k5eU1+FlERESQmpqqn4WblJSUABAVFQXAqlWrqKmpafAz6NOnD126dNHPoInZbDbmzZtHRUUFw4cP13vvITNmzODCCy9s8D6D/u03N21mQcumVlhYiM1mIy4ursH+uLg4Nm7c6KVStU15eXkAR/1Z1D8nTcdut3PbbbcxcuRI+vfvD5ifgb+/P5GRkQ2O1c+g6axdu5bhw4dTWVlJaGgo7733Hv369SM7O1vvvZvNmzeP1atXs3LlyiOe07/95kWhRkRcMmPGDH766SeWLFni7aK0Kb179yY7O5uSkhL++9//MnXqVBYvXuztYrV6O3fu5NZbb2XhwoUEBgZ6uzhyAmp+OkkxMTH4+Pgc0cM9Pz+f+Ph4L5Wqbap/v/WzcL+ZM2fy8ccf89VXX9GpUyfn/vj4eKqrqykuLm5wvH4GTcff35+ePXuSnJxMRkYGgwYN4oknntB772arVq2ioKCAIUOG4Ovri6+vL4sXL+bJJ5/E19eXuLg4vf/NiELNSfL39yc5OZnMzEznPrvdTmZmJsOHD/diydqebt26ER8f3+BnUVpayvLly/WzaCIOh4OZM2fy3nvvsWjRIrp169bg+eTkZPz8/Br8DDZt2sSOHTv0M3ATu91OVVWV3ns3GzNmDGvXriU7O9v5SElJYfLkyc5tvf/Nh5qfTkF6ejpTp04lJSWFYcOGMWfOHCoqKpg2bZq3i9bqlJeXs2XLFuf327ZtIzs7m6ioKLp06cJtt93G3/72N3r16kW3bt245557SEhIYMKECd4rdCsyY8YM3nzzTT744APCwsKcfQUiIiIICgoiIiKC6667jvT0dKKioggPD+eWW25h+PDhnHHGGV4ufcs3a9Yszj//fLp06UJZWRlvvvkmX3/9NZ9//rneezcLCwtz9h2rFxISQnR0tHO/3v9mxNvDr1q6p556ytGlSxeHv7+/Y9iwYY7vvvvO20Vqlb766isHcMRj6tSpDofDDOu+5557HHFxcY6AgADHmDFjHJs2bfJuoVuRo733gOM///mP85iDBw86br75Zke7du0cwcHBjksvvdSRm5vrvUK3Itdee62ja9euDn9/f0f79u0dY8aMcXzxxRfO5/Xee9bhQ7odDr3/zYnF4XA4vJSnRERERJqM+tSIiIhIq6BQIyIiIq2CQo2IiIi0Cgo1IiIi0ioo1IiIiEiroFAjIiIirYJCjYiIiLQKCjUiIiLSKijUiIiISKugUCMiIiKtgkKNiIiItAoKNSIiItIq/H/hUG+HfYZxbAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.plot(train_loss_history)\n",
    "plt.plot(val_loss_history)\n",
    "plt.legend(('train', 'val'))\n",
    "plt.savefig(os.path.join(output_dir, \"loss.pdf\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict and compute scores on validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "VALIDATION\n",
      "\n",
      "Micro AUPRC:           0.6144345431514495\n",
      "Micro F1-score (@0.5): 0.5529308836395451\n",
      "Macro AUPRC:           0.3666736994104178\n",
      "Coarse Tag AUPRC:\n",
      "      - 1: 0.7749544886438909\n",
      "      - 2: 0.3311363877198092\n",
      "      - 3: 0.18355672310440047\n",
      "      - 4: 0.27904221303689974\n",
      "      - 5: 0.5309844246832589\n",
      "      - 6: 0.07569277752197645\n",
      "      - 7: 0.7415846746837877\n",
      "      - 8: 0.01643790588931903\n"
     ]
    }
   ],
   "source": [
    "# Prediction on validation set\n",
    "\n",
    "print(\"VALIDATION\\n\")\n",
    "\n",
    "y_pred = predict(log_mel_spec_list, val_file_idxs, model)\n",
    "\n",
    "\n",
    "generate_output_file(y_pred, val_file_idxs, output_dir, file_list,\n",
    "                       'predictions_validation', 'coarse', taxonomy)\n",
    "\n",
    "prediction_file = os.path.join(output_dir, 'output_predictions_validation.csv')\n",
    "\n",
    "df_dict = evaluate(prediction_file,\n",
    "                  annotation_file,\n",
    "                  taxonomy_file,\n",
    "                  'coarse',\n",
    "                   'validate')\n",
    "\n",
    "micro_auprc, eval_df = micro_averaged_auprc(df_dict, return_df=True)\n",
    "macro_auprc, class_auprc = macro_averaged_auprc(df_dict, return_classwise=True)\n",
    "\n",
    "# Get index of first threshold that is at least 0.5\n",
    "thresh_0pt5_idx = (eval_df['threshold'] >= 0.5).to_numpy().nonzero()[0][0]\n",
    "\n",
    "print(\"Micro AUPRC:           {}\".format(micro_auprc))\n",
    "print(\"Micro F1-score (@0.5): {}\".format(eval_df[\"F\"][thresh_0pt5_idx]))\n",
    "print(\"Macro AUPRC:           {}\".format(macro_auprc))\n",
    "print(\"Coarse Tag AUPRC:\")\n",
    "\n",
    "for coarse_id, auprc in class_auprc.items():\n",
    "    print(\"      - {}: {}\".format(coarse_id, auprc))\n",
    "\n",
    "with open(os.path.join(output_dir, 'final_results_validation.txt'), 'w') as f:\n",
    "    f.write(\"Micro AUPRC:           {}\\n\".format(micro_auprc))\n",
    "    f.write(\"Micro F1-score (@0.5): {}\\n\".format(eval_df[\"F\"][thresh_0pt5_idx]))\n",
    "    f.write(\"Macro AUPRC:           {}\\n\".format(macro_auprc))\n",
    "    f.write(\"Coarse Tag AUPRC:\\n\")\n",
    "    for coarse_id, auprc in class_auprc.items():\n",
    "        f.write(\"      - {}: {}\\n\".format(coarse_id, auprc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predict on test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predict(log_mel_spec_list, test_file_idxs, model)\n",
    "\n",
    "\n",
    "generate_output_file(y_pred, test_file_idxs, output_dir, file_list,\n",
    "                       'predictions_test', 'coarse', taxonomy)\n",
    "\n",
    "prediction_file = os.path.join(output_dir, 'output_predictions_test.csv')"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "T4",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}