{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "### Programme inspecteur modèles BNP Paribas - module Monte-Carlo methods - 12 décembre 2019" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# TP 2 - Variance reduction methods" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 1. Control variate\n", "\n", "We consider a stochastic model defined by $Y = f(X)$. The goal is to estimate $\\mathbb{E}\\bigl[g(Y)\\bigl]$ for a given function $g$, such that $g(Y)$ is a square integrable random variable.\n", "\n", "We are given a simplified model $Y_c = f_c(X)$ such that the value of $m_c = \\mathbb{E}\\bigl[g(Y_c)\\bigl]$ is explicitly known and such that $\\mathbb{E}\\bigl[g(Y_c)^2\\bigl]<\\infty$.\n", "In practice, $f_c$ can be a function easier to evaluate (with lower computational cost) than the function $f$, or simply a function such that the expectation $m_c$ can be evaluated explicitly.\n", "\n", "We denote $(X_i)_{1\\leq i\\leq n}$ a sequence of i.i.d. copies of the input random variable $X$, and we set\n", "\n", " \\begin{eqnarray*}\n", " I_n&=&\\frac{1}{n}\\sum_{i=1}^n g(f(X_i)),\\qquad\n", " I_n^c = m_c + \\frac{1}{n} \\sum_{i=1}^n \\bigl( g(f(X_i)) - g(f_c(X_i)) \\bigr) .\n", " \\end{eqnarray*}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 1:\n", "Check that $I_n$ et $I_n^c$ are unbiased estimators of $\\mathbb{E}\\bigl[g(Y)\\bigl]$, and compute their variances." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 2 (a):\n", "\n", "We assume that the input random variable $X$ follows a uniform distribution on $[0,1]$, and that $f(x)=e^x$, $g(y)=y$, and we choose $f_c(x)=1+x$.\n", "\n", "Simulate the two estimators $I_n$ and $I^c_n$ and their $95\\%$ confidence intervals.\n", "\n", "Plot the trajectories $n \\mapsto I_n, I_n^c$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "N = 1000 # Sample size\n", "\n", "integers1toN = np.arange(1,N+1) \n", "\n", "############################################\n", "# Compute the exact value of m_c\n", "m_c = ????\n", "############################################\n", "\n", "##################################################\n", "# Draw the samples of the random variables Y = f(X)\n", "# and Y_c = f_c(X)\n", "##################################################\n", "Y = ????\n", "\n", "Y_c = ????\n", "############################################\n", "\n", "############################################\n", "## Evaluate the two estimators, their empirical\n", "## variances, and their confidence intervals\n", "\n", "I = ????\n", "emp_variance_MC = ????\n", "\n", "I_c = ????\n", "emp_variance_control = ????\n", "\n", "radius_confidence_int = ????\n", "radius_confidence_int_control = ????\n", "\n", "############################################\n", "# The true value of E[g(Y)] for comparison\n", "############################################\n", "Exp_gY = np.exp(1.) - 1.\n", "\n", "############################################\n", "# Plotting\n", "print(\"Sample size = %d\" %N)\n", "print(\"True expectation E[g(Y)] = %1.3f \\n\" %Exp_gY)\n", "\n", "print(\"Standard MC estimator: %1.3f\" %I)\n", "print(\"Confidence interval(95%%) = [%1.3f,%1.3f] \\n\" \\\n", " %(I - radius_confidence_int, I + radius_confidence_int))\n", "\n", "print(\"Control variate estimator: %1.3f \" %I_c)\n", "print(\"Confidence interval(95%%) = [%1.3f,%1.3f] \\n\" \\\n", " %(I_c - radius_confidence_int_control, I_c + radius_confidence_int_control))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "########################\n", "## Plot 10 trajectories\n", "########################\n", "\n", "##############################################################\n", "# Simulate M=10 trajectories of the estimators I_n et I_n^c\n", "# Wished size for the arrays: M x N\n", "##############################################################\n", "M = 10\n", "\n", "#Y = ?????\n", "#Y_c = ?????\n", "#I = ?????\n", "#I_c = ?????\n", "\n", "############\n", "## Plotting\n", "############\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "\n", "ax.plot(integers1toN, I[0], color=\"b\", label=\"MC\")\n", "ax.plot(integers1toN, I[1:].T, color=\"b\")\n", "\n", "ax.plot(integers1toN, Ic_[0], color=\"g\", label=\"Control\")\n", "ax.plot(integers1toN, Ic_[1:].T, color=\"g\")\n", "ax.axhline(Exp_gY, color=\"r\", label=\"True expectation\")\n", "\n", "ax.set_xlim(0, N)\n", "ax.set_ylim(1.4, 2.2)\n", "ax.legend(loc=\"best\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 2 (b):\n", "\n", "Plot the histograms of the (unnormalized) errors\n", "\n", "$$(I^j_n - m)_{1 \\le j \\le M} \\quad \\mbox{and} \\quad (I^{c,j}_n - m)_{1 \\le j \\le M}$$\n", "\n", "where $(I^j_n)_{1\\leq i\\leq M}$ and $(I^{c,j}_n)_{1\\leq i\\leq M}$ are samples of $M$ i.i.d. copies of the two estimators, for a fixed value of $n$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "N = 500 \n", "M = 1000 \n", "\n", "integers1toN = np.arange(1,N+1) \n", "\n", "Esp_gY = np.exp(1.)-1.\n", "m_r = 1.5\n", "\n", "############################################\n", "# Simulate the required samples of Y and Y_c\n", "Y = ????\n", "Y_c = ????\n", "############################################\n", "\n", "############################################\n", "# Samples of size M for the two estimators \n", "I_N = ?????\n", "Ic_N = ????\n", "\n", "#########################################\n", "## Plotting the histograms of the errors\n", "\n", "plt.hist( ????? , density=\"True\", bins=int(np.sqrt(M)), label=\"MC\")\n", "plt.hist( ????? , density=\"True\", bins=int(np.sqrt(M)), label=\"Control\")\n", "plt.title(\"Estimation with and without control variate, N = %1.0f\" %N)\n", "\n", "plt.legend(loc=\"best\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 3: Optimal control variate\n", "\n", "We now consider the estimator\n", "\n", "\\begin{eqnarray*}\n", "I_n^\\lambda = \\lambda \\, m_r + \\frac{1}{n}\\sum_{i=1}^n \\bigl(g(f(X_i)) - \\lambda \\, g(f_c(X_i))\\bigr),\n", "\\qquad \\lambda \\in \\mathbb R.\n", "\\end{eqnarray*}\n", "\n", "Propose a choice for the parameter $\\lambda$.\n", "\n", "Plot the trajectoires of the resulting estimator $I_n^\\lambda$ and compare them with $I_n$, then the histograms of their errors for a fixed value of $n$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "##############################################\n", "## Empirical estimation of the optimal lambda\n", "##############################################\n", "n_1 = 100\n", "X = np.random.rand(n_1)\n", "Y = np.exp(X)\n", "\n", "## Empirical optimal lambda\n", "lambda_opt = ?????\n", "\n", "print(\"Estimation of optimal lambda = %1.3f\" %lambda_opt)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#######################################\n", "### For the histograms: M x N samples\n", "#######################################\n", "M = 1000\n", "N = 500\n", "integers1toN = np.arange(1,N+1)\n", "\n", "Y = ?????\n", "Y_control = ?????\n", "\n", "#####################################\n", "### Plot the first 10 trajectories\n", "#####################################\n", "I_n = np.cumsum(Y[0:10,:], axis=1) / integers1toN\n", "Ic_n = np.cumsum(Y_control[0:10,:], axis=1) / integers1toN\n", "\n", "## Plot\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "\n", "ax.plot(integers1toN, I_n[0], color=\"b\", label=\"MC\")\n", "ax.plot(integers1toN, I_n[1:].T, color=\"b\")\n", "\n", "ax.plot(integers1toN, Ic_n[0], color=\"g\", label=\"Control\")\n", "ax.plot(integers1toN, Ic_n[1:].T, color=\"g\")\n", "ax.axhline(Esp_gY, color=\"r\", label=\"True expectation\")\n", "\n", "ax.set_xlim(0, N)\n", "ax.set_ylim(1.4, 2.2)\n", "ax.legend(loc=\"best\")\n", "plt.show()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "#############################################\n", "## For the histograms: evaluate the errors\n", "## of the two estimators I_N et I_N control\n", "#############################################\n", "error_N = ?????\n", "error_control_N = ?????\n", "\n", "plt.hist( error_N , density=\"True\", bins=int(np.sqrt(M)), label=\"MC\")\n", "\n", "plt.hist( error_control_N, density=\"True\", bins=int(np.sqrt(M)), label=\"Controle\")\n", "\n", "plt.title(\"Estimation with and without control variate, N = %1.0f\" %N)\n", "\n", "plt.legend(loc=\"best\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise 2. Importance sampling\n", "\n", "We consider input random variables $Y$ with standard Gaussian distribution $\\mathcal N(0,1)$.\n", "\n", "The aim is to evaluate $\\mathbb E [g(Y)] = \\mathbb E \\bigl[ (Y - 2)^+ \\bigr]$" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 1: standard Monte-Carlo\n", "\n", "Simulate the trajectories of the empirical mean estimator $I_n = \\frac 1 n \\sum_{i=1}^n (Y_i - 2)^+$ obtained from $n$ i.i.d copies $(Y_i)_{1 \\le i \\le n}$ of $Y$, then the histogram of the errors." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Function g\n", "def g(x):\n", " return np.maximum(x-2.,0.)\n", "\n", "N = 2000\n", "integers1toN = np.arange(1,N+1) \n", "\n", "############################################\n", "# Complete with N draws of the standard\n", "# Gaussian distribution\n", "Y = ?????\n", "\n", "# Evaluate the function g(Y) on the sample\n", "GY = ?????\n", "\n", "#############################################################\n", "# Stock into the variable 'mean' the MC estimation E[g(Y)],\n", "# into 'var' the empirical variabe, and into 'radius_CI'\n", "# the half-lenght of the 95% confidence interval\n", "mean = ?????\n", "var = ?????\n", "radius_CI = ?????\n", "\n", "print(\"MC estimation = %1.4f\" %mean )\n", "print(\"95%% confidence interval for E[g(Y)] = [ %1.4f , %1.4f ] \\n\" \\\n", " %(mean - radius_CI, mean + radius_CI))\n", "print(\"Relative error = %1.2f\" %(radius_CI/mean))\n", "\n", "######################################\n", "# Trajectories of the empirical mean\n", "######################################\n", "M = 10 # Number of trajectoires\n", "\n", "#############################################\n", "# Evaluate the M draws of the empirical mean\n", "# I_n for n ranging from 1 to N\n", "I_n = ?????\n", "\n", "##########################\n", "# Plot the trajectories\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "ax.plot(integers1toN, I_n.T, color=\"b\")\n", "\n", "ax.set_xlim(0, N)\n", "ax.set_ylim(0, 3.e-2)\n", "ax.legend(loc=\"best\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 2: Importance sampling\n", "\n", "**(a)** Show that, if $Z$ is a Gaussian random variable centered at $\\theta \\in \\mathbb R$ and with unit variance, then\n", "\n", "$$\n", "\\mathbb{E} [ g(Y) ]\n", "=\n", "\\mathbb{E} \\Bigl[ g(Z) \\, e^{-\\theta \\, Z + \\frac{\\theta^2}2} \\Bigr]\n", "$$\n", "\n", "\n", "What is the interest of such a formula?\n", "\n", "**(b)** \n", "Construct an estimator $J_n$ of $\\mathbb{E}[g(Y)]$ based on the simulation of the gaussian distribution with unit variance and centered at $\\theta=2$.\n", " \n", "Plot the trajectories of the estimator $J_n$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "theta = 2.\n", "\n", "N = 2000 # Taille echantillon\n", "integers1toN = np.arange(1,N+1)\n", "\n", "###############################################################################\n", "# Complete with N draws of the Gaussian distributioncentered at theta = 2\n", "# and evaluate the importance sampling estimator constructed with these N samples\n", "###############################################################################\n", "J_N = ????\n", "\n", "# Empirical variance and confidence interval\n", "var = ????\n", "radius_CI = ????\n", "\n", "print(\"Importance sampl estimator = %1.4f\" %J_N)\n", "print(\"95%% confidence interval for E[g(Y)] = [ %1.6f , %1.6f ]\" \\ \n", " %(J_N - radius_CI, J_N + radius_CI))\n", "print(\"Relative error = %1.4f\" %(radius_CI / J_N))\n", "\n", "####################################################\n", "# Trajectories of the importance sampling estimator\n", "####################################################\n", "M = 10\n", "\n", "#######################################################\n", "# Completer avec M tirages de l'estimateur d'importance\n", "# J_n pour n ranging from 1 to N\n", "J_n = ?????\n", "\n", "# Plot the M trajectories\n", "fig = plt.figure()\n", "ax = fig.add_subplot(111)\n", "ax.plot(integers1toN, J_n.T, color=\"b\")\n", "\n", "ax.set_xlim(0, N)\n", "ax.set_ylim(0, 3.e-2)\n", "ax.legend(loc=\"best\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Question 3: \n", "Compare the histograms of the errors of the estimators $I_n$ et $J_n$, for a fixed value of $n$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "N = 1000 \n", "M = 1000 # Sample size for the histogram of the error\n", "\n", "############################################\n", "# Complete with M x N draws \n", "# from the distribution N(0,1)\n", "# and from the distribution N(2,1)\n", "# and evalute the estimators\n", "GY = ?????\n", "\n", "GY_importance = ?????\n", "\n", "############################################\n", "# Evaluate the errors for the two estimators\n", "# Wished: sample of size M\n", "error_MC = ?????\n", "error_Importance = ?????\n", "\n", "# Plot the histograms of the errors\n", "plt.hist( ???, normed=\"True\", bins=int(np.sqrt(M)), label=\"error MC\")\n", "\n", "plt.hist( ???, normed=\"True\", bins=int(np.sqrt(M)), label=\"error Imp sampling\")\n", "\n", "plt.legend(loc=\"best\")\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### $\\blacktriangleright$ In practice:\n", "\n", "The optimal value $\\theta^*$ is the one that minimizes the variance of the importance sampling estimator: \n", "\n", "$$\n", "V(\\theta) = \n", "\\mathbb E \\left[ g(Y + \\theta) \\, e^{-2 \\, \\theta \\, Y - \\, \\theta^2} \\right]\n", "- \\mathbb E [g(Y)]^2.\n", "$$\n", "\n", "In practice, one can look for a value of $\\theta$ that minimizes an empirical approximation $V_n(\\theta)$ of $V(\\theta)$." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Application: estimation of VaR" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$\\newcommand{\\R}{\\mathbb{R}}$\n", "$\\newcommand{\\VR}{\\operatorname{VaR}}$\n", "Let $(X_i)_{i\\ge 1}$ be a sequence of i.i.d. random variables with density $f$, assumed to be continuous and strictly positive on an interval $I$ (and zero outside $I$, if $I$ is bounded).\n", "The related cdf is\n", "\n", "$$\n", "F(x) = \\mathbb{P}(X_1\\leq x) = \\int_{-\\infty}^x f(u) \\textrm{d} u, \\qquad x \\in \\R.\n", "$$\n", "\n", "The function $F:\\R \\rightarrow [0,1]$ being continuous and strictly increasing on $I$ (under the hypotheses above on the density $f$), the quantile function $Q : [0,1] \\rightarrow \\R \\cup \\{\\pm \\infty\\} $ is defined by:\n", "\n", "$$\n", "Q(u)=F^{-1}(u) \\quad \\textrm{ for all } \\quad u \\in (0,1), \n", "\\qquad Q(0)=\\inf I,\n", "\\qquad Q(1)=\\sup I.\n", "$$\n", "\n", "The goal of this exercise is to estimate the quantile function $Q$ from an i.i.d. sample $X_1, \\ldots, X_n$.\n", "When $X$ represents the loss of a position, the $\\operatorname{VaR}$ is defined by\n", "\n", "$$\n", "\\VR(\\alpha)=Q(1-\\alpha) \\qquad\n", "\\quad 0<\\alpha<1,\n", "$$\n", "\n", "for values of $\\alpha$ close to zero.\n", "\n", "We will consider $X_i$ with a standard Gaussian distribution (the functions `norm.cdf` and `norm.ppf` from `scipy.stats` can be used to compute the exact values of $F$ et $Q$.\n", "\n", "For every $n$, we denote $(X_{(i)})_{1\\leq i\\leq n}$ the variables $(X_i)_{1\\leq i\\leq n}$ sorted in the increasing order: \n", " \n", "$$\n", "X_{(1)} Q_n(u)\n", "########################################################\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "$\\newcommand{\\maj}{>}$\n", "#### Question 3: estimation of quantiles by Importance sampling \n", "\n", "The application of the importance sampling (IS) technique to the estimation of quantiles (hence of VaR) amounts to replacing the standard empirical approximation used above for the cdf of $X$ with the approximation obtained from IS.\n", "\n", "Since we are interested in the right tail of the distribution of $X$ (representing a loss value), we rather focus on the survival function $\\overline F(x) = \\mathbb P(X > x)$.\n", "It is immediate that\n", "\n", "$$\n", "Q(u) = \\inf\\{x \\in \\R : F(x) \\ge u\\}\n", "= \\inf\\{x \\in \\R : \\overline F(x) \\le 1- u\\},\n", "\\qquad u \\in (0,1).\n", "$$\n", "\n", "The central point of the method is the following: instead of the standard Monte-Carlo approximation $\\overline F_n(x) = \\frac 1n \\sum_{i=1}^n 1_{X_i \\maj x}$ of the survival function, we consider the approximation given by the IS method\n", "\n", "$$\n", "\\overline F_n^{\\theta}(x) =\n", "\\frac 1n \\sum_{i=1}^n\n", "1_{X_i + \\theta \\, \\maj x}\n", "\\,\n", "L(X_i),\n", "\\qquad \\mbox{where }\n", "L(G_i) = \\exp \\Bigl( -\\theta \\, X_i - \\frac{\\theta^2}2 \\Bigr).\n", "$$\n", "\n", "Note that, exactly as $\\overline F_n$, the function $\\overline F_n^{\\theta}$ is also piece-wise constant between the points $X_i$ (the difference being that its jumps at the points $X_i$ are not equal to $\\frac 1n$).\n", "The generalized inverse of $\\overline F_n^\\theta$, then, will also take values inside the sample $(X_i)_{1 \\le i \\le n}$.\n", "\n", "\n", "If we now define the _quantile by Importance sampling_ $Q_n^{\\theta}(u)$ by\n", "\n", "$$\n", "Q_n^{\\theta}(u)\n", "=\n", "\\inf\\{\n", "x \\in \\R:\n", "\\overline F_n^{\\theta}(x) \\le 1-u\n", "\\},\n", "$$\n", "\n", "we have\n", "\n", "$$\n", "Q_n^{\\theta}(u) = X_{(i(u))} + \\theta,\n", "\\qquad\n", "\\mbox{where } i(u) =\n", "\\max \\Bigl\\{ k : \\sum_{j = k}^n L(X_{(j)})\n", "\\le n (1 - u) \\Bigr\\}\n", "$$\n", "\n", "which is the estimator we want to implement." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Quantile level\n", "u = 0.95\n", "Var_u = sps.norm.ppf(u)\n", "\n", "# Simulations\n", "n = int(1.e3)\n", "X = np.random.randn(n)\n", "\n", "#######################################\n", "# The indexes that sort the array X\n", "# in the increasing order \n", "indexes = np.argsort(X)\n", "\n", "################################\n", "## Standard empirical quantile\n", "################################\n", "sorted_X = X[indexes]\n", "\n", "empirical_VaR = sorted_X[ int(n*u) ]\n", "\n", "###################\n", "## Quantile by IS\n", "###################\n", "\n", "############################################\n", "# A possible choice for the parameter theta \n", "theta = empirical_VaR\n", "\n", "weights_L = np.exp(-theta * sorted_X - (theta**2)/2)\n", "\n", "###############################################################\n", "# In order to compute the cumulated sums backwards from n to k,\n", "# we inverse the array of the weights L(G)\n", "weights_L_inverted = weights_L[::-1]\n", "\n", "cumulated_sum_weights = np.cumsum(weights_L_inverted)\n", "\n", "#############################################################\n", "# Complete with the evalution of the empirical quantile \n", "# by IS Q^{theta}_n defined above\n", "#############################################################\n", "empirical_VaR_IS = ???\n", " \n", "print(\"u = %1.2e, n = %1.0e \\n\" %(u, n) )\n", "\n", "print(\"True value of the VaR = %1.3f \\n\" %Var_u)\n", "\n", "print(\"Empirical VaR = %1.3f \\n\" %empirical_VaR)\n", "\n", "print(\"Empirical VaR by IS = %1.3f \\n\" %empirical_VaR_IS)\n", "\n", "\n", "################################\n", "# We can also plot the estimated\n", "# survival function: \n", "# Remove the if 0\n", "################################\n", "\n", "if 0:\n", " ###############################\n", " ## We fix some values for the x axis\n", " x_grid = sorted_X[n-80 : ]\n", "\n", " theoretical_values = sps.norm.sf(x_grid)\n", "\n", " plt.plot(x_grid, theoretical_values, \"k+\", label=\"True survival function\", markersize=13)\n", "\n", " ###############################\n", " # Standard MC estimate \n", " MC_estimates = np.arange(n, 0, -1, dtype=float) / n\n", "\n", " plt.plot(x_grid, MC_estimates[n-80:], color=\"red\", label=\"MC estimation\", linewidth=1.5)\n", "\n", " ###############################\n", " # Estimation by IS\n", " IS_estimates = np.zeros(x_grid.size)\n", "\n", " for i, x in enumerate(x_grid):\n", " IS_estimates[i] = np.mean( (X + theta > x) * np.exp(-theta * X -(theta**2)/2) )\n", "\n", " plt.plot(x_grid, IS_estimates, color=\"blue\", label=\"Estimation by IS\", linewidth=1.5)\n", "\n", " plt.title(r\"Estimation of the survival function $P(X > x)$\", fontsize=15)\n", " plt.xlabel(\"x\", fontsize=12)\n", " plt.legend(loc=\"best\", fontsize=12)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.5" } }, "nbformat": 4, "nbformat_minor": 2 }