2: Training Deep NNs (cont.); Introduction to Keras/Tensorflow; Application to Tabular Data

Quick Overview

Training a deep neural network involves designing the architecture, selecting a loss function appropriate for the output type (like binary cross-entropy for classification), and minimizing that loss using the gradient descent optimization algorithm, which generalizes the 19th-century method for high-dimensional parameter spaces like those in large models such as GPT-4.

Key Points: The rule of thumb for designing a network is to start with the simplest network and only increase complexity if performance is insufficient. The case study involves predicting heart disease diagnosis within one year using structured patient data from the Cleveland Clinic, featuring demographic and biomarker information. For the binary classification problem (heart disease yes/no), the network uses one hidden layer with 16 ReLU neurons and a sigmoid activation in the output layer to emit a probability. The input layer size was determined to be 29 features after one-hot-encoding some of the initial 13 input variables. The Mean Squared Error (MSE) is the loss function for regression, but for binary classification with sigmoid output, the Binary Cross Entropy loss function, expressed as minus log probability or minus log(1-probability), is utilized. Training involves minimizing the loss function by finding optimal weights and biases using Gradient Descent, an algorithm invented by Cauchy in 1847, implemented by iteratively updating parameters using the gradient (vector of partial derivatives) and a learning rate (alpha). In high-dimensional spaces (like models with billions of parameters), gradient descent stops near local minima or saddle points, which is empirically sufficient and avoids the risk of overfitting associated with finding a global minimum.

Context: The lecture continues the discussion on training deep neural networks, focusing on the practical steps and conceptual foundations required, such as optimization and loss functions. The discussion centers around a case study using structured data from the Cleveland Clinic to predict future heart disease diagnosis for patients presenting with other issues, setting up a classic binary classification problem to demonstrate the process using Keras/TensorFlow implementation.