1: Introduction to Neural Networks and Deep Learning; Training Deep NNs

Quick Overview

Deep learning fundamentally consists of neural networks, which are models that automatically learn data representations through layers of mathematical operations, primarily linear functions followed by non-linear activation functions, allowing them to process unstructured data where traditional machine learning failed due to manual feature engineering bottlenecks.

Key Points: The field of AI originated in 1956 at Dartmouth, with MIT researchers like Marvin Minsky, John McCarthy, and Claude Shannon being well represented among the founders. The traditional approach to AI relied on explicitly encoding human knowledge into brittle if-then rules, which failed because humans often know more than they can tell (Polanyi's paradox) and because edge cases are infinite. Machine learning succeeded the traditional approach by learning mappings from input-output examples using statistical techniques, but it required structured, numericalized data, struggling with unstructured data like images. Deep learning solves the unstructured data problem by automatically learning the necessary representations from raw input, effectively removing the massive human bottleneck of manual feature engineering. A node in a neural network is defined as a linear function followed by a non-linear activation function, and a vertical stack of these is called a layer; logistic regression is a neural network with no hidden layers. The Rectified Linear Unit (ReLU), defined as max(number, 0), is considered the hero of deep learning for hidden layers due to its empirical success, while output layer activation functions (like sigmoid or softmax) depend strictly on the required output format. Generative AI represents the latest breakthrough, sitting within deep learning, enabling the creation of unstructured data (text, images, audio) rather than just consuming it.

Context: This lecture introduces neural networks and deep learning, framing them within the historical evolution of Artificial Intelligence, which began formally in 1956. The discussion contrasts the initial rule-based AI approach with the subsequent success of machine learning and explains how deep learning, built upon neural networks, overcame the limitations of machine learning, especially concerning unstructured data processing.