Artificial Neural Networks (ANNs) are inspired by the biological brain, which consists of billions of neurons connected through synapses. Each neuron receives signals, processes them, and passes the result forward.
Similarly, an artificial neural network is made up of artificial neurons that mimics a biological neuron or nerve cell:
Receive inputs
Perform weighted computations
Produce outputs
The core motivation behind neural networks is simple:
We want machines to learn complex patterns directly from data, without explicitly programming rules.
Artificial Neuron vs Biological Neuron
A biological neuron consists of:
Dendrites → receive signals
Cell body (nucleus) → processes signals
Axon → transmits output via synapses
An artificial neuron mimics this behavior mathematically:
y^=
Check your understanding
Take a short quiz on this lesson before moving on.
This abstraction allows us to stack neurons into layers, forming a neural network.
Why Traditional Machine Learning Is Not Enough
Traditional machine learning models such as:
Linear Regression
Logistic Regression
Support Vector Machines
Decision Trees
work well for simple, structured problems. However, they struggle in real-world scenarios.
Why Do We Need Hidden Layers?
A network with only:
an input layer
an output layer
can only learn linear decision boundaries. We will understand that with examples here.
Hidden layers allow neural networks to:
Learn non-linear patterns
Build hierarchical feature representations
Solve complex real-world problems
Most real-world problems are impossible to solve using only input and output layers.
Linear Separability and Logic Gates
This image illustrates how the logical functions AND, OR, NAND, and NOR can be represented in a two-dimensional input space and separated using a single linear decision boundary. Each plot shows the four possible input combinations
(A,B), colored by their output class, with the dashed green line indicating the classifier’s decision boundary. Since a straight line is sufficient to separate the classes in all four cases, these functions are linearly separable and can be modeled using a single perceptron without any hidden layers.
That means:
A single straight line can separate the output classes
A single neuron (perceptron) can model them
XOR: The Breaking Point
The XOR function outputs:
1 when inputs are different
0 when inputs are the same
XOR is not linearly separable.
Unlike AND or OR, the positive and negative classes in XOR lie diagonally opposite to each other, making it impossible to separate them using a single straight decision boundary (shown by the dashed green line). This visualization highlights why XOR is not linearly separable and why solving it is not possible with just one neuron model.
This single example proves that single-layer models are insufficient.
Modeling AND and OR with a Single Neuron
AND Function Model
This diagram illustrates how the AND logical function can be implemented using a single artificial neuron (perceptron).
The neuron receives two binary inputs, x1 and x2, each scaled by their corresponding weights w1 and w2.
These weighted inputs are summed together along with a bias term b1, forming a linear combination w1x1+w2x2+b1.
This value is then passed through an activation function 𝜎(⋅), typically a sigmoid or step function, to produce the final predicted output y^.
We apply sigmoid here because we want to convert the neuron’s raw weighted sum into a bounded, interpretable output between 0 and 1.
By choosing appropriate values for the weights and bias, the neuron activates (outputs 1) only when both inputs are 1, and remains inactive (outputs 0) for all other input combinations. This demonstrates that the AND function is linearly separable and can be perfectly modeled using a single perceptron without any hidden layers.
Output behavior:
x1=1,x2=1⇒y^=1
All other inputs → y^=0
Consider the neuron:
y^=σ(z)
where:
z=20x1+20x2−30
Inputs:x1=0,x2=0
z=20(0)+20(0)−30=−30
y^=σ(−30)≈0
Inputs:x1=0,x2=1
z=20(0)+20(1)−30=−10
y^=σ(−10)≈0
Inputs:x1=1,x2=0
z=20(1)+20(0)−30=−10
y^=σ(−10)≈0
Inputs:x1=1,x2=1
z=20(1)+20(1)−30=10
y^=σ(10)≈1
Therefore, the neuron behaves like an AND gate:
x1
x2
y^
0
0
0
0
1
0
1
0
0
1
1
1
OR Function Model
Similarly, by choosing appropriate values for the weights and bias, the neuron activates (outputs 1) when at least one input is 1, and remains inactive (outputs 0) when both the inputs are 0.
Output behavior:
At least one input is 1 → y^=1
Consider the neuron:
y^=σ(z)
where:
z=20x1+20x2−10
Inputs:x1=0,x2=0
z=20(0)+20(0)−10=−10
y^=σ(−10)≈0
Inputs:x1=0,x2=1
z=20(0)+20(1)−10=10
y^=σ(10)≈1
Inputs:x1=1,x2=0
z=20(1)+20(0)−10=10
y^=σ(10)≈1
Inputs:x1=1,x2=1
z=20(1)+20(1)−10=30
y^=σ(30)≈1
Therefore, the neuron behaves like an OR gate:
x1
x2
y^
0
0
0
0
1
1
1
0
1
1
1
1
These examples show how:
Weights control feature importance
Bias shifts the decision boundary
Sigmoid approximates binary outputs
Why XOR Cannot Be Solved with a Single Neuron
For a single neuron:
y^=σ(w1x1+w2x2+b)
No choice of (w1,w2,b) can correctly model XOR.
This is a fundamental limitation of single-layer perceptrons.
Decomposing XOR into Simpler Functions
XOR can be rewritten as:
x1⊕x2=(x1∧¬x2)∨(¬x1∧x2)
Or equivalently:
x1⊕x2=(x1∨x2)∧¬(x1∧x2)
0⊕0=(0∨0)∧¬(0∧0)=0
0⊕1=(0∨1)∧¬(0∧1)=1
1⊕0=(1∨0)∧¬(1∧0)=1
1⊕1=(1∨1)∧¬(1∧1)=0
This shows that XOR is:
A composition of simpler logical functions
Each of which is linearly separable
Solving XOR Using a Hidden Layer
To solve XOR, we add:
One hidden layer
Two hidden neurons
Hidden layer
h1=σ(20x1+20x2−10)(Behaves like OR)
h2=σ(20x1+20x2−30)(Behaves like AND)
Output layer
y=σ(20h1−20h2−10)(Behaves like XOR)
The four diagrams below illustrate how different input combinations (x1,x2) activate the hidden neurons and lead to the correct output, demonstrating why hidden layers are necessary to solve non-linearly separable problems like XOR.
For inputs:(x1,x2)=(0,0)
h1=σ(20(0)+20(0)−10)=σ(−10)≈0
h2=σ(20(0)+20(0)−30)=σ(−30)≈0
Output:y^=σ(20(0)−20(0)−10)=σ(−10)≈
For inputs:(x1,x2)=(0,1)
For inputs:(x1,x2)=(1,0)
For inputs:(x1,x2)=(1,1)
This network:
Learns intermediate features
Combines them to produce XOR
Depth solves the problem.
From Logic Gates to Deep Learning
Modern neural networks extend this same idea. A neural network is a layered computational model composed of interconnected neurons that transform inputs into outputs through weighted connections and nonlinear activation functions.
It contains:
Input layer
Multiple hidden layers
Output layer
Weights wij(l)
Activations ai(l)
Input Layer
The input layer receives the raw features:
x1,x2,…,xn
These values are passed forward without modification and serve as the starting point of computation.
Hidden Layers
Each hidden layer consists of neurons that perform two operations:
A linear transformation
zj(l)=i∑wij(l)ai(l−1)+bj(l)
A nonlinear activation
aj(l)=σ(zj(l))
Here:
wij(l) is the weight connecting the i-th neuron in layer (l−1) to the j-th neuron in layer l
bj(l) is the bias of the j-th neuron in layer l
ai(l−1) is the activation from the previous layer
Hidden layers enable the network to learn hierarchical and non-linear representations of the input data.
Output Layer
The output layer applies the same computation to produce the final prediction:
y^=a(L)
Depending on the task, the activation function in this layer may vary (e.g., sigmoid for binary classification, softmax for multi-class classification).
Key Idea
By stacking multiple layers and learning the parameters w(l) and b(l), neural networks can approximate complex functions and model intricate patterns in data that are difficult or impossible to capture with linear models.
Neural networks learn representations, not rules.
Why Depth Matters
Hidden layers allow neural networks to:
Learn intermediate concepts
Build reusable feature hierarchies
Approximate complex non-linear functions
Examples:
Vision: edges → shapes → objects
Language: characters → words → meaning
Audio: frequencies → phonemes → speech
Depth is not optional — it is the source of intelligence.
Key Takeaways
Artificial Neural Networks are inspired by biological neurons, where inputs are combined using weights and bias and then transformed through an activation function.
A single artificial neuron, or perceptron, can learn only linearly separable patterns, such as the logical AND and OR functions.
The XOR problem demonstrates the limitation of single-layer networks because no single linear decision boundary can separate its outputs.
Hidden layers allow neural networks to solve complex, non-linear problems by learning intermediate representations of the data.
Deep Neural Networks learn a hierarchy of features, where early layers detect simple patterns and deeper layers combine them into higher-level abstractions.
The power of deep learning comes from stacking multiple layers, enabling networks to automatically learn complex functions directly from data.
