Articles / Linear Algebra · Part 8

Linear Combinations, Span, and Basis

Learn how vectors can be combined to construct new vectors, understand span geometrically, and explore how bases provide fundamental representations of vector spaces with step-by-step examples.

Linear Combinations, Span, and Basis

Introduction

In linear algebra, vectors are not just individual quantities. We can combine them, use them to construct other vectors, and describe entire vector spaces using a small collection of vectors.

For example, a vector in two-dimensional space can often be constructed by combining two other vectors. Similarly, vectors in three-dimensional space can be represented using combinations of three suitable vectors.

These ideas form the foundation of linear combinations, span, and basis.

They help us understand how vectors relate to one another, how spaces are constructed, and how a collection of vectors can represent many different possibilities.

In this article, we’ll explore:

  1. What is a linear combination?
  2. How to calculate linear combinations.
  3. What is the span of vectors?
  4. Geometrical interpretation of span.
  5. What is a basis?
  6. Standard basis vectors.
  7. Numerical examples.
  8. Applications in computer science and machine learning.

Let’s begin.


1. What Is a Linear Combination?

A linear combination is a way of combining vectors by multiplying each vector by a scalar and then adding the resulting vectors.

Suppose we have two vectors:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
2
\end{bmatrix},
\qquad
\mathbf{v}_2=
\begin{bmatrix}
3\
1
\end{bmatrix}
\]

We can multiply them by scalars (a) and (b), respectively, and add them:

\[
a\mathbf{v}_1+b\mathbf{v}_2
\]

This expression is called a linear combination of \(\mathbf{v}_1\) and \(\mathbf{v}_2\).

Here, (a) and (b) can be any real numbers.

For example, if (a=2) and (b=3), then:

\[
2\mathbf{v}_1+3\mathbf{v}_2
\]

Substituting the vectors:

\[
2
\begin{bmatrix}
1\
2
\end{bmatrix}
+
3
\begin{bmatrix}
3\
1
\end{bmatrix}
\]

First, multiply each vector by its scalar:

\[
\begin{bmatrix}
2\
4
\end{bmatrix}
+
\begin{bmatrix}
9\
3
\end{bmatrix}
\]

Now add them:

\[
\begin{bmatrix}
11\
7
\end{bmatrix}
\]

Therefore:

\[
2\mathbf{v}_1+3\mathbf{v}_2=
\begin{bmatrix}
11\
7
\end{bmatrix}
\]

We have constructed a new vector using two existing vectors.

General Definition

A linear combination of vectors \(\mathbf{v}_1,\mathbf{v}_2,\ldots,\mathbf{v}_n\) is an expression of the form:

\[
c_1\mathbf{v}_1+c_2\mathbf{v}_2+\cdots+c_n\mathbf{v}_n
\]

Where \(c_1,c_2,\ldots,c_n\) are scalars.

These scalars are called coefficients.

The coefficients determine how much of each vector contributes to the resulting vector.


2. Finding a Linear Combination

Let’s understand how to determine whether a vector can be expressed as a linear combination of other vectors.

Suppose:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
2
\end{bmatrix},
\qquad
\mathbf{v}_2=
\begin{bmatrix}
3\
1
\end{bmatrix}
\]

We want to express the vector:

\[
\mathbf{b}=
\begin{bmatrix}
7\
5
\end{bmatrix}
\]

As a linear combination of \(\mathbf{v}_1\) and \(\mathbf{v}_2\).

We write:

\[
a\mathbf{v}_1+b\mathbf{v}_2=\mathbf{b}
\]

Substituting the vectors:

\[
\begin{bmatrix}
7\
5
\end{bmatrix}
\]

Expanding:

\[
\begin{bmatrix}
7\
5
\end{bmatrix}
\]

Adding the vectors:

\[
\begin{bmatrix}
7\
5
\end{bmatrix}
\]

This gives us a system of linear equations:

\[
\begin{aligned}
a+3b&=7\
2a+b&=5
\end{aligned}
\]

From the first equation:

\[
a=7-3b
\]

Substitute into the second:

\[
2(7-3b)+b=5
\]

\[
14-6b+b=5
\]

\[
-5b=-9
\]

\[
b=\frac{9}{5}
\]

Substituting back:

\[
a=7-3\left(\frac{9}{5}\right)
\]

\[
a=\frac{8}{5}
\]

Therefore:

\[
\boxed{a=\frac{8}{5},\qquad b=\frac{9}{5}}
\]

The vector can be expressed as:

\[
\mathbf{b}=
\frac{8}{5}\mathbf{v}_1+
\frac{9}{5}\mathbf{v}_2
\]

This means our target vector can be constructed using the two given vectors.


3. What Is the Span of Vectors?

The span of a collection of vectors is the set of all possible linear combinations of those vectors.

Suppose we have vectors:

\[
\mathbf{v}_1,\mathbf{v}_2,\ldots,\mathbf{v}_n
\]

Their span is written as:

\[
\operatorname{Span}
{\mathbf{v}_1,\mathbf{v}_2,\ldots,\mathbf{v}_n}
\]

It contains every vector that can be constructed by choosing any scalar coefficients and combining the given vectors.

Formally:

\[
\left\{
c_1\mathbf{v}_1+\cdots+c_n\mathbf{v}_n
\mid c_i\in\mathbb{R}
\right\}
\]

In simple terms, span describes all the vectors we can reach using a given collection of vectors.

Example

Consider:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
0
\end{bmatrix}
\]

The span of this vector is:

\[
\left\{
c
\begin{bmatrix}
1\
0
\end{bmatrix}
\mid c\in\mathbb{R}
\right\}
\]

This produces vectors such as:

\[
\begin{bmatrix}
2\
0
\end{bmatrix},
\quad
\begin{bmatrix}
-3\
0
\end{bmatrix},
\quad
\begin{bmatrix}
5\
0
\end{bmatrix}
\]

All these vectors lie along the horizontal axis.

Therefore, the span of \(\mathbf{v}_1\) is the horizontal line passing through the origin.


4. Geometrical Understanding of Span

The span depends on the vectors we are given.

Case 1: One Nonzero Vector in \(\mathbb{R}^2\)

Suppose:

\[
\mathbf{v}_1=
\begin{bmatrix}
2\
1
\end{bmatrix}
\]

Any scalar multiple of this vector lies along the same line.

\[
c\mathbf{v}_1=
\begin{bmatrix}
2c\
c
\end{bmatrix}
\]

Therefore, its span is a line passing through the origin.

Case 2: Two Nonparallel Vectors in \(\mathbb{R}^2\)

Suppose:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
0
\end{bmatrix},
\qquad
\mathbf{v}_2=
\begin{bmatrix}
0\
1
\end{bmatrix}
\]

A linear combination gives:

\[
\begin{bmatrix}
a\
b
\end{bmatrix}
\]

Since (a) and (b) can be any real numbers, we can construct every vector in the two-dimensional plane.

Therefore:

\[
\operatorname{Span}{\mathbf{v}_1,\mathbf{v}_2}
=\mathbb{R}^2
\]

Case 3: Two Parallel Vectors

Suppose:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
2
\end{bmatrix},
\qquad
\mathbf{v}_2=
\begin{bmatrix}
2\
4
\end{bmatrix}
\]

Notice that:

\[
\mathbf{v}_2=2\mathbf{v}_1
\]

Both vectors point along the same line.

Combining them does not create any new direction.

Their span is still only a line through the origin, not the entire plane.

Case 4: Three Vectors in \(\mathbb{R}^3\)

Three vectors can potentially span the entire three-dimensional space.

For example:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
0\
0
\end{bmatrix},
\quad
\mathbf{v}_2=
\begin{bmatrix}
0\
1\
0
\end{bmatrix},
\quad
\mathbf{v}_3=
\begin{bmatrix}
0\
0\
1
\end{bmatrix}
\]

Their linear combinations produce:

\[
\begin{bmatrix}
a\
b\
c
\end{bmatrix}
\]

Since (a,b,c) can be any real numbers, these vectors span all of \(\mathbb{R}^3\).

However, simply having three vectors in three-dimensional space does not guarantee that they span the entire space. Their directions matter.


5. What Is a Basis?

A basis is a collection of vectors that can represent every vector in a vector space through linear combinations, without unnecessary redundancy.

A set of vectors forms a basis for a vector space if:

  1. The vectors span the entire space.
  2. The vectors are linearly independent.

We will study linear independence in detail in the next article. For now, understand that it means none of the vectors can be constructed as a linear combination of the others.

Example: Standard Basis of \(\mathbb{R}^2\)

The standard basis vectors are:

\[
\mathbf{e}_1=
\begin{bmatrix}
1\
0
\end{bmatrix},
\qquad
\mathbf{e}_2=
\begin{bmatrix}
0\
1
\end{bmatrix}
\]

Any vector in \(\mathbb{R}^2\) can be written as:

\[
x\mathbf{e}_1+y\mathbf{e}_2
\]

For example:

\[
4
\begin{bmatrix}
1\
0
\end{bmatrix}
+
7
\begin{bmatrix}
0\
1
\end{bmatrix}
\]

The two standard basis vectors span the entire plane and provide independent directions.

Therefore, they form a basis for \(\mathbb{R}^2\).

Standard Basis of \(\mathbb{R}^2\)

Similarly, the standard basis vectors of three-dimensional space are:

\[
\mathbf{e}_1=
\begin{bmatrix}
1\
0\
0
\end{bmatrix},
\quad
\mathbf{e}_2=
\begin{bmatrix}
0\
1\
0
\end{bmatrix},
\quad
\mathbf{e}_3=
\begin{bmatrix}
0\
0\
1
\end{bmatrix}
\]

Any vector:

\[
\begin{bmatrix}
x\
y\
z
\end{bmatrix}
\]

Can be represented as:

\[
x\mathbf{e}_1+y\mathbf{e}_2+z\mathbf{e}_3
\]

These three vectors form the standard basis of (\mathbb{R}^3).


6. A Basis Does Not Have to Be the Standard Basis

There can be many different bases for the same vector space.

Consider these vectors:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\\
1
\end{bmatrix},
\qquad
\mathbf{v}_2=
\begin{bmatrix}
1\\
-1
\end{bmatrix}
\]

They are not the standard basis vectors.

However, they are not parallel, so they provide two different directions in the plane.

Let’s represent an arbitrary vector:

\[
\begin{bmatrix}
x\\
y
\end{bmatrix}
\]

As a linear combination:

\[
a
\begin{bmatrix}
1\\
1
\end{bmatrix}
+
b
\begin{bmatrix}
1\\
-1
\end{bmatrix}
=
\begin{bmatrix}
x\\
y
\end{bmatrix}
\]

This gives:

\[
\begin{aligned}
a+b&=x\\
a-b&=y
\end{aligned}
\]

Adding the equations:

\[
2a=x+y
\]

\[
a=\frac{x+y}{2}
\]

Subtracting the second equation from the first:

\[
2b=x-y
\]

\[
b=\frac{x-y}{2}
\]

We can find values of \(a\) and \(b\) for any real \(x\) and \(y\).

Therefore, these vectors also span \(\mathbb{R}^2\).

They form another basis for the plane.

This shows that a vector space can have multiple different bases.


7. Basis and Coordinates

A basis allows us to describe vectors using coordinates relative to that basis.

Using the basis:

\[
\mathbf{v}_1=
\begin{bmatrix}
1\
1
\end{bmatrix},
\qquad
\mathbf{v}_2=
\begin{bmatrix}
1\
-1
\end{bmatrix}
\]

Consider the vector:

\[
\mathbf{x}=
\begin{bmatrix}
4\
2
\end{bmatrix}
\]

We want to represent it using these basis vectors.

\[
\mathbf{x}=a\mathbf{v}_1+b\mathbf{v}_2
\]

We obtain:

\[
\begin{aligned}
a+b&=4\
a-b&=2
\end{aligned}
\]

Solving:

\[
a=3,\qquad b=1
\]

Therefore:

\[
\mathbf{x}=3\mathbf{v}_1+\mathbf{v}_2
\]

The coordinates of \(\mathbf{x}\) relative to this basis are:

\[
[\mathbf{x}]_B=
\begin{bmatrix}
3\
1
\end{bmatrix}
\]

Notice that the coordinates of a vector depend on the basis we use.

In the standard basis, the vector has coordinates:

\[
\begin{bmatrix}
4\
2
\end{bmatrix}
\]

In the new basis, its coordinates are:

\[
\begin{bmatrix}
3\
1
\end{bmatrix}
\]

The vector itself has not changed. Only its representation has changed.

This idea is important in coordinate transformations, computer graphics, and many mathematical applications.


8. Linear Combinations, Span, and Basis in Machine Learning

These concepts are useful in understanding how data and features are represented.

Feature Combinations

Suppose a machine learning model has two features:

  • Study hours.
  • Attendance.

A model may combine these features using weights:

\[
z=w_1x_1+w_2x_2+b
\]

The weighted combination of features is closely related to the concept of a linear combination.

Neural networks repeatedly perform these weighted combinations.

Data Representation

Suppose a dataset contains vectors representing different observations.

We may investigate whether these vectors can be represented using a smaller collection of fundamental vectors.

This can help us understand redundancy and the structure of the dataset.

Dimensionality Reduction

In dimensionality reduction, we attempt to represent data using fewer dimensions while preserving important information.

PCA, which we will study later in this series, identifies important directions in data and represents observations using those directions.

These ideas connect directly to spans, vector representations, and bases.


9. Summary

In this article, we explored three important concepts in linear algebra.

Linear Combination: Combining vectors by multiplying them by scalars and adding the results.

Span: The set of all vectors that can be constructed using linear combinations of a given collection of vectors.

Basis: A set of vectors that spans a vector space and is linearly independent.

We learned that:

  • Linear combinations allow us to construct new vectors.
  • Span describes all possible vectors we can reach using given vectors.
  • A single nonzero vector in \(\mathbb{R}^2\) spans a line through the origin.
  • Two nonparallel vectors in \(\mathbb{R}^2\) span the entire plane.
  • A basis provides a fundamental set of vectors for representing a space.
  • Different bases can represent the same vector space.
  • Coordinates depend on the basis chosen.

These concepts are essential for understanding the structure of vector spaces and the relationships between vectors.

What’s Next?

We have learned how vectors can be combined, what spaces they can span, and how bases represent vector spaces.

But how do we determine whether a collection of vectors contains redundant vectors? How can we tell whether vectors are independent or dependent? And how many independent directions exist in a vector space?

These questions lead us to our next article:

Article 9: Dimension and Linear Independence

We’ll explore how to determine whether vectors are linearly independent, how to identify redundant vectors, and how dimension describes the number of independent directions in a vector space.

Quick quiz

5 questions
What is a linear combination of vectors?
What does the span of a set of vectors represent?
What is the span of a single nonzero vector in (mathbb{R}^2)?
Which condition must a set of vectors satisfy to form a basis of a vector space?
Can the same vector have different coordinates under different bases?

Keep a copy

Read offline as a printout or PDF.

farhanpaal.com

Keep reading

Linear Algebra · Part 20

Singular Value Decomposition (SVD)

Learn Singular Value Decomposition (SVD), calculate singular values and vectors, and explore its applications in compression, machine learning, and dimensionality reduction.

6 min read

Ask Farhan

Ask anything about Farhan