M2.4 · LESSON 4 OF 6

Cosine similarity

Compute vector operations, lengths, angles and projections and explain what each measures.

THE IDEA

Keep the angle, drop the length

The dot product mixes angle and length. To keep only the direction, divide by both lengths: cos(a, b) = a·b / (‖a‖‖b‖). From a·b = ‖a‖‖b‖cos θ this is exactly cos θ, the cosine of the angle between the vectors. It is 1 for the same direction, 0 for perpendicular, and -1 for opposite directions, whatever the lengths.

Worked example: a = [2, 3] and b = [4, 1]. The dot product is 11, the norms are √13 and √17, so the cosine is 11/√221 = 0.74, an angle of 42.274 degrees.

stretch b by k = 1

	dot	‖a‖·‖b‖	cos
a, b	5	7.071	0.707
a, 1·b	5	7.071	0.707

cos(a, b) = a·b / (‖a‖‖b‖)

= 5 / (3.162 · 2.236) = 0.707

θ = arccos(0.707) = 45 degrees

Drag a and b and read the angle. Then stretch b with the slider: the dot product changes, the cosine does not.

Why scale cancels

Replace b by k·b with k above 0. The numerator becomes k times larger (a·(k·b) = k·(a·b)) and so does the denominator (‖k·b‖ = k‖b‖). The k cancels. For negative k the direction flips and the cosine changes sign, so the cosine only ignores a positive stretch.

Predict first. Does doubling a nonzero vector change its cosine similarity with another vector?

Predict first. What is the cosine of [1, 0] and [-5, 0]?

Common mistakes

Rounding can push a cosine just past 1

import numpy as np
a = np.array([0.1, 0.1, 0.3])
b = np.array([0.03, 0.03, 0.09])      # b is 0.3 times a, so the angle is 0
c = a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
print(c)                 # 1.0000000000000002
print(np.arccos(c))      # nan: arccos only accepts -1 to 1
print(np.arccos(np.clip(c, -1, 1)))   # 0.0

Always clip to [-1, 1] before arccos. And remember the other convention trap: higher cosine means more similar, while higher distance means less similar. They sort in opposite directions.

Chapter context

A vector is an ordered set of coordinates in a specified space. It can describe a displacement, a data record or a learned representation. Operations are meaningful only when the coordinates refer to compatible dimensions.

TRY IT YOURSELF

Part 1. Do it on the plane

Rows of the table show a with b and a with k times b. The angle arc is drawn at the origin.

stretch b by k = 1

	dot	‖a‖·‖b‖	cos
a, b	5	7.071	0.707
a, 1·b	5	7.071	0.707

cos(a, b) = a·b / (‖a‖‖b‖)

= 5 / (3.162 · 2.236) = 0.707

θ = arccos(0.707) = 45 degrees

To do: Make cos(a, b) equal 0 with both vectors nonzero (a right angle)
To do: Make cos(a, b) equal -1 while b has a different length from a
To do: With a · b positive, stretch b by k = 2 or more and see the dot product grow while cos stays put

Part 2. Do it in code

Write cosine(a, b) with numpy. A zero vector has no direction, so raise ValueError.
Write angle_degrees(a, b). Rounding can push a cosine slightly above 1, and arccos of that is nan, so clip it to [-1, 1] first.

import numpy as np

def cosine(a, b):
    # dot product divided by both norms; ValueError for a zero vector
    pass

def angle_degrees(a, b):
    # the angle between the vectors, in degrees
    pass

print(cosine([2, 3], [4, 1]))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Does doubling a nonzero vector change its cosine similarity with another vector?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

a\cdot b=\sum_i a_i b_i,\quad \|a\|_2=\sqrt{\sum_i a_i^2}.

\cos(a,b)=\frac{a\cdot b}{\|a\|_2\|b\|_2},\quad \operatorname{proj}_b a=\frac{a\cdot b}{b\cdot b}b.

Both vectors must have the same dimension; cosine and projection need the denominator to be nonzero.

Worked example

For a=[2,3] and b=[4,1], matching products are 8 and 3, so the dot product is 11. Squared norms are 13 and 17. The cosine is 11/sqrt(221), approximately 0.73994. The projection of [3,4] onto [1,0] keeps only the first coordinate.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import sqrt, isclose
from operator import mul
a, b = [2, 3], [4, 1]
dot = sum(map(mul, a, b))
assert dot == 11
cosine = dot / (sqrt(sum(x*x for x in a))*sqrt(sum(x*x for x in b)))
assert isclose(cosine, 11/sqrt(221))
assert isclose(cosine, 0.7399400734, abs_tol=1e-9)

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Dot product is not elementwise multiplication alone. Similarity and distance have opposite ordering conventions. Zero vectors cannot be normalized by dividing by their norm.