M1.4 · LESSON 4 OF 6

Logarithms

Evaluate functions, compare growth rates and read a loss curve without confusing inputs and outputs.

THE IDEA

The undo button for exponents

An exponential asks "what do I get after x doublings?" The logarithm asks the reverse: "how many doublings do I need to reach y?" Written out, log₂(y) = x means 2ˣ = y. The two functions undo each other: log₂(2ˣ) = x, and 2^(log₂ y) = y for positive y. The base matters: base 2 counts doublings, base 10 counts zeros, and base e (about 2.7183, written ln) is the one calculus likes.

exponent x = 3

forward: 2^3 = 8

backward: log2(8) = 3

the log answers: which exponent gives 8?

The green dot (x, y) and the purple dot (y, x) swap their coordinates, which is why the curves mirror across the diagonal. A dot is missing when it falls outside the window.

Move the exponent x. The green dot (x, bˣ) and the purple dot (bˣ, x) swap their coordinates, so the curves are mirror images across the dashed diagonal.

Step 1 of 4

Start with a doubling table. 2⁰ = 1, 2¹ = 2, 2² = 4, 2³ = 8, 2⁴ = 16. The exponent counts the doublings.

Log rules: multiplying becomes adding

Because exponents add when you multiply powers (2³ · 2⁴ = 2⁷), the log of a product is the sum of the logs. Pick the exponents below and watch the equal columns. Notice the last row, which is not a rule.

a = 2^ = 8b = 2^ = 16

rule	left side	right side
product	log2(8·16) = 7	log2(8) + log2(16) = 3 + 4 = 7
quotient	log2(8/16) = -1	log2(8) − log2(16) = -1
power	log2(8³) = 9	3·log2(8) = 9
NOT a rule: sum	log2(8+16) = 4.585	log2(8) + log2(16) = 7

Multiplying numbers adds their exponents. That is the whole reason logs turn products into sums. The last row is the common mistake: there is no rule for the log of a sum, and the two sides differ.

With real numbers: multiplication on the left, addition on the right.

Predict first. What is log₂(8) + log₂(16)?

Why machine learning uses logs of probabilities

A model scoring a sequence multiplies the probability of each token. Tiny numbers multiplied together shrink fast. With 64-bit floats, a product of per-token probabilities of 0.001 reaches exactly 0 after 108 tokens, even though the sequence is not impossible, only unlikely. Take logs and the product becomes a sum of reasonable numbers (-6.9078 per token), and larger totals still mean "more likely", because log keeps the order of positive numbers.

tokens in the sequence n = 10

each token has probability	0.001
plain product of 10 probabilities	1.00e-30
sum of their natural logs	-69.0776

Repeated multiplication by 0.001 in ordinary 64-bit floats reaches exactly 0 at n = 108. The sum of logs is just -6.9078 per token, so it stays an ordinary number however long the sequence gets.

Drag the sequence length past the underflow point.

Chapter context

A function maps an input to an output under a stated rule. Graphs let us see the same rule at many inputs; formulas let us calculate a particular output exactly. Both views should describe the same quantity and units.

TRY IT YOURSELF

Part 1. Do it on the plot

Three small instruments share one state: the mirror plot (exponent and base), the rules table (two exponents), and the underflow meter (sequence length).

exponent x = 1

forward: 2^1 = 2

backward: log2(2) = 1

the log answers: which exponent gives 2?

The green dot (x, y) and the purple dot (y, x) swap their coordinates, which is why the curves mirror across the diagonal.

a = 2^ = 4b = 2^ = 8

rule	left side	right side
product	log2(4·8) = 5	log2(4) + log2(8) = 2 + 3 = 5
quotient	log2(4/8) = -1	log2(4) − log2(8) = -1
power	log2(4³) = 6	3·log2(4) = 6
NOT a rule: sum	log2(4+8) = 3.585	log2(4) + log2(8) = 5

Multiplying numbers adds their exponents. That is the whole reason logs turn products into sums. The last row is the common mistake: there is no rule for the log of a sum, and the two sides differ.

tokens in the sequence n = 10

each token has probability	0.001
plain product of 10 probabilities	1.00e-30
sum of their natural logs	-69.0776

Repeated multiplication by 0.001 in ordinary 64-bit floats reaches exactly 0 at n = 108. The sum of logs is just -6.9078 per token, so it stays an ordinary number however long the sequence gets.

To do: With base 2, move x until 2^x is exactly 0.125
To do: With base 2, find x so that 2^x is within 1 percent of 100 (the slider moves in steps of 0.05)
To do: Switch to base 10 and make 10^x equal 1000
To do: In the rules table, pick two different powers of 2 (neither equal to 1) whose product is 1024
To do: Lengthen the sequence until the plain product of probabilities underflows to exactly 0

Part 2. Do it in code

Write log_base(y, b): the exponent that turns base b into y. Use math.log. For y of 0 or less, math.log already raises ValueError; keep that behaviour.
Write doublings(n): how many times you must double 1 to reach at least n (so doublings(8) is 3). Use a loop with integers, not floating point log2. Raise ValueError for n below 1.
Write log_likelihood(ps): the sum of math.log(p) over probabilities ps. Raise ValueError if any p is 0 or less. Then compare it with the plain product, which underflows.

import math

def log_base(y, b):
    # log of y in base b
    pass

def doublings(n):
    # how many doublings of 1 reach at least n
    pass

def log_likelihood(ps):
    # sum of the natural logs of the probabilities
    pass

print(log_base(8, 2))

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Evaluate log base two of eight.

AEight, because log base two of eight is the number itself.BThree, because 2 cubed is eight.CFour, because log base two counts the digits of eight in binary.

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

f:X\to Y,\quad y=f(x),\quad \log_b(b^x)=x.

For the illustrative loss:

L(\theta)=(\theta-1.2)^2\ge0.

Theta is a tunable scalar, 1.2 is the target, and L is a nonnegative output. An exponent and its logarithm undo each other only on their valid domains.

Worked example

Evaluate L(0)=1.44, L(2)=0.64 and L(1.2)=0. The parameter 2 gives a larger input than 0 but a smaller loss. Squaring makes positive and negative distances contribute the same penalty. This is a synthetic landscape, not a claim that real models have one simple valley.

Reference code

The following example uses the Python standard library and does not call a remote service.

from math import isclose, log2
loss = lambda theta: (theta - 1.2) ** 2
assert isclose(loss(0), 1.44)
assert isclose(loss(2), 0.64)
assert loss(1.2) == 0
assert log2(8) == 3
assert sum(i*i for i in range(1, 4)) == 14

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

The output of a loss function is not its input parameter. Larger inputs do not always produce larger outputs. Logarithms require an explicit base when comparing information quantities.