ML1.1 · LESSON 1 OF 6

Features, labels and examples

Define a learning problem, establish a baseline and evaluate it on data not used for fitting.

THE IDEA

Machine learning starts with a table

Suppose you want to price a house. You could write rules by hand: "2.4 thousand per square metre, minus something for age". Machine learning flips this around. You collect past sales, and a learning algorithm chooses the rule from the data. For the rule to be learnable, the data must be organised in a very particular way.

An example is one row: one house.
A feature is one input column: area, age, distance to the centre.
The label is the column you want to predict: the sale price.

In code, the features become a 2D array X with one row per example and one column per feature, and the labels become a 1D array y with one entry per example. If you have 60 houses and 4 features, X.shape == (60, 4) and y.shape == (60,). Row i of X and entry i of y always describe the same house.

id	area_m2	bedrooms	age_yrs	dist_km	commission_k	price_k
1	101	2	35	17.2	6.6	221
2	104	2	29	17	6.5	216
3	122	4	17	8.5	8.8	292
4	55	1	25	17	3.4	112
5	78	2	3	15.1	5.9	198

First 5 of 60 synthetic houses (made by a seeded generator, not real sales). Tick X to use a column as a feature, choose y for the label.

X.shape == (60, 0) y.shape == (?,) X[0] == [] y[0] == ?

Pick one column as the label y.

7 columns: id = row number in the database; area_m2 = floor area in square metres; bedrooms = number of bedrooms; age_yrs = age of the building in years; dist_km = distance to the city centre in km; commission_k = agent commission paid at sale, in thousands; price_k = sale price in thousands.

A synthetic housing table (made by a seeded random generator, not real sales). Choose features and a label and watch X and y appear.

The rule for choosing features: only what exists at prediction time

A feature is allowed only if you will actually know it at the moment you make the prediction. The table has a column commission_k, the agent's fee. It is 3% of the price, so it predicts the price almost perfectly. But the fee is paid after the sale, so a house that is still for sale has no commission yet. A model trained with it looks brilliant in testing and is useless in use. This is called leakage: information from the outcome sneaking into the inputs.

Predict first. You want to predict tomorrow's sales total today. Which of these is a valid feature?

Shapes are the contract

Most beginner bugs here are shape bugs. Always print the shapes before training.

Building X and y with NumPy

import numpy as np
table = {"area_m2": [72, 110, 55], "age_yrs": [30, 12, 45], "price_k": [204, 331, 140]}
X = np.column_stack([table["area_m2"], table["age_yrs"]])   # shape (3, 2)
y = np.array(table["price_k"])                                  # shape (3,)
print(X.shape, y.shape)    # (3, 2) (3,)

Predict first. You have 1 feature and 50 examples. Which shape should X have?

Chapter context

A learning algorithm chooses a model from data rather than requiring a person to specify every decision rule. The resulting model is useful only if its predictions transfer to the intended future inputs under a clearly defined objective.

TRY IT YOURSELF

Part 1. Do it on the visual

The tasks tick from the real selection. Then you write the code that builds X and y yourself.

id	area_m2	bedrooms	age_yrs	dist_km	commission_k	price_k
1	101	2	35	17.2	6.6	221
2	104	2	29	17	6.5	216
3	122	4	17	8.5	8.8	292
4	55	1	25	17	3.4	112
5	78	2	3	15.1	5.9	198

First 5 of 60 synthetic houses (made by a seeded generator, not real sales). Tick X to use a column as a feature, choose y for the label.

X.shape == (60, 0) y.shape == (?,) X[0] == [] y[0] == ?

Pick one column as the label y.

To do: Make the sale price the label (y)
To do: Use area, age and distance as features
To do: Keep leaks and meaningless columns out: at least 3 features, none of them commission_k, id or the label
To do: Reach X.shape == (60, 4) with y.shape == (60,): all four honest features and the price as label

Part 2. Do it in code (NumPy)

Write build_xy(table, features, label): return X, y as NumPy float arrays. X has shape (n_examples, n_features) with columns in the order of features; y has shape (n_examples,).
build_xy must raise ValueError if features is empty, if the label is also listed as a feature, or if a column name is not in the table (and if the columns have different lengths).
Write drop_unavailable(features, unavailable): the features that remain after removing columns that would not exist at prediction time, keeping order and not changing the input list.
Write shape_summary(X, y): return (n_examples, n_features). Raise ValueError if X is not 2-dimensional or y has a different number of examples.

import numpy as np

# A tiny table: one list per column, one entry per house.
table = {
    "area_m2":      [72, 110, 55, 140],
    "bedrooms":     [2, 3, 1, 4],
    "age_yrs":      [30, 12, 45, 5],
    "commission_k": [6.1, 9.9, 4.2, 13.3],
    "price_k":      [204, 331, 140, 440],
}

def build_xy(table, features, label):
    # X: shape (n_examples, n_features), y: shape (n_examples,)
    pass

def drop_unavailable(features, unavailable):
    # features without the names in unavailable, same order
    pass

def shape_summary(X, y):
    # (n_examples, n_features), or ValueError for a bad shape
    pass

X, y = build_xy(table, ["area_m2", "age_yrs"], "price_k")
print(X.shape, y.shape)

Ctrl/⌘ + Enter runs it

Stuck, or something not working? You can come back to it any time.

CHECK YOUR UNDERSTANDING

Can tomorrow's final sales total be a feature when predicting tomorrow's sales today?

ANoBYes

Chapter reference: math, worked example & code

This section belongs to the whole chapter, so it is the same on every lesson in it and may cover a different lesson's topic. The code for the experiment you just ran is inside the experiment itself.

Math and notation

For labeled data $D=\{(x_i,y_i)\}_{i=1}^n$ , empirical risk is

\hat R(\theta)=\frac1n\sum_i\ell(f_\theta(x_i),y_i).

It estimates performance on the sampled population, not automatically on every deployment population. Accuracy is the fraction of correct predictions when classification is the task.

Worked example

A classifier predicts five held-out labels: [1,0,1,1,0] for actual labels [1,0,0,1,0]. Four are correct, so accuracy is 4/5=0.8. This number says nothing about its training accuracy or behavior on a differently collected test set.

Reference code

The following example uses the Python standard library and does not call a remote service.

actual = [1,0,0,1,0]
predicted = [1,0,1,1,0]
correct = sum(a == b for a,b in zip(actual,predicted))
assert correct == 4
assert correct/len(actual) == .8

Python examples are provided for study and local execution; this page does not execute Python.

Common mistakes

Learning a pattern does not establish causation. Validation data used repeatedly for model selection is no longer an untouched test. A random split is not always an independent split.