Decomposing Model Error

We can decompose the model error into three different parameters all caused from different sources of errors.

$T o t a l E r r o r = A p p r o x i m a t i o n_e r r o r + S t a t i s t i c a l_e r r o r + O p t i m i z a t i o n_e r r o r$

Error Type	Source
Approximation Error	The chosen model might not be complex enough to perfectly capture the underlying function or geometry of the data.
Statistical Error	Caused by the limited amount of data and the inherent randomness (noise) present in the data sampling process.
Optimization Error	Error when the algorithm used to train the model fails to find the best set of parameters. (Gradient descent gets stuck in a local minima for example).

Closely related to Sources of Uncertainty, but from a different point of view.

Statistical error will be reduced if we exploit invariances in the input space.