Representing Knowledge in a Neural Network

There are four general rules:

1. Similar inputs from similar classes should produce similar representations within the network and should be predicted as belonging to the same category.

   Similarity is based on distance measures which include the Euclidean distance, dot product and for input vectors from differing populations, the Mahalanobis distance which is defined as

   $$d^2_{ij} = ( \mathbf{x}_{i} - \mathbf{\mu}_{i} )^{T} \Sigma^{-1} ( \mathbf{x}_{i} - \mathbf{\mu}_{i} )$$
   where $\mathbf{\mu}_{i}$ is the mean vector for the population $\mathbf{x}_{i}$ and $\Sigma^{-1}$ is the inverse of the covariance matrix
2. Inputs coming from different classes should have widely differing represenations in the network
3. If a feature is important there should be a large number of neurons devoted to representing that feature
4. Network design should be simplified by building into it prior knowledge and invariance