Sparse coding with lateral inhibition in restricted Boltzmann machines and hierarchical competitive networks

Previous neurophysiological findings and computational simulations have revealed the use of sparse, over-complete representations in primary visual cortex. In this work, we utilize natural image and handwritten digit datasets to explore two sparse coding models: Restricted Boltzmann Machines (RBMs) with lateral inhibition and winner-take-all networks with small receptive fields. We find that applying fixed weight lateral inhibition to RBMs is efiective in sparsifying their representation. This enabled individual nodes to capture longer-range structure in the inputs, but this was limited in the case of natural images and required PCA dimensionality reduction with handwritten digits.

Winner-take-all learning was found to be more eficient and robust in capturing long-range structure. Using small receptive fields and a Gaus- sian pre-filter to encourage nodes to learn filters which are centred in their receptive field, allows us to minimize the number of required filters in a convolutional winner-take-all learning model. This network significantly increase the linear separability of handwritten digits. While second layer filters with larger receptive fields successfully learn coherent object parts, a computationally tractable number of such filters did not further increase the representation's separability.