Deep unsupervised learning on graphic processors
Unsupervised deep learning with Deep Belief Networks (DBNs)
Deep belief networks (DBNs) hold great promise for the simulation of human cognition because they show how structured and abstract representations may emerge from probabilistic unsupervised learning. These networks build a hierarchy of progressively more complex distributed representations of the sensory data by fitting a hierarchical generative model (Hinton & Salakhutdinov, 2006). We pioneerd the application of DBNs to cognitive modeling in the project "Generative models of human cognition" (GENMOD) funded by the European Research Council, with a notable example on modeling numerosity perception (Stoianov & Zorzi. 2012). In our early work we also developed DBN code exploiting low-cost graphic cards (GPUs) without any specific programming effort to perform large-scale simulations (involving millions of connection weights) on a desktop PC (Testolin et al., 2013). More recently, we introduced a novel iterative learning algorithm for DBNs that allows to jointly update the connection weights across all layers of the model (Zambra, Testolin, & Zorzi, 2023).
You can download the complete source code of our deep belief net implementations here (the original model can be found on Geoffrey Hinton's web page):
We also provide a series of useful Matlab routines that can be used to analyze deep belief networks, for example by plotting the shape of the receptive fields at different levels of the hierarchy or by performing simple linear read-outs to simulate explicit behavioral tasks. Two examples of deep belief networks trained on the MNIST data set can be downloaded here.
Code for the iDBN is available on the lab GitHub
Applications to cognitive modeling
In order to try unsupervised deep learning on the prototypical cognitive modeling problem of visual numerosity perception investigated by Stoianov & Zorzi (2012), you can download the complete dataset of visual images here and follow the instructions provided inside the archive. Code for the follow-up work by Testolin, Dolfi, Rochus, & Zorzi (2020) is available through the Open Science Framework
We provide the dataset (and the adapted source code) used in Di Bono & Zorzi (2013) to investigate word recognition with deep generative models here.
The confusion and similarity matrices produced by the unsupervised deep learning model of printed letters described in Testolin, Stoianov and Zorzi (2017) can be found here. The weights of the single-layer network trained on natural images can be found here. The full dataset containing uppercase, whitened Latin letters, printed using a variety of different fonts, styles and sizes (MATLAB format) is also available through the Open Science Framework.
If you find this code useful, please cite our work as:
Testolin, A., Stoianov, I., De Filippo De Grazia, M., & Zorzi, M. (2013). Deep unsupervised learning on a desktop PC : A primer for cognitive scientists. Frontiers in Psychology, 4 (251).
Zorzi, M., Testolin, A., & Stoianov, I. (2013). Modeling language and cognition with deep unsupervised learning: a tutorial overview. Frontiers in Psychology, 4 (515).
Zambra, M., Testolin, A. & Zorzi, M. (2023). A Developmental Approach for Training Deep Belief Networks. Cognitive Computation, 15, 103–120.