Access Restriction

Author Bengio, Yoshua ♦ Guyon, I. ♦ Dror, G. ♦ Lemaire, V. ♦ Taylor, G. ♦ Silver, D.
Source CiteSeerX
Content type Text
File Format PDF
Language English
Subject Domain (in DDC) Computer science, information & general works ♦ Data processing & computer science
Subject Keyword Transfer Learning Scenario ♦ Deep Learning Algorithm Seek ♦ Unknown Structure ♦ Higher-level Representation ♦ Lower-level Feature ♦ Good Representation ♦ Individual Feature ♦ Unsupervised Learning ♦ Deep Learning ♦ Unsupervised Pre-training ♦ Input Distribution ♦ Training Distribution ♦ Unknown Factor ♦ Multiple Level
Description Deep learning algorithms seek to exploit the unknown structure in the input distribution in order to discover good representations, often at multiple levels, with higher-level learned features defined in terms of lower-level features. The objective is to make these higher-level representations more abstract, with their individual features more invariant to most of the variations that are typically present in the training distribution, while collectively preserving as much as possible of the information in the input. Ideally, we would like these representations to disentangle the unknown factors of variation that underlie the training distribution. Such unsupervised learning of representations can be exploited usefully under the hypothesis that the input distribution P (x) is structurally related to some task of interest, say predicting P (y x). This paper focusses on why unsupervised pre-training of representations can be useful, and how it can be exploited in the transfer learning scenario, where we care about predictions on examples that are not from the same distribution as the training distribution.
In Proc. of ICML
Educational Role Student ♦ Teacher
Age Range above 22 year
Educational Use Research
Education Level UG and PG ♦ Career/Technical Study
Learning Resource Type Article
Publisher Date 2011-01-01