{"id":9,"url":"https://pm.philipcastiglione.com/papers/9.json","title":"Helpless infants are learning a foundation model","read":true,"authors":"Rhodri Cusack, Marc’Aurelio Ranzato, and Christine J. Charvet","year":2024,"auto_summary":"The paper \"Helpless infants are learning a foundation model\" by Rhodri Cusack, Marc’Aurelio Ranzato, and Christine J. Charvet explores the prolonged period of helplessness in human infants and proposes a novel explanation for this phenomenon. Traditionally, it has been thought that humans are born with immature brains due to maternal constraints, such as the obstetrical dilemma, which necessitates early birth. However, the authors argue that human brains are not exceptionally immature at birth compared to other species with shorter helpless periods.\n\nRecent neuroimaging studies reveal that many cognitive systems in human infants are surprisingly mature at birth, with brain connectivity and functional activation showing similarities to mature brains. This challenges the notion that human infants are helpless due to brain immaturity.\n\nThe authors draw parallels between infant brain development and machine learning, particularly the concept of \"foundation models\" in deep neural networks (DNNs). In machine learning, DNNs benefit from a pre-training phase where they learn foundational representations through self-supervised learning, allowing for better generalization to new tasks. Similarly, the authors propose that human infants use their helpless period to develop foundational mental representations that underpin later cognitive abilities.\n\nThe paper reviews evidence from evolutionary biology, neuroimaging, and machine learning to support this hypothesis. It suggests that the extended helpless period in humans allows for the development of rich sensory and cognitive representations, akin to the self-supervised learning in DNNs. This period of learning without immediate action or goal-directed behavior is seen as crucial for building a robust foundation for future cognitive development.\n\nThe authors also discuss the implications of this theory for understanding atypical development and the potential evolutionary pressures that shaped human infancy. They highlight the need for further research to test predictions from this theory, such as the recruitment of additional brain systems after the helpless period and the comparison of representation learning in infants and DNNs.\n\nOverall, the paper presents a compelling argument that the prolonged helplessness in human infants is an adaptive feature that facilitates the development of a foundation model for flexible and intelligent cognition, drawing insights from both biological and artificial systems.","notes":{"id":9,"name":"notes","body":"\u003ch1\u003eNotes\u003c/h1\u003e\u003cdiv\u003e\u003cbr\u003eHumans are often claimed to be super altricial. Truly, babies start out very helpless. Even surprisingly slow to acquire behaviours, in some sense, eg:\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cblockquote\u003e\u003cbr\u003eDespite hearing hundreds of thousands of words per month [2] and the enormous value that a few words could confer, language develops slowly over multiple years.\u003cbr\u003e\u003cbr\u003e\u003c/blockquote\u003e\u003cdiv\u003e\u003cbr\u003eThe typical explanation is that:\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cblockquote\u003e\u003cbr\u003ematernal constraints causing an early birth while the brain is still immature.\u003cbr\u003e\u003cbr\u003e\u003c/blockquote\u003e\u003cdiv\u003e\u003cbr\u003eHowever,\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cblockquote\u003e\u003cbr\u003eanatomical studies have shown that relative to other species, human newborn brains are not particularly immature.\u003cbr\u003e\u003cbr\u003e\u003c/blockquote\u003e\u003cdiv\u003e\u003cbr\u003eThe paper will assert that human babies are doing something akin to a huge DNN building a foundation model.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eKey markers neurogenesis and axonal extension suggest humans are “relatively precocial” and that birth is relatively late compared with many other species. This despite human newborns being helpless for years.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eIn fact humans are inconsistently described as altricial/precocial/secondarily altricial, explained by heterochronies (sensory systems are developed at birth, locomotor systems are underdeveloped).\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eImaging and comparison between mammals can now show us this.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eFunctional networks are also developed in a variety of areas in newborns.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eNewborns appear to be actively engaged in learning.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eThis theory predicts:\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cul\u003e\u003cli\u003esequential recruitment of subsequent brain areas following the helplessness period of self-supervised learning\u0026nbsp;\u003cul\u003e\u003cli\u003ein the anterior or medial temporal lobes for classification via (analogous to) supervised learning\u003c/li\u003e\u003cli\u003ein the frontal lobes for action policies as in reinforcement learning\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003cli\u003ewe should be able to compare representation learning in DNNs to the brain\u0026nbsp;\u003cul\u003e\u003cli\u003efuture developments and improvement in biological plausibility of learning mechanisms within artificial models will help here\u003c/li\u003e\u003c/ul\u003e\u003c/li\u003e\u003c/ul\u003e\u003cdiv\u003e\u003cbr\u003eThere may be (weak) evidence, or at least corollaries, between the observed process/journey of development of visual recognition in infants and self-supervised learning (eg. in DNNs).\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eSimilarly with motor function (observed by comparison with RL system policy development).\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eThirdly, it is proposed that early self-supervised learning of a deep foundational model may lead to better representations, explaining the greater general intelligence and problem solving capabilities observed in adults in species with a longer helplessness period.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eIt would be interesting to comparatively study other species also in this context.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eAs always, you can’t be a purist. Social learning and reward/feedback is also an important part of infant experience and might contribute to the helplessness period. Infants are not passive observers but active learners, curiosity directs their learning.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eThere are many areas of inquiry to continue with!\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003ch1\u003eQuestions\u003c/h1\u003e\u003cdiv\u003e\u003cbr\u003eWhich references should I read? There are many that look interesting.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eHow could we get better evidence for (or against) this theory? The predictions are quite loose.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eIs it realistic that some sharp/profound qualitative change in learning mode occurs? might there be an alternative in that other systems just begin? and the foundations don’t need to change, but continuously refine and learn over our lives? we could test that by defining a scenario where the foundation model needs to shift profoundly and seeing that it does (or doesn’t). Maybe if you grow up in a small white room, then as an adult, enter the real (rich) world, we could observe adaptation (or lack thereof). Something like this has probably already been done in mice…\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cul\u003e\u003cli\u003eaha - cataracts\u003c/li\u003e\u003c/ul\u003e\u003cdiv\u003e\u003cbr\u003eSimilarly, the helplessness period is not binary, right? Infants transition into toddlers, children, … So there is no “end to the helplessness period”.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003ch1\u003eTakeaways\u003c/h1\u003e\u003cdiv\u003e\u003cbr\u003e“Humans are altricial” is an oversimplification, and the conventional explanation of humans being born with underdeveloped brains is incomplete and partially inaccurate.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eEarly infant learning (and helplessness) has structural similarities to the foundation model pretraining stage in self-supervised deep neural networks.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eHumans ability to generalise and learn quickly as adults may relate beneficially/causally to this early period of helplessness.\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003eThe landscape is complex and this is all very young!\u003cbr\u003e\u003cbr\u003e\u003c/div\u003e\u003cdiv\u003e\u003cbr\u003e\u003c/div\u003e","record_type":"Paper","record_id":9,"created_at":"2024-12-10T04:11:40.967Z","updated_at":"2024-12-10T04:13:21.959Z"},"created_at":"2024-12-10T04:11:32.441Z","updated_at":"2024-12-10T04:13:21.959Z"}