Reconstructing and Recording Images the Mind Sees

1. Reconstructing and Recording Images the Mind Sees Anthony D’Onofrio BME 281 Section 2, Tuesdays October 23, 2012

2. What are Dreams Biggest mystery • Scientists do not know • No purpose at all • Mind’s just bored • Making connections loosely (Letting our mind wander) • Powered by emotion • Fright, Sadness, Happiness, Freedom • Dreams have been seen as a connection to the unconscious. • Controlled by imagination and previous memories as basis of dreams • Dreaming occurs mainly during REM sleep

3.  REM sleep • Rapid Eye Movement • Includes eye rapidly moving but also low muscle tone and a rapid, low-voltage EEG (electroencephalogram) • Recording of electrical activity along the scalp. • EEG measures voltage fluctuations resulting from current within the neurons of the brain. • REM sleep in adult humans typically occupies 20–25% of total sleep • 1–2 hours of a night's sleep. • 4-5 periods • Occurs close to morning. • Short at the beginning of the night and longer toward the end. •  It was identified and defined by Nathaniel Kleitman, Eugene Aserinsky, and Jon Birtwell in the early 1950s.

5. What if We Could Record Images? • The Gallant Lab at UC Berkeley • Dr. Gallant and Dr. Nishimoto • Main goal is to understand what portions of the brain become stimulated by a certain image • Develop a computer model that records those brain activities evoked and reconstructs them. • Whatever image we see in our mind, the computer model will read it, memorize it, and reconstruct it. • Prototype • It works

6. fMRI machine • Functional Magnetic Resonance Imaging • Instead of using EEG to read the brain waves • More accurate, computational representation • Procedure uses blood flow to detect connections between brain activation and task subject performs during the scan. • Specific to analyzing brain activity • Watch several hours of different movie clips

8. Computer Model: Looking at the Brain • Reads brain’s visual cortex • Determines the function of each brain section of the cortex • The computer will look at brain activity and notice certain patterns (shape, color, motion) that are similar in each movie • Example border color • Certain part of brain gets stimulated, when blue border appears • Computer reads that

10. Computer Model: The Data Collecting • Each neuron acts like a filter • Takes a visual stimulus as input, and produces a response as output. • Computer memorized all the outputs and data collected from brain activity • Builds a dictionary from that • Dictionaries create definitions for each neural output transmitted

11. The Experimental Process • Have same subject go into fMRI • Watch new set of movie clips • Computer does not know • Computer reads the brain activity • Deciphers the brain activity using the dictionaries created • Matches similar clips randomly downloaded from YouTube.

13. Recap and Conclusion • Record brain activity while subject watches several hours of movie trailers. • Build dictionaries to translate between shapes, edges and motion in the movies and measure brain activity. • Record brain activity to a new set of movie trailers that will be used to test the quality of the dictionaries and reconstructions. • Build a random library of 18,000,000 seconds of video downloaded at random from YouTube (that have no overlap with movies subjects saw in fMRI). • Put each of these clips through the dictionaries to generate predictions of brain activity. • Select the 100 clips whose predicted activity is most similar to the observed brain activity. • Average those clips together. • This is the reconstruction. • Surreal

14. https://www.youtube.com/watch?feature=player_embedded&v=nsjDnYxJ0bohttps://www.youtube.com/watch?feature=player_embedded&v=nsjDnYxJ0bo

15. Future • More accurate • Mapping and understanding more about which neurons are stimulated • Record dreams • Psychologically analyze people’s dreams • Schizophrenia • Post Traumatic Stress Syndrome • Put images into the head • Outputs would turn into inputs • Criminal Court rooms • Unaccurate • Ethical Concerns • Seeing memories • Help people in coma respond, stroke patients, dementia • People who can’t talk, talk through their images in head

16. Work Cited Page • Anwar, Yasmin. "Scientists Use Brain Imaging to Reveal the Movies in Our Mind." UC Berkeley News Center. Media Relations, 2011. Web. 18 Oct. 2012. <http://newscenter.berkeley.edu/2011/09/22/brain- movies/>. • "Brain Scanner Can Record Your Dreams on Video." DVICE (Syfy). The SyFy Online Network, 2011. Web. 18 Oct. 2012. <http://dvice.com/archives/2011/09/brain-imaging-c.php>. • "Dream Recording Device Around the Corner?" Neurogadget.com. N.p., 2011. Web. 18 Oct. 2012. <http://neurogadget.com/2011/09/28/dream-recording-device-around-the-corner/2708>. • Gallant, Jack L., and Shinji Nishimoto. "Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies - GallantLabUCB." Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies -GallantLabUCB. Google Sites, 2011. Web. 18 Oct. 2012. <https://sites.google.com/site/gallantlabucb/publications/nishimoto-et-al-2011>.

17. Work Cited Page • Gallant, Jack L., and Shinji Nishimoto. "Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies." Current Biology 21 (2011): 1641-646. Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies. Elsevier, 2011. Web. 18 Oct. 2012 • Ghosh, Pallab. "Dream Recording Device 'possible' Researcher Claims." BBC News. BBC, 27 Oct. 2010. Web. 18 Oct. 2012. <http://www.bbc.co.uk/news/science-environment-11635625>. • Moisse, Katie. "UC Berkeley Scientists 'See' Movies in the Mind." ABC News. ABC News Network, 22 Sept. 2011. Web. 18 Oct. 2012. <http://abcnews.go.com/Health/MindMoodNews/scientists- youtube-videos-mind/story?id=14573442>. • "Recording Dreams Science World." Recording Dreams « Science World. Science World, 2011. Web. 18 Oct. 2012. <http://blogs.voanews.com/science-world/2011/09/22/recording- dream/>.