Artificial Brain and Office Mate based on Brain Information Processing Mechanism

1. Artificial Brain and Office Mate based on Brain Information Processing Mechanism 2007. 9. 14 Young-Ik Kim Brain Science Research Center, KAIST

2. Contents • Introduction- About Brain Neuro-Informatics Research Programof BSRC, KAIST- Main functionalities of human brain • Implementation of artificial brain system and its mechanisms- Auditory part- Vision part- Agent (Service) part • Artificial Brain System Demo • Toward more challenging problems

3. Introduction • About Brain Neuro-Informatics Research Program- The third phase project of Brain Science Research Center (BSRC) in KAIST. - Funded by Korean Ministry of Commerce, Industry, and Energy. - Complete research period: 2004. 7 ~ 2008. 3 • Research focus:- Understanding brain information processing mechanism- Developing brain-like intelligent systems (Artificial Brain)

4. Introduction • Motivations- We have achieved a great development of computer technologies, but the ability of machines is limited to simple tasks which require human beings have to order what to do. - We lack the specific and concrete algorithms to solve practical problems in the real world. - A human brain is the best model in solving practical problems in the real world, and we came up with neural networks based on the human neural information processing

5. Main Functionalities of Human Brain

6. Artificial Brain System - Development Env. • The development team- 11 research groups in 6 universities- 3 parts:auditory, vision, agent (secretary) • Each group generates functional modules:- developed independently- integrated using the de-centralized system service (DSS) on the Microsoft .Net framework. - The common language runtime (CLR) property in .Net framework enables each module can be developed in any languages like C#, C++, Java, etc.

7. Artificial Brain System – Overall Configuration Expression Recognition Speech Separation Face Recognition Object Recognition Sound Localization Speaker Recognition Attention Area Speech Recognition Vision Module Auditory Module Stereo- Camera Stereo- Microphone TCP/IP Service Module (Agent) Speaker Text-to- Speech Robot Control Robot Head Movement Response Sentence Generation Knowledge-Base Context Analysis Dialog Manager

8. Auditory Part – Module Diagram • Flow diagram for auditory perception Speech Stereo- Microphone Active Noise Canceller Auditory Filterbank Voice Activity Detection Noises Speech Recognition BSS (ICA) Sound Localization Speaker Recognition Masking Keyword Recognition

9. Auditory Part – Mechanisms • The superior olivary complex (SOC) receives bilateral ascending input from the auditory ventral cochlea nucleus (AVCN) and descending input from the ipsilateral inferior colliculus (IC). • The medial superior olive (MSO) cells are sensitive to interaural time difference (ITD) and the lateral superior olive (LSO) cells are sensitive to interaural intensity difference (IID). I. Binaural pathways and sound localization

10. The auditory signal is represented by the time at which upward zero-crossing occurs and the peak amplitude within the zero-crossing interval (D. Kim et.al., 1999). • Binaural cue extraction:- detect zero-crossing times- measure zero-crossing interval powers- The ITD and IID

11. SNR estimation (Y. Kim et.al, 2007) • Identification of reliable ITD samples (a) filtered signal (b) measured ITDs(c) SNR estimation(d) selected ITDs with SNR>15 dB

12. Localization of multiple sound sources(a) SNR weighted ITD histogram(b) local peaks of the histogram(c) normalized by the largest peak (d) selected dominant peaks with threshold value 0.3

13. Auditory Part – Mechanisms II. Masking of interfering sounds • Cocktail party problem- Human speech perception is robust in the presence of diffusive noise and interfering sounds. - But, machine speech recognition remains problematic in such conditions. • Auditory masking?- When a sound is masked, it is eliminated from perception as if the sound never reached the ear. - Sound source can be segregated by identifying the segments of the sources in the time-freq. domain.

14. Directional mask estimation (Y. Kim et.al., 2006) - Assign each zero-crossing interval power to one of the nearest ITD source- Mask based on the target-to-interferers power ratio for each time-freq. segments • Example mask estimation (a mixture of 3 sounds)- Target and interfering speeches located at 0, -30, 30 degrees. (a) Ideal mask (b) Estimated mask

15. Vision Part – Module Diagram • Flow diagram of visual perception

16. Vision Part – Mechanisms Biological visual pathway of bottom-up and top-down processing

17. The segmentation problem?- finding different objects in the image.. - But what is the image of a “single object”?- Is a nose an object? Is a head one? • Finding salient regions in an image! - Human brain draws attention to the salient object in the image. - The saliency of an image may be determined by the combination of local and global aspects.

18. The architecture of bottom-up saliency map model (Choi et.al,2006)- I: intensity, E: edge, S: symmetry- CSD&N: center-surround difference and normalization- ICA: independent component analysis- SM: saliency map, SP: saliency point- IOR: inhibition of return

19. Experimental results of bottom-up selective attention- The saliency map model generates candidates of interesting regions.

20. Service Part - Modules

21. Service Part - Scenarios • Service domains of the OffceMate:- schedule management- patent search - new knowledge acquisition from the internet - object perception in an office • A Demo for the schedule management

22. Toward More Challenging Problems • Keyword spotting model with top-down attention • Context-dependent information processing

23. Selective Attention with an HMM (C. Lee et.al., 2007) • Train HMMs with training set • For testing pattern, calc. likelihood for all classes • Choose Nc best for candidates • For each model, • Set attention filter to 0. • Update attention filter • Calc. new likelihood of changed input • Repeat 2)-3) until likelihood converges • Calc. confidence measure M • Choose maximum M 23

24. Keyword Spotting Model with Attention Keyword Model 2 Keyword Model 1 Garbage Model FB VAD signal Compare Likelihood & Decision Making Attention Filter Confidence Measure OOV Rejection Keyword? Attention Filter Confidence Measure OOV Rejection Activation Attention

25. Keyword spotting performance with SA

26. Context-dependent information processing • What is a context?- In memory, our experiences are represented in structure that cluster together with related information.- Little is known about the neural underpinnings of contextual analysis and scene perception. • Searching for relevant mechanisms: - K-Line by M. Minsky, The Society of Mind, 1986. - Sequence seeking and counter streams by S. Ullman, Cereb. Cortex, 1995. - Proactive brain using analogies and associations by M. Bar, TRENDS in Cog. Sci., 2007.

27. Translating analogies to predictive association (M. Bar, 2007)

28. Context-dependent information processing • Some Big Questions! - What are the computational mechanisms mediating the transformation of a past memory into a future thought? - How does the brain handle completely novel situations where no reliable predictions can be generated? - ... • Try a keyword-based context generation: - In our service area, there are 4 static domains. - Using the keywords in the domain, we can change the context in our service domains. - But in real situations, the keyword cannot be pre-determined! - More dynamic context generation and management are needed for efficient services.

29. Conclusions • Human brain is the best model in solving practical problems in the real world. • Our artificial brain system “OfficeMate” incorporates many current findings of information processing mechanisms in human brain. • New and challenging research areas are waiting for our attentions! • Thank you~Any research idea or comments are welcomed!youngik@kaist.ac.kr