1. Making Watson Fast Daniel Brown HON111

2. Need for Watson to be fast to play Jeopardy successfully • All computations have to be done in a few seconds • Initial application speed: 1-2 hours processing time per question

3. Unstructured Information Management Architecture (UIMA): framework for NLP applications; facilitates parallel processing • UIMA-AS: Asynchronous Scaleout • UIMA chosen at start for these reasons; other optimization work only began after 2 years (after QA accuracy/confidence improved)

4. UIMA implementation of DeepQA

5. UIMA implementation of DeepQA • Type System • Common Analysis Structure (CAS) • Annotator • CAS multiplier (CM): creates new “children” CASes • Flow Controller • CASes can be spread across multiple systems (processed in parallel) for efficiency

6. Scaling out • Two systems: • Development (+question processing) • Meant to analyze many questions accurately • Production (+speed) • Meant to answer one question quickly

7. Scaling out: UIMA-AS • (UIMA-AS: Asynchronous Scaleout) • Manages multithreading, communication between processes necessary for parallel processing • Feasibility test: simulated production system with 110 processes, 110 8-core machines • Goal: less than 3 seconds; actual: more than 3 seconds • Two sources of latency: CAS serialization, network communication • Optimizing CAS serialization resulted in runtime of <1s

8. Scaling out: Deployment • 400 processes, 72 machines

9. How to find time bottlenecks in such a system? • Monitoring tool • Integrated timing measurements (in flow controller component)

10. RAM Optimizations • Wanted to avoid disk read/write time delays, so all (production system) data was put into RAM • Some optimizations: • Reference size reduction • Java object size reduction • Java object overhead • String size • Special hash tables • Java garbage collection with large heap sizes • *Full GC between games

11. Indri Search Optimizations • Indri search: used to find most relevant 1-2 sentences from Watson database • Using single processor, primary search takes too long (i.e. 100s) • Supporting evidence search even longer • Solution? • Divide corpus (body of information to search) into chunks, then assign each search daemon a chunk • (specifically, 50GB corpus of 6.8 million documents, 79 chunks of 100000 documents each, 79 Indri search daemons with 8 CPU cores each; end result, 32 passage queries could be run at once)

12. Preprocessing and Custom Content Services • Watson must first analyze the passage texts before being able to use them • Deep NLP analysis - semantic/structural parsing, etc. • Since Watson had to be self-contained, this analysis could be done before run time (preprocessed) • Used Hadoop (distributed file system software) • 50 machines, 16GB/8 cores each

13. Preprocessing and Custom Content Services • Retrieving the preprocessed data? • Preprocessed data much larger than unprocessed corpus (~300GB total) • Built custom content server – allocated data to 14 machines, ~20GB each • Documents then were accessed from these servers

14. End result • Parallel processing combined with a number of other performance optimizations resulted in a final average latency of less than 3 seconds. • No one “silver bullet” solution