Getting started with ElasticSearch on Windows and .NET with NEST

1. Getting started with ElasticSearch on Windows and .NET with NEST A shortintroduction Oslo/NNUG Meetup Tomas Jansson 29/01/2014

2. This is me Tomas Jansson Manager & Group Lead .NET BEKK Oslo @TomasJansson tomas.jansson@bekk.no github.com/mastoj blog.tomasjansson.com

3. TL;DR; https://github.com/mastoj/NestDemo

6. Audience N00b Expert Expert N00b

7. BackgrounD This is the data and weneedthisnewapplication

8. The masterplan

9. What I want to show you is... • Elasticsearch is awesome • Indexing using NEST • Querying using NEST • ... not about advanced elasticsearch hosting

11. Installation • Great news, install as a service added in 0.90.5 Powershell to the rescue

12. NEST Abstraction over Elasticsearch There is an lowlevelabstraction as wellcalledRawElasticClient

13. NEST Abstraction over Elasticsearch Fluent & Stronglytyped

14. FunctionalC#

15. FUNC DEMO • C:\Dev\git> scriptcs • scriptcs (ctrl-c or blank to exit) • > Func<int, int, int> add = (x, y) => x + y; • > add(1, 3) • 4 Func  executable

16. Simple Expression DEMO • > usingSystem.Linq.Expressions; • > Expression<Func<int, int, int>> addExpr = (x, y) => x + y; • > addExpr(1, 3) • (1,1): error CS1955: Non-invocablemember 'addExpr' cannot be used like a method. • > var binExpr = addExpr.Body as BinaryExpression; • > Console.WriteLine(binExpr); • (x + y) • > var add2 = addExpr.Compile(); • > add2(3, 1); • 4 Expression  ”function description”

17. More complex Expression DEMO • > public class SomeClass { public string MyString { get; set; } } • > Expression<Func<SomeClass, object>> propExpr = y => y.MyString + y.MyString; • > varcompExpr = propExpr.Compile(); • > varobj = new SomeClass { MyString = "Hello world" }; • > compExpr(obj) • Hello worldHelloworld • > varbody = propExpr.Body as BinaryExpression; • > Console.WriteLine(body); • (y.MyString + y.MyString) • > varleft = body.Left as MemberExpression; • > Console.WriteLine(left.Member.Name); • MyString

18. More complex Expression DEMO • > public class SomeClass { public string MyString { get; set; } } • > Expression<Func<SomeClass, object>> propExpr = y => y.MyString + y.MyString; • > varcompExpr = propExpr.Compile(); • > varobj = new SomeClass { MyString = "Hello world" }; • > compExpr(obj) • Hello worldHelloworld • > varbody = propExpr.Body as BinaryExpression; • > Console.WriteLine(body); • (y.MyString + y.MyString) • > varleft = body.Left as MemberExpression; • > Console.WriteLine(left.Member.Name); • MyString Enables us to translate from one domain to another in an ”easy” manner

19. Show me the code!

20. Elasticsearchconnection • publicclassElasticClientWrapper : ElasticClient • { • privatestaticstring _connectionString=Settings.ElasticSearchServer; • privatestaticConnectionSettings _settings = • newConnectionSettings(newUri(_connectionString)) //http://demoserver:9200 • .SetDefaultIndex(Settings.Alias) //"customer_product_mapping" • .UsePrettyResponses(); • publicElasticClientWrapper() • : base(_settings) • { • } • } • //usage • varclient=newElasticClientWrapper();

21. Mapping • publicclassProduct • { • publicdoubleUnitPrice { get; set; } • publicintTotalQuantity { get; set; } • [ElasticProperty(Index =FieldIndexOption.not_analyzed)] • publicstringProductName { get; set; } • [ElasticProperty(Index =FieldIndexOption.not_analyzed)] • publicstringCategoryName { get; set; } • } • publicclassCustomer • { • publicstringCustomerID { get; set; } • publicstringCompanyName { get; set; } • publicstring Address { get; set; } • publicstring City { get; set; } • publicstring Country { get; set; } • [ElasticProperty(Type =FieldType.nested)] • publicProduct[] Products { get; set; } • }

22. Mapping & Indexing • _client=newElasticClientWrapper(); • _client.CreateIndex("indexName", s => • s.AddMapping<Customer>(m =>m.MapFromAttributes())); • varcustomers= _customerRepo.GetCustomers(); • _client.IndexMany(customers, "indexName"); Mapping created from attributes Indexing will use the mapping for the specified index There is async versions of the methods

23. Alias • _client=newElasticClientWrapper(); • _client.Alias("indexName", "aliasName"); Alias Index_01

24. Swapping • _client=newElasticClientWrapper(); • _client.Swap("aliasName", new [] { "Index_01" }, new [] { "Index_02" } ); Alias Alias Create new index Swap Delete old index Index_01 Index_02

25. My query object (Will be used in the examples) • publicclassSearchModel • { • privateint? _numberToTake; • publicstring Query { get; set; } • publicDictionary<string, IEnumerable<string>> Filter { get; set; } • publicint?NumberToTake • { • get{ return _numberToTake.HasValue? _numberToTake.Value : 25; } • set{ _numberToTake=value; } • } • }

26. QueryinG Elasticsearch NEST • { • "query": { • "query_string": { • "query": "tomas" • } • } • } • _client.Search<Customer>(sd=>sd • .QueryString(Input.Query));

27. Fuzzy Elasticsearch NEST • { • "query": { • "fuzzy": { • "_all": { • "min_similarity": 0.6, • "prefix_length": 1, • "value": "tomas" • } • } • } • } • _client.Search<Customer>(sd=>sd • .Query(q =>q • .Fuzzy(fd=>fd • .OnField("_all") • .MinSimilarity(0.6) • .PrefixLength(1) • .Value(Input.Query)))); Will enableus to search for both «Thomas» and «Tomas» whenwriting «Tomas»

28. Fuzzy Improved (using bool query) - Elasticsearch • { • "query": { • "bool": { • "should": [{ • "match": { • "_all": { • "query": "tomas" • } • } • }, • { • "fuzzy": { • "_all": { • "boost": 0.1, • "min_similarity": 0.6, • "prefix_length": 1, • "value": "tomas" • } • } • }] • } • } • }

29. Fuzzy Improved (using bool query) - NEST • _client.Search<Customer>(sd=>sd • .Query(q =>q • .Bool(b =>b • .Should(newFunc<QueryDescriptor<Customer>, BaseQuery>[] • { • _ => _.Match(m =>m • .OnField("_all") • .QueryString(Input.Query)), • _ => _.Fuzzy(fd=>fd • .OnField("_all") • .MinSimilarity(0.6) • .PrefixLength(1) • .Value(Input.Query) • .Boost(0.1)) • }))));

30. Highlight result - Elasticsearch • { • "query": { • // seepreviousexample • }, • "highlight": { • "pre_tags": [ • "<span class='highlight'>" • ], • "post_tags": [ • "</span>" • ], • "fields": { • "companyName": { • "fragment_size": 100, • "number_of_fragments": 1 • } • } • } • }

31. Highlight result - NEST • _client.Search<Customer>(sd=>sd • .Query( /* See previousexample */ ) • .Highlight(h =>h • .PreTags("<span class='highlight'>") • .PostTags("</span>") • .OnFields(newAction<HighlightFieldDescriptor<Customer>>[] • { • _ =>_ • .OnField(c =>c.CompanyName) • .NumberOfFragments(1) • .FragmentSize(100) • })));

32. Facets - Elasticsearch • { • "query": { /* See previousexample */ }, • "highlight": { /* See previousexample */ }, • "facets": { • "products.productName": { • "nested": "products", • "terms": { "field": "products.productName", "size": 1000 } • }, • "products.categoryName": { • "nested": "products", • "terms": { "field": "products.categoryName", "size": 1000 } • }, • "country": { • "terms": { "field": "country", "size": 1000 } • } • } • }

33. Facets - NEST • _client.Search<Customer>(sd=>sd • .Query( /* See previousexample */) • .Highlight( /* See previousexample */) • .FacetTerm(f =>f • .Nested(c =>c.Products) • .OnField(c =>c.Products[0].ProductName) • .Size(1000)) • .FacetTerm(f =>f • .Nested(c =>c.Products) • .OnField(c =>c.Products[0].CategoryName) • .Size(1000)) • .FacetTerm(f =>f • .OnField(c =>c.Country) • .Size(1000)));

34. http://go-gaga-over-testing.blogspot.no/2011/09/solution-to-warning-in-quality-center.htmlhttp://go-gaga-over-testing.blogspot.no/2011/09/solution-to-warning-in-quality-center.html

35. Filters - Elasticsearch • { • "query": { • "filtered": { • "query": { /* See previousexample */ }, • "filter": { • "bool": { • "must": [ • { • "terms": { "country": ["usa"] } • }, • { • "nested": { • "query": { "terms": { "products.categoryName": ["Condiments", "Seafood"] } }, • "path": "products" • } • }, • { • "nested": { • "query": { "terms": { "products.productName": ["Chai"] } }, • "path": "products" • } • } • ] • } • } • } • }, • "facets": { /* See previousexample */}, • "highlight": { /* See previousexample */ } • }

36. Filters – NEST – Part 1, the Customers filter • privatestaticBaseFilterAddCustomerFilter(IEnumerable<string>items, Expression<Func<Customer, object>>propExpr) • { • returnFilter<Customer>.Terms(propExpr, items.ToArray()); • }

37. Filters – NEST – Part 1, the Products filter • privatestaticBaseFilterAddProductsFilter(IEnumerable<string> items, • Expression<Func<Customer, object>>propExpr) • { • returnFilter<Customer>.Nested(sel => sel • .Path(c =>c.Products) • .Query(q =>q.Terms(propExpr, items.ToArray()))); • }

38. Filters – NEST – Part 1, the Magic Dictionary • publicDictionary<string, Func<IEnumerable<string>, BaseFilter>>FilterDesc= • newDictionary<string, Func<IEnumerable<string>, BaseFilter>>() • { • {"products.productName", ps=>AddProductsFilter(ps, c => c • .Products[0].ProductName)}, • {"products.categoryName", cs=>AddProductsFilter(cs, c =>c • .Products[0].CategoryName)}, • {"country", cs=>AddCustomerFilter(cs, c =>c.Country)} • };

39. Filters – NEST – Part 1, All the helpers • privatestaticBaseFilterAddCustomerFilter(IEnumerable<string>items, Expression<Func<Customer, object>>propExpr) • { • returnFilter<Customer>.Terms(propExpr, items.ToArray()); • } • privatestaticBaseFilterAddProductsFilter(IEnumerable<string> items, • Expression<Func<Customer, object>>propExpr) • { • returnFilter<Customer>.Nested(sel => sel • .Path(c =>c.Products) • .Query(q =>q.Terms(propExpr, items.ToArray()))); • } • publicDictionary<string, Func<IEnumerable<string>, BaseFilter>>FilterDesc= • newDictionary<string, Func<IEnumerable<string>, BaseFilter>>() • { • {"products.productName", ps=>AddProductsFilter(ps, c => c • .Products[0].ProductName)}, • {"products.categoryName", cs=>AddProductsFilter(cs, c =>c • .Products[0].CategoryName)}, • {"country", cs=>AddCustomerFilter(cs, c =>c.Country)} • };

40. Filters – NEST – Part 2, The query • _client.Search<Customer>(sd=>sd • .Query(q =>q • .Filtered(fq=> • { • fq.Query(qs=> • { • if(!string.IsNullOrEmpty(Input.Query)) • { • qs.Bool( /* See previousexample */)); • } • else • { • qs.MatchAll(); • } • returnqs; • }); • if(Input.Filter.Count>0) • { • varfilters = • Input.Filter.Select(_ =>FilterDesc[_.Key](_.Value)).ToArray(); • fq.Filter(fs=>fs.Bool(bf =>bf.Must(filters))); • } • })) • .Highlight( /* See previousexample */) • .FacetTerm( /* See previousexample */) • .FacetTerm( /* See previousexample */) • .FacetTerm( /* See previousexample */);

41. Summary Elasticsearch NEST • Easy installation • Awesome search engine • Strongly typed client • Fluent • Abstraction over Elasticsearch

42. ResourceS • Demo code: https://github.com/mastoj/NestDemo • Nest documentation: http://nest.azurewebsites.net/ • Nest source code: https://github.com/Mpdreamz/NEST • Slideshare: http://www.slideshare.net/mastoj/getting-started-with-elasticsearch-and-net • Sense (greattool to queryelasticsearch in thebrowser): https://github.com/bleskes/sense

43. Questions?

44. Thank you! @TomasJansson