I400/590 No Way Big Data & Recommender Systems

1. I400/590 No Way Big Data & Recommender Systems Geoffrey Fox March 3 2014

2. Posting Assignment • Please post on either • a) "No way Big Data" Discuss areas where Big Data is really unimportant • OR • b) Novel applications of Recommender engines not discussed in Section 7 lectures. One example is auto completion of surveys discussed for census in section 5 (use cases 3 and 4). This can be same or different from Homework topic

3. No way Big Data I • Small Businesses e.g. Bakery, Florist, Coffee shop, Hair Salon, Local news,  log cutting or physical labor jobs  • Even these can benefit from Big Data • Depends on the scale and context of the operation using it for big data to be actually practical.  • Don’t have the resources or skills to improve from implementing a big data system. •  it would be hard to for small businesses to use to help them better their business since it would probably be hard to compile data to better their business but in my mind I still do not think the data would be unimportant. • Smaller businesses as they might not have the resources to take advantage of big data. However, that does not make it unimportant.  • niche that these types of entities occupy are really looking for a more personalized approach • Small businesses do not have the capability to support the 3 V's needed for Big Data.   • Their sample set could be very low, so it would be hard to come up with a consensus with the data. • specific time we may not have a big quantity of the data volume , like starting a small business in early time. But the big data  importance would be  improve by the business developing. • big data would not be as effective due to lack of actual data collected. In smaller organizations that do not keep track of day to day smaller transactions, it would be difficult to leverage this information.  • Big data is really unimportant for small businesses that do not wish to expand. Like the local restaurant in the next town over. Its been there for 30 years and doesn't need to change its way of business. 

4. No way Big Data II • Agriculture and sustainability.  • I do feel, however, that eventually there will be a time where every aspect of Big Data will be used and will benefit the company working with it.  • big data is so freely available now a days that almost all types of industries are using it in some way or the other.  • Information Privacy holds back big data • There are areas that don't necessarily have tangible numeric data but people are figuring out ways to analyze it. • I  do not believe big data to have a field or area where it could be classified as "unimportant".  • However, like most things, it is not the best solution to everything. might be situations that the Big Data approach isn't so effective. • It can be applied in some way to every area. • because the resources aren't there currently for someone to use big data doesn't mean that big data is simply unimportant for those people.  • Big data can sometimes be too much when your end goal is more simple than the data requires. • anything that involves some type of human judgment or opinion. • decision making in some international business.  • Since big data requires continuous, variable and  fair amount of data any filed that doesn’t have these “conditions” will find no use of big data. 

5. No way Big Data III • Security is an area that big data could be unimportant. • Big Data is not important to the creative arts. How would knowing more about Picasso's style of painting or which time signature Mozart used most often help with someone's own personal expression of what they feel?  • You could keep track of a lot of information about every piece of art and how the artist went about creating it, but people in that area, for the most part need to figure things out for themselves to find their own unique personality.  • anything that requires human intuition and judgment- for instance, law, philosophy, or the arts. It can only be used in areas that already employ mathematics or computation. •  anything with entertainment or art (which doesn't rely on any computation) cannot really benefit from big data.  • it is very difficult to think of an example or industry in which big data is not relevant. For the most part, the data can be used in some manner which will have positive impacts on the industry. However, I do think the fine arts industry does not really need big data. I • Automobile fuel sales.  • Selling homemade apple pies, making homemade iPhone cases and painting (art).  • It may not solve a problem the certain field is having, but it can provide information leading to problem solutions for most fields. • Gardening, Law, language, and religion. • For example, law enforcement might only use big data from traffic cameras to monitor license plates for stolen vehicles. A completely different industry such as commercial real estate may be able to use this big data in a completely different way to compute demographics based on the drivers passing by the cameras at a given intersection.  • Another example would be the judicial system. At the end of a day, the judge and jury is who decides the fate of many criminals, and that isn't going to change anytime soon.   • Even the field that doesn't really look like it really needs big data, i believe big data can be useful and beneficial to everything we are trying to do. 

6. No way Big Data IV • Big Data has been shown to be very important in many fields making it difficult to think of what fields where big data could not be important. However I can think of areas where big data may not be as important such as any situation/area where there is very small amounts of data to be gathered and analyzed. • The data on  household energy use, the times of local buses, the basic household activities to be precise is a field where big data cannot find any major application. Thus by far we can say that all fields that deal with small activities big data is really not important. • The application that mainly work with only textual data like Wikipedia and the many cooking websites that exist on the web do not have to implement Big Data for their success. 

7. No way Big Data V • An area that I would say Big Data will have little or no influence is in philosophy.  The methods inherent in philosophy resist being explained through algorithms.  For example, what about the "hard problem of consciousness".   No amount of data could bridge the gap to explain intentionality or explain the experience of certain qualia.   • Even if I was going down the route of experimental philosophy, the compare the different intuitions and reasoning of different people, I would still need a well defined independent and dependent variable, along with a well defined operational definition, so that I could come to correct conclusion.  • I do not want a large amount of unstructured data coming at a tremendous rate, I would want the standard statistical method. • I think that there is an easy metric for this is when the value of the information derived from Big Data is less than what it cost. The problem is that ROI calculations are messy and subjective.

8. Instructor’s Comments on NO Big Data I • Note DIKW pipeline that processes data systematically tending to reduce volume at each stage • Final stage can be small volume “little data” but early stages “big data” • In Justice, the early stages such as search on web cam are clearly big data but final stages are people analyzing “facts” established by prosecutor and defender • I mentioned the contrast in computer science field of artificial intelligence between “expert systems” (little data analogous to list of facts in criminal case) and “neural nets” (big data) • People use big data techniques with a massively parallel brain to analyze situations e.g. image processing to decide that there is a lion waiting to pounce • However final “logical analysis” is little data method

9. Instructor’s Comments on NO Big Data II • Agriculture does not use big data in final weeding of one’s garden but • Designing Genetically modified seeds or in large scale satellite photos to identify agricultural trends are big data • Cutting down a tree (logging) is little data • Analysis of logging in Amazon by satellite for climate change big data • Language translation and voice to text are big data as there is a huge corpus of translated books with same text in multiple languages • Allows pattern matching approach to translation (big data) rather than applying carefully researched grammar rules (little data) • Creative arts is not normally big data but music genome used by Pandora could impact music composition and has big data aspects

10. Use Recommender Engines I • LinkedIn people to people recommender engines • Recommender engine that was not talked about is being used by Steam, a multiplayer online gaming community for PC’s. This application recommends computer games through their online store.  • news story recommendation systems on actual news sites.  • navigation programs (specifically, mobile apps like Waze) which utilize both user generated, as well as crowd sourced, data to produce several recommendations for possible routes given selected user-preferences.  • providing many different services to registered voters including providing information on candidates for Americans who do not possess adequate time for research.

11. Use Recommender Engines II • The launch of Google Glass made a sterling breakthrough in the augmented reality paradigm that rendered a composite view of the world to the common man. If we could implement recommender systems on an augmented reality device. • The recommendation system can be used in an intelligent personal assistant and knowledge navigator systems like SIRI for Apple Device • Another example of recommendation systems is IMDb recommendation system. They take all of the movies and TV shows that a user has either rated or added to his Watchlist. Then, IMDb compares this data to ratings made by other users. • A novel approach in the implementation of these recommender systems is in google maps where the application recommends the preferred route to take based on weather conditions, traffic and other such criteria. • Readers enter their favorite book and the system suggests books which the reader may like and read next. It will take into account the genre of the book entered by him or her and suggest books of same genre or same author. A website providing such recommendations is whatshouldireadnext.com.

12. Use Recommender Engines III • One such novel approach not discussed in section 7 is prediction of market stocks.  • Walmart, Kroger where the system suggests the user for the products they have not purchased based on the past history. The system will also prompt the user on any discounts, offers on • marriage bureau websites, where people when they search for a partner, get to choose from many people with similar characteristics or a range of people with their required priorities.the items they regularly purchase. • Realm of recommendation in Twitter to categorize the existing works as well as to identify areas that need to be further studied. • Software agents that recommend options to users. They are becoming very popular in e-commerce applications to recommend the online purchase of some products. • Cyber security recommender system can detect, identify and mitigate cases of fraudulent activities and cyber threats. •  I could spot that the spell checker system on nearly all mobile devices and also the word suggestion systems are indeed recommendation systems

13. Use Recommender Engines IV • A recommender system is an information filter that presents only information that is relevant to the user. It filters the information based on:1. User's past behaviour (Personalised recommendation)2. Currently popular information among his cohorts (Social recommendation)3. Current search (Item recommendation) • At this time of the year, am positive that many students are actively applying for internships and full time and are maximum benefited by the recommendation systems in the job portal. An agent is created for every individual registered with the job portal and the agent keeps sending information about new jobs posted that matches the job seeker's preference. • One application of a Recommender System is Coursera.org. It provides world-class education by giving access to multiple open online courses. The company uses a Recommender System. A user who is enrolled into one or more courses is recommended to take different courses • One of the example of such a recommender system is one that offers optimum routes to drivers in city. Based on the location of the users, his interests would suggest him suitable information such as restaurants, gas stations, ATM, shopping malls, concerts in nearby locations and so on. 

14. Use Recommender Engines V • Google Now is an example of what looks like multi criteria and mobile recommender system. On android devices (at least on Nexus 5), I get recommendations from Google on restaurants in vicinity, coupons and so on. It provides options to filter out categories based on user's interests.  • online dating services and their "matching" features.  I'm not familiar with how their matching systems work, but recommender engine algorithms appear to be one method to do this kind of matching of people to people. • Google appears to have similar algorithms to recommend people for one's circles in Google+.  Grocery store loyalty programs that track customers' purchases and then mail personalized coupons also use recommender engine algorithms.

15. Use Recommender Engines VI • I work with institutional repositories http://en.wikipedia.org/wiki/Institutional_repository and I think it would be helpful to have a recommender system that offers help or instruction about the digital items that are found in these systems. • The recommender system would be located on the detailed item record and would be a recommended list of help or instruction documents related to aspects of use of the digital item in the detailed record. • This recommender system is different than the recommenders systems in section 7 because instead of recommending items of the same type [bibliographic records], the system recommends items of a different type [help or instruction documents].

16. Use Recommender Engines VII • I recently had to choose my seat assignments for round-trip airfare. I was given a seating chart of the plane to select my seat that gave me some information about my choice such as: window, aisle, emergency row, proximity to front or rear, etc. The chart also told me if the adjoining seats were empty or occupied. • What I would have found valuable is a similarity score on how my neighbors related to my preferences. If travelers had filled out a profile asking questions like: • Do you like to talk to strangers seated next to you? Always, Frequently,Sometimes,Occasionally, Never • Do you lean your seat back? Always, Frequently,Sometimes,Occasionally, Never • Do you close the window shade? Always, Frequently,Sometimes,Occasionally, Never • Based on the answers to such questions, a recommender system would be able to highlight seats that best matched me to like neighbors. Similar systems could place tables of diners within a restaurant or any seating event such as a play or concert.

17. Use Recommender Engines VIII • Google now is a perfect example of multi criteria recommender systems. It is amazing to see how multiple criteria decides what the user wants. I use Google now on my mobile and it gives me directions, suggestions on places to eat, travel and so on by tracking the day-to-day activity. To me, it is a mix of multi criteria and mobile recommender systems which looks like a future trend where multiple criteria will be required to provide user the best lifestyle experience. • A novel application of a recommender engine would be a system that would formulate a K-12 student’s ‘academic genome’ and use it to recommend majors that the student may want to consider at college, and the colleges where they are likely to find greater academic success. • An analysis of what similar students majored in; where they went to college; and how well they fared, would then be used to recommend majors that the student might want to consider and schools that they could explore applying to • A novel application of a recommender system would be to match people with political candidates.  Doing so is not a simple as movies or music because people's political point of view changes over time.  Also the politician's view can change as well.  A candidate who was recommended to a user today may not be an appropriate recommendation a week from now due to personal, political, or policy flip-flopping.  A system like this would need constant streaming input and would often be re-evaluating existing recommendations.