data analysis

Ever since I read Taleb's Black Swan and Fooled By Randomness I've had a heightened awareness of the limitations of our attempts to predict the future, and our tendency to post-rationalise events, create false narratives and illusions around our own influence and control. Prediction is difficult.

Sentiment analysis is a complex beast. Even for humans. Consider this statement: “The hotel room is on the ground floor right by the reception”. Is that neutral, or is it positive or negative? Well the answer is probably that it is different things to different people. If you want a high room with a view away from the noise or reception the review is negative. If have mobility issues and need a room with easy access it is positive.

I’ve been working with a client company and looking deeply into their customer calls to find patterns around why people call, how CSRs handle calls, and what issues customers are having with the company’s products and services.

Scott Golder recently wrote blog post at Cloudera entitled “Scaling Social Science with Hadoop” where he accounts for “how social scientists are using large scale computation.” He begins with a delightful quote from George Homans: The methods of social science are dear in time and money and getting dearer every day. He then turns to talk about the trajectory of social science:

Twitter shared a fascinating presentation with information on how they are handling analysis of it’s 100 Billion tweets. Much of the analysis is focused on numerous technical requirements but there are some really fascinating business requirements as well.

Watch this video by Bruce Nussbaum, BusinessWeek's innovation editor and veteran employee with the company. It's a fascinating illustration of the shifts in business we are seeing in real time. Media outlets in particular have been on the front lines of this shift. As I've said many times before, the Web and its latest social iteration has introduced ultra deep and pervasive niche content and experiences which directly compete with many business models.

Andrew McAfee’s blog is a great place to learn about how businesses can gain competitive advantage by their use of IT. But yesterday he took a left turn and discussed business situations where data crunching is not helpful to decisionmaking, and I loved it.

by: Gary Hayes

 by: Roger Dooley

Most market researchers earn their living by asking questions - what people did, why they did it, what they might do in the future, and so on. The methodology varies - focus groups, Web surveys, interviews, etc. - but in most cases the fundamentals are similar.

by: David Wigder