Category Archives: Uncategorized

Anomaly Detection – What, why and now!

#include <std_disclaimer.h>

I’ve been doing a little research lately to learn about anomaly detection and wanted to share. I can’t think of a better way to start that with a visceral example.

If you are old enough to be a working professional you already inherently know what anomaly detection is. We all learned it watching Sesame Street as children!

Do you remember the “1 of these things is not like the other” song? It brings back warm memories.

What is anomaly detection?

An anomaly is something that doesn’t belong. It’s an exception, an outlier, an aberration. It’s just peculiar. Not that there’s anything wrong with that.

In data mining, anomaly detection (or outlier detection) is the identification of items, events or observations that do not conform to an expected pattern or other items in a dataset. This is the formal definition from Wikipedia. In plain English, anomaly detection solutions use software algorithms to understand the streams of operational metrics and their inter-relationships to automatically identify events that shouldn’t be happening and the likely causes.

And how can we put this into perspective with things we already know? We can better relate this new machine learning and analytics to what we are familiar with by looking at the business intelligence (data analysis and reporting) maturity cycle:

Data     >>     Information     >>     Knowledge

In the early days of computing we used them primary to collect data, to record each item sold or produced, or inventoried. Computer systems, at the time, were very transaction oriented. And at the end of each month, management could see a report summarizing the transactions: total sales, total units.

In the 1980s and 1990s, enterprise use of data became much more informational—examining sales by region by month or week to manage productivity. Specialized business analysts with analytical reporting tools (OLAP) performing interactive, multi-dimensional analysis of metrics, pivoting through the data to spot important trends to the business used to be employed. Now we have those capabilities in Excel and every accounting department is creating pivot tables. This is traditional, mainstream business intelligence.

Once reserved for giant Telco’s and financial services, now knowledge bordering on foresight can be gleaned by leveraging powerful data mining algorithms on today’s ultrafast hardware applied to the thousands of metrics and millions of data points most businesses collect about user experience and business results.

Data mining and analytics are augmented cognition.

Why is anomaly detection important to business?

We have entered the Age of the Customer. Spurred by the iPhone’s consumerization of mobile touch technology, bringing always connected mobile computing, access to cloud services, and social networks to everyone including our children and our parents, businesses can no longer survive by earning customers by being good closers. Content, relationship, trust – substance is required to succeed with today’s customers whatever generation we want as customers.

Customers no longer want to buy products they want experiences. And creating digital experiences that are memorable and worth sharing is what’s driving the way tomorrow’s successful businesses will engage users.

For IT the world is different, too, then. We are no longer creating systems of record but systems of engagement, which directly impact business results. Execution of the customer experience through application delivery has a bigger impact on business results than ever.

Here’s a quick video about how Snap Interactive uses Anomaly Detection to power the decision making behind real-time campaign management.

Leveraging the data the business generates from business and technical operations can yield insights that improve business operations and results delivered to stakeholders. 

Why is anomaly detection important to IT Operations?

With application delivery having such a big impact on customer engagement these powerful new analytics are critical to managing the risk of IT operations.There are just too many metrics from too many tiers flowing out of today’s complex application delivery environments powered by Cloud, SaaS, virtualization, and third party content and components.

Business are functioning in real-time, so the IT that supports the business much be much more real-time. Without our software systems, often the business of our businesses comes to a halt. Add to that the new Continuous Deployment scenarios and things are just happening a lot faster than they used to.

Today’s application and IT operations monitoring are generating terabytes of big data. With more frequent collection intervals a lot more data is being generated. Many sensors today are generating data every few seconds. The complex infrastructures we are monitoring have many more data elements and are spread across more servers, app servers, networks, and third party components. And the importance of business systems to the operation of the business means that all of the data we are collecting needs to be analyzed in real-time, across all of the many dimensions.

The old methods are not enough.

Dashboards are insufficient to manage today’s complex, real-time businesses. How many metrics can a person really watch on a dashboard and understand? Maybe a dozen or so – not nearly enough. Threshold setting is also no longer effective. Manually setting static values that represent Red – Yellow – Green doesn’t work in dynamic environments and doesn’t scale. Besides, thresholds assume you already know what you are looking for.

Data mining and machine learning will augment our cognitive abilities to deliver user experiences with greater reliability and consistency.

Anomaly detection in action

The analytics behind anomaly detection use powerful statistical techniques such as: K-nearest neighbor, cluster analysis and neural networks to understand the data and train the algorithms about what’s normal.

Here’s a quick example of a multivariate Gaussian outlier analysis courtesy of holehouse.org.

The graph below shows how CPU and Memory utilization are fitted with an elliptical probability model that identifies the green X as an outlier using a formula something like this:

multivariate_formula

 

multivariate_gaussian

What benefits can I expect from using these new anomaly detection tools?

Able to process a nearly unlimited amount of data in real-time generating expert analysis, anomaly detection systems are automatic. Anomaly detection turns unknown business and technical events into known events giving you a better understanding of what’s going on with your business. These systems perform complex event correlation and are much better at isolating root-cause as opposed to just symptoms.

Analytics can see past broad averages and analyze trends by their dimensions quickly spotting issues on a single cluster or only affecting a particular region that might normally go unnoticed. The alerts produced are both more proactive and more reliable. These systems can find system bugs by quickly identifying troubling behavior during a 10% production deploy or identifying sudden abandonment points from visitor analytics or RUM.

Finally, this is all about risk mitigation.

Anomaly detection is a safety system for your business just like Traction Control is for your car.

Not that long ago traction control was only found on luxury cars but now you wouldn’t consider purchasing an automobile without this important safety capability that greatly mitigates the risk of accident in slippery / changing conditions.

Leading vendors and market dynamics

There are several leading vendors in the analytics spaced dubbed ITOA for IT Operations Analytics by Gartner. Prelert, Netuitive and Sumo Logic come to mind and these successful vendors are independently selling these analytics as add-ons to supplement the results from current tools. In fact, Prelert has a nice partnership with both Splunk and CA APM.

By now you can probably tell that I think that anomaly detection and event correlation is a capability of sensor systems and not something I believe can sustain a company or many companies in the long term.

Given the way technology product cycles are compressing, I wonder how long the market leaders will be independent?

The takeaway!

These powerful, and now accessible data mining algorithms are now available in commercially available products and are an important safety control system for your business.

Analytics turns unknown business and technical events into known events giving you a better understanding of what’s going on with your business.

You shouldn’t operate without them.

10 rules for how businesses need to ready themselves for the Matrix

#include <std_disclaimer.h>

We are in the matrix!

The digital world is now becoming more important than the real world.
We have become internet obsessed and the ubiquity that connected smartphones gives to mobile and social apps has made them ever-present in our lives.

Speaking of ubiquity, Pew Internet reports that 79% of adults between 18-29 now have a smartphone and 67% between 30-49. We are spending more time online at the expense of other activities.

Our time, both personal and professional, is shared between Facebook, Twitter, Linked-in, youtube moreso than interacting with real people. And children who were born in 2000 and later will never know a world without instant digital gratification.

Apple, Microsoft and Google were ranked #1, #3 and #4 in terms of largest market capitalizations at the end of 2013. Snapchat was just acquired by Facebook for $19B. That’s the most ever for a venture backed company. Shocked? Don’t be, they are the engines behind the digital experiences of the future.

Here’s that scene from the Matrix where Neo chooses the red pill for reality and the truth…

But there is no “red pill” for businesses. No way back to the old realities. To succeed and flourish you must change.

How do businesses need to ready themselves for the Matrix?

  1. The business needs to take ownership of the customer experience. The old bureaucracy is dead. Business can no longer afford to have bad outcomes and finger pointing. “It was ITs fault” or “We didn’t have clear requirements.” The business must be driving the ship and project managers and developers must be part of supporting the business goals.
  2. Take an integrated look at the people who are your users. Understand them and their needs. What do you need to do to keep them happy and loyal.
  3. Change the focus to outside-in. Now that you understand them better, think like the people who are your users. Construct and manage the user experience in a unified way across all the customer touch points. This means spending more time on usability.
  4. Learn to be a story teller. Traditional marketing content and tactics are becoming less and less effective. Tell a story that answers the question of “why.” Why will this help me or my company or my family or…well you get the idea :)
  5. Help your users tell your story. Send a follow up email asking them to return the product if they are not 100% happy and ask for a brief product review if they are.
  6. Be Agile. And by that I mean two things. Build the bare minimum at first. Don’t let committee scope creep thwart bringing new products to market. Try new things and be willing to fail…quickly. If the results don’t turn out as expected move on to the next thing to try.
  7. Consider gamification. Make using your product rewarding for the people who do it. Give them rankings or free credit or access to something exclusive.
  8. Use everything you know to help. I’m really avoiding big data here because I still think a terabyte is big since I built my first data warehouse with 2G fujitsu disks. But you know a lot about your users. What they purchase and what they don’t. What parts of your website they use and which parts they avoid. Probably a number of different demographics as well. Leverage all of that to help deliver the best experience possible.
  9. Start thinking about what you need to do in the future to ready yourself for when everyone walks around like a Cylon (am I allowed to make a Battlestar Galactica reference in the same post as the Matrix?) with their Google glass on and has digital assistants helping to find the products and services they want before they even think about it.
  10. Lastly, move faster. You must learn to run and operate your business in the new real-time reality of business.

What an exciting time this is to transform your business!

Hope you are starting the journey.

Ken

PX > CX > UX = #usable + #feels_fast + #was_emotive <-- the battle for business supremacy

Over the last few years I have become inspired or perhaps possessed with a certain awe about how touch interfaces, Mobile, Cloud and Social have converged to change the focus of most successful organizations from delivering usable products to delivering meaningful and pleasurable experiences worth sharing. Digital experience influences more and more of our business landscape from how customers find us, to how they learn about and perceive our reputation, to their on-boarding experience. It’s the experience to date (hey I just made up a new term ETD), the sum of the whole experience delivered to the PEOPLE who are our users, that drives this.

Delivering experiences that people feel good about, find memorable and want to share is the next battle for business supremacy.

I’ve suggested previously, because our users are people, and understanding the customer journey and how to deliver amazing experiences starts with people, this should not be the practice of customer experience (CX) but rather people experience (PX). Further, if we are focussing for today (and we are) on digital experiences then we are really talking about user experience (UX).

I would suggest to you that this equation holds true:

PX > CX > UX = #usable + #feels_fast + #was_emotive

Of course, this is rooted in the fact that software systems are now systems of engagement and not just systems of record. We count on our software systems to help improve our reputation with our customers and our software systems to help our employees improve our reputation with customers. Every software system built ultimately has an impact on People Experience. And I want to emphasize how important it is that even our internal systems provide pleasurable experiences to employees. Because happy employees make for happy customers.

Aberdeen Research interpretation of Andrew's CX hierarchy

Aberdeen Research interpretation of Andrew’s CX hierarchy

This concept is borrowed from a slideshare (slide 15) by Steven Anderson in 2006 and the clarified graphic is courtesy of Aberdeen Research.

They are both a refinement of some previous research from Carnegie Mellon on human computer interfaces in the early 90s.

This is what we all should be striving for in the software systems that drive our interactions with customers and prospects. And also the software systems that support real-world interactions that support our employees or inventory or return process. What does this refined CX pyramid look like to you? Does it remind you of Maslow’s Hierarchy of needs. Take a look.

maslow

Just like with Maslow’s Hierarchy the basic needs and basic tasks at the bottom are much easier to achieve than the needs at the top like self-actualization. And yet, that is what is required of everyone involved in designing customer experiences now.

How can we build applications that create pleasurable experiences? By understanding the PEOPLE who will be using them. That’s probably a lot easier than self-actualizing.

Understand the people who are your users and do more than help them get it done – strive for delight. Think about those smaller parts of the interaction that don’t require building the starship enterprise. The Kano model is a good strategy here. Where could you introduce parts of the interaction that are different and appealing?

Let’s look at one ingenious example in the Travel aggregation space – Hipmunk.com. Search for any flight…go ahead. Notice that cute little button in the sort bar that says sort by “agony.” How can that not make you smile if you’ve every travelled through airports?

What appealing little capabilities are you adding to your UX to help delight people?

Ken