Category Archives: Editorial

Exploring the methods of end-user experience monitoring for APM

#include <std_disclaimer.h>

Today’s application performance management (APM) marketplace is maturing and the best solutions bring a wide-ranging set of capabilities for measuring performance and understanding behavior for many aspects of the application delivery stack. One of the cornerstone’s of APM is end-user experience monitoring (EUM).

As defined by Gartner for the APM Magic Quadrant, EUM is:
“The capture of data about how end-to-end latency, execution correctness and quality appear to the real user of the application. Secondary focus on application availability may be accomplished by synthetic transactions simulating the end user.”

But what does that mean? What are those capabilities?

There are a number of methods to do end-user monitoring. Each has advantages, and one is not enough. It is important to look at end-user experience through a number of different sides of the prism to really try and understand how the metrics match up against user experience. As I was cataloging them for myself I thought it would be good food for thought to share my definitions.

Synthetic monitoring
Web performance monitoring started with synthetic monitoring in the 1990s. A synthetic monitor is not a real user of your application but an artificial robot user, thus synthetic. The robot periodically executes an interaction with your website, API or web application to verify availability and measure performance. It is one of the easiest monitoring to setup and provides almost immediate value by delivering visibility and hard data without having to install or configure anything within the application. An example of a synthetic monitor would be a web transaction monitor that ensures a online store is working by visiting the home page, searching for a product, viewing the product detail, adding it to the cart, and checking out. This is very similar to the pile of functional tests that should run every time

Although Gartner has relegated synthetic monitoring to an availability role, it still has a lot of value for performance monitoring that passive methods do not address. No other method can help you measure service delivery when real users are not on the system. Thus it is ideal for measuring SLAs. And it is the only way to see individual page resources (a la the waterfall report) as this is still not quite yet a real user monitoring (RUM) capability. Synthetics eliminate a lot of the independent variables that can make it difficult to compare real user monitoring data. Finally, the synthetic connection to the DevOps tool chain of tests run at build or in QA provides a continuous reference point from development environments, through test and production.

Web real-user monitoring (RUM)
When I first saw real-user monitoring back in 2008, I knew it was going to change the way we measure web performance. RUM works by extracting performance values using javascript. As actual users visit web pages performance metrics are beaconed back to the great reporting mothership. Originally, the only metric that could be captured by RUM was a basic page load number, but modern browsers now collect a slew of detailed performance metrics thanks to the w3c timings standards and soon will even provide access to page resource level detail.

RUMs great advantage vs. synthetic is that is can see what’s happening for all of your actual users on all of the web pages they visit. This means you can understand web performance metrics by page, geography, browser or mobile device type. While this provides a broader understanding of general performance, it also has many, many more independent variables making specific trending and comparison more challenging. RUM is also the front-end method by which transactions are “tagged” so they can be traced and correlated through the back-end for greater understanding of how software and infrastructure work together to deliver end-user experience and root-cause analysis.

RUMs greatest and perhaps least exploited value to business today is that it captures business activity information that represents the WHY we have a website to begin with. It is this business outcome data that should be our first canary in the coal mine for determining if something needs attention.

Mobile real-user monitoring
Mobile web applications can be monitored with traditional RUM; however, today’s native mobile apps require a different mechanism to measure the application experience. That is typically accomplished by adding an extra library into your mobile application that beacons mobile application performance data back for reporting. Like traditional RUM, this is also how transactions are “tagged” for mapping through delivery software and infrastructure.

With mobile web traffic now reaching 25% or total traffic and mobile being the #1 method for brands to engage consumers, mobile RUM will be of increasing importance to most organizations.

Network real-user monitoring
Hardware appliances that plug into a network switch’s span port to passively listen to network traffic provides network based RUM that very accurately represents end-to-end network performance of the application. This type of packet smashing leverages timestamps in the network packet headers to break performance down into client, server, and network components.

My own assessment is that network RUM is particularly good at monitoring HTTP API performance for services rather than the higher level end user experience of an application consumer.

Desktop agent based monitoring
A few tools focussed on the enterprise measure end-user performance and usage by installing an agent on the Windows desktop. These agents often use similar technology as network RUM to inspect client network traffic by IP address and Port. This method also provides visibility into usage of enterprise applications as well as general performance and availability.

How many sides of the prism is your organization looking at user experience through?

Hopefully, unless you are already a monitoring guru, you learned a little about the monitoring methods being offered by today’s crop of APM tools for understanding end-user experience. What is also interesting to explore is what capabilities do users get from the different tools leveraging these methods.

Perhaps good subject for a future post :)

Unified Monitoring – the new monitoring renaissance has a moniker

#include <std_disclaimer.h>

I’ve been seeing a lot of marketing leveraging the term Unified Monitoring lately. At times it’s made me smirk but mostly smile. Let me explain.

It’s made me smirk because once again what’s old is new.

Many of the infrastructure components of Unified Monitoring have been a part of Enterprise Systems Management tools for more than 20 years. IBM, BMC, and CA products have offered dashboards, event management, correlation, reporting and service level management for as long as I can remember and I have a fair amount of gray hair :)

What’s so compelling is to maximize customer experience in the real-time digital enterprise we are re-imagining traditional management systems. Existing network and systems management capabilities are being enhanced with easy-to-use web based access, big data powered analytics, and more focused APM capabilities including visitor behavioral data. Put all of that capability in a well-defined, self-service pricing model bringing it to hundreds of thousands of companies and not just blue chip enterprises and you can start to see the potential.

This is making me smile a big toothy grin!

I’ve suggested before that we are in the great monitoring renaissance and I think that the term Unified Monitoring is probably the arrowhead that all this is lining up behind.

For years we have heard the term business alignment to help IT do the right thing. In tomorrow’s successful digital enterprise there will be no clear lines between business and IT. There will just be teams of specialists all working on part of the customer experience, the business. And, those teams will include IT people and UX people and marketing people and customer support people.

Do you remember, back in middle-school, the way the science book used to have those cellophanes of the human body systems? The skeleton, muscular, circulatory, organs layers. I’ve always had this vision that we could do the same for our customer experience delivery stack. Business results come from user behaviors that are the result of user experience delivered by application performance supported by the technology delivery stack.

Layering the business like this allows the team to focus on business results. And it let’s the teams focus on building a user experience first and then the technology required to support that. Combine the above visualization of the layers with powerful anomaly detection and statistical algorithms and you now have a competent and logical artificial intellect helping you deliver, manage and optimize the customer experience. Add marketing analytics, financial and supply chain data and we might be able to imagine closed-loop, machine learning powered Business Resource Planning.

I’m excited about the future of Unified Monitoring and you should be too!

Am I being too utopian?

How to select the most important web performance metric as a KPI – #feelsfast

We all know intrinsically that website performance is important. It has a tremendous impact on all of the business KPIs that measure the success of our online endeavor. I think website performance gets so much attention for two reasons: (1) it’s the most obvious symptom for bad results and (2) and it is easy to measure.

In my larger philosophical views on Customer Experience (CX) I’ve suggested…

PX > CX > UX = #usable + #feelsfast + #emotive

Feelsfast here represents “a lack of perceived latency.”

15 years ago, when I first started thinking about web performance, we only had network oriented metrics to understand web page performance. Today, there are a larger set of collectible metrics to measure many aspects of the spectrum on User Experience (UX). And today’s web applications, because they are pretty fat clients, must take client-side performance into account as well.

We are constantly reminded of the importance of performance by vendors, the media and customers through their actions.

What is the most important metric to measure web performance as a KPI? It’s the one that best represents User Experience or a lack of perceived latency.

We have network metrics. These are old school metrics that focus on how long it takes your server and the network to delver web page resources to the browser’s network layer.

  • DNS lookup time – time to resolve DNS name
  • TCP connect time – time to TCP connect
  • SSL handshake time – time to perform SSL handshake
  • Time to first byte – time to receive the first packet of data
  • Time to receive the data – time to receive the data
  • Fullpage time – time to load the web page and all it’s resources

Most of today’s modern network browsers supplement this with a richer set of data based on the W3C standards for navigation timings:

  • navigationStart – time that the action was triggered
  • unloadEventStart – time of start of unload event
  • unloadEventEnd – time of completion of unload event
  • redirectStart – time http redirection begins
  • redirectEnd – time http redirection completes
  • fetchStart – time that request begins
  • domainLookupStart – time of start of DNS resolution
  • domainLookupEnd – time DNS resolution completes
  • connectStart – time when tcp connect request begins
  • connectEnd – time when tcp connect completes
  • secureConnectionStart – time just before secure handshake
  • requestStart – time that the browser requests the resource
  • responseStart – time that the browser receives first packet of data
  • responseEnd – time the browser receives the last byte of data
  • domLoading – time that the document object is created
  • domInteractive – time when the browser finishes parsing the document
  • domContentLoadedEventStart – time just before DomContentLoaded event
  • domContentLoadedEventEnd – time just after DOMContentLoaded event
  • domComplete – time when the load event of the document is completed
  • loadEventStart – time when the page load event is fired
  • loadEventEnd – time when the page load event completes

This is a nice visual of the W3C timings.

timing-overview

And we have visual timing metrics available from various tools:

  • IE11 brings us msFirstPaint as part of the browser timings
  • webpagetest.org gives us start render, filmstrip view, and the innovative speed index
  • AlertSite.com can provide visual capture and metrics for FirstPaint and Above the Fold using Firefox

How do you choose which web performance metric has the most value as a KPI when all of these have some value? The key is to identify, for any particular monitored application or web page, which metric best represents a users perception of latency, in other words, if it feels fast. This is likely one of the more modern metrics – loadEventEnd, FirstPaint, Speedindex, Above the fold.

Once selected, this #feelsfast metric should become a critical business KPI and tracked and managed as such.

Are you giving the web performance component of UX enough attention?

Ken

10 rules for how businesses need to ready themselves for the Matrix

#include <std_disclaimer.h>

We are in the matrix!

The digital world is now becoming more important than the real world.
We have become internet obsessed and the ubiquity that connected smartphones gives to mobile and social apps has made them ever-present in our lives.

Speaking of ubiquity, Pew Internet reports that 79% of adults between 18-29 now have a smartphone and 67% between 30-49. We are spending more time online at the expense of other activities.

Our time, both personal and professional, is shared between Facebook, Twitter, Linked-in, youtube moreso than interacting with real people. And children who were born in 2000 and later will never know a world without instant digital gratification.

Apple, Microsoft and Google were ranked #1, #3 and #4 in terms of largest market capitalizations at the end of 2013. Snapchat was just acquired by Facebook for $19B. That’s the most ever for a venture backed company. Shocked? Don’t be, they are the engines behind the digital experiences of the future.

Here’s that scene from the Matrix where Neo chooses the red pill for reality and the truth…

But there is no “red pill” for businesses. No way back to the old realities. To succeed and flourish you must change.

How do businesses need to ready themselves for the Matrix?

  1. The business needs to take ownership of the customer experience. The old bureaucracy is dead. Business can no longer afford to have bad outcomes and finger pointing. “It was ITs fault” or “We didn’t have clear requirements.” The business must be driving the ship and project managers and developers must be part of supporting the business goals.
  2. Take an integrated look at the people who are your users. Understand them and their needs. What do you need to do to keep them happy and loyal.
  3. Change the focus to outside-in. Now that you understand them better, think like the people who are your users. Construct and manage the user experience in a unified way across all the customer touch points. This means spending more time on usability.
  4. Learn to be a story teller. Traditional marketing content and tactics are becoming less and less effective. Tell a story that answers the question of “why.” Why will this help me or my company or my family or…well you get the idea :)
  5. Help your users tell your story. Send a follow up email asking them to return the product if they are not 100% happy and ask for a brief product review if they are.
  6. Be Agile. And by that I mean two things. Build the bare minimum at first. Don’t let committee scope creep thwart bringing new products to market. Try new things and be willing to fail…quickly. If the results don’t turn out as expected move on to the next thing to try.
  7. Consider gamification. Make using your product rewarding for the people who do it. Give them rankings or free credit or access to something exclusive.
  8. Use everything you know to help. I’m really avoiding big data here because I still think a terabyte is big since I built my first data warehouse with 2G fujitsu disks. But you know a lot about your users. What they purchase and what they don’t. What parts of your website they use and which parts they avoid. Probably a number of different demographics as well. Leverage all of that to help deliver the best experience possible.
  9. Start thinking about what you need to do in the future to ready yourself for when everyone walks around like a Cylon (am I allowed to make a Battlestar Galactica reference in the same post as the Matrix?) with their Google glass on and has digital assistants helping to find the products and services they want before they even think about it.
  10. Lastly, move faster. You must learn to run and operate your business in the new real-time reality of business.

What an exciting time this is to transform your business!

Hope you are starting the journey.

Ken

My Short Affair with Microsoft Surface Pro and the Retail User Experience

I couldn’t resist picking up one of those Surface Pro’s when they went on sale a few weeks back for $499. It is such a novel and interesting little device. A complete and powerful computer all neatly packed into a little tablet with a gorgeous 10.6 in. 1920 x 1080 hd touch screen. And of course I needed the accompanying Type 2 cover and keyboard.

I really wanted to love the Surface Pro but in the end I returned it to the Microsoft Store.

After two weeks I learned I couldn’t really find a way for it to work for me in a productive way. It still remained interesting, but it just felt like a device sitting next to me in the den while I used my recently rehabilitated 5 year old Macbook Pro.

Windows 8.1 was actually quite nice to use. After installing Classicshell.net I had a complete Windows 7 desktop environment as well as Microsoft’s device oriented Modern UI. The touch interface was nice in modern UI and I’ve caught myself smudging the Macbook’s glass swiping at it a time or two :). The Windows 8 snapping applications thing was pretty cool too. Not that you can’t do that in any windowing operating system just without the swipe and snap.

I did install Classicshell.net right after the Windows 8.1 updates which gave me the best of both worlds. A full Windows 7 desktop environment as well as Microsoft’s device oriented Modern UI. And I can completely see how for the mobile professional the Surface Pro could represent a solution that would let you have a complete professional computing environment both at the office, at the home office, and on the go.

But…it’s just not usable as a laptop and there’s the rub.

As much as I wanted to cherish this enticing device I could not be productive with it the way I compute most of the time. The kickstand was wonky when not at a desk. The type 2 cover keys had a really nice feel but a terrible mouse-touchpad thing. Even with the type 2 cover’s bit of rigidity the Surface was still too top-heavy to use effectively when not at a desk. It also got a little too warm as a tablet compared to…well…my iPad.

I was hoping the Surface Pro would cure my desire for a new 13 in. retina Macbook Pro. Instead, it was a short affair.

What Microsoft really needs to learn from this blog post has nothing to do with their hardware or software but rather their business systems and their understanding of how to interact with consumers. Because the $499. sale for last years Surface Pro was only over a weekend I ordered from the Microsoftstore.com online rather than risk not getting one at the retail store nearby in Boca Raton.

When I called Microsoft to let them know I wanted to make a return they said I “could” return it to the retail store but it sounded like that would be awkward. I just preferred to bring it back to return it to the store and close the case so to speak. When I went to the store the employees were very nice but couldn’t figure out how to use the case number I was given by the online Microsoftstore.com personnel. It took an incredibly long time for them to be able to process the return and 3 team members were involved to help get it to conclusion. The whole affair seemed painful to everyone and their systems. At one point someone mentioned that the online Microsoftstore.com and the retail store locations use different systems – really, really?

I will complement Microsoft’s employees who were very nice and made sure that I did everything required to have a successful return. In fact, it seemed like one or two went above and beyond to try and minimize the lack of cooperation from the systems. Honestly though, it was two 30+ minute calls to Microsoftstore.com support and 45 minutes at the Microsoft Store in Boca.

Microsoft, please take examine the retail user experience delivered by Apple. Seamless, swift, and never an unstated non-smile if you want to return something you decided against (as long as it’s in brand new condition).

Finally, and perhaps a contributor to my decision to return the Surface Pro, the recent browser usage statistics from New Relic show that IE is down to a paltry 14.8% share across more than 2 million application instances. If that’s not a signal that Microsoft’s role as THE predominant client is beyond repair I don’t know what is.

Maybe Microsoft should stick to what they have grown to be very good at – helping enterprises build and deliver corporate IT and applications and cede the consumer client space. The consumer client space will be dominated by those that focus on experiences and I’m just not feeling that competitiveness from this latest encounter.

Ken