I’ve always thought many Website monitoring services can make it difficult to understand exactly what you are getting and exactly what you are paying. Having spent eons building and delivering monitoring and caring for customers I thought I would take some time to try and clear things up.
When I say website monitoring services I’m specifically talking about online services (SaaS) that are subscribed to on a monthly or yearly basis to measure website performance and availability and alert when something is wrong. A check or test may run every 1-minute or 5-minutes or 15-minutes. The purpose of that test is to interact with your site, the way a user would, and verify your web application is working correctly. Since these are not “real” users this type of monitoring is typically referred to as synthetic monitoring.
There are 10 factors that should be considered when choosing the right Website monitoring for YOU:
- your needs
- reliability of reported errors
- basic testing or real-browser testing
- monitoring locations
- effort required to create and maintain monitoring
- status and diagnostics
- ability to create and share required reporting to constituents
- support you may need to be successful
- other services – load testing, RUM, APM
Your needs are of course of primary importance. Do you have just a single website or multiple? Are you mostly interested in site availability or is performance and user experience important? Do you need to test a single Web page or do you need a Web transaction monitor to verify your application is providing a good experience to users? Will monitoring performance from a single web browser like IE or Firefox be sufficient or do you need cross-browser monitoring?
The reliability of errors reported by the service is probably 2nd most important. Nobody wants to be woken in the middle of the night to alarms saying the website is down when it’s just some spurious Ad banner that didn’t display and wouldn’t have affected the end-user experience anyway.
Basic testing is an easy way to understand website availability and basic HTTP performance, and verifying and measuring the performance of meaningful Web application interactions requires a real-browser sensor. Basic testing just sends a simple protocol message asking for the web page. Real-browser monitoring fetches, renders, and measures each aspect of the performance from the perspective of the web browser. More sophisticated users are sometimes interested in monitoring from their top 2 or 3 different browsers like IE and Chrome. There is nothing wrong with basic monitors but they have limitations when collecting web performance metrics. Real-browsers collect w3c timings and and help you really understand web performance and user experience. This definitely ties back to your needs.
The geographic locations available for monitoring may also factor into your selection. Both available and performance metrics can be different based on where they are sampled from. Knowing which geographies your most important site visitors come from will help you select the best geographic locations to choose for monitoring. You can probably get a pretty good idea of this from your Google Analytics account.
Does the vendor provide tools to make it easy and convenient to define and maintain the scripts (monitor definitions) that perform your desired Web interaction? Do they provide a handy transaction recorder to make it easy to produce and maintain your scripts? Does it actually do what it’s supposed to do? Some more sophisticated users prefer the monitoring services that support Selenium scripts so they can leverage the expertise they already have.
You’ll want a high quality status and diagnostic display. Is the data your viewing in the status screen real-time or 10 minutes delayed? Is it easy to see the monitors experiencing errors? Can you drill into the errors to understand the duration, type of error and root symptom? I say root symptom because the root cause is often impossible to see from the outside looking in.
Can the service create and share the required reporting views you are trying to create to end users? Can you create customized dashboards? Do you need to create reports that filter for only certain pieces of content to keep track of 3rd party SLAs? Or will you be the only one looking and so this might be less important.
Pricing is of course a big factor. I’m going to leave off any comments now. I always try to break things down into a model that I’ll try to get down on paper with some top vendors compared to share tomorrow.
Can the vendor provide support to help you be successful? This is party related to needs and your skills and the complexity of the monitoring you are trying to perform. It may also be related to how well the service captures diagnostics and leads you through them when errors occur.
Lastly, do you need other services such as load testing, real-user-monitoring, or perhaps even full-fledged application performance monitoring (APM)? If you’re looking to get a number of things from a single vendor that may also help you limit the candidates. But don’t be afraid to get this here and that there is you really understand your needs.