www.nextslm.org: The Service Level Management Learning Community

Web Performance Metrics - Setting the Standard

by Richard Longland

Though the Internet bubble has burst, a class of enterprises remains stubbornly determined to define and measure the end-user experience to better serve its online audience. These leading companies are clearly committed to the Internet channel, run high-traffic Web sites and employ a mix of tools and methods attempting to measure their customers' online experience. They empathize with end users in a tangible way and constantly innovate to improve the performance and quality of online services they deliver. They have done their homework -- and recognize that there is a link between customer experience and Web site performance that has monetary value.

But how do these leading companies benchmark their Web site performance in order to improve it? Performance benchmarking is based on the real-time gauging of Internet Web site performance relative to industry peers, as identified by industry analysts with vertical expertise. Benchmarks are dynamic, not static, and they possess temporal, competitive and behavioral attributes, all of which contribute to a truly multidimensional measurement of performance.

This paper focuses on modern performance benchmarks that are relevant to measuring customer experience and offers readers insights to help them improve the quality of their online business channel.

Rationalizing Performance Metrics Data

Myriad opinions exist as to the best way to create valid performance data that attempts to define the end-user experience. What are the best ways to measure the end user's online experience, as well as his expectations? How can any performance data be made useful to a diverse group of enterprise decision makers with different needs?

Arguably, the utility of Internet performance data has improved as leading enterprises push performance vendors to provide more "actionable" data for decision support. The evolution of performance monitoring tools has affected the way organizations collect performance data -- and arguably how they apply that data for decision support. Collectively, vendors have tried to respond to the market's complex performance needs by creating products (though not necessarily services) that are easier to provision and use, operate in real-time and offer more flexible reporting.

While this trend may seem like good news to executive decision makers, there are still concerns about the overall integrity of Internet performance data. For performance data to be considered a worthy input into the decision making process, the user must consider it to be valid and applicable to the management issue du jour. Unfortunately, the methodology for collecting the data may be unclear, and it may force executives to make hasty decisions without the "right" data -- or worse, make hasty decisions with inaccurate data.1 This results in lingering questions about the validity and use of any Internet performance data.

This issue can be disconcerting for Internet performance analysts who recognize the lack of best practices and inconsistencies around Internet performance measurement. It also points out the risks associated with making capital investment decisions with seemingly questionable data. To better understand the dynamic nature of Internet performance data, it is worth exploring how it is collected and used at an enterprise level.

Leveling the Playing Field

Leading companies learned, albeit painfully, that organizing and collaborating around IT investment decisions makes good business sense. Indeed, recent research findings revealed in the landmark Net Impact Study illustrate that IT managers at larger enterprises are being evaluated much more closely on their ability to meet specific business objectives.2 In addition, the study found that managers were required to quantify financial returns in shorter time periods. Most important to understanding the stressful evolution of IT management is the finding that IT managers must "compete" on the same investment criteria as other business managers in reaching their objectives. While some planning managers find this competition constructive, others may find it tortuous due to the lack of meaningful, tactical data needed to support business cases.

Executives involved in IT strategy often share a challenging mandate: They're tasked with translating qualitative information into quantifiable intelligence to enable ROI planning. CIOs, in particular, are held accountable while coordinating the IT agenda with business unit goals. These are obviously daunting and multifaceted issues that have the potential to make or break a senior executive officer. It's no wonder that business intelligence and Web analytics firms are quickly gaining mind share at increasingly higher organizational levels -- within IT and the business sides of the organization.

Viewing Performance Data as a Value Chain

It is within this challenging context that executives must focus on Web-centric performance data collection. Since more executives are committing themselves to demonstrating Internet channel ROI, they are realizing the necessity of improving their customer's online experience. This, they belatedly understand, requires more sophisticated monitoring systems than what their organization originally deployed.

As both IT and business groups work more closely together to create more useful and accurate intelligence about their online customers, they are creating a value chain of information that forms the lifeblood of the enterprise. The stages of the "information value chain" should be viewed as evolutionary, given the Internet's dynamic nature. This dynamic condition is a result of multifaceted technological interdependencies (such as Web hosting options, peering arrangements, third-party content and service delivery deals), which impact decisions made at both the operational and strategic levels.3

Multidimensional Benchmarking Methods

The aforementioned information value chain framework is based on practices that emerged from a bygone era. And because every enterprise's framework is evolutionary and idiosyncratic, it will inevitably fail to provide robust performance data appropriate to both technical and business executives.

The current focus on end-user experience optimization represents a complex problem to these enterprises. How can an end user's experience be circumscribed in a tangible dimension? How can organizations correlate end-user performance data with a decision-making framework to understand, control and optimize the online environment for the intended audience? For data to be deemed worthy as a performance "benchmark," it must be assessed within geographical, temporal and competitive boundaries -- all within the expectations and behaviors exhibited by the online customer. It is a daunting task, rife with challenges, but some companies are already using advanced practices from previously unrelated disciplines to make remarkable and quantifiable improvements in their online customers' experience.4

By focusing on these challenges exclusively, leading businesses have designed new practices that weave competitive benchmarking, usability and end-user performance into a model that provides rich, multidimensional data that is sophisticated enough to be actionable for both IT and business users (see Figure 1, below).

And by collaborating on the previously disconnected issues, they are jointly creating benchmark data that has a longer lifetime and utility within the enterprise.

Many leading e-businesses design their sites to facilitate multiple types of buying transactions. Some sites will offer "dual paths" to online customers to obtain the same result -- an executed transaction -- but each may provide more choices to enrich the online users' "productivity path." The obvious value proposition for the e-business is that higher productivity levels from online users can lead to a higher conversion rate and can also be correlated to the resulting sales revenue.

Any of these "critical path" user transactions are easily emulated, using scripts that enable real-time monitoring, alerting and reporting. Even by themselves, these innovative performance metrics are helpful, since they combine best practices from performance monitoring and site usability principles. But to add even more dimensional depth, online business transactions that the enterprise deems critically important (such as buying a mutual fund) can be compared with competitors' Web sites.

There are other issues to illuminate before transitioning from a one-dimensional to a multidimensional benchmark environment. A far more insightful dimension of customer experience will appear if we address just a few of the challenging questions that have previously proven unsolvable:

Usability:

1. What are my critical business processes? Are they problematic for the online user in any way? How many steps does it take for my customers to execute a transaction? How long does this process take when compared with those of my direct competitors?

2. Are there changes I can embrace to make it easier for my customers to transact business? What design principles should I reconsider to improve Web site performance or efficiency?

Performance:

1. How available is my site for my target customers? Is it possible that my homepage is fully functional but my online customers cannot transact critical business tasks?

2. The Web site is fully functional, but I suspect a problem with my ISP. Can I diagnose and localize a performance problem in the proverbial Internet cloud?

3. What other factors might contribute to performance degradation of my customers' online experience?

Competitive Intelligence:

1. How is my competitor's site able to transact business four times more efficiently than my site?

2. How does my competitor's corporate site differ from mine in terms of design? Do any of these differences affect my ability to convert casual visitors into customers?

These are all difficult questions that require the infusion of expertise from previously disconnected fields, such as social science, computer science and even psychology. For example, when assessing online offerings, it is important to consider the tradeoff between functionality and performance. Borrowing an example from the brokerage industry, Figure 2 (below) shows a trader's quote page and illustrates the comparative differences in the functionality between the Web sites.

While the features on a given page can vary considerably according to the design, the resultant performance of the page and subsequent transactions vary as well. Figure 3 (below) shows the differences in the performance of the positions pages.

The corporate sites featured here are owned by companies with sophisticated Web design teams that continuously strive to improve their online environments. Other enterprises may not appreciate the implications of the performance/functionality balance. One of the lessons here is that some companies may design their corporate sites to be the fastest in terms of page download, yet offer limited content and functionality vis-à-vis the competition.5 To address these issues accurately and holistically, companies require expertise in the following areas:

Web Site Usability: Requires a unique blend of experience that includes user behavior psychology, heuristics and task flow analysis. At Gómez, we've conducted primary research that focuses on consumer attitudes toward performance and functionality and is developing standards based on both attributes. These new best practices yield utility ratings of Web sites and also include temporal and competitive components.6

Performance Monitoring and Analysis: Requires a mix of IP performance services that provide contextual data immediacy to both IT and business departments within the enterprise. Actionable advice may vary considerably and ranges from load balancing and page weight adjustment recommendations to suggestions for a more reliable ISP.

Competitive Intelligence: Requires deep comprehension of market drivers and competitive dynamics. You need insight into the strengths and weaknesses of a firm's online offering and delivers a snapshot of the competitive landscape. The analysis examines the effectiveness of business strategies and identifies and measures features that are critical to a Web site's success. Typical categories include:

1. Ease Of Use: The site's layout, content and functionality.

2. Customer Confidence: The site's reliability and accessibility.7

3. On-Site Resources: The range of products, services and information.

4. Relationship Services: The ability to build electronic relationships through personalization, among other features.

Enabling Multidimensional Benchmarking

Any company that considers the Internet as a critical business channel should consider the following actions:

1. Analyze the site's visual language, task efficiency and performance over a time period.

2. Identify the high-demand pages and critical transactions. Perform a step-by-step analysis of transaction flows.

3. Compare the site's utility and performance against top three competitors as well as the overall industry benchmark.

4. Record any performance issues, such as object errors, task constraints, failed transactions, etc.

5. Establish goals for site performance consistency:

Implement design- and performance-related changes based on initial discovery.

Create new service-level thresholds for both internal and external service-level providers.

Prioritize future enhancements to the site infrastructure based on performance/utility metrics.

Continuously monitor the site with an emphasis on performance metrics (such as critical transaction success rate).

By becoming familiar with these practices, leading enterprises can achieve a better understanding of the complexities of the relationship between Web site performance and customer experience.

Conclusion

While Web sites have become increasingly sophisticated, the performance metrics used to measure their utility and performance within the expectations of online customers have become elusive. This paper offers new methods to better characterize performance metrics that are capable of serving the diverse and sophisticated information needs of the modern enterprise. These practices are setting the new standard by which leading e-businesses are collaborating on IT investment decisions and focusing on customer experience. It is these "stubbornly determined" companies that are sharpening their competitive edge, delighting their online audience and setting the paradigm for Internet business in the new millennium.

Appendix - Stages of the Information Value Chain

Design. Most enterprises have existing legacy systems and do not have the luxury of building entirely new systems dedicated to monitoring and measuring customer experience. Many companies are forced to implement new tools and outsource Web site performance monitoring services, only to tackle the issue of routing third-party data back into their information systems.

Collection. While leading e-businesses have sophisticated monitoring instrumentation, most legacy systems were designed and deployed to meet the needs of the "techies" of the pre-Internet era. They lacked the ability to translate binary data into meaningful business impact terms and were not designed to enable performance data collection and distribution to both IT and business executives. This may partially explain the "Web performance disconnect" between IT and business units. Some leading enterprises have tasked both IT and marketing business units to jointly develop best practices that focus on optimizing the end-user experience -- with measurable results.

Distribution. A critical attribute of performance data is immediacy. Leading enterprises tend to invest more heavily in IT (as a percentage of annual revenue), and their corporate culture is more inclined to access and analyze performance data. Some may view this data immediacy and focus on operational metrics as a behavioral characteristic of leading e-business' performance data literacy, but it also implies that increasingly more executives (technical and business) are becoming conditioned to expect immediate access to valid, contextual data on demand. Therefore, having any data may be viewed as being better than having no data. Unfortunately, the sensitivity to accurate and repeatable data collection may not find its way into this stage of the value chain.

Consumption. For performance data to be utilized effectively, it must possess both prescriptive and predictive attributes. It must be predictive, since the data collection design should be influenced by both IT and business users. This means that when the data arrives at executive desktops, it empowers professionals to take actions based on a prescriptive remedy. It is because of this "design forethought" that data can be applied to benchmarking applications, while offering enough credibility to be used for setting IT priorities or making investment decisions. Yet design collaboration and forethought are luxuries that few enterprises can afford, leaving questions of data validity and utility to the majority of enterprises.

Action Facilitation. For many companies, performance monitoring is functional in nature and simply means providing alarms that help keep the infrastructure up to operational standards. But the needs of IT operations are quite different from those of other senior business managers. Modern information systems enable real-time, Web-centric, information flows to the executives who have the authority to influence constructive changes to any part of the Web environment to improve customer experience. And this surely represents a paradigm shift from legacy days.

Feedback. If the data life cycle is enabled through a well-designed enterprise information system, users can add value to the data at any point in the process. For example, if a CTO is trying to correlate online customer expectations with performance and is limited to page download time, what can be done to resolve the scarcity of actionable data? Does data exist elsewhere in the enterprise, or does the collection system lack "design forethought?" These questions demand flexibility and data immediacy that are not found in performance data information systems.

Notes:

1 The measurement methodology employed to gather accurate performance data is paramount. In the October 2001 issue of Business Communications Review, Peter Sevcik and John Bartlett of NetForecast clearly dissect the limitations and inaccuracies of popular measurement methodologies in the article Understanding Web Performance.

2 The Net Impact Study was a project conducted by Hal Varian of the University of California-Berkeley, Robert E. Litan of The Brookings Institution and Momentum Research Group, and sponsored by Cisco Systems. It was designed to measure the current and anticipated cost savings and revenue increases that organizations believe have been created by their investment in Internet business solutions. Results were published in January of 2002 and can be found at http://www.netimpactstudy.com/index.html.

3 See appendix for an organizational framework based on common business practices.

4 The Canadian Imperial Bank of Commerce (CIBC) uses the latest practices in Web performance management. Please refer to http://www.gomeznetworks.com/public/thought.asp?id=case

5 Customers do not always penalize a site for being slower than a competitive site. Refer to the GómezPro Brief, Performance Metrics in Context: http://www.gomeznetworks.com/public/thought.asp?id=perf.

6 The functionality of a particular Web site can be a highly subjective phenomenon. For this reason, Gómez relies on consumer research, analyst input and competitive assessments to define functionality attributes. Gómez continues to offer its landmark product, the "Customer Experience Analysis" which analyzes site utility and functionality within the context of online customers' expectations.

7 An optimally performing site is conducive to a positive customer experience. Gómez Performance Network measures availability, responsiveness, scalability; as well as object-level detail -- all in real time. Refer to: http://www.gomeznetworks.com/public/solutions.asp?id=gpn.

Richard Longland is Director of Product Marketing, Gómez Performance Network. He's responsible for the marketing of Gómez Performance Network and related performance measurement services, including the positioning and messaging of benchmarking, customer experience and service level reporting products to leading enterprises such as CIBC, Best Buy, Fidelity, DoubleClick and Wells Fargo. Prior to joining Gómez, Mr. Longland was a senior product manager at Genuity, where he developed SLAs for the Web hosting division. A former Air National Guardsman, Mr. Longland holds an MSM/IT degree from Lesley University and a BA from the University of Mass. in International Relations.