Using Business Intelligence to Improve all Operations in an Enterprise.

Business Intelligence

Subscribe to Business Intelligence: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Business Intelligence: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn


Business Intelligence Authors: William Schmarzo, AppDynamics Blog, Pat Romanski, Elizabeth White, Liz McMillan

Related Topics: SOA & WOA Magazine, Business Intelligence, Outsourcing on Ulitzer, Infrastructure 2.0 Journal

Article

Structural Quality Metrics in Outsourcing SLAs

How can you measure the quality of products being delivered by your outsourcer?

When I speak to customers and prospects trying to incorporate static code analysis into their software development processes, one of the most common questions that I get is "How do we incorporate the outputs of static analysis into SLAs?" Given the prevalence of outsourcing in Fortune 500 and Global 1000 companies, this question is not surprising. Companies have always struggled to measure the quality of products being delivered, beyond the typical defect densities measured after the fact.

To help organizations answer this and similar questions, I thought I would compile some frequently asked questions around introducing Structural Quality Metrics into SLAs.  Before I get into the details, I want to caution readers against using these metrics simply for monitoring, or as a tool to penalize vendors.  This approach invariably becomes counterproductive, and instead I recommend looking at these metrics as an opportunity to make the vendor-client relationship more transparent and fact-based-a win-win on both sides.

In addition to the FAQs below, for more on this topic you don't want to miss our next webinar on May 16th.  We are pleased to have Stephanie Moore, Vice President and Principal Analyst with Forrester Research discussing how to "Ensure Application Quality with Vendor Management Vigilance."  You can register here.

What kind of structural quality metrics can be included in SLAs?

  • Quality Indices: Code analysis solutions parse the source code and identify code patterns (rules) which could lead to
    potential defects. By categorizing these improper code patterns into application health factors such as Security, Performance, Robustness, Changeability and Transferability, you can aggregate and assign a specific value to each category, like the Quality Index in theCAST Application Intelligence Platform (AIP). You should set a baseline for each of these health factors and monitor the overall health of the over time.
  • Specific Rules: Quality indices provide a macro picture of the structural quality of the application, however there are often
    specific code patterns (rules) that you want to avoid.  For example, if the application is already suffering from performance issues, you want to make sure to avoid any rule that would further degrade the performance. These specific rules should be incorporated into SLAs as "Critical Rules" with Zero Tolerances.
  • Productivity: Amount charged per KLOC (kilo lines of code) or per Function Point. Static analysis solutions should provide the size of the code base that is added in a given release.  Along with KLOC, CAST AIP provides data on the number of Function Points that have been modified, added and deleted in a release.  This is a very good metric, specially in a multi-vendor scenario where you can see how different vendors are charging you and can set targets and monitor productivity for each vendor.

How do you set targets for Structural Quality Metrics?

The ideal way to set targets is to analyze your applications for a minimum of two to three releases and use the average scores as a baseline.

An alternative method is to use industry benchmark data.  CAST maintains data from hundreds of companies across different technologies and industries in a benchmarking repository called Appmarq, and it can be used to set targets based on industry averages or best-in-class performers.

When do you introduce Structural Quality Metrics into an SLA?

Of course, the best time to introduce Structural Quality Metrics into SLAs is at the beginning of the contract, when it is the easiest to set expectations on quality objectives based on the static analysis solution outputs.  However, if you are in the middle of a long-term contract with a vendor, you can try to make changes to the existing SLAs. A situation like this will require collaboration with the vendor to define common goals on why, how and when to use a static code analysis solution and what kind of metrics make the most sense in the context of those goals.

To hear an analyst perspective on achieving maturity in your outsourcing relationships, don't forget to register for our webinar on May 16th with Forrester Analyst Stephanie Moore.

More Stories By Jay Sappidi

Lev Lesokhin is responsible for CAST's market development, strategy, thought leadership and product marketing worldwide. He has a passion for making customers successful, building the ecosystem, and advancing the state of the art in business technology. Lev comes to CAST from SAP, where he was Director, Global SME Marketing. Prior to SAP, Lev was at the Corporate Executive Board as one of the leaders of the Applications Executive Council, where he worked with the heads of applications organizations at Fortune 1000 companies to identify best management practices.