Powered by Gartner Methodology

Download FrontRunners Material

Click image to download

      

Published FrontRunners:
Call Center June 2018
Construction Estimating June 2018
Fundraising June 2018
Hotel June 2018
Manufacturing June 2018
Mental Health June 2018
HRIS May 2018
Business Intelligence May 2018
Field Service May 2018
Enterprise Resource Management May 2018
Accounting May 2018
Residential Property Management* March 2018
Project Management* March 2018
Payroll* March 2018
Inventory Management* March 2018
ATS* February 2018
Maintenance Management* February 2018
CRM* February 2018
LMS* February 2018
EMR* January 2018
Retail* January 2018
Help Desk* January 2018

*Pre-May 2018 Methodology

FrontRunners Refresh Schedule:
July 2018 August 2018
Help Desk CRM
Retail Management ATS
Construction PM Maintenance Management
EMR  

FrontRunners Methodology (Version 2: May 2018 - present)


The Basics

The FrontRunners methodology calculates a score based on published user reviews for products on two primary dimensions: Usability on the x-axis and User Recommended on the y-axis.

The Usability score is a weighted average of two user ratings:

  1. End-user ratings of one to five stars on the product’s functionality.
  2. End-user ratings of one to five stars on the product’s ease of use.

The User Recommended score is a weighted average of two user ratings:

  1. End-user ratings of one to five stars on how valuable users consider the product to be relative to its price.
  2. End-user ratings of one to five stars on how likely they are to recommend the product to others.

There are two FrontRunners graphics for each market, one “Small Vendor” and one “Enterprise Vendor” graphic. The “Small Vendor” graphic highlights qualifiers from smaller (by employee size) vendors, while the “Enterprise Vendor” graphic displays qualifiers for larger (by employee size) vendors.1

Products that meet a market (or product category) specific minimum score for each axis for their respective size group (Small Vendor or Enterprise Vendor) are included as FrontRunners in their respective size group. The minimum score varies so that no more than 15 and no fewer than 10 products qualify for any FrontRunners graphic.

The minimum score cutoff to be included in the FrontRunners graphic varies by category, depending on the range of scores in each category. No product with a score less than 3.0 in either dimension is included in any FrontRunners graphic. For products included, the Usability and User Recommended scores determine their positions on the FrontRunners graphic.

 

For each individual rating in both the Usability and User Recommended criteria, the methodology weighs recent reviews more heavily.

Eligibility

Markets are defined by a core set of functionality, and to be eligible for FrontRunners, products must offer that core set of functionality. Core functionality required is determined by our research analysts, who provide coverage and have familiarity with products in that market. Additionally, a product must have at least 20 unique user-submitted product reviews across the three Gartner Digital Markets web properties: softwareadvice.com, capterra.com and getapp.com, which must be published within 18 months of start of the analysis period.

Inclusion in a “Enterprise Vendor” or “Small Vendor” FrontRunners graphics is based on the vendor’s employee count. Vendors eligible for the “Enterprise Vendor” graphic must have more than the median employee count for all vendors in the market, or 100 employees—whichever is greater. Vendors whose employee counts do not meet either of those thresholds qualify for the “Small Vendor” graphic.

In cases where fewer than 10 products qualify to be on either a “Small Vendor” and “Enterprise Vendor” FrontRunners graphic in a given market, the market will have only one graphic combining both enterprise and small vendors, which are color coded to show distinction between size.

Data

Data sources include published user reviews, public data sources and data from technology vendors. The user-generated product reviews data incorporated into FrontRunners is collected from submissions to all three Gartner Digital Markets sites (see above section for details). As a quality check, we ensure the reviewer is valid, that the review meets quality standards and that it is not a duplicate.

The employee count data comes from public sources, collected by Gartner associates. As a quality check, we compare this data against data submitted by the providers. We use this data to place a product in either the or “Small Vendor” or “Enterprise Vendor” version of the graphic. Employee size has no bearing on a product’s inclusion in the graphics beyond determining which version the product qualifies for.

See FrontRunners frequently asked questions (FAQ) for more information on the methodology.

External Usage Guidelines

Providers must abide by the FrontRunners External Usage Guidelines when referencing FrontRunners content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all FrontRunners reference(s) and graphic use:

FrontRunners scores and graphics are derived from: a) individual end-user reviews based on their own experiences, b) vendor-supplied information and c) publicly available product information; they do not represent the views of Gartner or its affiliates.

Terms and Conditions

Vendors should submit only publicly available information and should not submit any information that they deem to be confidential.

1In the event fewer than 10 products qualify for either a “Small Vendor” or “Enterprise Vendor” FrontRunners graphic, the vendors that do qualify will be combined into a single graphic.


FrontRunners Methodology (Version 1: September 2016-May 2018)


The Basics

The FrontRunners methodology assesses and calculates a score for products on two primary dimensions: Capability on the x-axis and Value on the y-axis.

The Capability score is an overall weighted average of scores including:

      1. End-user ratings of one to five stars on the product’s functionality.
      2. End-user ratings of one to five stars on the product’s ease of use.
      3. End-user ratings of one to five stars on the product’s customer support.
      4. A score, relative to other products in the market, for the product's inclusion of key functionality for the software category.
      5. A score, relative to other products in the market, representing the number of other products that integrate with it.

The Value score is an overall weighted average of scores including:

      1. End-user ratings of one to five stars on overall satisfaction with the product.
      2. End-user ratings of one to five stars on how valuable users consider the product to be relative to its price.
      3. End-user ratings of one to five stars on how likely they are to recommend the product to others.
      4. A score, relative to other products in the market, for the size of the product's customer base.
      5. A score, relative to other products in the market, for the number of professionals in the market who have experience with the product (e.g., users, developers, administrators).
      6. A score, relative to other products in the market, representing the total number of user reviews across the three Gartner web properties.
      7. A score, relative to other products in the market, representing the average number of times per month internet users search for the product on Google.

Markets are defined by a core set of functionality, and products considered for, and included in, FrontRunners must offer that core set of functionality. Additional related functionality can contribute to the capability score for a product. To qualify for consideration in a FrontRunners graphic, a product must have a minimum number of unique, user-submitted product reviews across the three Gartner Digital Markets web properties: softwareadvice.com, capterra.com and getapp.com. The minimum number of reviews required per product may differ by category, but will generally be between 10 and 20 unique reviews.

More Methodology Details

The FrontRunners methodology assesses products on two primary dimensions: Capability on the x-axis and Value on the y-axis. Products receive a score between one and five for each axis. Products that meet a minimum score for each axis are included as FrontRunners. The minimum score cutoff to be included in the FrontRunners graphic varies by category, depending on the range of scores in each category. For products included, the Capability and Value scores determine their positions on the FrontRunners graphic.

FrontRunners Graphic

The Capability score is based on three criteria: user ratings on capability, a functionality breadth analysis, and a business confidence assessment.

      1. The capability user ratings criterion captures user satisfaction with the product’s capabilities. The capability ratings score is a weighted average of the one- to five- star rating scores from three user ratings:
        1. Functionality
        2. Ease of use
        3. Customer support
      2. The functionality breadth analysis is based on:
        1. The product’s coverage of core software category functions
        2. The number of other products it integrates with, along with the number of other products that state they integrate with it

For each of these data points, the methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for FrontRunners consideration. That percentile ranking is then translated into a one to five score.

      1. The business confidence assessment is an indicator of whether the software company will likely continue to invest in the product for the next 12-18 months. The analysis is based on four data points:
        1. The product’s current customer base
        2. The annual growth rate of the product’s customer base
        3. The vendor’s current employee base
        4. The annual growth rate of the vendor’s employee base

If the company's size and product's customer base are both significant and growing, then the likelihood that the business will invest in the product is higher than in the alternative scenarios. For each of these four data points, the methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for FrontRunners consideration. That percentile ranking is then translated to a one to five score.

The overall one to five Capability score is a weighted average of the scores for user ratings, functionality breadth and business confidence.

The Value score is based on two criteria: user ratings on value and product adoption.

        1. The value user ratings criterion captures users' satisfaction with the business value provided by the product. The value ratings score is a weighted average of the one to five star rating scores from three user ratings:
          1. Overall ratings of the product.
          2. How likely users are to recommend the product to others.
          3. How valuable users consider the product to be relative to its price.
        2. The product adoption data analysis assesses if the product is positioned in the market as more of an industry standard with higher adoption (thus earning a higher score), or as an emerging competitor with more limited adoption (thus earning a lower score). The product adoption methodology analysis for each product is based on four data points:
          1. The size of the product's customer base.
          2. The number of professionals in the market who have experience with the product (e.g., users, developers, administrators).
          3. The total number of user reviews across the three Gartner web properties.
          4. The average number of times per month internet users search for the product on Google.

For each of these four data points, the methodology calculates the percentile ranking for each product relative to all other products in the software category that have qualified for FrontRunners consideration. That percentile ranking is then translated into a one to five score.

The overall one to five Value score is a weighted average of the scores for value user ratings and product adoption.

Data

Data sources include user reviews and ratings, public data sources and data from technology vendors. The user-generated product reviews data incorporated into FrontRunners is collected from submissions to all three Gartner Digital Markets sites (softwareadvice.com, capterra.com and getapp.com). As a quality check, we ensure the reviewer is valid, that the review meets quality standards and that it is not a duplicate.

The business confidence and product adoption data comes from public sources, collected by either a third-party data provider or by Gartner associates. As a quality check, we compare this data against data submitted by the providers. We use this data to calculate a product's percentile ranking, which allows us to determine how products compare relative to one another rather than determine an absolute number.

The functionality breadth data is collected from the technology providers. We check the data provided and challenge data that seems inflated or unlikely. We use this data to calculate a product's percentile ranking, which allows us to determine how products compare relative to one another rather than determine an absolute number.

See FrontRunners frequently asked questions (FAQ) for more information on the methodology.

External Usage Guidelines

Providers must abide by the FrontRunners External Usage Guidelines when referencing FrontRunners content. Except in digital media with character limitations, the following disclaimer MUST appear with any/all FrontRunners reference(s) and graphic use:

FrontRunners scores and graphics are derived from individual end-user reviews based on their own experiences, vendor-supplied information and publicly available product information; they do not represent the views of Gartner or its affiliates.

Terms and Conditions
Qualifying vendor's corporate logo is used to identify its position in the FrontRunners graphic. Qualifying vendor grants to Software Advice a non-exclusive, worldwide license to use its corporate logo for this purpose, provided that such use (i) adheres to qualifying vendor’s standard approved format; and (ii) is limited to the Software Advice website and related FrontRunners materials.  

Vendors should submit only publicly available information and should not submit any information that they deem to be confidential.