Methodology

General Comparison Model

Kaspersky Lab’s Guru Service enables users to evaluate the application control features of different endpoint protection solutions.

Based on the results of recent independent tests by AV-Test and Dennis Technology Labs, Kaspersky Guru uses a specially developed General Comparison Model to treat this test data equally, despite the differences in methodologies.

As the diagram opposite shows, this model is based on three main scenarios where endpoint protection is bundled with application control functionality: deployment and configuration, monitoring and support. Evaluation and comparison criteria focused on three key characteristics of any security solution: quality and functionality, ease of use and no adverse effects on system stability, performance or functionality.

Dennis Technology Labs’ comparison and evaluation model

Dennis Technology Labs’ methodology assesses solutions based on five scenarios: ability to identify software on the network, classify as appropriate/inappropriate, manage users and endpoints, provide in-depth logs and reporting, and minimal impact on business processes.

The diagram below shows Dennis Technology Labs’ model before and after its incorporation into the General Comparison Model.

Dennis sought to determine the capabilities of various endpoint protection products in solving two main problems relevant to the corporate user:

  • Ability to deny all non-work related, unauthorized or spurious software
  • Capacity to defend against advanced persistent threats

Dennis tested each product against multiple criteria, assigning score from 0 to 10. The score reflects the number of settings and features available in the product as well as quality of implementation and ease of use of each individual feature.

AV-Test comparison and evaluation model

AV-Test’s methodology also uses five scenarios to assess endpoint protection-application control bundles: deployment, configuration, monitoring, protection and support.

The below diagram shows the AV-Test model before and after its incorporation into the General Comparison Model:

To evaluate and compare products, AV-Test assessed each one against multiple test cases and assigned three scores each from 0 to 10. The scores reflected product characteristics such as: quality and functionality, ease of use and absence of adverse effects. As you can see, these three evaluation criteria were the basis of Kaspersky Guru’s General Comparison Model.

Aggregation of results

Comparing Dennis Technology Labs and AV-Test's evaluation and comparison models, it’s easy to see that AV-Test’s model covers more scenarios and is better balanced. Moreover, the criteria used to compare products are granular.

The below diagram illustrates how Dennis Labs and AV-Test scenarios were incorporated into the General Comparison Model for use in Kaspersky Guru:

Thus, Dennis Technology Labs’ results are a set of estimates for each of the products are tested - one score for each test scenario. This means the results are a mixture of the three criteria described in the AV-Test methodology. Because it’s impossible to split scores assigned by Dennis Technology, the results are used in Kaspersky Guru as evaluation of "Quality and functionality".

The table below shows how the AV-Test and Dennis Labs’ results were used by Kaspersky Guru.

Product ranking

Both AV-Test and Dennis Technology Labs based their assessments of endpoint protection solutions on their own perception of end-user needs. Kaspersky Guru was specially created to give users the ability to compare products based on their personal preferences and requirements.

Kaspersky Guru’s main interface consists of questions corresponding to general user needs, offering answer options that are closely linked to AV-Test and Dennis Technology Labs’ scores in each of their test scenarios. Kaspersky Guru analyzes these answers and compares products based on the user’s criteria. In this way, the user is able to determine the weighting of any particular feature in terms of the overall test score given. Products are sorted based on their overall score – the more a solution corresponds to the user preference determined by Kaspersky Guru, the higher it will feature.

The below diagram shows how user answers determine the importance of specific product characteristics in Kaspersky Guru. Shaded blocks are for answers selected by user.

AV-Test’s methodology includes a fourth criterion: “Efficiency” (in addition to quality and functionality, ease of use, absence of adverse effects). This criterion is calculated using a simple formula: [(Quality and functionality - Negative effects ) * Ease of use ].

The total score for each product is calculated as the average efficiency, which is itself calculated based on the average score (AV-Test and Dennis Technology Labs) multiplied by the user’s priorities. The diagram below shows the algorithm for calculating the total score, based on the results of AV-Test and Dennis Technology Labs tests.