You are on page 1of 9

BREACH DETECTION SYSTEMS COMPARATIVE

REPORT
Security Value Map™ (SVM)

OCTOBER 19, 2017
Author – Thomas Skybakmoen

Tested Products
Check Point Software Technologies 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Appliance
R77.30
Cisco FirePower 8120 v.6 & Cisco AMP v.5.1.9.10430

FireEye Network Security NX 10450 v7.9.2 & EX 8400 v7.9.0

FireEye Network Security 6500NXES-VA v7.9.2
Fortinet FortiSandbox-2000E v.FSA 2.4.1 & FortiClient (APT Agent) v.5.6.0.1075

Lastline Enterprise v7.25
Trend Micro Deep Discovery Inspector Model 4000 v3.8 SP5 & OfficeScan (OSCE) v.12.0.1807

Environment
Breach Detection Systems Test Methodology v4.0

Individual Test Reports are available for each product tested. Figure 1 – NSS Labs 2017 Security Value Map (SVM) for Breach Detection Systems (BDS) This report is Confidential and is expressly limited to NSS Labs’ licensed users. The SVM provides an aggregated view of the detailed findings from NSS’ group tests. The terms TCO per Protected Mbps and Value are used interchangeably throughout the Comparative Reports. and TCO.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Overview Empirical data from individual Test Reports and Comparative Reports is used to create NSS Labs’ unique Security Value Map™ (SVM). Comparative Reports provide detailed comparisons across all tested products in the areas of security. The SVM illustrates the relative value of security investment options by mapping the Security Effectiveness and the Total Cost of Ownership (TCO) per Protected Mbps (Value) of tested product configurations. performance. 2 .

3 . or Caution (bottom left).82. and the SVM. In addition. performance.36%. five products demonstrated value above the average. one product received a Security Recommended rating.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Key Findings ● Four products achieved a Recommended rating. This report is Confidential and is expressly limited to NSS Labs’ licensed users. Neutral (top left or bottom right). one product received a Neutral rating.nsslabs.0% Above average $35 Above average Recommended Figure 2 – NSS Labs’ 2017 Recommendations for Breach Detection Systems (BDS) This report is part of a series of Comparative Reports on security. ● The average Security Effectiveness rating was 93.0% Above average $16 Above average Recommended Lastline 100.2% and 100. five products received a Security Effectiveness rating above the average.2% Below average $43 Above average Neutral Fortinet 98. with most tested products costing less than US$44 per protected Mbps. and one product received a Caution rating.0%. For more information.  ● The average TCO per Protected Mbps (Value) was US$48.0% Above average $128 Below average Neutral (Security Recommended) FireEye NX & EX 81.0% Above average $25 Above average Recommended Trend Micro 100. and two demonstrated value below the average. TCO.2%.com.7% Below average $76 Below average Caution FireEye NXES-VA 80. ● False positive rates ranged from 0% to 0. and two received a Security Effectiveness rating below the average. ● TCO per Protected Mbps ranged between US$16 and US$128. NSS clients have access to an NSS Labs SVM Toolkit™ that allows for the incorporation of organization-specific costs and requirements to create a completely customized SVM. ● Five out of the seven products tested missed evasions.7% Above average $20 Above average Recommended Cisco 96. visit www. Value in US$ Product Security Effectiveness Overall Rating (TCO per Protected Mbps) Check Point 96. ● Overall Security Effectiveness ranged between 80. see the How to Read the SVM section of this document. Product Rating The Overall Rating in Figure 2 is determined by which section of the SVM the product falls within: Recommended (top right). For more information on how the SVM is constructed.

....................................................6 & Cisco AMP v..................... 7 Recommended ...................................................................................... 7 Fortinet FortiSandbox-2000E v............................................. 8 Test Methodology ..............................9..........1.................................................... 6 Analysis .......................................................................................1 & FortiClient (APT Agent) v....................... 7 Lastline Enterprise v7...............NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Table of Contents Tested Products ................................................................. 2 Figure 2 – NSS Labs’ 2017 Recommendations for Breach Detection Systems (BDS) ............................................................................... 2 Key Findings ........ 8 FireEye Network Security NX 10450 v7....................... 5 The y axis ..................................................................................................................................................................................................................................9.................0 .......................................... 7 Neutral ................................................5.................. 5 This report is Confidential and is expressly limited to NSS Labs’ licensed users....... 1 Overview................................................1075 ........................................................ 9 Table of Figures Figure 1 – NSS Labs 2017 Security Value Map (SVM) for Breach Detection Systems (BDS) .......................................................6........................... 3 Product Rating...... 1 Environment .. 3 Figure 3 – Example SVM ....................................9............................. 8 Caution............................FSA 2.....................1807 .......... 8 Cisco FirePower 8120 v............................................................................................................................................................................. 7 Trend Micro Deep Discovery Inspector Model 4000 v3...............................0......30 .................................................................................................................. 9 Contact Information ...........................................................................2 ..................................................................................... 3 How to Read the SVM.................................................................... 8 FireEye Network Security 6500NXES-VA v7...................12........................................................................5.....................................................9......... 7 Check Point Software Technologies 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Appliance R77....................................2 & EX 8400 v7.....................25 ..............................................................................................................................................0.....................................................................8 SP5 & (OfficeScan) OSCE v............................10430 ........................................................ 4 ............................................................................................................................................................... 5 The x axis ..................................4.............................................................................

The TCO incorporates capital expenditure (capex) costs over a three-year period.com. For more details on Security Effectiveness and TCO. NSS has developed a unique metric: TCO per Protected Mbps. Running a multi-device deployment provides a more accurate reflection of cost than running only a single BDS. The x axis displays the TCO per Protected Mbps in US dollars. In procuring a BDS solution for the enterprise. including initial acquisition and deployment costs and annual maintenance and update costs (software and hardware updates). This report is Confidential and is expressly limited to NSS Labs’ licensed users. in general. an enterprise must deploy 500 agents and/or one network device of 1. 5 . a log aggregation and/or event management unit). which decreases from left to right.nsslabs. In order to enable value-based comparisons of BDS products on the market. NSS has found that the malware detection rates of some BDS network devices drop when they operate at maximum capacity.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 How to Read the SVM The SVM depicts the value of a typical deployment of four BDS products plus one central management unit (and where necessary. making precise comparisons extremely difficult. see the Security and TCO comparative reports at www. This metric incorporates the 3-Year TCO with the Security Effectiveness score to provide a data point with which to compare the actual value of each product tested. it is essential to factor in both bandwidth and the number of users. NSS research has shown that.000 Mbps capacity. enterprise network administrators architect their networks for up to 2 Mbps of sustained throughput per employee. to support 500 users. For example. Figure 3 – Example SVM No two security products deliver the same security effectiveness or performance. The formula used is as follows: 3-Year TCO/ (Security Effectiveness x NSS- Tested Throughput).

The highest point of the gradient line represents Security Effectiveness based solely on block rate. However. This report is Confidential and is expressly limited to NSS Labs’ licensed users. These products provide a high level of detection and value for money. Security Effectiveness is greater toward the top of the y axis. which more realistically depicts the actual Security Effectiveness of a product. ● Caution: Products that map into the lower-left section of the SVM offer limited value for money given their 3- Year TCO and measured Security Effectiveness. Conversely. The Security Effectiveness score of some products is represented by two data points (a blue dot and a gradient line). Devices that are missing critical security capabilities will have a reduced Security Effectiveness score. These products would be suitable for environments where a slightly lower level of detection is acceptable in exchange for a lower TCO. The Security Effectiveness score of products that did not miss any evasions is represented by a single green dot. In all cases. the SVM should only be a starting point. Neutral products in the upper-left section score above the average for Security Effectiveness but below the average for TCO per Protected Mbps (Security Recommended).NSS Labs Breach Detection Systems Comparative Report — SVM_101917 The y axis displays the Security Effectiveness score as a percentage. Clients can also meet with NSS analysts if they wish to develop a custom SVM. These products are suitable for environments requiring a high level of detection. These lines divide the SVM into four unequally sized sections. which allows for the incorporation of organization-specific costs and requirements to create a custom SVM. Incorporating this additional information allows NSS to calculate a second. The SVM displays two dotted lines that represent the average for the Security Effectiveness and TCO per Protected Mbps ratings of all the tested products. 6 . NSS clients have access to the SVM Toolkit. albeit at a higher-than-average cost. lower score (represented by the blue dot). Neutral products in the lower-right section score below the average for Security Effectiveness but above the average for TCO per Protected Mbps. this is not the only measure of Security Effectiveness—NSS also factors in evasions. Where a product’s Security Effectiveness and TCO per Protected Mbps scores map on the SVM will determine which section it falls into: ● Recommended: Products that map into the upper-right section of the SVM score well for both Security Effectiveness and TCO per Protected Mbps. ● Neutral: Products that map into either the upper-left or lower-right sections may be good choices for organizations with specific security or budget requirements.

Stability and Reliability The product passed all stability and reliability tests. The product failed to detect 2% of the sandbox evasions it was tested against. The product proved effective against all evasion techniques it was tested Evasions against. Vendors are listed alphabetically within each section.0.0%.12.1 & FortiClient (APT Agent) v.8 SP5 & (OfficeScan) OSCE v.7%. the product was rated by NSS at 3. or Caution. Performance Rating During performance testing. The device failed to detect 50% of the web socket connection evasions it was Evasions tested against. Stability and Reliability The product passed all stability and reliability tests. Performance Rating During performance testing.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Analysis Each product may fall into one of three categories based on its rating in the SVM: Recommended. The product proved effective against all evasion techniques it was tested Evasions against. Each of the tested products receives only a single rating.667 Mbps. Fortinet FortiSandbox-2000E v.1807 The Trend Micro Deep Discovery Inspector Model 4000 and OSCE received a Detection Rate breach detection rating of 100.0. 7 . Stability and Reliability The product passed all stability and reliability tests.0%. Please see the Test Report for additional details.1075 The Fortinet FortiSandbox-2000E & FortiClient (ATP Agent) received a breach Detection Rate detection rating of 99. the product was rated by NSS at 5. Performance Rating During performance testing. the product was rated by NSS at 8.000 Mbps. Stability and Reliability The product passed all stability and reliability tests.6.667 Mbps.4. This report is Confidential and is expressly limited to NSS Labs’ licensed users.5. Lastline Enterprise v7.667 Mbps. Recommended Check Point Software Technologies 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Appliance R77. Neutral.25 Detection Rate The Lastline Enterprise received a breach detection rating of 100%. Performance Rating During performance testing. Trend Micro Deep Discovery Inspector Model 4000 v3. Evasions Please see the Test Report for additional details.30 The Check Point 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Detection Rate Appliance received a breach detection rating of 99. the product was rated by NSS at 8.FSA 2.

6%.0%.9.9.667 Mbps.5.2 & EX 8400 v7.10430 The Cisco FirePower 8120 & Cisco AMP received a breach detection rating of Detection Rate 99. Performance Rating During performance testing. This report is Confidential and is expressly limited to NSS Labs’ licensed users. 100% of web Evasions socket connection evasions.4%. Caution FireEye Network Security NX 10450 v7. the product was rated by NSS at 1. Stability and Reliability The product passed all stability and reliability tests.0 The FireEye Network Security NX 10450 & EX 8400 received a breach detection Detection Rate rating of 98. 8 .2 The FireEye Network Security 6500NXES-VA received a breach detection rating Detection Rate of 96.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Neutral Cisco FirePower 8120 v. 100% of web Evasions socket connection evasions. The product failed to detect 5.9% of the sandbox evasions it was tested against. Performance Rating During performance testing. and 20% of the HTTP evasions it was tested against. Please see the Test Report for additional details. the product was rated by NSS at 750 Mbps. The product failed to detect 2% of packer & compressor evasions. and 20% of the HTTP evasions it was tested against. FireEye Network Security 6500NXES-VA v7. Evasions Please see the Test Report for additional details.6 & Cisco AMP v.9.1. The product failed to detect 2% of packer & compressor evasions. Stability and Reliability The product passed all stability and reliability tests. Please see the Test Report for additional details. the product was rated by NSS at 5. Stability and Reliability The product passed all stability and reliability tests. Performance Rating During performance testing.000 Mbps.9.

EXEMPLARY. We are not liable or responsible for any damages. FITNESS FOR A PARTICULAR PURPOSE. needs. or specifications. and trade names of their respective owners. OR OTHER ASSETS. 3711 South MoPac Expressway Building 1.com www. CONSEQUENTIAL. NO WARRANTIES. 6. PUNITIVE.0 A copy of the test methodology is available on the NSS Labs website at www. DATA.com. TX 78746-8022 USA info@nsslabs. Inc.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Test Methodology Breach Detection Systems: Test Methodology v4. OR FOR ANY LOSS OF PROFIT. and trade names used in this report are the trademarks. The information in this report is subject to change by us without notice. e-mailed or otherwise disseminated or transmitted without the express written consent of NSS Labs. “You” or “your” means the person who accesses this report and any entity on whose behalf he/she has obtained this report. Please read the disclaimer in this box because it contains important information that binds you.nsslabs. All use of and reliance on this report are at your sole risk. This report does not constitute an endorsement. EXPRESS OR IMPLIED ARE GIVEN BY US. This report does not imply any endorsement. please contact NSS Labs. requirements. 3. Inc. © 2017 NSS Labs. 4. 9 . All rights reserved. This report is Confidential and is expressly limited to NSS Labs’ licensed users. or guarantee of any of the products (hardware or software) tested or the hardware and/or software used in testing the products. If you do not agree to these conditions. 5.nsslabs. stored on a retrieval system. but is not guaranteed. copied/scanned. sponsorship. recommendation. 2. All trademarks. Inc. and we disclaim any obligation to update it. you should not read the rest of this report but should instead return the report immediately to us. OR INDIRECT DAMAGES. EVEN IF ADVISED OF THE POSSIBILITY THEREOF. No part of this publication may be reproduced.nsslabs. COMPUTER PROGRAMS. To receive a licensed copy or report misuse. AND NON-INFRINGEMENT. service marks. affiliation. The testing does not guarantee that there are no errors or defects in the products or that the products will meet your expectations. service marks. REVENUE. or verification by or with any organizations mentioned in this report. INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY. (“us” or “we”). IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT.com This and other related documents are available at: www. 1. losses. Suite 400 Austin. ALL IMPLIED WARRANTIES. INCIDENTAL. The information in this report is believed by us to be accurate and reliable at the time of publication. or expenses of any nature whatsoever arising from any error or omission in this report. Contact Information NSS Labs. or that they will operate without interruption. ARE HEREBY DISCLAIMED AND EXCLUDED BY US.com.