You are on page 1of 9

BREACH DETECTION SYSTEMS COMPARATIVE

REPORT
Security Value Map™ (SVM)

OCTOBER 19, 2017
Author – Thomas Skybakmoen

Tested Products
Check Point Software Technologies 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Appliance
R77.30
Cisco FirePower 8120 v.6 & Cisco AMP v.5.1.9.10430

FireEye Network Security NX 10450 v7.9.2 & EX 8400 v7.9.0

FireEye Network Security 6500NXES-VA v7.9.2
Fortinet FortiSandbox-2000E v.FSA 2.4.1 & FortiClient (APT Agent) v.5.6.0.1075

Lastline Enterprise v7.25
Trend Micro Deep Discovery Inspector Model 4000 v3.8 SP5 & OfficeScan (OSCE) v.12.0.1807

Environment
Breach Detection Systems Test Methodology v4.0

2 . and TCO. Comparative Reports provide detailed comparisons across all tested products in the areas of security.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Overview Empirical data from individual Test Reports and Comparative Reports is used to create NSS Labs’ unique Security Value Map™ (SVM). Figure 1 – NSS Labs 2017 Security Value Map (SVM) for Breach Detection Systems (BDS) This report is Confidential and is expressly limited to NSS Labs’ licensed users. The SVM illustrates the relative value of security investment options by mapping the Security Effectiveness and the Total Cost of Ownership (TCO) per Protected Mbps (Value) of tested product configurations. The terms TCO per Protected Mbps and Value are used interchangeably throughout the Comparative Reports. Individual Test Reports are available for each product tested. The SVM provides an aggregated view of the detailed findings from NSS’ group tests. performance.

0% Above average $16 Above average Recommended Lastline 100. visit www. TCO. 3 . five products received a Security Effectiveness rating above the average. or Caution (bottom left).82. Value in US$ Product Security Effectiveness Overall Rating (TCO per Protected Mbps) Check Point 96. performance.0% Above average $35 Above average Recommended Figure 2 – NSS Labs’ 2017 Recommendations for Breach Detection Systems (BDS) This report is part of a series of Comparative Reports on security. and two received a Security Effectiveness rating below the average. and the SVM. see the How to Read the SVM section of this document. one product received a Security Recommended rating.7% Below average $76 Below average Caution FireEye NXES-VA 80. NSS clients have access to an NSS Labs SVM Toolkit™ that allows for the incorporation of organization-specific costs and requirements to create a completely customized SVM.7% Above average $20 Above average Recommended Cisco 96.2% and 100. ● False positive rates ranged from 0% to 0. ● Overall Security Effectiveness ranged between 80. and one product received a Caution rating. and two demonstrated value below the average. Product Rating The Overall Rating in Figure 2 is determined by which section of the SVM the product falls within: Recommended (top right). ● TCO per Protected Mbps ranged between US$16 and US$128.2%. with most tested products costing less than US$44 per protected Mbps. ● Five out of the seven products tested missed evasions. ● The average Security Effectiveness rating was 93.nsslabs.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Key Findings ● Four products achieved a Recommended rating.0%. one product received a Neutral rating. This report is Confidential and is expressly limited to NSS Labs’ licensed users. Neutral (top left or bottom right).0% Above average $25 Above average Recommended Trend Micro 100. For more information.0% Above average $128 Below average Neutral (Security Recommended) FireEye NX & EX 81.2% Below average $43 Above average Neutral Fortinet 98.com. In addition. five products demonstrated value above the average.  ● The average TCO per Protected Mbps (Value) was US$48. For more information on how the SVM is constructed.36%.

........................................................... 8 FireEye Network Security 6500NXES-VA v7......................................................0....................................................................... 2 Key Findings ..................................... 5 The y axis ........................ 9 Contact Information ....................................................................................................................10430 ........................................................................................9...................................2 & EX 8400 v7........................................................... 3 Figure 3 – Example SVM ...................................................................................... 1 Environment ........... 7 Trend Micro Deep Discovery Inspector Model 4000 v3...................0 ...................FSA 2...............................................................1 & FortiClient (APT Agent) v..................................................................................... 3 How to Read the SVM................................NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Table of Contents Tested Products . 7 Neutral .......................................................... 4 ...................................................................................................................................................... 7 Fortinet FortiSandbox-2000E v........................................................................................25 ..9..........6...................... 7 Lastline Enterprise v7........................ 9 Table of Figures Figure 1 – NSS Labs 2017 Security Value Map (SVM) for Breach Detection Systems (BDS) .... 8 Caution..... 7 Recommended ..............................................................................................1075 .................................................................................... 2 Figure 2 – NSS Labs’ 2017 Recommendations for Breach Detection Systems (BDS) ........................... 1 Overview............30 ............................................................1............1807 ................................................................................2 .......... 8 Cisco FirePower 8120 v............................................................................................................................ 5 The x axis .................................................... 6 Analysis ....... 5 This report is Confidential and is expressly limited to NSS Labs’ licensed users..............................................................................................................................5......................................9........0......... 3 Product Rating................ 7 Check Point Software Technologies 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Appliance R77..............................................................................................................8 SP5 & (OfficeScan) OSCE v........................................................................ 8 FireEye Network Security NX 10450 v7.... 8 Test Methodology ..............................................................................................4................................................................5.............................................................................................................................................................6 & Cisco AMP v..........12.............9..............................................................................................

This metric incorporates the 3-Year TCO with the Security Effectiveness score to provide a data point with which to compare the actual value of each product tested. 5 . to support 500 users. NSS has found that the malware detection rates of some BDS network devices drop when they operate at maximum capacity. Running a multi-device deployment provides a more accurate reflection of cost than running only a single BDS. For example. NSS has developed a unique metric: TCO per Protected Mbps. including initial acquisition and deployment costs and annual maintenance and update costs (software and hardware updates). which decreases from left to right. Figure 3 – Example SVM No two security products deliver the same security effectiveness or performance.000 Mbps capacity.nsslabs. NSS research has shown that. it is essential to factor in both bandwidth and the number of users. see the Security and TCO comparative reports at www. In procuring a BDS solution for the enterprise. in general. The TCO incorporates capital expenditure (capex) costs over a three-year period. In order to enable value-based comparisons of BDS products on the market. making precise comparisons extremely difficult. This report is Confidential and is expressly limited to NSS Labs’ licensed users. an enterprise must deploy 500 agents and/or one network device of 1.com. The x axis displays the TCO per Protected Mbps in US dollars. The formula used is as follows: 3-Year TCO/ (Security Effectiveness x NSS- Tested Throughput). a log aggregation and/or event management unit). For more details on Security Effectiveness and TCO. enterprise network administrators architect their networks for up to 2 Mbps of sustained throughput per employee.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 How to Read the SVM The SVM depicts the value of a typical deployment of four BDS products plus one central management unit (and where necessary.

● Neutral: Products that map into either the upper-left or lower-right sections may be good choices for organizations with specific security or budget requirements. The SVM displays two dotted lines that represent the average for the Security Effectiveness and TCO per Protected Mbps ratings of all the tested products. Where a product’s Security Effectiveness and TCO per Protected Mbps scores map on the SVM will determine which section it falls into: ● Recommended: Products that map into the upper-right section of the SVM score well for both Security Effectiveness and TCO per Protected Mbps. ● Caution: Products that map into the lower-left section of the SVM offer limited value for money given their 3- Year TCO and measured Security Effectiveness. which allows for the incorporation of organization-specific costs and requirements to create a custom SVM. In all cases. lower score (represented by the blue dot). NSS clients have access to the SVM Toolkit. Devices that are missing critical security capabilities will have a reduced Security Effectiveness score. Neutral products in the upper-left section score above the average for Security Effectiveness but below the average for TCO per Protected Mbps (Security Recommended). The Security Effectiveness score of products that did not miss any evasions is represented by a single green dot. These products are suitable for environments requiring a high level of detection. These products would be suitable for environments where a slightly lower level of detection is acceptable in exchange for a lower TCO. This report is Confidential and is expressly limited to NSS Labs’ licensed users. These lines divide the SVM into four unequally sized sections. Clients can also meet with NSS analysts if they wish to develop a custom SVM. The Security Effectiveness score of some products is represented by two data points (a blue dot and a gradient line). Neutral products in the lower-right section score below the average for Security Effectiveness but above the average for TCO per Protected Mbps. Security Effectiveness is greater toward the top of the y axis. the SVM should only be a starting point. 6 . However. Incorporating this additional information allows NSS to calculate a second. These products provide a high level of detection and value for money. The highest point of the gradient line represents Security Effectiveness based solely on block rate. which more realistically depicts the actual Security Effectiveness of a product. Conversely. this is not the only measure of Security Effectiveness—NSS also factors in evasions. albeit at a higher-than-average cost.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 The y axis displays the Security Effectiveness score as a percentage.

1807 The Trend Micro Deep Discovery Inspector Model 4000 and OSCE received a Detection Rate breach detection rating of 100. the product was rated by NSS at 3.6. Vendors are listed alphabetically within each section. the product was rated by NSS at 5. The device failed to detect 50% of the web socket connection evasions it was Evasions tested against. or Caution.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Analysis Each product may fall into one of three categories based on its rating in the SVM: Recommended. Stability and Reliability The product passed all stability and reliability tests. Stability and Reliability The product passed all stability and reliability tests. Each of the tested products receives only a single rating. Evasions Please see the Test Report for additional details.30 The Check Point 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Detection Rate Appliance received a breach detection rating of 99. The product failed to detect 2% of the sandbox evasions it was tested against.667 Mbps.4.0. Stability and Reliability The product passed all stability and reliability tests. the product was rated by NSS at 8. This report is Confidential and is expressly limited to NSS Labs’ licensed users.8 SP5 & (OfficeScan) OSCE v. Performance Rating During performance testing.000 Mbps.1 & FortiClient (APT Agent) v.0. Performance Rating During performance testing.667 Mbps. The product proved effective against all evasion techniques it was tested Evasions against. Fortinet FortiSandbox-2000E v. 7 .12. Neutral. Stability and Reliability The product passed all stability and reliability tests. Lastline Enterprise v7.0%.0%.667 Mbps. Please see the Test Report for additional details. The product proved effective against all evasion techniques it was tested Evasions against. Recommended Check Point Software Technologies 15600 Next Generation Threat Prevention & SandBlast™ (NGTX) Appliance R77. Performance Rating During performance testing. Trend Micro Deep Discovery Inspector Model 4000 v3.25 Detection Rate The Lastline Enterprise received a breach detection rating of 100%.1075 The Fortinet FortiSandbox-2000E & FortiClient (ATP Agent) received a breach Detection Rate detection rating of 99.5.7%. the product was rated by NSS at 8. Performance Rating During performance testing.FSA 2.

NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Neutral Cisco FirePower 8120 v. 100% of web Evasions socket connection evasions. The product failed to detect 2% of packer & compressor evasions. The product failed to detect 2% of packer & compressor evasions. 8 . the product was rated by NSS at 750 Mbps. Caution FireEye Network Security NX 10450 v7.9.2 & EX 8400 v7. Please see the Test Report for additional details.2 The FireEye Network Security 6500NXES-VA received a breach detection rating Detection Rate of 96.9.10430 The Cisco FirePower 8120 & Cisco AMP received a breach detection rating of Detection Rate 99.667 Mbps. The product failed to detect 5.0%. Evasions Please see the Test Report for additional details. This report is Confidential and is expressly limited to NSS Labs’ licensed users. 100% of web Evasions socket connection evasions. the product was rated by NSS at 5. Stability and Reliability The product passed all stability and reliability tests. the product was rated by NSS at 1. Performance Rating During performance testing.1. Performance Rating During performance testing.9.5.9% of the sandbox evasions it was tested against. Performance Rating During performance testing. Stability and Reliability The product passed all stability and reliability tests. and 20% of the HTTP evasions it was tested against.6 & Cisco AMP v. and 20% of the HTTP evasions it was tested against.9. Please see the Test Report for additional details.4%. FireEye Network Security 6500NXES-VA v7.000 Mbps.6%.0 The FireEye Network Security NX 10450 & EX 8400 received a breach detection Detection Rate rating of 98. Stability and Reliability The product passed all stability and reliability tests.

please contact NSS Labs. e-mailed or otherwise disseminated or transmitted without the express written consent of NSS Labs.com. 4.nsslabs. The information in this report is believed by us to be accurate and reliable at the time of publication. CONSEQUENTIAL. Inc. TX 78746-8022 USA info@nsslabs. This report is Confidential and is expressly limited to NSS Labs’ licensed users. (“us” or “we”).0 A copy of the test methodology is available on the NSS Labs website at www. stored on a retrieval system. service marks. and trade names of their respective owners. or guarantee of any of the products (hardware or software) tested or the hardware and/or software used in testing the products. requirements. ARE HEREBY DISCLAIMED AND EXCLUDED BY US. affiliation. recommendation. If you do not agree to these conditions. OR FOR ANY LOSS OF PROFIT. The testing does not guarantee that there are no errors or defects in the products or that the products will meet your expectations. Inc. “You” or “your” means the person who accesses this report and any entity on whose behalf he/she has obtained this report. 6. This report does not imply any endorsement.com www. We are not liable or responsible for any damages. AND NON-INFRINGEMENT. REVENUE. This report does not constitute an endorsement.nsslabs. EXPRESS OR IMPLIED ARE GIVEN BY US. Please read the disclaimer in this box because it contains important information that binds you. INCIDENTAL. All rights reserved. To receive a licensed copy or report misuse. All trademarks. copied/scanned. or specifications. ALL IMPLIED WARRANTIES. or expenses of any nature whatsoever arising from any error or omission in this report. 2. needs. service marks. PUNITIVE. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT. All use of and reliance on this report are at your sole risk. Contact Information NSS Labs. Suite 400 Austin. losses. 3.com This and other related documents are available at: www. NO WARRANTIES. EVEN IF ADVISED OF THE POSSIBILITY THEREOF. but is not guaranteed. and trade names used in this report are the trademarks. 1. DATA. COMPUTER PROGRAMS. 5. INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY. The information in this report is subject to change by us without notice. or that they will operate without interruption. No part of this publication may be reproduced. OR OTHER ASSETS. you should not read the rest of this report but should instead return the report immediately to us. sponsorship. 3711 South MoPac Expressway Building 1.com. OR INDIRECT DAMAGES. Inc. EXEMPLARY.NSS Labs Breach Detection Systems Comparative Report — SVM_101917 Test Methodology Breach Detection Systems: Test Methodology v4. 9 . © 2017 NSS Labs.nsslabs. or verification by or with any organizations mentioned in this report. FITNESS FOR A PARTICULAR PURPOSE. and we disclaim any obligation to update it.