NSS Labs vs. Palo Alto Networks
NSS Labs Inc. is causing a splash, once again, with its recent release of a new report concerning Next-Generation Firewalls. One vendor in particular was extremely displeased with their results, and angry enough to publicly dispute them. Palo Alto Networks, a known industry leader in the next-generation firewall sect, was tested alongside NGFWS such as Cisco Systems Inc., McAfee Inc., Barracuda Networks, Inc., Dell Inc., and other participants. Palo Alto’s PA-3020 received the lowest rating of any product in this year’s review, with an overall reduction of security effectiveness totaling to about 60.1%. Where in the previous year Palo Alto Networks received Palo Alto’s Senior VP of Product Management, Lee Klarich, had several things to say in response to NSS’s analysis.
Palo Alto Networks claims:
Palo Alto deliberately did not participate in the 2014 NSS Next-Generation Firewall Comparative Analysis report, and therefore is under the impression that unlike the other participating vendors, they did not have the advantage of fine tuning and configuring their product specifically for the NSS test. Their reasoning for lack of participation was that “overtime we have come to believe that the NSS model of allowing vendor test tuning prior to public test is a ‘pay-to-play’ approach and produces questionable objectivity and accuracy in results,” writes Klarich. In 2013, however, Palo Alto did participate in NSS testing and scored a 96.4%. Ever since, they insist that they have invested even more in their product’s security capabilities and do not understand how NSS could have come to such a radically different result in comparison to the 2013 test run against the same technology.
NSS writes a responding blog post:
Bob Walder, founder and Chief Research Officer at NSS labs confronts the claims made by Palo Alto and highlights their inaccuracies.
- “Palo Alto intentionally did not participate in the 2014 NSS Next-Generation Firewall Comparative Analysis” Participation in an NSS group test is not optional, writes Walder. NSS tests whatever products are sold in a particular market and whichever their enterprise clients would like to see tested. Palo Alto Networks was treated the exact same way as other participating vendors.
- “pay-to-play” NSS does not charge for any vendors for participation, the tests are performed in concordance with their own budget. In response to fine tuning, NSS uses the default configuration for each product and emphasizes repeatedly that no fine-tuning is permitted.
- “how could NSS reach such drastically different results from 2013-2014 testing the same technology” In 2013 version 4.1.9 of PAN-OS was tested and in 2014 PAN-OS 6.0.3 was tested. While version 4.1.9
He writes that Lee Klarich never addresses the main issue, which is that Palo Alto Networks NGFW misses several critical evasions leaving their customers at risk. Other vendors scored poorly in this round of testing as well, however, Walder highlights that the difference between these vendors and Palo Alto is that “the other vendor didn’t bother attacking NSS in public, but rather focused on fixing the issues and submitting their product for retesting.” Palo Alto Networks is one of the leading security platforms in the industry, and while NSS testing may not be enough to sway customers, the company does point out some interesting inconsistencies regarding their network and PR tactics.