Threat information is extremely time sensitive; knowing about a threat from weeks ago is useless. The success of the cloud-based system also depends on a bidirectional data flow: GTI gathers data from millions of client sensors and provides real-time intelligence back to these end products, at a rate of 100 billion queries per month.In order to provide up to date, comprehensive threat information, Mc Afee needs to quickly process terabytes of different data types (such as IP address or domain) into meaningful relationships: e.g. Mc Afee was unable to address these needs and effectively scale out to millions of records with their existing solutions.Mc Afee compensated for all the rebuilding and redeploying of Katta shards with “the usual scripting duct tape,” but what they really needed was a solution that could seamlessly handle the sharding and updating on its own.“We were spending more time building solutions in-house rather than focusing on threat research,” said Mc Afee IT Architect Wes Widner.“We needed a database engine to take care of itself and let us do our jobs – find interesting bits in the data, figure out who’s being naughty on the web at any given moment, and report that up the chain for whoever wants to use it.”Mc Afee selected Mongo DB, which had excellent documentation and a growing community that was “on fire.”The authoritative source for Mc Afee threat information, Mongo DB enables big data analytics and supports the real-time flow of cyberthreat data between GTI’s cloud-based system and end client products.For example, the HBase / Hadoop setup made it difficult to run interesting, complex queries, and experienced bugs with the Java garbage collector running out of memory.
It currently stores 4 billion documents – terabytes of data.
Auto-sharding makes it easy to add more servers at any time to handle GTI’s increasing data needs.
In 2010, it became clear that Mc Afee’s existing database solutions would not be able to handle the demands of exponential data growth.
The team spent a significant amount of time investigating workarounds and fixes, which created new cracks in the system.
Mc Afee turned to Mongo DB to achieve the scale, performance and flexibility required for big data analysis.Mc Afee GTI analyzes cyberthreats from all angles, identifying threat relationships, such as malware used in network intrusions, websites hosting malware, botnet associations, and more.