Apache Eagle: Real-time security monitoring solution


On January 10, 2017, the Apache Software Foundation, which consists of more than 350 open source projects and innovation initiatives, all developed by volunteer, governance volunteer and incubator volunteers, announced that Apache Eagle has graduated from the Apache Incubator Program.

Eagle originated in eBay, the first to solve large-scale Hadoop cluster monitoring issues. The team quickly realized that this would also be useful for the entire community, so in October 2015 the project was submitted to the Apache Incubator. Since then, Eagle has gained the attention of developers and organizations for its extensive usage scenarios, such as system / service monitoring, application performance monitoring, and security vulnerability detection. Eagle released next version apache-eagle-0.4.0-incubating.

Apache Eagle is an open source monitoring and alerting solution for intelligent real-time identification of security and performance issues on large data platforms such as Apache Hadoop and Apache Spark.

“We are proud that Eagle is able to successfully go through the incubation process and graduate as a top Apache project,” said Edward Zhang, vice president of Apache Eagle. “Communities are aggressively improving products to explore and analyze performance and security issues in large Hadoop clusters. ”

Key features of Apache Eagle include:

Highly extensible: Apache Eagle builds its core framework around application concepts; the application itself includes the collection, pre-processing, and canonicalization of source data to be monitored. Developers can use Eagle’s application framework to easily develop their own monitoring applications and deploy them to Eagle.

Ease of Use: Usability is also one of the core design principles of Eagle products. With Eagle’s Sandbox, users can set up their environment in just a few minutes and start experimenting. In order to make the user experience as simple as possible, there are lot of good examples, simply click a few mouse clicks, you can easily complete the strategy to create and add.

User Profile : Eagle built-in machine learning algorithm based on user behavior in Hadoop habits to establish user profiles. A variety of default machine learning algorithms that you can use to model different HDFS feature sets. With the historical behavior model, Eagle can detect abnormal user behavior and generate alerts in real time.

Lightweight distributed flow processing framework : DAG-based model for abstracting the general stream processing paradigm, in the development of user-defined control program based on DSL API stream processing logic, run-time and then select the actual physical execution environment, the default Support for single-process and Storm, but also to support the expansion of other execution environment, such as Spark Streaming or Flink and so on.

Real-Time Streaming Aggregation Engine: Provides easy-to-use, real-time streaming aggregation rules that define syntax, metadata-driven, dynamic deployment, and real-time monitoring of data streams for linear expansion.

Highly scalable: The project’s run-time environment is based on proven large data technology and uses an extensible core that adapts to the throughput of the data flow and the number of monitored applications.

Low latency: Provides the most advanced alarm engine to quickly identify security vulnerabilities and performance issues.

Dynamic collaboration: Users are free to enable or disable monitoring applications and dynamically change their alert policies without impacting the underlying runtime environment.

Distributed Policy Engine: distributed real-time warning rules execution engine, providing descriptive rules like SQL syntax and machine learning automatically extended to support a variety of extensions to the early warning rules of dynamic loading and partitioning.

Storage and query framework: Universal monitoring data storage framework can be used to store and query logs, indicators, alarms, events and other types of data, the default support Hbase , and for HBase a variety of optimization and expansion, such as coprocesser, Partition, etc., also supports other types of storage extensions such as RDBMS, etc., and provide a common ORM, REST API and easy-to-use class SQL query syntax.

Customizable Monitoring Report: Provides an interactive, real-time visualization analysis of the Notebook class. It also allows you to further select parts of the icon and define the layout as a dashboard for sharing or continuous monitoring.

Eagle JPA Application: Real-time monitoring of Hadoop or Spark and other operating platform on the current and historical implementation of the state, to provide multi-dimensional performance analysis of different granularity to support a variety of abnormal early warning and performance warnings, such as job run time is too long, Tilt, and failure rate. It can effectively provide early warning and performance suggestions before SLA can not be met. Meanwhile, it can combine the machine learning model, coordinate forecasting task or server node based on task distribution or index change, and integrate Remediation system to automatically repair the system.

Eagle DAM Application: Real-time monitoring of user behavior, to ensure data security, support for HDFS, HIVE and other different data types, to provide simple and efficient data stream access Plugin, support simple rule definition syntax, combined with machine learning algorithm for user behavior modeling ), Automatic detection of abnormal user behavior, can be integrated Dataguides and other sensitive data


monitoring, can also be integrated Apache Ranger and other restrictions on abnormal user behavior.

Ambari Plugin : Eagle support, such as Ambari Plugin easy way to install and integrate into existing clusters, and provide a user-friendly user interface management.

“What is exciting is that the widespread deployment of Apache Eagle is accompanied by a growing number of new use cases and good community collaboration,” Edward Zhang added.

Apache Eagle is a highly scalable and technology platform that supports the growing demand for intelligent monitoring and alerting in large-scale distributed computing environments,” said Jiff Inc. CTO and executive vice president Debashis Saha. Project, I am proud to see the continued expansion of the community with Apache Eagle to support the complex and diverse use cases in security, infrastructure, network and distributed service monitoring. Congratulations to this team and community, It grew to be a top Apache project. ”

Dataguise is proud to be a member of the Eagle committer group, a leader in data-centric, cloud-focused and large-data technology security. DgSecure Monitor is sensitive sensing monitoring product that uses Apache Eagle as the core engine, Subra Ramesh, vice president of products and engineering for Dataguise Inc. said, “Apache Eagle’s flexible architecture, proven scalability and cutting-edge design make DgSecure Monitor a highly responsive and extensible solution for in-house and cloud deployments, a data security solutions company. We look forward to continuing to participate in Eagle, because it has now become a top-level Apache project. ”

Zhong Hao, senior vice president of technology at Site 1 said, “We’ve been using Apache Eagle for about a year and are excited to see it become a top-level project, and Apache Eagle and its low latency real-time alert engine help us easily identify security and performance issues on Hadoop, In addition, Eagle ‘s architecture is highly scalable, and we look forward to using it in a real – time risk management system”.

Chad Chun, director of eBay Analytics Data Infrastructure said, “Apache Eagle is a great monitoring and alerting solution designed for large distributed environments“, It was originally designed for security monitoring and quickly became a common solution that allowed Domain experts create their own monitoring applications on top of Eagle.This is a good design, easy to use the power of community to create and share applications, look forward to the adoption of the industry.

The Apache Eagle community has done a lot of work throughout the incubation process, and I’m pleased to see it graduate as a top-level project,” said P. Taylor Goetz, ASF member and member of the Apache Eagle Project Management Committee (PMC). Data deployment to provide first-class security and performance monitoring and alerts.Eagle project has established a strong sustainable community and demonstrated a firm understanding of the Apache Way, the Eagle community to achieve this important milestone At the same time, I look forward to further innovation.”

It’s great to see Eagle graduate to the top of the Apache project in a year,” said Seshu Adunuthula, senior director of eBay Data Platforms. “This is a great product with a unique position because it fills the large-scale distributed monitoring and alerting This is a well-architected product that allows communities to easily implement custom monitoring and alerting applications in different technology areas, such as network and database clusters, and I look forward to a rapid growth in the community over the next few years!”


Leave a comment

Your email address will not be published. Required fields are marked *