Meta runs on over 100 million lines of code. Finding the root cause of an error is like searching for needles in a haystack.
To tackle this, Meta's engineering teams have built some of the World's most advanced debugging software.
These tools have evolved over a decade from simple alerts and retries, to complex AI models that can detect bugs and correct code!
To witness this evolution, we picked two systems from Meta Engineering: FBAR and HawkEye.
The former is from 2011, and the latter was written this year!
FBAR (Facebook Auto-Remediation)
The system is (conceptually) simple:
- A monitoring system detects failed jobs.
- FBAR continuously queries the monitoring system for alerts.
- It processes the alerts and calculates appropriate workflows for remediation.
Figure 1. The FBAR Repair Flow
The FBAR system is estimated to do the work of 200 system admins but is maintained by only two full-time engineers! It provided the base for future upgrades, optimizations, and evolution!
HawkEye (AI Debugging at Meta)
HawkEye is a toolkit used internally at Meta for monitoring, observability, and debugging of ML workflows.
Identifying the root cause of a machine learning model requires going through the model hyperparameters and system configuration.
Before HawkEye, debugging required specialized knowledge and coordination across different teams.
Figure 1. The HawkEye AI-Powered Repair Flow
HawkEye implements a decision tree (shown above) to streamline the debugging process and reduce the time spent on complex production issues.
Here is the basic algorithm:
- Workflows begin with alerts triggered by monitoring systems.
- HackEye identifies models causing degraded metrics. It does this by correlating derived metrics with top-line metrics.
- For example, the root cause of reduced sales is reduced registrations.
- The root cause of reduced registrations is reduced page views.
- Hawkeye finds multiple top-line metrics contributing to a derived metric, finds upstream systems causing the problem, and then reports to on-call engineers.
- HawkEye compares the current snapshot of a model with previous operationally stable snapshots to identify issues.
- Pray that we solve it in time!
As you can see, this algorithm is more involved and complex to implement. Each of these steps is a complex piece of engineering in itself.
HawkEye is a testament to the engineering skill and experience of Meta.
Conclusion
Meta is building on the shoulders of their home-grown engineering giants.
Their "practical" approach to problem-solving first looks for automation, followed by optimizations, eventually turning into major upgrades and full-fledged systems.
While the improvements are incremental every year, looking back at the initial systems shows how far they have come in 10 years.
And, the evolution continues. Automation for the win!
To learn more about Observability and Anomaly Detection, go to Observability in Distributed Systems.
Recommended Reads: