In 2018, I worked on a real-time analytics project.
Our customers were operations and sales teams. They wanted to be alerted if something went wrong.
"Sales are down by 20%!" "Our rating has gone down by 1 point!"
Our team was tasked with detecting and reporting anomalies.
The problem, simplified!
Speaking to the operations teams, we noted two important factors:
- Helpfulness
- Explain what went wrong. Where do I look it up, and who is the next person to contact?
- If multiple metrics are off, don't bombard me with multiple related alerts.
- Accuracy
- For the love of God, don't report false negatives.
- Unless you are sure, don't put artificial intelligence into it.
To start, we needed a suitable database. I started researching popular time-series databases, and one whitepaper stood out.
It was Facebook's GorillaDB: a time-series database that can monitor and analyze business metrics.
What struck me was the simplicity of their approach: Facebook would only store the last 26 hours of data in memory. The rest would be persisted on logs, and later pumped to OpenTSDB.
Since we had a real-time requirement, this paper heavily influenced the system we built.
What did we do?
We built a time series analyzer that consumed metrics from various upstream services (product, sales, marketing, operations, etc...) and looked for anomalies using a combination of ARIMA and Holt-Winters algorithms.
We would then report any detected anomalies along with "dependent" metrics. For example, if sales were dependent on website visits, we would mention that in the report. This helped teams track the relevant metrics together.
Metrics were classified as "dependent" and independent based on their Spearman's coefficient. You can think of the independent variables as stuff that decides the behavior of a system. For example, pricing is an independent variable decided by the business.
This significantly reduced the cognitive load on the operations teams. They would now focus on the independent variables only, trying to change what they had control over instead of fussing over dependent metrics.
What was the end result?
It didn't work out.
Our analytics engine had other, more obvious issues (showing incorrect data and frequent crashes).
In time, another team built a similar product with a stronger user base, effectively killing our project.
My learnings from this project were immense though. I also built an appreciation for time-series databases, two of which we cover at InterviewReady:
- Facebook Gorilla
- Google Monarch
Through a series of videos, we discuss the strengths and tradeoffs of these two systems, by diving into the whitepapers of GorillaDB and Monarch. Cheers!