.png)
In March 2016, I was a systems engineer working on a phone calling service similar to WhatsApp calling. Our app was popular among users in the USA who used it to call their families in India. We had an innovative calling strategy that combined VOIP and PSTN to reduce international calling costs by about 90%.
One day, a celebrity tweeted about our app, and our metrics shot up immediately. The product and marketing teams were ecstatic, but panic set in among the engineering team. The gateway was over 93% load, exceptions were tearing the server apart, and we deployed more servers to try to handle the increase in traffic.
However, we encountered a major problem. One service refused to improve, and every other request was dying without a response. Our calling service provider, which we had leased bandwidth from, was the single point of failure in our case, as the bandwidth was too thin to accommodate the massive wave of requests.
Despite our hotfixes, unhappy customers continued to increase, and word spread that our app took users' money without providing calling services. We had to shut down operations and apologize to all our users, with only 10% of them being served.
The root-cause analysis identified over 20 reasons for the failure, but the most important one was dynamic rate limiting. We learned a valuable lesson that when you receive more requests than you can handle, you shouldn't accept more than you should, rather than not accepting more than you can. Setting rates dynamically, propagating acceptance and failure rates across a distributed system, and managing peak load are challenges that all systems face, especially during peak periods like Black Friday, New Year's, Diwali, and Holi.
As a software engineer, I learned from this failure and recognized that it's our job to ensure that systems work and thrive under fire. For more detailed learnings, check out my video on distributed rate limiting in the system design course at InterviewReady.
Video Link: https://interviewready.io/learn/system-design-course/distributed_rate_limiting
Finally, I hope you never encounter a rate-limiting problem that you can't solve in time. Dynamic rate limiting is crucial to ensuring your systems work and thrive under fire, and it's essential to have the right tools and strategies in place to manage peak traffic. Don't let a systems failure like ours happen to you. Learn from our mistakes and be prepared for anything.