Back to overview
Downtime

Keka app is down

Sep 09 at 09:13am IST
Affected services
Login
Core HR
Mobile API
Core HR
Mobile API
Core HR
Mobile API

Resolved
Sep 09 at 10:13am IST

  • This issue is resolved and all affected services have been reinstated. We are closely monitoring the application as it gains stability. We apologise for the inconvenience caused during this period of application outage.
  • We are closely working with the related pods to plan for the next steps and how similar occurrences can be prevented in the future.
  • A detailed RCA will be shared with all affected customers by end of day today.

Updated
Sep 09 at 09:15am IST

  • Our team has identified that the database utilization has spiked due to a long running operation related to inbox items retrieval.
  • The reason for this occurrence was missing DB indexes which could have ensured faster data retrieval but due to a misconfiguration in a DB entity, this necessary step got missed and eventually resulted in over utilization of our database.
  • Since, Keka holds a centralized Auth server, the health checks for our main DB failed due to the long running DB operations resulting in login application collapse.
  • As all applications rely on login app for auth validation scoped to every request, users started to face accessibility issues across our application suite.
  • We are working proactively to reinstate the affected services and bring the complete application suite back online.

Created
Sep 09 at 09:13am IST

  • Our monitoring systems alerted us about a potential over utilization of database servers.
  • Along with the monitoring system alerts, we are also notified about the accessibility issues faced by the users across regions.
  • Upon addressing the customer concerns and monitoring alerts, our team started investigating the issue to identify the DB bottleneck which brought in the mentioned downtime.
  • We'll keep everyone posted with the necessary updates as soon as the reason for this downtime is identified.