Maintenance in Core Services

Incident Report for Care Hires

Postmortem

Post-Mortem Report on Recent Service Interruption - Incident Resolved

We are pleased to inform you that the recent service interruption has been successfully resolved. This post-mortem report is intended to provide a non-technical overview of the incident, the actions taken, and our commitment to preventing such occurrences in the future.

Incident Overview:

What Happened: We experienced a disruption in service due to an issue with traffic routing to cloud infrastructure
Duration: The issue persisted as an intermittent between 9am-10am during which our services were not operating optimally.

Response and Resolution:

Immediate Action: Our technical team members worked diligently to identify and resolve the issue.
Resolution: We implemented a fix that corrected the traffic routing, restoring our services to full functionality.

Impact:

The service interruption may have caused temporary inconvenience and disruption. We want to assure you that no data was compromised or lost during this period or due to this action.

Preventive Measures:

Review and Analysis: We are conducting a thorough analysis of this incident to understand the underlying causes.
System Improvements: Based on our findings, we will be making necessary improvements to our systems and processes.
Ongoing Monitoring: Enhanced monitoring protocols are being put in place to detect and prevent similar issues in the future.

Our Commitment: We are deeply committed to providing reliable and secure services. This incident, while unfortunate, has presented us with a valuable opportunity to enhance and further strengthen our processes, including upgrading to the latest versions. We are taking all necessary steps to enhance our systems and prevent such incidents from occurring again.

Thank You: We appreciate your patience and understanding during this time and thank you for your continued trust in our services.

Posted Jan 19, 2024 - 12:16 UTC

Resolved

We are pleased to report that the recent service interruption, caused by a traffic routing issue in our cloud infrastructure, has been successfully resolved. Our technical team worked promptly to identify and rectify the problem, restoring full functionality to our services.

To provide a brief overview:

- Issue Encountered: A disruption in traffic routing to our cluster, which impacted service delivery.
- Resolution: The technical team implemented a fix that corrected the traffic flow, ensuring the smooth operation of our services.

In our ongoing commitment to service excellence, we will continue to closely monitor our systems.

We understand the importance of reliable service and apologise for any inconvenience caused during this period. Your patience and understanding are greatly appreciated.

Posted Jan 19, 2024 - 12:07 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jan 19, 2024 - 11:57 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jan 19, 2024 - 11:57 UTC

Identified

We have pinpointed several issues in our cloud infrastructure, which appear to be a direct result of the recent security upgrades implemented to enhance service security. These upgrades have led to a package version inconsistency, identified as the root cause of the current problem.

Our team is diligently working to resolve these inconsistencies and restore normal service functionality as soon as possible. We are prioritising this task to ensure minimal disruption and a swift return to operational stability.

Next Steps:
1 - Continuous monitoring of the system for any further anomalies post-fix.
2 - Implementing additional checks to prevent similar issues in future upgrades.

Posted Jan 19, 2024 - 09:30 UTC

Investigating

We are currently experiencing a downtime in our core services. This is a critical situation, and our technical team is actively working to resolve it as swiftly as possible.

Posted Jan 19, 2024 - 08:56 UTC

This incident affected: API, Care Hires - Web (Care Hires - Neutral Vendor, Care Hires - Care ROTA SaaS Application), and Care Worker - Mobile (Care Hires - Worker Mobile Application (IOS), Care Hires - Worker Mobile Application (Android)).