Outage to IAM Cloud federated systems for EU based customers
Incident Report for IAM Cloud
Postmortem

IAM Cloud would like to thank all our EU customers for their patience this morning while we resolved unplanned disruption to the Authentication Service in the EU following major infrastructure maintenance this weekend. The scheduled maintenance this weekend was very complex and a major step forward in providing increased stability and efficiencies within the IAM Cloud Platform. All resources and planning was applied to ensure no disruption to our customers so we are extremely disappointed that any of our customers were affected today. We want to assure you that we take these incidents very seriously and have put clear steps in place to ensure we learn from this experience and further improve the IAM Cloud Service.

We hope the below RCA offers some insight and reassurances following this major incident.

Root Cause Analysis (RCA) Issued 10th July 2018

SUMMARY:

On Tuesday 10th July at 8:33 am BST our Infrastructure team were alerted of errors to the IAM Cloud Authentication Service which caused failed authentications for EU customers.

ROOT CAUSE:

The root cause of this issue was that following the scheduled maintenance this weekend the scaling algorithm failed to scale successfully meaning that the servers were unable to service the requests.

RESOLUTION

On Tuesday 10th July at 09:15am manual scaling was applied and existing services were re-cycled to clear the backlog of pending request. Full service was resumed at 09:40am.

MITIGATION

Engineers have fixed the scaling metrics and additional alerting and notifications on traffic build up have been applied.

Please do not hesitate to contact me direct if you would like to discuss this or any aspect of our service further.

Kind Regards Victoria Hamblin IAM Cloud's Chief Customer Officer

Posted Jul 10, 2018 - 10:53 UTC

Resolved
Access to federated services has now been restored. We will investigate the root cause of this and issue the RCA once we have collated the information.

If you still experience any authentication issues then please log a ticket with our support team and we'll get back to you ASAP.
Posted Jul 10, 2018 - 08:49 UTC
Investigating
Our engineers are currently investigating the outage to federated IAM Cloud systems for our EU based customers. Further updates to follow
Posted Jul 10, 2018 - 08:08 UTC
This incident affected: Authentication Service (Single Sign On, Smartlinks) and Cloud Drive Mapper, IAM Cloud Portal.