Email Delays in EMEA Region
Incident Report for Symantec Email Security
Resolved
The previous reported concern with email delays in the EMEA region has been resolved. The service is fully restored and there is no longer an impact to customers. This incident is resolved.
Posted Jun 29, 2020 - 20:26 UTC
Update
The service remains fully operational and customers should continue to see mail flowing normally.

We will continue to closely monitor the email service and will provide updates as new information becomes available.
Posted Jun 22, 2020 - 16:08 UTC
Update
Changes have been made and we are monitoring the results. The email service remains fully operational.

We will continue to closely monitor the email service and will provide updates as new information becomes available.
Posted Jun 21, 2020 - 17:39 UTC
Update
Our engineering team has recommended some configuration changes that are being reviewed and implemented. In addition, our network team has scheduled another routing change on June 21, 2020. Once this change is completed, we will be monitoring the performance results. See the following link for more details:

https://email-security.status.broadcom.com/incidents/t1jrbwjdyld2

We will continue to closely monitor the email service throughout the weekend and will provide updates as new information becomes available.
Posted Jun 19, 2020 - 20:08 UTC
Update
As of 06:00 UTC, email queues on the Broadcom infrastructure are back to normal levels. Customers sending and receiving emails using EMEA based infrastructure will no longer experience email delays.

Any remaining emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

We are closely monitoring the email service and will provide the next update at 20:00 UTC, or as new information becomes available.
Posted Jun 19, 2020 - 09:01 UTC
Update
Our network teams' analysis of the information gathered during the testing continues. Customers connecting to the EMEA infrastructure should no longer experience delays while sending or receiving emails. Any remaining emails queued on third party or customer MTAs will be delivered according to the MTAs retry schedule.

We continue our investigation of this issue and will provide the next update as new information becomes available.
Posted Jun 19, 2020 - 06:38 UTC
Update
As of 03:00 UTC, we have made a network routing change and are currently monitoring the results. Customers connecting to the EMEA infrastructure should no longer experience delays while sending or receiving emails. Any remaining emails queued on third party or customer MTAs will be delivered according to the MTAs retry schedule.

We continue our investigation of this issue and will provide the next update as new information becomes available.
Posted Jun 19, 2020 - 03:59 UTC
Update
The outputs of the tests have been provided to the networks team and they are currently investigating further. Currently customers connecting to the EMEA region should not be experiencing delays while sending or receiving emails. Any emails queued on third party or customer MTAs will be delivered according to the MTAs retry schedule.

We continue our investigation of this issue and will provide the next update at 04:00 UTC, or as new information becomes available.
Posted Jun 19, 2020 - 02:12 UTC
Update
We are currently running a series of tests with additional debugging to assist in determining the cause of this issue. Customers may continue to see intermittent delays while sending or receiving emails. Any emails queued on third party or customer MTAs will be delivered once the issue is resolved.

We continue our investigation of this issue and will provide the next update at 02:30 UTC, or as new information becomes available.
Posted Jun 19, 2020 - 01:13 UTC
Update
We are aware of customers provisioned on the EMEA based infrastructure experiencing intermittent delays sending or receiving emails. In addition, a small subset of customers may have received errors when connecting to the impacted infrastructure resulting in emails queuing on customer and/or third party MTAs. Any emails queued on third party or customer MTAs will be delivered once the issue is resolved.

We are currently investigating this issue and will provide the next update at 01:30 UTC or as new information becomes available.
Posted Jun 19, 2020 - 00:17 UTC
Update
On June 18, 2020, between 05:00 UTC to 06:00 UTC, customers provisioned on the EMEA based infrastructure may have experienced intermittent delays sending or receiving emails. In addition, a small subset of customers may have received errors when connecting to the impacted infrastructure resulting in emails queuing on customer and/or third party MTAs. This issue only impacted EU-Central infrastructure and the EU-West infrastructure remained operational and processed emails without any issue.

As of 06:00 UTC, the issue is resolved and any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules. We are closely monitoring the email service and will provide updates as soon as more information becomes available.
Posted Jun 18, 2020 - 06:27 UTC
Update
On June 17, 2020, between 00:00 UTC to 03:00 UTC and 05:00 to 05:50 UTC, customers provisioned on the EMEA based infrastructure may have experienced intermittent delays sending or receiving emails. In addition, a small subset of customers may have received errors when connecting to the impacted infrastructure resulting in emails queuing on customer and/or third party MTAs.

As of 05:50 UTC, the issue is resolved and any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

We are closely monitoring the email service and will provide updates as soon as more information becomes available.
Posted Jun 17, 2020 - 12:25 UTC
Update
Email queues on the Broadcom infrastructure are back to normal levels. Customers sending and receiving emails using EMEA based infrastructure will no longer experience email delays.

Any remaining emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

We are closely monitoring the email service and will provide updates as soon as more information becomes available.
Posted Jun 16, 2020 - 15:45 UTC
Update
The majority of existing queued emails on Broadcom infrastructure have successfully delivered. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

We are closely monitoring the email service will provide further updates as more information becomes available.
Posted Jun 16, 2020 - 12:26 UTC
Monitoring
As of 10:55 UTC, we have applied mitigation to address the underlying cause. Emails previously queued have started delivering. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

We will provide an update at 12:30 UTC, or as soon as more information becomes available.
Posted Jun 16, 2020 - 11:10 UTC
Update
We are handling this issue as a high priority and have engaged all appropriate teams at Broadcom. Any emails queued on our infrastructure, customer and/or third party MTAs will be delivered once service is restored.

Please note that there is no impact to protection services, and all other aspects of our services remain functional. We will provide our next update at 11:30 UTC, or as soon as more information becomes available.
Posted Jun 16, 2020 - 10:32 UTC
Investigating
We are aware of customers provisioned on the EMEA based infrastructure experiencing delays sending or receiving emails. In addition to email delays, a subset of customers may have received errors when connecting to the impacted infrastructure resulting in emails queuing on customer and/or third party MTAs.

We are currently investigating this issue and will provide the next update at 11:00 UTC or as new information becomes available.
Posted Jun 16, 2020 - 09:54 UTC
This incident affected: Email Security.cloud (Europe, Middle East & Africa).