Messaging Security

Service availability and scheduled maintenance for Email Security.cloud and Messaging Gateway.

Connectivity Issues in NAM Region
Incident Report for Messaging Security
Resolved
We have monitored the service and have not seen a re-occurrence of the email delivery issues. This incident is resolved.
Posted Sep 01, 2021 - 12:22 UTC
Monitoring
Email queues on the Broadcom infrastructure are back to normal levels. Customers sending and receiving emails using NAM based infrastructure will no longer experience email delays.

Any remaining emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

We are closely monitoring the email service and will provide update as soon as more information becomes available.
Posted Aug 31, 2021 - 23:06 UTC
Update
The majority of US West queues have processed. Remaining queues are expected to clear within an hour. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

A subset of customers may continue to experience delays while the remaining US West backlog is processed. We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 22:02 UTC
Update
US East mail queues have all cleared and the majority of US West queues are processed. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

A subset of customers may continue to experience delays while the remaining US West backlog is processed. We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 21:13 UTC
Update
Previously queued email on our NAM infrastructure continues to deliver. US East mail queues have cleared while US West is actively processing. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

A subset of customers may continue to experience delays as backlog is processed. We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 20:16 UTC
Update
The majority of the mail servers in US East and US West infrastructure have been placed back in service. We are performing manual load balancing for some servers to decrease the load. In addition, we have added additional compute capacity to help with the queued emails. Previously queued email on our NAM infrastructure continues to deliver. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules.

Customers will continue to experience delays as backlog is processed. We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 19:14 UTC
Update
We are continuing to work on this issue with the highest priority.

US East Infrastructure:
We have taken some overloaded servers out of service to stop new emails from being queued. We are in the process of adding compute capacity to help with backlog processing. US East infrastructure continues to process customer traffic but at a very slow rate.

US West Infrastructure:
US West infrastructure is now back in service and starting to process customer email traffic. Similar to the US East infrastructure, we are expecting high load to build up on these servers. As a proactive measure, we are adding more capacity to help with email processing.

Customers will continue to experience delays as emails are being processed. We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 18:21 UTC
Update
We are continuing to work on this issue with the highest priority.

US East Infrastructure:
US East infrastructure is processing customer traffic but at a very slow rate. We are investigating this issue further.

US West Infrastructure:
We have remediated the issue and are in the process of putting the site back in service. While US West is out of service, all customer traffic is being sent to the US East infrastructure.

Customers will continue to experience delays as emails are being processed. We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 17:51 UTC
Identified
We have applied mitigation to address the issue with US East infrastructure. Customers will no longer experience errors while connecting to the US East infrastructure. Any emails queued on third party or customer MTAs will start delivering based on their respective retry schedules. There are some delays expected while queued email is processed.

We continue to work on restoring US West infrastructure.

We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 17:00 UTC
Update
We are continuing to investigate and work towards identifying the underlying cause. In addition to Email Security.Cloud, this issue is also impacting DLP Cloud Service for Email and CloudSOC CASB customers.

We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 16:32 UTC
Update
We are continuing to investigate this issue with the highest priority. We are currently checking network routes and have engaged third-party vendors.

We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 15:53 UTC
Update
We are continuing to investigate this issue with the highest priority and have engaged all stakeholders from both product and infrastructure teams.

Customers may continue to experience connectivity errors and queued email. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules once the connectivity issue is resolved.

We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 15:26 UTC
Update
We are continuing to investigate this issue. Customers may continue to experience connectivity errors and queued email. Any emails queued on third party or customer MTAs will be delivered based on their respective retry schedules once the connectivity issue is resolved.

We will provide an update as soon as more information becomes available.
Posted Aug 31, 2021 - 15:02 UTC
Update
We are investigating connectivity issues in the NAM region. Customers may not be able to connect to NAM infrastructure and mails will queue on sending servers. We are investigating this issue as a priority.

Please note that there is no impact to protection services, and all other aspects of our services remain functional.

We will provide an update as soon as more information becomes available
Posted Aug 31, 2021 - 14:43 UTC
Investigating
We are aware of a subset customers provisioned on our NAM infrastructure experiencing delays sending and receiving emails. We are investigating this issue as a priority.

Please note that there is no impact to protection services, and all other aspects of our services remain functional.

We will provide an update as soon as more information becomes available
Posted Aug 31, 2021 - 11:27 UTC
This incident affected: Email Security.cloud (Americas, Asia Pacific, Japan).