Aug. 17, 2023 (5:45 am EST): We were made aware of the situation where you may not be getting your post endpoints triggered automatically. We can confirm there is an issue with post-processing - the messages are not being processed but we have them. If there are higher value endpoints that you need to get moving again, you can disable guaranteed delivery as a temporary solution. Once the issue is resolved, the backlog will process on its own.

Aug. 17, 2023 (9:00 am EST): The messages are moving again and we are monitoring the systems closely. We increased the processing capacity to get through the backlog faster and get the messages where they need to go.

Aug. 17, 2023 (10:30 am EST): The backlog has been processed.

UPDATE (Aug. 25, 2023):

As a follow-up to the delay in post processing event last week, we wanted to provide additional insight below.

The system that sends the post-data requests entered a degraded state on 8-16-23, triggered monitoring alarms, and recovered several minutes later. Around two and a half hours later no requests were being processed, and no monitoring alarms went off.

The following morning 8-17-23 we responded to complaints from clients about the delay in processing and found the root cause for messages being delayed. After restarting the system, the requests started to get processed again.

Moving forward we are taking steps to reproduce the condition consistently and to perform automated recovery steps. This will not handle every possible fault scenario, but will apply some basic recovery actions whenever a service disruption is detected.

If you have any follow-up questions, please don’t hesitate to reach out.