(June 6, 2023 8:30am EST): We have had recent reports of an issue where emailed reports are not sending all the notifications and only a few are coming through. We are actively looking into this and greatly appreciate your patience and understanding during this time. If you are experiencing this issue, please send an email to support@zerionsoftware.com with details such as Profile ID, Form Name, Email Address and the time-frame at which you noticed this behavior.

UPDATE (June 6, 2023 3:30pm EST): After being made aware that the email functionality has been disrupted, the Zerion team is actively working to correct all of the email related functions. We greatly appreciate your patience and understanding during this time.

UPDATE (June 6, 2023 11:15pm EST): We have deployed a fix for the record email notifications. Our team is actively validating other email scenarios to ensure the system operates as expected. We will provide a more detailed summary after the team can review all the findings.

UPDATE (June 8, 2023 12:30pm EST): We can confirm that password reset email and trial activation emails are still not sending at this time. If you are in need of resetting your password, please reach out to your Company or Server Admin to assist with resetting it manually on the backend. For trial activations, please reach out to sales@zerionsoftware.com. The team is continuing to work on getting the rest of the email functionalities fixed but do not currently have an ETA.

UPDATE (June 13, 2023 2:30pm EST): We sent out an emailed notice to the account champions on file about the incident that took place last week. This is our message:

Last week, many of our customers were affected by a disruption in our email infrastructure including record submission emails and password reset requests. After identifying the root cause and resolving the issue, kindly refer to the timeline of events and FAQ below.

Timeline of Events

Monday June 5th, our email host Sendgrid, upgraded their technology which resulted in email requests (record emails, password resets, package import notifications, etc) failing.

Tuesday June 6th, Zerion help desk requests indicated there was a broad problem with our email functions, and the team started to investigate with the help of Sendgrid. We confirmed the problem was related to a recent Sendgrid technology upgrade late Monday. Around 11:30 PM EST  we deployed a temporary update to correct the record email reports. After the deployment Tuesday night, our engineers reviewed the remaining functions to address, and came up with several options to resolve the disruption.
 
Wednesday June 7th, our engineers and infrastructure members reviewed the update from Tuesday and determined a new solution was needed for the remaining email functions. Four options were explored and we picked the most viable option to address all of the functions in a similar fashion.
 
Thursday June 8th, we perform the first round of QA testing on the proposed solution, and found no major flaws. Thursday evening we advanced the code to our official QA environment for more thorough testing.
 
Friday June 9th around 6PM EST, another updated was deployed that corrected the remaining email functions (password resets, package import notifications, etc).
 

​​​​​Q: Is there a way to get the emails that didn’t send out last week?
A: You can use the trigger post process feature in the admin interface, or take advantage of trigger post APIs for each record (articles below).

Trigger Post Process
Trigger Post via API6
Trigger Post via API8
 
Q: Can I increase the limit of post processing requests to trigger all my backlog?
A: Yes you can. Please file a help desk request so we can effectively track the requests and provide the best turn around.
 
Q: What are you doing to prevent a similar disruption in the future?
A: We are improving the way we monitor key suppliers for service disruptions and breaking changes through wider distribution of information and improved monitoring.

Q: What specific email functions were impacted by this outage?
A: All email functions were impacted, except post notification delays and failure emails when using guaranteed delivery.

Q: Would this disruption affect me if I was using my own email host?
A: If you are using your own email settings within your environment, this would not apply to you.

Q: Will the update change the email behavior?
A: Email behavior will remain unchanged, and you should have the same expectations as before regarding any email based function, even with the new approach.

We greatly apologize for the inconvenience that this may have caused you and your team. If you should have any follow-up questions, please don't hesitate to contact me directly. 

All the best,
Tony Ruth

truth@zerionsoftware.com