Delayed message sending

Messages and other background jobs have now caught up. We're sorry for the delay in these messages being sent out. We have included the technical explanation below for those that are interested, but if you have any queries please contact support.

At 5am BST this morning the crowdEngage database server automatically applied updates during its regularly scheduled maintenance window. During maintenance (which usually lasts for just a couple of seconds), the crowdEngage application is unable to connect to the database and goes into maintenance mode. As soon as scheduled maintenance finishes, the application reconnects and becomes available.

Today the main user-facing parts of crowdEngage behaved as expected and resumed service immediately after a few seconds. However unfortunately the system for processing background jobs didn't reconnect to the database as expected. Message sending (as well as other background tasks) happens on this system and so because it couldn't connect to the database these background jobs failed to complete.

crowdEngage is designed to catch up on delayed jobs, so once we were able to force the background jobs system to reconnect to the database all of the delayed messages were sent out straight away. At this stage the reason for the background jobs system failing to reconnect is unclear, but we will continue to investigate to find the root cause and implement additional monitoring so that we're alerted sooner if this does happen again.

Thank you for bearing with us, and sorry again for any disruption to your day.

We have found the problem and messages are now catching up.

We'll keep you updated here as we work out what is causing the problem.