Starting at approximately 20:40 UTC on July 18th, 2023, we began receiving reports of Marketplace Applications failing to deploy as expected.
In observing the low number of reports that were received during that period as well as a gap in the way that Marketplace Applications deployments are monitored, incident response procedures were not initiated until 09:42 UTC on July 20th. At this time, enough data had been gathered in order to escalate the situation, and a status page was launched to inform customers of the issue at 10:50 UTC.
At approximately 13:30 UTC, the root cause was identified as an error affecting a subset of Marketplace Applications. Due to missed dependencies in required packages for several of these scripts, these installations failed to execute as expected upon deployment.
At 14:11 UTC, an update to fix the broken dependencies in these applications was merged successfully, addressing the issue. New deployments of the impacted Marketplace Applications were tested and encountered no further problems. After a monitoring period, the incident was considered to be resolved at 17:05 UTC.
The teams involved in maintaining our Marketplace Applications are currently implementing changes to our existing update procedures for current and future offerings. This will provide new monitoring processes for these deployments, eliminate singular points of failure, and improve dependency checking to prevent this type of situation from reoccurring in the future.