tag:apify.statuspage.io,2005:/historyApify Status - Incident History2024-03-29T01:06:06+01:00Apifytag:apify.statuspage.io,2005:Incident/202632162024-03-16T09:09:06+01:002024-03-16T09:09:06+01:00Some background operations might be delayed<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>09:09</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:55</var> CET</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>12:55</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:10</var> CET</small><br><strong>Identified</strong> - Operations like publishing Actors, sending notifications, or adding new organization members might be delayed.<br />We have identified the cause and are preparing a fix.</p>tag:apify.statuspage.io,2005:Incident/202148612024-03-11T16:45:09+01:002024-03-11T16:45:09+01:00Partial outage<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>16:45</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved. There was no data loss as requests are being retried, but some of the Actor runs may have failed. You can resurrect the failed runs (https://docs.apify.com/platform/actors/running/runs-and-builds#resurrection-of-finished-run) to continue running.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>15:12</var> CET</small><br><strong>Update</strong> - While the proxy was the main affected component, the problem was related to our IAM service and other components like API, Actors or webhooks were affected as well.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>14:08</var> CET</small><br><strong>Monitoring</strong> - We have experienced an outage of the service managing IAM from 12:50 to 13:00 UTC. This was due to a misconfiguration while releasing a new version. We have reverted the release and everything is operational again.</p>tag:apify.statuspage.io,2005:Incident/198779292024-02-01T09:11:00+01:002024-02-01T09:11:00+01:00API performance degraded and outage of Actors<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>09:11</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>19:50</var> CET</small><br><strong>Monitoring</strong> - We have experienced two restarts of our servers that run Actors. It was caused by performance degradation of our API. There was no data loss. Some Actors may have failed and you can resurrect them to make them work again. We isolated the root cause and we're implementing measures to prevent this from happening in the future.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>19:09</var> CET</small><br><strong>Update</strong> - We are currently investigating degraded API performance and outage of Actors</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>19:06</var> CET</small><br><strong>Investigating</strong> - We are currently investigating degraded API performance</p>tag:apify.statuspage.io,2005:Incident/198648522024-01-30T10:53:21+01:002024-01-30T10:53:21+01:00API performance degraded<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:53</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:57</var> CET</small><br><strong>Monitoring</strong> - The issue was resolved, and only a single customer was affected. We are monitoring the situation.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:36</var> CET</small><br><strong>Investigating</strong> - We are experiencing a degraded performance of Apify API and are investigating the root cause. Only the performance (response time) is impacted, with no impact on data quality or anything else.</p>tag:apify.statuspage.io,2005:Incident/186359602023-09-29T10:12:13+02:002023-09-29T10:12:13+02:00Actor builds might fail and the overall performance might be degraded<p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:12</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>29</var>, <var data-var='time'>10:11</var> CEST</small><br><strong>Update</strong> - The health of upstream dependencies has been resolved and we are in turn no longer experiencing degraded performance.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>22:34</var> CEST</small><br><strong>Investigating</strong> - AWS is currently experiencing degraded performance and a higher error rate (https://health.aws.amazon.com/health/status). This has a cascading effect on different external systems, such as DockerHub, affecting in turn us as well. Most notably, we're experiencing a higher error rate on Actor builds. We will continue to monitor the health of upstream services and of our platform to mitigate any potential issues.</p>tag:apify.statuspage.io,2005:Incident/179562292023-07-25T04:00:00+02:002023-07-25T16:53:41+02:00Degraded stability of static proxy groups<p><small>Jul <var data-var='date'>25</var>, <var data-var='time'>04:00</var> CEST</small><br><strong>Resolved</strong> - We experienced a short incident during which the stability of Apify's static proxy groups (StaticUS1 and StaticUS3) was degraded. Default data center proxy groups were not affected by this incident. <br /><br />Problems were resolved in about 30 minutes. We are monitoring the situation to make sure that the situation is permanently resolved.</p>tag:apify.statuspage.io,2005:Incident/175630512023-06-14T08:52:10+02:002023-06-14T08:52:10+02:00Increased error rate and latency across all services<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>08:52</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>23:32</var> CEST</small><br><strong>Monitoring</strong> - The problem was fixed on the AWS cloud side. Everything is back to a normal state, and elevated error rates are resolved. <br /><br />The whole Apify platform stayed operational throughout the incident, but the elevated error rates of our API caused a performance drop. This may have caused some Actor runs to fail. In such cases, you can resurrect them (https://docs.apify.com/platform/actors/running/runs-and-builds#resurrection-of-finished-run), and the runs will continue from the point of failure.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:34</var> CEST</small><br><strong>Update</strong> - As the cloud provider has issues with identity access management, the error rate of all operations is higher than usual, resulting in Actors' degraded performance.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:21</var> CEST</small><br><strong>Identified</strong> - Due to an issue at our cloud service provider, most of our services are experiencing increased error rates and latencies. We are investigating the issue and will attempt to mitigate the impact.</p>tag:apify.statuspage.io,2005:Incident/173400512023-05-23T13:41:29+02:002023-05-23T13:41:29+02:00Platform is fully operational, status page is down<p><small>May <var data-var='date'>23</var>, <var data-var='time'>13:41</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>13:36</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>23</var>, <var data-var='time'>13:35</var> CEST</small><br><strong>Identified</strong> - We are only experiencing problems with the Status Page itself, so everything is displayed as down. But the Apify platform including all the components is fully operational.</p>tag:apify.statuspage.io,2005:Incident/171910722023-05-09T17:22:34+02:002023-05-09T17:22:34+02:00Residential proxy service down<p><small>May <var data-var='date'> 9</var>, <var data-var='time'>17:22</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 9</var>, <var data-var='time'>11:56</var> CEST</small><br><strong>Identified</strong> - Apify Residential Proxy service is down for many different countries. Problems started at about 9:45 UTC and are caused by one of our providers, traffic is being re-routed alternatively way and the service should recover in a couple of minutes.</p>tag:apify.statuspage.io,2005:Incident/169317732023-04-21T19:28:24+02:002023-04-21T19:28:24+02:00Service disruption<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>19:28</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>18:02</var> CEST</small><br><strong>Monitoring</strong> - A short service interruption (around 10 minutes) affected our services. We have identified the cause and resolved the issue. Actors continued running and there was no data loss. All services should be functional again. We are monitoring the situation and apologize for any inconvenience caused.</p>tag:apify.statuspage.io,2005:Incident/169134842023-04-20T08:26:42+02:002023-04-20T08:26:42+02:00API performance degraded<p><small>Apr <var data-var='date'>20</var>, <var data-var='time'>08:26</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>22:15</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>19</var>, <var data-var='time'>20:15</var> CEST</small><br><strong>Identified</strong> - We are experiencing lowered API performance. Problem is identified, and we are working toward a resolution.</p>tag:apify.statuspage.io,2005:Incident/162219822023-02-21T15:06:00+01:002023-02-21T15:38:08+01:00Outage of the French residential proxy<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>15:06</var> CET</small><br><strong>Resolved</strong> - There was a short outage of the residential proxy with the country selection France. The problems lasted for about 15 minutes. No other countries were affected. The problem is fixed, and we are monitoring the state.</p>tag:apify.statuspage.io,2005:Incident/159010902023-01-21T22:14:26+01:002023-01-21T22:14:26+01:00Residential proxy with "country=us" down<p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>22:14</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>21:26</var> CET</small><br><strong>Monitoring</strong> - The problem with the US region is fixed. We are monitoring the situation to make sure that the issue won't re-appear.</p><p><small>Jan <var data-var='date'>21</var>, <var data-var='time'>20:07</var> CET</small><br><strong>Investigating</strong> - We are experiencing partial downtime of our residential proxy service. Only the requests with the country set to "us" are affected. We are identifying the problem.</p>tag:apify.statuspage.io,2005:Incident/157714882023-01-04T15:30:00+01:002023-01-04T16:07:32+01:00Partial outage of actor runtime<p><small>Jan <var data-var='date'> 4</var>, <var data-var='time'>15:30</var> CET</small><br><strong>Resolved</strong> - Small percentage of actor runs could not be started and new actors could not be created from templates.<br />We have identified the issue and deployed a fix. This incident lasted 17 minutes.<br />There was no data loss. All the actors may have been restarted. Actors already running were not affected.</p>tag:apify.statuspage.io,2005:Incident/143878282022-12-05T17:16:33+01:002022-12-05T17:16:33+01:00Residential proxy outage<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>17:16</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>07:43</var> CET</small><br><strong>Update</strong> - We are sorry for the inconvenience caused. Problems were happening between 00:00 UTC and 6:30 UTC, causing residential proxy workloads to fail. We will be launching changes to our residential proxy service that will mitigate similar issues in the future.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>07:37</var> CET</small><br><strong>Monitoring</strong> - A fix has been implemented, and the residential proxy is back up. We will keep monitoring the system. Other proxy types were not affected.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>07:29</var> CET</small><br><strong>Identified</strong> - Apify residential proxy service is experiencing an outage due to a problem with one of our providers. We are working on a recovery.</p>tag:apify.statuspage.io,2005:Incident/136351802022-11-22T15:55:00+01:002022-11-22T16:03:21+01:00Actor runtime unavailable for about 5 minutes<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:55</var> CET</small><br><strong>Resolved</strong> - Due to an abnormally slow switch between the primary and replica of our database clusters, the worker machines that the host actor runs got restarted. The accumulated workload caused some runs to be delayed by about 3-5 minutes. Runs continued normally with no data loss but runs with a low timeout may have timed out.</p>tag:apify.statuspage.io,2005:Incident/127533552022-11-08T15:02:35+01:002022-11-08T15:02:35+01:00API performance degraded<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>15:02</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:51</var> CET</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:51</var> CET</small><br><strong>Monitoring</strong> - The performance has been restored and is under monitoring.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>12:43</var> CET</small><br><strong>Identified</strong> - The API of the performance is degraded. The root cause has been identified and we are currently working on a fix. The platform remains fully operational.</p>tag:apify.statuspage.io,2005:Incident/126340842022-10-27T13:00:00+02:002022-10-27T13:31:07+02:00Performance degraded<p><small>Oct <var data-var='date'>27</var>, <var data-var='time'>13:00</var> CEST</small><br><strong>Resolved</strong> - Due to an increased load and delayed scaling the performance of some services was degraded for short periods of time during the past 30 minutes <br /><br />This mainly affects the API response time and so the speed of Apify actors. There was no data loss or service disruption, but the overall performance was degraded at different levels.<br /><br />The underlying issue has been addressed, and the performance has been restored. We are implementing measures so that this won't occur again.</p>tag:apify.statuspage.io,2005:Incident/118945802022-10-12T17:51:13+02:002022-10-12T17:51:13+02:00Residential proxies outage<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>17:51</var> CEST</small><br><strong>Resolved</strong> - Issue has been resolved. We are monitoring the situation to beware of further disruptions.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>17:16</var> CEST</small><br><strong>Identified</strong> - We are experiencing an outage of residential proxies caused by a downtime of one of our providers. We are monitoring the situation and working with our partner on recovery.</p>tag:apify.statuspage.io,2005:Incident/112207872022-09-22T03:30:00+02:002022-09-22T15:47:28+02:00API performance temporarily degraded<p><small>Sep <var data-var='date'>22</var>, <var data-var='time'>03:30</var> CEST</small><br><strong>Resolved</strong> - We experienced a temporary degradation of the API performance while the rest of the system was operating normally.</p>tag:apify.statuspage.io,2005:Incident/111736852022-09-19T10:16:11+02:002022-09-19T10:16:11+02:00Partial proxy outage<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>10:16</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>02:41</var> CEST</small><br><strong>Monitoring</strong> - We have identified the issue and implemented a fix. All datacenter proxy groups should be back to usual performance. We will continue monitoring the proxy status.<br /><br />If your actor uses the "auto" proxy configuration, it could have experienced reduced performance due to higher error rate. If it uses a BUYPROXIES proxy group directly, it could have failed connecting to the proxy server.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>02:16</var> CEST</small><br><strong>Identified</strong> - Some datacenter proxy groups of Apify Proxy are having an outage. We have identified the issue and we are implementing a fix.</p>tag:apify.statuspage.io,2005:Incident/103144992022-06-16T08:07:37+02:002022-06-16T08:07:37+02:00Elevated error rate of Google SERP proxy<p><small>Jun <var data-var='date'>16</var>, <var data-var='time'>08:07</var> CEST</small><br><strong>Resolved</strong> - We are investigating an elevated Google SERP proxy error rate and timeouts that have been showing up over the past 12 hours. The success rate is back to normal and we are working on a solution to make sure that these drops won't happen in the future.</p>tag:apify.statuspage.io,2005:Incident/102002692022-06-09T16:43:47+02:002022-06-09T16:43:47+02:00Email sending problems<p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>16:43</var> CEST</small><br><strong>Resolved</strong> - Mailgun has fixed the issue. Email sending works again only the messages are being delayed.</p><p><small>Jun <var data-var='date'> 9</var>, <var data-var='time'>14:29</var> CEST</small><br><strong>Identified</strong> - Our email provider is experiencing a major outage so at the moment we are unable to send email notifications or password reset emails.<br /><br />This also affects https://apify.com/apify/send-mail actor. For more info follow the incident at https://status.mailgun.com/</p>tag:apify.statuspage.io,2005:Incident/101828692022-06-08T13:30:00+02:002022-06-08T14:44:57+02:00Degraded Request Queue performance<p><small>Jun <var data-var='date'> 8</var>, <var data-var='time'>13:30</var> CEST</small><br><strong>Resolved</strong> - Due to an overload on one of our service providers, some requests to our Request Queue API endpoints were failing with a 429 Too Many Requests error. We have identified the issue and applied an immediate fix, and the performance is back to normal. We are continuing to monitor the platform for further issues.</p>tag:apify.statuspage.io,2005:Incident/100721122022-05-25T20:50:27+02:002022-05-25T20:50:27+02:00Investigating performance drop of our API<p><small>May <var data-var='date'>25</var>, <var data-var='time'>20:50</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>16:06</var> CEST</small><br><strong>Monitoring</strong> - The performance of the platform should be back to normal and we are monitoring all systems.</p><p><small>May <var data-var='date'>25</var>, <var data-var='time'>15:26</var> CEST</small><br><strong>Investigating</strong> - We are investigating a performance drop of our API</p>