tag:status.campfirehq.net,2005:/historyCampfire HQ Status - Incident History2024-03-29T07:52:12ZCampfire HQtag:status.campfirehq.net,2005:Incident/94950862022-03-09T23:33:49Z2022-03-09T23:33:49ZRanking System<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>23:33</var> UTC</small><br><strong>Resolved</strong> - We have now resolved this issue. Ranking is now operational, though many cookies may have been expired due to a change pushed by Roblox.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>04:50</var> UTC</small><br><strong>Identified</strong> - Campfire/Hyra ranking is currently not performing as expected due to a change with how Roblox manages cookies. We are working on a solution.</p>tag:status.campfirehq.net,2005:Incident/61567342021-01-31T16:30:48Z2021-01-31T16:30:50ZMigrating GB Pods (44, 1922, 55)<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>11:39</var> UTC</small><br><strong>Scheduled</strong> - During the period 1400 UTC and 1600 UTC we will be migrating our GB services. No service disruption is expected, though it could be a possibility. <br /><br />Customers may request a migration from GB to another pod if they are concerned about any potential downtime. Please contact support to do this.</p>tag:status.campfirehq.net,2005:Incident/60620802021-01-20T00:42:57Z2021-01-20T00:42:58ZIssues with bots<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>00:42</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>23:55</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>23:20</var> UTC</small><br><strong>Update</strong> - We have identified the issue and are rolling out a fix.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>20:56</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>19:57</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating an issue with our bot network</p>tag:status.campfirehq.net,2005:Incident/60227972021-01-14T16:51:30Z2021-01-14T16:51:30ZMember Counter Maintenance<p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>13:51</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>14</var>, <var data-var='time'>13:50</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.campfirehq.net,2005:Incident/55789052020-11-16T16:51:36Z2020-11-16T16:51:36ZDelayed or no response on website<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>16:10</var> UTC</small><br><strong>Identified</strong> - This issue is caused by an upstream internet provider issue. We will provide more information as we receive it.</p>tag:status.campfirehq.net,2005:Incident/54094382020-10-26T01:00:36Z2020-10-26T01:00:41ZDaylight saving<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'>25</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>Scheduled</strong> - During daylight saving, some unexpected bugs may occur. Our team will be carrying out maintenance on this day to resolve any bugs that could potentially occur. The Campfire Team apologises about any inconvenience this could cause, and will try to keep disruption to a minimum.</p>tag:status.campfirehq.net,2005:Incident/53011792020-10-11T19:30:03Z2020-10-12T08:47:54ZDegraded ranking performance<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>19:30</var> UTC</small><br><strong>Resolved</strong> - Roblox has recovered and Campfire Ranking requests are now operational.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>17:57</var> UTC</small><br><strong>Identified</strong> - A Roblox issue causing decreased performance on the platform is impacting performance of Campfire Ranking.</p><p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>17:32</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.campfirehq.net,2005:Incident/52951392020-10-10T19:47:47Z2020-10-10T19:47:47ZDegraded ranking performance<p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>19:47</var> UTC</small><br><strong>Resolved</strong> - Roblox has recovered and Campfire Ranking requests are now operational.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>18:29</var> UTC</small><br><strong>Update</strong> - We're experiencing an elevated level of API errors due to a Roblox outage. The majority of requests are still going through, however a small percentage are failing.</p><p><small>Oct <var data-var='date'>10</var>, <var data-var='time'>17:57</var> UTC</small><br><strong>Identified</strong> - Ranking performance is currently degraded. We are looking into this issue and will provide an update shortly.</p>tag:status.campfirehq.net,2005:Incident/50208742020-09-05T14:10:30Z2020-09-05T14:10:31ZProvisioning Strivo<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>14:10</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>14:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>01:47</var> UTC</small><br><strong>Scheduled</strong> - We anticipate the possibility of downtime of less than 10 minutes while we provision a large backbone platform on our campfire.host infrastructure. This may cause some problems with our internal publish subscriptions API between our bot pods, but the downtime will be kept below 10 minutes. <br /><br />We will work diligently and swiftly to ensure that services are kept to a minimal disruption. We apologise about any inconvenience this may cause.</p>tag:status.campfirehq.net,2005:Incident/48539362020-08-14T14:24:32Z2020-08-14T14:24:32ZSecurity vulnerability<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>14:24</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>14:15</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>14:10</var> UTC</small><br><strong>Update</strong> - A fix has been implemented and these changes are now deploying across the CFHQ network.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>14:03</var> UTC</small><br><strong>Identified</strong> - We are working hard to roll out a fix as soon as possible for this security bug.</p>tag:status.campfirehq.net,2005:Incident/48191682020-08-10T00:57:35Z2020-08-10T00:57:36ZMember Counters<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>00:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>00:47</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>00:45</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>00:40</var> UTC</small><br><strong>Investigating</strong> - An issue has automatically been identified with member counters. A team is now investigating the issue.</p>tag:status.campfirehq.net,2005:Incident/47828122020-08-07T21:00:33Z2020-08-07T21:00:34ZUpdate to shout proxies<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>20:24</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>20:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'> 5</var>, <var data-var='time'>00:53</var> UTC</small><br><strong>Scheduled</strong> - We will be carrying out scheduled maintenance on our group shout proxies to remove “Powered by campfirehq.net” from the footer, making this product entirely white labelled. We anticipate this maintenance to last no longer than 60 minutes. <br /><br />Updates will be posted as required. We understand that this may cause some inconvenience for customers, and will work as quickly and as diligently as possible. Thank you for your ongoing support.</p>tag:status.campfirehq.net,2005:Incident/46442982020-07-17T21:43:34Z2020-07-17T21:43:35ZAll systems unavailable<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>21:43</var> UTC</small><br><strong>Resolved</strong> - We're marking this as resolved, further information can be found at https://www.cloudflarestatus.com/</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>21:32</var> UTC</small><br><strong>Investigating</strong> - There is currently an outage across Campfire products and services due to an upstream internet issue.</p>tag:status.campfirehq.net,2005:Incident/46399152020-07-17T12:44:55Z2020-07-17T12:44:55ZRanking services down<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>12:44</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>11:54</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.campfirehq.net,2005:Incident/46274162020-07-16T22:19:00Z2020-07-16T22:19:01ZMember Counters not operating as expected.<p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>22:19</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>10:31</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>10:29</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>10:20</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>08:53</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue regarding member counters not sending correct or at a reduced interval. As this product continues to grow in size, we are finding it increasingly difficult to manage the 1,728,000+ requests per day we send to Roblox. <br /><br />We are working closely with our DevOps Team to resolve the situation as soon as possible and we will update you as required. We apologise about the inconvenience.</p>tag:status.campfirehq.net,2005:Incident/45946932020-07-12T13:25:31Z2020-07-12T13:25:31ZPod 212 - Upgrade<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>13:25</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>12:25</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>12:22</var> UTC</small><br><strong>Scheduled</strong> - We are upgrading the resources on Pod 212. Services on Pod 212 will be down whilst this upgrade occurs.</p>tag:status.campfirehq.net,2005:Incident/45073622020-07-01T20:49:55Z2020-07-01T20:49:57ZRoblox.com down<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>20:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>18:56</var> UTC</small><br><strong>Monitoring</strong> - The site is now resumed. We are monitoring and will update if the situation changes.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>18:48</var> UTC</small><br><strong>Identified</strong> - Issues are still occuring. We will update you as required.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>18:44</var> UTC</small><br><strong>Monitoring</strong> - The site now back up. Our team are continuing to monitor and will keep you updated should issueso occur again.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>18:42</var> UTC</small><br><strong>Update</strong> - The site is now coming back online slowly.</p><p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>18:37</var> UTC</small><br><strong>Identified</strong> - Roblox.com is currently down.</p>tag:status.campfirehq.net,2005:Incident/44769612020-06-30T17:27:42Z2020-06-30T17:27:43ZCentralised Provider Change<p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:27</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:26</var> UTC</small><br><strong>Verifying</strong> - Maintenance tasks completed, verifying everything working as expected.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:24</var> UTC</small><br><strong>Update</strong> - Pod 1024 is now being moved.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:11</var> UTC</small><br><strong>Update</strong> - Pod 525 is now confirmed online.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:10</var> UTC</small><br><strong>Update</strong> - Pod 525 is now moving.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:06</var> UTC</small><br><strong>Update</strong> - Pod 666 and Pod 3662 are now fully operational.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:04</var> UTC</small><br><strong>Update</strong> - Pod 666 and 3662 are now being moved.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>17:02</var> UTC</small><br><strong>Update</strong> - Pod 212 is now restored and fully operation. We are now doing Pod 666.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Update</strong> - Pod 212 is currently being transferred. All bots on pod 212 are currently offline.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>16:46</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>16:01</var> UTC</small><br><strong>Update</strong> - We'll be moving all pods over to the new provider in 45 minutes. Some downtime is expected.</p><p><small>Jun <var data-var='date'>30</var>, <var data-var='time'>16:01</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>02:14</var> UTC</small><br><strong>Scheduled</strong> - We will be moving our Pod 212 to a new location during this time. We will try to keep service downtime to a minimum, during this time all customers using custom bots on Pod 212 will be re-routed to temporary Pod 790, located in Amsterdam. We apologise sincerely about any inconvenience this may cause to customers. <br /><br />Please also note that the following legacy bots will also be affected: <br />- Pastriez</p>tag:status.campfirehq.net,2005:Incident/44262932020-06-22T17:49:42Z2020-06-22T17:49:43ZMember Counters<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>16:58</var> UTC</small><br><strong>Update</strong> - We are working hard on a fix.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>10:08</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>09:44</var> UTC</small><br><strong>Update</strong> - We’re going to test a fix in production now. The only caveat of this is that there will be an increased delay between the counting - i.e. a longer amount of time will pass before we check for new members.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>00:49</var> UTC</small><br><strong>Identified</strong> - There is currently an issue where member counters may jump back and forth between two numbers. This is caused by a problem in the Roblox group count system, and we are looking into fixes for this problem. This service continues to operate as normal.</p>tag:status.campfirehq.net,2005:Incident/44130782020-06-20T21:51:16Z2020-06-20T21:51:17ZMember counters not operating<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>21:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>18:44</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>18:42</var> UTC</small><br><strong>Update</strong> - A fix will be rolling out very shortly.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>18:35</var> UTC</small><br><strong>Identified</strong> - Issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>16:22</var> UTC</small><br><strong>Update</strong> - Our DevOps Team are continuing to investigate the issue.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>12:45</var> UTC</small><br><strong>Update</strong> - DevOps Team are now working on a solution to this problem.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>02:59</var> UTC</small><br><strong>Update</strong> - All hooks and the ability to create and manage hooks has been disabled until DevOps Team becomes available. Apologies about the inconvenience.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>02:55</var> UTC</small><br><strong>Update</strong> - All DevOps team are currently unavailable. No expectation of resolution until DevOps Team become available again in 5-10 hours.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>02:53</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.campfirehq.net,2005:Incident/44034392020-06-19T09:44:21Z2020-06-19T09:44:23ZIssues with caching CDN<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>09:44</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>23:45</var> UTC</small><br><strong>Update</strong> - This issue may still be present for some users. We advise clearing your cache or using Ctrl + Shift + R / ⌘ + Shift + R</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>23:41</var> UTC</small><br><strong>Update</strong> - We expect this to be resolved for all users by 00:00 UTC.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>23:40</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p>tag:status.campfirehq.net,2005:Incident/42989992020-06-07T00:41:08Z2020-06-07T00:41:09ZBilling System Not Activating Bots<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>00:41</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>00:39</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>00:37</var> UTC</small><br><strong>Identified</strong> - The issue was identified as a type error. We’re now rolling out a patch.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>00:34</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.campfirehq.net,2005:Incident/40101692020-05-16T12:00:29Z2020-05-16T12:00:29ZZendesk - DNS Change<p><small>May <var data-var='date'>16</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>10:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>19:35</var> UTC</small><br><strong>Scheduled</strong> - Zendesk will be performing DNS maintenance during a scheduled window listed below. No downtime is expected.<br /><br />Details: https://support.zendesk.com/hc/en-us/articles/360047691573-Scheduled-Maintenance-May-16-2020-DNS-Change-All-Pods</p>tag:status.campfirehq.net,2005:Incident/40401512020-05-06T15:43:54Z2020-05-06T15:43:55ZRanking issues<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>15:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>13:08</var> UTC</small><br><strong>Monitoring</strong> - A fix has been deployed. We are now monitoring the fix.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>11:40</var> UTC</small><br><strong>Update</strong> - Our team are working on implementing rate limiting technology. We won’t ever hard rate limit, we’ll simply slow down your requests in transit. This is to prevent us from sending too many requests to Roblox’s servers. We will publish documentation to our Developer Docs when this has been implemented. <br /><br />Service will resume once this has been implemented. <br /><br />Thank you for your patience and cooperation.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>20:36</var> UTC</small><br><strong>Identified</strong> - We've identified the root cause as a denial of service attack on our ranking service. We are taking security precautions to resolve the issue.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>20:01</var> UTC</small><br><strong>Update</strong> - We have no ETA of when this can be resolved. The issues are beyond the scope of Campfire, and are with our providers.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:56</var> UTC</small><br><strong>Update</strong> - We've temporarily paused this service as we are facing issues on our backend database. We work with our team to resolve the issue.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:54</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'> 5</var>, <var data-var='time'>19:40</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating an issue with Campfire ranking and 429 errors. We're working closely with our DevOps Team to resolve the issues as quickly as possible.</p>tag:status.campfirehq.net,2005:Incident/40272352020-05-04T12:15:32Z2020-05-04T12:15:34ZUpgrading security of member counter & group shout proxy<p><small>May <var data-var='date'> 4</var>, <var data-var='time'>12:15</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed. Thank you for your patience.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>12:00</var> UTC</small><br><strong>Update</strong> - We're now temporarily pausing our hooks.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>09:45</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 4</var>, <var data-var='time'>08:57</var> UTC</small><br><strong>Scheduled</strong> - We are upgrading the security of the group shout proxy and member counters to support two groups with the same ID. Previously, it would cause an error when doing it and would actually edit another users configuration.</p>