tag:status.us.lifeomic.com,2005:/historyUS - LifeOmic Status - Incident History2024-03-29T01:23:49-04:00US - LifeOmictag:status.us.lifeomic.com,2005:Incident/144358362022-12-05T16:08:54-05:002022-12-05T16:08:54-05:00Login and data ingestion timeouts<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>16:08</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>15:58</var> EST</small><br><strong>Monitoring</strong> - Our API and applications are now fully operational. We are continuing to monitor the situation to ensure a full recovery.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>15:15</var> EST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.us.lifeomic.com,2005:Incident/86824232021-11-24T15:46:46-05:002021-11-24T15:46:46-05:00Major platform outage.<p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>15:46</var> EST</small><br><strong>Resolved</strong> - The platform and applications have fully recovered.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>15:03</var> EST</small><br><strong>Monitoring</strong> - Our API and applications are now fully operational. We are continuing to monitor the situation to ensure a full recovery.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>14:50</var> EST</small><br><strong>Update</strong> - We are seeing some improvement in the error rates and are continuing to work on a full resolution.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>14:17</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>14:17</var> EST</small><br><strong>Identified</strong> - We've identified the underlying issue and are working with our vendor to restore service.</p><p><small>Nov <var data-var='date'>24</var>, <var data-var='time'>13:44</var> EST</small><br><strong>Investigating</strong> - We are seeing high error rates in our API which is causing all our applications to be down.</p>tag:status.us.lifeomic.com,2005:Incident/64919412021-03-09T18:25:02-05:002021-03-09T18:25:02-05:00PHC, Lifeology, LIFE Fasting Tracker, and Life Extend outage.<p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>18:25</var> EST</small><br><strong>Resolved</strong> - The outage has been resolved. All systems are working properly.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>17:55</var> EST</small><br><strong>Monitoring</strong> - We've identified the problem, and a fix has been deployed. We are seeing some recovery and successful authentications. We are monitoring the situation.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>17:33</var> EST</small><br><strong>Investigating</strong> - Authentication and authorization into all of our applications and APIs is currently failing. We are investigating the problem.</p>tag:status.us.lifeomic.com,2005:Incident/48040032020-08-07T14:01:17-04:002020-08-07T14:01:17-04:00Errors with Organization Invitations<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>14:01</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>13:45</var> EDT</small><br><strong>Monitoring</strong> - We are no longer seeing errors with invitations. We will continue to monitor.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:59</var> EDT</small><br><strong>Identified</strong> - We are seeing an increase in errors again with circle and LIFE Extend organization invitations. The source of the errors has been identified.</p>tag:status.us.lifeomic.com,2005:Incident/48036682020-08-07T11:17:55-04:002020-08-07T11:17:55-04:00Errors with Organization Invitations<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:17</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:05</var> EDT</small><br><strong>Identified</strong> - We have identified the source of the errors with invitations.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>11:03</var> EDT</small><br><strong>Investigating</strong> - We are currently seeing an increase in errors with circle and LIFE Extend organization invitations.</p>tag:status.us.lifeomic.com,2005:Incident/46891202020-07-23T11:40:07-04:002020-07-23T11:40:07-04:00Jupyter Notebooks<p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>11:40</var> EDT</small><br><strong>Resolved</strong> - The Jupyter Notebooks issue has been resolved. Please contact us if you continue to experience any issues with Jupyter Notebooks.</p><p><small>Jul <var data-var='date'>23</var>, <var data-var='time'>07:42</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with Jupyter Notebooks.</p>tag:status.us.lifeomic.com,2005:Incident/43729162020-06-15T19:21:51-04:002020-06-15T19:22:25-04:00Jupyter Notebook Maintenance<p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>19:21</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed. Please contact us if you notice any issues with using Jupyter Notebooks in the PHC.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>19:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>15</var>, <var data-var='time'>12:25</var> EDT</small><br><strong>Scheduled</strong> - We will be performing some updates to Jupyter Notebooks during this hour. All running notebooks will be stopped during this time .</p>tag:status.us.lifeomic.com,2005:Incident/29688472019-09-08T19:37:09-04:002019-09-08T19:37:09-04:00Elevated API Errors<p><small>Sep <var data-var='date'> 8</var>, <var data-var='time'>19:37</var> EDT</small><br><strong>Resolved</strong> - The platform and applications are now back to their normal stable state.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>14:53</var> EDT</small><br><strong>Monitoring</strong> - We've resolved the issue that was causing the elevated error rate and are now monitoring the situation. It may take some time for the platform to return to its normal stable state.</p><p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>13:50</var> EDT</small><br><strong>Identified</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status.us.lifeomic.com,2005:Incident/25032952019-06-06T10:57:50-04:002019-06-06T10:57:50-04:00Elevated API Errors<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>10:57</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>10:18</var> EDT</small><br><strong>Monitoring</strong> - A fix has been deployed and we are monitoring the results.</p><p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>09:07</var> EDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status.us.lifeomic.com,2005:Incident/23679772019-05-03T14:45:56-04:002019-05-03T14:45:56-04:00Elevated API Errors and Delays<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>14:45</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>14:40</var> EDT</small><br><strong>Monitoring</strong> - Error rates are dropping and we are seeing improved performance. We are continuing to monitor the situation.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>14:17</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'> 3</var>, <var data-var='time'>14:14</var> EDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and delays and are currently looking into the issue.</p>tag:status.us.lifeomic.com,2005:Incident/23464252019-04-23T19:27:51-04:002019-04-23T19:27:51-04:00Elevated API Errors<p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>19:27</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>18:11</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>17:44</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'>23</var>, <var data-var='time'>17:44</var> EDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors and are currently looking into the issue.</p>tag:status.us.lifeomic.com,2005:Incident/22418672019-03-05T17:19:55-05:002019-03-05T17:19:55-05:00Elevated latency for clinical and health data writes.<p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>17:19</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>14:41</var> EST</small><br><strong>Monitoring</strong> - Latency is back down to normal levels. We are monitoring the system now to ensure it is fully operational.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>12:06</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 5</var>, <var data-var='time'>11:35</var> EST</small><br><strong>Investigating</strong> - We are seeing elevated latency in committing clinical and health data. There has been no data loss, but writes are delayed. This includes weight tracking via our LIFE mobile app.</p>tag:status.us.lifeomic.com,2005:Incident/22293882019-02-26T15:48:28-05:002019-02-26T15:48:28-05:00Elevated latency for clinical and health data writes.<p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>15:48</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>15:28</var> EST</small><br><strong>Monitoring</strong> - The fix has been deployed and latency is coming down. We will continue to monitor.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>14:58</var> EST</small><br><strong>Update</strong> - The problem has been identified and we are deploying a fix.</p><p><small>Feb <var data-var='date'>26</var>, <var data-var='time'>14:57</var> EST</small><br><strong>Identified</strong> - We are seeing elevated latency for clinical and health data which is causing delays in new data being visible in the PHC and LIFE Fasting Tracker (weight tracking only, fasts are unaffected).</p>tag:status.us.lifeomic.com,2005:Incident/21644922019-01-23T13:48:26-05:002019-01-23T13:48:26-05:00Circle feed distribution delays<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>13:48</var> EST</small><br><strong>Resolved</strong> - The platform has recovered and we are continuing to monitor status.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>13:22</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>13:02</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented and measured.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:12</var> EST</small><br><strong>Investigating</strong> - The platform is experiencing delays with feed distributions for fasting updates. We are currently investigating the issue.</p>tag:status.us.lifeomic.com,2005:Incident/20719242018-11-29T16:26:20-05:002018-11-29T16:26:20-05:00PHC New User sign up<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>16:26</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:51</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>15:01</var> EST</small><br><strong>Identified</strong> - Users signing up to use the Precision Health Cloud are experiencing unexpected behavior. We have identified the issue and will be releasing a fix soon. Existing users with an active account are unaffected at this time.</p>tag:status.us.lifeomic.com,2005:Incident/20484452018-11-19T09:15:01-05:002018-11-19T09:15:01-05:00API and App connection timeouts<p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>09:15</var> EST</small><br><strong>Resolved</strong> - The connection timeouts have subsided, and all services are fully operational. We are working with AWS to find the cause of the issue.</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>09:08</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Nov <var data-var='date'>19</var>, <var data-var='time'>08:47</var> EST</small><br><strong>Investigating</strong> - We have started to see an increasing number of connection timeouts across our API's and Web Applications. We are currently investigating the issue.</p>tag:status.us.lifeomic.com,2005:Incident/19577502018-10-08T22:05:05-04:002018-10-08T22:05:05-04:00Scheduled Maintenance<p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>22:05</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>22:01</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Oct <var data-var='date'> 8</var>, <var data-var='time'>21:41</var> EDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance that could affect users of the PHC web application. Users experiencing issues with the PHC web application may resolve them by refreshing the page.</p>tag:status.us.lifeomic.com,2005:Incident/19523112018-10-05T15:54:26-04:002018-10-05T15:54:26-04:00High API error rate<p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>15:54</var> EDT</small><br><strong>Resolved</strong> - AWS has resolved the issue with Amazon S3 in the US-EAST-2 region. All API's are operating normally.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>15:45</var> EDT</small><br><strong>Monitoring</strong> - AWS has reported increased error rates for Amazon S3 in the US-EAST-2 region that are causing some increased latency and random errors in the API's.</p>tag:status.us.lifeomic.com,2005:Incident/18879972018-09-02T00:19:24-04:002018-09-02T00:19:24-04:00Elevated LIFE application timeouts<p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>00:19</var> EDT</small><br><strong>Resolved</strong> - A longer term fix is in place and we are monitoring the update. Thanks for making LIFE better!</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>14:34</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results. A longer term fix is also in the works.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>12:49</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>10:53</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.us.lifeomic.com,2005:Incident/18663022018-08-17T16:21:58-04:002018-08-22T11:05:40-04:00Increased error rates and latency on our APIs.<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>16:21</var> EDT</small><br><strong>Resolved</strong> - Everything is back to normal.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>15:58</var> EDT</small><br><strong>Monitoring</strong> - Error rates and latency are back to normal levels. We are continuing to monitor the service.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>15:39</var> EDT</small><br><strong>Investigating</strong> - We are seeing increased latency and error rates on our APIs. The development team is investigating the issues.</p>tag:status.us.lifeomic.com,2005:Incident/18414722018-08-01T21:06:43-04:002018-08-01T21:07:47-04:00Facebook authentication outage.<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>21:06</var> EDT</small><br><strong>Resolved</strong> - Facebook users should now be able to use the app.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>20:05</var> EDT</small><br><strong>Monitoring</strong> - There was a spike in facebook application rate limits which have now past and we are monitoring the health.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>18:23</var> EDT</small><br><strong>Investigating</strong> - Users who sign in using Facebook are unable to log in to the application. Other forms of sign in are unaffected.</p>tag:status.us.lifeomic.com,2005:Incident/17994972018-07-03T16:55:28-04:002018-07-03T16:55:28-04:00PHC Web App Outage<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>16:55</var> EDT</small><br><strong>Resolved</strong> - The outage has been resolved.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>16:49</var> EDT</small><br><strong>Identified</strong> - The Precision Health Cloud web app is currently down. We are working to resolve the issue.</p>tag:status.us.lifeomic.com,2005:Incident/17667232018-06-12T11:02:26-04:002018-06-12T11:02:26-04:00FHIR API outage.<p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>11:02</var> EDT</small><br><strong>Resolved</strong> - The FHIR API is functional again.</p><p><small>Jun <var data-var='date'>12</var>, <var data-var='time'>10:58</var> EDT</small><br><strong>Identified</strong> - The FHIR API is failing with 500 errors. We are currently investigating the issue.</p>tag:status.us.lifeomic.com,2005:Incident/17492092018-05-31T12:00:00-04:002018-05-31T12:00:00-04:00Web application and API outage<p><small>May <var data-var='date'>31</var>, <var data-var='time'>12:00</var> EDT</small><br><strong>Resolved</strong> - Between 3:15 AM and 3:56 AM EDT, the LifeOmic web applications and API's were unavailable due to internet connectivity issues in the AWS US-EAST-2 region.</p>tag:status.us.lifeomic.com,2005:Incident/16762112018-04-06T14:34:38-04:002018-04-06T14:34:38-04:00PMP App Domain Change<p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>14:34</var> EDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>13:00</var> EDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 6</var>, <var data-var='time'>09:38</var> EDT</small><br><strong>Scheduled</strong> - https://pmp.us.lifeomic.com/ is moving to https://apps.us.lifeomic.com/. Please update your bookmarks and use https://apps.us.lifeomic.com/. At 1:00pm Eastern time https://pmp.us.lifeomic.com/ will be briefly unavailable and then will begin redirecting users to https://apps.us.lifeomic.com/. https://apps.us.lifeomic.com/ will be fully available during this time.</p>