All Systems Operational

Updated a few seconds ago

Scheduled Maintenance

Schedule

March 5, 2020 10:00PM - March 6, 2020 12:00AM MST

Components

Workfront Application Platform

Locations

CL01, CL02, CL03

Description

We will be performing database maintenance the evening of Thursday, March 5th. The Production environment may be unavailable for up to 30 minutes while maintenance takes place. Expected Time of Maintenance: Thursday, March 5th from 10:00 PM to 12:00 Midnight Mountain Time.

Workfront Application Platform




Operational

Workfront API




Operational

Workfront Event Subscription




Operational

Workfront Fusion




Operational

Workfront Library




Operational

WorkfrontDAM




Operational

Workfront Proof




Operational

Workfront Non-Production




Operational

Customer Experience Site




Operational

External Services

Workfront Uptime

100.00%




Workfront-Proof Uptime

100.00%




Avg. Round Trip

79.34ms




Total Requests

1854267.75 Hits




Workfront Uptime

100.00%




Workfront-Proof Uptime

100.00%




Avg. Round Trip

72.78ms




Total Requests

85785675.43 Hits




Workfront Uptime

100.00%




Workfront-Proof Uptime

99.95%




Avg. Round Trip

83.71ms




Total Requests

208451625.17 Hits




Maintenance Windows North America: CL01 | CL02 | CL03 | CL05 | CL06 2nd and 4th Saturday 8:00 pm to 12:00 am MT 1st and 3rd Thursday 8:00 pm to 12:00 am MT EMEA: CL04 1st and 3rd Thursday 8:00 pm to 12:00 am GMT CL01, CL02, CL03, CL05, and CL06 Preview environments refresh on Saturday CL04 Preview environment refreshes on Friday **We infrequently schedule maintenance in the windows listed above. If down time is necessary, it will be scheduled in the corresponding window and announced on this page.**

0

Active Incidents

0

Active Maintenances

1

Upcoming Maintenances

History (Last 30 days)

Incident Status

Operational


Components

Workfront Application Platform, Workfront Event Subscription


Locations

CL01, CL02, CL03, CL05, Workfront Event Subscription




February 18, 2020 2:22PM MST
[Investigating] We are seeing a delay in Event subscriptions and are investigating the root cause.

February 18, 2020 4:28PM MST
[Investigating] We are still seeing the delay in event subs. We have restarted all pods in event-sub namespace and cleaned up dead logs. We are continuing our investigation on the issue.

February 18, 2020 6:23PM MST
[Monitoring] At this time engineers have restored functionality to the event subscription service. Additional investigation is ongoing, and Workfront will continue to monitor the behavior of event subscriptions. If you continue to experience any issues, please contact support.

February 20, 2020 10:48AM MST
[Resolved] Engineering has completed monitoring over a 24 hour period for this issue, and there have been no further delays in event subscriptions for production lanes. Issue has been resolved successfully.

Incident Status

Component Disruption - SEV 3


Components

Workfront Fusion


Locations

Workfront Fusion




February 18, 2020 4:37AM MST
[Investigating] At this time fusion is experiencing issues that may lead to failure or performance issues. Workfront has been engaged with engineers to resolve these issues.

February 18, 2020 5:19AM MST
[Investigating] We discovered that the Fusion is experiencing issues that may lead to partition ‘write’ transaction abort on Fusion Document cards. We are working to fix the problem.

February 18, 2020 8:27AM MST
[Investigating] Customers are still experiencing Fusion FLOs Document downloading cards issue. We have escalated this issue to engineering within the Fusion product and are continuing to investigate the underlying cause.

February 18, 2020 10:46AM MST
[Identified] The problem has been identified and we are working to alleviate the issue.

February 18, 2020 11:41AM MST
[Monitoring] The issue has been solved and we are currently monitoring affected FLOs.

February 18, 2020 12:39PM MST
[Resolved] We have received confirmation that this issue is now resolved.

Incident Status

Operational


Components

Workfront Fusion


Locations

Workfront Fusion




February 14, 2020 4:45AM MST
[Identified] We have identified that Fusion FLOs are experiencing performance issues. At this time Workfront has been engaged with engineers to resolve these issues. If you continue to experience issues with Fusion, please contact support.

February 14, 2020 4:47AM MST
[Identified] We have identified that Fusion FLOs are experiencing performance issues. At this time Workfront has been engaged with engineers to resolve these issues. If you continue to experience issues with Fusion, please contact support.

February 14, 2020 7:52AM MST
[Identified] Customers are still experiencing Fusion FLOs performance issue. Workfront is working with engineering to identify the underlying cause.

February 14, 2020 10:24AM MST
[Identified] We are working to fix Fusion FLOs performance issues. The node engines have been restart, but had no positive effect. Additional resources are being pulled in from the engineering team.

February 14, 2020 11:54AM MST
[Identified] As of 11:45am mnt Fusion is down and inaccessible. We are on call with engineering to bring Fusion back up.

February 14, 2020 12:22PM MST
[Monitoring] We have blacklisted 4 workflows and bounced engine nodes and flows. Fusion coming back up for those other than the 4 FLOs that were blacklisted.

February 14, 2020 3:47PM MST
[Resolved] Issue has been resolved as of 12:45pm MT

Incident Status

Service Disruption - SEV 1


Components

Workfront Application Platform, Workfront Proof


Locations

CL01, CL02, CL03, CL05, Workfront Proof




February 14, 2020 8:43AM MST
[Identified] Proofing is not available now and Workfront clusters have performance degradation. At this time Workfront has been engaged with engineers to resolve these issues.

February 14, 2020 9:01AM MST
[Identified] Workfront Engineers are working to address the underlying cause. While this work is underway users may experience issues with delayed performance within Workfront and Proofing may be non-responsive or unavailable.

February 14, 2020 9:25AM MST
[Identified] Performance on clusters 1, 2, and 3 for the Workfront application should be working as normal. Cluster 5 is experiencing slowness. Proofing is currently unavailable.

February 14, 2020 9:43AM MST
[Monitoring] Speeds on clusters 1,2,3, and 5 should be normal and proofing service should be restored. We are watching to ensure that the underlying issue has been corrected.

February 14, 2020 9:56AM MST
[Resolved] Services have been restored.

Incident Status

Operational


Components

Workfront Application Platform


Locations

CL01, CL02, CL03, CL06




February 12, 2020 1:14PM MST
[Investigating] Email notifications are delayed and potentially undelivered, based on internal tests. We are performing additional tests and investigating the issue.

February 12, 2020 1:27PM MST
[Investigating] Emails do not appear to have been lost as part of this incident, but should only be delayed. We're continuing to investigate the issue.

February 12, 2020 1:47PM MST
[Resolved] We have investigated and resolved the issue, the results of which have been verified by our internal tests. Please reach out to our support teams if there are unexpected delays with email notifications from this point.

Incident Status

Component Disruption - SEV 3


Components

Workfront Fusion


Locations

Workfront Fusion




February 4, 2020 3:57PM MST
[Monitoring] A small portion of Fusion FLOs have been identified as experiencing performance issues. At this time Workfront has been engaged with engineers to resolve these issues. An infrastructure change has been made and we are investigating the results of this maintenance work. If you continue to experience issues with Fusion, please contact support. We expect an update on this overall issue at approximately 10 AM Mountain on 2/5/2020.

February 5, 2020 9:45AM MST
[Monitoring] Post maintenance behavior continues to be improved. We are still monitoring customer interactions directly and seeing success in many interactions which previously did not succeed. However, we have identified some ongoing issues for a smaller section of interactions and are continuing to follow up on those items. If you have any questions at this time, please contact support.

February 5, 2020 3:54PM MST
[Monitoring] Workfront has verified that while systemic impact preventing FLO execution from functioning has been resolved, ongoing issues with visual indicators may still appear to indicate FLOs have failed or not completed. We are continuing to look into the source of this visual defect and will provide updates when available. If you have additional questions, please contact support.

February 8, 2020 12:18PM MST
[Resolved] At this time FLO execution has returned to normal. If you have any questions or concerns please contact support.

Incident Status

Operational


Components

Workfront Application Platform


Locations

CL01, CL02, CL03, CL05




February 7, 2020 12:54PM MST
[Investigating] Customers on Cl01, Cl02, Cl03 and Cl05 may experience degraded performance when interacting with documents or avatars. We are currently resolving the cause of the issue.

February 7, 2020 1:12PM MST
[Identified] Users may encounter degraded performance on Cl05. We are working on restoring service.

February 7, 2020 1:45PM MST
[Monitoring] Service should be back to normal. We are continuing to monitor this issue.

February 7, 2020 4:47PM MST
[Resolved] The system has returned to normal operations. We will continue to monitor system performance involving documents closely.

Description

We will be performing database maintenance. The Production environment may be unavailable for up to 30 minutes while maintenance takes place. Expected Time of Maintenance: Thursday, February 6th from 10:00 PM to 12:00 Midnight Mountain Time.


Components

Workfront Application Platform


Locations

CL01, CL05


Schedule

February 6, 2020 10:00PM - February 7, 2020 12:00AM MST



February 6, 2020 10:10PM MST
[Update] We will be performing database maintenance on Friday. The Production environment may be unavailable for up to 30 minutes during maintenance.

February 6, 2020 11:10PM MST
[Update] Maintenance finished successfully.

Incident Status

Operational


Components

Workfront Library


Locations

Workfront Library




January 28, 2020 10:03PM MST
[Investigating] The Workfront service is experiencing a performance issue. Some users may have problems with logging in and timeout errors when using the service. We are working to resolve the situation as quickly as possible.

January 28, 2020 10:26PM MST
[Resolved] The issue is resolved at arround 22:17. Now customers will not see any issue.
Workfront Preview is not availableComponent Disruption - SEV 3

Incident Status

Component Disruption - SEV 3


Components

Workfront Non-Production


Locations

CL01 - Preview/Sandbox, CL02 - Preview/Sandbox, CL03 - Preview/Sandbox, CL04 - Preview/Sandbox, CL05 - Preview/Sandbox




January 28, 2020 1:16PM MST
[Investigating] Preview is currently inaccessible. There is a workaround by appending "redrock-" in front your domain. For example, "www.redrock-[customerdomain].preview.workfront.com" should work.

January 28, 2020 1:28PM MST
[Monitoring] Preview is available again. We are looking into root cause and monitoring the situation.

January 28, 2020 1:43PM MST
[Resolved] The issue has been resolved.