Duration: 6 hours and 31 minutes

Affected Components: Zultys

Apr 28, 20:42:20 GMT+0
Identified - Telus has now posted the incident on their website: "We are experiencing 1-way audio and DTMF issues with inbound calls routing via TELUS' Western Voice Core infrastructure. Our Network Integrity team is investigating the issue with priority.".

Apr 28, 21:42:13 GMT+0
Monitoring - Telus reports they have fixed the problem and our tests are now successful. We will continue to monitor for any further problems..

Apr 28, 23:08:15 GMT+0
Resolved - We have not detected any further issues since 2:42pm. This incident is now resolved. .

Apr 28, 16:37:23 GMT+0
Investigating - Problem: We are investigating reports of no audio on some inbound calls. Evidence currently points to an issue on the Telus Mobility network. We have escalated this issue to Telus..

Apr 28, 17:09:44 GMT+0
Identified - Update: We have confirmed the issue is with Telus Mobility. Some inbound calls from Telus Mobility cellphones are not passing audio to our network. We are continuing to work with Telus on a fix..

Apr 28, 18:32:59 GMT+0
Identified - We are continuing to work with Telus on a fix for this Mobility network issue. .

Apr 28, 19:30:52 GMT+0
Identified - Telus has now identified the issue and repair teams are actively working to resolve it. .

Affected Components: SMS/MMS

Apr 24, 04:37:43 GMT+0
Resolved - We identified and resolved an issue with our SMS delivery pipeline earlier today. Enhanced safeguards and logging have been deployed. We are actively monitoring and no further impact is expected..

Affected Components: SMS/MMS

Mar 31, 21:43:33 GMT+0
Resolved - An incident occurred today between approximately 10:00 AM and 12:00 PM PST that has now been fully resolved. During this window, a critical change to our backup infrastructure caused intermittent connectivity issues for texting clients. Affected clients experienced delayed reconnections, which resulted in some users being temporarily unable to send or receive messages. Other users may have experienced degraded performance but maintained partial functionality. The root cause has been identified and addressed. All SMS services are now operating normally. We will continue to monitor the situation to ensure stability. We apologize for any disruption this may have caused..

Duration: 1 hour and 16 minutes

Feb 5, 18:01:11 GMT+0
Monitoring - There is an active Telus Fiber outage occurring in Kamloops affecting the internet of Telus Fiber customers. More information can be found on the Telus Service Status page: <https://www.telus.com/en/bc/outages?linktype=ge-footer>.

Feb 5, 19:16:48 GMT+0
Resolved - Telus is reporting the outage has been resolved..

Duration: 4 hours and 31 minutes

Affected Components: , Inbound Call Routing, Outbound Call Routing (International), Toll-free Call Routing, Zultys, Outbound Call Routing (North America), Voice Services →

Oct 31, 19:40:00 GMT+0
Investigating - We are currently investigating an incident at one of our data centers causing a phone service disruption to a significant number of customers..

Oct 31, 20:43:20 GMT+0
Identified - We have identified an issue between Shaw/Rogers internet clients and one of our data centres. Engineers are working to resolve the issue as soon as possible. .

Oct 31, 21:34:07 GMT+0
Identified - Data Centre engineers are working directly with Rogers to resolve the issue as soon as possible. Impacted customers can use the Mobile ZAC application on their cellphone with wifi turned off to receive calls, if the cellphone provider is not Rogers..

Oct 31, 23:53:21 GMT+0
Identified - Data centre and Rogers engineers are still working on this. We found a technical "work-around" to restore service to many customer systems throughout the afternoon. We are continuing to monitor this issue and will update when new information is available..

Nov 1, 00:24:28 GMT+0
Resolved - This incident has now been resolved. We will continue to monitor throughout the weekend for any further problems. .

Duration: 4 hours and 30 minutes

Affected Components: Zultys

Oct 10, 20:30:00 GMT+0
Investigating - We began to receive some reports of call quality issues around 1:30pm. .

Oct 10, 23:00:00 GMT+0
Monitoring - More and more reports came in throughout the afternoon. We discovered a problem with one of Telus redundant call routing servers. We disabled this server and immediately noticed an improvement. .

Oct 11, 00:00:00 GMT+0
Monitoring - We found some "stuck" calls between the degraded Telus server and one of our servers. We rebooted our server to clear them. Our server did not boot up properly and some customers reported being unable to call out for about 10 minutes just before 5:00pm. We moved calls to our redundant server. .

Oct 11, 01:00:00 GMT+0
Resolved - Our server issue has been fixed and all calls are operational. We are continuing to monitor the failing Telus server, but this should have no impact on service over the weekend. .

Duration: 2 hours and 44 minutes

Affected Components: Zultys

Oct 10, 17:50:00 GMT+0
Monitoring - At 10:50am we detected one of our data centres went fully offline. It came back at 10:53, went down at 10:55, and came back at 10:58\. We believe there was a network problem that caused a hard failover scenario. We are monitoring for further issues and have opened a case with our data centre support. .

Oct 10, 20:34:26 GMT+0
Resolved - We have detected no further issues. .

Duration: 1 hour and 3 minutes

Affected Components: Zultys, Inbound Call Routing

Sep 16, 16:48:00 GMT+0
Monitoring - We believe the issue has been resolved at this time but will continue to monitor. Please report any incoming call issues to our \[dispatch@topsoffice.ca\](mailto:dispatch@topsoffice.ca) email..

Sep 16, 16:30:00 GMT+0
Investigating - We've detected a widespread problem with call routing affecting incoming calls. We believe the issue has been resolved at this time but will continue to monitor. Please report any incoming call issues to our \[dispatch@topsoffice.ca\](mailto:dispatch@topsoffice.ca) email..

Sep 16, 17:33:00 GMT+0
Resolved - This incident has been resolved. Problem: Customers reported inbound and outbound calls ringing forever, failing to connect, or getting a busy signal. Cause: This outage was caused by a software glitch in one of our maintenance tools that created a cascading failure beginning at approximately 9:30am. Once we determined the cause we were able to resolve it at 9:48am. Corrective Action: we will be patching and testing the maintenance tool offline before putting it back into service..

Duration: 15 minutes

Affected Components: Inbound Call Routing, Outbound Call Routing (North America), Outbound Call Routing (International), , Toll-free Call Routing, Zultys, Voice Services →

Aug 14, 20:56:33 GMT+0
Resolved - This incident has been resolved..

Aug 14, 20:42:02 GMT+0
Investigating - Problem: Our automated monitoring has detected a network issue affecting one of our three data centres. The impact is limited, and only a small portion of customers are affected. The issue began at approximately 12:20 PM PST and is ongoing. We are actively investigating to determine the cause and will provide further updates as more information becomes available..

Duration: 1 hour and 56 minutes

Jul 29, 21:27:28 GMT+0
Investigating - Status: investigating Problem: Our automated monitoring detected two brief network outages at one of our data centres. The first occurred at 13:52 PDT and lasted for \~2 minutes. The second occurred at 14:02 PDT and lasted \~2 minutes. The network is operational now and no further issues have been detected. We are investigating further to determine the cause. .

Jul 29, 23:23:43 GMT+0
Resolved - Status: resolved Update: We have confirmation the problem was caused by one of the internet providers to the data centre. We do not expect any further issues. .

Duration: 2 hours

Affected Components: Zultys

Jun 5, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 5, 02:00:00 GMT+0
Identified - We will be performing scheduled software updates during the following times: **June 3rd - 7:00pm-9:00pm** **June 4th - 7:00pm-9:00pm** If you received an email on May 23rd, your phones and phone applications may go offline for up to 30 minutes during one of these windows..

Jun 5, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours

Affected Components: Zultys

Jun 4, 02:00:01 GMT+0
Identified - Maintenance is now in progress.

Jun 4, 02:00:00 GMT+0
Identified - We will be performing scheduled software updates during the following times: **June 3rd - 7:00pm-9:00pm** **June 4th - 7:00pm-9:00pm** If you received an email on May 23rd, your phones and phone applications may go offline for up to 30 minutes during one of these windows. .

Jun 4, 04:00:00 GMT+0
Completed - Maintenance has completed successfully.

Duration: 2 hours and 30 minutes

Affected Components: Toll-free Call Routing, Outbound Call Routing (International), Zultys, Inbound Call Routing, Outbound Call Routing (North America)

Apr 22, 16:15:00 GMT+0
Investigating - Problem: several customers are reporting dropped calls or loss of audio during a call. Scope: We think the problem is only affecting a small portion of customers - under 10%.

Apr 22, 16:35:00 GMT+0
Identified - Update: We have identified high CPU usage on one of our core routing servers. We are rerouting traffic through our redundant server now. .

Apr 22, 18:00:00 GMT+0
Monitoring - Update: our software vendor identified a bug that was causing the high CPU usage and advised us to roll the software back. We completed the software rollback and initial observations appear the problem is resolved. .

Apr 22, 18:45:00 GMT+0
Resolved - Update: We have monitored traffic flows since we performed the rollback and confirmed the problem is resolved..

Duration: 2 hours and 9 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Outbound Call Routing (International), Outbound Call Routing (North America)

Apr 17, 21:26:07 GMT+0
Resolved - This incident has been resolved..

Apr 17, 19:17:00 GMT+0
Identified - Status: identified Problem: We had a minor problem with a virtual machine host in our data center that caused 9 systems to go offline, move to a different machine, and reboot. This outage lasted about 10 minutes and all customer systems are fully functional now. One of our core routing servers was among the 9 systems and has been corrupted. We are rebuilding it. This may cause some faxes to fail and some calls may take longer to connect. .

Duration: 2 hours

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Mar 22, 10:00:01 GMT+0
Identified - Maintenance is now in progress.

Mar 22, 12:00:00 GMT+0
Completed - Maintenance has completed successfully.

Mar 22, 10:00:00 GMT+0
Identified - Data Center support staff will be performing emergency maintenance between the hours of 3:00am-5:00am Saturday March 22nd. During this time, some customers may experience service interruptions. .

Duration: 3 days, 4 hours and 13 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Mar 21, 14:02:18 GMT+0
Identified - Status: identified Problem: Automated network monitoring has detected problems with our systems overnight. We believe this may cause phones and apps may be unable to connect to the system, or experience phone calls with no audio Scope: we believe this is impacting a subset of customers in one of our data centers Cause: We think this is related to the problems we had on Monday night. Update: We are working with support engineers at Rogers and Telus to resolve the issue as quickly as possible.

Mar 21, 14:11:31 GMT+0
Monitoring - Update: Our systems appear to be fully operational. We are doing further diagnosis and waiting for more information from support teams about the cause of the problem..

Mar 24, 18:15:33 GMT+0
Resolved - Update: We have received confirmation from our data center support that network maintenance performed by Rogers early in the morning on Saturday has resolved this issue. Our network diagnostics confirm increased stability since these changes. .

Duration: 6 days, 17 hours and 41 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Mar 24, 18:15:04 GMT+0
Resolved - Update: We have received confirmation from our data center support that network maintenance performed by Rogers early in the morning on Saturday has resolved this issue. Our network diagnostics confirm increased stability since these changes. .

Mar 18, 00:34:00 GMT+0
Investigating - Status: investigating Scope: unknown Problem: we are currently investigating a problem with one-way audio and login failures impacting several customers.

Mar 18, 14:54:34 GMT+0
Monitoring - Status: monitoring Scope: we believe this impacted all customers in one of our data centers that have Telus internet on their premise - estimated about 20% of customers Cause: there was a core networking "BGP" problem between Rogers and Telus. The source of this issue was identified to be a failing router in Rogers core network Update: Rogers has implemented a fix and we are monitoring all networks for any further issues.

Mar 18, 02:52:13 GMT+0
Identified - Status: identified Scope: we believe this is impacting 30% of our customers Problem: customers report intermittent connection issues with phones and apps, causing no audio on phone calls and unable to call out for 2-5 minute intervals Cause: there is a core networking "BGP" problem between Rogers and Telus Update: we have opened tickets with all involved parties and made them the highest priority.

Duration: 2 hours and 19 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Voicemail to Email, RingCentral, Voicemail Transcription, Text to Speech, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America), Advanced IVR/AA Services

Mar 11, 16:06:02 GMT+0
Investigating - Status: investigating Scope: some outbound faxes may fail; some customer systems may have experienced a brief service disruption Problem: investigating.

Mar 11, 17:11:38 GMT+0
Identified - Status: identified Scope: some outbound faxes may fail; some customers may experience congestion on inbound or outbound calls Problem: one of our call routing servers crashed and all traffic is currently being routed through our redundant server only Update: we are rebuilding the server now.

Mar 11, 18:24:33 GMT+0
Resolved - Status: resolved Update: the server has been rebuilt and is now processing calls Cause: the cause appears to be an issue with one of the server hosts in our data center. All machines were automatically migrated to a redundant host, however this migration caused the call routing server to crash.

Duration: 3 hours and 17 minutes

Affected Components: Zultys

Jan 26, 21:25:52 GMT+0
Identified - We are continuing maintenance, but no further impact to customers is expected. .

Jan 26, 23:17:12 GMT+0
Completed - Maintenance completed.

Jan 26, 20:00:00 GMT+0
Identified - We are currently performing maintenance activities. No impact is expected for most customers. A small number of customers may experience momentary app and/or phone call disconnects. .

Duration: 1 hour and 26 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Jan 3, 21:31:00 GMT+0
Resolved - Status: Resolved Update: We detected network connectivity issues between our data centers. We analyzed the impact and determined it to be no or very low impact to 99% of customers. We opened support tickets with appropriate channels. The problem is now resolved..

Jan 3, 19:31:00 GMT+0
Investigating - Status: investigating Scope: minor Problem: diagnostic tools have detected networking issues between our data centers; a few customers have reported one-way audio on some calls..

Duration: 1 hour and 51 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Dec 5, 17:28:05 GMT+0
Resolved - Update: The server has been repaired and all traffic is flowing normally This incident is now resolved.

Dec 5, 15:00:00 GMT+0
Identified - Problem: Our monitoring systems detected a problem with one of our core servers early this morning Status: Problem has been identified and repair is in progress Impact: Estimated 10% of customers will experience intermittent call failures or dropped calls until resolved .

Duration: 1 hour and 17 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Oct 15, 15:43:00 GMT+0
Investigating - Problem: customers are reporting choppy audio on phone calls, or phones continuing to ring and unable to be answered Status: investigating Impact: estimated about 30% of customer base Cause: diagnostic tools are reporting very high latency between our monitoring points and one of our data centers. We are investigating..

Oct 15, 16:34:00 GMT+0
Monitoring - Update: we have implemented a manual failover to our redundant firewall. We are monitoring the issue but initial results appear the problem is resolved..

Oct 15, 17:00:00 GMT+0
Resolved - Update: we are confident the problem is now resolved. Cause: the root cause is still under investigation..

Duration: 10 hours and 12 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Oct 2, 17:09:22 GMT+0
Identified - Problem: Two IP pools used by customer systems at one of our data centers are not connecting properly. This is causing all impacted systems to be unable to connect to phones and applications. Systems are still able to take voicemail messages. Impact: Roughly 40% of customers .

Oct 2, 18:28:25 GMT+0
Identified - Update: Data centre engineers are working with the internet providers at the highest priority to resolve this issue.

Oct 2, 23:39:07 GMT+0
Identified - Update: As of 4:30pm, half the impacted customers are now fixed. We will continue to work into the evening..

Oct 3, 02:57:39 GMT+0
Resolved - Resolution: All customers have been returned to service on an alternate fiber connection as of 7:30pm. A small number of customers will need to reboot their phones. We will be reaching out to these customers tomorrow..

Oct 2, 16:46:03 GMT+0
Investigating - Problem: Network connectivity has been lost to some customer phone systems Status: Under investigation Impact: Under investigation Cause: Under investigation.

Oct 2, 21:05:51 GMT+0
Identified - Update: We are implementing a manual workaround to regain access to all systems. Unfortunately this must be done one customer at a time. Impact: The number of customers impacted was about 40% at peak, it is now about 33% as our technicians work to reconfigure each system manually. Root Cause: A Rogers backbone fiber feeding the data center has been damaged. Rogers fiber team is onsite but there is no ETA for the repair, so work-arounds are being implemented at all levels..

Duration: 15 minutes

Affected Components: Zultys

Jun 14, 15:24:00 GMT+0
Investigating - Problem: We are currently investigating a significant outage.

Jun 14, 15:39:00 GMT+0
Resolved - Problem: significant outage Status: This outage is now resolved, all systems are functioning normally Impact: We think over 50% of customers were impacted for a short period of time Cause: still under investigation.

Jun 17, 16:04:45 GMT+0
Resolved - There were two different factors that caused this outage: 1\. Primary internet connection failure at one of our data centers 2\. Human error by the data center provider, which caused the redundant backup connection not to function properly. .

Duration: 4 hours and 14 minutes

Affected Components: Toll-free Call Routing

Jun 6, 20:00:00 GMT+0
Resolved - Status: The problem is resolved. All calls are working now. .

Jun 6, 15:45:35 GMT+0
Investigating - Problem: We have about 40 toll-free numbers that are receiving a busy signal when called right now Status: We are investigating this issue Impact: We think this is impacting 40 of our toll-free numbers Cause: We are investigating this issue.

Jun 6, 16:14:05 GMT+0
Identified - Problem: We have about 40 toll-free numbers that are receiving a busy signal when called right now. The problem is intermittent. Some calls are able to get through. Status: This is a service provider issue. Telus is investigating now..

Jun 6, 17:02:39 GMT+0
Identified - Problem: We have about 40 toll-free numbers that are receiving a busy signal when called right now. The problem is intermittent. Some calls are able to get through. Status: This issue is impacting a number of Telus large enterprise customers. Telus tier 4 team is troubleshooting urgently. .

Jun 6, 17:41:33 GMT+0
Identified - Status: Reports from Telus and our own internal testing indicate the problem is now resolved. .

Jun 6, 18:10:24 GMT+0
Identified - Status: This issue is still occurring when calling from Bell cellphones. Telus is investigating further. When calling from other major carriers the calls are successful. .

Jun 7, 23:34:18 GMT+0
Resolved - Cause: Telus explained the root cause of this incident was due to a minor programming error made by a Telus technician. The minor error propagated to the toll-free databases of all major carriers before it was discovered. Once it was corrected, it took additional time to propagate to all carriers. .

Duration: 2 hours and 34 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, RingCentral, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

May 29, 22:51:00 GMT+0
Resolved - Cause: The cause of this problem was a faulty server in the data center that began to flood the network with garbage packets at a high enough rate to reduce the performance of the entire network..

May 29, 16:30:00 GMT+0
Investigating - Problem: Customers are reporting degraded call quality on some or all calls Status: We are investigating this problem Impact: Currently unknown Cause: We are investigating this problem.

May 29, 16:40:00 GMT+0
Identified - Problem: Customers are reporting minor degraded call quality on some or all calls Status: We have identified the problem, data center engineers are working to resolve as soon as possible Impact: This problem may impact up to 50% of our customers Cause: The root cause is still under investigation.

May 29, 19:04:00 GMT+0
Resolved - Problem: Customers reported minor degraded call quality on some or all calls Status: The problem is now resolved Cause: The root cause is still under investigation.

Duration: 2 days and 17 hours

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

May 9, 20:25:00 GMT+0
Investigating - Problem: Some customers are reporting a complete outage - phones say no service, apps cannot not connect to systems Status: Investigating Impact: Roughly 30% of our customers are impacted Cause: We are investigating the cause of this problem.

May 9, 23:38:00 GMT+0
Resolved - Cause: The cause of this outage was due to a high level software update our data center provider attempted to perform. We were warned in advance of the update, but were told it would not cause any interruption to our systems. During the update process, the provider encountered an error which required them roll back the update. The roll back caused the brief interruption to our systems. The provider has stated they will perform future updates of this magnitude after-hours only. .

May 10, 04:01:00 GMT+0
Resolved - Status: This incident is now fully resolved..

May 9, 21:30:00 GMT+0
Resolved - Status: Temporarily resolved, further maintenance to be done after hours Impact: Roughly 30% of customers were impacted for a period of approximately 5 minutes, 4% of customers reported an extended outage lasting approximately 1 hour. Cause: We are still investigating the cause of this problem.

Duration: 4 hours and 4 minutes

Affected Components: Inbound Call Routing, Toll-free Call Routing, Zultys

Apr 12, 15:28:00 GMT+0
Identified - Problem: Inbound call failures Status: Engineers from Telus have already identified the problem Impact: This is a minor incident impacting approximately 1% of our customers Cause: The problem is with Telus and they are already working to fix it.

Apr 12, 18:24:09 GMT+0
Identified - Status Update: The Telus Tier 3 team is working diligently to rectify the situation.

Apr 12, 19:32:22 GMT+0
Resolved - Status Update: Telus reports the issue has been fully resolved. If you experience any further issues, please contact us..

Duration: 1 day and 15 hours

Affected Components: Zultys, Outbound Call Routing (International), Outbound Call Routing (North America)

Nov 29, 20:18:00 GMT+0
Resolved - We identified the problem and implemented a temporary fix. We are monitoring traffic and awaiting a permanent fix. .

Nov 29, 22:07:09 GMT+0
Resolved - This incident has been resolved..

Nov 29, 19:39:00 GMT+0
Investigating - We observed an increased number of outbound call failures starting at 11:39AM this morning. Some customers reported outbound call failures on some calls. When they tried dialing the number again it would go through. .