TOPS - Medium Incident – Incident details

Medium Incident

Resolved
Operational
Started about 2 months agoLasted 7 days

Affected

Unified Communication Systems

Degraded performance from 12:34 AM to 2:54 PM, Operational from 2:54 PM to 6:15 PM

Zultys

Degraded performance from 12:34 AM to 2:54 PM, Operational from 2:54 PM to 6:15 PM

Voice Services

Degraded performance from 12:34 AM to 2:54 PM, Operational from 2:54 PM to 6:15 PM

Inbound Call Routing

Degraded performance from 12:34 AM to 2:54 PM, Operational from 2:54 PM to 6:15 PM

Toll-free Call Routing

Degraded performance from 12:34 AM to 2:54 PM, Operational from 2:54 PM to 6:15 PM

Outbound Call Routing (North America)

Degraded performance from 12:34 AM to 2:54 PM, Operational from 2:54 PM to 6:15 PM

Updates
  • Resolved
    Resolved

    Update: We have received confirmation from our data center support that network maintenance performed by Rogers early in the morning on Saturday has resolved this issue. Our network diagnostics confirm increased stability since these changes.

  • Monitoring
    Monitoring

    Status: monitoring

    Scope: we believe this impacted all customers in one of our data centers that have Telus internet on their premise - estimated about 20% of customers

    Cause: there was a core networking "BGP" problem between Rogers and Telus. The source of this issue was identified to be a failing router in Rogers core network

    Update: Rogers has implemented a fix and we are monitoring all networks for any further issues

  • Identified
    Identified

    Status: identified

    Scope: we believe this is impacting 30% of our customers

    Problem: customers report intermittent connection issues with phones and apps, causing no audio on phone calls and unable to call out for 2-5 minute intervals

    Cause: there is a core networking "BGP" problem between Rogers and Telus

    Update: we have opened tickets with all involved parties and made them the highest priority

  • Investigating
    Investigating

    Status: investigating

    Scope: unknown

    Problem: we are currently investigating a problem with one-way audio and login failures impacting several customers