Unified - Internet2 IP Network Operations Report
09/07/2015 - 09/13/2015

Unified - Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

54983-Elevated 1-Critical Outage Resolved - I2 Optical DWDM System HART2-NEWY Cut or Damaged FiberVendor 09/08/2015 5:13 PM 09/09/2015 2:00 AM
55073-Elevated 1-Critical Brief Outage Resolved - I2 Optical DWDM System ATLA-JACK UndeterminedVendor 09/09/2015 5:39 AM 09/09/2015 5:42 AM
55411-Critical 1-Critical Outage Resolved - I2 Optical DWDM System MEMP-NASH Cut or Damaged FiberVendor 09/10/2015 8:23 PM 09/11/2015 8:09 AM

Unscheduled Outages Detail


Ticket No.:5498:144
Subject:Outage Resolved - I2 Optical DWDM System HART2-NEWY
Source Of Impact:Vendor
Affected:DWDM System HART2-NEWY
Optical Member ESnet
Optical Member NOX
Optical Member NOAA N-Wave
Optical Member OARnet
Optical Member NORDUnet
AL2S Backbone HART2-NEWY
AL2S Participant NORDUnet
IP Backbone CLEV-NEWY
IP Backbone CHIC-NEWY
IP Connector OARnet
Start Time:Tuesday, September 8, 2015, 5:13 PM (1713) UTC
End Time:Wednesday, September 9, 2015, 2:00 AM (0200) UTC
Description:The above listed assets were unavailable to the community. Level (3) Communications Engineers have reported the outage was due to a ladder that was left in a manhole which caused bent fibers. The fibers have been repaired. No further outages are expected.

Ticket No.:5507:144
Subject:Brief Outage Resolved - I2 Optical DWDM System ATLA-JACK
Source Of Impact:Vendor
Affected:DWDM System ATLA-JACK
AL2S Backbone(s) ATLA-JACK
IP Backbone(s) ATLA-HOUS
All Connectors/Peers
Start Time:Wednesday, September 9, 2015, 5:39 AM (0539) UTC
End Time:Wednesday, September 9, 2015, 5:42 AM (0542) UTC
Description:The above listed assets were briefly unavailable to the community. Level (3) Communications Engineers have investigated the reason for the outage. They were unable to determine an RFO.

Ticket No.:5541:144
Subject:Outage Resolved - I2 Optical DWDM System MEMP-NASH
Source Of Impact:Vendor
Affected:DWDM System MEMP-NASH
AL2S Backbone ATLA-JCSN
Optical Member ESnet
Optical Member NOAA N-Wave
Optical Member MissiON
IP Connector MEMP
Start Time:Thursday, September 10, 2015, 8:23 PM (2023) UTC
End Time:Friday, September 11, 2015, 8:09 AM (0809) UTC
Description:The above listed assets are now available to the community. Level(3) Communications Engineers have investigated a fiber cut between Mt Pleasant, TN and Nashville, TN. Splice crews have created an alternate route via aerial fiber as a temporary solution to restore services. Maintenance has been completed. Notifications for a planned maintenance to permanently repair the damage will be provided at a later date. Updates will be provided as they are received.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

None during this report period

Downtime and Availability



Unified - Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%

Aggregate Unified - Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%100.00000%100.00000%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Unified - Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
I2-ATLA-CLEV-VLAN-14832 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-CLEV-VLAN-14837 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14833 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14838 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14831 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14834 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.44938%99.79831%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 0 hr 3 min0 hr 3 min 99.97024%99.97024% 99.52186%99.96319%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.72735%99.84271%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.86510%99.99905%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.90798%99.99564%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 8 hr 47 min8 hr 47 min 94.77183%94.77183% 99.86377%99.89469%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.85523%99.88654%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.62280%99.81235%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.28241%99.41712%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.85390%99.97078%
R&E 10G Backbone: I2-LOSA-PAIX-VLAN-23105 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99184%100.00000%
Commodity 10G Backbone: I2-LOSA-PAIX-VLAN-23106 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99184%100.00000%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95959%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80077%99.88521%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80077%99.88521%
R&E 100G Backbone: I2-LOSA-WILC-VLAN-23103 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98918%99.99602%
Commodity 100G Backbone: I2-LOSA-WILC-VLAN-23104 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98918%99.99602%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89659%99.95959%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96471%99.99962%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96471%99.99962%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.72735%99.84271%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.65771%99.90912%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 3 min0 hr 3 min 99.97024%99.97024% 99.36779%99.72298%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93871%100.00000%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93245%99.99829%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95105%99.99981%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97647%99.99962%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93378%99.93815%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.77212%99.87306%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.74822%99.82734%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93321%99.99981%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93796%100.00000%
Legacy 10G Backbone: I2-LOSA-PAIX-10GE-05853 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98805%100.00000%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98103%100.00000%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.83493%99.85732%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%99.99981%
Legacy 100G Backbone: I2-LOSA-WILC-100GE-23893 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99317%99.99810%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99658%99.99981%

Aggregate All Unified - Internet2 IP Backbone Circuits 8 hr 53 min0 hr 0 min 99.89209%99.89209% 99.87693%99.94422%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:5528:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:
Start Time:Thursday, September 17, 2015, 10:45 PM (2245) UTC
End Time:Friday, September 18, 2015, 12:45 AM (0045) UTC
Description: Optical Member ESnet's Optical circuit ESNET-FNALFCC-STAR-10GE-12167 and ESNET-FNALFCC-STAR-10GE-07952 will be unavailable while ESnet Engineers perform maintenance. A brief outage is expected; however, the entire maintenance window has been reserved.

Ticket No.:5518:144
Subject:Maintenance *Tracking* 2 of 2 - I2 Various Optical Circuits (Northern Tier)
Affected:
Start Time:Friday, September 18, 2015, 6:00 AM (0600) UTC
End Time:Friday, September 18, 2015, 11:00 AM (1100) UTC
Description: The above listed assets should remain available while Zayo Engineering perform maintenance. The entire window is reserved.

Ticket No.:5480:144
Subject:Maintenance *Tracking* – I2 IP Node oob.newy1118th
Affected:oob.newy
Start Time:Saturday, September 19, 2015, 4:00 AM (0400) UTC
End Time:Saturday, September 19, 2015, 6:00 AM (0600) UTC
Description: The above listed asset may become unavailable while Level(3) engineers perform maintenance. No outage is expected; however, the entire window is reserved.

Ticket No.:5527:144
Subject:Maintenance - I2 Various Optical Circuits (MINN)
Affected:AL2S Backbone MINN-MISS2
Optical Member Canarie
Net+ Provider Code42
Optical Member NOAA N-Wave
Start Time:Sunday, September 20, 2015, 1:00 PM (1300) UTC
End Time:Monday, September 21, 2015, 1:00 AM (0100) UTC
Description: The above listed circuits may experience an outage of up to 3 hours while fiber provider Zayo performs a hot cut maintenance near Laurel, MT. The entire maintenance window is reserved.

Ticket No.:5571:144
Subject:Maintenance - I2 Optical Various Circuits
Affected:AL2S Backbone HART2-NEWY32AOA
IP Backbone LOSA-SEAT
IP Backbone CHIC-NEWY32AOA
IP Backbone CLEV-NEWY32AOA
IP Connector University of Oregon
Start Time:Monday, September 21, 2015, 9:00 AM (0900) UTC
End Time:Monday, September 21, 2015, 10:00 AM (1000) UTC
Description: The above listed circuits will be unavailable while Internet2 Engineers perform circuit maintenance. A brief outage of 5 minutes is expected, however the entire maintenance window has been reserved.

Ticket No.:5378:144
Subject:Maintenance 1 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
Optical Member ESnet
Optical Member MissiON
Optical Member NOAA N-Wave
AL2S Backbone ATLA-JCSN
Start Time:Tuesday, September 22, 2015, 5:00 AM (0500) UTC
End Time:Tuesday, September 22, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo Engineers perform maintenance to replace a splice case in Memphis, TN. An outage of up to 30 minutes is expected. The entire window is reserved. This is maintenance window 1 of 4.

Ticket No.:5379:144
Subject:Maintenance 2 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
Optical Member ESnet
Optical Member MissiON
Optical Member NOAA N-Wave
AL2S Backbone ATLA-JCSN
Start Time:Wednesday, September 23, 2015, 5:00 AM (0500) UTC
End Time:Wednesday, September 23, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo Engineers perform maintenance to replace a splice case in Memphis, TN. An outage of up to 30 minutes is expected. The entire window is reserved. This is maintenance window 2 of 4.

Ticket No.:5616:144
Subject:Maintenance - I2 Optical Circuit I2-BOIS-EQCH-10GE-09122
Affected:Optical Member Education Networks of America
Start Time:Wednesday, September 23, 2015, 9:00 AM (0900) UTC
End Time:Wednesday, September 23, 2015, 10:00 AM (1000) UTC
Description: ENA's Optical circuit I2-BOIS-EQCH-10GE-09122 will be unavailable while Internet2 Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:5380:144
Subject:Maintenance 3 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
Optical Member ESnet
Optical Member MissiON
Optical Member NOAA N-Wave
AL2S Backbone ATLA-JCSN
Start Time:Thursday, September 24, 2015, 5:00 AM (0500) UTC
End Time:Thursday, September 24, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo Engineers perform maintenance to replace a splice case in Memphis, TN. An outage of up to 30 minutes is expected. The entire window is reserved. This is maintenance window 3 of 4.

Ticket No.:5625:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:Optical Member ESnet
Start Time:Thursday, September 24, 2015, 7:00 AM (0700) UTC
End Time:Thursday, September 24, 2015, 9:00 AM (0900) UTC
Description: Member ESnet's Optical circuit ESNET-SACR-SEAT-100GE-07828 will be unavailable while Level 3 Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:5381:144
Subject:Maintenance 4 of 4 - I2 Optical DWDM System JCSN-MEMP
Affected:DWDM System JCSN-MEMP
Optical Member ESnet
Optical Member MissiON
Optical Member NOAA N-Wave
AL2S Backbone ATLA-JCSN
Start Time:Friday, September 25, 2015, 5:00 AM (0500) UTC
End Time:Friday, September 25, 2015, 10:00 AM (1000) UTC
Description: The above listed assets will be unavailable while Zayo Engineers perform maintenance to replace a splice case in Memphis, TN. An outage of up to 30 minutes is expected. The entire window is reserved. This is maintenance window 4 of 4.

Ticket No.:5606:144
Subject:Maintenance - I2 AL2S Backbone CHIC-COLU4
Affected:AL2S Backbone CHIC-COLU4
Start Time:Saturday, September 26, 2015, 5:00 AM (0500) UTC
End Time:Saturday, September 26, 2015, 7:00 AM (0700) UTC
Description: The above listed assets will be unavailable while Level 3 Engineers perform maintenance. An outage is expected, and the entire window is reserved.

Ticket No.:5543:144
Subject:Maintenance - I2 IP Node oob.wash
Affected:Level(3) High Speed IP Ethernet for OOB router in Washington DC
Start Time:Tuesday, September 29, 2015, 4:00 AM (0400) UTC
End Time:Tuesday, September 29, 2015, 10:00 AM (1000) UTC
Description: The above listed asset will be unavailable while Level(3) performs maintenance to migrate circuits to new optimum path to improve reliability. No impact is expected, however the entire window is reserved.