Unified - Internet2 IP Network Operations Report
12/07/2015 - 12/13/2015

Unified - Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

65654-Normal 1-Critical Brief Outage Resolved - I2 Optical DWDM System ATLA-CHAR PowerVendor 12/07/2015 4:02 PM 12/07/2015 4:07 PM
65974-Normal 2-High Outage Resolved- I2 IP Backbones ATLA-CLEV/CHIC-CLEV HardwareVendor 12/09/2015 1:17 PM 12/09/2015 2:40 PM
66211-Critical 1-Critical Outage Resolved - I2 Optical DWDM System MEMP-NASH Unannounced MaintenanceVendor 12/11/2015 12:00 PM 12/11/2015 1:37 PM
66354-Normal 1-Critical Brief Outage Resolved - I2 Optical DWDM System ASHB-PITT Unannounced MaintenanceVendor 12/12/2015 1:09 PM 12/12/2015 1:15 PM

Unscheduled Outages Detail


Ticket No.:6565:144
Subject:Brief Outage Resolved - I2 Optical DWDM System ATLA-CHAR
Source Of Impact:Vendor
Affected:DWDM System ATLA-CHAR
AL2S Backbones ATLA-WASH & ATLA-CHAR
IP Backbone ATLA-WASH
I2 Optical Member NOAA N-Wave
I2 Optical Member MCNC
I2 Optical Member ESnet
Start Time:Monday, December 7, 2015, 4:02 PM (1602) UTC
End Time:Monday, December 7, 2015, 4:07 PM (1607) UTC
Description:The above listed assets are currently available to the community after a brief outage. Level (3) Communications Engineers determined that the outage was caused by power disruption. Power has been restored and no further outages are expected.

Ticket No.:6597:144
Subject:Outage Resolved- I2 IP Backbones ATLA-CLEV/CHIC-CLEV
Source Of Impact:Vendor
Affected:AL2S Interconnect to IP
IP Backbone ATLA-CLEV
IP Backbone CHIC-CLEV
Start Time:Wednesday, December 9, 2015, 1:17 PM (1317) UTC
End Time:Wednesday, December 9, 2015, 2:40 PM (1440) UTC
Description:The above listed assets are available. Internet2 Engineers processed an Emergency Maintenance to restart the card. This process resolved the outage condition. No further outages are expected.

Ticket No.:6621:144
Subject:Outage Resolved - I2 Optical DWDM System MEMP-NASH
Source Of Impact:Vendor
Affected:DWDM System MEMP-NASH
AL2S Backbone ATLA-JCSN
Optical Member ESnet
Optical Member NOAA N-Wave
Optical Member MissiON
Start Time:Friday, December 11, 2015, 12:00 PM (1200) UTC
End Time:Friday, December 11, 2015, 1:37 PM (1337) UTC
Description:The above listed assets and entities are available. Level(3) engineers have completed processing extended maintenance to repair damaged cable in Brentwood,TN. No further outages are expected. *Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone. *Backbone traffic will re-route around the impacted assets.

Ticket No.:6635:144
Subject:Brief Outage Resolved - I2 Optical DWDM System ASHB-PITT
Source Of Impact:Vendor
Affected:DWDM System ASHB-PITT
AL2S Backbones ASHB-CHIC, ASHB-PITT, and CLEV-WASH
IP Backbones ATLA-CLEV, CHIC-WASH, and CLEV-WASH
Connector 3ROX (ASHB)
Customer NORDUnet
Start Time:Saturday, December 12, 2015, 1:09 PM (1309) UTC
End Time:Saturday, December 12, 2015, 1:15 PM (1315) UTC
Description:The above listed assets are currently available to the community after a brief outage. Level (3) Communications Engineers have reported this was due to a fiber disturbance on a maintenance site, outside the scheduled maintenance window.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

6507 3-Elevated 1-Critical Maintenance 2 of 2 Completed - I2 Optical DWDM System MEMP-NASH Circuit Vendor12/11/2015 6:04 AM 12/11/2015 12:00 PM
6540 3-Elevated 1-Critical Maintenance Completed 1 of 2 - I2 Optical DWDM System ASHB-PITT CircuitVendor 12/12/2015 9:42 AM 12/12/2015 10:05 AM

Scheduled Maintenances Detail

Ticket No.:6507:144
Subject:Maintenance 2 of 2 Completed - I2 Optical DWDM System MEMP-NASH
Source Of Impact:Vendor
Affected:DWDM System MEMP-NASH
AL2S Backbone ATLA-JCSN
Optical Member ESnet
Optical Member NOAA N-Wave
Optical Member MissiON
Start Time:Friday, December 11, 2015, 6:04 AM (0604) UTC
End Time:Friday, December 11, 2015, 12:00 PM (1200) UTC
Description: The above listed assets remain unavailable. Level(3) engineers are still in process on the maintenance to repair damaged cable in Brentwood,TN.. This maintenance window has expired. The event will be tracked further via an outage ticket, 6612:144.

Ticket No.:6540:144
Subject:Maintenance Completed 1 of 2 - I2 Optical DWDM System ASHB-PITT
Source Of Impact:Vendor
Affected:DWDM System ASHB-PITT
AL2S Backbones ASHB-CHIC, and ASHB-PITT
IP Backbones ATLA-CLEV, CHIC-WASH, and CLEV-WASH
Connector 3ROX (ASHB)
Customer NORDUnet
Start Time:Saturday, December 12, 2015, 9:42 AM (0942) UTC
End Time:Saturday, December 12, 2015, 10:05 AM (1005) UTC
Description: The above listed assets was unavailable while Level(3) Communications performed a fiber maintenance in Frostburg, MD. This window of the maintenance has been completed.


Downtime and Availability



Unified - Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability
ASHB MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%

Aggregate Unified - Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%100.00000%100.00000%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Unified - Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
Layer3 AL3S Backbone ASHB-ATLA R&E VLAN 55199 0 hr 5 min0 hr 5 min 99.95040%99.95040% 99.99905%99.99905%
Layer3 AL3S Backbone ASHB-ATLA Commodity VLAN 55204 0 hr 5 min0 hr 5 min 99.95040%99.95040% 99.99905%99.99905%
Layer3 AL3S Backbone ASHB-CHIC R&E VLAN 55198 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Layer3 AL3S Backbone ASHB-CHIC Commodity VLAN 55203 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99526%100.00000%
Layer3 AL3S Backbone ASHB-CLEV R&E VLAN 55197 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99886%100.00000%
Layer3 AL3S Backbone ASHB-CLEV Commodity VLAN 55202 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99886%100.00000%
Layer3 AL3S Backbone ASHB-NEWY R&E VLAN 55201 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Layer3 AL3S Backbone ASHB-NEWY Commodity VLAN 55206 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Layer3 AL3S Backbone ASHB-WASH R&E VLAN 55200 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98425%100.00000%
Layer3 AL3S Backbone ASHB-WASH Commodity VLAN 55205 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98425%100.00000%
I2-CHIC-WASH-VLAN-12131 0 hr 29 min0 hr 6 min 99.71230%99.94048% 99.95370%99.99753%
I2-CHIC-WASH-VLAN-12132 0 hr 29 min0 hr 6 min 99.71230%99.94048% 99.95370%99.99753%
I2-ATLA-CLEV-VLAN-14832 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-CLEV-VLAN-14837 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14833 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14838 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14831 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14834 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.67460%99.82525%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.72146%99.81861%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 5 min0 hr 5 min 99.95040%99.95040% 99.79148%99.84195%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 1 hr 23 min1 hr 23 min 99.17659%99.17659% 99.86699%99.96433%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92240%99.99583%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.82411%99.84726%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 0 hr 29 min0 hr 6 min 99.71230%99.94048% 99.85902%99.89071%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.49454%99.62223%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.58732%99.65430%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.91557%99.98027%
R&E 10G Backbone: I2-LOSA-PAIX-VLAN-23105 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98956%100.00000%
Commodity 10G Backbone: I2-LOSA-PAIX-VLAN-23106 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98956%100.00000%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89261%99.93264%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.77971%99.88559%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.77971%99.88559%
R&E 100G Backbone: I2-LOSA-WILC-VLAN-23103 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98918%99.99602%
Commodity 100G Backbone: I2-LOSA-WILC-VLAN-23104 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98918%99.99602%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.82411%99.89849%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.71596%99.74784%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.71596%99.74784%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318 0 hr 5 min0 hr 5 min 99.95040%99.95040% 99.79148%99.84195%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.86434%99.93606%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.81292%99.83796%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 5 min0 hr 5 min 99.95040%99.95040% 99.99279%99.99734%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95294%99.98065%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97666%99.99981%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 0 hr 29 min0 hr 6 min 99.71230%99.94048% 99.92524%99.93701%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.63323%99.68257%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98577%99.99962%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97344%99.99962%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98672%100.00000%
Legacy 10G Backbone: I2-LOSA-PAIX-10GE-05853 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98577%100.00000%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.90741%99.93359%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84081%99.85770%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%99.99981%
Legacy 100G Backbone: I2-LOSA-WILC-100GE-23893 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99450%99.99829%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.74480%99.74803%

Aggregate All Unified - Internet2 IP Backbone Circuits 3 hr 44 min1 hr 32 min 99.96357%99.97853% 99.91080%99.93892%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:6738:144
Subject:Maintenance - I2 AL2S Various Circuits (DALL3)
Affected:Connector LEARN
Start Time:Monday, December 28, 2015, 10:00 AM (1000) UTC
End Time:Monday, December 28, 2015, 12:00 PM (1200) UTC
Description: The above listed will be unavailable while Internet2 Engineers re-groom various customers utilizing Internet2 Core Router DALL3. An outage of 30 minutes is expected, however the entire maintenance window has been reserved.

Ticket No.:6735:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:
Start Time:Tuesday, December 29, 2015, 4:00 PM (1600) UTC
End Time:Tuesday, December 29, 2015, 8:00 PM (2000) UTC
Description: Member ESnet's Optical circuit ESNET-LBNL-SUNN-100GE-07835 will be unavailable while their engineers perform maintenance. An outage is expected. The entire maintenance window has been reserved.

Ticket No.:6753:144
Subject:Maintenance - I2 Optical Various Circuits (DALL)
Affected:
Start Time:Wednesday, December 30, 2015, 10:00 AM (1000) UTC
End Time:Wednesday, December 30, 2015, 12:00 PM (1200) UTC
Description: The above listed assets will be unavailable while Internet2 Engineers perform a maintenance to re-groom various customers that are utilizing Internet2 TR-CPS Router at DALL3. An outage of up to 30 minutes is expected; however the entire window is reserved.

Ticket No.:6657:144
Subject:Maintenance - I2 Optical DWDMs CLEV-DETR, DETR-EQCH DWDM
Affected:DWDM System CLEV-DETR
DWDM System DETR-EQCH
AL2S Backbones(s) ASHB-CHIC and CLEV-EQCH
IP Backbone CHIC-CLEV, CHIC-NEWY, and CHIC-WASH
Customer NORDUnet
Customer Merit Network
Optical Member ESnet
Optical Member NOAA N-Wave
Start Time:Thursday, January 7, 2016, 6:00 AM (0600) UTC
End Time:Thursday, January 7, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Internet2 Engineers move the optical node in Detroit to a new rack location. Outages are expected. The entire window is reserved.

Ticket No.:6648:144
Subject:Maintenance 1 of 2 - I2 Optical DWDM System ASHB-PITT
Affected:DWDM System ASHB-PITT
AL2S Backbone(s) ASHB-PITT and ASHB-CHIC
IP Backbone(s) ATLA-CLEV, CHIC-WASH, CLEV-WASH
Connector 3ROX (ASHB)
Customer NORDUnet
Optical Member ESnet
Optical Member NOAA N-Wave
Start Time:Saturday, January 9, 2016, 5:00 AM (0500) UTC
End Time:Saturday, January 9, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Level(3) Communications perform maintenance to relocate cable in Hancock, MD. An outage of 6 hours is expected; however, the entire window is reserved.

Ticket No.:6650:144
Subject:Maintenance 2 of 2 - I2 Optical DWDM System ASHB-PITT
Affected:DWDM System ASHB-PITT
AL2S Backbone(s) ASHB-PITT and ASHB-CHIC
IP Backbone(s) ATLA-CLEV, CHIC-WASH, CLEV-WASH
Connector 3ROX (ASHB)
Customer NORDUnet
Optical Member ESnet
Optical Member NOAA N-Wave
Start Time:Sunday, January 10, 2016, 5:00 AM (0500) UTC
End Time:Sunday, January 10, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Level(3) Communications perform maintenance to relocate cable in Hancock, MD. An outage of 6 hours is expected; however, the entire window is reserved. This is an alternate maintenance window if needed.

Ticket No.:6249:144
Subject:Maintenance - I2 Core Node SING
Affected:Core Node Singapore and its Connectors and Participants
Start Time:Thursday, January 14, 2016, 1:00 AM (0100) UTC
End Time:Friday, January 22, 2016, 1:00 PM (1300) UTC
Description: The above listed asset is expected to remain available while TATA performs power maintenance. This power maintenance is set to happen one feed at a time to avoid unnecessary impact. It is a multi-day event. No outages are expected. The entire window is reserved. All connections to the Core Node will be affected in the unlikely event of impact by this maintenance.