Unified - Internet2 IP Network Operations Report
06/08/2015 - 06/14/2015

Unified - Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

42034-Normal Outage Resolved - I2 AL2S Core Node (ATLA) SoftwareVendor 06/09/2015 2:45 PM 06/09/2015 3:22 PM
42624-Normal 1-Critical Brief Outage Resolved - I2 Optical DWDM System CINC-INDI Human ErrorVendor 06/12/2015 2:48 PM 06/12/2015 2:49 PM

Unscheduled Outages Detail


Ticket No.:4203:144
Subject:Outage Resolved - I2 AL2S Core Node (ATLA)
Source Of Impact:Vendor
Affected:Core Node ATLA and its Connector(s) and Participants.
IP Backbone ASHB-CHIC
IP Backbone ATLA-CHIC
IP Backbone ATLA-CLEV
IP Backbone ATLA-HOUS
IP Backbone ATLA-NEWY32AOA
IP Backbone ATLA-WASH
Start Time:Tuesday, June 9, 2015, 2:45 PM (1445) UTC
End Time:Tuesday, June 9, 2015, 3:22 PM (1522) UTC
Description:AL2S Core Node ATLA is currently available. Internet2 Network Engineering suspects a possible software issue caused an outage to AL2S (ALTA) and AL3S (ATLA) traffic passing through the core node. The reboot of the core node resolved the outage. An RFO is being pursued with the Vendor. No further outages are expected.

Ticket No.:4262:144
Subject:Brief Outage Resolved - I2 Optical DWDM System CINC-INDI
Source Of Impact:Vendor
Affected:DWDM System CINC-INDI
AL2S Backbone ATLA-CHIC
AL2S Backbone CINC-INDI
IP Backbone ATLA-CHIC
Start Time:Friday, June 12, 2015, 2:48 PM (1448) UTC
End Time:Friday, June 12, 2015, 2:49 PM (1449) UTC
Description:DWDM System CINC-INDI was briefly unavailable to the community. Level (3) Communications Engineers have reported the outage was due to trench digging by a local water department outside of Cincinnati, OH. There was damage to other fibers but Internet2's fiber was only bumped. Technicians were deployed to fix the neighboring fibers, which could have possibly caused our circuits to bounce again. This work has been completed and services are fully restored at this time. No further impact is expected.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

4068 3-Elevated Maintenance 1 of 2 Completed - I2 AL2S Core Node Insertion in Las Vegas, NV Hardware Internal06/09/2015 7:06 AM 06/09/2015 7:25 AM
4207 1-Critical Emergency Maintenance Completed- I2 AL2S Core Node (ATLA) HardwareInternal 06/09/2015 3:19 PM 06/09/2015 3:22 PM
4073 3-Elevated 1-Critical Maintenance 1 of 2 Completed - I2 Optical DWDM System CINC-LOUI Circuit Vendor06/10/2015 6:13 AM 06/10/2015 8:10 AM
4141 3-Elevated Maintenance Completed - I2 IP Core Node ATLA SoftwareInternal 06/10/2015 4:01 AM 06/10/2015 4:06 AM
4071 3-Elevated Maintenance 2 of 2 Completed - I2 AL2S Core Node Insertion in Las Vegas, NV Hardware Internal06/10/2015 3:12 AM 06/10/2015 4:07 AM
4245 3-Elevated Maintenance Completed - I2 Optical DWDM System RENO-SACR CircuitVendor 06/11/2015 7:20 AM 06/11/2015 7:48 AM
4275 3-Elevated 1-Critical Emergency Maintenance Completed - I2 Optical DWDM System CINC-INDI Circuit Vendor06/13/2015 12:47 AM 06/13/2015 3:14 AM

Scheduled Maintenances Detail

Ticket No.:4068:144
Subject:Maintenance 1 of 2 Completed - I2 AL2S Core Node Insertion in Las Vegas, NV
Source Of Impact:Internal
Affected:AL2S Core Node LASV
AL2S Backbone LASV-SALT
AL2S Backbone LASV-LOSA
IP Backbone LOSA-SALT
Start Time:Tuesday, June 9, 2015, 7:06 AM (0706) UTC
End Time:Tuesday, June 9, 2015, 7:25 AM (0725) UTC
Description: The above listed assets were unavailable while Internet2 Engineers performed maintenance on LOSA to SALT Backbone. Backbone traffic re-routed during this maintenance. Workgroup administrators received notifications from OESS regarding circuits re-routed on the backbone. Maintenance has been completed. No further outages are expected. *This was maintenance window 1 of 2*

Ticket No.:4207:144
Subject:Emergency Maintenance Completed- I2 AL2S Core Node (ATLA)
Source Of Impact:Internal
Affected:Core Node ATLA and its Connector(s) and Participants.
Start Time:Tuesday, June 9, 2015, 3:19 PM (1519) UTC
End Time:Tuesday, June 9, 2015, 3:22 PM (1522) UTC
Description: AL2S Core Node ATLA is currently available after an emergency reboot. Internet2 Network Engineering suspects a possible software issue caused a disruption to traffic passing through the core node. The reboot of the core node resolved the outage. An RFO is being pursued with the Vendor. This maintenance has been completed.

Ticket No.:4073:144
Subject:Maintenance 1 of 2 Completed - I2 Optical DWDM System CINC-LOUI
Source Of Impact:Vendor
Affected:DWDM System CINC-LOUI
AL2S Backbone(s) ATLA-CHIC
IP Backbone(s) ATLA-CHIC
IP Connector University of Memphis and its Participants
IP Connector Indiana Gigapop and its Participants
Start Time:Wednesday, June 10, 2015, 6:13 AM (0613) UTC
End Time:Wednesday, June 10, 2015, 8:10 AM (0810) UTC
Description: DWDM System CINC-LOUI was unavailable while Level(3) Communications performed cable relocation in Rexville, IN and Cincinnati, OH. Maintenance has been completed.

Ticket No.:4141:144
Subject:Maintenance Completed - I2 IP Core Node ATLA
Source Of Impact:Internal
Affected:IP Core Node ATLA and its Connector(s) and Participants
IP Backbone ATLA-CHIC
IP Backbone ATLA-HOUS
IP Backbone ATLA-WASH
IP Backbone ATLA-CLEV
IP Backbone ATLA-NEWY32AOA
Connector Indiana Gigapop
Connector KyRON
Start Time:Wednesday, June 10, 2015, 4:01 AM (0401) UTC
End Time:Wednesday, June 10, 2015, 4:06 AM (0406) UTC
Description: The above listed asset was unavailable while Internet2 Engineers performed a maintenance to swap the active routing engine. Maintenance has been completed. All connections to the Core Node were be affected.

Ticket No.:4071:144
Subject:Maintenance 2 of 2 Completed - I2 AL2S Core Node Insertion in Las Vegas, NV
Source Of Impact:Internal
Affected:AL2S Core Node LASV
AL2S Backbone LOSA-SALT
IP Backbone LOSA-SALT
Start Time:Wednesday, June 10, 2015, 3:12 AM (0312) UTC
End Time:Wednesday, June 10, 2015, 4:07 AM (0407) UTC
Description: The above listed assets were unavailable while Internet2 Engineers performed a core node insertion in Las Vegas. Maintenance has been completed. Backbone traffic re-routed during this maintenance. Workgroup administrators received notifications from OESS regarding circuits re-routed on the backbone. *This was maintenance window 2 of 2*

Ticket No.:4245:144
Subject:Maintenance Completed - I2 Optical DWDM System RENO-SACR
Source Of Impact:Vendor
Affected:DWDM System RENO-SACR
AL2S Backbone SALT-SUNN
Start Time:Thursday, June 11, 2015, 7:20 AM (0720) UTC
End Time:Thursday, June 11, 2015, 7:48 AM (0748) UTC
Description: DWDM System RENO-SACR is now available after Level(3) Communications relocated fiber in Auburn, CA.

Ticket No.:4275:144
Subject:Emergency Maintenance Completed - I2 Optical DWDM System CINC-INDI
Source Of Impact:Vendor
Affected:DWDM System CINC-INDI
AL2S Backbone ATLA-CHIC
AL2S Backbone CINC-INDI
IP Backbone ATLA-CHIC
IP Connector University of Memphis
IP Connector Indiana Gigapop
Start Time:Saturday, June 13, 2015, 12:47 AM (0047) UTC
End Time:Saturday, June 13, 2015, 3:14 AM (0314) UTC
Description: DWDM System CINC-INDI is now available after Level(3) Communications performed emergency work to repair damaged fiber outside of Cincinnati, OH. Services are now restored and no further impact is expected.


Downtime and Availability



Unified - Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%

Aggregate Unified - Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%100.00000%100.00000%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Unified - Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
I2-ATLA-CLEV-VLAN-14832 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-CLEV-VLAN-14837 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14833 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-NEWY32AOA-VLAN-14838 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14831 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CHIC-NEWY32AOA-VLAN-14834 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 5 hr 10 min0 hr 38 min 96.92460%99.62302% 99.47253%99.86187%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 0 hr 45 min0 hr 37 min 99.55357%99.63294% 99.60022%99.98900%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 45 min0 hr 37 min 99.55357%99.63294% 99.87800%99.96072%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.91917%99.99848%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94896%99.99848%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94061%99.99469%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.91291%99.94706%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84954%99.98672%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.62565%99.75087%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89375%99.98975%
R&E 10G Backbone: I2-LOSA-PAIX-VLAN-23105 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Commodity 10G Backbone: I2-LOSA-PAIX-VLAN-23106 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 1 hr 14 min0 hr 0 min 99.26587%100.00000% 99.96926%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94763%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94763%99.99905%
R&E 100G Backbone: I2-LOSA-WILC-VLAN-23103 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Commodity 100G Backbone: I2-LOSA-WILC-VLAN-23104 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94156%99.95940%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96793%99.99962%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96793%99.99962%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318 0 hr 45 min0 hr 37 min 99.55357%99.63294% 99.87800%99.96072%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 4 hr 30 min0 hr 1 min 97.32143%99.99008% 99.66511%99.97268%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.37026%99.72336%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93871%100.00000%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.93606%99.99829%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94023%99.99981%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97989%99.99962%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96376%99.99962%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%99.99867%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89982%99.99753%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.75770%99.82753%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92505%99.99924%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92505%99.99924%
Legacy 10G Backbone: I2-LOSA-PAIX-10GE-05853 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99621%100.00000%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99013%100.00000%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97818%99.99905%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99412%99.99981%
Legacy 100G Backbone: I2-LOSA-WILC-100GE-23893 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99696%99.99810%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%99.99981%

Aggregate All Unified - Internet2 IP Backbone Circuits 13 hr 19 min10 hr 49 min 99.83823%99.96963% 99.91667%99.97768%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:4321:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:
Start Time:Thursday, June 18, 2015, 4:00 PM (1600) UTC
End Time:Thursday, June 18, 2015, 8:00 PM (2000) UTC
Description: Member ESnet's Optical circuits ESNET-JGI-LBNL-10GE-10778 and ESNET- LBNL-SACR-10GE-07941 will be unavailable while ESnet Engineers perform maintenance. A brief outage is expected, however the entire maintenance window has been reserved.

Ticket No.:4248:144
Subject:Maintenance - I2 IP Core Node CLEV
Affected:IP Core Node CLEV and its Connector(s) and Participants
IP Backbone ATLA-CLEV
IP Backbone CHIC-CLEV
IP Backbone CLEV-NEWY
Ip Backbone CLEV-WASH
AL2S-IP Interconnect
Start Time:Friday, June 19, 2015, 4:00 AM (0400) UTC
End Time:Friday, June 19, 2015, 4:30 AM (0430) UTC
Description: The above listed asset will be unavailable while Internet2 Engineers perform a maintenance to swap the active routing engine. An outage of up to 10 minutes is expected; however, the entire window is reserved. All connections to the Core Node will be affected.

Ticket No.:4312:144
Subject:Maintenance - I2 AL2S Participant C-Light
Affected:AL2S Participant C-Light
Start Time:Friday, June 19, 2015, 4:00 AM (0400) UTC
End Time:Friday, June 19, 2015, 6:00 AM (0600) UTC
Description: The above listed asset will be unavailable while Internet2 Engineers perform maintenance. An outage is expected. The entire window is reserved.

Ticket No.:4249:144
Subject:Maintenance - I2 IP Core Node NEWY32AOA
Affected:IP Core Node NEWY32AOA and its Connector(s) and Participants
IP Backbone ATLA-NEWY32AOA
IP Backbone CHIC-NEWY32AOA
IP Backbone CLEV-NEWY32AOA
AL2S Interconnect to IP (NEWY)
Start Time:Friday, June 19, 2015, 4:30 AM (0430) UTC
End Time:Friday, June 19, 2015, 5:00 AM (0500) UTC
Description: The above listed asset will be unavailable while Internet2 Engineers perform a maintenance to swap the active routing engine. An outage of up to 15 minutes is expected; however, the entire window is reserved. All connections to the Core Node will be affected.

Ticket No.:4250:144
Subject:Maintenance - I2 IP Core Node WASH
Affected:IP Core Node WASH and its Connector(s) and Participants
IP Backbone ATLA-WASH
IP Backbone CHIC-WASH
IP Backbone CLEV-WASH
IP Backbone ASHB-WASH
IP Backbone NEWY32AOA-WASH
IP Backbone PHIL-WASH
IP Backbone PITT-WASH
AL2S Interconnect to IP (WASH)
Start Time:Friday, June 19, 2015, 5:00 AM (0500) UTC
End Time:Friday, June 19, 2015, 5:30 AM (0530) UTC
Description: The above listed asset will be unavailable while Internet2 Engineers perform a maintenance to swap the active routing engine. An outage of up to 15 minutes is expected; however, the entire window is reserved. All connections to the Core Node will be affected.

Ticket No.:3989:144
Subject:Maintenance - I2 Various Optical Circuits (Northern Tier)
Affected:AL2S Backbone MINN-MISS2
Net+ Provider Code42 (SEAT)
Start Time:Saturday, June 20, 2015, 1:00 PM (1300) UTC
End Time:Saturday, June 20, 2015, 11:00 PM (2300) UTC
Description: The above listed assets may be unavailable while Zayo Engineers perform circuit maintenance. An outage of up to 3 hours is expected, however the entire maintenance window has been reserved.

Ticket No.:4320:144
Subject:Maintenance - I2 Optical Circuits I2-COLU4-KANS-100GE-11579 and I2-COLU4-SAIN-100GE-11580
Affected:AL2S Backbone CHIC-COLU4
AL2S Backbone COLU4-KANS
Start Time:Tuesday, June 23, 2015, 5:00 AM (0500) UTC
End Time:Tuesday, June 23, 2015, 10:00 AM (1000) UTC
Description: The above listed will be unavailable while MOREnet Engineers perform maintenance. An outage of 5 hours is expected. The entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:4234:144
Subject:Maintenance – I2 IP Node oob.clev
Affected:IP Node oob.clev
Start Time:Thursday, June 25, 2015, 5:00 AM (0500) UTC
End Time:Thursday, June 25, 2015, 10:00 AM (1000) UTC
Description: Member Level(3)'s Optical circuit I2-CLEV-CLEV-CATX-07498 will be unavailable while Level(3) Engineers perform maintenance. An outage of up to 2 hours is expected; however, the entire maintenance window has been reserved.