Internet2 IP Network Operations Report
01/13/2014 - 01/19/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Start Time (UTC) End Time (UTC)

221782-High 2-High Instability Resolved - I2 IP Backbone CHIC-CLEV Circuit - Bumped Fiber 01/13/2014 11:12 PM 01/13/2014 11:15 PM
221822-High 2-High Brief Outage Resolved - I2 IP Backbone HOUS-LOSA Power 01/14/2014 4:52 PM 01/14/2014 4:57 PM
222022-High 2-High Outage Resolved - I2 IP Backbone CLEV-WASH Circuit - Damaged Fiber 01/16/2014 4:40 PM 01/17/2014 1:01 AM
222132-High 2-High Brief Outage Resolved - I2 IP Backbone KANS-SALT Hardware 01/18/2014 1:16 PM 01/18/2014 1:17 PM

Unscheduled Outages Detail


Ticket No.:22178:45
Subject:Instability Resolved - I2 IP Backbone CHIC-CLEV
Affected:Backbone CHIC-CLEV
Start Time:Monday, January 13, 2014, 11:12 PM (2312) UTC
End Time:Monday, January 13, 2014, 11:15 PM (2315) UTC
Description:Backbone CHIC-CLEV was briefly unstable. Level(3) Communications reports the reason for the instability was fiber disturbance caused by construction crews preparing for maintenance. No further instability is expected. IP traffic should have re-routed.

Ticket No.:22182:45
Subject:Brief Outage Resolved - I2 IP Backbone HOUS-LOSA
Affected:Backbone HOUS-LOSA
Start Time:Tuesday, January 14, 2014, 4:52 PM (1652) UTC
End Time:Tuesday, January 14, 2014, 4:57 PM (1657) UTC
Description:Backbone HOUS-LOSA was briefly unavailable. Internet2 Engineers are investigating the cause. Preliminary investigation has revealed that the outage was due to a power event in Pine Valley (Live Oak Springs), CA. IP traffic briefly re-routed.

Ticket No.:22202:45
Subject:Outage Resolved - I2 IP Backbone CLEV-WASH
Affected:Backbone CLEV-WASH
Start Time:Thursday, January 16, 2014, 4:40 PM (1640) UTC
End Time:Friday, January 17, 2014, 1:01 AM (0101) UTC
Description:Backbone CLEV-WASH is available. Level(3) Communications technicians have confirmed that Splicing has been completed. IP traffic re-routed during the outage. No further outages are expected.

Ticket No.:22213:45
Subject:Brief Outage Resolved - I2 IP Backbone KANS-SALT
Affected:Backbone KANS-SALT
Start Time:Saturday, January 18, 2014, 1:16 PM (1316) UTC
End Time:Saturday, January 18, 2014, 1:17 PM (1317) UTC
Description:Backbone KANS-SALT was briefly unavailable. Vendor Engineers report the outag was due to faulty hardware. IP traffic was re-routed.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Start Time (UTC) End Time (UTC)

22133 3-Elevated 2-High Maintenance Completed - I2 IP Backbones CHIC-CLEV and ATLA-WASH Circuit 01/14/2014 6:57 AM 01/14/2014 7:15 AM
22175 1-Critical 1-Critical Emergency Maintenance Completed - I2 IP Core Nodes CLEV, NEWY, and WASH Software 01/14/2014 10:05 AM 01/14/2014 10:32 AM
22128 3-Elevated 2-High Maintenance Completed - I2 IP Backbone NEWY-WASH Hardware 01/15/2014 5:26 AM 01/15/2014 3:49 PM
22196 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node KANS Software 01/18/2014 10:16 AM 01/18/2014 10:22 AM
22190 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node ATLA Software 01/18/2014 9:30 AM 01/18/2014 9:35 AM
22191 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node NEWY Software 01/18/2014 9:48 AM 01/18/2014 9:50 AM
22192 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node CLEV Software 01/18/2014 9:57 AM 01/18/2014 10:02 AM
22195 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node CHIC Software 01/18/2014 10:10 AM 01/18/2014 10:15 AM
22194 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node WASH Software 01/18/2014 9:39 AM 01/18/2014 9:45 AM
22197 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node HOUS Software 01/18/2014 10:25 AM 01/18/2014 10:30 AM
22198 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node SALT Software 01/18/2014 10:34 AM 01/18/2014 10:39 AM
22199 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node LOSA Software 01/18/2014 10:43 AM 01/18/2014 10:48 AM
22200 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Core Node SEAT Software 01/18/2014 10:52 AM 01/18/2014 10:57 AM

Scheduled Maintenances Detail

Ticket No.:22133:45
Subject:Maintenance Completed - I2 IP Backbones CHIC-CLEV and ATLA-WASH
Affected:Backbone CHIC-CLEV
Backbone ATLA-WASH
Connector Nysernet (CHIC)
Connector MREN (WASH)
Start Time:Tuesday, January 14, 2014, 6:57 AM (0657) UTC
End Time:Tuesday, January 14, 2014, 7:15 AM (0715) UTC
Description: The above listed was briefly unavailable while Level(3) Communications relocated backbone cable in Chicago, IL. Maintenance has been completed.

Ticket No.:22175:45
Subject:Emergency Maintenance Completed - I2 IP Core Nodes CLEV, NEWY, and WASH
Affected:Core Node CLEV and its Connector(s) and Participants
Core Node NEWY and its Connector(s) and Participants
Core Node WASH and its Connector(s) and Participants
Start Time:Tuesday, January 14, 2014, 10:05 AM (1005) UTC
End Time:Tuesday, January 14, 2014, 10:32 AM (1032) UTC
Description: The above listed Core Nodes were unavailable while Internet2 Engineers performed routing process restarts on Core Nodes CLEV, NEWY and WASH. Maintenance has been completed.

Ticket No.:22128:45
Subject:Maintenance Completed - I2 IP Backbone NEWY-WASH
Affected:IP Backbone NEWY-WASH
ION Backbone NEWY-WASH
Connector MAGPI
Connector MAX (NEWY)
Connector MARIA (NEWY)
Connector 3ROX (WASH)
Connector OARnet (WASH)
Start Time:Wednesday, January 15, 2014, 5:26 AM (0526) UTC
End Time:Wednesday, January 15, 2014, 3:49 PM (1549) UTC
Description: The above listed was unavailable while I2's Layer 1 hardware technicians installed a new node in Baltimore, MD. Maintenance has been completed.

Ticket No.:22196:45
Subject:Emergency Maintenance Completed - I2 IP Core Node KANS
Affected:Core Node KANS and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 10:16 AM (1016) UTC
End Time:Saturday, January 18, 2014, 10:22 AM (1022) UTC
Description: Core Node KANS was unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node were affected. Maintenance has been completed.

Ticket No.:22190:45
Subject:Emergency Maintenance Completed - I2 IP Core Node ATLA
Affected:Core Node ATLA and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 9:30 AM (0930) UTC
End Time:Saturday, January 18, 2014, 9:35 AM (0935) UTC
Description: Core Node ATLA was briefly unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node was affected.

Ticket No.:22191:45
Subject:Emergency Maintenance Completed - I2 IP Core Node NEWY
Affected:Core Node NEWY and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 9:48 AM (0948) UTC
End Time:Saturday, January 18, 2014, 9:50 AM (0950) UTC
Description: Core Node NEWY was briefly unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node was affected.

Ticket No.:22192:45
Subject:Emergency Maintenance Completed - I2 IP Core Node CLEV
Affected:Core Node CLEV and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 9:57 AM (0957) UTC
End Time:Saturday, January 18, 2014, 10:02 AM (1002) UTC
Description: Core Node CLEV was unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node were affected. Maintenance has been completed.

Ticket No.:22195:45
Subject:Emergency Maintenance Completed - I2 IP Core Node CHIC
Affected:Core Node CHIC and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 10:10 AM (1010) UTC
End Time:Saturday, January 18, 2014, 10:15 AM (1015) UTC
Description: Core Node CHIC was briefly unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node was affected.

Ticket No.:22194:45
Subject:Emergency Maintenance Completed - I2 IP Core Node WASH
Affected:Core Node WASH and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 9:39 AM (0939) UTC
End Time:Saturday, January 18, 2014, 9:45 AM (0945) UTC
Description: Core Node WASH was unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node were affected. Maintenance has been completed.

Ticket No.:22197:45
Subject:Emergency Maintenance Completed - I2 IP Core Node HOUS
Affected:Core Node HOUS and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 10:25 AM (1025) UTC
End Time:Saturday, January 18, 2014, 10:30 AM (1030) UTC
Description: Core Node HOUS was unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node were affected. Maintenance has been completed.

Ticket No.:22198:45
Subject:Emergency Maintenance Completed - I2 IP Core Node SALT
Affected:Core Node SALT and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 10:34 AM (1034) UTC
End Time:Saturday, January 18, 2014, 10:39 AM (1039) UTC
Description: Core Node SALT was briefly unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node was affected.

Ticket No.:22199:45
Subject:Emergency Maintenance Completed - I2 IP Core Node LOSA
Affected:Core Node LOSA and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 10:43 AM (1043) UTC
End Time:Saturday, January 18, 2014, 10:48 AM (1048) UTC
Description: Core Node LOSA was briefly unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node was affected.

Ticket No.:22200:45
Subject:Emergency Maintenance Completed - I2 IP Core Node SEAT
Affected:Core Node SEAT and its Connectors and Participants
Start Time:Saturday, January 18, 2014, 10:52 AM (1052) UTC
End Time:Saturday, January 18, 2014, 10:57 AM (1057) UTC
Description: Core Node SEAT was unavailable while Internet2 Engineers performed maintenance to swap the Routing engines in the process of upgrading software at the recommendation of Juniper to alleviate a bug. All connections to the Core Node were affected. Maintenance has been completed.


Downtime and Availability



Internet2 IP Core Nodes Down TimeReporting Period Availability52 Week Availability
ATLA MX960 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.98046% 99.99981%
CHIC MX960 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.96926% 99.99905%
CLEV MX960 0 hr 32 min0 hr 0 min 99.68254%100.00000% 99.99146% 99.99924%
HOUS MX960 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.96888% 100.00000%
KANS MX960 0 hr 6 min0 hr 0 min 99.94048%100.00000% 99.99772% 100.00000%
LOSA MX960 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.96680% 100.00000%
NEWY MX960 0 hr 29 min0 hr 0 min 99.71230%100.00000% 99.98179% 99.99867%
SALT MX960 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.97287% 100.00000%
SEAT MX960 0 hr 5 min0 hr 0 min 99.95040%100.00000% 99.96433% 100.00000%
WASH MX960 0 hr 33 min0 hr 0 min 99.67262%100.00000% 99.94099% 99.99962%

Aggregate Internet2 IP Core Nodes2 hr 10 min0 hr 10 min99.87103%100.00000%99.97346%99.99964%

** Less than 52 weeks of data available

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.98368%99.99943%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.98767%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.94668%99.99848%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.82032%99.87591%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.96433%100.00000%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 7 min0 hr 0 min 99.93056%100.00000% 99.98900%99.99943%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.98937%100.00000%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.98179%100.00000%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 0 hr 15 min0 hr 5 min 99.85119%99.95040% 99.75467%99.81690%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.97780%99.99981%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.98975%100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.96907%99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 8 min0 hr 0 min 99.92063%100.00000% 99.95200%99.95636%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.99051%99.99924%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.74233%99.81538%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05638 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.74101%99.81538%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.89507%99.96983%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05562 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.89394%99.97002%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05133 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.76738%99.86092%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.76871%99.86092%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.99772%99.99981%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-6366 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 31 min0 hr 3 min 99.69246%99.97024% 99.87591%99.89697%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08892 0 hr 31 min0 hr 3 min 99.69246%99.97024% 99.87591%99.89697%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.93701%100.00000%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.92790%100.00000%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08893 0 hr 7 min0 hr 0 min 99.93056%100.00000% 99.90760%99.95731%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 7 min0 hr 0 min 99.93056%100.00000% 99.91386%99.95731%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 8 hr 32 min8 hr 21 min 94.92063%95.02976% 99.84612%99.90190%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08890 8 hr 32 min8 hr 21 min 94.92063%95.02976% 99.84024%99.90190%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.58978%99.65657%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05561 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.58978%99.65657%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 15 min0 hr 5 min 99.85119%99.95040% 99.51749%99.68124%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05581 0 hr 15 min0 hr 5 min 99.85119%99.95040% 99.51711%99.68086%
Legacy 10G Backbone: I2-KANS-CHIC-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05138 0 hr 12 min0 hr 1 min 99.88095%99.99008% 99.77857%99.90912%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 12 min0 hr 1 min 99.88095%99.99008% 99.77876%99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 11 min0 hr 0 min 99.89087%100.00000% 99.82715%99.90912%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.74727%99.80666%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05571 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.75049%99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.73209%99.82278%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.73209%99.82601%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 10 hr 31 min0 hr 0 min 93.74008%100.00000% 99.73076%99.91101%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07613 10 hr 31 min0 hr 0 min 93.74008%100.00000% 99.65828%99.83853%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.95788%99.98596%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05573 0 hr 10 min0 hr 0 min 99.90079%100.00000% 99.97135%99.99943%

Aggregate All Internet2 IP Backbone Circuits 45 hr 49 min28 hr 44 min 99.43183%99.78815% 99.85638%99.91352%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

* Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.



Upcoming Scheduled Maintenances

Ticket No.:22304:45
Subject:Maintenance - I2 IP Core Node KANS
Affected:Core Node KANS and its Connector(s) and Participants
Connector LEARN
Net+ Provider Dell
Start Time:Saturday, February 8, 2014, 11:00 AM (1100) UTC
End Time:Saturday, February 8, 2014, 12:00 PM (1200) UTC
Description: Core Node KANS will be unavailable while Internet2 Engineers reboot the routing engine. An outage is expected, and the entire window is reserved. All connections to the Core Node will be affected.

Ticket No.:22231:45
Subject:Maintenance 1 of 2 - I2 IP Backbone LOSA-SALT
Affected:Backbone LOSA-SALT
Start Time:Tuesday, February 25, 2014, 8:00 AM (0800) UTC
End Time:Tuesday, February 25, 2014, 2:00 PM (1400) UTC
Description: Backbone LOSA-SALT will be unavailable while Level(3) Communications performs fiber maintenance in San Gabriel, CA as mandated by the California Department of Transportation. The entire window is reserved. IP traffic will re-route during this maintenance. An alternate window has been reserved for 2/28/14 at 0800-1400, should it be required.

Ticket No.:22234:45
Subject:Maintenance 2 of 2 - I2 IP Backbone LOSA-SALT
Affected:Backbone LOSA-SALT
Start Time:Friday, February 28, 2014, 8:00 AM (0800) UTC
End Time:Friday, February 28, 2014, 2:00 PM (1400) UTC
Description: Backbone LOSA-SALT will be unavailable while Level(3) Communications performs fiber maintenance in San Gabriel, CA as mandated by the California Department of Transportation. The entire window is reserved. IP traffic will re-route during this maintenance. This is an alternate window for ticket 22231:45, should it be required.