Internet2 IP Network Operations Report
07/14/2014 - 07/20/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

234472-High 2-High Outage Resolved - I2 IP Backbone Various Backbones HardwareVendor 07/14/2014 4:33 AM 07/14/2014 5:58 AM
234492-High 2-High Brief Outage Resolved - I2 IP Backbone HOUS-LOSA PowerVendor 07/14/2014 5:38 PM 07/14/2014 5:42 PM

Unscheduled Outages Detail


Ticket No.:23447:45
Subject:Outage Resolved - I2 IP Backbone Various Backbones
Source Of Impact:Vendor
Affected:Backbone LOSA-WASH
Backbone NEWY32AOA-WASH
Backbone CLEV-WASH
Backbone ATLA-WASH
Backbone CHIC-WASH
Connector OARnet
Connector 3ROX
Connector CAAREN
Connector SOX
Connector MAX GigaPOP
Connector MARIA
Peer SINET
Peer RedCLARA
Peer RNP
Peer CENIT
Peer CKLN
Peer GEANT
Participant ARE-ON
Peer Amazon
Participant University of Pennsylvania
Start Time:Monday, July 14, 2014, 4:33 AM (0433) UTC
End Time:Monday, July 14, 2014, 5:58 AM (0558) UTC
Description:The peers listed above are now available. Internet2 Engineers report that the reason for outage was a management module failure in the AL2S network which stopped pushing openflow rules to the switch. No further outages are expected. Backbone traffic re-routed during this outage event.

Ticket No.:23449:45
Subject:Brief Outage Resolved - I2 IP Backbone HOUS-LOSA
Source Of Impact:Vendor
Affected:Backbone HOUS-LOSA
Start Time:Monday, July 14, 2014, 5:38 PM (1738) UTC
End Time:Monday, July 14, 2014, 5:42 PM (1742) UTC
Description:Backbone HOUS-LOSA is now available following a brief outage. Internet2 Fiber Provider Engineers report the reason for outage was due to a power event that has been resolved. IP traffic will re-route.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

23425 3-Elevated 1-Critical Maintenance Completed - I2 IP Backbone ATLA-HOUS Software Vendor07/16/2014 5:01 AM 07/16/2014 8:12 AM
23429 3-Elevated 1-Critical Maintenance Completed - I2 IP Backbone HOUS-KANS SoftwareVendor 07/16/2014 6:10 AM 07/16/2014 7:31 AM
23401 3-Elevated 2-High Maintenance Completed - I2 IP Backbones ATLA-CHIC Hardware Vendor07/17/2014 7:16 AM 07/17/2014 7:19 AM

Scheduled Maintenances Detail

Ticket No.:23425:45
Subject:Maintenance Completed - I2 IP Backbone ATLA-HOUS
Source Of Impact:Vendor
Affected:Backbone ATLA-HOUS
Start Time:Wednesday, July 16, 2014, 5:01 AM (0501) UTC
End Time:Wednesday, July 16, 2014, 8:12 AM (0812) UTC
Description: Backbone ATLA-HOUS was unavailable while Internet2 Engineers performed a Software Upgrade. he maintenance has been completed. The upgrade has been reverted; notification will be provided for a new maintenance window. IP traffic was re-routed during this maintenance.

Ticket No.:23429:45
Subject:Maintenance Completed - I2 IP Backbone HOUS-KANS
Source Of Impact:Vendor
Affected:Backbone HOUS-KANS
Start Time:Wednesday, July 16, 2014, 6:10 AM (0610) UTC
End Time:Wednesday, July 16, 2014, 7:31 AM (0731) UTC
Description: Backbone HOUS-KANS was unavailable while Internet2 Engineers performed a Software Upgrade. The upgrade was reverted; notifcation will be provided for a new maintenance window. IP traffic was re-routed during this maintenance.

Ticket No.:23401:45
Subject:Maintenance Completed - I2 IP Backbones ATLA-CHIC
Source Of Impact:Vendor
Affected:Backbone ATLA-CHIC
ION Backbone ATLA-CHIC
Connector KyRON (CHIC)*
Connector Indiana Gigapop
Connector SOX
Start Time:Thursday, July 17, 2014, 7:16 AM (0716) UTC
End Time:Thursday, July 17, 2014, 7:19 AM (0719) UTC
Description: Backbone ATLA-CHIC was briefly unavailable while Internet2's fiber vendor performed hardware maintenance in Louisville, KY. Maintenance has been completed. No further outages are expected. IP traffic re-routed during this maintenance.


Downtime and Availability



Internet2 IP Core Nodes Down TimeReporting Period Availability52 Week Availability
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99355% 99.99867%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98596% 100.00000%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99089% 99.99924%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99734% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99639% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99184% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99317% 99.99981%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99639% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98843% 100.00000%
rtsw0.sing1.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
rtsw.sing2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%

Aggregate Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%99.99445%99.99981%

** Less than 52 weeks of data available

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 0 hr 3 min0 hr 0 min 99.97024%100.00000% 99.76624%99.95560%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 3 hr 11 min0 hr 0 min 98.10516%100.00000% 99.64443%99.89488%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 1 hr 25 min1 hr 25 min 99.15675%99.15675% 99.77136%99.82525%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.58978%99.74860%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92126%99.97723%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97230%99.99924%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 1 hr 25 min1 hr 25 min 99.15675%99.15675% 99.84479%99.96566%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 1 hr 21 min0 hr 0 min 99.19643%100.00000% 99.62337%99.71748%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.58428%99.68333%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93131%99.97002%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94592%100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95465%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94991%99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 1 hr 25 min1 hr 25 min 99.15675%99.15675% 99.92866%99.93985%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98710%99.99924%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98729%99.99943%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318 1 hr 25 min1 hr 25 min 99.15675%99.15675% 99.72924%99.81349%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 3 min0 hr 0 min 99.97024%100.00000% 99.89507%99.96774%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.34047%99.73399%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84707%99.85030%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84707%99.85030%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.74632%99.76852%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95048%99.98710%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95048%99.98710%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96053%99.99981%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.82886%99.83891%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.64879%99.75011%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 4 min0 hr 4 min 99.96032%99.96032% 99.51882%99.60705%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84384%99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84366%99.90912%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.74992%99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96243%99.99772%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95921%99.99772%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.78825%99.91177%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98956%99.99943%

Aggregate All Internet2 IP Backbone Circuits 10 hr 22 min4 hr 38 min 99.83323%99.90777% 99.83251%99.90162%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

* Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.



Upcoming Scheduled Maintenances

Ticket No.:23458:45
Subject:Maintenance - I2 IP Backbones KANS-SALT
Affected:Backbone KANS-SALT
ION Backbone KANS-SALT
Start Time:Wednesday, July 23, 2014, 6:00 AM (0600) UTC
End Time:Wednesday, July 23, 2014, 10:00 AM (1000) UTC
Description: Backbone KANS-SALT will be unavailable while Internet2's fiber provider performs hardware maintenance in Bethune, CO. Outages are expected; the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:23475:45
Subject:Maintenance - I2 IP Backbone KANS-SALT
Affected:Backbone KANS-SALT
ION Backbone KANS-SALT
Start Time:Friday, July 25, 2014, 10:00 AM (1000) UTC
End Time:Friday, July 25, 2014, 12:00 PM (1200) UTC
Description: The above listed will become available while vendor engineers perform maintenance to replace a faulty CFP in Denver, CO. Outages are anticipated and the entire maintenance window has been reserved.

Ticket No.:23478:45
Subject:Maintenance 1 of 2 - I2 IP Backbone ATLA-CHIC
Affected:Backbone ATLA-CHIC
Connector MissION
Connector KyRON
Start Time:Friday, August 1, 2014, 4:00 AM (0400) UTC
End Time:Friday, August 1, 2014, 10:00 AM (1000) UTC
Description: Backbone ATLA-CHIC will be unavailable while fiber provider Engineers perform maintenance. An outage is expected, and the entire window is reserved. IP traffic will re-route during this maintenance. This is the first of two maintenance windows.

Ticket No.:23479:45
Subject:Maintenance 2 of 2 - I2 IP Backbone ATLA-CHIC
Affected:Backbone ATLA-CHIC
Connector MissION
Connector KyRON
Start Time:Saturday, August 2, 2014, 4:00 AM (0400) UTC
End Time:Saturday, August 2, 2014, 10:00 AM (1000) UTC
Description: Backbone ATLA-CHIC will be unavailable while fiber provider Engineers perform maintenance. An outage is expected, and the entire window is reserved. IP traffic will re-route during this maintenance. This is the second of two maintenance windows.

Ticket No.:23465:45
Subject:Maintenance *Tracking* - I2 IP Node oob.kans & oob.chic
Affected:
Start Time:Thursday, August 7, 2014, 5:00 AM (0500) UTC
End Time:Thursday, August 7, 2014, 9:00 AM (0900) UTC
Description: Node oob.kans may be unavailable while Level 3 Engineers perform maintenance. Level 3 has indicated a possible outage of up to 30 minutes, but Internet2 does not expect this to be service-impacting.