Internet2 IP Network Operations Report
04/07/2014 - 04/13/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Start Time (UTC) End Time (UTC)

227542-High 2-High Outage Resolved - I2 IP Backbone HOUS-KANS Hardware 04/09/2014 3:59 PM 04/10/2014 5:37 AM
227673-Elevated 3-Elevated Degradation Resolved - I2 IP Various Backbones (CHIC) Hardware 04/10/2014 4:38 PM 04/10/2014 5:32 PM
227864-Normal 2-High Outage Resolved - I2 IP Backbone HOUS-LOSA Unannounced Maintenance 04/12/2014 5:38 AM 04/12/2014 6:19 AM

Unscheduled Outages Detail


Ticket No.:22754:45
Subject:Outage Resolved - I2 IP Backbone HOUS-KANS
Affected:Backbone HOUS-KANS
Start Time:Wednesday, April 9, 2014, 3:59 PM (1559) UTC
End Time:Thursday, April 10, 2014, 5:37 AM (0537) UTC
Description:Backbone HOUS-KANS is now available. Level(3) Communications has determined the problem to be a bad OCLD in opt.tuls. During the maintenance, the part was replaced and services have been restored. Maintenance has been completed

Ticket No.:22767:45
Subject:Degradation Resolved - I2 IP Various Backbones (CHIC)
Affected:AL2S Core Node (CHIC) and its Connector(s) and Participants
Backbones ATLA-CHIC, CHIC-CLEV, CHIC-KANS
Start Time:Thursday, April 10, 2014, 4:38 PM (1638) UTC
End Time:Thursday, April 10, 2014, 5:32 PM (1732) UTC
Description:The above are now available following a degraded condition. Vendor engineers indicated that there was a hardware issue that required a reboot to restore service. No further outages are expected.

Ticket No.:22786:45
Subject:Outage Resolved - I2 IP Backbone HOUS-LOSA
Affected:Backbone HOUS-LOSA
Start Time:Saturday, April 12, 2014, 5:38 AM (0538) UTC
End Time:Saturday, April 12, 2014, 6:19 AM (0619) UTC
Description:The above listed is now available. Level(3) Engineers were replacing a card in Phoenix, AZ that was not expected to be service impacting. Work has been completed and no further outages are expected.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Start Time (UTC) End Time (UTC)

22768 3-Elevated 1-Critical Emergency Maintenance Completed - I2 IP Various Backbones (CHIC) Hardware 04/10/2014 5:32 PM 04/10/2014 5:36 PM
22757 3-Elevated 2-High Emergency Maintenance Completed - I2 IP Backbones ATLA-CHIC and ATLA-WASH Hardware 04/11/2014 5:05 AM 04/11/2014 5:25 AM
22669 3-Elevated 2-High Maintenance Completed - I2 IP Backbone HOUS-LOSA Hardware 04/11/2014 7:10 AM 04/11/2014 7:26 AM
22773 3-Elevated 2-High Emergency Maintenance Completed - I2 IP Backbone HOUS-LOSA Hardware 04/12/2014 11:02 AM 04/12/2014 11:53 AM

Scheduled Maintenances Detail

Ticket No.:22768:45
Subject:Emergency Maintenance Completed - I2 IP Various Backbones (CHIC)
Affected:AL2S Core Node (CHIC) and its Connector(s) and Participants
Backbones ATLA-CHIC, CHIC-CLEV, CHIC-KANS
Start Time:Thursday, April 10, 2014, 5:32 PM (1732) UTC
End Time:Thursday, April 10, 2014, 5:36 PM (1736) UTC
Description: The above Backbone circuits were briefly unavailable while Internet2 Engineers performed hardware maintenance on AL2S Core Node (CHIC). Traffic was re-routed.

Ticket No.:22757:45
Subject:Emergency Maintenance Completed - I2 IP Backbones ATLA-CHIC and ATLA-WASH
Affected:Backbone ATLA-CHIC
Backbone ATLA-WASH
Start Time:Friday, April 11, 2014, 5:05 AM (0505) UTC
End Time:Friday, April 11, 2014, 5:25 AM (0525) UTC
Description: Backbones ATLA-CHIC and ATLA-WASH were unavailable while Level3 Engineers performed emergency maintenance. An outage of less than 30 minutes was observed. Maintenance has been completed. IP traffic was re-routed during this maintenance.

Ticket No.:22669:45
Subject:Maintenance Completed - I2 IP Backbone HOUS-LOSA
Affected:Backbone HOUS-LOSA
Start Time:Friday, April 11, 2014, 7:10 AM (0710) UTC
End Time:Friday, April 11, 2014, 7:26 AM (0726) UTC
Description: Backbone HOUS-LOSA was unavailable while Level3 Engineers performed maintenance to swap a card in an attempt to clear suspect pluggable alarms on opt.phoe. The maintenance was not successful in clearing these alarms. The maintenance has been completed. IP traffic was re-routed during this maintenance.

Ticket No.:22773:45
Subject:Emergency Maintenance Completed - I2 IP Backbone HOUS-LOSA
Affected:Backbone HOUS-LOSA
Start Time:Saturday, April 12, 2014, 11:02 AM (1102) UTC
End Time:Saturday, April 12, 2014, 11:53 AM (1153) UTC
Description: Backbone HOUS-LOSA was unavailable while Internet2 Engineers performed maintenance to install a new Layer 2 core node in Tucson, AZ. Maintenance has been completed . IP traffic re-routed during this maintenance.


Downtime and Availability



Internet2 IP Core Nodes Down TimeReporting Period Availability52 Week Availability
ATLA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99488% 100.00000%
CHIC MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98520% 99.99905%
CLEV MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99146% 99.99924%
HOUS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97685% 100.00000%
KANS MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99791% 100.00000%
LOSA MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99279% 100.00000%
NEWY MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
SALT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99317% 100.00000%
SEAT MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99734% 100.00000%
WASH MX960 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94175% 99.99962%

Aggregate Internet2 IP Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%99.98713%99.99979%

** Less than 52 weeks of data available

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 IP Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181 1 hr 18 min0 hr 54 min 99.22619%99.46429% 99.92695%99.98843%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98615%100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.78863%99.84138%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180 0 hr 58 min0 hr 54 min 99.42460%99.46429% 99.79774%99.85409%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183 0 hr 58 min0 hr 54 min 99.42460%99.46429% 99.95332%99.98975%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98786%99.99924%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95522%99.98179%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184 13 hr 38 min13 hr 38 min 91.88492%91.88492% 99.73873%99.77364%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189 1 hr 48 min0 hr 41 min 98.92857%99.59325% 99.72089%99.79660%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93947%99.97002%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97780%100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96395%99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96395%99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95124%99.95636%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98881%99.99924%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98900%99.99943%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.72374%99.81538%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05638 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.57290%99.66587%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.64917%99.98122%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05562 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.64803%99.98141%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05133 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.61787%99.71141%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.61919%99.71141%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84821%99.85030%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-6366 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96680%99.98786%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08892 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96338%99.98786%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%99.99981%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%99.99981%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08893 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94858%99.99829%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95560%99.99905%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.82886%99.88464%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08890 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80798%99.88464%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.60288%99.66625%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05561 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.60288%99.66625%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.49757%99.60686%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05581 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.49359%99.60667%
Legacy 10G Backbone: I2-KANS-CHIC-10GE-05781 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05138 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.81557%99.90930%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.81557%99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.81861%99.90912%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.78180%99.80666%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05571 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.78503%99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80400%99.87117%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80077%99.87117%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.73076%99.91101%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07613 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.73076%99.91101%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99127%99.99943%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05573 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99127%99.99943%

Aggregate All Internet2 IP Backbone Circuits 18 hr 40 min1 hr 39 min 99.77778%99.79742% 99.84004%99.90115%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

* Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.



Upcoming Scheduled Maintenances

Ticket No.:22741:45
Subject:Maintenance 1 of 2 - I2 IP Backbone HOUS-KANS
Affected:Backbone HOUS-KANS
Connector University of Arizona (HOUS)*
Connector MissiON (HOUS)*
Connector LONI
Start Time:Tuesday, April 15, 2014, 5:00 AM (0500) UTC
End Time:Tuesday, April 15, 2014, 11:00 AM (1100) UTC
Description: Backbone HOUS-KANS will be unavailable while Level(3) Communications relocate cable in Dallas, TX. An outage of 1 hour is expected; however the entire window is reserved. IP traffic will re-route during this maintenance. An alternate window has been reserved for 4/16 at 0500 UTC, should it be required. *Denotes the connector is multi-homed and therefore will remain reachable

Ticket No.:22711:45
Subject:Maintenance - I2 IP Backbone ATLA-HOUS
Affected:Backbone ATLA-HOUS
Start Time:Wednesday, April 16, 2014, 4:00 AM (0400) UTC
End Time:Wednesday, April 16, 2014, 6:00 AM (0600) UTC
Description: Backbone ATLA-HOUS will be unavailable while I2 Engineers perform maintenance. An outage is expected; however, the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:22742:45
Subject:Maintenance 2 of 2 - I2 IP Backbone HOUS-KANS
Affected:Backbone HOUS-KANS
Connector University of Arizona (HOUS)*
Connector MissiON (HOUS)*
Connector LONI
Start Time:Wednesday, April 16, 2014, 5:00 AM (0500) UTC
End Time:Wednesday, April 16, 2014, 11:00 AM (1100) UTC
Description: Backbone HOUS-KANS will be unavailable while Level(3) Communications relocate cable in Dallas, TX. An outage of 1 hour is expected; however the entire window is reserved. IP traffic will re-route during this maintenance. This is Maintenance window 2 of 2. *Denotes the connector is multi-homed and therefore will remain reachable

Ticket No.:22733:45
Subject:Maintenance 1 of 2 - I2 IP Backbones ATLA-CHIC and ATLA-WASH
Affected:Backbones ATLA-CHIC
Backbone ATLA-WASH
Connector Indiana Gigapop
Connector KyRON
Connector MissiON
Start Time:Wednesday, April 16, 2014, 5:00 AM (0500) UTC
End Time:Wednesday, April 16, 2014, 11:00 AM (1100) UTC
Description: *This is the first of two maintenance windows* Backbones ATLA-CHIC and ATLA-WASH will be unavailable while fiber provider Engineers perform maintenance. Outages are expected, and the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:22734:45
Subject:Maintenance 2 of 2 - I2 IP Backbones ATLA-CHIC and ATLA-WASH
Affected:Backbones ATLA-CHIC
Backbone ATLA-WASH
Connector Indiana Gigapop
Connector KyRON
Connector MissiON
Start Time:Thursday, April 17, 2014, 5:00 AM (0500) UTC
End Time:Thursday, April 17, 2014, 11:00 AM (1100) UTC
Description: *This is the second of two maintenance windows* Backbones ATLA-CHIC and ATLA-WASH will be unavailable while fiber provider Engineers perform maintenance. Outages are expected, and the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:22708:45
Subject:Maintenance - I2 IP OOB (ATLA)
Affected:oob.atla.net.internet2.edu
Start Time:Thursday, April 17, 2014, 7:00 AM (0700) UTC
End Time:Thursday, April 17, 2014, 10:00 AM (1000) UTC
Description: OOB monitoring will become unavailable, while Level(3) Engineers perform maintenance in Atlanta, GA. No network impact is expected.

Ticket No.:22792:45
Subject:Maintenance 1 of 2 - I2 IP Backbone CINC-INDI
Affected:I2-ATLA-CHIC-VLAN-09218 ATLA-CHI BB
I2-ATLA-CHIC-VLAN-09219 ATLA-CHIC BB (ION)
I2-ATLA-CHIC-VLAN-09320 ATLA-CHIC BB
I2-CHIC-NASH-10GE-05845
I2-CHIC-NASH-10GE-06055 (SOX)
I2-INDI-ATLA-10GE-04182 (InGP)
Start Time:Thursday, April 24, 2014, 4:00 AM (0400) UTC
End Time:Thursday, April 24, 2014, 10:00 AM (1000) UTC
Description: The above listed will be unavailable while Level(3) Communications relocate cable due to mandated bridge installation project between Greensburg, IN and Indianapolis, IN. An outage of 30 minutes is expected; however the entire window is reserved. *This is the first of two maintenance windows*

Ticket No.:22774:45
Subject:Maintenance - I2 IP Backbone CHIC-KANS
Affected:Backbone CHIC-KANS
Start Time:Thursday, April 24, 2014, 7:00 AM (0700) UTC
End Time:Thursday, April 24, 2014, 11:00 AM (1100) UTC
Description: Backbone CHIC-KANS will be unavailable while Level(3) Engineers perform maintenance. An outage of 30min is expected; however, the entire window is reserved. IP traffic will re-route during this maintenance.