Generated Report for Internet2 IP

Internet2 IP Network Operations Report
07/21/2014 - 07/27/2014

Internet2 IP Network Operations Center
at Indiana University

New Connections

None

Upgrades

None

New Services

None

Unscheduled Outages Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Outage Type	Source Of Impact	*Start Time (UTC)*	*End Time (UTC)*

23470	2-High	2-High	Outage Resolved - I2 IP Backbone Circuit KANS-SALT	Hardware	Vendor	07/21/2014 3:00 AM	07/21/2014 2:49 PM
23485	2-High	2-High	Brief Outage Resolved - I2 IP Backbone ATLA-HOUS	Other	Vendor	07/23/2014 7:05 PM	07/23/2014 7:06 PM
						07/23/2014 8:13 PM	07/23/2014 8:13 PM
23493	2-High	2-High	Brief Outage Resolved - I2 IP Backbone ATLA-HOUS	Human Error	Vendor	07/24/2014 4:14 PM	07/24/2014 4:15 PM
23499	2-High	2-High	Instability Resolved - I2 IP Backbone KANS-SALT	Hardware	Vendor	07/25/2014 4:32 AM	07/25/2014 1:30 PM

Unscheduled Outages Detail

Ticket No.:	23470:45
Subject:	Outage Resolved - I2 IP Backbone Circuit KANS-SALT
Source Of Impact:	Vendor
Affected:	Backbone KANS-SALT
Start Time:	Monday, July 21, 2014, 3:00 AM (0300) UTC
End Time:	Monday, July 21, 2014, 2:49 PM (1449) UTC
Description:	Backbone KANS-SALT is now available. Internet2 Engineers have resolved the issue, a hardware failure in Denver, CO. No further outages are expected. IP traffic re-routed during this event.

Ticket No.:	23485:45
Subject:	Brief Outage Resolved - I2 IP Backbone ATLA-HOUS
Source Of Impact:	Vendor
Affected:	Backbone ATLA-HOUS ION Backbone ATLA-HOUS
Start Time:	Wednesday, July 23, 2014, 7:05 PM (1905) UTC
End Time:	Wednesday, July 23, 2014, 7:06 PM (1906) UTC
Start Time:	Wednesday, July 23, 2014, 8:13 PM (2013) UTC
End Time:	Wednesday, July 23, 2014, 8:13 PM (2013) UTC
Description:	The above listed circuits are currently available to the community after a brief outage. Internet2's fiber vendor have attributed this to severe weather in the Northern Florida location. Updates will be provided when they become available. Backbone traffic re-routed during the event.

Ticket No.:	23493:45
Subject:	Brief Outage Resolved - I2 IP Backbone ATLA-HOUS
Source Of Impact:	Vendor
Affected:	Backbone ATLA-HOUS
Start Time:	Thursday, July 24, 2014, 4:14 PM (1614) UTC
End Time:	Thursday, July 24, 2014, 4:15 PM (1615) UTC
Description:	Backbone ATLA-HOUS was briefly unavailable. Level 3 Communications has indicated that the reason for the outage is a fiber bump. IP traffic re- routes.

Ticket No.:	23499:45
Subject:	Instability Resolved - I2 IP Backbone KANS-SALT
Source Of Impact:	Vendor
Affected:	Backbone KANS-SALT
Start Time:	Friday, July 25, 2014, 4:32 AM (0432) UTC
End Time:	Friday, July 25, 2014, 1:30 PM (1330) UTC
Description:	Backbone KANS-SALT is now stable. Internet2 and Level(3) Communications Engineers found that the outage was due to failed hardware, which was replaced. No further outages are expected. IP traffic re-routed during the outage.

Scheduled Maintenances Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Maintenance Type	Source Of Impact	*Start Time (UTC)*	*End Time (UTC)*

23455	3-Elevated	3-Elevated	Maintenance Completed - I2 IP Backbone CHIC-KANS	Hardware	Vendor	07/21/2014 5:48 AM	07/21/2014 7:11 AM
23456	3-Elevated	3-Elevated	Maintenance Completed - I2 IP Backbone HOUS-LOSA	Circuit	Vendor	07/22/2014 5:07 AM	07/22/2014 5:21 AM
23475	3-Elevated	2-High	Maintenance Completed - I2 IP Backbone KANS-SALT	Hardware	Vendor	07/25/2014 6:00 AM	07/25/2014 6:40 AM

Scheduled Maintenances Detail

Ticket No.:	23455:45
Subject:	Maintenance Completed - I2 IP Backbone CHIC-KANS
Source Of Impact:	Vendor
Affected:	Backbone CHIC-KANS
Start Time:	Monday, July 21, 2014, 5:48 AM (0548) UTC
End Time:	Monday, July 21, 2014, 7:11 AM (0711) UTC
Description:	Backbone CHIC-KANS was unavailable while Level(3) Communications performed maintenance to replace a card in Plainfield, IL. IP traffic was re-routed during this maintenance. Maintenance did not go as planned when vendor technicians discovered that the replacement card was dead on arrival. A future maintenance will be scheduled to perform card replacement. Technicians are hands off and affected connections are available and stable. Maintenance has been completed.

Ticket No.:	23456:45
Subject:	Maintenance Completed - I2 IP Backbone HOUS-LOSA
Source Of Impact:	Vendor
Affected:	Backbone HOUS-LOSA
Start Time:	Tuesday, July 22, 2014, 5:07 AM (0507) UTC
End Time:	Tuesday, July 22, 2014, 5:21 AM (0521) UTC
Description:	Backbone HOUS-LOSA was unavailable while fiber provider Engineers performed maintenance. The maintenance has been completed. IP traffic was re-routed during this maintenance.

Ticket No.:	23475:45
Subject:	Maintenance Completed - I2 IP Backbone KANS-SALT
Source Of Impact:	Vendor
Affected:	Backbone KANS-SALT ION Backbone KANS-SALT
Start Time:	Friday, July 25, 2014, 6:00 AM (0600) UTC
End Time:	Friday, July 25, 2014, 6:40 AM (0640) UTC
Description:	The above listed was unavailable while vendor engineers performed maintenance to replace a faulty CFP in Denver, CO. We had to expedite this maintenance due to an outage on the path. Maintenance has been completed and the faulty CFP in Denver has been replaced, However this did not fix the issue and the affected is still down. The outage is being tracked in ticket 23499. Please note that the Times listed above are for when the maintenance was performed, not for downtime of the Circuit. It was down before and after those times.

Downtime and Availability

Internet2 IP Core Nodes	Down Time		Reporting Period Availability		52 Week Availability
ATLA MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99412%	99.99867%
CHIC MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98691%	100.00000%
CLEV MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99165%	99.99924%
HOUS MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99734%	100.00000%
KANS MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99734%	100.00000%
LOSA MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99203%	100.00000%
NEWY MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99943%	100.00000%
SALT MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99317%	99.99981%
SEAT MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99734%	100.00000%
WASH MX960	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98937%	100.00000%
rtsw0.sing1.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
rtsw.sing2.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%

Aggregate Internet2 IP Core Nodes	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99489%	99.99981%

** Less than 52 weeks of data available

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 IP Backbone Circuits	Down Time		Reporting Period Availability		52 Week Availability
R&E 100G Backbone: I2-ATLA-CHIC-VLAN-09181	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.76624%	99.95560%
R&E 100G Backbone: I2-ATLA-HOUS-VLAN-09182	0 hr 1 min	0 hr 1 min	99.99008%	99.99008%	99.64424%	99.89469%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09179	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.77136%	99.82525%
R&E 100G Backbone: I2-CHIC-CLEV-VLAN-09180	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.58978%	99.74860%
R&E 100G Backbone: I2-CHIC-KANS-VLAN-09183	1 hr 23 min	0 hr 0 min	99.17659%	100.00000%	99.90551%	99.97723%
R&E 100G Backbone: I2-CLEV-NEWY32AOA-VLAN-09177	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97230%	99.99924%
R&E 100G Backbone: I2-CLEV-WASH-VLAN-09178	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.84479%	99.96566%
R&E 100G Backbone: I2-HOUS-KANS-VLAN-09184	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.62337%	99.71748%
R&E 100G Backbone: I2-HOUS-LOSA-VLAN-09189	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.58428%	99.68333%
R&E 100G Backbone: I2-KANS-SALT-VLAN-09185	21 hr 25 min	20 hr 46 min	87.25198%	87.63889%	99.68750%	99.73361%
R&E 100G Backbone: I2-LOSA-SALT-VLAN-09188	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.94592%	100.00000%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09187	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.95465%	99.99905%
R&E 100G Backbone: I2-LOSA-SEAT-VLAN-09328	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.94991%	99.99905%
R&E 100G Backbone: I2-NEWY32AOA-WASH-VLAN-09176	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.92866%	99.93985%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09186	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98710%	99.99924%
R&E 100G Backbone: I2-SALT-SEAT-VLAN-09327	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98729%	99.99943%
R&E 100G Backbone: I2-WASH-ATLA-VLAN-09179	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
R&E 100G Backbone: I2-ATLA-WASH-VLAN-09318	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.72924%	99.81349%
Legacy 10G Backbone: I2-ATLA-CHIC-10GE-05419	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.89507%	99.96774%
Legacy 10G Backbone: I2-ATLA-HOUS-10GE-05423	0 hr 2 min	0 hr 2 min	99.98016%	99.98016%	99.34009%	99.73361%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-05251	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.84707%	99.85030%
Legacy 10G Backbone: I2-ATLA-WASH-10GE-09142	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.84707%	99.85030%
Legacy 10G Backbone: I2-CHIC-CLEV-10GE-08891	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.74632%	99.76852%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05781	1 hr 23 min	0 hr 0 min	99.17659%	100.00000%	99.93473%	99.98710%
Legacy 10G Backbone: I2-CHIC-KANS-10GE-05976	1 hr 23 min	0 hr 0 min	99.17659%	100.00000%	99.93473%	99.98710%
Legacy 10G Backbone: I2-CLEV-NEWY32AOA-10GE-08894	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96053%	99.99981%
Legacy 10G Backbone: I2-CLEV-WASH-10GE-08889	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.82886%	99.83891%
Legacy 10G Backbone: I2-HOUS-KANS-10GE-05560	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.64879%	99.75011%
Legacy 10G Backbone: I2-HOUS-KANS-LAG-08934	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
Legacy 10G Backbone: I2-HOUS-LOSA-10GE-05559	0 hr 14 min	0 hr 0 min	99.86111%	100.00000%	99.51617%	99.60705%
Legacy 10G Backbone: I2-KANS-SALT-10GE-05566	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.84384%	99.90930%
Legacy 10G Backbone: I2-KANS-SALT-LAG-08935	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.84384%	99.90930%
Legacy 10G Backbone: I2-LOSA-SALT-10GE-05563	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.74992%	99.80666%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05564	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96243%	99.99772%
Legacy 10G Backbone: I2-LOSA-SEAT-10GE-05572	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.95921%	99.99772%
Legacy 10G Backbone: I2-NEWY32AOA-WASH-10GE-07612	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.78825%	99.91177%
Legacy 10G Backbone: I2-SALT-SEAT-10GE-05565	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98956%	99.99943%

Aggregate All Internet2 IP Backbone Circuits	25 hr 51 min	5 hr 2 min	99.58414%	99.66511%	99.82456%	99.89522%

* Internet2 IP 100G Backbone Circuits Ride over AL2S for connectivity.

Upcoming Scheduled Maintenances

Ticket No.:	23481:45
Subject:	Maintenance - I2 IP Various Backbones (WASH)
Affected:	Backbone NEWY32AOA-WASH Backbone CLEV-WASH Backbone ATLA-WASH Backbone CHIC-WASH IP to AL2S Interconnect
Start Time:	Tuesday, July 29, 2014, 4:00 AM (0400) UTC
End Time:	Tuesday, July 29, 2014, 6:00 AM (0600) UTC
Description:	The above listed backbones and interconnect with I2 AL2S will be unavailable while Internet2 Engineers execute power maintenance. Outages are expected during the window. The entire window is reserved.

Ticket No.:	23478:45
Subject:	Maintenance 1 of 2 - I2 IP Backbone ATLA-CHIC
Affected:	Backbone ATLA-CHIC Connector MissION Connector KyRON
Start Time:	Friday, August 1, 2014, 4:00 AM (0400) UTC
End Time:	Friday, August 1, 2014, 10:00 AM (1000) UTC
Description:	Backbone ATLA-CHIC will be unavailable while fiber provider Engineers perform maintenance. An outage is expected, and the entire window is reserved. IP traffic will re-route during this maintenance. This is the first of two maintenance windows.

Ticket No.:	23479:45
Subject:	Maintenance 2 of 2 - I2 IP Backbone ATLA-CHIC
Affected:	Backbone ATLA-CHIC Connector MissION Connector KyRON
Start Time:	Saturday, August 2, 2014, 4:00 AM (0400) UTC
End Time:	Saturday, August 2, 2014, 10:00 AM (1000) UTC
Description:	Backbone ATLA-CHIC will be unavailable while fiber provider Engineers perform maintenance. An outage is expected, and the entire window is reserved. IP traffic will re-route during this maintenance. This is the second of two maintenance windows.

Ticket No.:	23502:45
Subject:	Maintenance - I2 IP Backbone KANS-SALT
Affected:	Backbone KANS-SALT
Start Time:	Tuesday, August 5, 2014, 6:00 AM (0600) UTC
End Time:	Tuesday, August 5, 2014, 8:00 AM (0800) UTC
Description:	Backbone KANS-SALT will be unavailable while Level(3) Communications perform hardware maintenance in Bethune, CO. Outages are expected, the entire window is reserved. IP traffic will re-route during this maintenance.

Ticket No.:	23465:45
Subject:	Maintenance Tracking - I2 IP Node oob.kans & oob.chic
Affected:
Start Time:	Thursday, August 7, 2014, 5:00 AM (0500) UTC
End Time:	Thursday, August 7, 2014, 9:00 AM (0900) UTC
Description:	Node oob.kans may be unavailable while Level 3 Engineers perform maintenance. Level 3 has indicated a possible outage of up to 30 minutes, but Internet2 does not expect this to be service-impacting.