Generated Report for Internet2 Advanced Layer 2 Service

Internet2 Advanced Layer 2 Service Network Operations Report
02/24/2014 - 03/02/2014

Internet2 Advanced Layer 2 Service Network Operations Center
at Indiana University

New Connections

Connector	Type	Date
NEREN / UVM (ALBA)	Participant	2/25/2014
NEREN / UMS (ALBA)	Participant	2/25/2014
NEREN / UNH (ALBA)	Participant	2/25/2014
NEREN / DART (ALBA)	Participant	2/25/2014

Upgrades

 OESS 1.1.2b, 02/26/2014 @ 08:50 UTC

New Services

None

Unscheduled Outages Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Outage Type	*Start Time (UTC)*	*End Time (UTC)*

1389	2-High	2-High	Outage Resolved - I2 AL2S Backbone RALE - WASH	Hardware	02/24/2014 8:24 PM	02/24/2014 9:04 PM
1392	2-High	2-High	Instability Resolved - I2 AL2S Backbone ASHB-PITT	Circuit - Bumped Fiber	02/25/2014 5:01 PM	02/25/2014 6:17 PM
1395	2-High	2-High	Brief Outage Resolved - I2 AL2S Backbone ASHB-PITT	Circuit - Damaged Fiber	02/26/2014 7:40 PM	02/26/2014 7:44 PM
1400	2-High	2-High	Degraded Condition Resolved - I2 AL2S Core Node DENV	Hardware	02/27/2014 5:50 PM	02/27/2014 6:27 PM
1401	1-Critical	1-Critical	Emergency Maintenance Completed - I2 AL2S Core Node DENV	Unannounced Maintenance	02/27/2014 6:25 PM	02/27/2014 6:27 PM
1402	2-High	2-High	Brief Outage Resolved - I2 AL2S Backbone ATLA-CHIC	Circuit - Bumped Fiber	02/27/2014 7:56 PM	02/27/2014 7:57 PM
1406	2-High	2-High	Brief Outage - I2 AL2S Backbone ATLA-JCSN	Undetermined	03/02/2014 10:24 PM	03/02/2014 10:25 PM

Unscheduled Outages Detail

Ticket No.:	1389:135
Subject:	Outage Resolved - I2 AL2S Backbone RALE - WASH
Affected:	Backbone RALE-WASH
Start Time:	Monday, February 24, 2014, 8:24 PM (2024) UTC
End Time:	Monday, February 24, 2014, 9:04 PM (2104) UTC
Description:	Backbone RALE-WASH is now available after an outage. Internet2 Engineers have advised that a hardware issue was resolved to restore service. No further outages are expected. Traffic re-routed during this event.

Ticket No.:	1392:135
Subject:	Instability Resolved - I2 AL2S Backbone ASHB-PITT
Affected:	Backbone ASHB-PITT
Start Time:	Tuesday, February 25, 2014, 5:01 PM (1701) UTC
End Time:	Tuesday, February 25, 2014, 6:17 PM (1817) UTC
Description:	Backbone ASHB-PITT is now stable. Internet2's fiber provider has indicated that the reason for the outage is multiple fiber bumps associated with non- Internet2 fiber testing. Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:	1395:135
Subject:	Brief Outage Resolved - I2 AL2S Backbone ASHB-PITT
Affected:	Backbone ASHB-PITT
Start Time:	Wednesday, February 26, 2014, 7:40 PM (1940) UTC
End Time:	Wednesday, February 26, 2014, 7:44 PM (1944) UTC
Description:	Backbone ASHB-PITT was briefly unavailable. Internet2 Engineers have indicated that the reason for the outage is fiber damage. Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:	1400:135
Subject:	Degraded Condition Resolved - I2 AL2S Core Node DENV
Affected:	Core Node DENV
Start Time:	Thursday, February 27, 2014, 5:50 PM (1750) UTC
End Time:	Thursday, February 27, 2014, 6:27 PM (1827) UTC
Description:	Core Node DENV is available after being in a degraded state at the above listed times. Internet2 Engineers processed a reboot of a line card in the Core Node located in Denver, CO to resolve this issue. A maintenance will be required to permanently resolve this issue. This is expected to happen today, 2/27/14.

Ticket No.:	1401:135
Subject:	Emergency Maintenance Completed - I2 AL2S Core Node DENV
Affected:	Core Node DENV (Line Card 1) Backbone DENV-KANS Backbone DENV-SALT
Start Time:	Thursday, February 27, 2014, 6:25 PM (1825) UTC
End Time:	Thursday, February 27, 2014, 6:27 PM (1827) UTC
Description:	AL2S Core Node DENV (Line Card 1) was rebooted by Internet2 Engineers to resolve a hardware issue. The reboot resolved the hardware issue. No further outages are expected.

Ticket No.:	1402:135
Subject:	Brief Outage Resolved - I2 AL2S Backbone ATLA-CHIC
Affected:
Start Time:	Thursday, February 27, 2014, 7:56 PM (1956) UTC
End Time:	Thursday, February 27, 2014, 7:57 PM (1957) UTC
Description:	The AL2S backbone ATLA-CHIC listed above was briefly unavailable. The vendor indicated that there was a fiber bump that caused the issue. No further outages are expected related to this issue.

Ticket No.:	1406:135
Subject:	Brief Outage - I2 AL2S Backbone ATLA-JCSN
Affected:	Backbone ATLA-JCSN
Start Time:	Sunday, March 2, 2014, 10:24 PM (2224) UTC
End Time:	Sunday, March 2, 2014, 10:25 PM (2225) UTC
Description:	Backbone ATLA-JCSN was briefly unavailable. Internet2 Engineers are investigating the cause; however, the reason for the outage is pending. Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone.

Scheduled Maintenances Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Maintenance Type	*Start Time (UTC)*	*End Time (UTC)*

1236	3-Elevated	2-High	Maintenance 1 of 2 Completed - I2 AL2S Backbone LOSA-SALT	Circuit	02/25/2014 9:25 AM	02/25/2014 10:28 AM
1373	3-Elevated	2-High	Maintenance Completed - I2 AL2S Backbone BATO-JACK	Hardware	02/27/2014 6:08 AM	02/27/2014 6:12 AM
1394	3-Elevated	1-Critical	Emergency Maintenance Completed - I2 AL2S Core Node DENV	Hardware	02/27/2014 9:31 PM	02/27/2014 10:02 PM
1405	3-Elevated	2-High	Emergency Maintenance Completed - I2 AL2S Backbone ASHB-PITT	Circuit	02/28/2014 9:12 AM	02/28/2014 10:31 AM

Scheduled Maintenances Detail

Ticket No.:	1236:135
Subject:	Maintenance 1 of 2 Completed - I2 AL2S Backbone LOSA-SALT
Affected:	Backbone LOSA-SALT
Start Time:	Tuesday, February 25, 2014, 9:25 AM (0925) UTC
End Time:	Tuesday, February 25, 2014, 10:28 AM (1028) UTC
Description:	Backbone LOSA-SALT was unavailable while Level(3) Communications performed fiber maintenance in San Gabriel, CA as mandated by the California Department of Transportation. Backbone traffic was re-routed during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone. An second window has been reserved for 2/28/14 at 0800-1400.

Ticket No.:	1373:135
Subject:	Maintenance Completed - I2 AL2S Backbone BATO-JACK
Affected:	Backbone BATO-JACK
Start Time:	Thursday, February 27, 2014, 6:08 AM (0608) UTC
End Time:	Thursday, February 27, 2014, 6:12 AM (0612) UTC
Description:	Backbone BATO-JACK was briefly unavailable while Vendor Engineers performed maintenance to change out a line module on the node in Slidell, LA. Maintenace has been completed. Backbone traffic re-routed during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:	1394:135
Subject:	Emergency Maintenance Completed - I2 AL2S Core Node DENV
Affected:	Core Node DENV (Line Card 1) Backbone DENV-KANS Backbone DENV-SALT
Start Time:	Thursday, February 27, 2014, 9:31 PM (2131) UTC
End Time:	Thursday, February 27, 2014, 10:02 PM (2202) UTC
Description:	Core Node DENV (Line Card 1) is available after Internet2 Engineers, with the assistance of Level(3) Communications, performed maintenance to replace the failing line card. The maintenance has been completed and no further outages are expected.

Ticket No.:	1405:135
Subject:	Emergency Maintenance Completed - I2 AL2S Backbone ASHB-PITT
Affected:	Backbone CLEV-WASH
Start Time:	Friday, February 28, 2014, 9:12 AM (0912) UTC
End Time:	Friday, February 28, 2014, 10:31 AM (1031) UTC
Description:	Backbone CLEV-WASH was unavailable while vendor Engineers performed maintenance. Maintenance has been completed. Backbone traffic re-routed during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Downtime and Availability

Internet2 Advanced Layer 2 Service Core Nodes	Down Time		Reporting Period Availability		52 Week Availability
sdn-sw.albu.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
sdn-sw.ashb.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99772%	100.00000%
sdn-sw.atla.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98254%	100.00000%
sdn-sw.bost.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99658%	100.00000%
sdn-sw.char.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
sdn-sw.chic.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.96357%	100.00000%
sdn-sw.clev.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98748%	100.00000%
sdn-sw.dall.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
sdn-sw.denv.net.internet2.edu	0 hr 37 min	0 hr 37 min	99.63294%	99.63294%	99.84650%	99.85902%
sdn-sw.elpa.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98824%	100.00000%
sdn-sw.houh.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.91633%	99.92885%
sdn-sw.jack.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98710%	100.00000%
sdn-sw.jcsn.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98520%	99.98520%
sdn-sw.kans.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98748%	100.00000%
sdn-sw.losa.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98767%	100.00000%
sdn-sw.minn.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98653%	100.00000%
sdn-sw.newy32aoa.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98786%	100.00000%
sdn-sw.phil.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
sdn-sw.phoe.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99791%	100.00000%
sdn-sw.pitt.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99962%	100.00000%
sdn-sw.port.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
sdn-sw.rale.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.71482%	99.73550%
sdn-sw.salt.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98710%	100.00000%
sdn-sw.seat.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99374%	100.00000%
sdn-sw.star.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98672%	100.00000%
sdn-sw.sunn.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98748%	100.00000%
sdn-sw.tuls.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98748%	100.00000%
sdn-sw.wash.net.internet2.edu	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.98691%	100.00000%

Aggregate Internet2 Advanced Layer 2 Service Core Nodes	0 hr 37 min	0 hr 37 min	99.98689%	99.98689%	99.97295%	99.98245%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 Advanced Layer 2 Service Backbone Circuits	Down Time		Reporting Period Availability		52 Week Availability
I2-ASHB-PITT-100GE-07737	2 hr 39 min	1 hr 20 min	98.42262%	99.20635%	99.86510%	99.88065%
I2-ATLA-CHAR-100GE-07738	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97970%	99.97970%
I2-ATLA-CHIC-100GE-07740	0 hr 1 min	0 hr 1 min	99.99008%	99.99008%	99.70647%	99.79945%
I2-ATLA-HOUH-10GE-07742	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
I2-ATLA-JACK-100GE-07739	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99412%	100.00000%
I2-ATLA-JCSN-100GE-10734	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
I2-ATLA-JCSN-100GE-10784	0 hr 1 min	0 hr 1 min	99.99008%	99.99008%	99.77402%	99.77402%
I2-ATLA-WASH-100GE-07738	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
I2-BATO-HOUH-100GE-08860	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.92847%	99.92847%
I2-BATO-JACK-100GE-11005	0 hr 4 min	0 hr 0 min	99.96032%	100.00000%	99.98918%	99.99981%
I2-CHAR-RALE-100GE-10633	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.97970%	99.97970%
I2-CHIC-KANS-100GE-07745	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.95200%	99.97970%
I2-CHIC-STAR-100GE-07743	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.75581%	99.77004%
I2-CLEV-NEWY32AOA-100GE-07735	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.66720%	99.68086%
I2-CLEV-PITT-100GE-10334	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99336%	100.00000%
I2-CLEV-STAR-100GE-07736	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.94991%	99.98368%
I2-DALL-HOUH-100GE-07751	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.92885%	99.92885%
I2-DALL-TULS-100GE-11300	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
I2-DENV-KANS-100GE-07746	1 hr 10 min	0 hr 39 min	99.30556%	99.61310%	99.83075%	99.88141%
I2-DENV-SALT-100GE-07747	1 hr 10 min	0 hr 39 min	99.30556%	99.61310%	99.92676%	99.96983%
I2-ELPA-HOUH-100GE-07749	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.36912%	99.48125%
I2-ELPA-LOSA-100GE-07750	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99981%	100.00000%
I2-HOUH-JCSN-100GE-10783	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.91405%	99.91405%
I2-HOUH-TULS-100GE-07751	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.68029%	99.74461%
I2-KANS-TULS-100GE-07753	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.91462%	99.91519%
I2-LOSA-SALT-100GE-07757	1 hr 3 min	0 hr 0 min	99.37500%	100.00000%	99.91481%	99.93226%
I2-LOSA-SUNN-100GE-07755	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.89716%	99.89830%
I2-NEWY23AOA-PHIL-100GE-07759	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
I2-PHIL-WASH-100GE-10867	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.89451%	100.00000%
I2-PORT-SEAT-100GE-08997	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	100.00000%	100.00000%
I2-PORT-SUNN-100GE-10343	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.99905%	99.99962%
I2-RALE-WASH-100GE-08888	0 hr 40 min	0 hr 40 min	99.60317%	99.60317%	99.94422%	99.97742%
I2-SALT-SUNN-100GE-07766	0 hr 0 min	0 hr 0 min	100.00000%	100.00000%	99.94725%	99.96850%

Aggregate All Internet2 Advanced Layer 2 Service Backbone Circuits	6 hr 48 min	3 hr 28 min	99.87735%	99.93988%	99.89989%	99.92022%

Upcoming Scheduled Maintenances

Ticket No.:	1355:135
Subject:	Maintenance 1 of 2 - I2 AL2S Backbone MINN-SEAT
Affected:	Backbone MINN-SEAT
Start Time:	Sunday, March 9, 2014, 6:01 AM (0601) UTC
End Time:	Sunday, March 9, 2014, 11:00 AM (1100) UTC
Description:	Backbone MINN-SEAT will be unavailable while fiber provider Engineers perform maintenance. An outage up to one hour is expected; however, the entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone. This is the first of two maintenance windows.

Ticket No.:	1397:135
Subject:	Maintenance 1 of 2 - I2 AL2S Backbone MINN-SEAT
Affected:	Backbone MINN-SEAT
Start Time:	Sunday, March 9, 2014, 7:01 AM (0701) UTC
End Time:	Sunday, March 9, 2014, 12:00 PM (1200) UTC
Description:	Window 1 of 2 Backbone MINN-SEAT will be unavailable while Zayo Engineers perform maintenance. An outage is expected, and the entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:	1356:135
Subject:	Maintenance 2 of 2 - I2 AL2S Backbone MINN-SEAT
Affected:	Backbone MINN-SEAT
Start Time:	Monday, March 10, 2014, 6:01 AM (0601) UTC
End Time:	Monday, March 10, 2014, 11:00 AM (1100) UTC
Description:	Backbone MINN-SEAT will be unavailable while fiber provider Engineers perform maintenance. An outage up to one hour is expected; however, the entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone. This is the second of two maintenance windows.

Ticket No.:	1398:135
Subject:	Maintenance 2 of 2 - I2 AL2S Backbone MINN-SEAT
Affected:	Backbone MINN-SEAT
Start Time:	Monday, March 10, 2014, 7:01 AM (0701) UTC
End Time:	Monday, March 10, 2014, 12:00 PM (1200) UTC
Description:	Window 2 of 2 Backbone MINN-SEAT will be unavailable while Zayo Engineers perform maintenance. An outage is expected, and the entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.