Internet2 Advanced Layer 2 Service Network Operations Report
01/27/2014 - 02/02/2014

Internet2 Advanced Layer 2 Service Network Operations Center
at Indiana University

New Connections

None


Upgrades
OESS 1.1.2a, 01/28/2014 @ 23:00 UTC

None


New Services

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Start Time (UTC) End Time (UTC)

12602-High 2-High Outage Resolved - I2 AL2S Various Backbones Software 01/28/2014 7:21 PM 01/28/2014 9:08 PM
12722-High 2-High Outage Resolved - I2 AL2S Backbone CLEV-STAR Unannounced Maintenance 01/30/2014 8:56 PM 01/30/2014 9:57 PM
12732-High 2-High Outage Resolved - I2 AL2S Backbone MINN-SEAT Unannounced Maintenance 01/31/2014 1:00 PM 01/31/2014 1:48 PM

Unscheduled Outages Detail


Ticket No.:1260:135
Subject:Outage Resolved - I2 AL2S Various Backbones
Affected:Backbone ATLA-CHAR
Backbone ATLA-CHIC
Backbone CHAR-RALE
Backbone CHIC-KANS
Backbone DENV-KANS
Backbone DENV-SALT
Start Time:Tuesday, January 28, 2014, 7:21 PM (1921) UTC
End Time:Tuesday, January 28, 2014, 9:08 PM (2108) UTC
Description:Internet2 isolated a degraded condition issue to an interaction between the OE-SS software and a newly deployed version of OSCARS. The OSCARS software tried to delete a circuit more than once, triggering a bug in the OE-SS software that interpreted the removal as a wildcard rule removal. Internet2 has deployed a patch to resolve the issue and expects no further outages.

Ticket No.:1272:135
Subject:Outage Resolved - I2 AL2S Backbone CLEV-STAR
Affected:Backbone CLEV-STAR
Start Time:Thursday, January 30, 2014, 8:56 PM (2056) UTC
End Time:Thursday, January 30, 2014, 9:57 PM (2157) UTC
Description:Backbone CLEV-STAR now available after Level(3) Engineers performed Emergency maintenance. This is a temporary fix to be with a future maintenance for permanent correction. Backbone traffic re-routed during this outage. The maintenance has been completed and no further outages are expected.

Ticket No.:1273:135
Subject:Outage Resolved - I2 AL2S Backbone MINN-SEAT
Affected:Backbone MINN-SEAT
Start Time:Friday, January 31, 2014, 1:00 PM (1300) UTC
End Time:Friday, January 31, 2014, 1:48 PM (1348) UTC
Description:Backbone MINN-SEAT is now available after Zayo Engineers performed maintenance. The maintenance ran past maintenance window due to a need to correct splicing performed during the window. Backbone traffic re-routed during this outage. Workgroup administrators received notifications from OESS regarding circuits re- routed on the backbone. The reason for outage is pending.

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Start Time (UTC) End Time (UTC)

1235 3-Elevated 2-High Maintenance Completed - I2 AL2S Backbone DENV-SALT Hardware 01/28/2014 6:11 AM 01/28/2014 6:25 AM
1185 3-Elevated 2-High Maintenance 1 of 2 Completed - I2 AL2S Backbone ATLA-CHIC Circuit 01/29/2014 6:01 AM 01/29/2014 9:24 AM
1178 3-Elevated 2-High Maintenanc 2 of 2 Completed- I2 AL2S Backbone MINN-SEAT Circuit 01/31/2014 10:40 AM 01/31/2014 1:00 PM

Scheduled Maintenances Detail

Ticket No.:1235:135
Subject:Maintenance Completed - I2 AL2S Backbone DENV-SALT
Affected:Backbone DENV-SALT
Start Time:Tuesday, January 28, 2014, 6:11 AM (0611) UTC
End Time:Tuesday, January 28, 2014, 6:25 AM (0625) UTC
Description: Backbone DENV-SALT was briefly unavailable while Level(3) Communications performed maintenance to troubleshoot an Amp Pump Failure between Denver, CO and Platteville, CO. Backbone traffic will re- route during this maintenance. Maintenance has been completed. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:1185:135
Subject:Maintenance 1 of 2 Completed - I2 AL2S Backbone ATLA-CHIC
Affected:Backbone CHIC-ATLA
Start Time:Wednesday, January 29, 2014, 6:01 AM (0601) UTC
End Time:Wednesday, January 29, 2014, 9:24 AM (0924) UTC
Description: Backbone CHIC-ATLA was unavailable while vendor Engineers performed maintenance. Outages were observed throughout the maintenance window. Backbone traffic was re-routed during this maintenance. Workgroup administrators should have received notifications from OESS regarding circuits re-routed on the backbone. Maintenance has been completed. This is the first of two maintenance windows.

Ticket No.:1178:135
Subject:Maintenanc 2 of 2 Completed- I2 AL2S Backbone MINN-SEAT
Affected:Backbone MINN-SEAT
Start Time:Friday, January 31, 2014, 10:40 AM (1040) UTC
End Time:Friday, January 31, 2014, 1:00 PM (1300) UTC
Description: Backbone MINN-SEAT was unavailable while Zayo Engineers performed maintenance. Backbone traffic will re-route during thisoutage. Workgroup administrators will receive notifications from OESS regarding circuits re- routed on the backbone.


Downtime and Availability



Internet2 Advanced Layer 2 Service Core NodesDown TimeReporting Period Availability52 Week Availability
sdn-sw.albu.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
sdn-sw.ashb.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99772% 100.00000%
sdn-sw.atla.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97495% 100.00000%
sdn-sw.bost.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99658% 100.00000%
sdn-sw.char.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
sdn-sw.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95560% 100.00000%
sdn-sw.clev.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970% 100.00000%
sdn-sw.dall.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
sdn-sw.denv.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84574% 99.86604%
sdn-sw.elpa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98027% 100.00000%
sdn-sw.houh.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970% 100.00000%
sdn-sw.jack.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970% 100.00000%
sdn-sw.jcsn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98520% 99.98520%
sdn-sw.kans.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97951% 100.00000%
sdn-sw.losa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97989% 100.00000%
sdn-sw.minn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98653% 100.00000%
sdn-sw.newy32aoa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98008% 100.00000%
sdn-sw.phil.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
sdn-sw.phoe.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99791% 100.00000%
sdn-sw.pitt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99962% 100.00000%
sdn-sw.port.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
sdn-sw.rale.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.69642% 99.73550%
sdn-sw.salt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970% 100.00000%
sdn-sw.seat.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99374% 100.00000%
sdn-sw.star.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97894% 100.00000%
sdn-sw.sunn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970% 100.00000%
sdn-sw.tuls.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970% 100.00000%
sdn-sw.wash.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97913% 100.00000%

Aggregate Internet2 Advanced Layer 2 Service Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%99.97093%99.98524%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Internet2 Advanced Layer 2 Service Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
I2-ASHB-PITT-100GE-07737 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89811%99.89868%
I2-ATLA-CHAR-100GE-07738 1 hr 47 min1 hr 47 min 98.93849%98.93849% 99.97970%99.97970%
I2-ATLA-CHIC-100GE-07740 5 hr 10 min1 hr 47 min 96.92460%98.93849% 99.55734%99.67023%
I2-ATLA-HOUH-10GE-07742 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96148%99.96148%
I2-ATLA-JACK-100GE-07739 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99412%100.00000%
I2-ATLA-JCSN-100GE-10734 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-WASH-100GE-07738 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99526%100.00000%
I2-BATO-HOUH-100GE-08860 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99962%99.99962%
I2-BATO-JACK-100GE-11005 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%99.99981%
I2-CHAR-RALE-100GE-10633 1 hr 47 min1 hr 47 min 98.93849%98.93849% 99.97970%99.97970%
I2-CHIC-KANS-100GE-07745 1 hr 47 min1 hr 47 min 98.93849%98.93849% 99.95200%99.97970%
I2-CHIC-STAR-100GE-07743 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.75581%99.77004%
I2-CLEV-NEWY32AOA-100GE-07735 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.57366%99.60819%
I2-CLEV-PITT-100GE-10334 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99336%100.00000%
I2-CLEV-STAR-100GE-07736 1 hr 1 min1 hr 1 min 99.39484%99.39484% 99.93625%99.98368%
I2-DALL-HOUH-100GE-07751 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-DALL-TULS-100GE-11300 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-DENV-KANS-100GE-07746 1 hr 47 min1 hr 47 min 98.93849%98.93849% 99.84403%99.88881%
I2-DENV-SALT-100GE-07747 2 hr 1 min1 hr 47 min 98.79960%98.93849% 99.93321%99.97932%
I2-ELPA-HOUH-100GE-07749 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.37310%99.48144%
I2-ELPA-LOSA-100GE-07750 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%100.00000%
I2-HOUH-JCSN-100GE-10783 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98520%99.98520%
I2-HOUH-TULS-100GE-07751 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.65619%99.74461%
I2-KANS-TULS-100GE-07753 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.91462%99.91519%
I2-LOSA-SALT-100GE-07757 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92638%99.93188%
I2-LOSA-SUNN-100GE-07755 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89716%99.89830%
I2-NEWY23AOA-PHIL-100GE-07759 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-PHIL-WASH-100GE-10867 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.89451%100.00000%
I2-PORT-SEAT-100GE-08997 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-PORT-SUNN-100GE-10343 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%99.99962%
I2-SALT-SUNN-100GE-07766 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94725%99.96850%

Aggregate All Internet2 Advanced Layer 2 Service Backbone Circuits 15 hr 20 min3 hr 37 min 99.70558%99.77503% 99.90151%99.92335%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:1236:135
Subject:Maintenance 1 of 2 - I2 AL2S Backbone LOSA-SALT
Affected:Backbone LOSA-SALT
Start Time:Tuesday, February 25, 2014, 8:00 AM (0800) UTC
End Time:Tuesday, February 25, 2014, 2:00 PM (1400) UTC
Description: Backbone LOSA-SALT will be unavailable while Level(3) Communications performs fiber maintenance in San Gabriel, CA as mandated by the California Department of Transportation. The entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re- routed on the backbone. An alternate window has been reserved for 2/28/14 at 0800-1400, should it be required.

Ticket No.:1239:135
Subject:Maintenance 2 of 2 - I2 AL2S Backbone LOSA-SALT
Affected:Backbone LOSA-SALT
Start Time:Friday, February 28, 2014, 8:00 AM (0800) UTC
End Time:Friday, February 28, 2014, 2:00 PM (1400) UTC
Description: Backbone LOSA-SALT will be unavailable while Level(3) Communications performs fiber maintenance in San Gabriel, CA as mandated by the California Department of Transportation. The entire window is reserved. Backbone traffic will re-route during this maintenance. Workgroup administrators will receive notifications from OESS regarding circuits re-routed on the backbone. This is an alternate window for ticket 1236:135, should it be required.