Unified - Internet2 Advanced Layer 2 Service Network Operations Report
12/14/2015 - 12/20/2015

Unified - Internet2 Advanced Layer 2 Service Network Operations Center
at Indiana University

New Connections

None


Upgrades

None


New Network Resources

None


Unscheduled Outages Summary

Ticket Number Customer Impact Network Impact Title Outage Type Source Of Impact Start Time (UTC) End Time (UTC)

66414-Normal 1-Critical Brief Outage Resolved - I2 Optical DWDM System HOUS-JCSN Cut or Damaged FiberVendor 12/14/2015 4:41 PM 12/14/2015 4:42 PM
66664-Normal 2-High Brief Outage - I2 AL2S Backbone MINN-MISS2 InvestigatingVendor 12/16/2015 5:18 AM 12/16/2015 5:19 AM
66862-High 2-High Outage Resolved - I2 AL2S Backbone HART2-NEWY32AOA HardwareVendor 12/17/2015 12:02 PM 12/18/2015 2:53 AM
67061-Critical 1-Critical Outage Resolved - I2 Optical DWDM System BATO-JACK Cut or Damaged FiberVendor 12/18/2015 8:20 PM 12/19/2015 1:30 PM
67122-High 2-High Outage Resolved - I2 AL2S Backbone MISS2-SEAT HardwareVendor 12/19/2015 8:37 AM 12/19/2015 3:05 PM

Unscheduled Outages Detail


Ticket No.:6641:144
Subject:Brief Outage Resolved - I2 Optical DWDM System HOUS-JCSN
Source Of Impact:Vendor
Affected:DWDM System HOUS-JCSN
AL2S Backbone HOUS-JCSN
Optical Member NOAA N-Wave
Optical Member ESnet
Optical Member MISSiON
Start Time:Monday, December 14, 2015, 4:41 PM (1641) UTC
End Time:Monday, December 14, 2015, 4:42 PM (1642) UTC
Description:The above listed assets were briefly unavailable to the community. Level (3) Communications Engineers have investigated and determined the outage was caused by a partial fiber cut in Jackson, MS. Fiber crews have repaired the damaged fiber and service migration has successfully restored impacted customers. No further outages are expected.

Ticket No.:6666:144
Subject:Brief Outage - I2 AL2S Backbone MINN-MISS2
Source Of Impact:Vendor
Affected:AL2S Backbone MINN-MISS2
Net+ Provider Code42
Start Time:Wednesday, December 16, 2015, 5:18 AM (0518) UTC
End Time:Wednesday, December 16, 2015, 5:19 AM (0519) UTC
Description:The above listed asset is available following a brief outage. Zayo Engineers are investigating the cause. The reason for the outage is pending. Workgroup administrators receive notifications from OESS regarding circuits re-routed on the backbone.

Ticket No.:6686:144
Subject:Outage Resolved - I2 AL2S Backbone HART2-NEWY32AOA
Source Of Impact:Vendor
Affected:AL2S Backbone HART2-NEWY
IP Backbone CHIC-NEWY
IP Backbone CLEV-NEWY
Start Time:Thursday, December 17, 2015, 12:02 PM (1202) UTC
End Time:Friday, December 18, 2015, 2:53 AM (0253) UTC
Description:This ticket was initially sent out indicating a possible issue with IP services in New York, NY. Internet2 Engineers have determined that there is an issue with the above listed AL2S Backbone from HART2-NEWY. This issue caused a monitoring problem that presented a wide scope of possible impact. The actual impact has been determined as that AL2S Backbone and the associated impact is listed above. Internet2 Engineers emergency maintenance effort was successful in the resolution of this issue. The outage was resolved after a line card was replaced. Due to a configuration error, traffic was degraded briefly while engineers restored the link. Workgroup administrators received notifications from OESS regarding circuits re-routed on the backbone. Backbone traffic re-routed during the impacted times. Issue has been resolved; no further outages are expected.

Ticket No.:6706:144
Subject:Outage Resolved - I2 Optical DWDM System BATO-JACK
Source Of Impact:Vendor
Affected:DWDM System BATO-JACK
AL2S Backbone BATO-JACK
IP Backbone ATLA-HOUS
Start Time:Friday, December 18, 2015, 8:20 PM (2020) UTC
End Time:Saturday, December 19, 2015, 1:30 PM (1330) UTC
Description:The above listed assets are currently available to the community. Level3 Communications Engineers reported the outage was due to fiber damage caused by excavation work being performed in Ponce De Leon, FL. Field Services have completed splicing cable at the damage location. Services have restored and no further outages are anticipated. Closing ticket

Ticket No.:6712:144
Subject:Outage Resolved - I2 AL2S Backbone MISS2-SEAT
Source Of Impact:Vendor
Affected:AL2S Backbone MISS2-SEAT
Net+ Provider Code42
Start Time:Saturday, December 19, 2015, 8:37 AM (0837) UTC
End Time:Saturday, December 19, 2015, 3:05 PM (1505) UTC
Description:The above listed asset is now available. Zayo Engineershave determined the reason for the outage is a hardware issue. The card has been replaced. No further outages are expected. Alarms have cleared, closing ticket

Scheduled Maintenances Summary

Ticket Number Customer Impact Network Impact Title Maintenance Type Source Of Impact Start Time (UTC) End Time (UTC)

None during this report period

Downtime and Availability



Unified - Internet2 Advanced Layer 2 Service Core NodesDown TimeReporting Period Availability52 Week Availability
sdn-sw.albu.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000% 100.00000%
sdn-sw.alba.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99260% 99.99545%
sdn-sw.ashb.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97875% 99.97875%
sdn-sw.atla.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98805% 99.98843%
sdn-sw.bato.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.bost.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.char.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99032% 99.99545%
sdn-sw.clev.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99412% 99.99545%
sdn-sw.colu4.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99393% 99.99545%
sdn-sw.dall.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.denv.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.elpa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.eqch.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99393% 99.99545%
sdn-sw.houh.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98501% 99.99545%
sdn-sw.jack.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.jcsn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.kans.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98900% 99.99545%
sdn-sw.losa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99526% 99.99545%
sdn-sw.minn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.miss2.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.newy32aoa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.phil.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.phoe.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99374% 99.99393%
sdn-sw.pitt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.port.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.rale.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99146% 99.99393%
sdn-sw.salt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99545% 99.99545%
sdn-sw.seat.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98900% 99.99393%
sdn-sw.star.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97154% 99.99469%
sdn-sw.sunn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96243% 99.99545%
sdn-sw.tuls.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99317% 99.99545%
sdn-sw.wash.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99222% 99.99545%
sdn-srv2.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981% 100.00000%
sdn-srv3.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981% 100.00000%

Aggregate Unified - Internet2 Advanced Layer 2 Service Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%99.99217%99.99501%

NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.

Unified - Internet2 Advanced Layer 2 Service Backbone CircuitsDown Time Reporting Period Availability52 Week Availability
I2-ALBA-BOST-100GE-09210 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99602%100.00000%
I2-ALBA-CLEV-100GE-07735 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99260%100.00000%
I2-ASHB-CHIC-100GE-11803 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.86358%99.92866%
I2-ASHB-PITT-100GE-07737 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94346%99.94820%
I2-ASHB-WASH-100GE-09106 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92524%99.92790%
I2-ASHB-WASH-100GE-11823 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.91272%99.95579%
I2-ATLA-CHAR-100GE-07739 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-CINC-100GE-07740 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96357%99.99981%
I2-ATLA-HOUH-10GE-07742 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-JACK-100GE-07739 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96452%99.99621%
I2-ATLA-JCSN-100GE-10734 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ATLA-JCSN-100GE-10784 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.72014%99.84555%
I2-ATLA-CHAR-100GE-07738 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96718%99.99886%
I2-ATLA-WASH-100GE-54536 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%99.99905%
I2-BATO-HOUH-100GE-08860 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98824%100.00000%
I2-BATO-JACK-100GE-11005 17 hr 10 min17 hr 10 min 89.78175%89.78175% 99.57119%99.64177%
I2-BOST-HART2-100GE-09211 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93777%99.94763%
I2-CHAR-RALE-100GE-10633 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.87819%99.88065%
I2-CHIC-INDI-100GE-50993 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92733%99.93245%
I2-CINC-INDI-100GE-51692 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97173%99.99962%
I2-CINC-LOUI-100GE-54197 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99981%99.99981%
I2-CHIC-COLU4-100GE-11554 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.91670%99.97420%
I2-CHIC-KANS-100GE-07745 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95276%99.99886%
I2-CHIC-STAR-100GE-07743 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97021%99.99924%
I2-CLEV-EQCH-100GE-07736 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97078%99.98084%
I2-CLEV-NEWY32AOA-100GE-07735 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-CLEV-PITT-100GE-10334 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99564%100.00000%
I2-CLEV-STAR-100GE-07736 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-COLU4-KANS-100GE-11548 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98900%99.99810%
I2-DALL-HOUH-100GE-07751 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.75619%99.83512%
I2-DALL-TULS-100GE-11300 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.80191%99.83739%
I2-DENV-ELPA-100GE-07748 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99146%99.99867%
I2-DENV-KANS-100GE-07746 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98349%99.99962%
I2-DENV-SALT-100GE-07747 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93511%99.98065%
I2-ELPA-HOUH-100GE-07749 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.65676%99.71387%
I2-ELPA-PHOE-100GE-07750 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-ELPA-TUCS-100GE-07750 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%99.99905%
I2-EQCH-STAR-100GE-09299 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93435%99.99924%
I2-EQCH-MINN-100GE-12184 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.69737%99.72393%
I2-HART2-NEWY32AOA-100GE-12947 14 hr 51 min14 hr 51 min 91.16071%91.16071% 99.72829%99.73057%
I2-HOUH-HOUS-100GE-09379 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.77174%99.80571%
I2-HOUH-JCSN-100GE-10783 0 hr 1 min0 hr 1 min 99.99008%99.99008% 99.95048%99.98273%
I2-HOUH-TULS-100GE-07751 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%100.00000%
I2-KANS-MINN-100GE-12675 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97951%99.98235%
I2-KANS-TULS-100GE-07753 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.93739%99.94972%
I2-LASV-LOSA-100GE-53802 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.90798%99.93359%
I2-LASV-SALT-100GE-07757 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98596%100.00000%
I2-LOSA-PHOE-100GE-09190 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99791%99.99829%
I2-LOSA-SUNN-100GE-07755 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96338%99.99981%
I2-MINN-MISS2-100GE-09130 0 hr 1 min0 hr 1 min 99.99008%99.99008% 98.90729%99.18260%
I2-MISS2-SEAT-100GE-13237 6 hr 28 min6 hr 28 min 96.15079%96.15079% 99.77838%99.81595%
I2-NEWY23AOA-PHIL-100GE-07759 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99374%99.99981%
I2-PHIL-WASH-100GE-10867 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.83701%99.90077%
I2-PHOE-TUCS-100GE-11990 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.94687%99.94706%
I2-PORT-SEAT-100GE-08997 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96224%99.99791%
I2-PORT-SUNN-100GE-10343 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.85219%99.88635%
I2-RALE-WASH-100GE-08888 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98444%99.99791%
I2-SALT-SEAT-100GE-08998 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.74139%99.74632%
I2-RENO-SALT-100GE-53725 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98387%100.00000%
I2-RENO-SUNN-100GE-07766 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.69433%99.74556%

Aggregate All Unified - Internet2 Advanced Layer 2 Service Backbone Circuits 38 hr 31 min0 hr 0 min 99.61789%99.61789% 99.89962%99.92740%


NOTE: Numbers in italics represent downtime with scheduled maintenance excluded. Internet2's definition of scheduled maintenance is any anticipated outage announced at least 72 hours in advance of the outage start time.



Upcoming Scheduled Maintenances

Ticket No.:6738:144
Subject:Maintenance - I2 AL2S Various Circuits (DALL3)
Affected:Connector LEARN
Start Time:Monday, December 28, 2015, 10:00 AM (1000) UTC
End Time:Monday, December 28, 2015, 12:00 PM (1200) UTC
Description: The above listed will be unavailable while Internet2 Engineers re-groom various customers utilizing Internet2 Core Router DALL3. An outage of 30 minutes is expected, however the entire maintenance window has been reserved.

Ticket No.:6735:144
Subject:Maintenance - I2 Optical Member ESnet
Affected:
Start Time:Tuesday, December 29, 2015, 4:00 PM (1600) UTC
End Time:Tuesday, December 29, 2015, 8:00 PM (2000) UTC
Description: Member ESnet's Optical circuit ESNET-LBNL-SUNN-100GE-07835 will be unavailable while their engineers perform maintenance. An outage is expected. The entire maintenance window has been reserved.

Ticket No.:6753:144
Subject:Maintenance - I2 Optical Various Circuits (DALL)
Affected:
Start Time:Wednesday, December 30, 2015, 10:00 AM (1000) UTC
End Time:Wednesday, December 30, 2015, 12:00 PM (1200) UTC
Description: The above listed assets will be unavailable while Internet2 Engineers perform a maintenance to re-groom various customers that are utilizing Internet2 TR-CPS Router at DALL3. An outage of up to 30 minutes is expected; however the entire window is reserved.

Ticket No.:6657:144
Subject:Maintenance - I2 Optical DWDMs CLEV-DETR, DETR-EQCH DWDM
Affected:DWDM System CLEV-DETR
DWDM System DETR-EQCH
AL2S Backbones(s) ASHB-CHIC and CLEV-EQCH
IP Backbone CHIC-CLEV, CHIC-NEWY, and CHIC-WASH
Customer NORDUnet
Customer Merit Network
Optical Member ESnet
Optical Member NOAA N-Wave
Start Time:Thursday, January 7, 2016, 6:00 AM (0600) UTC
End Time:Thursday, January 7, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Internet2 Engineers move the optical node in Detroit to a new rack location. Outages are expected. The entire window is reserved.

Ticket No.:6648:144
Subject:Maintenance 1 of 2 - I2 Optical DWDM System ASHB-PITT
Affected:DWDM System ASHB-PITT
AL2S Backbone(s) ASHB-PITT and ASHB-CHIC
IP Backbone(s) ATLA-CLEV, CHIC-WASH, CLEV-WASH
Connector 3ROX (ASHB)
Customer NORDUnet
Optical Member ESnet
Optical Member NOAA N-Wave
Start Time:Saturday, January 9, 2016, 5:00 AM (0500) UTC
End Time:Saturday, January 9, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Level(3) Communications perform maintenance to relocate cable in Hancock, MD. An outage of 6 hours is expected; however, the entire window is reserved.

Ticket No.:6650:144
Subject:Maintenance 2 of 2 - I2 Optical DWDM System ASHB-PITT
Affected:DWDM System ASHB-PITT
AL2S Backbone(s) ASHB-PITT and ASHB-CHIC
IP Backbone(s) ATLA-CLEV, CHIC-WASH, CLEV-WASH
Connector 3ROX (ASHB)
Customer NORDUnet
Optical Member ESnet
Optical Member NOAA N-Wave
Start Time:Sunday, January 10, 2016, 5:00 AM (0500) UTC
End Time:Sunday, January 10, 2016, 11:00 AM (1100) UTC
Description: The above listed assets will be unavailable while Level(3) Communications perform maintenance to relocate cable in Hancock, MD. An outage of 6 hours is expected; however, the entire window is reserved. This is an alternate maintenance window if needed.

Ticket No.:6249:144
Subject:Maintenance - I2 Core Node SING
Affected:Core Node Singapore and its Connectors and Participants
Start Time:Thursday, January 14, 2016, 1:00 AM (0100) UTC
End Time:Friday, January 22, 2016, 1:00 PM (1300) UTC
Description: The above listed asset is expected to remain available while TATA performs power maintenance. This power maintenance is set to happen one feed at a time to avoid unnecessary impact. It is a multi-day event. No outages are expected. The entire window is reserved. All connections to the Core Node will be affected in the unlikely event of impact by this maintenance.