Internet2 Advanced Layer 2 Service Network Operations Report
07/21/2014 - 07/27/2014

Internet2 Advanced Layer 2 Service Network Operations Center
at Indiana University

New Connections

None


Upgrades

None

New Network Resources

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Source Of Impact
Start Time (UTC)

End Time (UTC)





20012-High
2-High

Outage Resolved - I2 AL2S Backbone DENV-KANS
HardwareInternal
07/21/2014 3:00 AM
07/21/2014 2:49 PM


20022-High
2-High

Stability - I2 AL2S Backbone MINN-SEAT
Cut or Damaged FiberVendor
07/21/2014 7:45 AM
07/21/2014 8:16 AM


07/21/2014 11:08 PM
07/21/2014 11:18 PM


20052-High
2-High

Brief Outage - I2 AL2S Backbone ATLA-JCSN
InvestigatingUndetermined
07/21/2014 5:30 PM
07/21/2014 5:31 PM


20192-High
2-High

Brief Outage Resolved - I2 AL2S Backbone BATO-JACK
OtherVendor
07/23/2014 7:05 PM
07/23/2014 7:06 PM


07/23/2014 8:12 PM
07/23/2014 8:13 PM


20202-High
2-High

Brief Outage Resolved - I2 AL2S Backbone BATO-JACK
Human ErrorVendor
07/24/2014 4:13 PM
07/24/2014 4:14 PM


20232-High
2-High

Instability Resolved - I2 AL2S Backbone DENV-SALT
HardwareVendor
07/25/2014 4:32 AM
07/25/2014 1:30 PM






Unscheduled Outages Detail






Ticket No.:2001:135



Subject:Outage Resolved - I2 AL2S Backbone DENV-KANS



Source Of Impact:Internal



Affected:Backbone DENV-KANS



Start Time:Monday, July 21, 2014, 3:00 AM (0300) UTC



End Time:Monday, July 21, 2014, 2:49 PM (1449) UTC


Description:Backbone DENV-KANS  is now available. Internet2 Engineers identified a
hardware issue in Denver, CO.  The issue was able to be resolved via
remote hands operations. Workgroup administrators receive
notifications from OESS
regarding circuits re-routed on the backbone.







Ticket No.:2002:135



Subject:Stability - I2 AL2S Backbone MINN-SEAT



Source Of Impact:Vendor



Affected:Backbone MINN-SEAT



Start Time:Monday, July 21, 2014, 7:45 AM (0745) UTC



End Time:Monday, July 21, 2014, 8:16 AM (0816) UTC


Start Time:Monday, July 21, 2014, 11:08 PM (2308) UTC



End Time:Monday, July 21, 2014, 11:18 PM (2318) UTC


Description:Backbone MINN-SEAT is stable. The Internet2 fiber provider has
advised that there was a span loss event in the Fargo, ND region. This
circuit has been stable for 4+ hours.  Workgroup administrators
received notifications from OESS regarding circuits re-routed on the
backbone.







Ticket No.:2005:135



Subject:Brief Outage - I2 AL2S Backbone ATLA-JCSN



Source Of Impact:Undetermined



Affected:Backbone ATLA-JCSN



Start Time:Monday, July 21, 2014, 5:30 PM (1730) UTC



End Time:Monday, July 21, 2014, 5:31 PM (1731) UTC


Description:Backbone ATLA-JCSN  is available after a brief outage. Internet2 Fiber
Vendor Engineers are
investigating the cause; however, the reason for the outage is
pending. Workgroup administrators receive notifications from OESS
regarding circuits re-routed on the backbone.







Ticket No.:2019:135



Subject:Brief Outage Resolved - I2 AL2S Backbone BATO-JACK



Source Of Impact:Vendor



Affected:Backbone BATO-JACK



Start Time:Wednesday, July 23, 2014, 7:05 PM (1905) UTC



End Time:Wednesday, July 23, 2014, 7:06 PM (1906) UTC


Start Time:Wednesday, July 23, 2014, 8:12 PM (2012) UTC



End Time:Wednesday, July 23, 2014, 8:13 PM (2013) UTC


Description:Backbone BATO-JACK is currently available to the
community after a brief outage.  Internet2's fiber vendor have
attributed this to severe weather in the Northern Florida
area.  Workgroup administrators receive notifications from OESS
regarding circuits
re-routed on the backbone.







Ticket No.:2020:135



Subject:Brief Outage Resolved - I2 AL2S Backbone BATO-JACK



Source Of Impact:Vendor



Affected:Backbone BATO-JACK



Start Time:Thursday, July 24, 2014, 4:13 PM (1613) UTC



End Time:Thursday, July 24, 2014, 4:14 PM (1614) UTC


Description:Backbone BATO-JACK was briefly unavailable.  Level 3  Communications 
has indicated that the reason for the outage is a fiber bump.  Workgroup 
administrators receive notifications from OESS regarding circuits re-routed 
on the backbone.







Ticket No.:2023:135



Subject:Instability Resolved - I2 AL2S Backbone DENV-SALT



Source Of Impact:Vendor



Affected:Backbone DENV-SALT



Start Time:Friday, July 25, 2014, 4:32 AM (0432) UTC



End Time:Friday, July 25, 2014, 1:30 PM (1330) UTC


Description:Backbone DENV-SALT is now stable. Internet2 and Level(3)
Communications Engineers found that the outage was due to failed
hardware, which was replaced.  No further outages are expected.  The 
workgroup administrators receive notifications from OESS regarding circuits 
re-routed on the backbone.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Source Of Impact
Start Time (UTC)
End Time (UTC)





1988
3-Elevated
3-Elevated

Maintenance Completed - I2 AL2S Backbones CHIC-KANS and CHIC-COLU4
Hardware
Vendor07/21/2014 5:48 AM
07/21/2014 7:11 AM


1989
3-Elevated
3-Elevated

Maintenance Completed - I2 AL2S Backbone ELPA-HOUH
CircuitVendor
07/22/2014 5:07 AM
07/22/2014 5:21 AM




Scheduled Maintenances Detail





Ticket No.:1988:135



Subject:Maintenance Completed - I2 AL2S Backbones CHIC-KANS and CHIC-COLU4



Source Of Impact:Vendor



Affected:Backbone CHIC-KANS
Backbone CHIC-COLU4



Start Time:Monday, July 21, 2014, 5:48 AM (0548) UTC



End Time:Monday, July 21, 2014, 7:11 AM (0711) UTC


Description:   Backbones CHIC-KANS and CHIC-COLU4 were unavailable while Level(3)
Communications performed maintenance to replace a card in Plainfield, 
IL. Maintenance did not go as planned when vendor technicians 
discovered that the replacement card was dead on arrival. A future 
maintenance will be scheduled to perform card replacement.  
Technicians are hands off and affected connections are available and 
stable. Maintenance has been completed.



  







Ticket No.:1989:135



Subject:Maintenance Completed - I2 AL2S Backbone ELPA-HOUH



Source Of Impact:Vendor



Affected:Backbone ELPA-HOUH



Start Time:Tuesday, July 22, 2014, 5:07 AM (0507) UTC



End Time:Tuesday, July 22, 2014, 5:21 AM (0521) UTC


Description:   Backbone ELPA-HOUH was unavailable while fiber provider Engineers
performed maintenance.  The maintenance has been completed.  Backbone
traffic was re-routed during this maintenance. Workgroup
administrators  received notifications from OESS regarding circuits
re-routed on the backbone.



  










Downtime and Availability







Internet2 Advanced Layer 2 Service Core NodesDown TimeReporting Period Availability52 Week Availability


sdn-sw.albu.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.alba.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%  100.00000%
sdn-sw.ashb.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%  100.00000%
sdn-sw.atla.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
sdn-sw.bato.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%  100.00000%
sdn-sw.bost.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99772%  100.00000%
sdn-sw.char.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96471%  99.98975%
sdn-sw.clev.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
sdn-sw.colu4.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.dall.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.denv.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.85864%  99.85902%
sdn-sw.elpa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
sdn-sw.eqch.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.houh.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92828%  99.92885%
sdn-sw.jack.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96319%  100.00000%
sdn-sw.jcsn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98425%  99.98520%
sdn-sw.kans.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99886%  99.99943%
sdn-sw.losa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
sdn-sw.minn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98596%  100.00000%
sdn-sw.newy32aoa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99962%  100.00000%
sdn-sw.phil.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.phoe.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.pitt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99962%  100.00000%
sdn-sw.port.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.rale.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99810%  99.99886%
sdn-sw.salt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%  100.00000%
sdn-sw.seat.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%  100.00000%
sdn-sw.star.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%  100.00000%
sdn-sw.sunn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
sdn-sw.tuls.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98444%  100.00000%
sdn-sw.wash.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98273%  99.98387%
      
Aggregate Internet2 Advanced Layer 2 Service Core Nodes0 hr 0 min0 hr 0 min100.00000%100.00000%99.98857%99.99203%

 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  
	    

	    
Internet2 Advanced Layer 2 Service Backbone CircuitsDown Time
Reporting Period Availability52 Week Availability


I2-ALBA-BOST-100GE-09210
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98634%100.00000%
I2-ALBA-CLEV-100GE-07735
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97932%99.99981%
I2-ASHB-CHIC-100GE-11803
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89811%99.98975%
I2-ASHB-PITT-100GE-07737
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.77117%99.88065%
I2-ASHB-WASH-100GE-09106
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89261%99.98368%
I2-ASHB-WASH-100GE-11823
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89261%99.98311%
I2-ATLA-CHAR-100GE-07738
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96812%99.97932%
I2-ATLA-CHIC-100GE-07740
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.52546%99.71577%
I2-ATLA-HOUH-10GE-07742
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-JACK-100GE-07739
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.95617%100.00000%
I2-ATLA-JCSN-100GE-10734
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-JCSN-100GE-10784
               0 hr 1 min0 hr 1 min
               99.99008%99.99008%
               99.44824%99.50345%
I2-ATLA-WASH-100GE-07738
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-BATO-HOUH-100GE-08860
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.90361%99.92847%
I2-BATO-JACK-100GE-11005
               0 hr 3 min0 hr 3 min
               99.97024%99.97024%
               99.62413%99.73228%
I2-BOST-NEWY32AOA-100GE-09211
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98160%100.00000%
I2-CHAR-RALE-100GE-10633
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96774%99.97970%
I2-CHIC-COLU4-100GE-11554
               1 hr 23 min0 hr 0 min
               99.17659%100.00000%
               99.94403%99.97856%
I2-CHIC-KANS-100GE-07745
               1 hr 23 min0 hr 0 min
               99.17659%100.00000%
               99.72905%99.91006%
I2-CHIC-STAR-100GE-07743
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.67099%99.78370%
I2-CLEV-EQCH-100GE-07736
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-CLEV-NEWY32AOA-100GE-07735
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99962%100.00000%
I2-CLEV-PITT-100GE-10334
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.90323%100.00000%
I2-CLEV-STAR-100GE-07736
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.86737%99.98596%
I2-COLU4-KANS-100GE-11548
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98197%99.98235%
I2-DALL-HOUH-100GE-07751
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.82980%99.87325%
I2-DALL-TULS-100GE-11300
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.82886%99.84460%
I2-DENV-ELPA-100GE-07748
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97875%99.99260%
I2-DENV-KANS-100GE-07746
               11 hr 48 min11 hr 48 min
               92.97619%92.97619%
               99.70458%99.74727%
I2-DENV-SALT-100GE-07747
               8 hr 58 min8 hr 58 min
               94.66270%94.66270%
               99.83132%99.86775%
I2-ELPA-HOUH-100GE-07749
               0 hr 14 min0 hr 0 min
               99.86111%100.00000%
               99.33155%99.39170%
I2-ELPA-PHOE-100GE-07750
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89507%99.91595%
I2-ELPA-TUCS-100GE-07750
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-EQCH-STAR-100GE-09299
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96300%99.97439%
I2-EQCH-MINN-100GE-12184
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99753%100.00000%
I2-HOUH-JCSN-100GE-10783
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.91177%99.91405%
I2-HOUH-TULS-100GE-07751
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.68807%99.75068%
I2-KANS-TULS-100GE-07753
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89413%99.99981%
I2-LOSA-PHOE-100GE-09190
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.78408%99.80874%
I2-LOSA-SALT-100GE-07757
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.88730%99.93226%
I2-LOSA-SUNN-100GE-07755
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99355%99.99943%
I2-MINN-SEAT-100GE-09130
               0 hr 41 min0 hr 41 min
               99.59325%99.59325%
               99.30897%99.53647%
I2-NEWY23AOA-PHIL-100GE-07759
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99431%99.99981%
I2-PHIL-WASH-100GE-10867
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.87269%99.98368%
I2-PORT-SEAT-100GE-08997
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98900%100.00000%
I2-PORT-SUNN-100GE-10343
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98843%99.99810%
I2-RALE-WASH-100GE-08888
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.77193%99.81197%
I2-SALT-SUNN-100GE-07766
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97287%99.99545%
      
Aggregate All Internet2 Advanced Layer 2 Service Backbone Circuits
                   24 hr 31 min3 hr 0 min
                   99.69597%99.73318%
                   99.86061%99.90947%


 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  

Upcoming Scheduled Maintenances





Ticket No.:2014:135



Subject:Maintenance - I2 AL2S Core Node WASH



Affected:Core Node WASH
AL2S to IP Interconnect



Start Time:Tuesday, July 29, 2014, 4:00 AM (0400) UTC



End Time:Tuesday, July 29, 2014, 6:00 AM (0600) UTC



Description:   Core Node WASH will be unavailable while Internet2 Engineers execute
power maintenance. Outages are expected during the window. The entire
window is reserved. All connections to the core node will be affected.



  







Ticket No.:2007:135



Subject:Maintenance - I2 AL2S Backbone COLU4-KANS



Affected:Backbone COLU4-KANS



Start Time:Tuesday, July 29, 2014, 5:00 AM (0500) UTC



End Time:Tuesday, July 29, 2014, 10:00 AM (1000) UTC



Description:   Backbone COLU4-KANS is involved in a maintenance being executed by an
Internet2 fiber vendor.  No outages are expected.  The entire window
is reserved.



  







Ticket No.:2010:135



Subject:Maintenance 1 of 2 - I2 AL2S Backbones  ATLA-CHIC and ATLA-JCSN 



Affected:Backbone ATLA-CHIC
Backbone ATLA-JCSN



Start Time:Friday, August 1, 2014, 4:00 AM (0400) UTC



End Time:Friday, August 1, 2014, 10:00 AM (1000) UTC



Description:   Backbones ATLA-CHIC and ATLA-JCSN will be unavailable while fiber
provider Engineers perform maintenance.  An outage is expected, and
the entire window is reserved.  Backbone traffic will re-route during
this maintenance. Workgroup administrators will receive notifications
from OESS regarding circuits re-routed on the backbone. This is the
first of two maintenance windows.



  







Ticket No.:2011:135



Subject:Maintenance 2 of 2 - I2 AL2S Backbones ATLA-CHIC and ATLA-JCSN



Affected:Backbone ATLA-CHIC
Backbone ATLA-JCSN



Start Time:Saturday, August 2, 2014, 4:00 AM (0400) UTC



End Time:Saturday, August 2, 2014, 10:00 AM (1000) UTC



Description:   Backbones ATLA-CHIC and ATLA-JCSN will be unavailable while fiber
provider Engineers perform maintenance.  An outage is expected, and
the entire window is reserved.  Backbone traffic will re-route during
this maintenance. Workgroup administrators will receive notifications
from OESS regarding circuits re-routed on the backbone. This is the
second of two maintenance windows.



  







Ticket No.:2025:135



Subject:Maintenance - I2 AL2S Backbone DENV-KANS 



Affected:Backbone DENV-KANS



Start Time:Tuesday, August 5, 2014, 6:00 AM (0600) UTC



End Time:Tuesday, August 5, 2014, 8:00 AM (0800) UTC



Description:   Backbone DENV-KANS will be unavailable while Level(3)
Communications perform hardware maintenance in Bethune, CO. Outages
are expected, the entire window is reserved.  Backbone traffic
will re-route during this maintenance. Workgroup administrators will
receive notifications from OESS regarding circuits re-routed on the
backbone.