Internet2 Advanced Layer 2 Service Network Operations Report
12/16/2013 - 12/22/2013

Internet2 Advanced Layer 2 Service Network Operations Center
at Indiana University

New Connections

None


Upgrades
OESS 1.1.1.1, 12/16/2013 @ 10:00 UTC

OESS 1.1.1.1a, 12/19/2013 @ 08:00 UTC

New Services

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Start Time (UTC)

End Time (UTC)





11182-High
2-High

Brief Outage Resolved - I2 AL2S Backbone ASHB-PITT
Circuit - Bumped Fiber
12/16/2013 6:52 PM
12/16/2013 6:53 PM


12/16/2013 6:54 PM
12/16/2013 6:55 PM


11242-High
2-High

Outage Resolved - I2 AL2S Backbone CHIC-STAR
Hardware
12/17/2013 8:00 AM
12/17/2013 6:53 PM


11302-High
2-High

Brief Outage - I2 AL2S Backbone BATO-HOUH
Undetermined
12/20/2013 10:43 AM
12/20/2013 10:44 AM


11352-High
2-High

Brief Outage - I2 AL2S Backbone DENV-SALT
Undetermined
12/20/2013 4:28 PM
12/20/2013 4:29 PM


11362-High
2-High

Brief Outage Resolved - I2 AL2S Backbone SALT-SUNN
Circuit - Bumped Fiber
12/20/2013 5:34 PM
12/20/2013 5:35 PM


11432-High
2-High

Outage Resolved - I2 AL2S Backbones ATLA-JCSN and HOUH-JCSN
Hardware
12/22/2013 8:25 AM
12/22/2013 9:43 AM


11441-Critical
1-Critical

Outage Resolved - I2 AL2S Core Node JCSN
Hardware
12/22/2013 8:25 AM
12/22/2013 9:43 AM






Unscheduled Outages Detail






Ticket No.:1118:135



Subject:Brief Outage Resolved - I2 AL2S Backbone ASHB-PITT



Affected:Backbone ASHB-PITT



Start Time:Monday, December 16, 2013, 6:52 PM (1852) UTC



End Time:Monday, December 16, 2013, 6:53 PM (1853) UTC


Start Time:Monday, December 16, 2013, 6:54 PM (1854) UTC



End Time:Monday, December 16, 2013, 6:55 PM (1855) UTC


Description:Backbone ASHB-PITT was briefly unavailable. Internet2's fiber provider
has reported that the RFO is a fiber bump.  Workgroup administrators 
receive notifications from OESS regarding circuits re-routed on the 
backbone.







Ticket No.:1124:135



Subject:Outage Resolved - I2 AL2S Backbone CHIC-STAR



Affected:Backbone CHIC-STAR



Start Time:Tuesday, December 17, 2013, 8:00 AM (0800) UTC



End Time:Tuesday, December 17, 2013, 6:53 PM (1853) UTC


Description:Backbone CHIC-STAR is now available. The reason for the outage was
do to hardware failure during troubleshooting on AL2S Core Node STAR.
The failed card has been replaced and no further outages are to be
expected.







Ticket No.:1130:135



Subject:Brief Outage - I2 AL2S Backbone BATO-HOUH



Affected:Backbone BATO-HOUH



Start Time:Friday, December 20, 2013, 10:43 AM (1043) UTC



End Time:Friday, December 20, 2013, 10:44 AM (1044) UTC


Description:Backbone BATO-HOUH was briefly unavailable. Internet2 Layer 1
Engineers are
investigating the cause; however, the reason for the outage is
pending.  Workgroup administrators receive notifications from OESS
regarding circuits re-routed on the backbone.







Ticket No.:1135:135



Subject:Brief Outage - I2 AL2S Backbone DENV-SALT



Affected:Backbone DENV-SALT



Start Time:Friday, December 20, 2013, 4:28 PM (1628) UTC



End Time:Friday, December 20, 2013, 4:29 PM (1629) UTC


Description:Backbone DENV-SALT  was briefly unavailable. Internet2 Engineers are
investigating the cause; however, the reason for the outage is
pending.  Workgroup administrators receive notifications from OESS
regarding circuits re-routed on the backbone.







Ticket No.:1136:135



Subject:Brief Outage Resolved - I2 AL2S Backbone SALT-SUNN



Affected:Backbone SALT-SUNN



Start Time:Friday, December 20, 2013, 5:34 PM (1734) UTC



End Time:Friday, December 20, 2013, 5:35 PM (1735) UTC


Description:Backbone SALT-SUNN  was briefly unavailable. Vendor engineers report
the reason for outage was a fiber bump. No further outages are
expected.  Workgroup administrators received notifications from OESS
regarding circuits re-routed on the backbone.







Ticket No.:1143:135



Subject:Outage Resolved - I2 AL2S Backbones ATLA-JCSN and HOUH-JCSN



Affected:Backbone ATLA-JCSN
Backbone HOUH-JCSN



Start Time:Sunday, December 22, 2013, 8:25 AM (0825) UTC



End Time:Sunday, December 22, 2013, 9:43 AM (0943) UTC


Description:Backbones ATLA-JCSN and HOUH-JCSN are now available. Internet2
Engineers report the reason for the outage was Routing Engine failure on
AL2S Core Node JCSN.  No further outages are expected. Workgroup
administrators receive notifications from OESS regarding circuits
re-routed
on the backbone.







Ticket No.:1144:135



Subject:Outage Resolved - I2 AL2S Core Node JCSN



Affected:Core Node JCSN and its Connector(s) and Participants



Start Time:Sunday, December 22, 2013, 8:25 AM (0825) UTC



End Time:Sunday, December 22, 2013, 9:43 AM (0943) UTC


Description:Core Node JCSN is now available. Internet2 Engineers report
the reason for the outage was Routing Engine failure on AL2S Core Node
JCSN. All connections to the Core Node were affected.  No further outages 
are expected.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Start Time (UTC)
End Time (UTC)





1122
3-Elevated
2-High

Emergency Maintenance Completed - I2 AL2S Backbone CHIC-STAR
Hardware
12/17/2013 6:57 AM
12/17/2013 7:59 AM


1115
3-Elevated
2-High

Maintenance 2 of 2 Completed - I2 AL2S Backbone HOUH-JACK
Hardware
12/20/2013 6:12 AM
12/20/2013 8:36 AM




Scheduled Maintenances Detail





Ticket No.:1122:135



Subject:Emergency Maintenance Completed - I2 AL2S Backbone CHIC-STAR



Affected:Backbone CHIC-STAR
Connector ICCN/BlueWaters



Start Time:Tuesday, December 17, 2013, 6:57 AM (0657) UTC



End Time:Tuesday, December 17, 2013, 7:59 AM (0759) UTC


Description:   Backbone CHIC-STAR and Connector ICCN/BlueWaters was unavailable while
Internet2 Engineers performed maintenance.   Maintenance has been
completed. Traffic re-routed during
this maintenance.



  







Ticket No.:1115:135



Subject:Maintenance 2 of 2 Completed - I2 AL2S Backbone HOUH-JACK



Affected:Backbone HOUH-JACK



Start Time:Friday, December 20, 2013, 6:12 AM (0612) UTC



End Time:Friday, December 20, 2013, 8:36 AM (0836) UTC


Description:   Backbone HOUH-JACK was unavailable while Internet2 Engineers performed
maintenance to install a core node in Baton Rouge, LA. Maintenance has
been completed. Traffic re-routed during this maintenance. This was
alternate window for ticket 1098:135.



  










Downtime and Availability







Internet2 Advanced Layer 2 Service Core NodesDown TimeReporting Period Availability52 Week Availability


sdn-sw.albu.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.ashb.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99772%  100.00000%
sdn-sw.atla.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97495%  100.00000%
sdn-sw.bost.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99658%  100.00000%
sdn-sw.char.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.95560%  100.00000%
sdn-sw.clev.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%  100.00000%
sdn-sw.denv.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.84574%  99.86604%
sdn-sw.elpa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98027%  100.00000%
sdn-sw.houh.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%  100.00000%
sdn-sw.jack.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%  100.00000%
sdn-sw.jcsn.net.internet2.edu 1 hr 18 min1 hr 18 min 99.22619%99.22619% 99.98520%  99.98520%
sdn-sw.kans.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97951%  100.00000%
sdn-sw.losa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97989%  100.00000%
sdn-sw.minn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98748%  100.00000%
sdn-sw.newy32aoa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98008%  100.00000%
sdn-sw.phil.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.phoe.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99791%  100.00000%
sdn-sw.pitt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99962%  100.00000%
sdn-sw.port.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.rale.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.69642%  99.73550%
sdn-sw.salt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%  100.00000%
sdn-sw.seat.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99374%  100.00000%
sdn-sw.star.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97894%  100.00000%
sdn-sw.sunn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%  100.00000%
sdn-sw.tuls.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97970%  100.00000%
sdn-sw.wash.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.97913%  100.00000%
      
Aggregate Internet2 Advanced Layer 2 Service Core Nodes1 hr 18 min1 hr 18 min99.97134%99.97134%99.96989%99.98469%

 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  
	    

	    
Internet2 Advanced Layer 2 Service Backbone CircuitsDown Time
Reporting Period Availability52 Week Availability


I2-ASHB-PITT-100GE-07737
               0 hr 2 min0 hr 2 min
               99.98016%99.98016%
               99.99317%99.99374%
I2-ATLA-CHAR-100GE-07738
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-CHIC-100GE-07740
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.61616%99.69054%
I2-ATLA-HOUH-10GE-07742
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.37462%99.37462%
I2-ATLA-JACK-100GE-07739
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99886%100.00000%
I2-ATLA-JCSN-100GE-10734
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-WASH-100GE-07738
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99526%100.00000%
I2-BATO-HOUH-100GE-08860
               0 hr 1 min0 hr 1 min
               99.99008%99.99008%
               99.99981%99.99981%
I2-BATO-JACK-100GE-11005
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-CHAR-RALE-100GE-10633
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-CHIC-KANS-100GE-07745
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97230%100.00000%
I2-CHIC-STAR-100GE-07743
               11 hr 54 min10 hr 52 min
               92.91667%93.53175%
               99.75600%99.77004%
I2-CLEV-NEWY32AOA-100GE-07735
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.57366%99.60819%
I2-CLEV-PITT-100GE-10334
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99336%100.00000%
I2-CLEV-STAR-100GE-07736
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.95257%99.99639%
I2-DENV-KANS-100GE-07746
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.86453%99.90930%
I2-DENV-SALT-100GE-07747
               0 hr 1 min0 hr 1 min
               99.99008%99.99008%
               99.96110%99.99981%
I2-ELPA-HOUH-100GE-07749
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.20556%99.31390%
I2-ELPA-LOSA-100GE-07750
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99981%100.00000%
I2-HOUH-JCSN-100GE-10783
               1 hr 18 min1 hr 18 min
               99.22619%99.22619%
               99.98520%99.98520%
I2-HOUH-TULS-100GE-07751
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96528%99.99355%
I2-KANS-TULS-100GE-07753
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.91462%99.91519%
I2-LOSA-SALT-100GE-07757
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.92638%99.93188%
I2-LOSA-SUNN-100GE-07755
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89716%99.89830%
I2-NEWY23AOA-PHIL-100GE-07759
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-PHIL-WASH-100GE-10867
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-PORT-SEAT-100GE-08997
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-PORT-SUNN-100GE-10343
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99924%99.99981%
I2-SALT-SUNN-100GE-07766
               0 hr 1 min0 hr 1 min
               99.99008%99.99008%
               99.95541%99.96850%
      
Aggregate All Internet2 Advanced Layer 2 Service Backbone Circuits
                   13 hr 17 min1 hr 2 min
                   99.72735%99.74856%
                   99.89311%99.90858%


 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  

Upcoming Scheduled Maintenances





Ticket No.:1127:135



Subject:Maintenance - I2 AL2S Backbone PHIL-WASH



Affected:Backbone PHIL-WASH



Start Time:Tuesday, January 7, 2014, 4:00 AM (0400) UTC



End Time:Tuesday, January 7, 2014, 8:00 AM (0800) UTC



Description:   Backbone PHIL-WASH will be unavailable while
Level(3) Communications performs maintenance between Mclean, VA
and Millersville, MD. Outages are expected throughout the maintenance
window. Backbone traffic will re-route during this maintenance.
 Workgroup administrators will receive notifications from OESS
regarding circuits re-routed on the backbone.



  







Ticket No.:1111:135



Subject:Maintenance 1 of 3 - I2 AL2S Backbone ATLA-JACK



Affected:Backbone ATLA-JACK



Start Time:Wednesday, January 8, 2014, 5:00 AM (0500) UTC



End Time:Wednesday, January 8, 2014, 11:00 AM (1100) UTC



Description:   Backbone ATLA-JACK will be unavailable while vendor Engineers perform
circuit maintenance in Jacksonville, FL.  Outages are expected, and
the entire window is reserved.  Backbone traffic will re-route during
this maintenance. This is the first of three maintenance windows.



  







Ticket No.:1112:135



Subject:Maintenance 2 of 3 - I2 AL2S Backbone ATLA-JACK



Affected:Backbone ATLA-JACK



Start Time:Thursday, January 9, 2014, 5:00 AM (0500) UTC



End Time:Thursday, January 9, 2014, 11:00 AM (1100) UTC



Description:   Backbone ATLA-JACK will be unavailable while vendor Engineers perform
circuit maintenance in Jacksonville, FL.  Outages are expected, and
the entire window is reserved.  Backbone traffic will re-route during
this maintenance. This is the second of three maintenance windows.



  







Ticket No.:1077:135



Subject:Maintenance - I2 AL2S Backbone HOUH-TULS



Affected:Backbone HOUH-TULS



Start Time:Friday, January 10, 2014, 6:01 AM (0601) UTC



End Time:Friday, January 10, 2014, 12:00 PM (1200) UTC



Description:   Backbone HOUH-TULS will be unavailable while Level(3)
Communications implement scheduled maintenance to splice backbone
cable as mandated by the Oklahoma Department of Transportation due
road widening project.  An outage is expected, and the entire window is
reserved. Backbone traffic will re-route during this maintenance.



  







Ticket No.:1113:135



Subject:Maintenance 3 of 3 - I2 AL2S Backbone ATLA-JACK



Affected:Backbone ATLA-JACK



Start Time:Saturday, January 11, 2014, 5:00 AM (0500) UTC



End Time:Saturday, January 11, 2014, 11:00 AM (1100) UTC



Description:   Backbone ATLA-JACK will be unavailable while vendor Engineers perform
circuit maintenance in Jacksonville, FL.  Outages are expected, and
the entire window is reserved.  Backbone traffic will re-route during
this maintenance. This is the third of three maintenance windows.