Internet2 Advanced Layer 2 Service Network Operations Report
04/28/2014 - 05/04/2014

Internet2 Advanced Layer 2 Service Network Operations Center
at Indiana University

New Connections

None


Upgrades

None

New Services

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Start Time (UTC)

End Time (UTC)





16222-High
2-High

Instability Resolved - I2 AL2S Backbone MINN-SEAT
Hardware
04/28/2014 2:00 AM
04/28/2014 1:25 PM


16422-High
2-High

Brief Outage - I2 AL2S Backbone BATO-JACK
Undetermined
04/30/2014 6:40 AM
04/30/2014 6:41 AM


16492-High
1-Critical

Brief Outage - I2 AL2S Core Node KANS 
Human Error
04/30/2014 8:42 PM
04/30/2014 8:45 PM


16552-High
2-High

Outage Resolved - I2 AL2S Backbone ATLA-CHIC and ATLA-JCSN
Unannounced Maintenance
05/01/2014 5:01 AM
05/01/2014 7:47 AM


16632-High
2-High

Outage Resolved - I2 AL2S Backbone BATO-JACK
Cut or Damaged Fiber
05/02/2014 4:23 PM
05/03/2014 1:34 AM






Unscheduled Outages Detail






Ticket No.:1622:135



Subject:Instability Resolved - I2 AL2S Backbone MINN-SEAT



Affected:Backbone MINN-SEAT



Start Time:Monday, April 28, 2014, 2:00 AM (0200) UTC



End Time:Monday, April 28, 2014, 1:25 PM (1325) UTC


Description:Backbone MINN-SEAT is stable. Zayo Engineers have reported that the
reason for the instability was due to faulty hardware in Elbow Lake,
MN. The hardware has been successfully replaced and no further outages
are expected.  Workgroup administrators receive
notifications from OESS regarding circuits re-routed on the backbone.







Ticket No.:1642:135



Subject:Brief Outage - I2 AL2S Backbone BATO-JACK



Affected:Backbone BATO-JACK



Start Time:Wednesday, April 30, 2014, 6:40 AM (0640) UTC



End Time:Wednesday, April 30, 2014, 6:41 AM (0641) UTC


Description:Backbone BATO-JACK was briefly unavailable. Level(3) Communications
Technicians are investigating the cause; however, the reason for the
outage is pending. Workgroup administrators receive notifications from
OESS
regarding circuits re-routed on the backbone.







Ticket No.:1649:135



Subject:Brief Outage - I2 AL2S Core Node KANS 



Affected:Core Node KANS



Start Time:Wednesday, April 30, 2014, 8:42 PM (2042) UTC



End Time:Wednesday, April 30, 2014, 8:45 PM (2045) UTC


Description:Internet2 AL2S Core Node KANS connectivity was disrupted.  Engineers
onsite in KANS mistakenly rebooted management switch causing a brief
outage to Core Node KANS.  Connectivity has been restored and no
further outages are expected.







Ticket No.:1655:135



Subject:Outage Resolved - I2 AL2S Backbone ATLA-CHIC and ATLA-JCSN



Affected:Backbone ATLA-CHIC and ATLA-JCSN



Start Time:Thursday, May 1, 2014, 5:01 AM (0501) UTC



End Time:Thursday, May 1, 2014, 7:47 AM (0747) UTC


Description:Backbone ATLA-CHIC and ATLA-JCSN are now available.  Fiber provider
Engineers report the outage was due to planned maintenance between
Manchester and Jasper, TN. The maintenance has been completed, and no
further outages are expected. Workgroup administrators receive
notifications from OESS regarding circuits re-routed on the backbone.







Ticket No.:1663:135



Subject:Outage Resolved - I2 AL2S Backbone BATO-JACK



Affected:Backbone BATO-JACK



Start Time:Friday, May 2, 2014, 4:23 PM (1623) UTC



End Time:Saturday, May 3, 2014, 1:34 AM (0134) UTC


Description:Backbone BATO-JACK is now available. Our vendor's field crew reported
that a fiber cut between Milton, FL and Defuniak Springs, FL was
caused by boring work in the area. Fiber crews have completed work to repair cable.
No further outages are anticipated.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Start Time (UTC)
End Time (UTC)





1629
3-Elevated
2-High

Emergency Maintenance 1of 2 Completed - I2 AL2S Backbone MINN-STAR
Circuit
04/30/2014 3:34 AM
04/30/2014 3:53 AM


1668
3-Elevated
2-High

Emergency Maintenance Completed - I2 AL2S Backbone BATO-HOUH
Hardware
05/03/2014 8:42 PM
05/03/2014 8:54 PM




Scheduled Maintenances Detail





Ticket No.:1629:135



Subject:Emergency Maintenance 1of 2 Completed - I2 AL2S Backbone MINN-STAR



Affected:Backbone MINN-STAR



Start Time:Wednesday, April 30, 2014, 3:34 AM (0334) UTC



End Time:Wednesday, April 30, 2014, 3:53 AM (0353) UTC


Description:   Backbone MINN-STAR was unavailable while Zayo Engineers performed
emergency maintenance to troubleshoot light levels in Chicago, IL.
Maintenance has been completed. No further outages are expected.
Backbone traffic re-routed during this maintenance. Workgroup
administrators will receive notifications from OESS regarding circuits
re-routed on the backbone. 
This is maintenance window 1 of 2.



  







Ticket No.:1668:135



Subject:Emergency Maintenance Completed - I2 AL2S Backbone BATO-HOUH



Affected:Backbone BATO-HOUH



Start Time:Saturday, May 3, 2014, 8:42 PM (2042) UTC



End Time:Saturday, May 3, 2014, 8:54 PM (2054) UTC


Description:   Backbone BATO-HOUH was unavailable while Internet2 and Level 3
Engineers performed hardware maintenance.
maintenance, which is completed.  Backbone traffic re-routed during this
maintenance. Workgroup administrators will receive notifications from OESS
regarding circuits re-routed on the backbone.



  










Downtime and Availability







Internet2 Advanced Layer 2 Service Core NodesDown TimeReporting Period Availability52 Week Availability


sdn-sw.albu.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.alba.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99696%  100.00000%
sdn-sw.ashb.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99677%  100.00000%
sdn-sw.atla.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%  100.00000%
sdn-sw.bost.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99564%  100.00000%
sdn-sw.char.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 100.00000%  100.00000%
sdn-sw.chic.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.96376%  99.98975%
sdn-sw.clev.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%  100.00000%
sdn-sw.dall.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.denv.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.85770%  99.85902%
sdn-sw.elpa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99943%  100.00000%
sdn-sw.houh.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.92752%  99.92885%
sdn-sw.jack.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99829%  100.00000%
sdn-sw.jcsn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98425%  99.98520%
sdn-sw.kans.net.internet2.edu 0 hr 3 min0 hr 3 min 99.97024%99.97024% 99.99810%  99.99943%
sdn-sw.losa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99886%  100.00000%
sdn-sw.minn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98596%  100.00000%
sdn-sw.newy32aoa.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99905%  100.00000%
sdn-sw.phil.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.phoe.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99715%  100.00000%
sdn-sw.pitt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99962%  100.00000%
sdn-sw.port.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99924%  100.00000%
sdn-sw.rale.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.98937%  99.99886%
sdn-sw.salt.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99829%  100.00000%
sdn-sw.seat.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99279%  100.00000%
sdn-sw.star.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99791%  100.00000%
sdn-sw.sunn.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%  100.00000%
sdn-sw.tuls.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99867%  100.00000%
sdn-sw.wash.net.internet2.edu 0 hr 0 min0 hr 0 min 100.00000%100.00000% 99.99810%  100.00000%
      
Aggregate Internet2 Advanced Layer 2 Service Core Nodes0 hr 3 min0 hr 3 min99.99897%99.99897%99.98855%99.99176%

 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  
	    

	    
Internet2 Advanced Layer 2 Service Backbone CircuitsDown Time
Reporting Period Availability52 Week Availability


I2-ALBA-BOST-100GE-09210
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99393%99.99924%
I2-ALBA-CLEV-100GE-07735
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99279%99.99981%
I2-ASHB-CHIC-100GE-11803
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98900%99.98975%
I2-ASHB-WASH-100GE-11823
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-CHAR-100GE-07738
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97970%99.97970%
I2-ATLA-CHIC-100GE-07740
               2 hr 46 min2 hr 46 min
               98.35317%98.35317%
               99.79394%99.86434%
I2-ATLA-HOUH-10GE-07742
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-JACK-100GE-07739
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99412%100.00000%
I2-ATLA-JCSN-100GE-10734
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-ATLA-JCSN-100GE-10784
               2 hr 46 min2 hr 46 min
               98.35317%98.35317%
               99.67953%99.72355%
I2-ATLA-WASH-100GE-07738
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               100.00000%100.00000%
I2-BATO-HOUH-100GE-08860
               0 hr 12 min0 hr 0 min
               99.88095%100.00000%
               99.92543%99.92847%
I2-BATO-JACK-100GE-11005
               9 hr 12 min9 hr 12 min
               94.52381%94.52381%
               99.88350%99.89488%
I2-BOST-NEWY32AOA-100GE-09211
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98634%100.00000%
I2-CHAR-RALE-100GE-10633
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97894%99.97970%
I2-CHIC-COLU4-100GE-11554
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98881%99.98956%
I2-CHIC-KANS-100GE-07745
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.96774%99.96888%
I2-CHIC-STAR-100GE-07743
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.84859%99.86358%
I2-CLEV-NEWY32AOA-100GE-07735
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98615%99.99924%
I2-CLEV-PITT-100GE-10334
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99336%100.00000%
I2-CLEV-STAR-100GE-07736
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.94991%99.98368%
I2-COLU4-KANS-100GE-11548
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.98254%99.98254%
I2-DALL-HOUH-100GE-07751
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.92619%99.92885%
I2-DALL-TULS-100GE-11300
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.84403%99.84479%
I2-DENV-KANS-100GE-07746
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.83075%99.88141%
I2-DENV-SALT-100GE-07747
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.92676%99.96983%
I2-ELPA-HOUH-100GE-07749
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.37272%99.48163%
I2-ELPA-PHOE-100GE-07750
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89716%99.91595%
I2-HOUH-JCSN-100GE-10783
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.91310%99.91405%
I2-HOUH-TULS-100GE-07751
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.68788%99.75068%
I2-KANS-TULS-100GE-07753
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99924%99.99981%
I2-LOSA-PHOE-100GE-09190
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.80779%99.80912%
I2-LOSA-SALT-100GE-07757
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.91481%99.93226%
I2-LOSA-SUNN-100GE-07755
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99829%99.99943%
I2-MINN-SEAT-100GE-09130
               11 hr 24 min11 hr 24 min
               93.21429%93.21429%
               99.37557%99.54463%
I2-MINN-STAR-100GE-09299
               0 hr 19 min0 hr 0 min
               99.81151%100.00000%
               99.88293%99.90475%
I2-NEWY23AOA-PHIL-100GE-07759
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99924%100.00000%
I2-PHIL-WASH-100GE-10867
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.89375%100.00000%
I2-PORT-SEAT-100GE-08997
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99829%100.00000%
I2-PORT-SUNN-100GE-10343
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.99829%99.99962%
I2-RALE-WASH-100GE-08888
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.79926%99.82829%
I2-SALT-SUNN-100GE-07766
               0 hr 0 min0 hr 0 min
               100.00000%100.00000%
               99.97533%99.99658%
      
Aggregate All Internet2 Advanced Layer 2 Service Backbone Circuits
                   26 hr 39 min0 hr 31 min
                   99.62231%99.62963%
                   99.90371%99.92497%


 NOTE: Numbers in italics represent downtime with scheduled  maintenance excluded. Internet2's definition of scheduled
  maintenance is any anticipated outage announced at least 72
  hours in advance of the outage start time.  

Upcoming Scheduled Maintenances





Ticket No.:1652:135



Subject:Maintenance 1 of 2 - I2 AL2S Backbone ALBA-CLEV



Affected:Backbone ALBA-CLEV



Start Time:Thursday, May 8, 2014, 4:00 AM (0400) UTC



End Time:Thursday, May 8, 2014, 10:00 AM (1000) UTC



Description:   Backbone ALBA-CLEV will be unavailable while Level(3) Communications
performs maintenance to repair a damaged cable in Buffalo, NY. An
outage of 45 minutes is expected; however the entire window is
reserved.  Backbone traffic will re-route during this maintenance.
Workgroup administrators will receive notifications from OESS
regarding circuits re-routed on the backbone.

*This is the first of 2 maintenance windows



  







Ticket No.:1654:135



Subject:Maintenance 2 of 2 - I2 AL2S Backbone ALBA-CLEV



Affected:Backbone ALBA-CLEV



Start Time:Friday, May 9, 2014, 4:00 AM (0400) UTC



End Time:Friday, May 9, 2014, 10:00 AM (1000) UTC



Description:   Backbone ALBA-CLEV will be unavailable while Level(3) Communications
performs maintenance to repair a damaged cable in Buffalo, NY. An
outage of 45 minutes is expected; however the entire window is
reserved.  Backbone traffic will re-route during this maintenance.
Workgroup administrators will receive notifications from OESS
regarding circuits re-routed on the backbone.

*This is the second of 2 maintenance windows



  







Ticket No.:1596:135



Subject:Maintenance - I2 AL2S Backbone MINN-SEAT



Affected:Backbone MINN-SEAT



Start Time:Saturday, May 10, 2014, 1:00 PM (1300) UTC



End Time:Saturday, May 10, 2014, 11:00 PM (2300) UTC



Description:   Backbone MINN-SEAT will be unavailable while our fiber provider's
third party provider performs a cable relocation in Judson ND. This work
must take place during daylight hours as the cable is located along the
railroad. An outage of up to 3 hours is expected; however the entire
window
has been reserved.