Internet2 IP Network Operations Report
12/10/2007 - 12/16/2007

Internet2 IP Network Operations Center
at Indiana University

New Customers

None


Upgrades

None

New Services

None



Unscheduled Outages Summary







Ticket Number
Customer Impact
Network Impact
Title
Outage Type
Start Time (UTC)

End Time (UTC)





114422-High
2-High

Internet2 IP Network Backbone Circuit LOSA-SEAT Available
Undetermined
12/13/2007 5:12 PM
12/13/2007 5:13 PM


114542-High
2-High

Internet2 IP Network Backbone Circuit ATLA-HOUS Available
Circuit - Other
12/14/2007 12:07 AM
12/14/2007 3:00 PM






Unscheduled Outages Detail






Ticket No.:11442:45



Subject:Internet2 IP Network Backbone Circuit LOSA-SEAT Available



Affected:Backbone Circuit LOSA-SEAT



Start Time:Thursday, December 13, 2007, 5:12 PM (1712) UTC



End Time:Thursday, December 13, 2007, 5:13 PM (1713) UTC


Description:Internet2 IP Network Backbone Circuit LOSA-SEAT
experienced a brief outage. Level(3) technicians have
reported the outage was due to bumped fibers.







Ticket No.:11454:45



Subject:Internet2 IP Network Backbone Circuit ATLA-HOUS Available



Affected:Backbone Circuit ATLA-HOUS
Connectors LONI and USF
Peer ESnet



Start Time:Friday, December 14, 2007, 12:07 AM (0007) UTC



End Time:Friday, December 14, 2007, 3:00 PM (1500) UTC


Description:Internet2 IP Network Backbone Circuit ATLA-HOUS is
unavailable.  Level(3) Communications report that they
have isolated the problem to be with hardware at the
equipment site in Tarrytown, GA.  The faulty amplifier
card has been replaced and the backbone circuit has
been re-enabled.







Scheduled Maintenances Summary





Ticket Number
Customer Impact
Network Impact
Title

Maintenance Type
Start Time (UTC)
End Time (UTC)





11423
3-Elevated
2-High

Internet2 IP Network Backbone Circuit ATLA-WASH Emergency Maintenance Completed
Circuit
12/12/2007 7:35 AM
12/12/2007 7:58 AM


11433
3-Elevated
1-Critical

Internet2 IP Network Core Node NEWY Emergency Maintenance Completed
Software
12/13/2007 4:00 AM
12/13/2007 4:02 AM


11434
3-Elevated
1-Critical

Internet2 IP Network Core Node SEAT Emergency Maintenance Completed
Software
12/13/2007 4:10 AM
12/13/2007 4:12 AM


11435
3-Elevated
1-Critical

Internet2 IP Network Core Node ATLA Emergency Maintenance Completed
Software
12/13/2007 4:20 AM
12/13/2007 4:22 AM


11436
3-Elevated
1-Critical

Internet2 IP Network Core Node HOUS Emergency Maintenance Completed
Software
12/13/2007 4:30 AM
12/13/2007 4:32 AM


11437
3-Elevated
1-Critical

Internet2 IP Network Core Node WASH Emergency Maintenance Completed
Software
12/13/2007 4:40 AM
12/13/2007 4:42 AM


11438
3-Elevated
1-Critical

Internet2 IP Network Core Node SALT Emergency Maintenance Completed
Software
12/13/2007 4:50 AM
12/13/2007 4:52 AM


11439
3-Elevated
1-Critical

Internet2 IP Network Core Node CHIC Emergency Maintenance Completed
Software
12/13/2007 5:00 AM
12/13/2007 5:02 AM


11440
3-Elevated
1-Critical

Internet2 IP Network Core Node LOSA Emergency Maintenance Completed
Software
12/13/2007 5:10 AM
12/13/2007 5:12 AM


11441
3-Elevated
1-Critical

Internet2 IP Network Core Node KANS Emergency Maintenance Completed
Software
12/13/2007 5:20 AM
12/13/2007 5:22 AM




Scheduled Maintenances Detail





Ticket No.:11423:45



Subject:Internet2 IP Network Backbone Circuit ATLA-WASH Emergency Maintenance Completed



Affected:IP Backbone Circuit ATLA-WASH
Ciena Backbone Circuit ATLA-WASH
ESNet IP Backbone ATLA-WASH
ESNet SDN Backbone ATLA-WASH
Peers MCNC, redCLARA, and RNP



Start Time:Wednesday, December 12, 2007, 7:35 AM (0735) UTC



End Time:Wednesday, December 12, 2007, 7:58 AM (0758) UTC


Description:   Internet2 IP Network Backbone Circuit ATLA-WASH was
briefly unavailable as Circuit Provider Level(3)
Communications performed emergency circuit
maintenance. Maintenance has been completed.



  







Ticket No.:11433:45



Subject:Internet2 IP Network Core Node NEWY Emergency Maintenance Completed



Affected:Core Node NEWY and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 4:00 AM (0400) UTC



End Time:Thursday, December 13, 2007, 4:02 AM (0402) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has  been completed.



  







Ticket No.:11434:45



Subject:Internet2 IP Network Core Node SEAT Emergency Maintenance Completed



Affected:Core Node SEAT and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 4:10 AM (0410) UTC



End Time:Thursday, December 13, 2007, 4:12 AM (0412) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has  been completed.



  







Ticket No.:11435:45



Subject:Internet2 IP Network Core Node ATLA Emergency Maintenance Completed



Affected:Core Node ATLA and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 4:20 AM (0420) UTC



End Time:Thursday, December 13, 2007, 4:22 AM (0422) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has been completed.



  







Ticket No.:11436:45



Subject:Internet2 IP Network Core Node HOUS Emergency Maintenance Completed



Affected:Core Node HOUS and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 4:30 AM (0430) UTC



End Time:Thursday, December 13, 2007, 4:32 AM (0432) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has been completed.



  







Ticket No.:11437:45



Subject:Internet2 IP Network Core Node WASH Emergency Maintenance Completed



Affected:Core Node WASH and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 4:40 AM (0440) UTC



End Time:Thursday, December 13, 2007, 4:42 AM (0442) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has been completed.



  







Ticket No.:11438:45



Subject:Internet2 IP Network Core Node SALT Emergency Maintenance Completed



Affected:Core Node SALT and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 4:50 AM (0450) UTC



End Time:Thursday, December 13, 2007, 4:52 AM (0452) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has been completed.



  







Ticket No.:11439:45



Subject:Internet2 IP Network Core Node CHIC Emergency Maintenance Completed



Affected:Core Node CHIC and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 5:00 AM (0500) UTC



End Time:Thursday, December 13, 2007, 5:02 AM (0502) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has  been completed.



  







Ticket No.:11440:45



Subject:Internet2 IP Network Core Node LOSA Emergency Maintenance Completed



Affected:Core Node LOSA and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 5:10 AM (0510) UTC



End Time:Thursday, December 13, 2007, 5:12 AM (0512) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has been completed.



  







Ticket No.:11441:45



Subject:Internet2 IP Network Core Node KANS Emergency Maintenance Completed



Affected:Core Node KANS and Its Connectors, Participants, and Peers



Start Time:Thursday, December 13, 2007, 5:20 AM (0520) UTC



End Time:Thursday, December 13, 2007, 5:22 AM (0522) UTC


Description:   Internet2 engineers performed a maintenance
upgrade of the Internet2 IP backbone. Each router was
expected to experience a 5 minute outage as the new
software took ownership and BGP sessions
re-established.  Traffic was re-routed around the
routers as they were upgraded, but the IP network
experienced an aggregate route instability during the
upgrade window.  Maintenance has been completed.



  










Downtime and Availability







Internet2 IP Core NodesDown TimeReporting Period Availability52 Week Availability


** CHIC T6400 hr 0 min100.00000%100.00000%
** NEWY T6400 hr 2 min99.98016%99.99618%
** SEAT T6400 hr 2 min99.98016%99.99847%
** WASH T6400 hr 0 min100.00000%100.00000%
** HOUS T6400 hr 2 min99.98016%99.99886%
** LOSA T6400 hr 2 min99.98016%99.99924%
** ATLA T6400 hr 2 min99.98016%99.99580%
** SALT T6400 hr 2 min99.98016%99.99828%
** KANS T6400 hr 2 min99.98016%99.99809%
      
Aggregate Internet2 IP Core Nodes0 hr 14 min99.98457%99.99832%

** Less than 52 weeks of data available

Internet2 IP Backbone CircuitsDown TimeReporting Period Availability50 Week Availability




** I2-CHIC-ATLA0 hr 0 min100.00000%99.28690%
** I2-CHIC-NEWY0 hr 0 min100.00000%99.47857%
** I2-CHIC-KANS0 hr 0 min100.00000%99.99544%
** I2-CHIC-WASH0 hr 0 min100.00000%99.87996%
** I2-NEWY-WASH0 hr 0 min100.00000%99.98115%
** I2-WASH-ATLA0 hr 23 min99.77183%99.97976%
** I2-ATLA-HOUS14 hr 53 min91.14087%99.39286%
** I2-KANS-ATLA0 hr 0 min100.00000%99.82956%
** I2-KANS-SALT0 hr 0 min100.00000%100.00000%
** I2-SALT-SEAT0 hr 0 min100.00000%100.00000%
** I2-HOUS-LOSA0 hr 0 min100.00000%99.96369%
** I2-LOSA-SALT0 hr 0 min100.00000%99.91964%
** I2-LOSA-SEAT0 hr 1 min99.99008%99.99841%
** I2-KANS-HOUS0 hr 0 min100.00000%99.97837%
      
Aggregate All Internet2 IP Backbone Circuits15 hr 17 min99.35020%99.83459%






Upcoming Scheduled Maintenances





Ticket No.:11404:45



Subject:Internet2 IP Network Backbone Circuit CHIC-WASH Maintenance Rescheduled



Affected:Backbone Circuit CHIC-WASH
ESNet SDN Backbone Circuit CHIC-WASH
I2 Grooming Backbone CLEV-PITT
Connector OARNet



Start Time:Thursday, December 20, 2007, 5:00 AM (0500) UTC



End Time:Thursday, December 20, 2007, 11:00 AM (1100) UTC



Description:   ***Please Note Rescheduled Maintenance Date
Internet2 IP Network Backbone Circuit CHIC-WASH will
undergo circuit maintenance as Level(3) Communications
performs a hot cut at Beaver Falls, PA.   An outage of
2 hours is expected, however the entire window is reserved.