Generated Report for Internet2 IP

Internet2 IP Network Operations Report
09/17/2007 - 09/23/2007

Internet2 IP Network Operations Center
at Indiana University

New Customers

None

Upgrades

None

New Services

None

Unscheduled Outages Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Outage Type	*Start Time (UTC)*	*End Time (UTC)*

10968	1-Critical	1-Critical	Internet2 IP Network Core Node HOUS Available	Undetermined	09/19/2007 4:16 PM	09/19/2007 4:18 PM
10986	1-Critical	1-Critical	Internet2 IP Network Core Node CHIC Available	Undetermined	09/21/2007 6:31 PM	09/21/2007 6:33 PM
10990	2-High	2-High	Internet2 IP Network Backbone Circuit HOUS-LOSA Available	Undetermined	09/22/2007 7:26 AM	09/22/2007 8:19 AM

Unscheduled Outages Detail

Ticket No.:	10968:45
Subject:	Internet2 IP Network Core Node HOUS Available
Affected:	Core Node HOUS and its Connectors, Peers, and Participants Backbone Circuit HOUS - LOSA Backbone Circuit ATLA - HOUS
Start Time:	Wednesday, September 19, 2007, 4:16 PM (1616) UTC
End Time:	Wednesday, September 19, 2007, 4:18 PM (1618) UTC
Description:	Internet2 IP Network Core Node HOUS was briefly unavailable to the community. Internet2 Engineers have been contacted, however no reason for the outage is known.

Ticket No.:	10986:45
Subject:	Internet2 IP Network Core Node CHIC Available
Affected:	Core Node CHIC
Start Time:	Friday, September 21, 2007, 6:31 PM (1831) UTC
End Time:	Friday, September 21, 2007, 6:33 PM (1833) UTC
Description:	Internet2 IP Network Core Node Chicago was briefly unavailable to the community. Internet2 Engineers report that this was due to a routing engine failure in Chicago. The backup routing engine took over and connectivity was restored.

Ticket No.:	10990:45
Subject:	Internet2 IP Network Backbone Circuit HOUS-LOSA Available
Affected:	Backbone Circuit HOUS-LOSA
Start Time:	Saturday, September 22, 2007, 7:26 AM (0726) UTC
End Time:	Saturday, September 22, 2007, 8:19 AM (0819) UTC
Description:	Internet2 IP Network Backbone Circuit HOUS-LOSA is available to the Internet2 community. Level(3) Communications have been contacted and are investigating the RFO.

Scheduled Maintenances Summary

Ticket Number	Customer Impact	Network Impact	*Title*	Maintenance Type	*Start Time (UTC)*	*End Time (UTC)*

10942	3-Elevated	2-High	Internet2 IP Network Backbone Circuit ATLA-HOUS Maintenance Completed	Circuit	09/18/2007 2:00 AM	09/18/2007 2:03 AM
10973	3-Elevated	2-High	Internet2 IP Network Backbone Circuit ATLA-HOUS Emergency Maintenance Completed	Circuit	09/20/2007 2:02 AM	09/20/2007 2:06 AM
10946	3-Elevated	1-Critical	Internet2 IP Network Core Node NEWY Maintenance Completed	Software	09/21/2007 3:00 AM	09/21/2007 3:03 AM
10947	3-Elevated	1-Critical	Internet2 IP Network Core Node SEAT Maintenance Completed	Software	09/21/2007 3:10 AM	09/21/2007 3:12 AM
10949	3-Elevated	1-Critical	Internet2 IP Network Core Node LOSA (Qwest) Maintenance Completed	Software	09/21/2007 3:30 AM	09/21/2007 3:32 AM
10950	3-Elevated	1-Critical	Internet2 IP Network Core Node LOSA (Level3) Maintenance Completed	Software	09/21/2007 3:40 AM	09/21/2007 3:43 AM
10951	3-Elevated	1-Critical	Internet2 IP Network Core Node WASH Maintenance Completed	Software	09/21/2007 3:51 AM	09/21/2007 3:53 AM
10953	3-Elevated	1-Critical	Internet2 IP Network Core Node HSTN (Qwest) Maintenance Completed	Software	09/21/2007 4:10 AM	09/21/2007 4:13 AM
10954	3-Elevated	1-Critical	Internet2 IP Network Core Node KANS Maintenance Completed	Software	09/21/2007 4:20 AM	09/21/2007 4:22 AM
10955	3-Elevated	1-Critical	Internet2 IP Network Core Node SNVA Maintenance Completed	Software	09/21/2007 4:30 AM	09/21/2007 4:32 AM
10956	3-Elevated	1-Critical	Internet2 IP Network Core Node CHIC Maintenance Completed	Software	09/21/2007 4:40 AM	09/21/2007 4:42 AM
10957	3-Elevated	1-Critical	Internet2 IP Network Core Node SALT Maintenance Completed	Software	09/21/2007 4:50 AM	09/21/2007 4:52 AM

Scheduled Maintenances Detail

Ticket No.:	10942:45
Subject:	Internet2 IP Network Backbone Circuit ATLA-HOUS Maintenance Completed
Affected:	Backbone Circuit ATLA-HOUS I2-ALBU-ATLA-ESNET-00075 I2-ATLA-BATO-I2-00100 I2-BATO-JACK-I2-00047
Start Time:	Tuesday, September 18, 2007, 2:00 AM (0200) UTC
End Time:	Tuesday, September 18, 2007, 2:03 AM (0203) UTC
Description:	Internet2 IP Network Backbone Circuit ATLA-HOUS was briefly unavailable to the community while Level(3) Communications technicians restored a faulty amp in Jacksonville, FL. Maintenance has been completed.

Ticket No.:	10973:45
Subject:	Internet2 IP Network Backbone Circuit ATLA-HOUS Emergency Maintenance Completed
Affected:	Backbone Circuit ATLA-HOUS I2-ALBU-ATLA-ESNET-00075
Start Time:	Thursday, September 20, 2007, 2:02 AM (0202) UTC
End Time:	Thursday, September 20, 2007, 2:06 AM (0206) UTC
Description:	Internet2 Backbone Circuit ATLA-HOUS was unavailable to the community. Level(3) Communications technicians reported a possible faulty amp in Jacksonville, FL. The faulty amp was replaced.

Ticket No.:	10946:45
Subject:	Internet2 IP Network Core Node NEWY Maintenance Completed
Affected:	Core Node NEWY and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 3:00 AM (0300) UTC
End Time:	Friday, September 21, 2007, 3:03 AM (0303) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The outage was limited to five (5) minutes per node, while the route-engines change over, though the entire thirty (30) minute window per node was reserved. All BGP sessions with the core node reset. Backbone traffic was routed around the router being upgraded.

Ticket No.:	10947:45
Subject:	Internet2 IP Network Core Node SEAT Maintenance Completed
Affected:	Core Node SEAT and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 3:10 AM (0310) UTC
End Time:	Friday, September 21, 2007, 3:12 AM (0312) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The outage was limited to five (5) minutes per node, while the route-engines change over, though the entire thirty (30) minute window per node was reserved. All BGP sessions with the core node reset. Backbone traffic was routed around the router being upgraded.

Ticket No.:	10949:45
Subject:	Internet2 IP Network Core Node LOSA (Qwest) Maintenance Completed
Affected:	Core Node LOSA (Qwest) and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 3:30 AM (0330) UTC
End Time:	Friday, September 21, 2007, 3:32 AM (0332) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. All BGP sessions with the core node did reset. Backbone traffic routed around the router being upgraded. Maintenance has been completed.

Ticket No.:	10950:45
Subject:	Internet2 IP Network Core Node LOSA (Level3) Maintenance Completed
Affected:	Core Node LOSA (Level3) and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 3:40 AM (0340) UTC
End Time:	Friday, September 21, 2007, 3:43 AM (0343) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. All BGP sessions with the core node did reset. Backbone traffic routed around the router being upgraded. Maintenance has been completed.

Ticket No.:	10951:45
Subject:	Internet2 IP Network Core Node WASH Maintenance Completed
Affected:	Core Node WASH and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 3:51 AM (0351) UTC
End Time:	Friday, September 21, 2007, 3:53 AM (0353) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. All BGP sessions with the core node did reset. Backbone traffic routed around the router being upgraded. Maintenance has been completed.

Ticket No.:	10953:45
Subject:	Internet2 IP Network Core Node HSTN (Qwest) Maintenance Completed
Affected:	Core Node HSTN (Qwest) and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 4:10 AM (0410) UTC
End Time:	Friday, September 21, 2007, 4:13 AM (0413) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The maintenance has been completed and the connection restored.

Ticket No.:	10954:45
Subject:	Internet2 IP Network Core Node KANS Maintenance Completed
Affected:	Core Node KANS and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 4:20 AM (0420) UTC
End Time:	Friday, September 21, 2007, 4:22 AM (0422) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The maintenance has been completed and the connection restored.

Ticket No.:	10955:45
Subject:	Internet2 IP Network Core Node SNVA Maintenance Completed
Affected:	Core Node SNVA and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 4:30 AM (0430) UTC
End Time:	Friday, September 21, 2007, 4:32 AM (0432) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The outage was limited to five (5) minutes per node, while the route-engines change over, though the entire thirty (30) minute window per node was reserved. All BGP sessions with the core node reset. Backbone traffic was routed around the router being upgraded.

Ticket No.:	10956:45
Subject:	Internet2 IP Network Core Node CHIC Maintenance Completed
Affected:	Core Node CHIC and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 4:40 AM (0440) UTC
End Time:	Friday, September 21, 2007, 4:42 AM (0442) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The outage was limited to five (5) minutes per node, while the route-engines change over, though the entire thirty (30) minute window per node was reserved. All BGP sessions with the core node reset. Backbone traffic was routed around the router being upgraded.

Ticket No.:	10957:45
Subject:	Internet2 IP Network Core Node SALT Maintenance Completed
Affected:	Core Node SALT and Its Connectors, Participants, and Peers
Start Time:	Friday, September 21, 2007, 4:50 AM (0450) UTC
End Time:	Friday, September 21, 2007, 4:52 AM (0452) UTC
Description:	Internet2 Engineers performed software updates on the Internet2 core nodes. The outage was limited to five (5) minutes per node, while the route-engines change over, though the entire thirty (30) minute window per node was reserved. All BGP sessions with the core node reset. Backbone traffic was routed around the router being upgraded.

Downtime and Availability

Internet2 IP Core Nodes	Down Time	Reporting Period Availability	52 Week Availability
** CHIC T640	0 hr 0 min	100.00000%	100.00000%
DNVR T640	0 hr 0 min	100.00000%	99.99962%
HSTN T640	0 hr 3 min	99.97024%	99.99409%
LOSA T640	0 hr 5 min	99.95040%	98.95490%
** NEWY T640	0 hr 3 min	99.97024%	99.99733%
SNVA T640	0 hr 2 min	99.98016%	99.99657%
** SEAT T640	0 hr 2 min	99.98016%	99.99886%
** WASH T640	0 hr 0 min	100.00000%	100.00000%
** HOUS T640	0 hr 2 min	99.98016%	99.99962%
** LOSA T640	0 hr 0 min	100.00000%	100.00000%
ATLA M5	0 hr 0 min	100.00000%	99.97844%
** ATLA T640	0 hr 0 min	100.00000%	99.99676%
** SALT T640	0 hr 2 min	99.98016%	99.99924%
** KANS T640	0 hr 2 min	99.98016%	99.99905%

Aggregate Internet2 IP Core Nodes	0 hr 21 min	99.98512%	99.92246%

** Less than 52 weeks of data available

Internet2 IP Backbone Circuits	Down Time	Reporting Period Availability	38 Week Availability
DNVR-SNVA OC-192	0 hr 0 min	100.00000%	100.00000%
DENV-STTL OC-192	0 hr 0 min	100.00000%	99.97024%
HSTN-LOSA OC-192	0 hr 0 min	100.00000%	99.97964%
LOSA-SNVA OC-192	0 hr 0 min	100.00000%	100.00000%
ATLA-HSTN OC-192	0 hr 0 min	100.00000%	99.61336%
SNVA-STTL OC-192	0 hr 0 min	100.00000%	99.77783%
** I2-CHIC-ATLA	0 hr 0 min	100.00000%	99.08077%
** I2-CHIC-NEWY	0 hr 0 min	100.00000%	99.48256%
** I2-CHIC-KANS	0 hr 0 min	100.00000%	100.00000%
** I2-CHIC-WASH	0 hr 0 min	100.00000%	100.00000%
** I2-NEWY-WASH	0 hr 0 min	100.00000%	99.97729%
** I2-WASH-ATLA	0 hr 0 min	100.00000%	99.98225%
** I2-ATLA-HOUS	0 hr 7 min	99.93056%	99.91385%
** I2-KANS-ATLA	0 hr 0 min	100.00000%	99.77574%
** I2-KANS-SALT	0 hr 0 min	100.00000%	100.00000%
** I2-SALT-SEAT	0 hr 0 min	100.00000%	100.00000%
** I2-HOUS-LOSA	0 hr 53 min	99.47421%	99.95249%
** I2-LOSA-SALT	0 hr 0 min	100.00000%	100.00000%
** I2-LOSA-SEAT	0 hr 0 min	100.00000%	99.99948%

Aggregate All Internet2 IP Backbone Circuits	1 hr 0 min	99.96867%	99.86871%

Upcoming Scheduled Maintenances

Ticket No.:	10980:45
Subject:	Internet2 IP Network Backbone Circuits HOUS-LOSA Maintenance
Affected:	Backbone Circuit HOUS-LOSA
Start Time:	Thursday, October 4, 2007, 5:00 AM (0500) UTC
End Time:	Thursday, October 4, 2007, 7:00 AM (0700) UTC
Description:	Backbone circuit HOUS-LOSA will be briefly unavailable to the Internet 2 IP Network community. Level(3) Communications will be performing fiber maintenance in Austin TX. No impact is expected. However, the entire maintenance window will be observed.