Report to the Joint Committee on Public Safety and Technology & Information Services Pursuant to Resolution No. 080694 -- Regarding the Failure of the City’s Public Safety Radio System on July 22, 2008
Overview
On July 22nd of this year, the City experienced an outage of the Police B system for approximately 40 minutes. The A side of the System, which is used by Fire and Municipal Departments, was not affected. However, the consequences of this event could have had serious implications with respect to providing public safety response for those in our City. Fortunately, there were no immediate adverse consequences as a result of the incident; however, any future occurrence of an event of this nature is totally unacceptable. This report will briefly describe what caused the incident, what corrective actions have been taken to date, and the future plans of this Administration to remedy this situation.
The July 22nd Incident
The outage of the Police B system on July 22nd was caused by a series of events, where the system initially functioned as designed but ultimately failed due to an operator error by Motorola during routine maintenance earlier in the day. Due to a high volume of radio traffic, the system went into a condition of fail-soft, as designed, and rolled over to a secondary controller to ensure continued functionality. However, when maintenance was performed earlier in the day on a component of the secondary controller, necessary settings were not properly entered into the component. As a result, when the secondary controller was required to function, it failed to do so and the Police B radio system failed. However, Bands J, T, and M that reside on both the Police B and Fire A system all continued to function throughout the incident. There was no adverse impact on the A side of the system, as all fire communications remained intact. When the failure was recognized, a technician was immediately dispatched to the Domino Lane tower site, and the component settings corrected, returning the system to operability. This took approximately 40 minutes.
During the 40 minutes when the primary and secondary controllers were in “fail-soft,” the Police dispatch center was still able to communicate with officers on the street using three back-up modes using various components of the installed console equipment. While these back-up systems were being used by the dispatchers, it became apparent that improved written instructions were required and additional training was needed on the use of these systems.
Corrective Actions Taken To Date
To ensure operational reliability of the Police radio system, changes were instituted, post-incident, to improve operational performance of both maintenance and dispatch personnel. These changes were aimed specifically at preventing the recurrence of a similar incident that occurred on July 22nd. These steps included revising existing maintenance and operational procedures that were aimed at improving Police dispatch and maintenance personnel performance during both normal and off-normal operational conditions. On-going training and testing using these improved procedures has demonstrated a marked improvement of our operational capability with respect to off-normal operation of the Police radio system. This testing continues each weekend with Police dispatch to insure that all shifts and dispatchers are completely familiar with all back–up components of the system. This has been supplemented with modified written procedures of what steps are required should the system again go into a fail-soft mode.
Additional training has been offered at no charge to all users of the system. Steps are being taken to include this training as officers and firefighters take required training on other aspects of their duties.
Immediate steps were taken to improve the performance of Motorola’s technical staff. New maintenance procedures were prepared for Motorola technicians, and job aids in the form of placards were created to help ensure proper settings are entered into system components when they are replaced. Motorola now requires that all maintenance done on any portion of the controllers be tested when the controller is in an active state. Also there have been replacement controllers in a cold standby mode, placed at both of the prime sites of the system -- City Hall and Domino Lane, at no charge to the city of Philadelphia. In addition to the above, the City has allowed Motorola technicians remote access to the controllers of the system, this had been removed previously due to security concerns which have been corrected.
Path Forward
At present, the City is evaluating a number of permanent solutions to the make the City’s 800 MHz emergency radio system as reliable as possible, considering the current state of the technology available. Both the cost and the time to implement improvements to the current system are critical factors when making decisions on the path forward. Our current maintenance agreement with Motorola expires in June 2010. Although extending this agreement is under evaluation, it does not appear to be the most desirable option since the current system is no longer manufactured, and replacement components may no longer be available.
Cost estimates to upgrade the currently installed system range as high as $40M. We have currently retained an independent consultant to assess Motorola’s proposal to provide the needed upgrades to the current system. The results of this assessment will determine what the best course of action is for the City. Replacing the entire system is another option, but the complexities associated with such a replacement could require a minimum of 5 years to accomplish, and incur costs that would be most likely double the current cost estimated to upgrade the system. It is clearly apparent that we are faced with a number of difficult decisions, particularly in light of the current fiscal constraints being experienced by the City. After a hard look at our options we will choose the most cost effective and technologically superior solution that meets the needs of the City without compromising public safety.
Reliable communications are the corner stone of the City’s ability to respond to emergency situations. All those who live and work in Philadelphia must have confidence that that when an incident involving public safety is reported anywhere in our City -- that there is a timely and effective response from Police, Fire, Medical or any other responder. To ensure this level of public confidence, we must provide our responders with an emergency communication system that has a high degree of reliability, and equally important, provide the procedures and training for operational personnel to effectively use the system we provide.
Our goal is to implement an emergency radio system with state-of-the-art reliability that creates a high level of confidence City-wide and enables us to provide the highest level of public safety.
