The Milwaukee Police Department uses the MediaSolv system for recording prisoner interviews (see SOP 750 here). The data stored on that system became inaccessible in early January 2015 due to a hardware malfunction and has been sent to a data recovery vendor for repair. The vendor reports the recovery is nearing completion. The Milwaukee Police Department is not aware of any cases that have been compromised by this malfunction. The District Attorney’s Office is aware of those cases where the data will not be available until the recovery is complete. Funds have been allocated for both the hardware repair work currently underway and the installation of a backup server to reduce the likelihood of future malfunctions. Earlier this year, the Milwaukee Police Department provided the following information to its members regarding the malfunction of the MediaSolv system.
The following information regarding the Milwaukee Police Department’s MediaSolv system was compiled by our Information Services Division and our Geographic and Investigative commands.
What is MediaSolv?
MediaSolv is the vendor for our interview recording system. We use the system to record the video and audio of our interviews. The storage system has redundant storage features and is designed to withstand up to two simultaneous disc failures.
How does it work?
When a member conducts an interview and determines that the contents may be evidentiary in nature, they are able to flag that recording for the purpose of retention. The member is also able to use the system to transfer the recording to a DVD for presentation to a prosecutor for consideration of charges, as any recorded interview to be used as evidence is transferred to a DVD. Members also document the substance of their interviews in written reports.
On January 2, the MediaSolv system suffered what is known as a simultaneous triple failure. This simultaneous triple failure resulted in our inability to retrieve recorded interviews from the system. The failure was caused by a hardware malfunction (i.e., three separate disc drives within the system failed at the same time). The system did not fail because of a lack of storage capacity. The system did not fail because of human error. The system did not fail because of human interference. We have been in contact with the system vendor who has informed us that they are unaware of any other simultaneous triple failures in their deployed hardware.
What was our response?
When we were made aware of the disc failure, we informed all members of the Department via memoranda and provided instructions regarding the interim procedures for recording and transferring interviews to DVDs during the repair of the system. The interim procedures require a member interested in obtaining a DVD of an interview to inform Information Technology staff who will then ask the system vendor to manually extract the requested interview.
We also notified the District Attorney’s Office to inform them of the malfunction and our interim procedures.
At the same time, we contacted the system vendor and began working with them to recover the data held on the malfunctioned discs. When the vendor was unable to recover the data themselves, we identified and contracted with a third-party vendor to handle the data recovery effort. The system vendor and the third-party vendor are working closely to complete the data recovery. As of this morning, we have been informed that the third-party vendor has completed approximately 80% of the programming work necessary to begin recovering data from the discs.
What’s the impact of the malfunction?
Presentation of evidentiary interviews.
Our interim procedures still allow members to conduct interviews and have DVD recordings of those interviews created for presentation to prosecutors. While the interim procedures may, in some cases, result in a slight delay in the production of a DVD copy of an interview, we prioritize those cases that require a DVD for a charging decision to ensure that violent and dangerous criminals are held in jail during the preparation of their cases.
Prosecution of criminal cases.
The great majority of criminal cases involving evidentiary interviews stored on the failed discs already had copies of those interviews recorded on DVDs prior to the January 2 failure. Based on documented requests for interview recordings from the District Attorney’s Office, we have identified seven cases in which evidentiary interviews are stored on the failed discs and the interviews have not yet been retrieved. In each of these cases, the suspects have been charged based on our investigators’ reports and other information and evidence that met the standards for criminal charges. One of the cases is a homicide investigation in which multiple interviews of a suspect were conducted but not all of the interviews were retrieved prior to the system failure. Despite this, the homicide suspect has been charged and remains in custody.
Any cases involving evidentiary interviews conducted after the disc failure were conducted under the interim procedures and are therefore available for transfer to DVD and presentation to prosecutors.
The failed discs are under warranty from the vendor and will be replaced at no cost to the City. The data recovery vendor is contracted to provide services for $49,500. To date, we have paid the vendor $25,000 from our existing Information Technology budget, with the balance due upon completion of the data recovery work.
In our ongoing efforts to improve the security and functionality of our Information Systems technology, we successfully requested funding to create an automated backup and redundant online storage solution for emergency recovery of recorded interviews. The anticipated cost for this currently-underway project is $50,000, and we are negotiating with video storage vendors to consolidate our agency-wide video storage needs – to include the MediaSolv data – in our efforts to both keep up with the evolution of video storage technology and exercise fiscal prudence.