The benefits of HD IP cameras are well known, with such solutions now rapidly replacing older analogue and early IP camera systems. Whether the images are being stored for longer, or the higher resolution of the cameras is generating more data, it’s clear that more storage is needed. What’s the best solution, then, for practising end users? Alastair McLeod offers his take on the matter.
According to IHS Markit, the amount of data produced in just one day by all of the new video surveillance cameras installed worldwide in 2015 was 566 Petabytes. Thankfully, the recent launch of 12 TB and 14 TB hard drives are strong indicators that the industry is responding to customer needs.
Given the increasing demand for greater forensic detail and analysis, higher image quality, 24/7 recording and longer retention times, even a medium-sized system can end up requiring a good lot of disk capacity. As storage systems for video surveillance can often represent 40% to 50% of the overall system cost, managing that storage solution is critical. Unless a purpose-designed system is used, the Total Cost of Ownership (TCO) can be much greater than originally envisaged with a shorter lifetime than promised. End users must consider the lifetime TCO – and what that lifetime is, or should be, as initial expectations are frequently not met.
Gaining an appreciation of the very significant differences between general IT data and surveillance video data will save the end user money. Systems designed for IT database management are not designed for the nature of video data: there are substantial implications for optimal system design, lifetime and TCO. If ignored, such differences can constitute a very expensive mistake over both the short and the longer term.
It’s a common misconception that digital video data are just like any other data. In fact, there are important and crucial differences between video data and generalised IT data. These differences will be amplified all the more in large-scale surveillance systems.
Typically, small to medium-sized business’ IT storage and retrieval systems are database-driven. The data set is of limited size (compared with video), being added to incrementally at a very modest pace. The systems are indexed and optimised for very high speed, I/O variable-rate concurrent searches across potentially multiple databases (during office hours, at least).
Bandwidth use is also an important parameter. With general IT data, there are huge peaks and troughs in bandwidth usage due to fluctuating demands. Brief delays in data delivery due to network bandwidth peaks are, in most cases, easily tolerated by caching. However, data sets of accounting systems and medical records, etc are often mission-critical, requiring that every single bit of information is present and correct. Such data are usually recorded on enterprise-level storage arrays, backed-up multiple times across several – often physically separate – systems and usually stored indefinitely.
Expected equipment lifetimes are shorter with IT systems. IT budgets are often larger and have higher priority than security in many organisations, so changing systems every five years, perhaps with disk changes every three years, is not an uncommon occurrence.
Video surveillance data have entirely different characteristics. Consisting of multiple streams of sequential data with no ‘slow’ or ‘quiet’ time, the timing and order of the information is critically important to avoid video ‘judder’ and frame drop-outs. Networks used for video must have sufficient bandwidth overhead to avoid severe timing delays as and when the overall network traffic increases. The data rate from IP cameras generally increases when the viewing scenes are ‘busy’ or when viewing poorly-lit night scenes which generates more image noise.
Surveillance video data are almost unidirectional into storage, with 100% of the data being recorded and only a small fraction – from 1% to 3% – ever being retrieved for review and playback. With very long retention times (eg periods of 120 days are typical in Middle Eastern installations), this can drop to a fraction of a percent.
Video surveillance recordings are effectively evidence and must be treated as such. File retention times are generally fixed for a particular project or installation, and often defined by national legislation or otherwise depend upon the type of application. Video must be kept for this defined period after which they must be deleted or made inaccessible, most commonly by overwriting with new data.
There are a few other practical and technical considerations which should be considered by designers of video surveillance storage systems. Systems or storage media which cache data can make IT systems much more responsive, but are not particularly useful for video and, in fact, can sometimes create problems. The sequential nature of video data can be exploited to improve the efficiency of purpose-built surveillance storage systems.
Also, in some circumstances, it can be extremely beneficial to be able to physically locate specific video recordings and rapidly extract them for transporting bulk data or handing bulk data over to law enforcement authorities. Removal of disks in IT systems, where the location of specific data is highly abstracted and often scattered over a number of storage arrays, is generally not possible. Removable disks which retain data as stand-alone media volumes are, therefore, extremely advantageous.
Video surveillance storage design
Storage application type is critical and general purpose IT storage arrays are less than ideal for video surveillance storage. Storage arrays which exploit the sequential nature of video allow the design of a system which significantly reduces the three key causes of hard disk failure: temperature, vibration and wear.
Recording sequentially within a disk (sequential write, random read) removes vibration as the read/write head of the drive moves slowly across the drive platter, rather than constantly ‘jumping and seeking’ as it does in traditional IT systems. This is only possible because video is generated sequentially and can be written sequentially. It also reduces the operating temperature of the drive as it’s having to work less hard. Further, it rather obviously reduces wear and tear on the read/write head movement mechanisms.
Recording to disks sequentially across an array allows all non-writing disks to be switched off, saving further on wear, but also further reducing power consumption, reducing disk exposure to power spikes and safely preserving stored video data. Given the extended file retention time of, for example, 120 days or more, there’s little point in having all 120 days’ worth of disks constantly powered up and running (as happens with traditional IT storage approaches), and especially so when powered-down disks can be made available within seconds as soon as a playback command is received for data on that disk.
With low-power control electronics, an innovative mirrored overlapping-pair writing pattern and careful system design, the end result is a video surveillance storage array which uses less than half a Watt of power per Terabyte of storage. It can use the highest capacity (lowest cost per TB) ‘archive’-style SMR disks and allows the physical location and removal of specific video recording time spans.
The most important benefit of this approach is that hard disks employed in the system exhibit very high reliability indeed – even surpassing the idealised MTBF figures quoted in disk manufacturers’ technical data sheets. Useful life expectancy for even ordinary desktop drives exceeds seven years, as evidenced by long-term installations which have been in place from 2009/2010.
With a power consumption which is lower by a factor of ten, sequential storage system design requires minimal cooling and far less expensive air conditioning equipment, plus a much lower expenditure on Uninterruptible Power Supply back-up systems. These factors, together with the extended useful lifetime of the system including the hard disks, dramatically reduce the overall TCO of storing and managing surveillance images.
Alastair McLeod is Group CEO of Veracity