Video Analytics: At the Edge

By Dr Rustom Kanga, CEO, iOmniscient Corporation.

iOmniscient is the winner of the Global Security Challenge for Crowded Places awarded by the UK Home Office and BAE. The company is widely regarded as the technology leader in Video Analytics. Dr Kanga, its co-founder and CEO is also one of the authors of "Automated Surveillance" the first book on the use on the practical use of video analysis in industry which sold out its first edition within a month of being published. This assessment on when to use Edge devices is a summary of Chapter 10 in that book.

iOmniscient itself provides its solution in Edge Systems, in Centralized systems and in hybrid systems and is therefore unbiased in its views on the subject.

There is plenty of hype in the market about video analysis "on the edge" (which is essentially analysis software that is ported to a chip which is then embedded into a device close to the camera). Several of the major players have ported their software into a DSP chip which is mounted onto an embedded platform such as an encoder or camera. Is this the ideal way to provide "intelligence" in a system? Is there is benefit over leaving the intelligence on a server in some more central location. When is one approach superior to the other?

Let us discuss the factors that need to be considered to assist the user in deciding whether to use a server based solution or an "edge device".

What is an Implant?

Video Analysis software needs a computer to run on. The computer can be a standard computer of the type you have on your desk. At the core of this computer is a chip and the software runs on this chip. The rest of the computer is there to support the computing that is done on this chip. The rest of the computer manages the data, organises the networking and prioritises multiple tasks, but the main computing is done on the chip within the computer.

It is possible to have a similar chip embedded within other devices such as cameras or encoders. When this is done the video analysis software still runs on this chip. The chip is located on a computer motherboard which is programmed to perform some of the other functions performed by the computer (such as the networking) and it is this board that is actually embedded in the camera or other device.

See image: Intelligent Super Edge Device capable of running 4 channels of software with an Intelligence level of IQ Infinity - courtesy AnalyticsReady Ltd

Some functions like data management may continue to be performed at or near the chip.

More often than not that function is separated out and will be done remotely on a different server.

First Choose the Intelligence

Deciding on the architecture and platform has to be the second question. The first must always be whether the system chosen can perform the functions required.

Without such an analysis it would be equivalent to worrying about whether a plane has the right coloured upholstery without first ensuring that it can fly.

See chart: IQ Rating Chart showing level of Intelligence required for different capabilities

Choosing a system with the right level of intelligence is important. Intelligence can be rated using the IQ rating system  shown here. If the application requires an IQ 120 level system but the Edge device can only perform IQ 100 level functions there is a clear misfit between what the device can achieve and what is required of the system to perform the detection reliably.

Because of the limitations of the DSP chips used in edge devices (in terms of the amount of computing power and memory that is usually available) many vendors often restrict the functionality that they port over to the device. Hence they might have a highly functional Counting system on their server while offering a much more restricted version on their edge device.

Determining whether your environment is mission critical or whether it is acceptable to have less robust intelligence is extremely important and may make all the subsequent considerations irrelevant. Hence to reiterate, it is critically important to understand the level of intelligence that you require first as this may dictate the platform that is selected.

Understand the needs of the application

The major advantage of edge devices is that the intelligence is placed close to the sensor (which is usually a camera in the case of video based surveillance). This means that the Edge device does not have to transmit information continuously to a central location, thus saving on network bandwidth. It also means that the analysis can continue even when the network connection is down. Note of course that if a number of alarms occur while the network is unavailable some of them could be lost as the edge device might have very limited local storage capability.

Technically it is feasible to put very large storage capacity on each edge device but this greatly increases their cost relative to a server based system where large storage capacity can be shared across many cameras.

The major advantage of Server based systems (in addition to the relatively lower cost of storage just mentioned) is that much of the original development for most suppliers of video analysis products is done on server based platforms that have the computing power to deliver reliable detection. This means that for most suppliers, the Server based solution is usually several versions ahead of the edge solution and it is always more reliable and robust as it will have the advantage of the extra computing power and would have been through more prolonged testing.

It is also much easier to customise Server based applications and this could be a determining factor if customisation is important to the user.

One organisation, a railway in Europe, needed to put a very large number of cameras on poles along the railway line. They wished to know if there was an incident on the track. Most of the time on most of the cameras nothing happens and it does not make sense to transmit large amounts of unnecessary data to a central control room. However if an event occurs it is important for the central authority to receive the alarms immediately. This is a perfect application for an edge device. The intelligence is located at the camera at each pole but nothing is transmitted to the Centre unless if there is an event. If all the video from all the cameras had to be sent to a server to be processed the network load would be unmanageable.

See Image: Intelligent Camera with iOmniscient Intelligence Implant - courtesy AnalyticsReady Ltd

Consider another application for the same organisation. They wanted to protect their railway stations from abandoned bags or parcels, they want to count the number of people on their platforms to prevent overcrowding and they want to manage their public liability by using a Slip and Fall detection system. In theory they could use either an Edge or Server based device. However they had an additional requirement. They wanted the system to check on whether the ticket vending machines were working (by keeping an eye on the little light on top of the machine to make sure it was green and not red). This required a small amount of customisation and this made a Server based system most appropriate.

There was a second reason for choosing a server based solution in this case. A Slip and Fall detection system can be useful in that it permits the organisation to see such events in real time and take immediate action to help a person who has fallen down. Studies have shown that such action significantly reduces the number of liability claims. However these systems cannot be expected to capture a hundred percent of such incidents. The person may have been half hidden by a table when he fell down and a system may miss that event. In such cases it is useful to be able to return to the recording (based on date and time) if such an incident is reported, to analyse it. Since Edge devices are not suitable for continuous recording, a server based solution would be needed. In situations where one cannot be assured of 100% detection and network capacity is not a constraining issue, a server based application would be the preferred option.

Consider the Recording Policy

Every organisation has its own recording policy. In some airports for instance legislation dictates that they need to keep video footage for a certain number of days. In other situations (e.g. the rail organisation described earlier which put thousands of cameras on poles along the track) they only require event based recording.

Edge devices are ideally suited to event-based recording. The analysis can be done close to the camera. If all the video has to be transmitted and stored centrally then it would make more sense to take advantage of the more comprehensive storage offerings usually available on a server platform.

Is Real Estate Important?

Servers are usually placed in racks and these have a footprint that consumes expensive real estate in control rooms. Edge devices would usually have a higher footprint for the same number of cameras. However since they can be mounted on walls and ceilings near the camera or even inside the camera, the real estate costs is often not easily noticed or quantified. If there is very restricted space in the rack in the control room, edge devices can be the solution. However this is never the only consideration in using these devices (the previous factors will usually carry much more weight in the decision).

Does Distribution mean Edge?

As a general rule, the computing industry has moved towards distributed rather than centralised computing. Edge products are invariably distributed. This does not mean that Server based systems are centralised. Servers do not have to be located in racks in central control rooms. In modern facilities they can be distributed close to the cameras as well and many servers today may come in sizes that are very similar to the implanted devices. However since a single server can handle multiple cameras, even a distributed server solution is likely to be somewhat more centralised than an edge based solution in the sense that many cameras feed into a single server.

 This distinction is quickly vanishing as servers are available with very small form factors and a single chip in an edge based encoder can now handle multiple cameras.

Will using edge devices make the network more reliable?

A network can fail at many points. A server may stop working, individual cameras may fail or the network itself may stop working. There are different methods available for ameliorating such situations, usually by building in some level of redundancy into each element that has a risk of failure.

If an individual camera fails, both edge and server based solutions would be impacted. In mission critical situations it would make sense to have two cameras mounted in totally separate locations to observe the scene from totally different directions. This is rarely done in practice as the cost of duplication is very high. Nonetheless as cameras become less expensive this becomes an option to consider.

Network failures can be managed by building multiple paths in the network to ensure sufficient redundancy. Hence if one connection in the network is broken, the network should be able to self-adapt and continue to operate effectively.

The value of edge devices is more evident if the server crashes for whatever reason. In this case a large number of cameras may be disabled. If edge devices are used and one stops working only that single camera is affected. In this situation, where it is preferable to lose one camera rather than eight or twelve at a time, the edge device has an advantage.

If on the other hand the user wants absolute reliability and cannot afford to lose a single camera at any time, then alternative strategies need to be adopted. For instance, clustered servers which provide hot back up can be used (this technology has been available for over twenty years). Of course this solution comes with a higher cost in computing power and real estate (in terms of more rack space).

Will there be a price advantage?

The prices of both server and edge based systems continue to fall. At any given time one may be a slightly less expensive option than the other. Today server based systems still have a cost advantage over edge devices but this may change as volumes increase for the latter.

Since vendors will often "sell" the particular architecture that they have, it is important for users to get a balanced and unbiased assessment of the alternative architectures before making a decision to go down one path. Since both architectures can be useful in different scenarios, the ideal vendor will offer both capabilities without bias.  

Edge Devices becoming more powerful

 In recent years Edge devices have become increasingly powerful. The introduction of Solid State Disks for storage and smaller and more affordable hard disk technology has meant that the Edge device can actually store a reasonable amount of information.

Different types of chips are now being used for Edge devices. Whereas previously most Edge devices were powered by DSP chips from companies like Texas Instruments, Intel chips are becoming more prevalent now.

Essentially the device is a powerful mini-PC which has been stripped of unnecessary functionality so that it can operate optimally in a real time environment.

The cooling system for such a machine used to be an issue. While being much more powerful, it required more power and emanated more heat which had to be dissipated. This required a fan. Of course, as soon as one puts a mechanical device such as a fan in an Edge device it becomes the most likely point of failure.

AnalyticsReady Ltd overcame this limitation with the introduction of Heat Pipe Cooling technology for Intelligent Edge devices. This new technology eliminated the need for fans making it possible to put these powerful little machines in very unfriendly environments.

The use of these new generation edge devices has changed the financial equation with respect to their use. Suddenly the price of Edge devices has virtually halved making distributed processing much more viable than it was even a year ago with the traditional DSP based devices.

Questions to ask in determining whether to use Edge devices or Server based systems

1. What is the particular capability that the user requires and is this capability available on an Edge device. (If NO then consider a server based system. If YES then either architecture can be used).

2. What is the recording strategy - is it necessary to do continuous recording (If YES then consider a server based system).

3. Is any customisation going to be required? (if YES then consider a server based system)

4. Is real estate very restricted in the rack in the control room? (If YES then consider an EDGE based system).

5. What is the redundancy strategy for the network?