Risk Analysis of a Video-Surveillance System
Leon Rothkrantz, Iulia Lefter
Abstract: The paper describes a surveillance system of cameras installed at lamppost of a military area. The surveillance system has been designed to detect unwanted visitors or suspicious behaviors. The area is composed of streets, building blocks and surrounded by gates and water. The video recordings are monitored by operators at regular times. Next some operators inspect the area by car. To compute the probability of unobserved intrusion we used (Hidden) Markov Models. We also investigated the weak spots. We considered three intrusion scenarios of suspicious individuals, traveling by car, by boat and by feet. The military area has been split up in different areas/zones, partly covered by cameras or by the inspection team. The probability of detection in some forbidden areas has been extracted from the surveillance system and from the time tables of the inspection operators. The transition probabilities between areas are computed using data from analysis of video recordings and based on interviews with operators.
.
Key words: Surveillance systems, Hidden Markov Models, Risk Analysis.
INTRODUCTION
At this moment many video camera systems have been installed to detect unwanted behaviour of some people. Unwanted behaviour such as aggression or violence acts against people or material. Military areas should be protected against unwanted intruders aiming at stealing weapons from the arsenal/armoury or violence acts by bombing or whatever. At this moment most systems are only used for recording of data. In case of an incident the video recordings will be analysed after the incident. Real time monitoring of video data by human operators takes a lot of time and money. At this moment there are more than 2 million cameras installed at the city of London. Inspection by human operators is not an option. There is a need for automatic surveillance. At Delft University of Technology there is a project running on automatic surveillance by multimodal cameras. The aim is to use the camera recording in combination with sound recordings and automatically detect if the situation is normal or not and alarm the operator [8].
Camera surveillance systems monitored by human operators is currently used for surveillance of limited areas such as industrial areas, areas around buildings or military areas. In this paper we study a military area composed of streets, building, squares and grass fields as depicted in Figure 1. The area is surrounded by a fence or by water. To enter the area a visitor has to pass a guarded barrier, or use a valid access card. During daytime we can observe traffic from visitors moving to the lecture halls, dormitories etc. In exceptional cases we can observe people moving to the arsenal. Of special interest are the unusual rarely observed tracks, people climbing the gate, hitting the barrier or walking on the grass fields instead of using the regular roads. In this paper we want to analyse the tracks people take in the military area and detect unusual and possible suspicious behaviour.
The paper is organised as follows. In the next section we give an overview of surveillance systems, then we report our models and show how we can compute the probability of a path. We end this paper with an overview of results and conclusion and future work. The last section contains references to related work.
Figure 1 - Map of the military area at Den Helder
RELATED WORK
In Europe we can observe many camera systems installed along the highways and along streets in the city. These systems are used for speed control but also for navigation reasons. Companies as Siemens and Philips developed many road sensor systems. In the Netherlands, Vialis is a company specialized in developing sensor-systems to control and navigate traffic streams [1].
Object and anomaly detection are researched in [2]. In a surveillance scenario, they discriminate between normal and abnormal tracks. Their approach is based on modelling pixel level probability distribution functions of object speed and size from the tracks and was used for detecting local as well as global anomalies in object tracks. The research presented in [3] focuses on understanding human behaviour and interactions in complex dynamic scenes. They are able to find rules governing a scene like traffic sequences order, based on learning spatio-temporal dependencies in the scene. The first step is to learn dependencies between motion patterns and therefore extract local temporal rules of the scene. The second step is to jointly learn co-occurring activities and their time dependencies, generating global temporal rules.
A smart surveillance system named CASSANDRA aimed at detecting instances of aggressive human behaviour in public environments is presented in [4]. A surveillance system that uses audio and video sensors to reveal and track the presence of an intruder in an off-limit area is presented in [5].The system is composed of a mobile agent and several static agents cooperating in the tracking task. The mobile agent is a vision agent composed of an omnidirectional vision system and a mobile robot. The static agents are acoustic agents composed of self-steerable microphone arrays and a vision agent implemented on an omnidirectional vision system.
The multimodal workbench for automatic surveillance applications presented in [6] is cantered on the shared memory paradigm, the use of which allows for loosely coupled asynchronous communication between multiple processing components. This decoupling is realized both in time and space. The shared memory in the current design of the framework takes the form of XML data spaces. This suggests a more human-modeled alternative to store, retrieve and process data. The framework enhances the data handling by using a document cantered approach to tuple spaces. All the data is stored in XML documents and these are subsequently received by the data consumers following specific XML queries. In addition, the framework consists of a set of software tools to monitor the state of registered processing components, to log different types of events and to debug the flow of data given any running application context.
In [7] a comparison has been made between HMM’s and Bayesian networks to model strings of observations and to compute the probabilities. In this paper we used the software to implement our HHM. In [8] we have researched different methodologies and technologies to analyse the car behaviour in a military area.
MODEL
Hidden Markov Models
A first order Markov chain is a stochastic process, whose outcome is a sequence of T observations O1, O2,….OT such that each observation belongs to a finite set of states {S1,…..SN}. Any observation depends only upon the immediately preceding observation and not upon other previous observation. For every pair of states {Si,Sj}, aij denotes the probability that Si occurs immediately after Sj has occurred (Figure 2). Hidden Markov models inherit all properties of Markov models, but the states of the model are no longer associated with one observation, but with a probability distribution over all possible observations.
Figure 2 - HMM model with transition probabilities
The military area has been portioned in zones, such as roads, gate surroundings, grass fields, squares etc. The zones correspond to the states in the HMM. The transition probabilities between zones have to be computed. Cars follow the road network but at crossings the routing table will be computed based on surveillance data from the past. In every zone the operators can observe cars, pedestrians, cyclists or don’t observe any object. The probability of every observation has been computed based on the surveillance scheme. Once we have the transition probabilities and observation probabilities we are able to compute the probability of every sequence of observations.
Figure 3 – Schematic map of the area (< barrier, = gate)
Table 1 - Transition probabilities between states (zones) and observation probabilities of observed car (Oc) and no-observed-car (Onc) in every zone (R=road, F=fence, G=grass field)
R1 R2 R3 R4 R5 --- R15 F1 ---- F8 G1 G2 G3 Oc Onc
R1 0.3 0.1 0.0 0.6 0.0 0.0 0.0 0.01 0.01 0.01 0.83 0.07
R2 0.2 0.1 0.2 0.1 0.4 0.0 0.0 0.0 0.0 0.01 0 0.83 0.07
R3 0.0 0.2 0.5 0.1 0.0 0.2 0.0 0.0 0.0 0.01 0.01 0.83 0.07
R4 0.0 0.2 0.1 0.5 0.0 0.0 0.0 0.0 0.0 0.0 0.01 0.83 0.07
R5 0.4 0.1 0.0 0.0 0.1 0.0 0.0 0.0 0.01 0.0 0.0 0.83 0.07
-----
R15 0.0 0.0 0.2 0.0 0.0 0.5 0.0 0.0 0.01 0.01 0.01 0.83 0.07
F1 0.0 0.0 0.0 0.0 0.0 0.0 0.5 0.4 0.01 0.01 0.0 0.19 0.81
-----
F8 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0.3 0.01 0.01 0.0 0.19 0.81
G1 0.2 0.0 0.0 0.0 0.1 0.0 0.0 0.01 0.3 0.0 0.0 0.01 0.99
G2 0.2 0.2 0.2 0.0 0.0 0.0 0.01 0.0 0.0 0.3 0.0 0.01 0.99
G3 0.0 0.0 0.3 0.3 0.0 0.01 0.01 0.0 0.0 0.0 0.3 0.01 0.99
Transition probabilities
The military area is composed of buildings, streets, squares, grass fields etc. It has been partitioned in zones (see Figure 3). The whole area is surrounded by a fence. There are six possibilities to enter the area. Three are blocked by barriers and the other three by removable gates. To enter, a digital access-card/visitor-pass is needed or to contact the guard via the Intercom. The area has a street network. All the streets in the military area can be monitored via cameras installed at lampposts. The street segments between crossings are considered as Zones. The 4 segments in front of the gate or barrier are considered as hot spots. Visitors can go from one zone to another with different probabilities. Visitors by cars are composed of the following groups. Employees of the Military academy park their car in front or at the back of the building. Students park their cars close to the dormitories. We assume that they take the shortest path from the main entrance to the parking lot and by counting the parked cars we computed the probability that a car at a crossing is going straight ahead, turns left or turns right. Visitors by car have to inform the guard about their goal of their visit and they are also assumed to take the shortest route. Using this data we were able to compute the transition probabilities between states/zones of the frequently used routes. We notice that the probabilities are time dependent. The peak hours in the morning, lunchtime and end of the afternoon are different from probabilities during the evening. During the evening we expect no cars going to the lecture halls, but only students driving to the dormitories. In exceptional cases we observed other car routes. Cars are assumed to follow the roads and obey the traffic rules. It could happen that some cars go off road via the grass field and even damage a barrier or a gate. The probabilities of these exceptional cases were not observed in the recorded data but the probabilities were set to less than 5%.Take for example most cars entering the main entrance via R1 turn to the left (R5, 95%, only a minority to the right (3 %) and entering G1 or G2 is set to 1%.
The computation of the transition probabilities is based on observed traffic streams. But the density of the traffic streams has not been taken into account. For example most visitors enter via the main entrance and a minority via the back doors. For the silent routes we take equal distribution (at a T-crossing 50% to the left, 50% to the right). Less data is available about intruders such as how many times people climbing the fence, or the gate or take an off-the road route via the grass field area. We set all these transition probabilities to 1%. In Table 1 we provide some probabilities based on this data analysis.
Observation probabilities
Next we have to compute that an incoming car, cyclist, walker will be observed. At this moment there are 8 monitors available in the control room. Four of them display the videos recorded at the hotspot and 4 of them display the videos of the remaining cameras during 3 minutes, then the recordings of other zones are displayed during 3 minutes. The operators are supposed to monitor the videos all the time. We have 14 Road segments, excluding 4 hotspots implies 2x10=20 cameras. We have 8 gate segments controlled by 2x8=16 cameras. Cameras are monitored in groups of 4 so the detection probability is about 11%. From every hour 15 minutes are used for outdoor inspection. The guards take fixed route around the area and are able to observe areas not covered by cameras. We assume that they observe less than 1/3 of the area. The detection probability is 3/4x1/3=8% Based on this data we computed the detection probabilities, spitted up in observation in control room and outdoor inspection
- Hotspot (75% + 8%)
- Regular zone (11% + 8%)
- Off the road zone (11% + 8%).
SCENARIOS
One of the goals of our research is to compute Pd,r the probability of detection along route r. The probability that an intruder will not be detected is 1-Pd,r This can be computed as the product of the probability of non-detection along every composing segment of the route r. Let us assume we have a string of observations Oi,Oj,Ok,Ol, assuming that observation Om has been realised at State m. We can compute the probability that an intruder will not be detected (Pnd,r) along route r as follows:
Pnd,r(Oi,Oj,Ok,Ol) = P(Si) x P(Oi,nc) x aij x P(Oj,nc). (1)
Our assumption is that an intruder travels like an employee or student. Our transition probabilities are based on the traffic behaviour of regular visitors. The probability of a string of non-detection is computed correctly, but it is questionable if that probability corresponds with the probability of intrusion. One can argue that an intruder doesn’t prefer the main route. So detection of intruders should be focussed on less probable strings of observation. From the other side it can be argued that an intruder prefers as long as possible the main route because this is not suspicious. A sneaky route, with a low probability is suspicious by definition.
An interesting topic is to research if there are routes with a high probability of non-detection, ending at suspicious places i.e. arsenal. Consider the route R4-R2. This route has a high probability of non detection, so a low probability that it will be detected. The main reason is that the transition probabilities are high. A solution could be to bloc the option that travellers at the end of R4 can turn to the right. Another option is to place the road segment R2 on the list of hot spots which will be supervised all the time. In this case the probability of detection will be increased.
Given the destination of the arsenal room at R2 we can compute the probability of all paths without detection ending at R2. From the regular paths R1-R2 is the most probable track and F1-G2-R2 hitting the gate G1 and crossing the grass field F2 is the most probable from the illegal paths.
CONCLUSIONS AND FUTURE WORK
In this paper we modelled a video surveillance system of a military area using HMM. Usually a lot of data is needed to train HMMs. Surveillance of the area for a long time provides the necessary data to compute the transition probabilities. And because the work of the surveillance operators was strictly organised according to a fixed schedule we were able to compute the observation probabilities using this schema.
After implementing the HMM models using a simplified version of HTK toolkit we were able to compute intrusion probabilities for some scenarios.
Next future we will consider multiple observations and consider a surveillance system based on multimodal cameras, that is to say the cameras will be provided with a microphone. The observation will be increased with sound observations.
……………………………………………………………………………………………..……….…
REFERENCES
[1] http://www.vialis.com
[2] A. Basharat, A. Gritai, and M. Shah. Learning object motion patterns for anomaly detection and improved object detection. In IEEE Int. Conf. Of Computer Vision and Pattern Recognition, 2008.
[3] D. Kuettel, M. D. Breitenstein, L. Van Gool, and V.Ferrari. Whats going on? discovering spatio-temporal dependencies in dynamic scenes. In IEEE Conference on Computer Vision and Pattern Recognition, June 2010.
[4] W. Zajdel, J.D. Krijnders, T.C. Andringa, and D.M. Gavrila. Cassandra: audio-video sensor fusion for aggression detection. Proc. IEEE Conference on Advanced Video and Signal Based Surveillance AVSS, page 200-205, 2007.
[5] E. Menegatti, E. Mumolo, M. Nolich, and E. Pagello. A surveillance system based on audio and video sensory agents cooperating with a mobile robot. In Intelligent Autonomous Systems 8, pages 335-343, IOS Press, 2004.
[6] D. Datcu, Z. Yang, and L. Rothkrantz. Multimodal workbench for automatic surveillance applications. Multimodal Surveillance: Sensors, Algorithms, and Systems, 2007.
[7] M.C. Popa, L.J.M Rothkrantz, D. Datcu, P. Wiggers, R. Braspenning, C. Shan. 2010. A comparative study of HMMs and DBNs applied to Facial Action Units Recognition. Neural network World. 6:737–760.
[8] I. Lefter, L.J.M Rothkrantz, P. Bouchner, P. Wiggers. 2010. A multimodal car driver surveillance system in a military area. Driver Car Interaction & Interface 2010.
……………………………………………………………………………………………..……….…
ABOUT THE AUTHOR
Prof. Leon Rothkrantz, PhD, Department of Mediamatica, Delft University of Technology, Section Sensor, Weapons and Command and Control, Netherlands Defence Academy, Phone: +31152787504, Е-mail: L.J.M.Rothkrantz@tudelft.nl.
Ir. Iulia Lefter, Department of Mediamatica at Delft University of Technology, Section Sensor, Weapons and Command and Control at Netherlands Defence Academy, TNO Defence Security and Safety. E-mail: i.lefter@tudelft.nl.