At TUDelft there is a project running on the design of digital assistants based on automatic assessment of the behavior of people during human machine interaction. The system will observe and analyze human-behavior such as facial expressions, gestures, way of speaking, etc. The goal is to assess the emotional state of the car driver and to analyze if his behavior fulfills the expected goals and requirements. One of the application areas is the analysis of the car driver behavior. We aim at analyzing human behavior patterns, speed and duration, such as posture, movements of the upper body, pointing, waiving, raising of hands, etc. We use special software to extract features from image frames. Next we developed a knowledge based system to reason about the behavior. The model and the employed software tools will be described in the paper including the undertaken experiments.