Affective Computing

Research Team: Enzo Pasquale Scilingo (coordinator), Gaetano Valenza, Alberto Greco, Andrea Guidi, Mimma Nardelli

The BIOLAB group at Centro di Ricerca "E.Piaggio" is mainly involved in the following research activities:

• Advanced Biomedical Signal processing
• Machine learning and artificial intelligence algorithms for biomedical data.
• Development of novel  for monitoring of physiological parameters in humans and animals
• Eye-tracking technologies.
• Innovative smart textile-based sensing technologies.
• Applications including, but not limited to, the assessment of autonomic nervous system activity on cardiovascular control, brain-heart interactions, affective computing, assessment of mood and mental disorders, human-horse interaction.

 

Advanced Biomedical Signal Processing

All the signals acquired through wearable and non-wearable systems are analysed by using novel techniques of signal processing. Our research is mainly focused on:

• Autonomic nervous systems signals: ElectroCardioGram (ECG), ElectroDermal Activity (EDA), Respiration.
• ElectroEncephaloGrafic (EEG) signals

Concerning the analysis of cardiovascular signals, we have been studying novel methods in the field of nonlinear dynamic analysis, especially applied to the Heart Rate Variability (HRV) series. This signal is extracted from the RR series, a series given by the time duration of the intervals between each R peak and the successive in ECG.

Figure 1. Example of RR series processed using Kubios HRV software.

Many studies in the recent literature demonstrated that the use of nonlinear methods in HRV analysis provide a relevant additional prognostic information on the autonomic regulation during physiological and pathological conditions.

We applied several nonlinear techniques to HRV series, obtaining great results in the recognition of psychophysical diseases and in the frame of affective computing, in order to recognize emotional responses to sensorial elicitation (visual, acoustic and tactile).

Figure 3. Example of the trend of the area of Poincaré Plot as a function of normalzed HF power

 

As regards the EDA analysis we developed a novel method for the decomposition of the signal into its two main components (tonic and phasic), called cvxEDA.

The EDA signal process is modeled as a convolution process between the SudoMotor nerve activity (SMNA), as part of the sympathetic nervous system, and the Impulse Response Function (IRF) under the hypothesis that EDA is controlled by SMNA resulting in a sequence of distinct impulses which regulate the eccrine sweat glands dynamics. Formally, it is possible to write:
EDA = SMNAIRF

where SMNA = (DRIVERtonic + DRIVERphasic ). SMNA is unknown and it is evaluated by deconvolving the EDA signal with the IRF. In order to decompose the obtained SMNA signal into the DRIVERtonic and DRIVERphasic components, several algorithmic steps have been processed.

 

We use EEG signals to detect the differences in the central nervous system response to emotional stimuli.

Figure 5. Example of EEG signals

The Power Spectral Density (PSD) is estimated in each electrode and in every bandwidths.

 

Figure 6. Statistical comparison (p-value maps) between unpleasant and pleasant sub-sessions of each arousal session, performed on EEG-PSD estimates calculated in β band.

 

 

Wearable systems

An exemplary wearable system is the PSYCHE system, developed in the frame of PSYCHE European project. The goal of the project was to continuously monitor physiological and behavioral signs in bipolar patients, and promoting the communication between patients, their peers and health professionals. The garment was designed for females and males and was made up of elastic piezoresistive fibres that in addition to tight adhesion to the user’s body allow monitoring of fabric stretching (and consequently respiration activity) and metallic fibres knitted to create fabric electrodes to monitor the ECG. Several physiological signals as well as behavioural parameters were taken into account (Autonomic Nervous System-related vital signs, voice, activity index, sleep pattern alteration, electrodermal response, biochemical markers) to identify and to predict changes in pathological mental states. The PSYCHE platform was able to continuously acquire physiological data, and store them in a Micro SD card for up to 24 h, using a lithium battery.

This wearable system will be used in another European project, NEVERMIND, which just started this year, and will involve patients affected by depressive symptoms during severe primary somatic diseases. In particular, NEVERMIND will have enhanced sensing capabilities by adding a behavioural monitor, a module that keeps track of the information coming from the use of the mobile phone (number of phone calls, text messages, use of social networks, contact list, etc). Physiological signals and behavioural parameters (Autonomic Nervous System-related vital signs, voice, activity index, sleep pattern alteration, biochemical markers) will be continuously monitored for up to 36h. Previously developed pre-processing algorithm will be adapted to provide real-time information to the NEVERMIND real-time Decision Support System module.

The HATE-move is a new head mounted eye tracking system developed by our group. One of the main feature of the proposed system is the ability in detecting and monitoring the eye movement.

The HAT-Move eye tracking system was developed in two configurations: the “baseball” hat (see Figure 8) and the head band (see Figure 3). They are technically and functionally equivalent, although the former can be considered aesthetically more pleasant. It is comprised of a wireless camera that is light and small, with an Audio/Video (A/V) transmitter of up to 30m of distance. The camera has a resolution of 628 x 586 pixels with F2:0, D45° optic, and 25 frames per second.

In addition, the InfraRed filter, which is actually present in each camera, was removed and a wide-angle-lens was added allowing to enlarge the view angle and acquire natural infrared components increasing both the image resolution and the contrast between pupil and iris. This system is able to simultaneously record the visual scene in front of the subject and eye position. This is achieved through a mirror (5x0.6cm) placed in front of the user’s head (see Figure 9).

Figure 9. Version II: “Head band” configuration.

The pupil tracking algorithm is comprised of several blocks in which the acquired eye image is first binarized in order to separate the pupil from the background by using a threshold in the image histogram; then a geometrical method was used to reconstruct pupil contours and to remove outliers belonging to the background. Following the geometrical detection of the points belonging to the pupil, an ellipse fitting algorithm is implemented for pupil contour reconstruction and for detecting the center of the eye (see Figure 10).

We also developed a multi-frequency, sensorized glove able to acquire the ElectroDermal Activity (EDA).

In this glove, integrated textile electrodes were placed at the distal phalanges of the index and middle fingers (see Figure 11). Textile electrodes, provided by Smartex s.r.l.(Pisa, Italy), are made up of 80% polyester yarn knitted with 20% steel wire, with a dimension of 1 × 2 cm. In one of our previous studies, we performed a comparison of textile sensors with Ag/AgCl electrodes demonstrating comparable performance. 

In order to acquire ElectroEncephaloGrafic (EEG) signals, we use commercial wearable and portable systems.