Vision and Cognitive Systems

C Introduction

Imagesvenerdì 6 marzo 2020 19:52

Definizioni di AI

1983 def: AI is the study of how to make computers do things at which, at the moment, people are better.

2018 def: AI is the theory and development of computer systems able to perform tasks normally requiring human intelligence, such as visual perception, speech recognition, decision making, and translation between languages.

Some definitions of artificial intelligence

• Analyze the environment: perception, sensing, vision, NLP, speech analysis, IoT processing…
• Display intelligent behavior: reasoning, learning, understanding, planning, having emotions…
• Take actions: moving and stopping, grasping, suggesting and recommending…

With a certain degree of autonomy, human can take control of the system for a specific goal. AI is an intelligence different from human one, but controlled by humans.

Vision

Use intelligence to understand the world from visual stimuli.

• Visual intelligence: a cognitive capability to think and reason through mental images.
• Computer vision is the science that tries to reconstruct the 3D world and understand the world from images and videos.

From AI to computer vision

• Artificial Intelligence: the scientific field that studies how to create computers and computer software that are capable of intelligent behaviors, using sensing, perception, knowledge, reasoning, and learning.
• Machine Learning: the scientific discipline that studies how to construct algorithms that can learn from and make predictions on data, for getting computers to act without being explicitly programmed.
• Deep Learning: a branch of Machine Learning for modeling and implementing deep neural network architectures and algorithms.
• Pattern Recognition: the scientific discipline that studies how to classify or recognize patterns and observed data using a-priori knowledge, statistical information and learning.
• Computer Vision: the scientific discipline that studies how computers can perceive and understand the world through visual data (and provide a visual intelligence).
• Cognitive Computing: describes technology platforms that are based on scientific disciplines of Artificial Intelligence and Signal Processing. These platforms include machine learning, reasoning, natural language processing, speech and vision, human-computer interaction, dialog and narrative generation and more.

From computer vision to machine vision

• Image Processing: the scientific discipline that studies how to automatically modify images to create other images with changed or enhanced properties (ex. filtering, denoising, compressing).
• Image (Video) Analysis: the scientific discipline that studies how to automatically process images and videos to extract visual information (ex. measuring object shapes, counting people in a mall).
• Imaging: the science that studies how to modify, manipulate, process images manually or automatically, for further tasks (ex. photo correction, medical imaging).
• Machine Vision: the engineering field that studies how to build computer vision-based systems, services, and solutions, typically for industrial environments.

ABC of images and processing operators

Different image definitions

• The image is a discrete representation of a 2D continuous function.
• The image is a pixel matrix.
• The image is a 2D tensor used as input for many algorithms.
• For image processing, the image is a bidimensional signal, sampled and quantized.

Photometry: studies the measures of the brightness.

Colorimetry: studies the wavelength and color emission.

Histograms are good to represent images, but we lose the spatial information of the images. It is a plot of the number of pixels for each tonal value. The left side of the horizontal axis represents the dark areas, the middle represents mid-tone values and the right-hand side represents light areas. The vertical axis represents the size of the area (total number of pixels) that is captured in each one of these zones.

Normalized Histogram is the histogram divided by pixels and represents the probability distribution. Cumulative Histograms let us also define the cumulative distribution function corresponding to (prob dist or normalized hist). The cumulative hist is the probability of having a pixel value less than a certain threshold.

Image Entropy specifies the uncertainty in the image values. It measures the averaged amount of information required to encode the image values. A high value indicates a non-equal distribution of the color.

Saturated arithmetic is used in imaging when the result of the image processing should be another image, it is not correct from a mathematical point of view, but is still visible to humans.

Image processing operators

• Point operators: the value of each pixel of the resulting image depends only on the original pixel in the same image spatial position.
• Local (Neighborhood) operators: the value of each pixel depends only on the original pixel in the same image spatial position and in a local neighborhood.
• Global operators: the value of each pixel depends on all the pixels of the original image.

Point operators offer the possibility to parallelize the operations.

Linear point operator

If a linear transformation transforms an image into another image through a processing operation, we can write it using a scale factor called gain or contrast, and an offset constant called bias or brightness.

Contrast stretching is the expansion of the gray (or color) level of pixels in a dynamic range, given the histogram.

C Filtering

sabato 14 marzo 2020 20:15

Point processing gets intensity of a single pixel as input and it is unaware of spatial information. Neighborhood processing instead, takes spatial information into account.

Noise

Noise is everything that is not a signal or everything that is not useful information. Noise is also something relevant we cannot see in the image. Images are normally affected by noise. Image noise is a random (not present in the object image) variation of brightness or color information. It can be produced by sensors, cameras, transmissions, artifacts, etc…

We can define 3 types of noise:

• Signal noise: the most important, it is an additive noise (something added to the image).
• Computational noise: it is an error, acting as a noise, produced by computational tasks with approximations or computational limitations (ex. saturated arithmetics).
• Perceptual noise: called distractors, it is everything in the image which is not the target of the image itself. It is something that distracts us from the image and decreases the perception and understanding of the rest.

In order to provide compute vision tasks, noise signals must be removed. We would like to eliminate all types of noise. The noise can be generally:

• Salt and pepper noise: random occurrences of black and white pixels.
• Impulse noise: random occurrences of white pixels.
• Gaussian noise: variations in intensity drawn from a Gaussian normal distribution.

The signal noise in an image is supposed to be white. White noise is a random signal (or process) with a flat power spectral density. If a time series is normally distributed with zero mean and standard deviation, the series is called Gaussian white noise.

Image Restoration

The purpose of image restoration is to improve the image quality by eliminating noise and artifacts.

Types of noise for image restoration

• Without other information, noise is supposed to be additive and Gaussian, that is a stochastic value added to the pixel value, completely uncorrelated with the signal and the noise added to other pixels. It is always present, especially from low-cost cameras.
• Impulsive noise (salt and pepper) is a random modification of pixel values.
• Multiplicative noise, in which we have pixel values multiplied according to the type of sensor.
• Speckle noise, in which we have both additive and multiplicative noise.

Noise reduction and filtering

In surveillance, images are usually affected by noise because we use low-cost cameras with low quality. To reduce the noise, we can do a smoothing process. The simplest is the average of the values, where we replace each pixel with an average of all values in its neighborhood, in order to smooth the high frequencies. We can find a transformation that starts with a signal and goes to the signal, which is the average around the neighborhood.

Doing the average (mean) filter, that is a linear filter, makes an image blurred because our brain also does a Gaussian filtering, so we don't see the edges.

Linear filtering

Given an image, linear filtering consists in a process which gives in output a new image, where each location is a weighted sum of the original pixel values neighborhood, using the same set of weights each time.

The result is:

• Shift-invariant: the output value depends only on the pattern in the image neighborhood and not on the position in the image.
• Linear: the output obtained by the sum of two images is the same obtained by summing the two outputs separately (superposition of effects).

The pattern of weights used for a linear filter is usually called kernel of the filter. The process that applies the kernel or filter is usually called correlation or convolution.

Filtering

Given a signal, a filter has a variability between inputs and outputs and it is applied as a dot product or (cross-)correlation. Correlation and convolution are identical in computer vision, since kernel coefficients are symmetric.

Difference between continuous and discrete signals

As we said before, linear filters have some properties:

• Linearity: maintains linearity of the transformation.
• Scalarity: the output is scaled appropriately.
• Shift-invariance: maintains invariance to shifts in input.

Padding

If we apply a filter to the border, we obtain wrong results because we don't have information in the border. To resolve this border problem, we use padding:

• Zero padding: insert 0 pixels.
• Constant padding: insert a specific color in the border.
• Clamp to edges: repeat the edge value.
• Wrap: loop around in a toroidal configuration.
• Mirror: reflect the edges.

In computer vision, the best is zero padding because we only need information in the central part and we lose information at each processing step. When we do convolutions, the information that has a dimension of half of the kernel is lost forever, because it is wrong.

Smoothing

Mean filter: is obtained by moving an average filter (smoothing or blurring). A low pass blurring is given by averaging the pixel with the neighbor ones, it corresponds in convolving the image with a kernel of 1 values and then scaling (ex. kernel filled by a certain constant). We can apply as many filters as we want. Cons: it limits also the information with the same spatial frequency (blurring) and does not work on salt-pepper noise.

In this image, we want to maintain the indicator.

Gaussian filter: is the best filter to smooth Gaussian noise. It is an isotropic mask given by a Gaussian function with zero average and a given standard deviation, convolved with the image. The filter must be discretized choosing the filter size and the standard deviation.

Vision and cognitive systems

The filter must be discretized choosing a specific parameter, that is the filter size, and the standard deviation. In a filter, we choose a value of about a certain range. For a barcode, we need to apply a Gaussian with a kernel that is smaller with respect to the information we have.

Linear filters can be combined and applied in cascade. Example: sharpening filter.

Average filter

Great ideas of CNNs include using only the receptive field and not all the inputs, using the same set of convolutive kernel weights, using many kernels in parallel to create a feature map or feature vector, processing feature maps through non-linear activation functions and compressing by pooling to work in multiresolution. Feature vectors are used for final inference, using fully connected layers to take decisions.

Non-linear filters

Min Max filter: the minimum and maximum value in the moving region of the original image is the result of the min and max filter respectively (ex. max pooling).

Median filter: non-linear filter, useful to clean and mitigate the effect of salt-pepper or impulsive noise. The output pixel is the median value of the neighborhood. It is also possible to compute a weighted median filter. Application of the median filter.

If we apply a Gaussian filter to the first image, we spread the noise.

Variable-valued filters

Bilateral filter: combine a weighted filter kernel with outlier rejection. Every sample is replaced by a weighted average of its neighbors. These weights reflect how close and how similar are the neighbor and the center sample (larger weight to close and similar samples).

Bilateral weighted filter: in a neighborhood of the image, the result is a normalized weighted sum: Range kernel and Domain kernel. Bilateral filter is not shift-invariant.

Denoising

Block-matching and 3D filtering: exploits non-local statistics of the image (different patches in the same image are often similar in appearance).

Deblurring

Blur: degradation of sharpness and contrast of the image, causing loss of high frequencies. It is due to camera motion or camera defocus.

Blur is a generative process. This function is only invertible if the noise is zero. If there is noise, the recovered image has bad quality.

C Edges

lunedì 16 marzo 2020 17:47

Typical process pipeline

• Low level vision: filtering edge detection & selection segmentation.
• Image analysis: labeling, visual features extraction.
• Camera calibration.
• High level vision: clustering, unsupervised classification.

Features extraction

Features extraction: is the task that performs the extraction of a -dimensional vector representing some visual properties. The features have to summarize the image content, it is a quantization or a compression problem. It is necessary to obtain a more compact representation and also for data generalization to avoid overfitting.

Visual features criteria to take into account in order to select the features:

• Discriminant property: features must assume values that are significantly different for objects belonging to different classes.
• Reliability property: features must assume values that are similar for objects belonging to the same class.
• Independent property: features must be independent of each other (select the best features to avoid overfitting, not linear combinations).
• Minimum cardinality property: features must be as few as possible.

Gestalt also defines other properties for computer science:

• Invariant properties: scaling, translation, geometric contraction, rotation…
• Subjective relevance property: independence from luminance changes.

To find contours in an image we can use local variations, for example, we can look at the areas where luminance changes.

Edges

Contours and borders are global properties of a region, instead edges are local properties. Edge is a local property of a pixel and its neighborhood to have rapid intensity variation. Edge is a vector with magnitude and direction. It depends on the luminance variation and we can compute it as a gradient. The edge has the direction perpendicular to the gradient direction.

The edges can be created by different sources of luminance variation, mainly 4:

• Edges can be defined as points or set of points where there is a high gradient.

To detect borders we need to

• Use an edge detection operator (edge detector).
• Select strong edges with some given criteria.
• Link the edges (labeling).

The problem is that noise creates many false edge points.

Edge detection algorithms

• Methods based on first derivative computation (Sobel).
• First derivative and regularization techniques using filtering and optimal masks (Canny).
• Border following local techniques based on neighborhood operation of labeled edges and then segmentation (Border tracing).
• Classification (Deep learning).

Discrete detectors

Images have discrete values, so we use discrete derivatives. We have 3 types:

• Forward difference.
• Backward difference.
• Central difference.

Sobel edge detector operator

Sobel operator uses a first central derivative, smoothed in the opposite direction. We use the Sobel filter and compute the magnitude of the gradient. If the gradient is higher than the threshold, it is an edge.

Canny edge detector

A good operator should have 3 criteria:

• Good detection: we need to maximize the signal-to-noise ratio (SNR).
• Good localization: true edge detection (precise and position of the edge).
• One response to single edge: one answer for one edge, low false positive edges (false edges due to noise or artifacts).

Since finite difference filters respond too strongly to noise, we have to apply a Gaussian filter and the derivative of the Gaussian (DOG). The DOG is equal to the Gaussian of the derivative.

Canny proposed the best continuous filter in 1D and then discretized and extended it to 2D:

• Add a smoothing filter to keep high the signal-to-noise ratio.
• Find the true directions of the gradients in order to extract only one edge.
• Suppress the false edges.

Canny algorithm:

• Smooth image with a 2D Gaussian filter (convolution of the Gaussian with the image).
• Find the local edge normal direction for each pixel.
• Compute the edge magnitude of Grad, to understand the strength of the gradient for each position.
• Non-maximum suppression, that is to take just only the maximum of the gradient.