appunti esame Prof Zorzi

Marr’s three levels of analysis

In every way we want to use computation, it is fundamental to take into account all the three levels of analysis proposed by Marr (1982). At the top, there is the computational level: we ask what the goal of computation is, the possible strategies to reach the goal. This is the most abstract level. Then there is the representation and algorithm level and the questions become what the appropriate algorithm is to reach the goal and how to transform the input in output. At the end, the implementation level, which deals with the physical implementation. We need to work at different levels. Kraukauer asked for a more pluralistic neuroscience, in which the focus won’t be only on neurons (I can say that describing a neuron, its structure and functioning (implementation level) is not sufficient to understand the behavior that a reductionist approach wants to be casually determined by the neurons. “Detailed examination of brain parts or their selective perturbation is not sufficient to understand how the brain generates behavior. It is very hard to infer the mapping between the behavior of a system and its lower-level properties by only looking at the lower-level properties. Relying solely on the collection of neural data, with behavior incorporated as an after-thought (and typically over-constrained), will not lead to meaningful answers.

Describe an artificial neuron

Neurons (nodes) receive signals and send them to other neurons through connections. Each connection has a weight that represents its strength (it is the same as biological synapses). Weights can be positive (excitatory) or negative (inhibitory). So, the input to a neuron from another connected neuron is computed by multiplying the incoming signal by the connection weight. But, B can receive input by more than one neuron, so we need to compute the total input. The net (total) input to a neuron is the sum across all inputs from other neurons. The final activation state is computed through an activation function. Since there’s a maximum firing rate, so that the neuron doesn’t fire more than that, the function is not linear, because there’s a limit. Indeed, the activation function is a sigmoid. The output of the neuron, then, is propagated to other downstream neurons. The artificial neuron model captures the essential of how a real neuron works. The inhibition or excitation depends only on connections, so the same artificial neuron can inhibit a neuron and excite another neuron at the same time. Nevertheless, this is not real. In fact, neurons in our brain are either excitatory or inhibitory.

Describe the neural networks

Neural networks are used in psychology and cognitive neuroscience to reproduce, through simulation on a computer, cognitive abilities and human behavior. The aim is not a reductionist one, i.e., to reduce psychological functions to neurophysiology, but rather of “reconstructing” these functions as emergent properties of neural systems. In an architecture of neurons, there are always an input layer, receiving inputs directly from the environment, and an output layer, producing the final output of the network, that is the category, the name (think you see an image and have to identify the object). In some networks, there are neurons that do not make direct contact with input or output and are called hidden neurons. Conversely, input and output units are visible neurons, sometimes named. Neurons can be connected to each other in different ways (connectivity scheme):

• Feed-forward network: There are only unidirectional links from input to hidden to output neurons (bottom-up); what happens at upper levels doesn’t influence lower levels.
• Recurrent network: There are bidirectional links, in which activation can spread backwards (top-down or feedback).
• Fully recurrent network: As a recurrent network but there are also intra-layer (lateral) connections.
• Deep network: There is more than one hidden layer; it can be feedforward or recurrent.

Describe the Hubel and Wiesel experiment on oriented edge detectors in V1

Much of the current understanding of the functional organization of the visual cortex had its origin in the pioneering studies of David Hubel and Torsten Wiesel at Harvard Medical School. The method they used was primarily microelectrode recordings in anesthetized animals that reported the responses of individual neurons in the lateral geniculate nucleus and the cortex to various patterns of retinal stimulation. The responses of neurons in the lateral geniculate nucleus were found to be remarkably similar to those in the retina, with a center–surround receptive field organization and selectivity for luminance increases or decreases. However, the small spots of light that were so effective at stimulating neurons in the retina and lateral geniculate nucleus were largely ineffective in the visual cortex. Instead, most cortical neurons in cats and monkeys responded vigorously to light–dark bars or edges, and only if the bars were presented at a particular range of orientations within the cell’s receptive field. The responses of cortical neurons are thus tuned to the orientation of edges, much as cone photoreceptors are tuned to the wavelength of light; the peak in the tuning curve (the orientation to which a cell is most responsive) is referred to as the neuron’s preferred orientation. By sampling the responses of a large number of single cells, Hubel and Wiesel demonstrated that all edge orientations were roughly equally represented in the visual cortex. As a result, a given orientation in a visual scene appears to be “encoded” in the activity of a distinct population of orientation-selective neurons.

Describe the features of the localist connectionist model

In localist connectionist models, there is no learning. The connectivity and the strength of connections are set by hand and levels of representations are stipulated (pre-defined). Each node in the network corresponds to a representation with a specific role (e.g., word nodes) (localist representation). The ability is investigated not from its acquisition but by the capacity of the model to reproduce the function. So, the emphasis is on behavioral data.

The interactive activation model and the context effects in letter perception

In letter perception there is a context effect. You are able to distinguish the two words, event and went, and so their different meaning, although they look written in the same way. The ambiguous stimuli don’t create problems for us, because due to the context of the sentence we understand anyway. This phenomenon is well shown and represented by the Interactive Activation Model (IAM) by McClelland and Rumelhart (1981). It consists of three levels of nodes. In the bottom level, there were the features detectors; in the middle one, the letter detectors and at the top there was a node for every word, that can be composed by those letters. This model contained almost all English words of 4 letters. In the scheme of the model below, the top-down and bottom-up connections are visible. Some connections end with a dot instead of an arrow so you can distinguish the inhibition ones. The credit of McClelland and Rumelhart was to have introduced for the first time the lateral inhibition, i.e. inhibition connections between nodes of the same level. The most activated node sends inhibitory signals to the other nodes of the same level (the winner takes it all). Interactive, recurrent processing accounts for context effects (e.g., ambiguous letters, word superiority effect). What word is it? The problem concerns the understanding of the last letter. It can be an R or a K. As shown in the graphs below, before the separation between R and K, the word WORK is raising up, because it is the most consistent and so sends feedback to letter level, activating its components. Hence, K increases and sends inhibition to R. Word recognition starts as soon as the activation in the letter level is upper than 0 and so starts to send info to upper level. Therefore, there is a mix of bottom-up and top-down processes.

What are the neural correlates of illusory contours? (Lee and Nguyen)

An illusory contour like this one consists of a change of luminance that is not real, but it is perceived as another triangle. Lee and Nguyen (2001) investigated the neural correlates of the phenomenon with four control conditions. They hypothesized that this illusion can be explained as a top-down process effect. To test this, they look at the timing of activation for the illusion, compared to a real line and other conditions, in the primary visual cortex and in the secondary visual cortex. The two graphs below show, at the left, the activation in V1 and, at the right, the activation in V2. In V2 population of neurons, there is an activation for the illusory contours (red line) after about 50 ms from the presentation of the stimulus. In V1, the peak for the red line is after about 100 ms. Hence, the peak in V2 occurs before the one in V1. This can mean that V1 response is an effect of a top-down signal, the V2 activation. Nothing comes from the retina or from the lateral geniculate nucleus. However, it is not possible to conclude about causality for these findings, because there is no manipulation, i.e., to knock out V2 and see whether there is a rise of signal in V1 or not.

Describe the connectionist model of learning

In the connectionist models of learning, the representation of a concept is not localized but is distributed among more nodes. Not everything is predefined about the network. Indeed, we allow learning to set levels. It is an emergent property and we have to look inside the network to understand what’s going on. Sometimes you can have something without explanatory value, which is a black box. In 1958, Rosenblatt, in order to build a learning machine to recognize shape, developed a network, “perceptrons”, with pixels of an image as input and the shape of the image as output. Connection weights were modified to decrease the discrepancy between the network output and the desired output (target). This is called supervised learning. Hence, learning is based on the correction of a mismatch. However, this network was criticized by Minsky and Papert (1969), because it cannot learn an important class of problems, but just those ones that can be solved with a linear separation. In 1986, Rumelhart, McClelland and the PDP group discovered a way to go over the limitation of perceptrons, introducing the concept of back-propagation.

Describe what are the three types of learning and how learning works in a neural network

• Supervised learning: The machine is also given desired outputs y1, y2, …, and its goal is to learn to produce the correct output given a new input (generalization). It is not very human-like.
• Unsupervised learning: The goal of the machine is to build representations from that can be used for reasoning, decision making, predicting things, communicating etc.
• Reinforcement learning: The machine can also produce actions which affect the state of the world, and receives rewards (or punishments) r1, r2. Its goal is to learn to act in a way that maximizes rewards in the long term.

Learning consists in changing the synaptic weights, that is the computation of D w with respect to the current value of w. The update may take place after each pattern (online learning) or after each epoch (batch learning). Initial values of synaptic weights are randomly assigned (e.g., between -0.1 and +0.1) or set to zero. To avoid the deletion of previously learned knowledge, only a fraction of the computed synaptic change must be used, which is defined through the constant η-learning rate.

Define the concepts of generalization and overfitting

The aim of models like these is to obtain generalization, i.e., an appropriate output (labels) for patterns in the test data (that is, data that were not used for training). However, sometimes a poor generalization on test data despite good performance on training data occurs, called overfitting. It can occur due to an unregular X-Y relation or noise.

What is linear separability? Is it always good?

Imagine an example with two classes of elements to separate (a cat or a dog), considered two features, X and Y (height and weight), every element of every class can be in a Cartesian plane according to its X-Y relation. Now the problem is to separate these elements in a way that at a side of a traced line there is a class and at the other side, the other one. Which classifier can be used? A linearseparation is not appropriate every time (Minsky’s accusation to perceptrons), because it can happen that there are too many exceptions. Hence, a predictor too inflexible cannot capture the pattern. Otherwise, a predictor too flexible, that tries to avoid any exceptions, can also be bad, because it fits noise in the data and the network learns also the noise in the training data. A good classifier is flexible and fits well with the data, although there might be some exceptions, but as seen trying to avoid exceptions can be worse.

Describe how to partition examples into datasets, including the cross-validation

We need the training data to be as large as possible. If there are lots of examples, one possible division is (in percentages):

• 50, Training dataset: examples used for training the classifier, that is to find the classifier parameters
• 25, Validation dataset: used for tuning the parameters of a classifier
• 25, Testing dataset: used for assessing the performance of the classifier