Second Semester 2009/2010, 6 EC

Lecturer: PD Dr. Reinhard Blutner
ILLC, University of Amsterdam

Lectures: Wednesday 9-11, P.014 (Plantage Muidergracht 24)
 

Office Hours: by appointment
Science Park 904, Room C3.128
 

Outline 

Examinations

This course will be graded based on

• A powerpoint presentation and/or a term research paper  is 40% of course grade. 
  Final deadline for the paper: June 6 (grade is reduced if work is late: -1 per day!)
  
• Written test (=45 minute exam),  count 30%. 
• Practical exercises. Submit the  tlearn exercise 8, count 30%. 
  Final deadline for the exercise: June 6 (grade is reduced if work is late: -1 per day!)
• Class attendance and doing homework is expected

Schedule

Part  A

• Week 1. Introduction  [slides]. Exercise 1
      tlearn exercise 1. Learning to use the simulator (starting with chapter 3)

• Week 2. Neural Nets and Learning [slides]. Exercise 2
      tlearn exercise 2  (chapter 3)

• Week 3. Neural Nets and Learning  continued. Exercise 3
      tlearn exercise 3  (chapter 4). Learning internal representations

• Week 4. Hopfield Networks  and Stability [slides]. Exercise 4
      tlearn exercise 4  (chapter 5). Auto-association
      Additional task: experiment with the Hopfield network as associative memory

• Week 5. Distributed representations and auto-associative memory [slides].  Exercise 5. Obligatory reading: Jordan B. Pollack (1990). Recursive Distributed Representations
       tlearn exercise 5 (chapter 7). Translation invariance

• Week 6. Structure in time: Recurrent Networks [slides]. Exercise 6. Obligatory reading: Jeffrey L. Elman (1990). Finding structure in time. 
      tlearn exercise 6 (chapter 8). Simple recurrent networks

• Week 7. No course, exam preparation
     

Part B

• Week 1. Is language governed by rules? Models of  Past-tense Aquisition [slides]. Exercises 7. Obligatory reading: Pinker and Prince (1988). On Language and Connectionism: Analysis of a Parallel Distributed Processing Model of Language Acquisition . Additional reading: Kim Plunkett & Virginia Marchman (1991). U-shaped learning and frequency effects in a multi-layered perceptron.
      tlearn exercise 7 (chapter 11). Learning the English past tense

• Week 2. Practical session: discussing earlier tlearn exercises. Test

• Week 3. Opening the Connectionist-Symbolist Debate  [slides]; The nature of systematicity [slides].  Student presentation
      Obligatory reading: Jerry A. Fodor and Zenon W. Pylyshyn (1988) Connectionism and Cognitive Architecture: A Critical Analysis

• Week 4. Integrating Connectionism and Symbolism: Nonmonotonic Reasoning and Hopfield Nets. [slides].
  Modelling propositional reasoning in neural networks. Student presentation

• Week 5. ???.  Student presentation

• Week 6. Introduction into spiking networks. Backpropagation for spiking networks. Student presentation

• Week 7. Assemblies, synfire chains, echo/fluid machines. Student presentation
      Obligatory reading: Abeles, Heyon, Lehmann (2004): Modeling Compositionality by Dynamic Binding of Synfire Chains

Possible topics for projects/essays

1. What is systematicity?
   Tim van Gelder and Lars Niklasson: Classicalism and Cognitive Architecture
   Blutner, Hendriks, de Hoop, Schwartz: When Compositionality Fails to Predict Systematicity
   
   
2. Modelling conceptual combination
   Edward E. Smith, Daniel N.Osherson, Lance J. Rips, and Margaret Keane: Combining Prototypes: A Selective Modification Model
   If prototype theory is to be extended to composite concepts, principles of conceptual composition must be supplied. This is the concern of the present paper. In particular, we will focus on adjective-noun conjunctions such as striped apple and not very red fruit, and specify how prototypes for such conjunctions can be composed from prototypes for their constituents. While the specifics of our claims apply to only adjective-noun compounds, some of the broader principles we espouse may also characterize noun-noun compounds such as dog house.
   
   Barry Devereux & Fintan Costello: Modelling the Interpretation and Interpretation Ease of Noun-Noun Compounds Using a Relation Space Approach to Compound Meaning
   How do people interpret noun-noun compounds such as gas tank or penguin movie? In this paper, we present a computational model of conceptual combination. Our model of conceptual combination introduces a new method of representing the meaning of compounds: the relations used to interpret compounds are represented as points or vectors in a high-dimensional relation space. Such a representational framework has many advantages over other approaches. Firstly, the highdimensionality of the space provides a detailed description of the compound’s meaning; each of the space’s dimensions represents a semantically distinct way in which compound meanings can differ from each other. Secondly, representation of compound meanings in a space allows us to use a distance metric to measure how similar of different pairs of compound meanings are to each other. We conducted a corpus study, generating vectors in this relation space representing the meanings of a large, representative set of familiar compounds. A computational model of compound interpretation that uses these vectors as a database from which to derive new relation vectors for new compounds is presented.
   
   
3. Relating and unifying connectionist networks and propositional logic
   Gadi Pinkas (1995). Reasoning, connectionist nonmonotonicity and learning in networks that capture propositional knowledge. [1,6 MB!]
   Reinhard Blutner (2005): Neural Networks, Penalty Logic and Optimality Theory
   
   
4. Symbolic knowledge extraction from trained neural networks
   A.S. d’Avila Garcez, K. Broda, & D.M. Gabbay (2001). Symbolic knowledge extraction from trained neural networks: A sound approach.
   Although neural networks have shown very good performance in many application domains, one of their main drawbacks lies in the incapacity to provide an explanation for the underlying reasoning mechanisms. The “explanation capability” of neural networks can be achieved by the extraction of symbolic knowledge. In this paper, we present a new method of extraction that captures nonmonotonic rules encoded in the network, and prove that such a method is sound.   
   
   
   
5. Natural deduction in connectionist systems
   William Bechtel (1994): Natural Deduction in Connectionist Systems
   The relation between logic and thought has long been controversial, but has recently influenced theorizing about the nature of mental processes in cognitive science. One prominent tradition argues that to explain the systematicity of thought we must posit syntactically structured representations inside the cognitive system which can be operated upon by structure sensitive rules similar to those employed in systems of natural deduction. I have argued elsewhere that the systematicity of human thought might better be explained as resulting from the fact that we have learned natural languages which are themselves syntactically structured. According to this view, symbols of natural language are external to the cognitive processing system and what the cognitive system must learn to do is produce and comprehend such symbols. In this paper I pursue that idea by arguing that ability in natural deduction itself may rely on pattern recognition abilities that enable us to operate on external symbols rather than encodings of rules that might be applied to internal representations. To support this suggestion, I present a series of experiments with connectionist networks that have been trained to construct simple natural deductions in sentential logic. These networks not only succeed in reconstructing the derivations on which they have been trained, but in constructing new derivations that are only similar to the ones on which they have been trained.
   
   
6. Modelling logical inferences: Wason's selection task and connectionism
   Steve J. Hanson,  Jacqueline P. Leighton , & Michael R.W. Dawson: A parallel distributed processing model of Wason’s selection task
   Three parallel distributed processing (PDP) networks were trained to generate the ‘p’, the ‘p and not-q’ and the ‘p and q’ responses, respectively, to the conditional rule used in Wason’s selection task.  Afterward, each trained network was analyzed for the algorithm it developed to learn the desired response to the task. Analyses of each network’s solution to the task suggested a ‘specialized’ algorithm that focused on card location. For example, if the desired response to the task was found at card 1, then a specific set of hidden units detected the response. In addition, we did not find support that selecting the ‘p’ and ‘q’ response is less difficult than selecting the ‘p’ and ‘not-q’ response. Human studies of the selection task usually find that participants fail to generate the latter response, whereas most easily generate the former. We discuss how our findings can be used to (a) extend our understanding of selection task performance, (b) understand existing algorithmic theories of selection task performance, and (c) generate new avenues of study of the selection task. 
   
   
7. Infinite RAAM: A principled connectionist substrate for cognitive modelling
   Simon Levy and Jordan Pollack (2001): Infinite RAAM
   Unification-based approaches have come to play an important role in both theoretical and applied modeling of cognitive processes, most notably natural language. Attempts to model such processes using neural networks have met with some success, but have faced serious hurdles caused by the limitations of standard connectionist coding schemes. As a contribution to this effort, this paper presents recent work in Infinite RAAM (IRAAM), a new connectionist unification model. Based on a fusion of recurrent neural networks with fractal geometry, IRAAM allows us to understand the behavior of these networks as dynamical systems. Using a logical programming language as our modeling domain, we show how this dynamical-systems approach solves many of the problems faced by earlier connectionist models, supporting unification over arbitrarily large sets of recursive expressions. We conclude that IRAAM can provide a principled connectionist substrate for unification in a variety of cognitive modeling domains.
   
   
8. Encoding nested relational structures in fixed width vector representations.
   Tony A. Plate (2000): Analogy retrieval and processing with distributed vector representations
   Holographic Reduced Representations (HRRs) are a method for encoding nested relational structures in fixed width vector representations. HRRs encode relational structures as vector representations in such a way that the superficial similarity of the vectors reflects both superficial and structural similarity of the relational structures. HRRs also support a number of operations that could be very useful in psychological models of human analogy processing: fast estimation of superficial and structural similarity via a vector dot-product; finding corresponding objects in two structures; and chunking of vector representations. Although similarity assessment and discovery of corresponding objects both theoretically take exponential time to perform fully and accurately, with HRRs one can obtain approximate solutions in constant time. The accuracy of these operations with HRRs mirrors patterns of human performance on analog retrieval and processing tasks.

    

9. New solutions to the binding problem
   Abeles, Heyon, Lehmann (2004): Modeling Compositionality by Dynamic Binding of Synfire Chains
   This paper examines the feasibility of manifesting compositionality by a system of synfire chains. Compositionality is the ability to construct mental representations, hierarchically, in terms of parts and their relations. We show that synfire chains may synchronize their waves when a few orderly cross links are available.We propose that synchronization among synfire chains can be used for binding component into a whole. Such synchronization is shown both for detailed simulations, and by numerical analysis of the propagation of a wave along a synfire chain. We show that global inhibition may prevent spurious synchronization among synfire chains. We further show that selecting which synfire chains may synchronize to which others may be improved by including inhibitory neurons in the synfire pools. Finally we show that in a hierarchical system of synfire chains, a part-binding problem may be resolved, and that such a system readily demonstrates the property of priming. We compare the properties of our system with the general requirements for neural networks that demonstrate compositionality.
   
   See also: van der Velde (2005): Neural blackboard architectures
   
   
10. The role of symbolic grounding within embodied cognition / Grounding symbols with neural nets
    Michael L. Anderson: Embodied Cognition: A field guide
    Stevan Harnad: Grounding symbols in the analog world with neural nets -- A hybrid model (Target Article on Symbolism-Connectionism)
    Bruce J. MacLennan: Commentary on Harnad on Symbolism-Connectionism
    Stevan Harnad: Symbol Grounding and the Symbolic Theft Hypothesis
    
    
11. Subsymbolic language processing using a central control network
    Risto Miikkulainen: Subsymbolic case-role analysis of sentences with embedded clauses
    A distributed neural network model called SPEC for processing sentences with recursive relative clauses is described. The model is based on separating the tasks of segmenting the input word sequence into clauses, forming the case-role representations, and keeping track of the recursive embeddings into different modules. The system needs to be trained only with the basic sentence constructs, and it generalizes not only to new instances of familiar relative clause structures, but to novel structures as well. SPEC exhibits plausible memory degradation as the depth of the center embeddings increases, its memory is primed by earlier constituents, and its performance is aided by semantic constraints between the constituents. The ability to process structure is largely due to a central executive network that monitors and controls the execution of the entire system. This way, in contrast to earlier subsymbolic systems, parsing is modelled as a controlled high-level process rather than one based on automatic reflex responses.

Books used to prepare the lecture

Related Websites 

• Tlearn software

• Flexible simulation program LENS including doumentation

• Martin Ryder's connectionism links

• The Experimental Philosophy Laboratory

• Behavioral and Brain Sciences

• Rutgers Optimality Archive

• A Brief History of Connectionism (David A. Medler)

• William Bechtel

• Jeff Elman

• Tim van Gelder

• Stephen Grossberg

• Stevan Harnad

• Günther Palm 

• Jordan B. Pollack

• Alan Prince

• Zenon Pylyshyn

• Lokendra Shastri

• Paul Smolensky

• Ron Sun

• Thomas Wennekers