Project 10: Hybrid episodic-abstract computational modelling: HSP
Although standard HSP models have had considerable success in simulating some of the basic phenomena of spoken word recognition, any further progress will be critically dependent on constructing models that pay proper attention to the FPD available in the speech signal. This project aims to achieve this by constructing a new computational model that uses FPD to guide the construction of prosodic and grammatical structures that can drive the recognition process.
Computationally, Prosynth/Polysp prosodic trees are XML structures with nodes in the prosodic tree linked to nodes in the grammatical tree. The aim is to establish whether these trees and their relationships are all that are needed to model speech perception. If so, an experienced listener can be seen as placing feature values on the nodes of a general tree or linked set of trees. The main difference between this and a standard perception model is that the FPD maps onto every type of node, at any height in the tree, not just the bottom level of the tree that in a standard model specifies phonemes/phonological features. This conceptually simple difference is challenging to model.
Methods and materials We will progress from a ‘proof of principle’ evaluation of Prosynth/Polysp, to the development of a complete computational model of speech recognition.
In phase 1 we will construct a modelling framework with flexible, complex data structures that can represent prosodic, phonological, morphological and lexical information, their inter-relationships, and the FPD that accompanies these structural distinctions. Flexibility is important, as the representations specified by Prosynth/Polysp will evolve during the course of the project. Because each utterance type has a unique structure, we will work first with a small set of structures whose FPD is perceptually salient, so that clear predictions can be made based on behavioural data.
In phase 2, the model will be tested on annotated handcrafted input representing the FPD that we know is extracted from the signal. This will be used to incrementally construct Polysp structures consistent with the current input. These structures, the values of whose nodes will be assigned probabilistic weightings according to their consistency with the input, will then be used to constrain all hypotheses about the signal’s linguistic structure: featural, segmental, syllabic, prosodic, grammatical and lexical. The structured representations will be revised and updated as new evidence arrives in the same kind of updating and re-evaluation that occurs in existing ‘abstractionist’ models like SHORTLIST, but with the advantage of being further constrained by attention to FPD and knowledge about what it signifies. Standard search procedures will be used to generate a continuously evolving measure of the probability of alternative structural hypotheses (cf the methods used in SpeM for deriving ‘word activation’ scores). Weightings for specific nodes and properties will be derived from perceptual data and judgments of the auditory salience of specific utterances’ acoustic properties in the ambient noise conditions.
In phase 3 we will try to incorporate automatic extraction of a limited set of FPD features from real speech, using easily-segmentable utterances with differing grammatical status, so that the focus is not on word recognition per se, but on the pattern and temporal order in which systematically varying FPD is mapped onto the nodes of the data structures.
Young researchers One ESR (Piccolino) is based at Cambridge. His likely visits are to York, to learn prosodic phonology, and to Leuven, to learn hybrid ASR computational modelling techniques.
Links This project has close but complementary links with Projects 7, 9 & 11 and can use FPD insights from Theme I and, in time, Project 8. The primary link is between psychological modelling and experimental phonetics; and we anticipate extensions to engineering models of ASR.
Working on this project: » Dr Dennis Norris » Dr Richard Ogden » Prof Sarah Hawkins » Marco Aldo Piccolino Boniforti
