Maths: Auditory interaction with mathematics

Keywords

mathematics, blindness, visual disability, visual impairment, speech, non-speech sounds, SGML

Background

The main aim of the Maths Project is to develop a multi-media workstation for the presentation, and manipulation of mathematics for students with visual disabilities. One of the design principles is that the workstation should make as much use as possible of multiple communications media. Input of mathematics and control commands will use speech recognition and keyboard and output will be in the form of synthetic speech, non-speech sounds, braille and an enhanced visual display (for partially sighted users).

Sounds

Having a variety of output formats means that users can use them in combination (listening to an equation being spoken at the same time as reading it on the braille display, for instance) or concentrate on one format which is appropriate to their current task (braille is generally better for information in a tabular format, for instance).

Although the multiple output format is important, we at York are concentrating on the sounds output. Most of the ideas are based on those developed by Robert Stevens within his Mathtalk program.

Reading

Just being able to read mathematics is very difficult for visually disabled people. Non-visual representations (such as braille) are not as powerful and flexible as visual ones. Take for instance the following well-known equation: $x equals the fraction, numerator negative b plus or minus the square root of b squared minus 4 a c, denominator 2a$

This uses the two dimensions of the paper to represent the fraction as a clear single component. It also uses the relative positions of different components in a semantically meaningful way. For example the bar on the top of the square root symbol extends over all the symbols governed by it. There is also no redundancy in the representation (unlike written text); if any one symbol were deleted the meaning of the equation would be changed completely.

Just giving visually disabled people access to equivalent representations will be a major achievment of the Maths project. An important aspect of the project is that the central representation will be SGML (the Standardized General Markup Language). It is important that this is a standard and an increasing number of books (including mathematics ones) are being published in that format. (It is also closely related to HTML - the format used for Web pages). Thus, it can be expected that blind students will be able to access the same maths books as their sighted peers.

A problem with reading mathematics in non-visual formats is to be able to control the reading. For instance, if one listens to a whole equation at once, one only remembers a small part of it. When reading visually people can focus on the parts of the equation which are of interest and ignore the rest. There is therefore a requirement to give users of the Maths Workstation a similar level of control. This will be based on the browsing commands developed within Mathtalk.

Interaction

A step further on from being able to read mathematical material is to be able to manipulate it, to create new equations and modify existing ones. Within the Maths Workstation manipulations will be based on concepts of selection and re-writing. One of the principles behind the Maths Workstation design is that the workstation will not do any of the mathematical maniplations for the user; it will not be a computer-based tutor. As a teaching tool, it will still rely on the teacher to teach the mathematical concepts to the student. The workstation should not give the student any advantage over a sighted student using a pencil and paper.

In the future it may be desirable for people to be able to use the workstation with symbolic manipulation packages and to that end there will be `hooks' in the system to connect to such pacakges. This will be useful for more advanced students who have already learned the basic mathematics and do not want to be bothered by the low-level manipulations.

Representations

The Maths project is concentrating on textual representations of mathematics (algebra and related notations). There are many other visual representations of mathematical information such as diagrams, tables and graphs but within this project we chose to not be over-ambitious and to stick to textual representation. In a separate project at the University of York, we have developed Soundgraphs as a means of presenting simple line graphs to blind people.

Personnel

People at York who are working on this project are:

Alistair Edwards
Robert Stevens

Publications

Edwards, A. D. N., Stevens, R. D. and Pitt, I. J. Repreésentation non visuelle des mathématiques , (translated by A. Assimacopoulos) in A. B. Safran and A. Assimacopoulos (editors) Le Déficit Visuel,Éditions Masson, pp. 169-178 (1995).
Edwards, A. D. N and Stevens, R. D., (1994) Une interface multimodale pour l'accèss aux formules mathématiques par des élèves ou étudiants aveugles, Comme les Autres: Interfaces multimodales pour handicapés visuels, Special number 1, pp. 97-104. published by INSERM (Université Pierre et Marie Curie, B23, 9 Quai Saint-Bernard, 75252, Paris Cedex 05) and ANPEA (ISSN 0010-2520).
Stevens, R. D., Wright, P. C., and Edwards, A. D. N. (1994). Prosody improves a speech based interface. D. England (editor) Ancillary Proceedings of HCI'94, London: British Computer Society.
Edwards, A. D. N. and Stevens, R. D., Mathematical representations: Graphs, curves and formulas, in D Burger and J-C Sperandio (Eds) Non-Visual Human-Computer Interactions: Prospects for the visually handicapped (Proceedings of the INSERM Seminar Non-visual presentations of data in human-computer interactions, Paris, March 1993), John Libbey Eurotext, pp 181-194.
Stevens, R. D. and Edwards, A. D. N., A Sound Interface to Algebra, Proceedings of the IEE Colloquium on Special Needs and the Interface, London, January 1993 (IEE Digest no. 1993/005)
Edwards, A. D. N., Pitt, I. J, Brewster, S. A and Stevens, R. D. Multiple modalities in adapted interfaces In A. D. N. Edwards (editor) Extra-Ordinary Human-Computer Interaction, New York: Cambridge University Press, (in press).
Stevens, R. D., Brewster, S. A., Wright, P. C. and Edwards, A. D. N. Design and evaluation of an auditory glance at algebra for blind readers. In G. Kramer, (editor), Auditory Display: The Proceedings of the Second International Conference on Auditory Display. Addison-Wesley (in press) 1995.