Practical Aspects of Using Video in the Foreign Language Classroom

Christine Canning-Wilson
christine.canning [at] hct.ac.ae
The Center of Excellence for Research and Training, Higher Colleges of Technology (Abu Dhabi)

Current Theory on the Use of Video as an Educational Medium of Instruction

Video is at best defined as the selection and sequence of messages in an audio-visual context. Considerable confidence is placed in the value of audio-visual aids to enhance the learning of foreign languages, yet there is little empirical data and research to support the proposition that video facilitates in the learning of foreign languages. However, with the amount of time devoted to using video in the Foreign or Second Language (F/SL) classroom, research is warranted to show how audio-visual aids enhance the language learning process. Currently, research over the past two decades, shows that there are several limitations to be recognized when examining results. First, many studies have been done with visual aids and not with actual foreign and second language videos. Secondly, many video studies use intact groups instead of random groups who were studying only one foreign language. Thus, studies using different groups and languages may yield different results than those found in the literature. As Omaggio (1979) points out, to find varied results, researchers would need to ask whether or not the same findings would hold true in different languages.

Another limitation on video studies deals with the long-term effects of using video in the classroom. It can be argued that video instruction should be discouraged because there is scant empirical proof to verify comprehension. For example, how can long-run effects of video be measured and how much exposure to video would make a significant difference in the language learning process? In addition to these queries, researchers would need to ask if the visual significantly affects listening comprehension (Mueller, 1980). Research would also need to address how video manifests itself differently than prolonged exposure to a visual aid that does not have an audio component? With such unanswered questions yet to be proven with quantitative measures practitioners must ask themselves if there is sufficient evidence to support continued use of audio-visual aids in the learning of foreign languages to justify the allocation of resources for them in the foreign or second language classroom.

However, in recent years, some of these questions have begun to be addressed, but in limited contexts. For example, Balatova (1994) suggests that unlike a student, who listened in sound only conditions, the use of video and sound conditions were more consistent in their perception of the story, in the sense that difficult and easy passages formed a pattern. The study notes that scenes where utterances were backed up by an action and/or body language and that were relatively shorter, were considered easier to understand by students. Less lively scenes, which involved relatively long stretches of conversation, were labeled as more difficult. These comments illustrate that visual cues are important, since they either facilitated or distracted from understanding. In addition, her research also notes that "It is also interesting to point out that students in the sound-only conditions in the two experiments were less successful in maintaining the interest and concentration in listening".

Research by Herron, Hanley and Cole (1995) indicates that the visual support in the form of descriptive pictures significantly improved comprehension scores with language videos for English speaking students learning French. The results of the study indicate that extensive listening is facilitated by the richness of the context that visual organizers, such as educational videos, provide. Heron (1994) finds that advanced organizers based on videos helped learners improve comprehension and aid in the retention of information.

A recent large-scale survey by Canning-Wilson (2000) suggests that the students like learning language through the use of videos. One of the results of her survey shows that learners prefer action/entertainment films to language films or documentaries in the classroom. She states although these films may seem to hold student interest, she believes that it could be inferred that student comprehension of the video may be due to the visual clues instead of the auditory components.

Heron, Hanley and Cole also hypothesize that the more meaningful an advanced organizer is the more impact it can have on comprehension and retention. Their results of using twelve different videos with foreign language learners indicates that scores improved when advanced organizers, such as a pictures and/or visual stimuli, are used with the video. Perhaps the findings from these studies can be attributed to the fact that video offers contextual support and/or helps learners to visualize words as well as meanings.

Individuals process information in different ways. The strategies used by one learner are likely to differ from those used by a different learner. It has been proven that what benefits one group of learners may actually hinder the performance of a different group of learners (Bovy, 1981). Issues of the value of video as a teaching tool are often questioned. Omaggio (1979) suggests that "the profession has virtually no empirical basis for promoting the use of visuals as aids to comprehension in the second language; we know practically nothing about how students benefit from visuals" (1979, p.107). Recently, in a lecture on the use of visuals in research, Canning-Wilson (2000) claims that the use of illustrations, visuals, pictures, perceptions, mental images, figures, impressions, likenesses, cartoons, charts, graphs, colors, replicas, reproductions, or anything else used to help one see an immediate meaning in the language may benefit the learner by helping to clarify the message, provided the visual works in a positive way to enhance or supplement the language point. She reports that images contextualized in video or on its own can help to reinforce the language, provided the learner can see immediate meaning in terms of vocabulary recognition in the first language. Furthermore, her research suggests that visuals can be used to help enhance the meaning of the message trying to be conveyed by the speakers through the use of paralinguistic cues.

Additional factors must be considered when looking at video as an instructional medium to teach a foreign or second language. In 1999, Coombe and Kinney stated that "Learners learn primarily because of what they bring to their classroom experience in terms of their perceived needs, motivations, past experiences, background knowledge, interests and creative skills" (1999,p.21). Furthermore, experts suggest that instructional design and cognitive processing considerations are more salient than media used to deliver the instruction (Clark, 1983). Hannafin (1986) suggests that the incorporation of criterion-based questions in video instruction is likely to improve intended learning, but scant empirical data has unequivocally proven this hypothesis. It is important to note that in the late 1980s, Hannafin admitted, "there was no research to support one side over the other, so we are left with logic and common sense to form an initial hypothesis" (Hannafin 1986). In 1994, Balatova¼s studies indicated that visual cues found invideos were informative and enhanced comprehension in general, but did not necessarily stimulate the understanding of a text. It was also found that teaching with video had some affective advantages. If the results of their findings are true and the same results can be replicated, perhaps practitioners will no longer have to rely solely on anecdotal evidence.

Using Video in the F/SL Classroom

What are the practical implications of using video in the classroom? At the most basic level of instruction, video is a form of communication and it can be achieved without the help of language, since we often interact by gesture, eye contact and facial expression to convey a message. Video provides visual stimuli such as the environment and this can lead to and generate prediction, speculation and a chance to activate background schemata when viewing a visual scene reenacted. It can be argued that language found in videos could help nonnative speakers understand stress patterns. Videos allow the learner to see body rhythm and speech rhythm in second language discourse through the use of authentic language and speed of speech in various situations. Videos allow contextual clues to be offered. In addition, video can stimulate and motivate student interest. The use of visuals overall can help learners to predict information, infer ideas and analyze the world that is brought into the classroom via the use of video instruction. In a teaching or testing situation video can help enhance clarity and give meaning to an auditory text; it can create a solid link between the materials being learned and the practical application of it in a testing situation; the video can act as a stimulus or catalyst to help integrate materials or aspects of the language; videos can help manipulate language and at the same time be open to a variety of interpretations.

Arthur (1999) claims that:

"Video can give students realistic models to imitate for role-play; can increase awareness of other cultures by teaching appropriateness and suitability; can strengthen audio/visual linguistic perceptions simultaneously; can widen the classroom repertoire and range of activities; can help utilize the latest technology to facilitate language learning; can teach direct observation of the paralinguistic features found in association with the target language; can be used to help when training students in ESP related scenarios and language; can offer a visual reinforcement of the target language and can lower anxiety when practicing the skill of listening."

Video used in a classroom should be interpretive and to the point. The visual should show reasonable judgement and enhance comprehension, heighten sensory acuteness, and illustrate the target language being used. Practitioners should avoid the use of distracters, over-crowded or violent stimuli. Visuals are ineffective in the learning process when the visual is too small; when the visual or video uses stereotypes; when the visual or video is a poor reproduction; when the picture is to far away from the text illustration; when the video has irrelevant captioning; when the video or visual offers to much information related or unrelated to the picture; when the video or visual is poorly scaled; and when the picture is not esthetically meaningful. A visual cue may be accompanied by a written cue to focus on a lexical item being furnished. Videos can make the task, situation or language more authentic. More importantly, video can be used to help distinguish items on a listening comprehension test, aid in the role of recall, help to sequence events, as well as be adapted, edited or changed in order to meet the needs of the language learner (Canning, 1998).

Suggestions to the Classroom Practitioner

If video is to be used in the classroom to improve listening comprehension, it should be shown in segments and not as a whole. These segments should be broken down to exploit the macro-listening skills and the micro-listening skills from the audio-component of the video. There is scant, if any, empirical evidence to indicate that videos shown in their entirety improve listening comprehension scores of nonnative speakers of English. For as much as the visual may aid in understanding the scenario or general gist of the film¼s plot, it may detract from the individual messages produced by the speakers. In fact the constant visual stimuli may detract from the auditory component. Empirical evidence has shown that attention spans are lowered when watching videos used to teach foreign languages (Balatova, 1994). As Balatova (1994) states: "The first signs of distraction in those groups appeared after the first minute, and by the end of four minutes, distraction spread all over the groups, while in the video conditions several more students became distracted after six minutes, more students lost concentration after ten minutes and around one third of them kept watching until the end."

Key Considerations for Using Video in the Classroom with Nonnative Speakers

Although video may be a popular tool to use with students, as F/SL educators we must not loose sight of the educational purpose it has in the language classroom. It is suggested that language teachers should ask themselves the following questions before implementing a video lesson with F/SL learners:

• How will the language learner benefit from the use of video in the classroom?
• How will the visual component enhance the auditory component?
• Who will select the video? Is it the class, the teacher or the curriculum developers?
• Who decides which language should be exploited from the video? Is it the class, teacher or curriculum developers?
• How do you plan to exploit the 1-10 minute segment of the video? What are the alternative methods of exploiting the clip for further reinforcement in the classroom?
• Whose responsibility is it to select key vocabulary and structures from the video?
• Who decides how many times the video is played?
• How can students and teachers develop academic listening and conversational listening activities based on the video?
• How is the video used in a classroom context?
• How does video support the curriculum?
• Can the comprehension of the video be measured without visual support?
• Can the comprehension of the video be measured without auditory support?
• How will you assess the comprehension of the video by the language learners?
• How practical is the video to improve a learner¼s academic listening and/or conversational listening skills?
• What is the educational purpose for showing the video? How will you later assess its effectiveness with the learners ability to comprehend information?

Conclusion

Because academic listening tasks are often tested rather than taught, video offers foreign and second language learners a chance to improve their ability to understand comprehensible input. Videos allow teachers to ask both display and referential questions. Video tasks used in the F/SL classroom, can include but are not limited to creating advanced organizers, other visual representations and descriptors.

Video tasks should be multi-layered in order to exploit all information and elements contained in the aural and visual texts. Additionally, it is essential that video tasks and lessons be perceived by the language leaner as a challenging and requiring effort. Be sure that students are able to answer questions based solely upon what they see instead of what they hear. Otherwise it is possible to imply that practitioners are measuring their visual literacy and not their ability to comprehend aural input.

With the increase in educational technology, video is no longer imprisoned in the traditional classroom; it can easily be expanded into the computer aided learning lab (Canning 1998). Interactive language learning using video, CD ROM, and computers allow learners the ability to view and actively participate in lessons at their desired pace. It is recommended that institutions and practitioners encourage the use of instructional video in the F.SOL classroom as it enables them to monitor and alternate instruction by fostering greater mental effort for active learning instead of passive retrieval of visual and auditory information.

References:

• Arthur, P. "Why use video? A teacher's perspective", VSELT 2:4 (1999): 4.
• Baltova, I." Impact of video on the comprehension skills of core French students" Canadian Modern Language Review, 50, 3 (1994): 506-531.
• Bovy, R. "Successful instruction methods: A cognitive information processing approach" Educational Communication and Technology Journal, 29, (1981): 203-217.
• Canning, C., "Visual support and language teaching", TESOL Arabia News, Volume 5:4, (1998): 3-4
• Canning, C.,"Maximizing the effect of visuals in software programs" EMCEE, 4:3, (1998): 7-8.
• Canning, C., "Educational Applications for information technology: re-evaluating computer aided instruction in the classroom", EMCEE, 4:3, (1998). 3-4
• Canning, C. "Theoretical use of Visuals". In Canning, C. and Koester, J., Illustrated Visual Aids for Academic English, 1:1, (1997): 2-4
• Canning-Wilson, C., Research in Visuals, Invited Paper for the Video Special Interest Group at the International TESOL Arabia 2000 Conference, Hilton Ballroom, Hilton Hotel, April 12-14, 2000.
• Canning-Wilson, C., "Role of Video in the F/SL Classroom", (2000): 69-76. In S. Riley, S Troudi and C. Coombe. (ed.) Teaching, Learning and Technology, TESOL Arabia 1999 Conference Proceedings, TESOL Arabia 1999 Conference March 8-10, 1999.
• Clark, R. "Reconsidering research on learning from the media" Review of Educational Research, 53, (1983): 445-459
• Coombe, C. and Kinney, J., "Learning Center Listening Assessment", FORUM, 37:2, April-June (1999) : 21-23
• Coombe, C., Kinney, J. and C. Canning "Issues in the Evaluation of Academic Listening Tests", in Caroline Chapman and Dianne Wall (eds.) Language Testing Update (International Testing Journal), v.24, (1998): 32-45
• Coombe, C., Kinney, J. and C.Canning, (1997) "Issues in Foreign and Second Language Academic Listening Assessment_"In C. Coombe (ed.) Current Trends in English Language Testing, v.2, October 1997/1998, pp. 27-36, TESOL Arabia Conference Proceedings held at Al Ain University, Al Ain, May 15, 1997.
• Herron, C, Hanley, J. and S. Cole,_" A comparison study of two advance organizers for introducing beginning foreign language students to video", Modern Language Journal, 79:3, (1995): 387-394.
• Herron, C. "An investigation of the effectiveness of using an advance organizer to introduce video in the foreign language classroom." Modern Language Journal, 78, (1994): 190-198.
• Mueller, G. "Visual Contextual Clues and Listening Comprehension: An experiment." Modern Language Journal 64 (1980): 335-40
• Omaggio, A "Pictures and Second Language Comprehension: Do They Help?" Foreign Language Annals 12 (1979): 107-16.

---

The Internet TESL Journal, Vol. VI, No. 11, November 2000
http://iteslj.org/

---

http://iteslj.org/Articles/Canning-Video.html

Aligning Learning Activities and Assessment Strategies in the ESL Classroom

Robert F. Tambini
robu_sensei [at] yahoo.com
Centenary College (New Jersey, USA)

Introduction

Accurately gauging the extent to which student progress meets the goals that have been set is one of the most difficult tasks which educators must perform. In order to do this effectively, it is essential that instructors design their lesson plans in such a way that expectations placed upon students regarding performance standards, instructional objectives and learning activities identified for use in the classroom, and assessment strategies employed in evaluating student learning are well-aligned with one another.

As a means of demonstrating the form that this alignment might take, I will discuss this alignment within the context of a lesson plan which I have used in since 1996. The class for which this lesson plan was developed was (and remains) part of Centenary College's spring Intensive English Language Program (IELP), a six-week program offered each year to students of Centenary's sister college in Tokyo, Japan. The original class for which this lesson plan was designed consisted of twenty 19 year-old female Japanese students at Obirin Junior College. This lesson plan has been subsequently used in both the spring and summer Intensive English Language Programs at Centenary College.

This discussion will begin with a presentation of the standards which informed the design of the lesson plan, and a description of the performance standards set out in the plan which students are expected to meet. The lesson plan itself follows. Finally, I will attempt to form a conclusion addressing the success with which the various elements of the plan have been aligned.It is important to note, and I wish to emphasize, that the particular lesson plan under discussion here is an actual lesson plan which has seen three years use in the classroom, not a hypothetical one constructed solely for the purpose of serving as the object of this study.

The Standards

Any English as a Second Language exercise is founded upon a single standard against which students will be measured: the relative character and quality of the English used by students as compared to the character and quality of the English used by native speakers. Of course, when we assign levels to second language (L2) students, we are judging them not only against the standard set by native speakers, but also relative to a) the received academic standard, and b) other L2 students. These standards, English as used by native speakers and English as used by L2 students, are built of a large number of components, including (but not limited to) the following:

1. Vocabulary - number and quality of items known; ability to use items correctly and appropriately
2. Grammar - knowledge of morphology and syntax; ability to apply knowledge correctly and appropriately
3. Oral production - the ability to create utterances comprehensible to native speakers
4. Listening comprehension - the ability to comprehend utterances produced by native speakers
5. Written production - the ability to manipulate symbols (letters, characters) and produce language structures comprehensible in a visual context, and to express feelings, concepts and ideas in writing
6. Reading comprehension - the ability to comprehend language symbols and structures in a visual or tactile context, and to understand feelings, concepts, and ideas expressed in writing

Within each of these areas any number of items might be identified by the teacher as the object of a specific lesson. The ultimate goal of ESL teaching is mastery of the English language; however, prior to becoming fluent in their use of English, students must first acquire a clear understanding of and proficiency in each of the areas outlined above.

About the Plan

This lesson plan was developed for use in the Intensive English Language Program, a six-week ESL program offered annually during the spring and summer semesters at Centenary College in Hackettstown, New Jersey. This program is attended by students from Obirin Junior College, Tokyo, Japan. Daily, the structure of the program is as follows:

• English Language and Conversation (three hours)
• Lunch (one hour)
• Language Lab (two hours)
• American Culture (two hours)

After dinner, students are expected to study and complete homework for approximately three hours. Upon completion of the IELP, students are awarded three transfer credits in English which they may apply as an elective toward their Obirin degrees. The lesson plan which will be described below is used in support of a unit on "apartment hunting" in the conversation class.

The Objective

The objective of this lesson is to actively engage students in the process of seeking out and making inquiries regarding the availability and suitability of living space in as authentic a manner as possible. In addition to this introduction to the apartment hunting process, students are required to make actual contacts and inquiries based on their identification of available and potentially desirable living spaces advertised in local newspapers. This requirement is intended to force students, who otherwise tend to become overwhelmed by and shy at the prospect of interacting with a native speaker, to use the English skills they possess in an authentic, real-life situation which many of them are likely to encounter again.

Pursuant to this objective, students will demonstrate: a) proficiency in their knowledge and use of English vocabulary related to housing; b) the ability to form appropriate wh- questions intended to elicit information which will assist in determining the suitability of a particular apartment; c) critical thinking skills, as evidenced in descriptions (written and oral) of the reasoning which led to the ultimate acceptance or rejection of a particular housing unit.

The Learning Activities

Various learning activities, designed to introduce students to the subject matter and to improve their proficiency relative thereto, are employed in support of this lesson. They are:

• In-class presentation of vocabulary items, accompanied by teacher explanations and pronunciation practice/drills
• An in-class game in which students are provided with descriptions of four available units and asked to match fictional prospective tenants with the apartment best suited to them
• In-class review of newspaper classified advertisements
• Library work, in which students locate the classified section of a local newspaper and identify an apartment which they feel might be suited to their current circumstances (re: rent, location, etc.)
• In-class, teacher-guided composition of wh- questions to be used during a telephone inquiry regarding the apartment
• As homework, students make actual inquiry of the landlord/agency using the questions they have composed
• Students report results of inquiry and determination of suitability both orally in class and to the teacher in writing

Evaluating Student Performance

The assessment strategies employed in evaluating student performance related to this lesson are as varied as the tasks which students are assigned. However, in the interest of brevity and for the purpose of this study, I will concentrate on the strategy used in only one area of the lesson - the actual contact between student and landlord/agency. As mentioned above, students must report the results of the inquiry with the landlord/agency offering the apartment for rent. This report takes two forms: a) an in-class presentation during which students describe their experiences interacting with a native speaker, as well as any information which they were able to gather regarding the apartment and rental terms; b) a written description of the conversation with the landlord/agency, the student's assessment of his own performance during the interview, and a justification for the student's decision to accept or reject the apartment.

Assessing Oral Perfomance

Assessment of the oral component of the report is conducted through teacher observation of student performance and teacher questioning during both the in-class presentation and a follow-up, one-on-one student/teacher conference. Students are evaluated primarily on the quality and appropriateness of the wh- questions which they have formulated, and on the effectiveness with which they were able to elicit information from the landlord/agency. The standard against which students are measured are the examples that were provided in class; additionally, students are judged on the quality and depth of their reflection on their own work.

Assessing Written Perfomance

Similarly, student written performance is also measured against the standard set by examples discussed in class. These examples, whether drawn from the textbook, provided to students in the form of handouts, or presented by the teacher during class, remain available throughout the course of the lesson as a means by which students might judge the progress and quality of their work. In addition, students receive an in-class explanation regarding those aspects of their report on which they will be evaluated and the teacher's expectations in these areas. The criteria for evaluation include: spelling, punctuation, word choice, grammatical correctness, sentence structure and variety, clarity, coherence, organization, and overall effectiveness.

Providing Feedback

Feedback is an essential component of this lesson, and is provided in any number of ways: in-class teacher response, one-on-one student/teacher conferences, written comments, etc. Most important, however, is the fact that during this lesson the lines of communication are not filtered through a one-way valve. Of course, feedback is provided to the student by the teacher; but, students are also encouraged and expected to evaluate the quality and appropriateness of the services they receive, and to offer constructive criticisms relative thereto. For example, the one-on-one student/teacher conferences are not intended merely as opportunities for the teacher to "preach from the mountaintop," but are rather meant to provide students with the chance to offer input as to how their learning might be better facilitated.

Conclusion

Clearly, the alignment of performance standards, instructional objectives, and assessment strategies is necessary if student learning and progress are to be evaluated accurately. I hope that, through the presentation and discussion of the lesson plan presented above, I have been able to demonstrate at least some of the manifold forms that this alignment might take. During the design this lesson plan, the alignment of standards, objectives, and strategies was a matter of primary concern. I believe that this alignment - insofar as is possible within the context of a single lesson plan - has been successfully achieved: students are given the opportunity to learn, the teacher is provided effective means to gauge that learning, and communication between student and teacher is encouraged and supported.

---

The Internet TESL Journal, Vol. V, No. 9, September 1999
http://iteslj.org/

---

http://iteslj.org/Articles/Tambini-Aligning.html

Testing Communicative Competence

S. Kathleen, Kitao Doshisha Women's College (Kyoto, Japan)
s.kitao [at] lancaster.ac.uk
Kenji Kitao, Doshisha University (Kyoto, Japan)
k.kitao [at] lancaster.ac.uk

Testing language has traditionally taken the form of testing knowledge about language, usually the testing of knowledge of vocabulary and grammar. However, there is much more to being able to use language than knowledge about it. Dell Hymes proposed the concept of communicative competence. He argued that a speaker can be able to produce grammatical sentences that are completely inappropriate. In communicative competence, he included not only the ability to form correct sentences but to use them at appropriate times. Since Hymes proposed the idea in the early 1970s, it has been expanded considerably, and various types of competencies have been proposed. However, the basic idea of communicative competence remains the ability to use language appropriately, both receptively and productively, in real situations.

The Communicative Approach to Testing

What Communicative Language Tests Measure

Communicative language tests are intended to be a measure of how the testees are able to use language in real life situations. In testing productive skills, emphasis is placed on appropriateness rather than on ability to form grammatically correct sentences. In testing receptive skills, emphasis is placed on understanding the communicative intent of the speaker or writer rather than on picking out specific details. And, in fact, the two are often combined in communicative testing, so that the testee must both comprehend and respond in real time. In real life, the different skills are not often used entirely in isolation. Students in a class may listen to a lecture, but they later need to use information from the lecture in a paper. In taking part in a group discussion, they need to use both listening and speaking skills. Even reading a book for pleasure may be followed by recommending it to a friend and telling the friend why you liked it.

The "communicativeness" of a test might be seen as being on a continuum. Few tests are completely communicative; many tests have some element of communicativeness. For example, a test in which testees listen to an utterance on a tape and then choose from among three choices the most appropriate response is more communicative than one in which the testees answer a question about the meaning of the utterance. However, it is less communicative than one in which the testees are face- to-face with the interlocutor (rather than listening to a tape) and are required to produce an appropriate response.

Tasks

Communicative tests are often very context-specific. A test for testees who are going to British universities as students would be very different from one for testees who are going to their company's branch office in the United States. If at all possible, a communicative language test should be based on a description of the language that the testees need to use. Though communicative testing is not limited to English for Specific Purposes situations, the test should reflect the communicative situation in which the testees are likely to find themselves. In cases where the testees do not have a specific purpose, the language that they are tested on can be directed toward general social situations where they might be in a position to use English.

This basic assumption influences the tasks chosen to test language in communicative situations. A communicative test of listening, then, would test not whether the testee could understand what the utterance, "Would you mind putting the groceries away before you leave" means, but place it in a context and see if the testee can respond appropriately to it.

If students are going to be tested over communicative tasks in an achievement test situation, it is necessary that they be prepared for that kind of test, that is, that the course material cover the sorts of tasks they are being asked to perform. For example, you cannot expect testees to correctly perform such functions as requests and apologies appropriately and evaluate them on it if they have been studying from a structural syllabus. Similarly, if they have not been studying writing business letters, you cannot expect them to write a business letter for a test.

Tests intended to test communicative language are judged, then, on the extent to which they simulate real life communicative situations rather than on how reliable the results are. In fact, there is an almost inevitable loss of reliability as a result of the loss of control in a communicative testing situation. If, for example, a test is intended to test the ability to participate in a group discussion for students who are going to a British university, it is impossible to control what the other participants in the discussion will say, so not every testee will be observed in the same situation, which would be ideal for test reliability. However, according to the basic assumptions of communicative language testing, this is compensated for by the realism of the situation.

Evaluation

There is necessarily a subjective element to the evaluation of communicative tests. Real life situations don't always have objectively right or wrong answers, and so band scales need to be developed to evaluate the results. Each band has a description of the quality (and sometimes quantity) of the receptive or productive performance of the testee.

Examples of Communicative Test Tasks

Speaking/Listening

Information gap. An information gap activity is one in which two or more testees work together, though it is possible for a confederate of the examiner rather than a testee to take one of the parts. Each testee is given certain information but also lacks some necessary information. The task requires the testees to ask for and give information. The task should provide a context in which it is logical for the testees to be sharing information.

The following is an example of an information gap activity.

Student A

You are planning to buy a tape recorder. You don't want to spend more than about 80 pounds, but you think that a tape recorder that costs less than 50 pounds is probably not of good quality. You definitely want a tape recorder with auto reverse, and one with a radio built in would be nice. You have investigated three models of tape recorder and your friend has investigated three models. Get the information from him/her and share your information. You should start the conversation and make the final decision, but you must get his/her opinion, too.

(information about three kinds of tape recorders)

Student B

Your friend is planning to buy a tape recorder, and each of you investigated three types of tape recorder. You think it is best to get a small, light tape recorder. Share your information with your friend, and find out about the three tape recorders that your friend investigated. Let him/her begin the conversation and make the final decision, but don't hesitate to express your opinion.

(information about three kinds of tape recorders)

This kind of task would be evaluated using a system of band scales. The band scales would emphasize the testee's ability to give and receive information, express and elicit opinions, etc. If its intention were communicative, it would probably not emphasize pronunciation, grammatical correctness, etc., except to the extent that these might interfere with communication. The examiner should be an observer and not take part in the activity, since it is difficult to both take part in the activity and evaluate it. Also, the activity should be tape recorded, if possible, so that it could be evaluated later and it does not have to be evaluated in real time.

Role Play. In a role play, the testee is given a situation to play out with another person. The testee is given in advance information about what his/her role is, what specific functions he/she needs to carry out, etc. A role play task would be similar to the above information gap activity, except that it would not involve an information gap. Usually the examiner or a confederate takes one part of the role play.

The following is an example of a role play activity.

Student

You missed class yesterday. Go to the teacher's office and apologize for having missed the class. Ask for the handout from the class. Find out what the homework was.

Examiner

You are a teacher. A student who missed your class yesterday comes to your office. Accept her/his apology, but emphasize the importance of attending classes. You do not have any extra handouts from the class, so suggest that she/he copy one from a friend. Tell her/him what the homework was.

Again, if the intention of this test were to test communicative language, the testee would be assessed on his/her ability to carry out the functions (apologizing, requesting, asking for information, responding to a suggestion, etc.) required by the role.

Testing Reading and Writing

Some tests combine reading and writing in communicative situations. Testees can be given a task in which they are presented with instructions to write a letter, memo, summary, etc., answering certain questions, based on information that they are given.

Letter writing. In many situations, testees might have to write business letters, letters asking for information, etc.

The following is an example of such a task.

Your boss has received a letter from a customer complaining about problems with a coffee maker that he bought six months ago. Your boss has instructed you to check the company policy on returns and repairs and reply to the letter. Read the letter from the customer and the statement of the company policy about returns and repairs below and write a formal business letter to the customer.

(the customer's complaint letter; the company policy)

The letter would be evaluated using a band scale, based on compliance with formal letter writing layout, the content of the letter, inclusion of correct and relevant information, etc.

Summarizing. Testees might be given a long passage--for example, 400 words--and be asked to summarize the main points in less than 100 words. To make this task communicative, the testees should be given realistic reasons for doing such a task. For example, the longer text might be an article that their boss would like to have summarized so that he/she can incorporate the main points into a talk.

The summary would be evaluated, based on the inclusion of the main points of the longer text.

Testing Listening and Writing/Note Taking

Listening and writing may also be tested in combination. In this case, testees are given a listening text and they are instructed to write down certain information from the text. Again, although this is not interactive, it should somehow simulate a situation where information would be written down from a spoken text.

An example of such a test is as follows.

You and two friends would like to see a movie. You call the local multiplex theater. Listen to their recording and fill in the missing information in the chart so that you can discuss it with your friends later.

Theater Number        Movie           Starting Times

  1                 Air Head

  2                                  4:00, 6:00, 8:00

  3                                  4:35, 6:45, 8:55

  4                 Off Track

Summary

Communicative language tests are those which make an effort to test language in a way that reflects the way that language is used in real communication. It is, of course, not always possible to make language tests communicative, but it may often be possible to give them communicative elements. This can have beneficial backwash effects. If students are encouraged to study for more communicative tasks, this can only have a positive effect on their language learning.

---

The Internet TESL Journal, Vol. II, No. 5, May 1996
http://iteslj.org/

---

http://iteslj.org/Articles/Kitao-Testing.html

Testing Grammar

S. Kathleen, Kitao Doshisha Women's College (Kyoto, Japan)
s.kitao [at] lancaster.ac.uk
Kenji Kitao, Doshisha University (Kyoto, Japan)
k.kitao [at] lancaster.ac.uk

The testing of grammar is one of the mainstays of language testing. While such tests test the ability to either recognize or produce correct grammar and usage, they do not test the ability to use the language to express meaning.

However, it can be argued that a basic knowledge of grammar underlies the ability to use language to express meaning, and so grammar tests do have an important part to play in language programs.

Types of Tests

Multiple Choice Tests

Probably the most common way of testing grammatical knowledge is the multiple choice test. These tests have the advantage of being easy to grade and being able to cover a lot of grammatical points quickly.

The most common type of multiple choice grammatical item is one in which the test maker gives the testee a sentence with a blank and four or five choices of a word or phrase which completes the sentence correctly. For example,

Because my mother was sick, I _____ to go home last week.
a) had b) have c) has d) hadn't

To give slightly more context, this type of question sometimes makes use of a short dialogue, with one person saying something and the other person responding.

A way of testing short answers and responses is to give the testees an utterance, and have them decide which of four or five utterances is an appropriate response. This can be either a test of comprehension or a test of grammar. For example,

"I think that tuition is much too high here."
a) I do so. b) Do I so. c) I so do. d) So do I.

Another way to test grammatical knowledge using multiple choice items is to give testees a sentence and ask them to choose which of four or five alternatives has the same meaning.

"The school should have expelled him."
a) The school didn't expel him, which was wrong.
b) The school expelled him, because it was necessary.
c) The school might have expelled him, if it had known.
d) The school will probably expel him in the near future.

Again this is a test of reading comprehension as well as grammar, but in order to understand the meaning of the sentence, the reader does have to understand the grammar.

The test maker must find a balance between giving enough context and giving too much. One way to give more context and make the language more natural is to give the items in the form of a short reading passage rather than individual sentences. This gives the testees more context and, if the passage is chosen carefully, is also much more interesting than reading individual, uncontextualized sentences. However, it may be more difficult to test a range of grammatical points, since the grammatical points are restricted by the content of the passage.

A variation on this idea is to use a piece of prose written by a non-native English speaker. This is particularly useful for making a grammar test for testees who are all of the same language group, since the errors made by the writer can be used as distracters.

Error Correction

Error correction items are also useful for testing grammar. An error correction item is one in which the testee is given a sentence with an error. Four words or phrases in the sentence marked with letters, and the testee needs to decide which of the words or phrases has the error. For example,

(a) Most of students (b) believe that they (c) should be getting better grades (d) than they are.

The teacher may also mix in some sentences that have no errors, and students are required to indicate that there is no error. In addition, the students might be required to correct the error. Errors from students' actual writing are a good source of ideas for this type of exercise.

Items to Test Knowledge of Word/Sentence Order

Other types of items can be used to test testees' knowledge of word order. The traditional way is to present the testee with four alternative word orders. For example,

I wonder how she knows _____.
a) how it costs much.
b) how much it costs.
c) it costs how much.
d) it how much costs.

Another possibility is to give testees the four words and ask them to put the words in order. For example,

I wonder how she knows __________
A. how B. it C. much D. costs
/__/__/__/__/

This can also be done in a way that actually requires the writer to do some writing. For example,

I wonder how she knows ___________________.
how / it /much / costs

Understanding of appropriate sentence order can also be tested in a similar way by giving testees several sentences and asking them to put them in order. This type of test tests knowledge of references, cohesive devices, etc.

Completion Items

Completion items are items in which the testees are asked to fill in blanks in sentences.

For example,

Give the book to ______ woman in the blue dress.

For the purpose of a grammar test, the words which fit in the blanks should be function words, such as articles and prepositions. (Completion items intended to test reading ability or vocabulary knowledge, in contrast, use content words.) The advantage of completion items is that they test production, not just recognition. The disadvantage is that they need to be marked by hand and there will be some cases where the marker needs to make judgements about whether a response is correct. It is not always easy to write items for which there is only one possible answer. Using a piece of continuous prose rather than disconnected sentences is one way of cutting down on possible different interpretations of what goes into a particular blank, but it is probably impossible to entirely eliminate the possibility of different answers.

Also, it is possible to require a phrase instead of a word in each blank. However, while this method presents a more realistic situation, it does become more difficult to mark. While it is probably not realistic for large- scale testing situations, it is something that is useful for classroom teachers who want to help their students develop an ability to produce appropriate grammatical forms in context.

Transformation Items

Another type of grammar item makes use of transformations. In this type of item, testees are given a sentence and the first few words of another sentence to change the original sentence without changing the meaning. For example,

1. Jim hasn't been home in a long time.
It's been a long time _________________________.

2. I don't need to go to the grocery store this week.
It isn't __________________________________________.

3. It is difficult to study when it is so noisy.
Studying ________________________________________.

There are variations on this type of item in which the word which starts the transformed sentence is underlined, or the testee is given one word to use in the new sentence. For example,

I don't need to go to the grocery store this week. (necessary)

Again, this type of test is difficult to grade because the teacher has to be aware of the variety of possible answers. Another problem is that it does not in any way test the testees' knowledge of when each of the possible transformations would be most appropriate. For example, the testee might be perfectly able to transform an active sentence to a passive sentence but not know when to use passive rather than active. However, it is still sometimes a useful test of grammatical knowledge.

Word Changing Items

Another type of item is one in which the testees are given a sentence and a word which they need to fit into the sentence by changing the form of the word. For example,

1. I have never _____ to Australia. (be)
2. I will be with you __________. (moment)

This type of grammar test item tests students' knowledge of different word forms and how they are used in sentences.

Sentence Combining Exercises

Sentence combining exercises can play a part in testing grammar as well as its more traditional use as part of composition testing and training. For example, testees might be instructed to combine the following sentences using a relative pronoun.

I met a man.
The man went to the same high school I did.

I met a man who went to the same high school I did.

Summary

While the testing of grammatical knowledge is limited--it does not necessarily indicate whether the testee can use the grammatical knowledge in a communicative situation--it is sometimes necessary and useful. When considering the testing of grammar, the teacher has to make decisions about such factors as ease of marking, the degree of control, and the degree of realism.

---

The Internet TESL Journal, Vol. II, No. 6, June 1996
http://iteslj.org/

---

http://iteslj.org/Articles/Kitao-TestingGrammar.html

Testing Listening

S. Kathleen, Kitao Doshisha Women's College (Kyoto, Japan)
s.kitao [at] lancaster.ac.uk
http://www.ling.lancs.ac.uk/staff/visitors/kathi/kathi.htm

Kenji Kitao, Doshisha University (Kyoto, Japan)
k.kitao [at] lancaster.ac.uk
http://www.ling.lancs.ac.uk/staff/visitors/kenji/

Testing listening involves a variety of skills. At the lowest level, it involves discrimination among sounds, discrimination among intonation and stress patterns, and comprehension of short and long listening texts. While the first two are part of listening, they are, of course, not sufficient.

Testing Phoneme Discrimination

Sounds are sometimes difficult to discriminate in a language other than one's native language, especially if the sounds are not distinguished in the native language. There are several ways to test phoneme discrimination, that is, ability to tell the difference between different sounds. One way to test phoneme discrimination is to have the testees look at a picture and listen to four words and decide which word is the object in the picture. The words chosen as alternatives should be close to the correct word. However, it is often difficult to find common enough words with similar sounds, and if unfamiliar words are used, they will not make good alternatives. Alternatively, the testees could be presented with four pictures and be asked to choose the picture that matches the word that they hear. Another possibility is to give testees three words and ask them to indicate which two are the same. Finally, testees can listen to a spoken sentence and be asked to identify which one of four similar words were used in the sentence.

Items with full sentences have the drawback that testees can make use of not just phoneme discrimination but also knowledge of grammar and lexical items. If one of the words that the alternatives does not fit grammatically or semantically in the sentence, then testees who realize that have an advantage.

This type of discrimination item is one that can be used for diagnostic purposes to see whether students have particular problems with distinguishing between phonemes. However, it does not give the teacher any information about the testees' ability to comprehend spoken English.

Discriminating Stress and Intonation

The ability to recognize stress can be tested by having testees listen to a sentence that they also have in front of them. Testees are instructed to indicate the word that carries the main stress of the sentence. While recognizing stress patterns is useful in English, the problem with this type of test is that it lacks a context. Testees need to show that they can recognize the difference between "John is going today" and "John is going today," but they do not need to show that they understand that there is a difference in the meaning of the two sentence or what the difference is.

Ability to understand the meaning of difference in intonation can be tested by having the testees listen to a statement and choose from three interpretations of the statement. For example, testees might be given the statement "Vera is a wonderful musician" and be asked to decide whether the speaker is making a straightforward statement, a sarcastic statement, or a question. Since the context is neutral, however, it is sometimes difficult to avoid ambiguity. In real communication, listeners make use of their background knowledge, the context, etc., as well as the intonation to help them interpret the communicative meaning of an utterance.

Understanding Sentences and Dialogues

A teacher can also test the students' understanding of individual sentences and dialogues. In the simplest form, this type of item consists of a single sentence which testees listen to and four written statements from which they choose the one closest in meaning to the original spoken sentence. For example:

Spoken:
I had hoped to visit you while I was in New York.

Written:
A. I was in New York but did not visit you.
B. I will be in New York and hope to visit you.
C. I visited you in New York and hope to again.
D. I am in New York and would like to visit you.

Another type of item is one in which the testees listen to an utterance and choose from among four responses the most appropriate response. In that case, the testees are not being asked directly what the meaning of the utterance is. Rather they are being asked to show that they know what it means by showing that they recognize an appropriate response. This tests both the testees' listening ability and their knowledge of appropriate second pair parts of adjacency pairs.

An example of this type of item is as follows.

Spoken: Would you mind if I visited you next time I came to New York?

Written:
A. Yes, of course. I'd love to visit New York.
B. No, I don't really think that much of New York.
C. Yes, I would. You can come any time.
D. No, not at all. I'd really love to have you.

(At a slightly higher level, both the first statement and the responses can be spoken, but in that case, it might be better to have only three responses, since it would be difficult to keep all four responses in mind.)

In this example, the testees need to know that "Would you mind if I..." is a form used for asking permission, and that a positive response begins with "no (I don't mind)." Because this type of item requires two different types of information, there is a certain amount of controversy about it. Some theorists argue that it is not a good item type, because it requires these two types of knowledge. Testees could possibly understand the utterance perfectly well but not know how to respond to it. Also, since the utterances are presented in isolation and out of context, the situation is not realistic. However, this type of item can be useful if these limitations are kept in mind. It is a more communicative type of task than many listening tasks, so it may have beneficial backwash effects, and it is relatively easy to administer.

Tasks Using Visual Materials

Matching and True/False Tasks

Some types of tasks make use of visual materials along with the spoken material to test listening. The simplest form of this task is to present testees with a picture or other visual information (for example, a chart, graph, etc.) along with spoken true/false statements. Testees look at the visual and decide if the statements are true or false. An alternative is to present testees with a series of similar pictures and to have them match the pictures with spoken sentences describing them. Similarly, the testees can listen to a short dialogue, rather than just a statement, and decide which of the pictures matches the dialogue.

The advantage of using visual materials in this way is that they can be used to test listening alone without involving other skills very much (if the tester considers that an advantage)--though it is impossible to entirely eliminate the use of other skills. However, they do not reflect the sorts of listening done in the real world.

Map Tasks

Another way to use visual materials in testing listening is to use maps. One activity involves having testees listen to directions for how to get somewhere and follow along on the map. They respond by drawing their route or indicating where they would be at the end of the directions. Another possibility is to have testees listen to a conversation referring to various locations on the map and having the testees identify the locations.

Drawing Tasks

Testees can also do drawing tasks according to instructions. For example, they can be given a simple line drawing and be asked to complete it according to certain instructions. The testees can be presented with a diagram of a room with the bed represented by a rectangle and be asked to add a table, a bookcase, a door, etc., in certain locations in the room. It is important to keep such activities simple so that the drawing task itself does not demand too much of testees. The activity is more interesting if it can be done as part of a simple story rather than as a list of statements, though this may depend on the level of the testees. In addition, the students should have a chance to try out this type of activity before having to do it as part of a test.

Tasks Involving Talks and Lectures

For students who will be using English in schools where it is the medium of instruction, there will be situations where they need to listen to lectures or talks in English and take notes and/or answer questions on the lecture or talk. Therefore, listening tests can involve listening to formal or informal talks.

One way of using talks in listening tests is to have the testees listen to the talk and then fill in the blanks in a written summary of the talk. The words chosen from blanks should be ones that the testees cannot figure out from the context of the summary, without listening to the talk, but they should also be ones that are related to the main idea of the talk, so that filling in the blanks does not require remembering small details of the talk.

Another way of doing this involves giving testees questions to answer as they listen to the talk. These questions can be short answer/completion, multiple choice, or true/false. The difficulty with using short answer or completion questions is that they require the testees to both read and write while they are listening, something that can be difficult even for native speakers. Multiple choice questions may require a lot of reading, something that may also be a problem. True/false questions may be the best type for this type of task, since they require relatively little reading compared to multiple choice questions and relatively little writing compared to short answer/completion questions. Also, true/false questions can also have a "no information available" option, meaning that the information required to answer the question is not included in the talk. This decreases the amount of guessing and reduces the element of chance.

Another type of task that can be used is a chart that the testees fill in while listening to the talk. Answers in some of the blanks in the chart may be filled in for the testees. Testees should be given the chart in advance of hearing the talk, and be given time to familiarize themselves with it and make sure they understand what is expected of them.

Finally, testees can be allowed to take notes while listening to the talk, and then use the notes to answer questions after the talk is over. Depending on the length and complexity of the talk, testees might be given a list of the major topics included in the talk in order to help them in taking notes.

Summary

There are a number of ways to test listening, but, particularly when testees' listening proficiency gets more advanced, testing listening becomes more complicated. It becomes more difficult to separate listening from other skills, and combining skills can put great demands on the testee. In addition, some ways of testing listening do not reflect real-world listening tasks. In choosing tasks for listening, the teacher should be aware of these problems.

---

The Internet TESL Journal, Vol. II, No. 7, July 1996
http://iteslj.org/

---

http://iteslj.org/Articles/Kitao-TestingListening.html

A Review of TOEIC

Susan Gilfert
sugi_ngo [at] excite.com
Aichi Prefectural University, British-American Studies Dept.

What is a TOEIC? What does it test? Is it useful? How is it used? What do the scores mean? Are the scores comparable to TOEFL in any way? These and other questions will be answered in this paper. Knowledge of the background of this test will help users understand it, so a short history will be provided; followed by a general structure discussion with examples of question types, subsection by subsection. A brief discussion of how results of the exam are used or misused finishes this review.

History of TOEIC

The TOEIC came from the Japanese Ministry of International Trade and Industry requests to the Educational Testing Service in the middle 1970's. It is "designed to measure the English-language listening comprehension (LC) and reading (R) skills of individuals whose native language is not English. The TOEIC is used primarily by corporate clients, worldwide." (Wilson, 1989). Most examinees are in their mid-twenties to late forties, and working for a corporation. However, TOEIC test-takers have recently included many university graduating seniors, because corporations are requiring TOEIC scores for new employees more and more often. From its beginning nearly 20 years ago, the use of TOEIC has spread from Japan throughout Asia, and it is becoming more frequently used throughout Europe and South America.

Structure

The TOEIC is a multiple-choice instrument designed to measure an examinee's receptive English skills, and is increasingly becoming considered a reasonable predictor of these skills. The general register of the TOEIC is "real-life, business-type English.." The TOEIC was created by Educational Testing Services in Princeton, New Jersey, but is now entirely owned and operated by the Japanese TOEIC office in Tokyo. The structures of the TOEIC is not radically different from the TOEFL (see Figure 1). The topic treated in each test, however, is different.

In the listening subtest, visual stimuli (in the form of photographs) is first used by ETS for English-language testing purposes. Another relatively novel section is the response section, where no stimuli other than the voices on the tape are given. No printed information whatsoever is given in this section. Other sections include listening to conversations and listening to short talks, like many other English-language tests. The type of questions asked within the Listening comprehension section (main idea, vocabulary, idioms, minimal pairs, or inference) is similar to other English-language tests.

In the reading comprehension subtest, two subsections evaluate the testee's ability to use English grammar in a relatively formal manner. The TOEIC uses business letters, short news items, and advertisements as stimuli in the readings. However, the type of questions asked in the Reading comprehension section (main idea/ topic, inference, attitude/tone, vocabulary, idioms, or details/ application within the passage) is similar to other English- language tests.

Specific Comparison of Listening Comprehension

Section 1 of the TOEIC tests the ability of the examinee to recognize vocabulary in the context of the photo prompt (see Figure 2). The test-takers see a photo and hear four sentences describing the photo. Examinees tend to feel that the photo prompt, providing visual context, is reassuring, even though both the question and possible answers are only heard, not printed.

Section 2 of the TOEIC assesses the examinee's ability to listen to a prompt and choose the appropriate response (see Figure 3). Some Japanese examinees have commented that this section seems to be mostly a structure test, listening for the grammatically correct response. Most examinees feel that this part of the TOEIC the most difficult part of the listening component since both the prompt and the possible answers are only heard, not printed.

Section 3 of the TOEIC is listening to short conversations (see Figure 4). This section follows the pattern of Speaker A:, Speaker B:, Speaker A; and the question and four possible answers are printed in the test booklet. Examinees tend to feel that the TOEIC is relatively easy to understand in this section because both the question and possible answers are printed in the test book, which provides examinees more context into which to fit the conversation. There is only one question per conversation.

Section 4 of the TOEIC is the longer conversations section (see Figure 5). The TOEIC "short talks" subtest tends to have short talks (1-1.5 minutes) and asks 3-5 questions per talk. The content of the TOEIC are typically extended conversations (5 or 6 extended exchanges) between two people talking about office matters, or single speakers giving a news report or other information. Idioms and vocabulary in context are tested extensively in this section. Examinees tend to feel that the TOEIC material is comparatively less difficult than other tests, since both the amount of spoken language and the number of questions are limited.

General Comments on the Listening Section

There are 100 questions on tape in the TOEIC; testing time is about 50 minutes. Timing of questions is roughly 30 seconds per question. There is a "thinking gap" of about 10 seconds per question on the tape. Vocabulary and idioms in context are evaluated throughout the test, as well as grammar. The general register of the TOEIC listening subtest is "business," with a high frequency of idioms being spoken and relatively few polysyllabic words.

Reading Comprehension

The next two subsections of the TOEIC, Incomplete Sentences and Error Recognition, assess the examinee's knowledge of English structure, or grammar. These TOEIC subtests are supposed to "measure ability to recognize language that is appropriate for standard written English," (ETS, 1993).

The example in Figure 6 tests demonstrative pronoun usage. The example in Figure 7tests word order. This section of the TOEIC is not unusual; in fact, it is practically the same as the comparative subsections of the TOEFL.

The example in Figure 8 is a news report. The types of reading comprehension questions are not unusual: main idea, details, inference, and/or author's attitude. The TOEIC tends to have 3-4 questions per passage, and about 6 or 7 reading passages on each test.

There are 100 questions in the Reading Comprehension section; 60 questions in the Grammar subtest and 40 questions in the Reading subtest, with a total time of 75 minutes. The Grammar and Reading Comprehension sections are timed together. Examinees should allow about 25 minutes (or less) for the Grammar questions. If an examinee can quickly (and accurately) go through the Grammar section, then more time is left for the Reading Comprehension questions.

Purposes of the Tests

The stated purposes of the TOEIC is to show the examinees some general measure of their English ability. However, some institutions misuse the tests for purposes which should not be measured on these tests.

The TOEIC is correctly used to assess a examinees' overall English proficiency in a business context. TOEIC scores are increasingly being required by corporate employers of either entering employees or of employees who are being considered for promotion and/or overseas assignments. Employers use TOEIC scores as a screening device, hiring only those who meet a certain pre-determined TOEIC score (see Figure 9). As a result of this practice, Japanese colleges, universities, and tertiary-level vocational schools are now offering TOEIC-preparation courses in greater numbers than five years ago. TOEIC-preparation courses have already been offered by language schools throughout Japan for many years now. Some corporate employers use the TOEIC incorrectly, by requiring their domestic employees (who do not use English on a regular basis) to obtain a certain score for promotion or raises.

Score Usage

Many researchers and students of testing believe that the TOEIC shows the differences between low-beginner-to-high-intermediate levels very well. A TOEIC score of 450 is frequently considered acceptable for hiring practices, with the understanding that the employee will continue English studies. A TOEIC score of 600 is frequently considered the minimum acceptable for working overseas. Domestically-based engineers who have a TOEIC score of 500 are considered reasonably proficient in English . If the same engineer is being considered for a posting overseas, he or she must usually try for a TOEIC score of about 625. A domestically-based desk-worker with a TOEIC score of 600 is considered reasonably proficient in English. For the same desk-worker to go overseas, she or he must usually have a TOEIC score of 685.

What do the Scores Mean?

The TOEIC office in Tokyo, Japan has published a comparison between the Oral Proficiency Index (the OPI is used by the U.S. Foreign Service), TOEFL scores, TOEIC scores, other tests, and the Japanese Eiken1. All these tests assess English reading, listening, and grammar proficiency. The OPI and Eiken series further test speaking ability. The Oral Proficiency Index is considered one of the best tests since it provides a means of testing the examinees' productive language skills, as well as their receptive language skills. However, due to time and cost considerations, the OPI is an impractical test to administer for large numbers of people. The Educational Testing Service also administers the Test of Spoken English (TSE) and the Test of Written English (TWE). Of the 12 "official" (International) TOEFL tests administered every year, 5 include a TWE in addition to the regular TOEFL. The TSE has its own testing schedule, since it requires making an audio tape. Many schools do not require these additional tests for much of their admissions procedures, although non-native English speaking graduate students who wish to become Graduate Teaching Assistants are increasingly required to pass the TSE in order to get their assistantship.

Both TOEFL and TOEIC test receptive skills (listening and reading) rather than productive skills (speaking and writing). It is possible for students to score very high on the TOEFL, but not be able to use oral or written English in context . Many examinees become expert in taking language tests, but do not learn how to use the language. Therefore, the authors maintain that TOEFL and TOEIC tests operate in an "artificial reality." The tests, when used alone, are valid and reliable in themselves, but not in a larger sense. Examinees who score well on these tests may have self-confidence in the language classroom, but using their language skills in the real world may be quite a different thing.

In theory, an examinee with a score of 650 would be expected to operate in a English-speaking business context better than a examinee with a score of 600. In the real world, examinees will be reading and generating faxes and reports, listening to and making presentations, and using the telephone. Examinees who excel in taking paper tests, yet are unable to use their language productively, will be at a loss in the real world.

Comparing TOEFL and TOEIC Scores

What is the difference between the TOEFL and the TOEIC? Can they be compared?

The scoring system is different and the number of questions is different, as is the amount of time needed to take each test. The register is also different ("Academic English for TOEFL and "Business English" for TOEIC). The reasons for taking each test (the examinees' motivation) can be different (except perhaps in the area of securing employment), and the ways of using the results of the tests are different. The vocabulary in the two tests has areas of similarity, but there are some noticeable differences due to register of English tested. Many examinees feel that the TOEIC is easier than the TOEFL. Many students of testing consider that the TOEFL is a more accurate discriminator for higher-level examinees, and the TOEIC is a more accurate lower-level discriminator.

The tests were both created by Educational Testing Service, and test American English. ETS has calculated a number of reliability and validity checks on both tests, so they are both considered accurate and useful when used within the guidelines published by ETS. The grammar subtests of both tests are quite similar and the types of questions asked in the Reading Comprehension subtest (main idea, details, inference, and/or author's attitude) are similar.

In short, with proper understanding of the TOEIC, it can be useful, but it must be used properly, with full knowledge of its limitations.

Figures

Figure 1: General Comparison of Tests

TOEFL                             TOEIC
3 major subtests;                 2 major subtests;
  5 subsections                     7 subsections
150 questions                     200 questions
scaled score ranges from 200      scaled score ranges from 10      to 990      
                   to 990
examinees tend to be students     examinees tend to be    
  (18-25 yrs)                       corporate-level employees
                                    (25-50 yrs old)
results tend to determine         results tend to determine
  schools to be attended and        overseas postings and
  academically related matters      other business related
                                    matters


1. Listening Comprehension      1.  Listening Comprehension
                                   I.   One photograph, spoken
                                        sentences (20 qs)
                                   II.  Spoken utterances,
                                        spoken response (30 qs)
I. Short conversation (25 qs)      III. Short conversation (30 qs)
II. Short talks (25 qs)            IV. Short talks (20 qs)


2. Structure & Written          2.  Reading Comprehension
   Expression
III. Incomplete sentences          V.  Incomplete sentences
      (15 qs)                           (40 qs)
IV.  Error recognition (25 qs)     VI. Error recognition (20 qs)


3.  Reading Comprehension
V. Reading comprehension           VII. Reading comprehension       (30 qs)    
                  (40 qs)

Figure 2: Question Example: Listening Comprehension

I. One picture, spoken sentences (Gilfert & Kim, 1996):

Seen:
a photo of two men talking across a table. An unused computer is in the background.

Heard:
(A) The two men are computing.
(B) The computer is having a meeting with the men.
(C) The two men are talking.
(D) One man is buying a computer.

(C) is the correct answer, since it is closest in meaning to what is shown in the photo.

Figure 3: Question Example: Listening Comprehension

II. Spoken utterances, spoken response (Gilfert & Kim, 1996)

Heard:
Hello, I'm John.

Heard:
(A) Hi, John. How are you?
(B) Who's John?
(C) Good-bye, see you later.

The correct response is (A), since it is the most likely response to this greeting.

Figure 4: Question Example: Listening Comprehension

III. Short Conversation, Four printed answers (Gilfert & Kim, 1996)

Heard:
A: May I help you?
B: Yes, do you have this shirt in size 12?
A: Certainly. I'll get one for you.

Read:
Where is this conversation most likely taking place?
(A) in a hotel
(B) in a department store
(C) in a post office
(D) in an airport

The correct response is (B), since the conversation appears to be happening between a sales clerk and a customer

Figure 5: Question Example: Listening Comprehension

IV. Short Talks: (Gilfert & Kim, 1996)

Heard:
Sunshine is forecast for today after two damp days. Westerly winds will freshen by afternoon and chilly air will be transported across the metropolitan area. Clouds will overtake clear skies by morning. Chance of rain is thirty percent today and forty percent this evening. Highs in the low sixties; that's in the high teens Celsius; and lows in the mid forties Fahrenheit or just under ten degrees Celsius today.

Read:
1. How was the weather earlier this week?
(A) Sunny
(B) Cool and dry
(C) Damp
(D) Chilly

2. What kind of weather is expected tomorrow?
(A) Cool and cloudy
(B) Sunny and dry
(C) Damp and windy
(D) Cold and sunny

3. What is a likely high temperature today?
(A) 10¼ C
(B) 17¼ C
(C) 42¼ C
(D) 63¼ C

For question 1, (C) is the best answer. The announcer notes that the last two days have been damp.

For question 2, (A) is the best answer. The announcer notes that "chilly air" is expected overnight.

For question 3, (B) is the best answer. It is within the range of the "high teens."

Figure 6: Question Example: Reading Comprehension

V. Incomplete Sentences: (Gilfert & Kim, 1995)

Read:
_____ girl over there is my sister.
(A) This (C) Those (B) These (D) That

(D) is the answer that is grammatically correct.

Figure 7: Question Example: Reading Comprehension

VI. Error Recognition: (Gilfert & Kim, 1996)

Read:
In today's (A) class middle, both (B) parents have to work in order (C) to pay all (D)their bills.

(A) is an error in word order, making it the correct answer.

Figure 8: Question Example: Reading Comprehension

VII. Reading Comprehension (Gilfert & Kim, 1996)

Questions 1-3 refer to the following news report

LOS ANGELES (NNN)--Rains, accompanied by high winds, closed a number of roads and schools on Monday, but the storm was welcomed here in southern California because it brought some relief from a fifth straight year of drought. Fire cautions remain high, however.

1. Why might the rain close schools?
(A) California schoolchildren want to go outside to study rain and flooding.
(B) California parents don't want their children to get wet.
(C) The roads were closed because of flooding and the school buses couldn't drive through.
(D) The rains and winds damaged the school buildings.

2. What has the weather been like in California for the last five years?
(A) wet (B) windy (C) dry (D) mild

3. Why might people welcome rain?
(A) California people enjoy walking in the rain.
(B) California roads need relief from the sun.
(C) California schoolchildren want to study rain.
(D) There has been no rain for five years.

The correct answer for Question 1 is (C). This is the only possibility mentioned specifically in the report.

The correct answer for Question 2 is (C). "Drought" means a time of no rain, and therefore very dry.

The correct answer for Question 3 is (D). See Question 2.

Figure 9: TOEIC Score Level (Matsumoto, p. 9-11)

This is an estimated chart, using TOEIC scores and company recommendations from various sources in the Tokyo area.

TOEIC Score - Level - Comments

990-860 A level

Ability to communicate on a variety of topics, both personal and professional, with native speakers. Vocabulary, grammar and pronunciation are reasonably accurate and understandable. Senior office staff posted overseas.

860-730 B level

Ability to communicate with success in various situations where the testee has some expertise. Vocabulary, grammar &c. may not always be the best choice or completely accurate, but testee will be understood. Junior office staff posted overseas.

730-470 C level

Ability to communicate about everyday matters and daily news, although limited in business- matter communication. Fluency is not rapid but not a major hindrance. Senior domestic office staff dealing in English matters. Engineers posted overseas.

470-220 D level

Ability to communicate at the lowest level on everyday matters. Fluency is slow; changes conv. subjects with difficulty. Uses simple grammar structures & vocabulary. Junior domestic office staff dealing with English matters. Engineers dealing domestically with English matters.

220-10 E level

Ability to communicate in English is very limited.

References

• ETS. (1993). TOEFL 1993-4 Bulletin of Information. Princeton, NJ: Educational Testing Service
• Gilfert, S. and Kim, V. (1990). TOEFL Strategies. Nagoya, Japan: Kawai Juku Press.
• Gilfert, S. and Kim, V. (1996). TOEIC Strategies, Tokyo: Macmillan Japan K.K.
• Heaton, J.B. (1990). Classroom Testing. London: Longman Group UK.
• Hinkel, E., Director of English Language and American Culture Program, Xavier University, Cincinnati Ohio, personal communication.
• Matsumoto, S and West, D. (1990). TOEIC Š®‘S–ÌŽŽ [Kanzen Moshi--Perfect Model Test]. Tokyo: ARUKU Co. Ltd.
• Wilson, K. M. (1989). TOEIC Research Report #1. Princeton, NJ: Educational Testing Service
• Woodford, P. (1982). "The Test of English for International Communication." In C. Brumfit, (Ed.) English for international communication. Oxford: Pergamon Press.

Notes

1. The Japanese Eiken is a 6-part series of non-standardized tests produced by the Society for Testing English Proficiency (STEP), published in Tokyo, Japan. The STEP test levels are level 4 (low beginning), 3 (high beginning), pre-2 (low intermediate), 2 (high intermediate), pre-1 (low advanced) and 1 (high advanced). STEP is offered by pre-registration at a relatively low cost (´2,500 -´3,000 per person per test) twice a year in Japan. There is no limit on how often a person can take the test.

In contrast, the International TOEFL is available, by pre-registration only, 12 times a year, anywhere in the world. The cost of taking a TOEFL is U.S. $42 on a Friday administration or $35 on a Saturday administration, payable in U.S. (or Canadian) funds only. Five times a year, the TWE is part of the TOEFL at no extra cost. The International TOEFL is the "official" TOEFL; the scores are sent at the examinee's request to examinee-selected schools. Five of the International TOEFL administrations are Disclosed. Examinees can choose to give a self-addressed envelope and postage for 43 grams from the U.S. to the test administrators. The examinee's test booklet will be mailed a week or so later to the examinee.

There are other versions of the TOEFL, called Institutional TOEFL. Institutions may choose to purchase and offer the TOEFL to their students or employees at any time they wish, as long as that date does not conflict with an International TOEFL. These scores are used within the institution which offers the TOEFL; the scores do not leave the institution for use in applying to schools in the U.S. or Canada.

The TSE is offered at other times, for U.S. $80 (for TSE-A) or $110 (for TSE-P).

The TOEIC is offered, by pre-registration only, 6 times a year for ´6,500 per test, payable in Japanese yen or the equivalent in local funds. Institutional TOEIC administrations can be arranged at other times by institutions, as in TOEFL above.

This information may change. Please check the latest bulletin of information for the latest prices, test dates and availability. Bulletins are available free of charge at many bookstores and university or college campuses in the U.S. In Japan, there may be a small charge for test information at bookstores.

2. In Japan, engineers are generally considered to need fewer language skills than other workers, since engineers tend to communicate in formulae and numbers, transcending linguistic limitations. This may not always be true; in fact, limited communication ability may cause progress to be slowed or to come to a stop.

---

The Internet TESL Journal, Vol. II, No. 8, July 1996
http://iteslj.org/

---

http://iteslj.org/Articles/Gilfert-TOEIC.html