part2 of method

In order to answer research question D, we needed to have several tests because we wanted to test different types of word knowledge. We selected three tests, which were: 1) a word-form recognition test; 2) a multiple-choice (prompted recognition) test; and, 3) a meaning by translation (unprompted recognition) test. The three tests were extensively piloted with a group of eight subjects of similar ability and background. These subjects were not part of the main study. The aim of the piloting was to confirm that words were easy pronounceable by Japanese subjects, that the tests contained enough words and the text was not too long and could be read in about one hour at a reasonable reading speed. The word-form recognition test required the subjects to circle any words they recognized from the text. The test contained the twenty-five substitute words that they had met in the text, plus an additional seventeen distractors to investigate the level of guessing. Piloting had shown that seventeen distractors were enough to provide us with a reliable check for errant guessing. Data
RFL 15.2 – At what rate do learners learn and retain new vocabulary from reading a graded reader?
http://nflrc.hawaii.edu/rfl
139 were collected for the number of correct recognitions and the number of false recognitions (i.e., when they circled a substitute word that did not appear in the text.). The test is in Appendix A. The multiple-choice recognition test is a standard prompted recognition four-choice test with the correct meaning and three distractors. An I do not know option was added to allow subjects to indicate when they did not know an item so as to reduce the effect of guessing. The subjects were asked to circle the words they thought were nearest to these words in meaning. These choices are the same part of speech. For example, the substitute word mand means face. Face is a concrete noun, so the four choices are concrete nouns. Care was taken to ensure that the distractors were from different semantic sets to allow for small amounts of knowledge to be demonstrated. The test appears in Appendix B. The meaning (translation) test presented the twenty-five substitute words in a list. The subjects were asked "What do these words mean? Write the meaning in Japanese." Second and third answers options were given to give them a chance to provide plausible alternative answers to discover whether the subjects could provide a near rather than an exact synonym. The test appears in Appendix C. The tests were given in a strict order. The word-form recognition test, (the one requiring the least amount of word knowledge) was given first. If the subjects had taken the meaning (translation) test after the multiple-choice recognition test they would have been able to remember some meanings and taken this information to the translation test. Therefore the meaning (translation) test was given second, and the multiple-choice recognition test was given last. As this research tried to establish how much vocabulary was learned from natural reading, it was decided not to test the words in context. This was because if context had been used (whether contrived, or from the text itself), the subjects may have been able guess what the word meant by working it out at test time. Therefore, we would not know if the word was learned from reading, or guessed at test time.
Procedure The full text ofA Little Princess, with the substitute words, was printed for each subject and was put into book form along with a test booklet. The subjects were asked to "read this story as usual and enjoy it." They were told there would be a test after their reading, but no detailed information was given about the test. The subjects were not told that there would be any unfamiliar words in the text. During the test, the researchers checked that the subjects were not turning to look at the test words, and ensured that the subjects did not look at the text when taking the tests. Subjects were not allowed to use a dictionary. As soon as each subject finished the text, she was required to take the tests in the strict order outlined above. The time taken to read the book was collected for each subject. After reading, each subject was asked if she thought the text was a) easy to read, b) a little difficult, or c) very difficult. Each subject was also interviewed informally after the reading. During the reading, there were some
RFL 15.2 – At what rate do learners learn and retain new vocabulary from reading a graded reader?
http://nflrc.hawaii.edu/rfl
140 comments from the subjects, such as "there are a lot of unknown words," but they were only told "please enjoy it." Seven to ten days after the first test administrations, the subjects took the tests a second time. The aim of this was to determine the rate of forgetting that had occurred from the learning from context. Approximately three months after the reading, the subjects were tested again. This allowed us to test research question C. Both of these test administrations were unannounced. The subjects took the same tests without reading the story again and they never met the words again. At each test administration, the order of presentation of the test items on each test was changed, but the order in which the tests were given was retained. Table 3 has a summary of the types of tests given at each administration. Table 3: Summary of the procedure
Test Time Test Type (in order of presentation) Immediately after reading (n = 15) Reading the text 1. word-form recognition test 2. meaning (translation) test 3. multiple-choice recognition test One week later (n = 15) 1. word-form recognition test 2. meaning (translation) test 3. multiple-choice recognition test 3 months later (n = 14) 1. word-form recognition test 2. meaning (translation) test 3. multiple-choice recognition test
Marking The correct answers on the word-form recognition test were counted as one point. False recognitions were also counted to determine how much guessing was happening. One point was awarded for correct answers and one point was awarded for each of the words "selected in error". These are the "correct" and the "selected in error" scores found in Table 6. On the meaning (translation) test, correct answers were given one point and a word with a similar meaning was given a half point. For example, if the test word's correct answer was shame, one point was given, but if the subject supplied pity, or regrettable, because it is a near synonym, a half point was awarded. A total of only 15 response items were given a half point for the 15 subjects over the three tests over the three test administrations and would not have significantly affected the results overall. On the multiple-choice recognition test only correct scores were counted.