& f Digitized by the Internet Archive in 2019 with funding from Princeton Theological Seminary Library / https://archive.org/details/tentativestandarOOmean Vol. XXX No. Whole No. 134 1921 ! JUt 27 PSYCHOLOGICAL REVIEW PUBLICATIONS Psychological Monographs EDITED BY JAMES ROWLAND ANGELL, 522 Fifth Avenue, New York. HOWARD C. WARREN, Princeton University ( Review) JOHN B. WATSON, Johns Hopkins University (/. of Exp. Psychol .) SHEPHERD I. FRANZ, Govt. Hosp. for Insane ( Bulletin ) and MADISON BENTLEY, University of Illinois (Index) A Tentative Standardization of a Hard Opposites Test BY ✓ MARIE HACKL MEANS, Ph.D. PSYCHOLOGICAL REVIEW COMPANY PRINCETON, N. J. axd LANCASTER, PA. Agents: G. E. STECHERT & CO., London (2 Star Yard, Carey St., W. C.) Paris (16, rue de Conde) Grateful acknowledgment is due Dr. Edward K. Strong who suggested the problem and Dr. Joseph Peterson under whose guidance and help the work was completed. The writer also wishes to thank those students of Peabody College who acted as subjects, as well as the following men and women who so kindly gave of their time to make the completion of this research pos¬ sible : Dr. Harvey Carr, Miss Cora Bratton, Dr. George Arps, Miss Mary Small, Dr. Walter Hunter, Dr. Florence Richard¬ son, Dr. Benjamin Simpson, Dr. Norman Triplett, Dr. Franklin Smith, Dr. Homer Bean, Dr. E. A. Gamble, Mr. E. A. Kirk¬ patrick, Dr. C. A. Ruckmick, Dr. M. R. Trabue, Dr. George W. Camp, Dr. W. B. Pillsbury. It would be most ungrateful for the writer not to acknowledge the help and encouragement received from her mother. I CONTENTS Section Page I The Problem and its Importance. i II Results of Previous Investigators. 2 III Previous Efforts toward Standardization. 16 IV The Selection of the Stimuli for the Present Investi¬ gation . 19 V The Choice of the Acceptable Responses. 24 VI The Computation of the Relative Difficulty of the Stimuli . 33 VII Conversion into a Group Test. 48 VIII Establishment of Norms. 51 IX Interpretation of the Results. 57 X Suggestions and Directions for the Use of the Oppo¬ sites Test. 59 XI Summary and Conclusions. 62 XII Appendix . 64 Bibliography . 65 A TENTATIVE STANDARDIZATION OF A HARD OPPOSITES TEST I. The Problem and its Importance There have been many attempts to apply mental tests to college students, but the results have not been on the whole satisfactory. The very homogeneity of the college group, brought about by the natural process of the elimination of the unfit, should have warned us not to expect results comparable to those obtained from children and unselected adults. The minute individual differences of degrees of accuracy or rate of performance can be detected only by the use of a most delicate scale. Therefore to raise our correlations of mental tests and esti¬ mated intelligence it is necessary that greater attention should be paid to the selection of the tests themselves. Where the higher functions are involved it is better to avoid tests of co-ordination and sense discrimination as well as those tests, success in which is largely dependent upon speed of performance rather than ac¬ curacy. Unfortunately the humbler task of standardization for convenient use has been neglected and until investigators have at their disposal a number of such tests, our attitude toward the validity of mental testing of college students should be highly tentative. This present investigation was attempted in the hope of con¬ tributing a small amount toward the preparation of some ade¬ quately standardized tests. To evolve out of a test in common use a vastly more difficult test, to determine the relative dif¬ ficulty of its parts, to standardize the responses which are accept¬ able, and to set up workable norms with which comparisons could be made—this specifically was our aim. For this purpose we have chosen an opposites test. No effort has been made to obtain correlations with other tests and no claim is made that the test, as it now stands, has mentally diagnostic values of high importance for the individual. » II. The Results Obtained by Previous Investigators The diversified use of the opposites test, with its different methods of presenting and scoring, has yielded results which, while not altogether comparable, are not without interest. Dr. Bonser 1 in an investigation of the reasoning ability of children in the upper division of the fourth, fifth, and sixth grades of the public schools of Passaic, New Jersey, tested three hundred and eighty-five boys and three hundred and seventy-two girls within the period of February 13 to 27, 1906. The tests employed were designed to exercise the most fundamental phases of reasoning ability, namely, mathematical judgment, controlled association, selective judgment, and the analytic and synthetic thinking necessary for the intellectual interpretation of litera¬ ture. For controlled association, three types of tests were used. First, in two sets of ten sentences presented, a significant word was omitted from each sentence, which was to be filled in by the pupils. Second, two other sets of ten sentences were given, in each of which two significant words were placed one above the other. The pupils were instructed to draw a line through the wrong word, leaving the sentence so it would read correctly. Third, three sets of twenty words each were given to the pupils with the instructions that they were to write beside each re¬ spective word a word just its opposite in meaning. For selective judgment two types of tests were used. The first consisted of two sets of two series each of ten reasons why some given fact is true, some of which are correct, the others incorrect. The pupils were to check the former. Second, there were given two sets of three series each, of five definitions of a given thing or term, some of which were correct, others incor- 1 Bonser, Frederick G., “The Reasoning Ability of Children of the Fourth, Fifth, and Sixth School Grades,” Col. Univ. Coni. Ed., 37, 1906, 1-101. STANDARDIZATION OF HARD OPPOSITES TEST 3 rect or irrelevant. The pupils were to choose the right definition. In the table below these tests are indicated as V and VI respect¬ ively. For literary interpretation the pupils were asked to write the meaning of two stanzas of poetry. Below is shown the coefficients of correlation obtained by cor¬ relating each test with the combined score of all the tests. TABLE i Bonser’s Correlations of Each Test with the Totals for All Tests in Reasoning Ability. Opposites and Combined Score .85 Selective Judgment and Combined Score (V). 73 Mathematical Judgment and Combined Score.59 Selective Judgment and Combined Score (VI).58 Controlled Association and Combined Score.55 Interpreting Poems and Combined Score .37 Spelling and Combined Score.22 Dr. Bonser has arranged for comparison separate tables for the younger and the older group in each grade. He finds the younger group superior in the opposites test for every grade, but this superiority diminishes as the years in school increase. This would indicate, he thinks, that the test is one which reveals some sort of native ability, which is concealed in the upper grades where the test is relatively simple. Miss Norsworthy 2 in a study of the comparison of defective and normal children found that the defectives were farthest removed from normal children in ability to deal with abstract data. Of the 137 cases ranging in age from eight years up, none of the defectives surpassed the median score for normal children in the opposites tests. This is shown in the following table. Woolley and Fischer 3 in their work in connection with the in¬ dustrially employed children in Cincinnati obtained results by the use of opposites tests which are highly significant. Over eight hundred children fourteen years of age were given the following 2 Norsworthy, Naomi, “The Psychology of Mentally Deficient Children,” New York, Columbia University, 1906. 3 Woolley, Helen Thompson, and Fischer, Charlotte Rust, “Mental and Physical Measurements of Working Children,” Psychol. Monog., 1914, XVIII, 77, 213-241. 4 MARIE HACKL MEANS TABLE II Norsworthy’s Percentages of Children Scoring above 2 P. E., i P. E and Median for Normal Children in Series of Tests. % above— 1 % above —2 % above Med. P. E. (or P. E. (or for ordinary lowest 25% lowest 9% children of ordinary of ordinary children) children) Height . 45 61 77 Weight . 44 66 77 Pulse . 49 69 86 Temperature . 26 59 77 Weight Test. 18 28 39 A-T Test. 1 14 28 Memory of Unrelated Words 6 18 27 Dictation . 10 10 21 Memory of Unrelated Words 5 19 30 Part-Whole Test . 9 17 27 Genus-Species Test . 9 16 17 First Opposite Test. 0 0.9 ** a Second Opposite Test. 0 1 7 physical and mental tests upon their entrance into the industrial world: Height, Weight, Visual Acuity, Auditory Acuity, Vital Capacity, Strength of Hand, Steadiness of Hand, Tapping, Card Sorting, Cancellation Test, Memory (Digits), Substitution, Completion of Sentences, Association by Opposites, and Puzzle Box Test. A year later six hundred and seventy-nine of these boys and girls were re-tested. In giving the opposites test one of the eight lists of twenty words printed one under another was presented to the subject who was requested to write beside each word another word op¬ posite in meaning. The time was recorded for the total list but the scores were based on the percentage of accuracy alone. Cred¬ its of one, one-half, or zero were assigned to the responses given. Misspelled words were given full credit but adjectives written in place of adverbs received only half credit. A positive correlation was obtained between the school grade completed at fourteen years of age and the ranking in every one of the mental tests. The general order is as follows: Memory, STANDARDIZATION OF HARD OPPOSITES TEST 5 Association by Opposites, Sentence Test, Substitution Test, Can¬ cellation, and Puzzle Box. The public school children were found to be superior to the parochial school group, and their superiority was most decided in opposites and in the puzzle box tests, the two tests which, accord¬ ing to the investigators, are farthest removed from a relationship with school drill. ‘‘There is a marked and consistent positive correlation with school grade in this test (opposites) for both sexes and at both ages. The differences from grade to grade are so large and so consistent that their significance cannot be questioned." 4 While director of the Department of Psychology at the New York State Reformatory for Women, at Bedford Hills, Dr. Weidensall 5 began a series of experiments upon the women de¬ tained in that institution. It was hoped that a number of tests might be found which would prove prophetic of the convicted woman’s reformability and would thus eliminate such cases which, because of inability to learn, consumed a disproportionate amount of time. The major portion of the monograph deals with the records of the criminal woman as compared with the norms of normal working girls fourteen and fifteen years of age. The norms were being formulated at the time by Dr. Woolley 6 in her work as director of the Bureau of Vocational Guidance, Cincinnati, Ohio. Since comparisons were to be made, the tests which were being used by Dr. Woolley were adopted by Dr. Weidensall and given in approximately the same manner. The women tested were admitted between the first of January and the end of October, 1913. Two hundred and eight were committed to Bedford during that period, but of that number only a hundred were tested. Because of lack of facility in the English language of twelve foreigners, the percentile tables and curves are based on eighty-eight records. All tests were given 4 Woolley, Helen Thompson, and Fischer, Charlotte Rust, Op. cit., 222. 5 Weidensall, Jean, “The Mentality of the Criminal Woman,” Baltimore, Warwick & York, 1916, 3-266. 6 Woolley, Helen Thompson, and Fischer, Charlotte Rust, Op. cit. 6 MARIE HACKL MEANS individually and during the first two weeks of confinement while the subject was still in quarantine. The method followed in connection with the opposites test was to present a list of twenty words and to record the total time required for the subject to write the opposites. Results were based on the percent of accuracy, as it was found that the cor¬ relation between rank in time and accuracy in this test was as high as .83, P. E. .029. All identical accuracy scores were arranged in point of time of performance and the rankings correlated with the native ability of the group as estimated by the director of the Industrial School of the Reformatory after she had worked with these women for from eighteen months to two years. This correlation was -f- .79, P. E. .026, and was obtained by the formula 6 2 d 2 n (n 2 — 1) This correlation, which was higher than that for any other test, would probably have been even higher had it not been necessary for the director to base her judgment of ten of the women on the reports of matrons. It was the difference in rank accorded these ten which was responsible for some of the largest variations from the rank of the test. It is interesting to note that of the Bedford eighty-eight but thirty-nine percent attain or surpass the median record of the working girl of fifteen. Dr. Weidensall feels that the easy op¬ posites test proved the most reliable of all tests given, for clinical purposes. In 1912 Dr. Benjamin R. Simpson 7 selected two groups as widely different in intellectual status as possible, the one repres¬ ented by seventeen professors and advanced students in Columbia University, the other by inmates of charitable institutions, with the exception of two who were recognized by their associates as being dull. The fifteen tests, which were given, were admin¬ istered individually and in the same order. The hard opposites test not only separated the two groups com- 7 Simpson, Benjamin R., “Correlations of Mental Abilities/' Col. Univ. Cont. Ed., 1912, No. 53. STANDARDIZATION OF HARD OPPOSITES TEST 7 pletely but correlated with the general intelligence of the good group, as estimated by the members which composed it, more highly than any of the other tests. The individuals of the good group were rated in order of merit for general intelligence, each by the rest of the group, four years after the tests were given. Two rankings made by the experimenter a month apart were included. The judgments, correlated with the various tests, are as follows: TABLE III Simpson’s Correlations of the Estimated Intelligence of his Superior Group with the Results of Eleven Tests Estimated Intelligence and Hard Opposites.96 “ Ebbinghaus Completion Test.89 “ Memory for Words .93 “ Memory of Passages .35 Easy Opposites .82 “ “ “ Adding .72 Learning Pairs .34 “ Completing Words . 1.00* “ “ “ “A” Test .21 “ Geometrical Forms .07 “ Drawing Lengths ...—.20 *This coefficient, according to Dr. Simpson, is not to be considered reliable, since the reliability coefficients of the Completing Words test in the Good group is only .27. On account of the high correlation between the hard opposites and the Ebbinghaus Completion Test, Dr. Simpson feels that they test the same mental function, namely selective thinking. Bronner s has endeavored to determine the intellectual status of the delinquent girl as compared with the intelligence of several other groups engaged in occupations and pursuits which, of neces¬ sity, required varied degrees of education and ability. Thirty gills living at Waverly House, a detention home maintained by the New York Probation Association, composed the delinquent group. The college group contained thirty-six girls, all mem¬ bers of the freshmen and sophomore classes of Barnard and Bronner, Augusta. F., A Comparative Study of the Intelligence of De¬ linquent Girls,” Col. Univ. Cont. Educ., 1914, No. 68. 8 MARIE HACKL MEANS Teachers’ College. Thirty-four Jewish girls, who spent their evenings at the University Settlement or the Harlem Branch of the Y. W. C. A., engaging in educational pursuits, formed the evening class group. The fourth group was composed of twenty- nine girls who had never engaged in any wage-earning occupa¬ tion except domestic service. None of these girls was pursuing studies whereby she hoped to prepare herself for a different oc¬ cupation nor had she been guilty of offenses which had brought her in conflict with the law. In addition to an ethical discrimination test, those tests em¬ ployed comprise the Easy Opposites, the Hard Opposites, the Memory of Words, the Memory of Passages, and the Ebbinghaus Completion test. These tests were the same as those used by Dr. Benjamin Simpson in his study of Correlations of Mental Abili¬ ties . In the following table, which is a reorganization of five tables presented by Bronner, D represents the delinquent group, C the college group, E the evening class group, and S the domestic service group. These data show, as Bronner states, that the TABLE IV Comparisons of the Four Groups Easy Opposites % of D reaching the 25 percentile of the C group.... % of E reaching the 25 percentile of the C group.... % of S reaching the 25 percentile of the C group.... Hard Opposites % of D reaching the 25 percentile of the C group.... % of E reaching the 25 percentile of the C group_ % of S reaching the 25 percentile of the C group_ Memory of Unrelated Words % of D reaching the 25 percentile of the C group.... % of E reaching the 25 percentile of the C group.... % of S reaching the 25 percentile of the C group_ Memory of Passages % of D reaching the 25 percentile of the C group.... % of E reaching the 25 percentile of the C group.... % of S reaching the 25 percentile of the C group_ Ebbinghaus Completion Test % of D reaching the 25 percentile of the C group.... % of E reaching the 25 percentile of the C group.... % of S reaching the 25 percentile of the C group.... 3 - 3 % 15.0% 7.0% 0.0% 12.0% 0.0% 20.0% 59-0% 7.0% 10.0% 15.0% 3.0% 6 . 6 % 9-0% 3-o% STANDARDIZATION OF HARD OPPOSITES TEST 9 college girls excel in all the tests. The delinquent girl is quite as capable as the domestic service girl, while both are surpassed by the girls attending evening classes. The superiority of the evening class girls over the other two groups displayed in each test, cannot be attributed to more favor¬ able educational advantages for none of the members had attended high school, whereas of the delinquent group, four had reached the eighth grade, two the first year of high school, two the second year, one was graduated, and one had attended a normal school. The Hard Opposites Test separated the college group from the others almost entirely. Next in order comes the Ebbinghaus Completion Test, with the Easy Opposites ranking third. It is not without interest that Dr. Simpson found the Hard Opposites separated his good group entirely from his poor group and the Easy Opposites in this respect surpassed the Ebbinghaus Test. Let us now examine the record of the Opposites Test when applied to the highly selected group, represented by the college student. In 1914 Dr. Kitson 9 began a series of tests upon the freshmen in the College of Commerce and Administration of the University of Chicago. One of the sixteen tests used was an opposites test. Two lists of twenty words each were presented and the time recorded for the subject to call the opposites of each list. Five points were deducted for each wrong word or for failure to respond within fifteen seconds, from the one hundred points allowed each list. The final score for the individual was obtained by dividing the time score by the accuracy score. The stimuli were of such little difficulty that they measured for the most part merely speed of association. Almost half of the subjects obtained an accuracy score of one hundred in each list. Of forty students, thirty-one scored one hundred in accuracy in the easy list, and twenty-one scored a hundred in the hard list. In his results Kitson has included the following table of cor¬ relations. Of the fifteen tests used by Kitson, by computing the multiple and the partial correlations and the regression coefficients, 9 Kitson, H. D., “The Scientific Study of College Students,” Psychol. Monog., 1917, 23. 10 MARIE HACKL MEANS TABLE V Correlations of Standings in Each Test with Standings in the Net Score (Method of Rank Differences) Correlations P. E. Logical Material Seen (Deferred) . .60 .07 Opposites . -53 -°8 Hard Directions (Printed) . -49 -°8 Objects Seen . .48 -°8 Loss in Logical Material Seen . .4 7 -09 Logical Material Heard (Deferred) . -45 -°9 Word Building . -45 -09 Loss in Logical Material Heard . .43 *09 Sentence Building . -4 2 -09 Constant Increment . .38 .10 Business Ingenuity . .33 .10 Logical Material Seen (Immediate) . .29 .10 Numbers Heard . .27 .10 Hard Directions (Oral) . .23 .11 Logical Material Heard (Immediate) . .23 .11 Number Checking . .18 .11 Rosenow 10 concludes that five of the tests carry all the meaning with reference to school marks and hence all diagnostic value. In the table below the writer has rearranged the results of Rosenow's investigation. From it one learns that the probability is 1300 : 1 that the Logical Memory is significant. \ TABLE VI Rearrangement of the Results of Rosenow’s Investigation Probability is 1300 : 1 that the Logical Memory Test is Significant. 25 : 1 that the Constant Increment Test is Significant. 30 : 1 that the Sentence Building Test is Significant. 23 : 1 that Auditory Presentation is Superior to Visual. 140 : 1 that the Loss or Gain in Logical Auditory Memory is significant. that the Hard Directions Test has Negative Significance. that the Objects Seen Test has Negative Significance. The remaining tests, including the Opposites, have no diag¬ nostic value. In view of the fact that the stimuli used in the Opposites Test were the Woodworth and Wells standardized list, Rosenow’s con- 10 Rosenow, Curt, “The Analysis of Mental Functions,” Psychol. Monog., 1917, 24. STANDARDIZATION OF HARD OPPOSITES TEST ii elusions are not surprising. The words were of so little difficulty that they tested only speed of association. King and M’Crory 11 in testing two hundred and seventy-six women and two hundred and sixty-eight men in the freshmen class at the University of Iowa, applied seven tests, including both easy and hard opposites. The following table indicates the correlations which were obtained between the Opposites and other tests as well as university grades. The Opposites Test, they find, correlates most highly of any of the tests with the university marks of both men and women. TABLE VII Correlations of Opposites with Other Tests and University Grades Correlations Women Men Opposites and Completion.3 1 -79 “ “ Arithmetic (Speed) .03 Neg. “ “ Arithmetic (Accuracy) .01 Neg. “ “ Analogies .52 .77 “ “ Information .24 .56 “ Visual Imagery .07 .56 “ “ Logical Memory .32 .38 “ Test Average .51 .88 “ University Grades . 45 .84 Dr. King 12 is of the opinion that the Opposites Test, if thor¬ oughly standardized and used in conjunction with other tests, will yield results of great importance. In the Spring of 1916, while the writer was yet engaged in the selection of appropriate stimuli for the present investigation, an opportunity 13 presented itself of correlating the grades of seventy- three students in psychology with the results secured by the use of the Opposites Test. An effort had been made to keep the two lists composed of one hundred and fifty words each, as nearly 11 King, Irving, and M’Crory, J. L., “Freshmen Tests at the State Univer¬ sity of Iowa,” Jour. Educ. Psychol., 1918, IX, 32-46. 12 King, Irving, and Gold, Hugo, “A Tentative Standardization of Certain Opposites Tests,” Jour. Educ. Psychol., 1916, VII, 459-482. 13 Dr. Edward K. Strong, at that time Professor of Psychology at George Peabody College for Teachers, kindly furnished these data. The grades were compiled from the records of fourteen tests given during the quarter. 12 MARIE HACKL MEANS equal as possible. The two lists are given below. The words are arranged in random order. Those words which had not been used by previous investigators are in italics. List i.—Defective, Late, Sinful, Easy, Hilly, Superior, Girl, White, Cool, Large, Evil, Queen, Deep, True, Public, Sink, Future, Adroit, Dangerous, Day, Ugly, Quick, Poor, Diligent, Wicked, Round, Ceiling, Broken, Gentle, Vague, Brief, Ani¬ mated, Slovenly, Dim, Out, Rude, Lazy, Injurious, Conservative, Wet, Asleep, Stingy, Fertile, Wise, Calm, Tardy, Hinder, Re¬ spect, Big, Gain, Great, Profit, Young, Few, Summer, Above, Glad, Masculine, Remember, Off, Beginning, Love, Straight, War, Joy, Naked, Pride, Apart, Brave, Noisy, Fickle, Create, Wild, Despondent, Frequently, Timid, Hollow, Belief, Bad, Up, Sick, Empty, Strong, Inside, Front, After, Broad, Sharp, Sweet, Succeed, Add, Happy, Raise, Aristocratic, East, Short, Thick, Result, Rare, Stale, North, Hostile, Laugh, Obnoxious, Ex¬ pensive, Near, Join, Hot, Forcible, Preserve, Strict, Handsome, Friend, Miser, Exciting, Rough, Brother, Light, Careful, Push, Haughty, Impoverish, Busy, Much, Graceful, Ocean, Precise, Barbarous, Ignorant, Reckless, Odd, Victorious, Repulsion, Per¬ mit, Positive, Pessimistic, Extravagant, Durable, Analytical, Par¬ simony, Orthodoxy, Acute, Exoteric, Antonym, Dorsal, Longi¬ tude, Divide, Infinity, Dynamic, Posterior. List II.—Gay, Foolish, Drop, Giving, Cloudy, Blunt, Beauti¬ ful, Backwards, Well, Top, Success, Soft, New, Refined, Weary, Spend, Break, Male, Country, Dark, Weak, Black, Disastrous, Rigid, Elation, Hindrance, Savage, Degrade, Ripe, Shaky, Sepa¬ rate, Liquid, Sell, Honest, Difficult, Dirty, Wrong, Winter, Help¬ less, Obscure, Expand, Insignificant, Sleepy, Sad, Little, Enemy, Open, Yours, Yes, Conservative, Soothing, Doubtful, Sacred, Sure, Reveal, Stupid, Motion, Sickly, Slowness, Outside, Same, Cowardly, Float, Foreign, Strength, Sane, Level, Simple, Many, Lost, Something, Sour, Enrage, Serious, Long, In, Take, Tight, Prompt, Patient, Permanent, Genuine, Morning, Smooth, Heavy, Full, Grand, Humility, Tall, Over, First, Strife, Follower, Hold, Proficient, Vertical, Shallow, Absent, Rapid, Rich, Purity, Lo¬ quacious, Imaginary, Silly, Increase, Wider, Nowhere, Upper, STANDARDIZATION OF HARD OPPOSITES TEST 13 Woman, Generous, Careless, Scarce, Height, Always, Wife, Best, Below, Thin, High, Early, Suave, Lack, Advance, Harmonious, Bless, Both, Cruel, Cheap, Ancient, Less, Forget, Come, Slow, Good, Negative, Optimistic, Economical, Perishable, Synthetical, Prodigality, Heterodoxy, Chronic, Esoteric, Synonym, Ventral, Latitude, Multiply, Zero, Static, Anterior. The test was given individually and orally. The experimenter read the stimulus and recorded the response of the subjects as well as his reaction time. The accuracy score in percentage and the time score arrived at by computing the median time of all reactions were tabulated for each student. In grading these papers, credits of one, one-half, or zero were assigned to the responses. What credit a word deserved was determined solely by the writer, as she was not yet ready to standardize the responses for so bulky a list. In order to allow equal credit for time and accuracy the follow¬ ing formula was used: Dt Da Individual score =- 1 -in which OT QA 2 Da = the deviation of the individual accuracy scores from the accuracy scores of the group. Dt = the deviation of the individual time score from the time score of the group. Since signs were regarded, where an individual had a lower time score than the group his deviation was positive. QA= the quartile of the accuracy scores of the group. QT = the quartile of the time scores of the group. The coefficient of correlation obtained by the method of rank differences was .54, P. E. = .08. Speed and accuracy were likewise correlated, giving a coefficient of .33, P. E. = .10. This coefficient would hardly justify the statement that, in so far as this particular test is concerned, either accuracy or speed could be disregaded without affecting the rankings in the test. In Figure I is shown the relative positions of the several mem¬ bers of the class in psychology and in the Opposites Test. The 14 MARIE HACKL MEANS vertical line indicates the rank in the test, the horizontal line, the rank in psychology. A glance at the figure will show that on the whole those good in the test were good in psychology. The greatest exception is to be found in the position of the student indicated by the figure 5 in the graph. This woman of splendid ability was permitted by her physician to remain in school only on the condition that she would do the least possible amount of work to secure credit for the course in which she was enrolled. Case 4, a widow, had been out of school and school work for a number of years. The increased cost of living had forced her to return. Case 3 had registered in psychology because it was one of the required courses. Figure i The relations between rank based on psychology grades and rank based on the Opposites Test score are shown here. Each circle indicates the posi¬ tion of one of the seventy-three college students. Units on the vertical line denote rank in the test; horizontal units are ranks in psychology. STANDARDIZATION OF HARD OPPOSITES TEST 15 On the other hand, it seems, one could predict with less cer¬ tainty the standing in the class of those students who are some¬ what poor in the test. This may be due in part to the inadequacy of the test itself, in part to the inequality of interest in psy¬ chology. Cases 1 and 2 are typical of those students whose class work represents their best effort. These cases are selected be¬ cause they are the only ones falling within the writer’s experience. A careful analysis of the individual records, if possible, might reveal results of significance. While the results indicate that the test as used by the different investigators reveals some sort of native ability, just what this ability is and to what extent it is revealed remains to be seen. Certainly, results of various experimenters up to date are not comparable, nor will they be until identically the same test is given with the same method of presenting and scoring. III. Previous Efforts toward Standardization The first attempt to meet this need of standardization was made by Woodworth and Wells 1 in 1911. After testing six sub¬ jects with a long list of words, forty words were selected, which in turn were tried with forty other subjects. A few more words were substituted from tests of a few subjects, then the revised col¬ lection was tried with thirteen fresh subjects (all college and graduate students) and a few more minor corrections introduced. In the experiment the four lists of ten stimulus words were presented visually but the time of the single reaction as well as the total time was roughly taken. An attempt was made to have the halves of the list of equal difficulty, and since it was found impossible to prepare a list of twenty stimuli of equal difficulty, the words were combined in pairs, “so that pairs should be of equal difficulty, as judged by the sum of the reaction times to the two members of each pair.” One pair, for instance, consisted of the hardest and the easiest word in the list and another pair of two words of medium dif¬ ficulty. In the event that the test might be given with a time limit, the authors have placed the words of medium difficulty in the list where most of the subjects will be stopped, namely, from about the eighth to the sixteenth word. “If then the time limit is so chosen that the great majority of subjects shall be stopped in this list, the separate words may, without much error on the aver¬ age, be counted as equal units.” The test as given by Woodworth and Wells is scored only on the basis of time. The words were so selected that none of the subjects could fail to have a perfect accuracy score. As such, it is, as the authors intended, merely a test of the speed of associa¬ tion. 1 Woodworth, R. S., and Wells, F. L., “Association Tests,” Psychol. Monog., 1911, XIII. STANDARDIZATION OF HARD OPPOSITES TEST 17 In 1916 a further attempt was made by King and Gold 2 to standardize the test. The four lists of twenty easy words and the four lists of twenty hard words, as used by Simpson, 3 were pre¬ sented individually to nine faculty members, twenty-three gradu- are students, forty-seven seniors, and twenty-one juniors of the Departments of Education and Psychology in the University of Iowa. The subjects were instructed to take one list at a time, go down the list and name orally the opposite of each word in rapid suc¬ cession and to avoid wherever possible prefixing a syllable to the stimulus word. The time was recorded for each list separately and the responses were taken down in shorthand. The easy and hard opposites were scored differently. In the case of the easy opposites a credit of one, one-half, or zero was given to responses, and the penalty for an omitted word was four seconds. To the various responses to the hard opposites, credits of one, two-thirds, one-third, and zero were assigned, and an omitted word or incorrect word was penalized eighteen seconds. Regardless of the fact that the words were of varying difficulty, the penalty remained the same, namely, four seconds for easy words and eighteen seconds for hard words. On the basis of these results (one hundred records for each word) the percent of failures for each word was computed. For each stimulus is recorded the value in terms of accuracy per¬ centage as well as the acceptable responses and the frequency of each. In the work of Harry A. Greene 4 we find the first attempt to assign to each word a point value based on its relative difficulty. Greene presented the stimuli used by King and Gold 5 to two groups of freshmen in the University of Iowa in the fall of 1916- 1917. Nine hundred and ninety individuals were tested by the first half of the list and seven hundred and ten by the second 2 King, Irving, and Gold, Hugo, “A Tentative Standardization of Certain Opposites Tests,” Jour. Educ. Psychol., 1916, VII, 459-482. 3 Simpson, Benjamin R., Op. cit. 4 Greene, Harry A., “A Standardization of Certain Opposites Tests,” Jour. Educ. Psychol., 1918, IX, 559-566. 5 King, Irving^ and Gold, Hugo, Op. cit. i8 MARIE HACKL MEANS half. The words were graded by the King and Gold standardiza¬ tion with the exception that words given two-thirds credit by them were given one-half credit and those receiving one-third credit were scored wrong. The percentage of failures was then determined by allowing a value of one unit for correct responses and one-half for half- correct responses and subtracting this total from nine hundred and ninety. In the case of those words from the last half of the list for which there were only seven hundred and ten responses available, these totals were increased in the proportion of nine hundred and ninety to seven hundred and ten, thus making all the words in the list comparable. By reading directly from the table based upon the area of the probability curve and assuming that the base line is broken ar¬ bitrarily at +3 sigma, the percentage scores were changed into percentile values. These values were then totaled and each value in turn divided by the total, thus converting the percentile values into relative point values. The points were based on accuracy alone, no account being taken of time. IV. The Selection of Stimuli for the Present Investigation At the very outset, the writer was confronted with the problem of the method in which the test was to be presented. Chiefly because of the opportunity it afforded of studying each word individually, the oral method was chosen. Each word thus be¬ came a unit. The list could be lengthened or shortened at will without influencing to any appreciable extent the words retained. Let it be held in mind that at this point our interest was two-fold, namely, the selection of suitable stimuli and the computation of the value in points to be assigned to each word. To have given the test in groups would have simplified the scoring and would have dispensed with much effort, but the at- tendent disadvantages made it necessary to abandon this method as a possible choice. In the first place, it was intended that the test should be more difficult than previous tests of this character. This meant the substitution of entirely new stimuli, stimuli to be acquired only by the slow process of trial and error. Furthermore it was en¬ cumbent upon the writer, due to the small number of subjects at her disposal, to avoid a method which would involve discard¬ ing the entire record of a subject because of the use of one or more undesirable stimuli. As given, each word was a problem apart from every other word and only the records of those words not in the list as finally decided upon, were thrown into the discard. If the group method is to be employed, a decision must be made as to whether time, accuracy, or a combination of both will be considered a measure of the difficulty of the word. Time considered alone, the test becomes one largely of speed of asso¬ ciation. If the test is to be given with a time limit, the individual responses lose their identity as units and the percentage of failures 20 MARIE HACKL MEANS per stimulus will vary with the time allowed. For instance, if the time limit is reduced from six to three minutes, the percentage of failures will vary considerably, particularly in the case of the more difficult words which should be placed at the end of the list. Thus one can see how a shortened time limit might give identical accuracy scores with words of unequal difficulty. To illustrate the effect of varying the time limit not only upon the percentage of failures but also upon the nature of the re¬ sponses themselves, let us consider the responses to the word “sacred” given, first, by a group of fifty-six students tested orally and individually, allowing ten seconds for the response, second, by another group of fifty-six students similarly tested, and third, by a group of fifty-six who were allowed only six minutes to write the opposites to the list of sixty-eight words eventually re¬ tained. In the table below is shown the number of failures and the exact responses given by the three groups of fifty-six. Notice the greater number of failures in Group III and also the homo¬ geneity of responses. This may be explained by the fact that the word is not reached in the list or that uncertain words are passed up for those in which the subject feels confident of success. On the other hand to lengthen the time limit considerably, while it would give more nearly correct accuracy scores, would be unfair to the individuals tested, if these same records were to be used as the basis for norms. With a lengthened time limit, students would be able to complete the test at different intervals. Evidently of two students, both of whom make perfect scores, the one finishing the work in the shorter period is the more ef¬ ficient. But we should have no measure of this efficiency. To give the test as a group test without taking into considera¬ tion the time consumed, would seem to imply either that addi¬ tional time would fail to increase the accuracy percentage, or that without exception the more difficult a word, the longer the time required to think of an opposite. The former assumption is probably true within certain limits. It is conceivable that a time limit might be selected beyond which there would be no improve¬ ment, but it is obvious that the time limit might be so shortened STANDARDIZATION OF HARD OPPOSITES TEST 21 TABLE VIII Showing the Different Responses Given as an Opposite to the Word “Sacred” when the Time Limit is Changed Number of Times Each Response is given by the Several Groups Different Responses Given by the Groups Group I Group II Group III Common . I i Secular . 2 4 Profane. . 5 7 4 Sinful . 2 I Worldly . 2 I Unholy . . 9 I 2 Irreverent . . 2 3 I Wicked . . 2 2 I Sacrilegious . . 3 7 2 Unsacred . . 8 5 0 Vulgar . . i 3 0 Public . . 2 I 0 Mean . i 0 Ungodliness . . 0 i 0 Heathen . I 0 Sacrilege . I 0 Obnoxious . . I 0 0 Ungodly . . I 0 0 Unnoticed . . I 0 0 Idolatrous . . I 0 0 Irreligious .. 0 0 Infidel . 0 o Hypocritical . 0 o Failure to Respond. . 15 16 39 Number of Different Responses per Group.... 16 9 that the accuracy scores would be greatly affected. In general it is true that words with a high accuracy score have a corre¬ spondingly low time score, but a glance at Tables XI and XII will show that such is not always the case, for words with differ¬ ent accuracy scores may have identical time scores and words with identical accuracy scores have different time scores. How¬ ever, the correlation between the two is undoubtedly high. To give the test individually and record the time for the total list has all the disadvantages in point of labor and none of the advantages that accrue when the test is given individually and orally. 22 MARIE HACKL MEANS Having determined the procedure to be followed, the writer began in the fall of 1915 the task of selecting suitable stimuli for the present investigation. After eleven records were obtained with a list containing the three hundred and twenty-three words used by previous experimenters and forty new words, the list was divided and during the remainder of the year two lists were used which consisted on the average of about one hundred and fifty words each. From time to time words found inadequate were dropped and others substituted as they occurred to the writer. Hence the inequality in the number of subjects tested with a given word. In this manner one hundred and thirty-nine students at George Peabody College for Teachers were tested during the school year 1915-1916. Commonly misunderstood words were weeded out as were those with an accuracy score of one hundred percent. Likewise attempts were made to discard those words which had an oppo¬ site formed by adding the prefix “un” in frequent and reputable use. Meanwhile original words were subjected to the same pro¬ cess of examination and elimination. The point of interest was the individual word, but as these data were to be used in another connection care was taken that the conditions remain constant throughout. Since each test was given individually, it was an easy matter to change the order of the stimuli and thus avoid practice effects. The reaction time in fifths of a second was recorded with a stop-watch along with the response. The following directions were read to each subject: “As soon as I read a word you are to give me the best opposite you can think of. For instance, if I read 'black’ you are to say 'white/ Do not give me phrases, nor words beginning with the prefix 'non.’ The word you give must belong to the same part of speech as the word in the list. Your time will be recorded, so answer each word as quickly as possible. Under no circum¬ stances will I allow you over ten seconds for a word. Let me show you how long ten seconds really is. (Experimenter illus¬ trates with stop-watch.) Remember in each case to respond as quickly as possible. Do you understand what you are to do?” STANDARDIZATION OF HARD OPPOSITES TEST 23 In the fall of 1918 1 the work was resumed. It was decided expedient to use only one list as the number of possible subjects to be obtained at Peabody was small. Two lists would of course cut the records for each word in two. Besides the second list contained for the most part opposites of the first. Of the one hundred words with which the work was begun in the fall, thirty- two were dropped, leaving the sixty-eight which comprise the list as it now stands. These words, arranged in the ascending order of difficulty, are given below. The original words are given in italics. Full, Negative•, After, Dim, Blunt, Success, Pessimistic, Joy, Public, Profit, Spend, Always, Graceful, Strength, Ancient, Expand, Barbarous, Hinder, Despondent, Vague, Fertile, Doubtful, Injurious, Busy, Abstract, Advance, Foreign, Create, Simple, Extravagant, Aristocratic, Rare, Dan¬ gerous, Slovenly, Defective, Stingy, Reveal, Diligent, Join, Im¬ poverish, Permanent, Elation, Sinful, Obnoxious, Conservative, Victorious, Obscure, Proficient, Rigid, Repulsion, Imaginary, Permit, Orthodoxy, Analytical, Extrinsic, Sacred, Dynamic, Lo¬ quacious, Heterogeneous, Spurious, Disastrous, Facility, Pride, Result, Adroit, Parsimony, Suave, Esoteric. One hundred and twelve subjects were tested with this list. This means that later in determining the relative difficulty of the stimuli, the calculations are based on at least one hundred and twelve records for each word, varying up to one hundred and eighty-seven for others. 1 This research was conducted under the supervision of Dr. Edward K. Strong up to this point, thereafter under the guidance of Dr. Joseph Peterson. V. The Choice of the Acceptable Responses The next step was to standardize and evaluate the responses. In order to facilitate matters, beneath each stimulus was written the long list of words which had been given as opposites by those subjects experimented upon. The credit due each of these re¬ sponses was determined by five judges, including the writer. They were besides the writer: Miss Lula O. Andrews, Professor of English; Miss Mary Clay Hiner, Instructor in English; Mr. S. H. Phelps, Instructor in School Administration, and Dr. Joseph Peterson, Professor of Psychology, all of George Peabody College. Each judge was ignorant of the credit assigned any word by any other judge. The following directions, a copy of which was handed each judge, will make the matter clear: Directions for Grading Responses I. Make use of any available source of information such as < the dictionary or book of antonyms. II. The responses are to be graded with a grade of “one,” which means an exact opposite, or “one-half,” which means only an approximate opposite, or “zero,” which means a failure. III. More than one response to a word may be given a credit of “one.” IV. Words belonging to a different part of speech are to be graded “zero,” also words with the prefix “non.” V. If a word belongs to the same part of speech and is nearly an opposite, give it “one-half” credit. VI. Add any response that may suggest itself to you and grade it as directed above. The new words suggested were in turn passed on by the other four judges. In addition, when later the test was converted into a group test and hundreds of records secured, a number of STANDARDIZATION OF HARD OPPOSITES TEST 25 different responses, given by those tested, were submitted to the same process of evaluation. The credits assigned each response were then averaged and if the result was nearer “zero” than “one-half,” the word re¬ ceived no credit; if nearer “one-half” than “zero” or “one” it received “one-half” credit; and if nearer “one” than “one-half,” it was given a credit of “one.”' For instance the response “young” in answer to “ancient'’ received credits as follows: TABLE IX I II III / IV Credit V Total Aver. Assigned Credits given to “young” by five j udges.. 1 0 0 y 2 2 .4 y 2 On the following pages is to be found in Column II the stim¬ ulus word, in Column III the responses which are due a credit of “one’ (the full value assigned to the word), and in Column IV those responses which receive only “one-half” credit (half the value assigned the word). The value of the word is given in points in Column I. The manner in which we arrived at these values will be discussed in the following section. TABLE X A List of the Stimuli with Assigned Values and Accepted Responses Value Stimulus Word Correct Responses Half Correct Responses 1 Full Empty Meagre 1 Negative Affirmative Positive Assertive 1 After Before Preceding Fore I Dim Bright ' Light Clear Plain Distinct Luminous Vivid 1 Blunt Keen Acute Pointed Polite Sharp Sensitive Tactful Suave 26 MARIE HACKL MEANS TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses i Success Defeat Downfall Disaster Failure Loss i Pessimistic Optimistic Joyful I Joy Gloom Depression Grief Despondency Misery Displeasure Sadness Pain Sorrow Woe 2 Public Private Domestic Personal Secluded Secret 2 Profit Loss Deficit Lose 2 Spend Earn Accumulate Hoard Get Husband Hold Keep Make Save Receive 2 Always Never Infrequently Seldom 2 Graceful Awkward Uncouth Clumsy Ungraceful Gawky Ungainly Unsightly 2 Strength Feebleness Delicacy Frailty Weakness Insecurity 3 Ancient Modern New Present Recent Young 3 Expand Contract Compress Decrease Diminish Shrink Shrivel Narrow 3 Barbarous Civilized Chivalrous Humane Civil Cultured Educated Gentle Kind Polite STANDARDIZATION OF HARD OPPOSITES TEST 27 TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses 3 Hinder 3 Despondent 3 Vague 3 Fertile 4 Doubtful Aid Assist Expedite Fore Forward Further Facilitate Help Promote Buoyant Cheerful Ecstatic Elated Exuberant Glad Happy Hopeful Joyful Jubilant Clear Definite Distinct Exact Plain Specific Arid Barren Poor Sterile Unproductive Assured Certain Evident Hopeful Sure 4 Injurious Advantageous Beneficial Helpful Innocuous Wholesome 4 Busy Idle Unemployed Unoccupied Advance Encourage Bright Encouraged Exhilarated Gay Hilarious Joyous Merry Optimistic Sanguine Apparent Concise Explicit Fruitless Impotent Impoverished Infertile Unimaginative Apparent Believable Clear Confident Credible Known Positive True Truthful Unquestioned Harmless Healthful Safe Uninjurious Dull _ Inactive Indolent Loafing 28 MARIE HACKL MEANS TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses 4 Abstract Concrete Specific 4 Advance Decline Degrade Recede Hinder Retard Retire Retreat Retrograde Withdraw Withhold 4 Foreign Domestic American Germane Home Native Indigenous Pertinent Local Natural 4 Create Annihilate Abolish Demolish Disintegrate Destroy Dismember Exterminate Obliterate Undo Waste 4 Simple Complex Bright Complicated Confusing Compound Difficult Elaborate Gorgeous Intricate Grand Ornate Hard Wise Learned Sensible Smart 4 Extravagant Economical Careful Frugal Close Miserly Conservative Niggardly Economizing Parsimonious Penurious Saving Stingy Thrifty 4 Aristocratic Bourgeois Humble Common Low Democratic Lowly Plebeian Ordinary 4 Rare Abundant Commonplace Common Often Dense Ordinary Frequent Profuse Numerous Regular Plentiful Plenteous Usual Occasional 4 Dangerous Harmless Safe Peaceful STANDARDIZATION OF HARD OPPOSITES TEST 29 TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses 5 Slovenly Neat Careful Neatly Carefully Tidy Gracefully Tidily Precise Trim 5 Defective Complete Correct Faultless Effective Normal Good Perfect Sound Whole Healthy 5 Stingy Generous Bounteous Lavish Bountiful Liberal Extravagant Prodigal Freehearted Magnanimous Unselfish Unsharing Wasteful 5 Reveal Conceal Cover Hide Secrete Obscure Suppress Withhold 5 Diligent Dilatory Careless Indolent Negligent Lazy Inattentive i Slothful Idle Shiftless 5 Join Abandon Detach Disjoin Disassociate Disconnect Disband Dismember Leave Divide Resign Part Tear Separate Undo Sever Untie Sunder Withdraw 6 Impoverish Enrich Aggrandize Replenish Fertilize Nourish Strengthen 7 Permanent Ephemeral Ephemerate Evanescent Fluctuating Fleeting Impermanent Passing Perishable Temporary Shifting Transient Transitional Transitory Unsubstantial 30 MARIE HACKL MEANS TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses 7 Elation Dejection Depression Despair Despondence Despondency Grief Sadness Shame Sorrow 7 Sinful Blameless Righteous Sinless Godly Holy Innocent Just Religious Perfect Pious Pure Upright Virtuous 7 Obnoxious Agreeable Beneficial Inoffensive Pleasing Acceptable Attractive Congenial Desirable Harmless Helpful Likable Pleasant Winning 7 Conservative Extreme Liberal Progressive Radical Aggressive Extravagant Extremist Wasteful 7 Victorious Conquered Defeated Vanquished Beaten Unsuccessful Unvictorious Whipped 7 Obscure Clear Conspicuous Eminent Lucid Plain Prominent Renowned Reveal Apparent Bright Disclose Distinct Evident Famous Noticeable Notorious Obvious Unambiguous 8 Proficient Deficient Inapt Unskilled Unskillful Backward Incapable Incompetent Inefficient Lacking Unprepared STANDARDIZATION OF HARD OPPOSITES TEST 3i TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses 8 Rigid Elastic Easy Flexible Flabby Lax Flaccid Lenient Loose Limber Limp Plastic Relaxed Pliable Supple Yielding 8 Repulsion Attraction Acceptance Cohesion Agreeableness Admiration Love 9 Imaginary Real Actual Prosaic Realistic True 9 Permit Forbid Deny Prevent Disallow Refuse Disapprove Restrain Object Prohibit 9 Orthodoxy Heresy Catholicity Heterodoxy Liberalism Unconventionality Unsoundness 10 Analytical Synthetic Synthetical Unanalytical 10 Extrinsic Intrinsic Essential Internal 11 Sacred Defiled Common Profane Desecrated Secular Sacrilegious Unholy Unconsecrated Ungodly • Vulgar 11 Dynamic Inert Potential Static Powerless Weak 11 Loquacious Laconic Mute Reticent Reserved Taciturn Quiet Silent Untalkative 12 Heterogeneous Homogeneous Alike Similar 32 MARIE HACKL MEANS TABLE X—Continued Value Stimulus Word Correct Responses Half Correct Responses 12 Spurious Authentic Genuine Real * Actual Legitimate Pure Sound True Truthful Unadulterated 13 Disastrous Advantageous Beneficial Helpful Safe Favorable Fortunate Gainful Harmless Lucky Prosperous 14 Facility t k Difficulty Awkwardness Clumsiness Disability Effort Inconvenience 14 Pride Humility Lowliness Meekness Debasement Degradation Humbleness Humiliation 15 Result Causation Cause Beginning Commencement Origin Purpose Start 15 Adroit Awkward Crude Clumsy Unskilled 17 Parsimony 22 Suave Inexpert Maladroit Unskillful Extravagance Generosity Lavishness Liberality Prodigality Brusque Impolite Tactless Unpolished Freeheartedness Magnanimity Wastefulness Abrupt Blunt Crude Gruff Harsh Rough Rude Uncouth Undiplomatic Unpleasant Untactful Esoteric Private 26 Exoteric VI. The Computation of the Relative Difficulty of the Stimuli As stated previously, when the test was administered, each response given by the subject as well as the reaction time of that response was recorded. On the basis of these records the following data were tabulated for each stimulus: the different responses, the reaction time of each response, and the failures to respond. Records were so kept that information could be obtained not only regarding the number of subjects giving a certain response but also regarding the reaction times of that response. For instance, instead of merely indicating the number of people who gave “young” as an opposite to “ancient,” the records show that “young” was given one time in six-fifths of a second, another time in nine-fifths, another in seven, etc. Furthermore, in order to ascertain when a sufficiently large number of subjects had been tested for the results to be reliable and to be able to indicate the extent of this reliability, as deter¬ mined by the change brought about by additional sampling, sepa¬ rate records were kept of all tests made prior to the fall of 1918. These will be designated as Group I. Group II consists of the fifty-six records secured in the early fall of 1918, and Group III of the remaining fifty-six records. Since some of the stimuli were added to the list at different times the number of records of subjects in Group I varies from zero to seventy-nine. After the acceptable responses had been determined these record sheets were scored and the number of correct responses the number of half correct responses, and the number of failures, as well as the total of all three, were tabulated. The number of failures includes both the failures to give any opposite and the failures to give the correct opposite. These results are to be found in Table XI, columns III, IV, V and VI. Referring to this table, we read that for the word “full,” there were in Group 34 MARIE HACKL MEANS i, sixty correct responses, no half correct responses, and three failures, making a total of sixty-three records for that word ob¬ tained before the fall of 1918. Reading further in Group II for the same word, we have fifty-four correct and one half coirect responses with one failure, thus totaling fifty-six. In Group III we find the following distribution: fifty-five correct and one half- correct. In computing the percentage of failures, to be found in column VII, two half-correct responses were considered equivalent to one failure. The formula will make clear the procedure followed. H F + T Percentage of failures = 1 - in which * N F = the number of failures, H = the number of half-correct responses, N = the total number tested. Applying this formula to the word “full” we have in Group I, o 3 H- 2 -y-= -047 63 Group II does not represent the percentage of failures in Group II alone but a combination of Groups I and II. Applying the 4 “t” -5 formula again for the same word we have -= .037 119 in which 4 equals the sum of the failures in groups I and II, .5 equals half of the half-correct responses in groups I and II, and 119 is the sum of the total number tested. Similarly Group III represents the sum of groups I, II, and III. This procedure of estimating the three different percentages of failures, instead of only one was adopted because it indicated the degree of reliability of the measure. Very radical changes obviously would mean little reliability. Efforts were made to have our sampling as representative of all the students as possible. Therefore the number tested include graduate students as well as members of the four different classes of undergraduates. STANDARDIZATION OF HARD OPPOSITES TEST 35 TABLE XI Data from Which the Percentage of Failures for Each Stimulus was Computed I Stimulus II Group hi Number of Correct Responses IV Number of Half-Correct Responses V Number of Failures VI Total Number Tested VII Percentage of Failures Full I 60 0 3 63 .047 II 54 I I 56 •037 III 55 I 0 56 .028 Negative I 48 0 2 50 .040 II 53 0 3 56 .047 III 56 0 0 56 .030 After I 7 1 0 3 74 .040 II 55 0 1 5b .030 III 53 0 3 56 •037 Dim I 66 8 1 75 .066 II 50 3 3 56 .072 III 42 14 0 56 .089 Blunt i 60 0 3 63 •947 n 48 2 6 56 ‘.084 hi 50 0 6 56 .091 Success i 59 0 4 63 .063 ii 5 i 1 4 56 .071 hi 44 3 9 56 .108 Pessimistic i 47 0 2 49 .040 ii 49 1 6 56 .080 hi 47 0 9 56 .108 Joy i 65 0 3 68 .044 ii 49 3 4 56 .068 in 48 2 6 56 .086 Public i 63 0 5 68 •073 ii 46 3 7 56 .108 in 49 3 4 56 .105 Profit i 55 7 5 67 .126 ii 52 0 4 56 .101 hi 4 i 5 10 56 •139 Spend i 46 9 8 63 .198 ii 5 i 1 4 56 .142 in 5 i I 4 56 .122 Always i 52 1 10 63 .166 ii 44 0 12 56 .188 hi 43 2 11 56 .197 36 MARIE HACKL MEANS TABLE XI—Continued I Stimulus II Group in Number of Correct Responses IV Number of Half-Correct Responses V Number of Failures VI Total Number Tested VII Percentage of Failures Graceful I 50 13 5 68 .169 II 44 7 5 56 .161 III 45 4 7 56 .161 Strength I 56 0 8 64 •125 II 43 0 13 56 • 175 III 4 i 0 15 56 .204 Ancient I 33 26 3 62 .258 II 32 22 2 56 •245 III 40 14 2 56 .218 Expand I 45 1 11 57 .201 II 43 2 11 56 .207 III 41 0 15 56 .227 Barbarous I 57 11 0 68 .080 II 34 17 5 56 .153 III 34 12 10 56 .194 Hinder I 52 3 13 68 .213 II 44 0 12 56 .213 III 39 5 12 56 .227 Despondent I 48 12 3 63 .142 II 37 15 4 56 .172 III 39 8 9 56 .191 Vague I 55 2 10 67 .164 II 42 2 12 56 •195 III 38 1 1 7 56 .231 Fertile I 54 1 15 7 o .221 n 40 0 16 56 .250 in 47 0 9 56 .222 Doubtful i 38 9 10 57 •254 11 39 9 8 56 .238 hi 38 10 8 56 .236 Injurious i 48 10 10 68 .220 ii 39 8 9 56 .225 hi 36 6 14 56 •249 Busy i 39 4 24 67 .388 ii 35 1 20 56 .378 hi 4 i 1 14 56 •340 Abstract i 0 0 0 0 .000 ii 35 2 19 56 •357 in 33 4 19 56 •365 STANDARDIZATION OF HARD OPPOSITES TEST 37 TABLE XI—Continued I Stimulus II Group III Number of Correct Responses IV Number of Half-Correct Responses V Number of Failures VI Total Number Tested VII Percentage of Failures Advance I 48 0 10 58 .172 II 32 4 20 56 .280 III 34 1 21 56 •314 Foreign I 37 TO 6 53 .207 11 34 10 12 56 .256 III 34 10 12 56 .272 Create I 50 3 14 67 .231 11 38 1 1 7 56 .292 III 35 2 19 56 .296 Simple I 13 21 9 43 •453 II 34 15 7 56 •343 III 32 14 10 56 .328 Extravagant I 4 i 8 7 56 .196 II 42 6 8 56 .196 III 35 9 12 56 .229 Aristocratic I 52 4 12 68 .205 II 34 5 17 56 .270 III 32 1 23 56 .316 Rare I 37 16 8 61 .262 II 38 7 11 56 .260 III 38 „6 12 56 .262 Dangerous I 35 1 15 5 i .303 II 36 0 20 56 •331 III 35 0 21 56 •346 Slovenly I 44 10 14 68 .279 II 27 11 18 56 •342 III 28 4 24 56 .380 Defective I 47 6 15 68 .264 II 38 6 12 56 .264 III 27 8 21 56 322 Stingy I 43 11 14 68 .286 II 30 11 15 56 .322 III 18 13 25 56 •397 Reveal I 38 4 21 63 .365 II 31 4 20 56 .378 III 34 4 18 56 •371 Diligent I 43 4 20 67 .328 II 33 15 8 56 .304 III 26 9 21 56 •351 38 MARIE HACKL MEANS i TABLE XI—Continued I Stimulus II Group hi Number of Correct Responses IV Number of Half-Correct Responses V Number of Failures VI Total Number Tested VII Percentage of Failures Join I 33 11 11 55 .300 II 38 0 18 56 .310 III 30 7 19 56 •341 Impoverish I 38 4 27 69 .420 n 22 3 3i 56 .492 hi 27 0 29 56 .500 Permanent i 37 5 21 63 •373 n 24 1 3i 56 .462 hi 34 1 21 56 •437 Elation i 33 10 19 62 •387 ii 17 3 36 56 .521 hi 18 3 35 56 •563 Sinful i 20 40 8 68 .411 ii 13 25 18 56 .471 hi 17 19 20 56 .488 Obnoxious i 43 11 13 67 .276 ii 14 22 20 56 .402 hi 16 14 26 56 .460 Conservative i 39 3 27 69 •413 ii 15 15 26 56 .496 hi 14 15 27 56 •533 Victorious i 43 4 24 67 .388 ii 22 9 25 56 •451 hi 24 8 24 56 .466 Obscure i 27 4 26 57 .491 ii 25 3 28 56 .508 hi 25 6 25 56 .505 Proficient i 18 30 15 63 .476 ii 16 18 22 56 .512 hi 14 13 29 56 •551 Rigid i 27 14 16 57 403 ii 17 18 21 56 .469 hi 14 18 24 56 .508 Repulsion i 29 4 34 67 •537 ii 20 2 34 56 •5 77 hi 18 3 35 56 .600 Imaginary i 22 4 28 54 •555 ii 21 2 33 5b .581 hi 20 2 34 56 •596 STANDARDIZATION OF HARD OPPOSITES TEST 39 TABLE XI—Continued I Stimulus II Group III Number of Correct Responses IV Number of Half-Correct Responses V Number of Failures VI Total Number Tested VII Percentage of Failures Permit I 28 12 28 68 .500 II 24 7 25 56 004 III 24 8 24 56 .502 Orthodoxy I 15 0 40 55 .727 II 9 0 47 56 •783 III 6 0 50 56 .820 Analytical I 29 1 27 57 .482 II 10 1 45 56 .646 III 11 0 45 56 .698 Extrinsic I 1 0 1 2 .500 II 16 0 40 56 .706 III 15 0 4 i 56 .719 Sacred I 24 4 30 58 •551 II 15 7 34 56 .609 III 10 11 35 56 .647 Dynamic I 5 4 30 39 .820 II 3 4 49 56 •873 III 7 2 47 56 .867 Loquacious I 14 20 28 62 .612 II 13 3 40 56 •673 III 13 5 38 56 .689 Heterogeneous I 2 0 1 3 •333 II 9 4 43 56 •779 III 12 0 44 56 .782 Spurious I 6 12 3 i 49 •755 i II 4 3 49 56 .833 III 4 3 49 56 .857 Disastrous I 18 2 30 50 .620 II 15 6 35 56 .650 III 21 2 33 56 .635 Facility I 10 1 34 45 .766 II 6 3 47 56 .821 III 5 0 51 56 •853 Pride I 17 3 53 73 .746 II 8 2 46 56 .786 III 6 1 49 56 .816 Result I 15 15 32 62 .636 II 10 6 40 56 .698 III 4 6 46 56 •755 40 MARIE HACKL MEANS TABLE XI—Continued I Stimulus II Group III Number of Correct Responses IV V Number of Number Half-Correct of Responses Failures VI Total Number Tested VII Percentage of Failures Adroit I 19 2 40 61 .672 II 7 3 46 56 756 III 3 0 53 56 .817 Parsimony I 11 2 35 48 •750 II 7 0 49 56 .816 III 6 6 44 56 .823 Suave I 2 10 34 46 .847 II 2 9 45 56 .867 III 1 8 47 56 .882 Exoteric I 3 0 49 52 .942 II 1 0 55 56 .962 III 0 0 56 56 •975 Since the value of the stimuli was to be in terms of both ac¬ curacy and speed, the median time of all responses, both correct and half correct, was computed. The time score taken with a stop-watch represents fifths of a second. As in the case of the accuracy score, three time scores were reckoned for each word, the last score representing the combined time of all three groups. See column III, Table XII. The percent of failures of the pre¬ ceding table is copied in order to make clear the process by which the values of the several stimuli are reached. To arrive at the relative difficulty of the stimuli, the percen¬ tage of failures was multiplied by the median time. If credit is to be given for both measures, some more or less arbitrary scheme must be adopted. This procedure seemed justifiable since by it words of equal accuracy scores received different point values if the difference in time varied to any appreciable extent. This was precisely what was desired. Any method which covered up this difference was to be avoided. In order to determine a point value for the stimuli, the stan¬ dard deviation of the different values, as indicated in the fourth column of the following table, was obtained. Beginning at the average and calling it zero, steps of one-fifth standard deviation were then made, positively and negatively. These steps being STANDARDIZATION OF HARD OPPOSITES TEST 4i equal in a sense, the furtherest negative step was assigned a value of one, the next two, and so on up to twenty-six, the most difficult word in the list. These point values are given in the fifth column of Table XII. In selecting the stimuli for the completed test, those words which had received a hundred percent accuracy score were dis¬ carded, for thus we were better enabled to locate a zero point. Obviously we cannot find or infer the place where the ability to give opposites begins, as we can in the physical sciences for weight or length. Therefore the zero point will of necessity be an arbitrary one selected to represent very little of the trait in question. In eliminating all words with a hundred percent ac¬ curacy score, but retaining those next in difficulty, we assume that a college student who is unable to score a point on the test as it now stands has zero ability in naming opposites. Since we have located the distance from the median in terms of standard deviation, we have equally difficult steps. Roughly we assigned to the furtherest negative step a value of one. Since we are not concerned as were Woody 1 and Hering 2 with finding the dis¬ tances between different grades or groups of pupils, it seems impracticable to be too particular about the absolute point of the zero. In addition any error in slightly misplacing the point might be of less importance than that made in weighting time and accuracy. It is regretted that, especially in the case of the more difficult words, the measure of time is based on so small a number of records. As an indication of the reliability of the median time, we have computed the median deviation and coefficient of variability for each word. The probable error of the median is calculated 5 Q dis by the formula, P. E. med. = --. 3 4Vn 1 Woody, Clifford, “Measurements of Some Achievements in Arithmetic,” Col. Univ. Cont. Educ., 1916, LXXX. 2 Hering, John P., “Derivation of a Scale to Measure Abilities in Scientific Thinking,” Jour. Educ. Psychol., 1919, IX, 417-431. 3 Thorndike, E. L., “Mental and Social Measurements,” New York, 1916, p. 195- 42 MARIE HACKL MEANS TABLE XII The Point Value of the Stimuli Computed from Table IX VII I Stimulus II Percent of Failures III IV Median Relative Time Value V Point Value ' VI Mediar Deviatic Full .047 7 .329 •037 6 .222 .028 7 .209 1 1.48 Negative .040 7 .280 .047 7 •329 .030 7 .210 1 1.22 After .040 6 .240 .030 7 .210 •037 7 .259 . 1 1.20 Dim .066 8 .528 .072 7 •504 .089 7 .623 1 1.64 Blunt .047 6 .282 .084 7 .588 .091 7 .637 1 1.40 Success .063 6 .378 .071 7 •497 .108 6 .648 1 1.50 Pessimistic .040 6 .240 .080 6 .480 .108 6 .648 1 1.11 Joy .044 7 .308 . .068 8 •544 .086 8 .688 1 10 b CO Public •073 7 .511 .108 7 .756 4 .105 8 .840 2 2.03 Profit .126 7 .882 .101 8 .808 •139 8 1.112 2 2.26 Spend .198 10 1.980 .142 10 1.420 .122 10 1.220 2 2.80 Always .166 8 1.328 .188 8 1.504 .197 8 1.576 2 1.90 Graceful .169 10 1.690 .161 10 1.610 .161 10 1.610 2 3-36 VIII n of Error of Variability Median .21 •i 7 .17 •23 .20 •25 .18 .26 •25 .28 .28 .23 .14 .12 .11 •15 •13 .14 .11 .20 .19 .22 .27 .19 •33 •32 STANDARDIZATION OF HARD OPPOSITES TEST 43 TABLE XII—Continued I Stimulus II Percent of Failures HI IV Median Relative Time Value y vi vii viii Point Median Coefficient Probable Value Deviation of Error of Variability Median Strength •125 7 .875 • 175 8 1.400 .204 8 1.632 2 1-95 .24 .20 Ancient .258 8 2.064 .245 7-5 1.837 .218 8 1.744 3 2.30 •25 .22 Expand .201 9 1.809 •297 8 1.656 .227 8 1.816 3 1.28 .16 •13 Barbarous .080 11 .880 •153 11 1.683 .194 11 2.134 3 2.90 .26 .28 Hinder .213 9 1.917 .213 10 2.130 .227 10 2.270 3 348 •34 .36 Despondent .142 14 1.988 .172 12 2.064 .191 12 2.292 3 3-54 •29 •35 Vague .164 9 1.476 • 195 10 1.950 .231 11 2.541 3 3.23 •29 •34 Fertile .221 12 2.652 .250 12 3.000 .222 12 2.664 3 3-85 •32 .40 Doubtful •254 12 3.048 .238 12 2.858 .236 12 2.832 4 3-89 .32 .40 Injurious .220 11 2.420 .225 12 2.700 .249 12 2.988 4 447 •37 .46 Busy .388 10 3.880 •378 10 3.780 •340 9 3.060 4 2.64 •29 •30 Abstract .000 0 0.000 •357 9 3-213 •365 8.5 3.102 4 2.10 •24 •30 Advance .172 9 1.548 .280 9 2.520 •314 10 3.140 4 2.90 •29 •33 44 MARIE HACKL MEANS TABLE XII—Continued I Stimulus II Percent of Failures HI Median Time IV Relative Value V VI VII VIII Point Median Coefficient Probable Value Deviation of Error of Variability Median Foreign .207 12 2.484 .256 12 3.072 .272 12 3.264 4 3-8o •31 45 Create .231 10 2.310 .292 11 3.212 .296 11 3.256 4 3 . 3 i •30 .36 Simple 453 8 3.624 •343 10 3-430 .328 10 3.280 4 3.23 •32 .28 Extravagant .196 14 2.744 .196 15 2.940 .229 15 3435 4 4.14 •27 43 Aristocratic .205 12 2.460 .270 11 2.970 •316 11 3476 4 4.20 •38 46 Rare .262 14 3.668 .260 14 3-640 .262 14 3.668 4 3-39 .24 •35 Dangerous •303 11 3-333 .331 11 3-641 .346 11 3.806 4 3-75 •34 45 Slovenly .279 11 3-069 .342 12 4.104 .380 11 4.180 5 3-62 .32 •63 Defective .264 13 3432 .264 13 3432 .322 13 4.186 5 5-29 40 .58 Stingy .286 10 2.860 .322 12 3.864 •397 11 4467 5 3-34 •30 •37 Reveal •365 11 4.015 .378 11 4.158 •371 12 4452 5 f 3-83 .31 •44 Diligent .328 13 4.264 .304 13 3-952 • 35 i 13 4.563 5 3-97 •30 43 Join .300 14 4.200 .310 13 4.030 • 34 i 14 4-774 5 4.90 •35 .56 STANDARDIZATION OF HARD OPPOSITES TEST 45 TABLE XII—Continued I Stimulus II Percent of Failures HI Median Time IV Relative Value y vi vii viii Point Median Coefficient Probable Value Deviation of Error of Variability Median Impoverish 420 12 5.040 .492 11 5-412 .500 11 5-500 6 •341 •31 49 Permanent •373 13 4.849 ,462 13 6.006 •437 14 6.118 7 4.50 .32 •55 Elation .387 10 3.870 •521 12 6.262 .563 12 6.756 7 3-79 .31 •51 Sinful .411 13 5-343 .471 16 7.536 .488 14 6.832 7 5.10 .36 .64 Obnoxious .2 76 13 3.588 .402 14 5.628 .460 15 6.900 7 5.36 •35 .61 Conservative •413 11 4-543 .496 11 6.448 •533 13 6.929 7 4.30 •33 •53 Victorious .388 14 5-432 •451 15 6.765 .466 15 6.990 7 4-75 .31 .56 Obscure •491 12 5.892 .508 13 6.604 .505 14 7.070 7 6.06 43 •79 Proficient 476 11 5.236 .512 13 6.656 .551 13 7.163 8 545 .41 .65 Rigid 403 15 6.045 .469 15 7.035 .508 15 7.620 8 4-97 •33 •59 Repulsion •537 11 5.907 •5 77 13 7-501 .600 13 7.800 8 5-30 .40 •75 Imaginary •555 11 6.105 •581 12 6.972 .596 14 8-344 9 4.56 •32 .67 Permit .500 16 8.000 •504 16 8.064 .502 17 8.534 9 6.50 .38 .80 46 MARIE HACKL MEANS TABLE XII—Continued I Stimulus II Percent of Failures hi iv Median Relative Time Value V VI VII VIII Point Median Coefficient Probable Value Deviation of Error of Variability Median Orthodoxy .727 11 7-997 •783 11 8.613 .820 11 9.020 9 2.50 .22 •57 Analytical .482 16.5 7-953 .646 14 9.044 .698 14 9.772 10 4.27 •30 •74 Extrinsic .500 9 4-500 .706 17 12.000 .719 14 10.066 10 5.30 •37 1.17 Sacred • 55 i 15 8.265 .609 16 9-744 .647 16 10.352 11 6.17 .38 .91 Dynamic .820 12 9.840 .873 12 10.476 .867 12 10.404 11 3-75 • 3 i •93 Loquacious .612 15 9.180 •673 15 10.095 .689 16 11.024 11 5.08 • 3 i .76 Heterogeneou s -333 14-5 4.828 ■779 13 10.127 .782 15 11.730 12 4.88 •32 1.17 Spurious •755 14 10.570 .833 14 11.662 •857 14 11.998 12 5.50 •39 1.21 Disastrous .620 17 10.540 .650 19 12.350 •635 20 12.700 13 7.08 •35 1.10 Facility .766 16 12.256 .821 18 14.778 •853 16 13-648 14 7.00 •43 1.75 Pride .746 16 11.936 .786 17 13.362 .816 1 7 13.872 14 5-88 •34 1.20 Result .636 20 12.740 .699 22 I 5.378 •755 20 15.100 15 8.00 .40 1-33 Adroit .672 16 10.752 •756 17 12.852 .817 18.5 I 5 -II 4 15 6.23 •33 1-33 STANDARDIZATION OF HARD OPPOSITES TEST 47 TABLE XII—Continued I II III IV V VI VII VIII Percent Median Relative Point Median Coefficient Probable Stimulus of Time Value Value Deviation of Error of Failures Variability Median Parsimony 750 25 18.750 .816 22 17.952 .823 21 17.183 1 7 6.60 .31 1-45 Suave •847 24.5 20.751 .867 25 21.675 .882 25 22.050 22 9.17 •36 2.02 Exoteric .942 36 33.912 .962 27-5 26.455 •975 27.5 26.812 26 9-50 •34 5-93 VII. The Conversion into a Group Test In order to make it possible for even the inexperienced to use the test with facility, it was converted into a group test. Thus, obviously, much time in administering and labor in scoring were dispensed with. In addition, the personal element became almost negligible. The words were arranged in the ascending order of difficulty and six minutes allowed for the test. This time limit was determined empirically, after experimenting upon a dozen or more students at the George Peabody College and some sixty students at the North Georgia Agricultural College. 1 In arriving at the time to be allowed, two considerations were held in mind. It was desired that sufficient time be given for individual scores to cover a considerable range, and thus avoid a large undistributed group of poor records. On the other hand the time should be so planned that a perfect score is impossible. Two individuals with a perfect score might vary in efficiency, but under the conditions of the test no measure of their difference could be obtained. The test sheets were printed upon a good quality of white paper, eight and a half by eleven inches. The directions printed on the back of the sheet explained to the experimenter as well as to the subject, exactly what was to be done. A copy is included in the Appendix. By mistake the word “rare” was placed be¬ tween “defective” and stingy.” It should follow “aristocratic.” In scoring the papers, however, proper credit was assigned the word. The displacement was so slight that it is highly improb¬ able that the final results were at all affected. Three thousand copies of the test were sent to psychologists in twenty different normal schools, colleges, and universities throughout the country. Of this number, one thousand six hun¬ dred and twenty-eight copies were returned for grading from 1 These results, which were obtained through the kindness of Professor George Camp, are not included in the norms because the sheets were mimeo¬ graphed instead of printed. STANDARDIZATION OF HARD OPPOSITES TEST 49 fourteen widely different institutions. Unfortunately, a number of papers arrived too late to be included in the norms herein pre¬ sented. The table below indicates the number of subjects tested in each school and the class to which both the men and women then belonged. TABLE XIII The Number of Men and Women Tested in Each School and Class in Which They Were Enrolled Freshmen Soph’s Juniors Seniors Graduates Total Women Men Women Men Women Men Women Men Women Men Wellesley College 43 95 138 Brooklyn Training School University 148 5 153 of Illinois University 4 2 44 46 60 34 35 16 2 243 of Montana Alabama 23 2 10 5 24 1 66 State Normal University 50 50 of Chicago Teachers’ 2 6 40 74 7 13 20 9 2 173 College Vanderbilt 2 13 2 9 28 54 University University 1 I 12 14 of Kansas 1 32 27 11 6 3 4 I 85 University of Tennessee University I 25 19 7 2 4 1 2 60 of Louisiana Fitchburg 2 1 14 9 5 14 8 4 I 58 Normal School Kansas 54 1 7 26 17 2 116 State Normal Ohio State 49 6 38 I 15 3 112 University 59 63 40 58 33 22 14 13 2 2 306 Total 319 101 375 237 262 102 121 5 i l6 44 1628 The final results of the sixteen hundred records indicated class variations as well as slight sex differences. In order to ascertain a possible cause, the writer has computed the percentage of stu¬ dents of both sexes in each of the classes. This distribution is shown in the following table. In grading the papers two classes of errors were encountered, namely, the misspelled word, and the abbreviated word. In the former case full credit was allowed whenever the spelling clearly 50 MARIE HACKL MEANS TABLE XIV The Percentage of Students of Both Sexes in Each of the Classes Freshmen Sophomores Juniors Seniors Graduates Women .291 •343 .239 .110 .014 Men .188 •440 .190 .095 .082 Both Sexes .27 •37 .22 .10 •03 showed that an acceptable word was meant. However, when a word similar in sound was correctly spelled no credit was given. As an example, “save” was not accepted as an opposite of “dan¬ gerous/' but “richous” (actual case) as an opposite of “sinful” was given full credit. This rule was adopted because without it gradings by different individuals would vary considerably. Where the response was abbreviated it was given no credit. The only two abbreviations were “opt” as the opposite of pes¬ simistic and “dem” as opposite of aristocratic. In both cases the subject might have intended to write “optimist” and “demo¬ crat,” both of which would have been scored a failure. Even had he had in mind the adjective instead of the noun, his score would have been raised only five points. No doubt in abbreviat¬ ing the words, he was thus enabled, at an unfair advantage to others, to write the opposite of a more difficult word. In view of the fact that a time limit was placed on the test, not to penalize abbreviations seems unjust to those students who have followed the directions carefully. Fortunately the number of students abbreviating was negligible. The sum of the point values of the stimuli as indicated in Table XII is four hundred and thirty-eight. Since each response may receive the assigned value, one-half that value, or zero, almost any score between zero and this number is possible. This wide range of distribution eliminates the probability of a large group of subjects of different ability receiving identical scores. The actual scores based on the records obtained varied from zero to three hundred and fifty-one. VIII. The Establishment of Norms for College Students Sixteen hundred and twenty-eight test papers were graded and the norms for college students by the group test were based on these records. In order to ascertain the form of the distribution of the scores made by the several classes, steps of fifty beginning with zero were made and the percent of students included between these steps was estimated. Examination of Table XV reveals the fact that the largest percentage of scores falls between one hundred and one hundred fifty. This is perhaps more clearly shown in the graphs (Figures 2, 3, 4, 5, 6) based on this table. TABLE XV Percent of Students of Several Classes Attaining Scores Within the Indicated Limits Score Freshmen Sophomores Juniors Seniors Graduates 0 - 49-5 .069 •033 .016 .005 .000 50 - 99-5 .356 .272 .190 t—1 00 .066 100-149.5 •397 .346 .318 .406 •300 I50-I99-5 .124 .225 .247 .214 .250 200-249.5 •045 .078 .156 .111 .163 250-299.5 .007 .032 .063 .047 •133 300-349.5 .002 .014 .010 .028 .086 350-399.5 .000 .000 .000 .005 .000 The norms for college students are presented in the form of percentile tables because this method, statistically simple, admits of convenient use and permits comparisons to be made with other tests, similarly treated. Let us suppose that a directions test has been standardized and the norms compiled in terms of percentiles. Scores made in the opposites test can be compared with scores made in the directions test when both are converted into percentiles. The two percentiles can be averaged or the median computed. This is statistically impossible when the two scores are given in terms of the score in points made on 52 MARIE HACKL MEANS o •* © ■o to o o to o - n. £ w o CO CO as • 4 _> as u T 3 C to 4 ) l* o as • to OS n Kr c —■ bCiii rn C £ 2 a> as 3 ^ u>ri ri, oO as 00 u dj as x J C 5 ' M as >« E c Oj o as S c >- 03 OS ■ 4—1 u as > V to as v H to 4 ) i- o S o o 5 to n 0 u as £> u as *0 4 ) to .5 OS 04 8 8 Jg ^ U £:=: © Vh be as c as OS - 4 —» as > as 4 C H as as as u * STANDARDIZATION OF HARD OPPOSITES TEST 53 o ■ft to § to o o CO CD be cd •*-> G G ~a c/D G £ C/3 o CD r| U "lO O gg cd c/3 OJ u 3 be P* O *s u o CD o T-t G c/D u O o to 03 a v > OJ I—< H G 3 s o m to o o to o •o ct CD to cd 8 ° a 3 a « -*-* > DG _ .G +-> o G cd to * CJ t 4 «U £! G « c/D g Vh — o T-t rtf • -h O C o to Ih CD b* ' CD -C H CD to 54 MARIE HACKL MEANS 90 80 80 10 10 0 50 100 150 200 230 300 350 Figure 6 The vertical line in the graph shows the percentage of graduates receiving the scores indicated on the base line. each test. Obviously, if the possible grades on the opposites test range from zero to four hundred and thirty-eight and on the hypothetical directions test from zero to sixteen, to add or average the two scores made by any individual would practically amount to weighting the opposites test. But the percentile scores can be averaged and the final result will represent a com¬ bination of the two tests to which equal credit is given. In order to make clearer the meaning of the percentile table the following explanation of the manner in which it was derived is offered. For instance in the case of the freshmen, as indicated in Table XVI, all scores were arranged in numerical order, be¬ ginning with the lowest and ending with the highest. The num¬ ber of scores was ascertained, which in this case was four hun¬ dred and twenty. Ten percent of the number of scores was counted off, beginning with the lowest score. The score upon which the forty second count fell became the tenth percentile. A STANDARDIZATION OF HARD OPPOSITES TEST 55 glance at the table will show that in this instance it was fifty- eight and a half. This means that this grade is not exceeded by the lowest ten percent of the freshmen, and that any student making this grade is equalled or surpassed by ninety percent of the college students. An additional ten percent is counted off to secure the next higher percentile, and so on. TABLE XVI Norms for College Students in Terms of Percentile Scores in the Group Test Percentile Freshmen Sophomores Juniors Seniors Graduates All Comb’d IO 58.5 66.5 77.0 84.0 107.0 68.5 20 74-5 84.0 97.0 100.0 124.0 78.0 30 87.0 99.0 116.0 116.5 138.5 IOI.O 40 97-0 112.0 130.5 125.5 151.0 114-5 50 107.0 125.0 146.0 137.5 155.5 128.5 6o 120.0 139.9 161.5 147.0 I 9 I -5 142.0 70 132.5 156.0 181.0 166.0 200.0 157.0 8o 147.0 178.5 203.0 187.5 247.0 181.5 90 172.0 214.0 239-0 231.0 298.0 220.5 100 303.5 338.0 3130 35 i.o 3 i 3 .o 3510 Number Tested 420 612 364 172 60 1628 Frequently comparisons of different groups are stated in terms of the percent of individuals of the first group reaching or ex¬ ceeding the twenty-five or seventy-five percentile of the second group. In the table below the twenty-five, fifty, and seventy-five percentiles are indicated for each class. TABLE XVII Twenty-five, Fifty, and Seventy-five Percentile Scores for Each Class Percentile Freshmen Sophomores Juniors Seniors Graduates All Comb’d 25 82.5 92.5 106.0 wd8.o 136.0 89.5 50 107.0 125.0 146.0 137.5 155-5 128.5 75 139.0 165.5 192.0 173-0 213.0 169.0 The sophomores surpass the freshmen and they are in turn outstripped by the juniors. The gap between seniors and gradu¬ ates is wide; but the former are little if at all superior to the juniors. A consideration of the lowest and highest score made by the several classes reveals the fact that although ten percent of 56 MARIE HACKL MEANS the juniors barely reach the lowest score made by the graduates, still the highest score attained by each class is identical. In Table XVIII is shown the gradual rise of the lowest score real¬ ized by the different classes, contrasted with the irregularity of the highest score. TABLE XVIII Table Showing the Highest and Lowest Score Made by Each Class Freshmen Sophomores Juniors Seniors Graduates Lowest Score . o 18.5 38.0 46.0 75.0 Highest Score . 303-5 338-0 313.0 351.0 313.0 Although men are slightly superior in the test, to women belongs the highest as well as the lowest score. The greatest sex difference is to be found in the fortieth and fiftieth percentiles which in each case are surpassed by the next lowest percentile secured by the men. The table below shows the ten percentiles for both sexes and the number tested in each case. It also in¬ cludes, for the benefit of those interested in sex differences, the more commonly employed twenty-five and seventy-five per¬ centiles. TABLE XIX Percentile Scores for Men and Women Percentiles Men Women 0 10 72.0 60.0 20 92-5 85.5 25 99.0 92.0 30 106.0 98.5 40 122.5 IOI.O 50 133-5 1150 60 147-0 139.0 70 163.5 155.5 75 176.0 167.0 80 189.0 178.0 90 221.0 222.0 100 338.0 351-0 Number Tested 535 1093 A consideration of the range of scores made by both sexes would seem to indicate that women have a wider range than men in the trait in question. Whereas men range only from i8 -5 t0 338, women beginning with an initial score of o reach 351 points as their highest score. IX. Interpretation of the Results Whether the test is, as Simpson 1 believes, a selective thinking test, we cannot from the very nature of the data conclude. How¬ ever, we feel justified in assuming that it is a test success in which is dependent upon native ability rather than number of years schooling. The higher scores attained by the graduates confirm rather than refute the contention, for although there is a constant process of weeding out the mentally inferior, selection operates particularly at the conclusion of work offered for the bachelor's degree. The graphs (figures 2, 3, 4, 5, and 6) reveal a considerable amount of overlapping from year to year. Be¬ tween ten and twenty percent of the freshmen reach the median score of the graduates and practically thirty percent, the median score of the seniors. Not without significance is the range of scores of the several classes. An examination of Table XVIII discloses the fact that the highest individual score obtained by either a junior or gradu¬ ate is surpassed by a sophomore. This condition is not to be at¬ tributed to the simplicity of the test, which might admit of a number of nearly perfect records. On the contrary three hundred and fifty-one is the highest score reached, whereas a possible four hundred and thirty-eight could be attained in the allotted time were an individual familiar with the opposites. The lowest score made by the several classes ascends regularly from fresh¬ men to graduates, all of which seems to indicate that instead of the scores progressing in absolute value as the years in school increase, the limit of attainment is about reached in the freshmen year, and thereafter a dropping off of the poorer students raises the percentage of high scores made by the more advanced students. The form of the distribution of the scores, skewed as it is to 1 Simpson, Benjamin R., Op. cit. 58 MARIE HACKL MEANS the lower end, indicates that the time allotted is too short. Had eight minutes instead of six been allowed for the test, the scores would have been scattered over a wider area and the form of distribution would probably have approached the normal. As it stands the test provides a large number of possible scores which can be attained by the college student. In addition, because of its difficulty, it offers an opportunity of testing a more highly selected group. Sex differences favor the male. King and M’Crory 2 found that the same condition obtained with the hard opposites test they used. In the present investigation twice as many women were tested as men. An equal number might alter the situation. This difference might be due to the fact that the normal schools largely attended by women were slightly inferior in the test to the colleges and universities. In addition, proportionally more women belonged to the freshman class and fewer were enrolled in the graduate school, as is indicated in Table XIV. Until an equal number of unselected cases from each sex is tested, the subject is debatable. 2 King, Irving, and M’Crory, J. L., “Freshmen Tests at the State Uni¬ versity of Iowa,” Jour. Educ. Psychol., 1918, IX. X. Suggestions and Directions for the Use of the Opposites Test The following suggestions are recommended for using the test: Distribute the test sheets with the face downward and read aloud the written directions found on the back of the sheets. In timing the class, make use of a stop-watch. Allowing the point value of the word for correct responses and half that value for half correct responses, grade the papers according to the acceptable responses presented in Table X. Add the points and record this sum as the individual’s score. If an entire class is to be compared with the norms herein contained, compute the median score of the class and compare with the fifty percentile to ascertain whether the class is above or below standard. The ratio between the percentile score of the individual or of the class and the fifty percentile of the group may be taken as index of brightness. Frequently it is necessary to have more than one list of oppo¬ sites. To meet such a contingency, the list has been divided so that both parts contain an equal number of points and words. The lists are given below, arranged in the order of difficulty. The assigned point value is at the left of the stimulus. The writer feels that the Opposites Test as a group test has sacrificed to time an important element in that it fails to detect certain aspects of the subject’s mind, aspects upon which per¬ sonal efficiency is so largely dependent. Most significant and enlightening i Full I Dim i Blunt 1 Pessimistic 2 Public 2 Spend 2 Graceful 3 Ancient 3 Barbarous \ is the attitude of the subject toward the test. Despondent Doubtful Busy Abstract Foreign Simple Extravagant Dangerous Defective LIST I 5 Reveal 5 Join 7 Permanent 7 Obnoxious 7 Victorious 8 Proficient 8 Rigid 9 Orthodoxy 10 Analytical 11 Loquacious 11 Sacred 12 Spurious 14 Facility 14 Pride 17 Parsimony 22 Suave 6o MARIE HACKL MEANS LIST II 1 Negative 3 Vague 5 Diligent 9 Permit 1 After 3 Fertile 6 Impoverish 10 Extrinsic 1 Success 4 Injurious 7 Elation 11 Dynamic 1 Joy 4 Advance .7 Sinful 12 Heterogeneous 2 Profit 4 Create 7 Conservative 13 Disastrous 2 Always 4 Aristocratic 7 Obscure 15 Result 2 Strength 4 Rare 8 Repulsion 15 Adroit 3 Expand 3 Hinder 5 Slovenly 5 Stingy 9 Imaginary 26 Exoteric Whether these attitudes, which Ruger 1 has entitled subjective and objective, so easily perceived by the experimenter, carry over into other lines of work, while highly probable, is a matter that yet remains to be proved. The poor record in the Opposites Test does not necessarily indicate the subjective attitude but an exceptionally good record does more probably indicate the absence of it. The student be¬ comes easily confused and his attention is divided between the opposite to be given and fear of a bad showing. The score goes low on account of frequent interruptions of which the following are typical: “I know the opposite but I can’t think of it right now,” “I can’t think of the opposite but I know the word per¬ fectly,” “This is a vocabulary test and my vocabulary is limited.” This type of subject seldom admits that he is familiar neither with the word nor the opposite. He prefers in the individual test to waste the allotted ten seconds attempting to create the impression that he knows the opposite but this time he can not quite get it. Frequently he forgets the present stimulus in re¬ gretting the one just missed. However if time is wasted in administering the test, it is more than counterbalanced by the speed with which he makes his exit. He does not tarry to ask about some of the words he has failed on, nor is he curious about the records of others. With a hasty explanation of his own inefficiency he hurries from the room, leaving the experimenter with a vague sort of feeling that the latter has purposely inflicted a mortal injury. In striking contrast is the subject whose interest is centered upon the test itself. He works rapidly offering neither apologies 1 Ruger, H. A., “The Psychology of Efficiency, A rchiv. of Psychol., 1910, STANDARDIZATION OF HARD OPPOSITES TEST 61 nor explanations. Unashamed he denies knowledge of some of the more difficult words, but rarely does he depart without this knowledge. He is eager to know what others are able to do under the same conditions. Interest in self is overshadowed by interest in the problem in hand. Peterson, 2 with his Rational Learning Test, notes these same attitudes, and the writer knows of no better test in which to ob¬ serve them, for not only are they evident during the administra¬ tion of the test, but the traits in question can be easily detected in the individual records. Determination of the extent to which the opposites test reveals native ability, is not attempted in this research. It is hoped that someone will carry on the work here begun, and by a series of correlations empirically determine the scope and limitation of the test. The writer is of the opinion, that if used with other stan¬ dardized tests, the Opposites Test will be found of value in mental diagnosis. 2 Peterson, Joseph, “Experiments in Rational Learning,” Psychol. Review> 1918, 25, 462-463. XL Summary and Conclusions I. The Hard Opposites Test herein presented enables the instructor within a brief period to examine a large number of students. Even the most inexperienced will find little difficulty in evaluating the papers, so statistically simple is the test. II. Initial experimentation began with all words previously employed but eventually the list was narrowed so as to include only fifty-four of these stimuli. Similarly the fourteen original words, for the most part of greater difficulty, were selected from a long list which was subjected to the same process of investiga¬ tion. III. Subjects were tested both orally and individually so the experimenter might study each stimulus from the standpoint of both time and accuracy. Records were kept of each individual response and reaction time. On the basis of these records the relative difficulty of the words was computed. The calculations are based on at least one hundred records for each word, varying up to one hundred and eighty-seven for others. IV. The acceptable responses were chosen by five judges, from all those responses given by the subjects experimented upon. Additional responses which suggested themselves to each judge were recorded and in turn passed on by the others. Effort was made to have the responses include every possible opposite, not merely the most common ones. These judgments were averaged and a value of o, or i was accorded each response. V. The relative difficulty of the words was determined by multiplying the median time of all correct and half-correct re¬ sponses by the percentage of failures, allowing one-half a point for half-correct responses. These values were then converted into standard deviation and steps of 1/5 standard deviation were marked off. Having selected for the easiest word, a word which for college students approached zero difficulty, we assigned to it STANDARDIZATION OF HARD OPPOSITES TEST 63 a value of one, to the next step a value of two. The hardest word in the list received a value of twenty-six. VI. For convenient use the test was converted into a group test. The stimuli were arranged in the ascending order of dif¬ ficulty and printed on a good qualtiy of white paper. On the back of the sheet, directions make clear to both the subject and instructor what is to be done. Three thousand copies of the test were mailed to psychologists in normal schools, colleges, and universities throughout the country. From this number the records of 1628 students were obtained, on the basis of which norms for college students were established. These standards are presented in terms of percentiles for each class, including graduates, and for each sex. VII. The list of stimuli has been divided into an equal num¬ ber of words and points, for the convenience of any experi¬ menter who may desire two lists rather than one. As it stands, the test can be easily incorporated into a group of tests and com¬ parisons made, provided the other tests are presented in terms of percentiles. The test, because of its large number of possible points and its difficulty, offers an opportunity of testing ade¬ quately a less highly selected group than that represented by col¬ lege students. VIII. The results of this investigation tend to show that suc¬ cess in the test is dependent upon native ability rather than years of schooling. This conclusion seems justifiable in view of, first, the tremendous amount of overlapping among the several classes, and, second, the gradual rise of the lowest scores from year to year while the highest scores remain constant. The fact that graduates are superior to seniors, and sophomores to freshmen is to a considerable extent to be attributed to the operation of the law of selection. 64 MARIE HACKL MEANS APPENDIX Name. School.-. Sex. Underscore the class of which you are a member: Freshmen; Sophomore; Junior; Senior; Graduate. Directions for Giving the Test Before distributing the test blanks, request the students not to turn over the sheets until the signal is given. Ask them to fill in the blank spaces at the top of the page and then make clear what is to be done by reading aloud the following directions: On the other side of this page are a number of words beside each of Which you are to write as quickly as possible the exact opposite. For instance if the word “Black” occurs you should write White.” The opposite you write must belong to the same part of speech as the word in the list. Phrases and words formed by prefixing “Non” are counted wrong. Begin at the top and work downward, but in case you do not know the opposites, pass on to the next word, and later, if you have time, come back to those you have omitted. You will be given just six minutes, so when the Instructor says “Ready” be prepared to write, and when he says “Go” turn over the sheet and begin. Continue until the signal is given to stop. Full Busy Obscure Negative Abstract Proficient After Advance Rigid Dim Foreign Repulsion Blunt Create Imaginary Success Simple Permit Pessimistic Extravagant Orthodoxy Joy Aristocratic Analytical Public Rare Extrinsic Profit Dangerous Sacred Spend Slovenly Dynamic Always Defective Loquacious Graceful Stingy Heterogeneous Strength Reveal Spurious Ancient Diligent Disastrous Expand Join Facility Barbarous Impoverish Pride Hinder Permanent Result Despondent Elation Adroit Vague Sinful Parsimony Fertile Obnoxious Suave Doubtful Conservative Exoteric Injurious Victorious BIBLIOGRAPHY Bonser, Frederick G., “The Reasoning Ability of Children of the Fourth, Fifth, and Sixth School Grades,” Col. Univ. Cont. Educ., 1906, XXXVII. Bronner, A. F., “A Comparative Study of the Intelligence of Delinquent Girls,” Col. Univ. Cont. Educ., 1914, LXVIII. Greene, Harry A., “A Standardization of Certain Opposites Tests,” Jour. Educ. Psychol., 1918, IX, 559-566. Hering, John P., “Derivation of a Scale to Measure Abilities in Scientific Thinking,” Jour. Educ. Psychol., 1919, IX, 417- 43 1 - King, Irving, and Gold, Hugo, “A Tentative Standardization of Certain Opposites Tests,” Jour. Educ. Psychol., 1916, vii, 459-482. King, Irving, and M’Crory, J. L., “Freshman Tests at the State University of Iowa,” Jour. Educ. Psychol., 1918, IX, 32-46. Kitson, H. D., “The Scientific Study of College Students,” Psychol. Monog., 1917, XXIII. Norsworthy, Naomi, “The Psychology of Mentally Deficient Children,” New York, Columbia University, 1906. Peterson, Joseph, “Experiments in Rational Learning," Psychol. Review, 1918, XXV, 462-463. Rosenow, Curt, “The Analysis of Mental Functions,” Psychol. Monog., 1917, XXIV. Ruger, H. A., “The Psychology of Efficiency," Archiv. of Psychol., 1910, XV. Simpson, B. R., “Correlations of Mental Abilities,” Col. Univ. Cont. Educ., 1912, LIII. Thorndike, E. L., “Mental and Social Measurements,” New York, 1916. Weidensall, Jean, “The Mentality of the Criminal Woman,” Baltimore, 1916. Woodworth, R. S., and Wells, F. L., “Association Tests,” Psychol. Monog., 1911, LVII. Woody, Clifford, “Measurements of Some Achievements in Arithmetic," Col. Univ. Cont. Educ., 1916, LXXX. Woolley, Helen Thompson, and Fisher, Charlotte Rust, “Mental and Physical Measurements of Working Children,” Psychol. Monog., 1914, LXXVII.