Creative Education
2013. Vol.4, No.5, 322-334
Published Online May 2013 in SciRes (http://www.scirp.org/journal/ce) http://dx.doi.org/10.4236/ce.2013.45048
322
Validating a Research-Based Monograph for Teaching
Post-Secondary EFL Reading Teachers the Meta-Cognitive
Aspects of How to Teach Summarizing Strategies for Expository
Text: Phase II of a Harvard Business School Type Case Study
Wei Xu1, James Carifio2, Lorraine Dagostino2
1College of International Education, Shanghai International Studies University, Shanghai, China
2School of Education, University of Massachusetts, Lowell, USA
Received March 19th, 2013; revised April 20th, 2013; accepted May 5th, 2013
Copyright © 2013 Wei Xu et al. This is an open access article distributed under the Creative Commons Attribu-
tion License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original
work is properly cited.
This article reports the results of the second phase of a Harvard Business School type case study on the
evaluation of a comprehensive research-based English language monograph for teaching Chinese EFL
reading teachers the metacognitive aspects of how to teach summarizing strategies for English language
expository texts to Chinese undergraduate students. This monograph could be used by native English
speaking EFL teachers to teach native English speaking students the same skills, but the focus of this
study was on the bilingual and bicultural aspects of such a monograph (text) and its development as a
general model of such cross-language and cross-culture instructional materials development problems
which are becoming increasingly more prevalent now and are a harbinger of the future of instructional
materials. A cross-panel replicated expert reviewer (native Chinese EFL practitioners and academics) de-
sign was used to validate the monograph developed using the Carifio-Perla instructional materials devel-
opment model as a guide. The expert reviewers used a 30-item previously validated structured responding
protocol that reflected 7 evaluative criteria and 4 open-ended responding questions to review and rate the
monograph chapter by chapter and then again for all 8 chapters. The reviewers unanimously agreed that
the general Metacognitive Knowledge Framework, devised as a result of the literature reviews, analyses
done, and numerous problems identified in Phase I of this study concerning views, definitions and strate-
gies for analyzing and teaching summarizing strategies metacognitively, was appropriately constructed
and effectively communicated and represented in The Monograph for the target audiences. The uniformly
positive ratings by the two expert panels validated the high quality and consistency of the monograph in
terms of the 7 evaluative criteria used. These results also showed aspects of skills, knowledge, under-
standings, and metacognitions both transcend and can be represented and communicated successfully
across languages and cultures and to different professional audiences as well.
Keywords: Instructional Materials Development Models; Metacognition; Metacognitive Knowledge;
Bilingual and Bi-Cultural Instructional Contexts; Summarizing Strategies Instruction;
Expository Text; Teacher Training
Introduction
When one teaches anyone something such as domain-based
knowledge, skills, strategies, procedures, or sensibilities (i.e.,
values or attitudes), one should have a fairly complete and
well-conceptualized schema and understanding of the domain
in question, and one must analyzed the contexts and events of
instruction and make decisions as to what to do and how to
proceed, both before the fact and while the teaching (instruction)
is occurring. More specifically, one must have metacognitive
knowledge of 1) the domain to be taught, 2) the instructional
pedagogies to be used or adapted to achieve the goals of in-
struction one has set, 3) the students’ to be taught metacogni-
tive skills, processes and knowledge, and 4) one’s own personal
metacognitive skills, processes, and knowledge. Teaching is not
a simple and uncomplicated act and it is usually more complex
and complicated than learning. Teaching someone to teach
something expotentiates the complexity and complications even
more, as does having systems, models and theories for each of
the four frames just identified to track and keep clear these four
frames and not “jumble them”, as very often happens at both
the individual and professional level, and in scholarly and in-
structional writings about these four frames, which in turn pro-
duces many problems and confusions. Further, it should be
clearly noted that metacognition, metacognitive processes and
metacognitive knowledge of various kinds are typically re-
quired and typically occur in each of the four frames identified
above for various event and outcomes to occur successfully.
However, the concept of metacognition itself is a complex
and fuzzy construct that varies from theorist to theorist and

W. XU ET AL.
field to field as well as from subject-matter area to subject-
matter area. So the discussion here will be brief and directed at
establishing some key points and the reader will be referred to
other sources for more detailed explication of the points made
in this article. A concept as complex and fuzzy as metacogni-
tion demands that beyond such commonly acceptable definition
as “thinking about thinking” or “cognition about cognition”,
researchers should elaborate the construct theoretically and
practically with regard to different content domains, various
situations, and specific processes to make the construct more
useful, concrete and grounded relative to specific pedagogical
contexts and situations. Flavell’s (1979) seminal metacognition
model, it should be noted, includes two kinds of metacognition.
The first kind is metacognitive knowledge about self, task, and
strategies and the second kind of metacognition is metacogni-
tive experience or experiencing, which are “items of metacog-
nitive knowledge that have entered consciousness” (p. 908).
Based on Flavell’s metacognition model (1979), various studies
have been carried out to explore the diverse functions metacog-
nition serves in different content (subject matter) areas. More
specifically, in the field of language arts and literacy, a re-
search-based belief has been established that metacognition can
be taught and learned (e.g. Flavell, 1979; Pressley & Woloshyn,
1995; Lovett, 2008; Lai, 2011) on the bases of such research
evidence as the key role of metacognition in successful reading
(e.g. Dagostino & Carifio, 1994a, 1994b; Brown et al., 1981;
Brown, 1985; Baker & Brown, 1984), and the increasingly
accumulated knowledge about what “the metacognitively so-
phisticated reader” is (Pressley, 2002).
With this understanding, reading teachers have been called
upon to promote metacognition in students, or in Pressley’s
(2002) term, to be “the metacognitively sophisticated teacher”.
This view is in accordance with Hartman (2001) call “to teach
for metacognition”, referring to teachers’ thinking about “how
their instruction will activate and develop their students’ meta-
cognition, or think about their own thinking as learners” (p.
149). This view and call is essentially focused on component
three in the four component framework described at the begin-
ning of this article. Teachers have also been expected to teach
with metacognition; that is, teachers should think about their
own thinking regarding their teaching before, during, and after
conducting lessons to increase instructional effectiveness (Hart-
man, 2001). This view, definition, and focus on metacognition
is essentially a focus on component four only in the four com-
ponent framework previously-described.
Flavell also contends that his three kinds of metacognitive
knowledge are the basis for particular metacognitive processes
and experiences (e.g. Flavell, 1979; Garner, 1987). Thus for
Flavell, metacognitive knowledge and metacognitive experi-
ences are joined and dependent on each other with metacogni-
tive knowledge being the gateway variable to specific meta-
cognitive experiences and processes for both teachers and stu-
dents. Therefore, it is important for teachers to acquire enough
metacognitive knowledge if they are expected to teach for
and/or with metacognition to have the metacognitive experi-
ences and processes they need to have to achieve the desired
goal for and/or with metacognition.
But there is another rub. All of the above four components of
metacognitive knowledge are nevertheless complex in their
own right and need to be understood both individually and rela-
tionally to teach or learn something well, as well as to teach
someone how to teach other (pre or in-service) teachers the
different elements of a given domain (i.e., knowledge, skills,
strategies, procedures, or sensibilities), once they have suffi-
ciently mastered them to do so to the targeted goals or levels set.
This latter task involves components 2), 3), and 4) above, each
of which involves or is heavily laden with various metacogni-
tive processes, skills, and knowledge of one kind or another that
are relevant to the tasks to be accomplished successfully in each
component and across components by the teacher, the student
or the teacher of teachers depending on the frame and compo-
nent one is considering. It would seem logical, then, to both
define and discuss metacognition and metacognitive knowledge,
skills and processes, since they are so central to learning and
teaching and teaching (pre or in-service) teachers about teach-
ing some domain or one of its elements, or subelements such as
strategies for summarizing expository text, the focus and sub-
ject of this article and study.
Unfortunately, only preliminary and formative work has been
done along these lines in education and in teacher training and
professional development for (pre or in-service) teachers. In
addition, another problem is that Flavell and many of who built
upon his work and model both jumble and confound the four
components or frames associated with metacognition described
at the beginning of this work, and the two types of metacogni-
tion Flavell identified (kinds of knowledge and experiences),
and it is often more than just difficult to disentangle the four
components and two types clearly in these works, which is a
source of various confusions, misunderstandings, and miscom-
munications observed in this literature as it relates to both tea-
cher and student metacognitions. It is for this reason that we use
our “4 by 2” (components or frames by metacognition types)
model to classify views and assertions to aid clarification and
communication about this fuzzy construct and area.
The various models and theories used in our work, including
the general one for guiding the development and production of
a research-based monograph for teaching (pre or in-service)
teachers how to effectively teach strategies for summarizing
expository text (see Carifio & Perla, 2010; Xu, Carifio, & Da-
gostino, 2012, for details), are heuristic frameworks, aids and
guides for understanding and conceptualizing the four compo-
nents or frames of metacognition identified above, both indi-
vidually and in relationship to each other, and for keeping them
more distinguishable from each other for in terms of both ex-
plication and discussion. More will be said on this point below.
Prior Research
Limited research has been conducted to explore what meta-
cognitive knowledge reading teachers should have so as to be
able to teach for and/or with metacognition and to be able to
judge when, how, and whether or not to teach metacognitively.
Such explorations are especially rare for Chinese post-secon-
dary in-service reading teachers in China. Most of the discus-
sions on teacher knowledge in China have been conducted in
terms of domain-based knowledge teachers should have, or
component one only in the four component framework de-
scribed at the beginning of this article. Next, as a pioneer in
educational reform oriented studies on teacher knowledge,
Shulman (1986, 1987) conceived that a knowledge base for
teaching should be an amalgam of knowledge, skills, and dis-
positions that underlies the capacity to teach effectively, and he
developed a typology to characterize teacher knowledge that
other researchers have used, expanded, and refined later (e.g.,
Cochran et al., 1993; Grossman & Richert, 1988; Grossman,

W. XU ET AL.
1990; Gudmundsdottir, 1991). Shulman’s view is essentially
component two in the four component framework described at
the beginning of this article. Discussions of a knowledge base
in TESOL, always nested in reviewing the general knowledge
base of teacher education, have been fostered from two tradi-
tions: one is from the theoretical perspective, attempting to
“identify and develop the knowledge teachers need to demon-
strate,” and the other, the practice and policy perspective, in-
tending to “enhance the teacher preparation processes and
teaching practices” (Fradd & Lee, 1998). In a recent study that
examined the degree to which the requirements in teacher edu-
cation programs in the United States reflected current theory
and practice for teachers of English language learners (ELLs) in
their coursework, each state’s teacher preparation policies and
requirements were still focused only on cognitive level of sub-
ject-matter knowledge, knowledge of pedagogy, knowledge of
linguistics, and knowledge of cultural and linguistic diversity,
without even a mention of metacognition or teaching metacog-
nitively (see Lopez et al., 2013, for details). Furthermore, Fen-
stermacher (1994) tried to distinguish two major types of
knowledge about teaching: formal knowledge and practical
knowledge. “Formal knowledge”, described as knowledge for
teachers, is the knowledge that is primarily known and pro-
duced by researchers. “Practical knowledge”, referred to as
knowledge of teachers, is the knowledge that is principally
known and produced by teachers themselves. In this article, the
focus of attention is on what Fenstermacher noted, teachers’
formal knowledge, but from the metacognitive perspective and
in the area of reading strategies instruction.
At the metacognitive level, Pressley (2002) hypothesized that
the metacognitively sophisticated reading teacher should know
“what good readers know…” (p. 305) and can teach compre-
hension strategies when needed (p. 306). The question that
logically and obviously follows from Pressley’s assertion, then,
is “What metacognitive knowledge and metacognitions do
reading teachers need to acquire specifically in order to teach
metacognitively?” Our work and study investigated this ques-
tion in detail with regard to a particular group of reading teach-
ers’ (post-secondary in-service EFL teachers), specific teaching
tasks (summarizing strategies for expository text) and a specific
group of students (EFL undergraduates).
Another view of metacognition is that metacognitive knowl-
edge and Flavell-like metacognitive experiences are used to
control the cognitive processes used to achieve the goals of a
specific cognitive task (Garner, 1987; Griffith & Ruan, 2005),
while other kinds of knowledge, such as automatic functioning
and automatic processing that occur in complex, well-learned
and habit-driven cognitive activities like reading and studying a
textbook chapter, do not have such “controlling influences” that
can adjust or change these latter kinds of activities mid-process.
In spite of this view and point, no qualitative difference exists
between cognitive and metacognitive knowledge in Flavell’s
view and model (1985) of metacognition, which is conscious-
ness driven with no acknowledgement of unconscious auto-
matic processes and functions. This flaw and fuzziness in Flav-
ell’s and many other models of metacognition is a difficult
hurdle to get by in exploring metacognitive knowledge and its
functions in learning and teaching, as what at one time is con-
scious metacognitive experience can with learning, time and
use become automatic unconscious processing and functions
which creates many conceptual, theoretical and research diffi-
culties and confusions. The key here, we believe, is that Flavell
and many other metacognition theorist have a hidden assump-
tion in their models and theories that the learner is a relatively
novice learner in terms of the tasks to be learned, as compared
to a more expert and experienced learners, and thus few if any
automatic processes have been acquired relative to the task(s)
to-be-learned, and most metacognitive experiences and proc-
esses will be conscious for the learner and thus metacognitive
(self, tasks, and strategies) knowledge dependent. Flavell and
other theorists, moreover, are not talking about the adult learner
or even the young adult learner, which is another hidden as-
sumption/restriction in their models and theories which make
them only fuzzy or approximate fits to pre and in-service teach-
ers and undergraduate students. Learners in Flavell-like models
are fairly novice learners relative to the knowledge-domain and
tasks to be learned, and are not adult learners in Knowles’ sense
and definition of the term (Knowles, Holton, & Swanson, 2005).
Knowles’ notion of adult learners corresponds to many aspects
of the upper levels of “self-regulated” learners in Zimmerman’s
(2008) and Purdie & Hattie’s (1996) conceptualization and
models of this view and type of metacognition and controlling
of cognitive processes, which, it should be clearly noted, can be
both conscious and/or automatic and unconscious. Understand-
ing such limitations of Flavell-like models of learners allows
various aspects of Flavell’s model and theory to be reasonably
used to both model and teach a fairly simple and simplified
model of metacognition to pre and in-service teachers and un-
dergraduate students that is not overly complex or complicated,
but also not grossly distorted or inadequate for the task. This
approach is the approach we adopted in this work and some of
the reasons why we adopted it. Further, according to our data-
base searches, there is no prior work or studies like the ones we
have done conceptualizing, operationalizing, and developing a
research-based monograph for teaching the types of metacogni-
tion required by post-secondary in-service EFL reading teach-
ers to teach summarizing strategies for expository texts and
then validating that monograph using a detailed panel review of
the monograph chapter by chapter by highly experienced prac-
ticing EFL teachers and of expert EFL professors and compar-
ing the results and views of the monograph by these two dif-
ferent kind of expert panels (for further and more detailed ela-
boration of this point, see Xu, Carifio, & Dagostino, 2012).
Purpose
This article reports the second phase of the Harvard Business
School like case study we did; namely, the validation of the
monograph developed according to the formal monograph de-
velopment model used (see below for details). Before a detailed
report of this validation process is presented, a brief overview
of the design of the entire study and the completion of Phase I
of the study (the creation and production of the research-based
monograph) is given to help the reader understand the process
by which the monograph was developed and refined before the
validation process (Phase II) of this study started (see Xu, Cari-
fio, & Dagostino, 2012, for more details). The focus of this arti-
cle, then, is on the method and findings of the monograph (pro-
duct) validation and panel review processes done in Phase II.
Design of the Case Study and Conduct of
Phase I Study: An Overview
As previously stated, the purpose of this study was to de-
velop and validate a monograph to enrich post-secondary in-
324

W. XU ET AL.
service EFL reading teachers’ understanding of the processes
and metacognitive knowledge involved in teaching summariz-
ing strategies for expository text to EFL undergraduates. An
extensive and comprehensive literature review and content
critical evaluation phase was required during the process of
developing and validating the first draft of the monograph rela-
tive to various areas of research, such as metacognition, text
comprehension, reading strategies instruction, TESOL, and so
on. This process yielded certain theoretical frameworks, syn-
theses, ideas, and views, both anticipated and unanticipated
beforehand that formed the core of this work and the mono-
graph developed. To gauge and parameterize all expected and
unexpected findings, the design of this study adhered to a for-
malized model of instructional materials development initially
proposed by Carifio (1975, 1977) and further elaborated by
Perla (2006) and Perla and Carifio (2011). This model is de-
picted in Figure 1 for the reader for the purpose of following
the discussion here and the model is explained in detailed in
Perla and Carifio (2011).
According to the Carifio-Perla model, the process of creating
and developing academic materials involves three macro com-
ponents, starting from developing a “Critical & High Quality
Knowledge Base” (CHQKB) in a particular domain which
identifies the key and critical features of the CHQKB and
which is a form of meta-knowledge about the CHQKB as well.
This “Critical and High Quality Knowledge Base” is then
translated into “Appropriate Representations and Communica-
tions” (ARC’s) in the form of instructional or academic materi-
als. The “Appropriate Representations and Communications”
(ARC’s) then go through “Validation and Field Testing for
Effectiveness” (VFTE). During the process of going from
CHQKB to ARC’s, and then from ARC’s to VFTE, two types
of unanticipated findings might emerge respectively, which the
developer should be particularly sensitive to as they are re-
search findings. The first type of unanticipated findings con-
cerns inadequacies in the theories, frameworks, and claims
encountered in the domain and contradictions between alterna-
tive views of all three found in the domain. The second type of
unanticipated findings concerns inadequacies in the pedagogi-
cal theories, frameworks, and claims encountered in the domain
and contradictions between alternative views of all three found
in the domain. As both types of unanticipated findings must be
reconciled to some degree, model driven and guided instruc-
tional materials development can help to refine and improve
both the theory and the pedagogical knowledge in a given do-
main if done appropriately, which is not the commonly held
Figure 1.
A simplified version of Carifio-Perla Model (2011) for instructional
materials development. CHQKB = Critical and high quality knowledge
Base; ARC = Appropriate representations and communications; VFTE
= Validated and field-tested for effectiveness. See original figures in
Perla and Carifio (2011).
view of the instructional materials development process today.
Phase I, then, was to utilize the Carifio-Perla model (Perla
and Carifio, 2011) to create a research-based monograph (re-
ferred to as The Monograph hereafter). During Phase I, an un-
expected finding was the many difficulties and contradictions
that were encountered in constructing a general Metacognitive
Knowledge Framework (MKF). The elaboration of this MKF
was a process of exploring answers to the question “What ex-
actly does a post-secondary in-service EFL reading teacher’s
metacognitive knowledge consist of as to teaching summarizing
strategies with expository text?” The general MKF that was
constructed was based on Flavell’s (1979) tripartite metacogni-
tive knowledge model and other researchers’ conceptualization
of metacognitive knowledge (e.g. Baker, 1989; Garner, 1987,
1990; Glaser & Chi, 1988; Pressley, Borkowski, & Schneider,
1987; Reynolds, 1992; Schneider & Pressley, 1989). The MKF
that was constructed from this process turned out to be a
two-dimensional matrix as shown in Table 1 (for further and
more detailed elaboration of this MKF, see Xu, Carifio, & Da-
gostino, 2012).
As can be seen from Table 1, Flavell’s one dimensional view
of metacognitive knowledge (i.e. person, task, and strategies
variables) can be analyzed in terms of a second dimension as
well; namely, metacognitive declarative, procedural, and condi-
tional knowledge, and vice versa. It has to be pointed out again
as it was done at the beginning of this article that the discussion
of metacognitive knowledge on person variables is too complex,
complicated, and jumbled (confounded) to be useful or to be
included in this study. Thus, the focus of this study was only on
exploring the metacognitive knowledge of Flavell’s task and
strategy variables. This is to say, post-secondary in-service EFL
reading teachers’ metacognitive knowledge of summarizing
strategies instruction with expository text was investigated only
from the six aspects numbered “1” to “6” in Table 1 and not in
terms of those numbered 7 to 9 in Table 1. The monograph,
therefore, elaborated the general MKF in terms of task and
strategy variables for the dimension of metacognitive declara-
tive, procedural, and conditional knowledge. The nature of this
elaboration is given in detail in another article (Xu, Carifio, &
Dagostino, 2012).
The process for establishing the finalized content outline and
the content of The Monograph (i.e., the ARC’s for this study)
was an iterative, spiral and dynamic process. The finalized
content for the version of The Monograph sent for review in
phase II had eight chapters. The main idea of each chapter is
listed as follow:
1) Chapter 1 outlined and summarized the focus of The
Monograph.
2) Chapter 2 presented the construction of a general Meta-
cognitive Knowledge Framework (MKF) via thorough analyses
of different researchers’ conceptualization of “metacognition”
and “metacognitive knowledge”.
Table 1.
A Two-dimensional Matrix of Metacognitive Knowledge Framework
(MKF).
Metacognitive
Knowledge CategoriesTask Variables Strategy Variables Personal Variables
Declarative Knowledge1 2 7
Procedural Knowledge 3 4 8
Conditional Knowledge5 6 9

W. XU ET AL.
3) Chapter 3 presented and elaborated a reading model to
support the constituents of MKF, helping reading teachers bet-
ter understand the nature and process of summarizing complex
text and expository text comprehension at the post-secondary
level.
4) Chapter 4 synthesized an instructional model, focusing on
maximizing comprehension along with a range of comprehen-
sion improving strategies to further elaborate the theoretical
MKF for reading teachers on teaching summarizing strategies
in ESL/EFL instructional contexts.
5) Chapter 5 explored the constructs and teaching demands
of summarizing strategies instruction with expository text,
which comprises the metacognitive declarative knowledge of
task variables in MKF.
6) Chapter 6 addressed and illustrated the specifics and de-
tails of teaching summarizing strategies with expository text
(Category 2 in the framework) with a strong focus on the sub-
teaching tasks of summarizing strategies instruction.
7) Chapter 7 continued to address the specifics and details of
teaching summarizing strategies with expository text, concen-
trating on the selection of cognitive and metacognitive strate-
gies with some accompanying conditional knowledge involved
in teaching summarizing strategies.
8) Chapter 8 applied the previously discussed metacognitive
knowledge and provided scenarios to show how a reading
teacher’s metacognitive knowledge can be translated into de-
signing a lesson plan for teaching certain summarizing strate-
gies with expository text to EFL undergraduates.
Because little literature was found in the various database
searched (e.g. ProQuest, EBSCOhost, SAGE, JSTOR) that was
directly related to the research topic and focus of this study,
which surprised us, the meta-cognitive framework we con-
structed, as well as the pedagogical framework for teaching it,
was very tentative and novel. We had apparently found a large
“hole” in the literature and the various exhortations for teachers
to teaching “metacognition” and to teach “metacognitively,”
repeatedly stated by many recent articles we encountered in this
literature, as there was little that was concretely worked out and
available in the literature to actually use to implement these
numerous exhortations, and particularly so in the area of read-
ing and expository text processing/summarization skills which
are at the core of most academic instruction. Given these points
and the numerous decisions and selections made in the devel-
opment process, The Monograph developed needed to be fur-
ther reviewed, evaluated, and cross-validated (or not) by inde-
pendent and “expert” third-parties. Therefore, the completed
monograph was sent to panels of expert reviewers for inde-
pendent review, evaluation, and cross-validation. The method-
ology and results of the validation process; namely, Phase II of
this study, are discussed in the following sections.
Validation of the Metacognitive Knowledge
Exploration: Phase II Study
As previously stated, Phase II aimed to validate The Mono-
graph. The Monograph was sent to two formal reviewer panels
(seven EFL reading teacher expert practitioners and seven EFL
academic reading professors of education) in China for external
third party validation. This type of general research design and
panel review methodology, which is an adaptation and imple-
mentation of Campell and Fiscke’s (1959) convergent and dis-
criminant validation design, was developed by Carifio (1990)
and has been successfully used by others (See Dagostino &
Carifio, 1994a; Flores, 2005; Erikson, 2006; Kwong, 2008).
Perla (2006) summarized Kerlinger and Lee’s (2000) explana-
tion of the two concepts, convergent and discriminant, in his
study as the following:
Convergence here refers to different sources (observers or mo-
des of observation) used to measure a construct or “traits” pro-
ducing similar results, whereas discriminability refers to simi-
lar traits that can be empirically differentiated from one another
or identifying traits that are not correlated or negatively corre-
lated with each other or the construct (p. 168).
Therefore, the agreement and disagreement between data
sources and reviewers provided triangulation evidences to as-
sess and validate The Monograph created as well as the model
and theory used to create The Monograph validated. Specifi-
cally, the validation process in Phase II was intended to address
three questions:
1) How adequate and comprehensive was the general Meta-
cognitive Knowledge Framework generated in the views of all
reviewers?
2) How well and completely was the general Metacognitive
Knowledge Framework elaborated in the views of all review-
ers?
3) Were there important or key differences between the two
panels of reviewers in evaluating The Monograph?
Methodology of the Validation
A cross-panel replicated expert reviewer (practitioners and
academics) design was used to validate The Monograph.
Review Panels
The independent formal reviewers were grouped into two
panels. Panel 1 was 7 post-secondary EFL teacher educators
(the academic expert panel) who taught pre-service EFL teach-
ers at a university in Zhejiang Province, China. Two of Panel 1
reviewers were senior teacher educators with around 30 years
of experience teaching preparing EFL teachers. The other five
panelists were more junior EFL teacher educators, who once
were students of the two senior teacher educators during their
undergraduate studies at the same university. One of the junior
teacher educators also had a doctorate in Comparative Litera-
ture, while another one was also currently a doctoral candidate
in Comparative Linguistics at the time The Monograph was
reviewed. The other four Panel 1 reviewers also had master de-
grees in either English Language and Literature or English Lan-
guage and Linguistics. The members of Panel 1 were well-
educated as well as well experienced practitioners.
Panel 2 was made up of 7 post-secondary EFL professors/
teachers of reading from several universities in Shanghai, who
taught EFL undergraduates in various majors. Most members of
Panel 2 (the expert practitioner group) had master degrees in
either English Language and Literature or English Language
and Linguistics. All the EFL academics who were Panel 2
members also had many years of EFL teaching experience at
the post-secondary level. Their professional opinions, therefore,
should be more than just well-informed and “academic” relative
to designing effective comprehension strategies instruction, and
especially in terms of the summarizing strategies instruction
discussed in The Monograph.
It should be pointed out that since it was very difficult to get
qualified formal reviewers who were Chinese and living and
326

W. XU ET AL.
teaching in China (one of the key criteria for being a member of
these review panels), the formal reviewers for this study were
chosen from the first author’s former colleagues and classmates
in China, based on their availability and interest of doing this
kind of reviewing work. Two further points need to be made
here however.
First as The Monograph was written in English for native
Chinese professionals and students who would be learning and/
or teaching expository text summarizing skills in English to
undergraduate Chinese EFL students, it was key to both the
evaluation and validation of this Monograph that the review
panel members were native Chinese experts who were bilingual
and who taught English and the reading and understanding of
English to undergraduate Chinese EFL students and not experts
who were just English speaking EFL experts evaluating an EFL
Monograph on the teaching of expository text summarizing
skills. The latter groups would be highly flawed (and possibly
biased) validation panels in several different ways, where the
former groups (the panels we created and used) would not be,
as they were the experts with the correct set of skills, training
and experiences to more fully and comprehensively carry out
the tasks, evaluation and validation of The Monograph. One
feature of this study that is relatively unique and important,
therefore, and unique and important relative to the research
studies typically done in the EFL area is that it used bi-lingual
and bi-cultural panelists to evaluate an instructional/academic
monograph written in English and produced in the USA that
was intended for EFL teachers, preparing teachers, and students
in China; namely, in another country, culture and educational
system contextually and implicitly regulated by different poli-
cies and values. Whether The Monograph would be judged to be
of worth and a quality work of value in this context is not just
external validation and an external validity question, but rather
is ecological validation, and an ecological validity question is
typically rarely addressed or done in most research works,
making the views and reviews of The Monograph by the two
bi-lingual and bi-cultural panels used important and unique in
this larger and broader sense relative to multilingual and multi-
cultural works and studies and international education.
The second point that needs to be addressed above is that
members of the two review panels used were former colleagues
of the senior author of this article. In Western cultures and con-
texts, the use of colleagues as reviewers and judges might be
seen as a flaw and a biasing factor, but in China and China
today, any biasing would be far less, as it would be culturally
and personally as well as academically dishonest and unac-
ceptable to render anything other than the frankest of evalua-
tions and criticisms of a work and a work that was a social ob-
ject that would be used in social contexts with social conse-
quences even if it was the work of a known old colleague. Does
this mean that there would not perhaps be unconscious biases
operating in the review process? Not likely; only that they
would most probably be comparative less due to differing cul-
tural imperatives and effects as well as “averaged down” even
further over the number of expert reviewers used in this study
making this bias quite low for a panel review design.
Data Collection
The two panels of reviewers were provided a detailed review
protocol to formally review and evaluate each of the eight
chapters of The Monograph and The Monograph itself. This
protocol was a modified version of an already field-tested and
validated 30-item Likert responding format objective text pro-
tocol plus 7 open-ended written questions (See Carifio, 2003;
Dagostino & Carifio, 1994a; Pellitier, 2004; Flores, 2005; Perla,
2006; Erikson, 2006). The protocol’s purpose was to get both
open and closed structured feedback about the appropriateness
of the constructed MKF, and the effectiveness of the translation,
representation and communication of the MKF in The Mono-
graph. The protocol asked each formal reviewer to make spe-
cific judgments about each chapter as well as the whole work
on a scale ranging from “NA (Not applicable)” to “RP (Rift
with Problems)” to “E (Excellent)” in terms of eight categories
of feedback. These categories were: 1) Accuracy, Saliency and
Relevance of Content; 2) Thoroughness; 3) Quality of Sup-
porting Theory, Research, and Scholarship; 4) Presence of
Multiple and Alternative Views; 5) Tone; 6) Clarity of Writing
Relative to Audiences; 7) Specificity and Concreteness of Key
Points and Recommendations; and 8) Missing Elements (see
Erikson, 2006 for details). These eight criteria were originally
developed by Carifio (2003) and later effectively applied to re-
search studies of this instructional materials development
model by other researchers (e.g., Perla, 2006; Erickson, 2006;
Kwong, 2008). All reviewers were also encouraged and al-
lowed to write comments directly on The Monograph.
The whole monograph was sent to each individual reviewer
chapter by chapter accompanied by the formal evaluation re-
view protocol. A new chapter would not be sent until the pre-
vious chapter was finished by the reviewers so that halo effects
were reduced and chapters were reviewed as independently
from other chapters as possible. The same protocol was sent to
each reviewer again to evaluate the whole document after they
had finished reviewing all eight chapters separately.
Upon the completion of the formal review, a follow-up 15 to
20 minutes informal interview was also performed with re-
viewers on the phone if the interviewer (the first author of this
article) felt the necessity of having the panelists further explain
their responses or to help ensure no misunderstandings of their
points and views were occurring. Thus, the discussed contents
in each interview were different, and no formal interview
guidelines were used. Notes were made both during and after
these interviews and were referenced during the formal data
analyses. The Likert data and other rating data in this study
were analyzed as parametric data as this was the appropriate
way in which these data should be analyzed (see Carifio &
Perla, 2007, for details).
Data Analyses
Of the 14 external third party reviewers, only 6 (3 teacher
educators and 3 in-service reading teachers) were able to com-
plete the whole reviewing process. The other reviewers failed to
finish the reviewing work due to either their heavy teaching
load or personal issues. All data from the 6 reviewers were then
analyzed both quantitatively and qualitatively as the core
analyses in this study as they were a complete data set with no
missing observations.
For the 30-item objective text protocol, the responding keys
were converted to numerical data from 0 to 6 and uploaded to
SPSS for sequential statistical analyses except for Items 7 and 8
because of reviewers’ inconsistent responses to these two items
due to their misunderstandings and concerns in terms of the
meaning and function of “NA (Not Applicable)” term and des-
ignation, which was a cross-cultural discovery for us.
First, mean and weighted mean combined responses by the

W. XU ET AL.
teacher educators and the in-service reading teachers were cal-
culated separately for each item for each chapter, across chap-
ters, and all chapters as a whole, namely, The Monograph itself.
Second, before conducting the next steps of data analyses, all
items were aggregated into the first seven major evaluation
criteria chosen for each chapter. The original eighth criterion
“Missing Elements” was excluded because no reviewer pointed
out any missing elements from The Monograph.
Third, mean responses as well as the overall mean responses
for each criterion and each chapter were then calculated for the
two review panels respectively.
Finally, a frequency matrix of mean ratings for each category
was conducted respectively for Panel 1 reviewers, Panel 2 re-
viewers, so that they were directly comparable, and then re-
viewers of both panels combined.
A cut-off criterion of 3.5 on the 0 to 6-point scale was used
as the minimally acceptable mean level of quality on each item,
each category, and each chapter in the aggregate, as well as on
the items and the categories across the chapters for the objec-
tive responding review protocol.
As to the open-ended written questions, the external review-
ers’ responses were analyzed using a thematic and simple tal-
lies scoring system. The responses were coded using a simple
8-point Very Positive comment to Very Negative comment
rating scale (see details in Erikson, 2006: p. 98). This rating
scale had the following categories: (−4) Extremely Negative,
(−3) Very Negative, (−2) Negative, (−1) Somewhat Negative,
(0) Neutral/Unsure, (+1) Somewhat Positive, (+2) Positive, (+3)
Very Positive, and (+4) Extremely Positive. The cut-off crite-
rion of +1.0 was chosen as the minimally accepted level of
quality on each chapter and the chapters in the aggregate. Sim-
ple tallies and means, both weighted and unweighted, were thus
obtained to analyze the overall quality of the eight chapters
relative to those positive and negative reviewer comments for
both panels. Thus, the number and kind of responses to the
questions could be reported, and the overall qualities of The
Monograph could be assessed. Further, agreements and dis-
agreements between Panel 1 and Panel 2 reviewers were also
analyzed formally. With regard to the comments directly made
on The Monograph, they were not coded using the +4 to −4
scoring rubric because not all reviewers commented directly on
the text. However, these comments along with the results of the
analyses reported below were used to revise The Monograph.
Results of the Validation
The following sections report the major findings of the vali-
dation of study done; namely, the formal review of The Mono-
graph and the scoring and coding of these reviews as described
above.
Structured Format Ratings
Table 2 is a summary table that presents the means and
weighted combined means for both the structured format rat-
ings and the rated open-ended responses by both the teacher
educators and the reading teachers for the eight chapters as well
as the overall averages for all chapters by the two panels of
reviewers.
As can be seen from Table 2, the weighted combined means
for all chapters for the structured format means ranged from
4.92 (highly acceptable) to 5.68 (good plus). This indicated that
the ratings for all chapters were well above good ratings,
Table 2.
Summary of means and weighted combined means for the structured
format ratings made by the academic teacher educators (n = 3) and the
practitioner in-service reading teachers (n = 3) for the eight chapters of
the monograph on a zero to six point response scale.
Structured Format
Mean Ratings
Academic
Teacher
Educators
Practitioner
Reading
Teachers
Combined
Mean
Chapters
1 = Introdution/
Overview 5.40 4.92 5.16
2 = A Model of
Metacognitive
Knowledge
5.63 5.38 5.51
3 = A Reading Model 5.62 5.29 5.46
4 = ESL/EFL
Instructional Contexts 5.36 5.23 5.30
5 = Constructs and
Teaching Demands 5.28 5.32 5.30
6 = The Sub-teaching
Tasks 5.60 5.28 5.44
7 = The Selection of
Strategies 5.61 5.17 5.39
8 = A Scenario 5.68 5.10 5.39
Average all Chaters 5.52 5.21 5.37
Scale Key
6 = Excellent
5 = Good
4 = Acceptable
3 = Weak
2 = Poor
1 = Rift With Problem
0 = Not Applicable This Chapter
and so were the averages for all chapters. The mean ratings by
the three teacher educators (academic experts) were markedly
higher than those by the three in-service reading teachers (prac-
titioner experts) for seven chapters of the eight chapters of
monograph, the exception being Chapter Five (Constructs and
Teaching Demands). For Chapter Eight, and Chapter One, these
mean rating differences were almost half a scale point or more
for the academic experts, indicating obvious differences be-
tween the two panels of reviewers’ viewpoints.
Relative to the findings presented in Table 2, the first three
chapters of The Monograph are very theoretical and introduced
the background of the conceptualization of metacognitive
knowledge, while Chapter Five was about the constructs and
teaching demands of summarizing strategies instruction. Thus,
the teacher educators (the academic experts) seemed to buy
more into those theoretically oriented chapters, but the reading
teachers (the practitioner experts) were more interested in the
chapters on the analyses of practical teaching issues. The aca-
demic teacher educators (professors) rated Chapter Eight (A
Scenario) the highest, and in fact much higher than the practi-
tioner reading teachers did, who also rated this chapter “good
plus.” This particular finding meant that though Chapter Eight
was supposed to provide the readers a practical example, it
could not be fully appreciated without the full understanding of
its theoretical background elaborated in the previous chapters.
Thus, overall the three teacher educators were more positive
than the three in-service reading teachers on the 30 item struc-
tured rating scale. However, the difference in variability for the
structured ratings between the two panels of reviewers was not
that much. Actually, the range of the ratings for the structured
328

W. XU ET AL.
format means by both panels was 0.40 with the teacher educa-
tors’ ratings basically 0.36 higher than those of the reading
teachers.
Open-Ended Responses
In terms of the average rated comment means for the chap-
ters, which were done on a −4.0 to +4.0 scale, Table 3 shows
that all chapters except Chapter One (Introduction/Overview)
more than met the cut-off criteria of +1. The low overall rating
for Chapter One (+0.9) was caused by the low (negative) rating
of this chapter by the teacher educators (+0.7). As previously
stated, Chapter one was perhaps the most theoretical of the
chapters. The recorded comments indicated that the reviewers
expected more background description in Chapter One in terms
of why the researcher chose this research topic and what spe-
cifically led to this research focus.
The rated comment means by the three academic teacher
educators were markedly more positive than those by the three
practitioner in-service reading teachers about the five chapters
of the monograph: Chapter Two, Chapter Three, Chapter Four,
Chapter Seven and Chapter Eight, two of which (Chapter Two
and Seven) even higher than the double of the ratings by the
reading teachers. The practitioner reading teachers tended to be
somewhat more positive than the academic teacher educators
about Chapter One (Introduction) and Chapter Five (Constructs
and Teaching Demands), and significantly more positive about
Chapter Six (The Sub-teaching Tasks). Thus, the means for all
chapters by all reviewers except that for Chapter One by the
academic teacher educators were above the acceptable rating of
+1, and some were highly above +1 with the highest mean of
+2.5 occurring for the academic teacher educators for Chapter
Seven (The Selection of Strategies) and Chapter Two (A Model
of Metacognitive Knowledge). The highest mean rating (posi-
tive comments) by the practitioner reading teachers was Chap-
ter Six (The Sub-teaching Tasks).
These differences in making positive and negative comments
on the chapters among the reviewers echoed the differences
Table 3.
Summary of means and weighted combined means for the ratings of
comments made by the academic teacher educators (n = 3) and the
practitioner in-service reading teachers (n = 3) for the eight chapters of
the monograph on a one to eight (−4.0 to +4.0) point response scale.
Rated Comment Means Academic Teacher
Educators
Practitioner
Reading
Teachers
Combined
Mean
Chapters
1 = Introduction/Overview 0.7 1.1 0.9
2 = A Model of Metacognitive
Knowledge 2.5 1.1 1.8
3 = A Reading Model 2.3 1.4 1.9
4 = ESL/EFL Instructional
Contexts 2.0 1.6 1.8
5 = Constructs and Teaching
Demands 1.7 2.1 1.9
6 = The Sub-teaching Tasks 1.5 2.5 2.0
7 = The Selection of Strategies 2.5 1.2 1.9
8 = A Scenario 2.3 2.1 2.2
Average all Chapters 1.9 1.6 1.8
shown by their structured ratings. These differences once again
revealed their differences in views while reading an academic
monograph. The academic teacher educators seemed to buy
more into those theoretically oriented chapters, such as Chapter
Two (A Model of Metacognitive Knowledge), Chapter Three
(A Reading Model), and Chapter Four (ESL/EFL Instructional
Contexts), while the practitioner reading teachers were more
interested in the chapters on the analyses of practical teaching
issues, such as Chapter Five and Chapter Six. However, the real
point of importance here is that both the academic teacher edu-
cators and the practitioner in-service reading teachers needed
all 8 of the chapters in The Monograph really regardless of their
preference and that point came out later in their overall evalua-
tion.
All reviewers thought highly of Chapter Eight (A Scenario)
as shown from their extremely positive comments (the average
of 2.3 and 2.1 respectively by the two panels) according to the
average all chapters and the total combined mean ratings. Thus,
it can be concluded that all reviewers thought that Chapter
Eight successfully applied theory into practice and effectively
demonstrated how a reading teacher’s metacognitive knowl-
edge was involved in her decision making while designing a
lesson plan for summarizing strategies instruction.
The average of all chapters by the teacher educators was 1.9,
higher than that by the reading teachers (1.6). Thus, basically
the three teacher educators were more positive than the three
in-service reading teachers in their responses to the open-ended
questions, but with a bit of more variability in their evaluations
(range 1.8) as compared to the reading teachers who seemed
less variable in their responses (range 1.4). This variability, it
should be noted, makes it easier to understand the observed
variability in the combined weighted means for the open-ended
comments.
Table 4 presents the total combined means of the structured
and comment ratings for the eight chapters of The Monograph
made by the academic and practitioner expert reviewers in this
study. As can be seen from Table 4, overall both the academic
and practitioner reviewers were quite positive about the chap-
ters of The Monograph in terms of the evaluation criteria used
with the pattern of differences by chapter being as expected
beforehand. Overall, the academic reviewers were slightly more
positive than the practitioner reviewers which was also some-
what of an expected outcome.
Evaluation Criteria Analyses and Results
The 30-item structured survey also reflected seven major
evaluation criteria for each chapter as mentioned earlier. An-
other and equally important way of depicting results as well as
differences between the two review panels is in terms of each
of the seven evaluation criteria used. Table 5 presents the mean
responses by each of the seven evaluation criteria for the aca-
demic and practitioner review panels. The results depicted in
Table 5 indicate that when The Monograph was reviewed
chapter by chapter, an overall combined weighted mean for all
criteria were either “good plus” or “excellent minus.” These
results further supported the high quality of content and consis-
tency of The Monograph. Again, the academic teacher educa-
tors rated all seven criteria for each chapter higher than the
practitioner reading teachers did. The largest variability was
within the criterion of Professional Tone for which the mean
rating by the practitioner reading teachers was 0.41 below that
of the academic teacher educators.

W. XU ET AL.
Table 4.
Total combined means of structured and comment ratings made by the
academic teacher educators (n = 3) and the practitioner in-service
reading teachers (n = 3) for the eight chapters of the monograph.
Total Com bi ned Mean
Ratings
Academic
Teacher
Educators
Practitioner
Reading
Teachers
Total
Combined
Mean
Chapters
1 = Introduction/Overview 6.10 6.02 6.06
2 = A Model of
Metacognitive Knowledge 8.13 6.48 7.31
3 = A Reading Model 7.92 6.69 7.31
4 = ESL/EFL Instructional
Contexts 7.36 6.83 7.10
5 = Constructs and Teaching
Demands 6.98 7.42 7.20
6 = The Sub-teaching Tasks 7.10 7.78 7.44
7 = The Selection of
Strategies 8.11 6.37 7.24
8 = A Scenario 7.98 7.20 7.59
Average all Chapters 7.46 6.85 7.16
Table 5.
Mean and weighted mean combined responses made by the academic
teacher educators (n = 3) and the practitioner in-service reading teach-
ers (n = 3) on the seven major evaluation criteria on a one to six point
responses scale.
Evaluation C riteria Aca demic Teacher
Educators
Practitioner
Reading
Teachers
Combined
Mean
1. Accuracy of Content 5.54 5.26 5.40
2. Thoroughness of Text 5.63 5.29 5.46
3. Quality of Supporting
Research/Theory 5.34 5.01 5.18
4. Multiple Perspective
Presented 5.39 5.25 5.32
5. Professional Tone 5.54 5.13 5.34
6. Clarity of Writing 5.54 5.20 5.37
7. Adequacy of Key Points/
Recommendations 5.67 5.35 5.51
Average all Criteria 5.52 5.21 5.37
Scale Key
6 = Excellent
5 = Good
4 = Acceptable
3 = Weak
2 = Poor
1 = Rift With Problem
0 = Not Applicable This Chapter
When The Monograph was evaluated as a whole according
to the seven criteria, the mean rating for Professional Tone by
the teacher educators was also much higher than that of the
reading teachers (0.34 higher, see Table 6). This difference
again showed that the academic teacher educators appreciated
professional and scholarly discourse and presentation more than
the practitioner reading teachers, but this result is relative as
both were appreciative and satisfied with the tone of The
Monograph. Again, this difference most probably can be attrib-
uted to the teacher educators’ higher level of professional de-
velopment from their doctoral degrees, which made them more
Table 6.
Mean and weighted mean combined responses made by the academic
teacher educators (n = 3) and the practitioner in-service reading teach-
ers (n = 3) on the seven major evaluation criteria for the monograph as
a whole on a one to six point responses scale.
Evaluation C riteria Aca demic Teacher
Educators
Practitioner
Reading
Teachers
Combined
Mean
1. Accuracy of Content 5.38 5.19 5.29
2. Thoroughness of Text 5.67 5.67 5.67
3. Quality of Supporting
Research/Theory 5.58 5.33 5.46
4. Multiple Perspective
Presented 5.67 5.56 5.62
5. Professional Tone 5.67 5.33 5.50
6. Clarity of Writing 5.56 5.26 5.41
7. Adequacy of Key Points/
Recommendations 5.78 5.39 5.60
Average all Criteria 5.62 5.35 5.49
Scale Key
6 = Excellent
5 = Good
4 = Acceptable
3 = Weak
2 = Poor
1 = Rift With Problem
0 = Not Applicable This Chapter
accustomed to communications in a professional academic tone,
perhaps making them more “trilingual” than their bilingual
practitioner counterparts. It was interesting to see that the mean
and weighted mean combined responses made by both panels
of reviewers for The Monograph as a whole (see Table 6) were
higher than those for The Monograph reviewed chapter by
chapter on almost all seven criteria except the criterion of Ac-
curacy of Content. This result meant that after reading the
whole work, all reviewers most likely came to see clearer the
big picture of The Monograph and understood the contents of
The Monograph better, so they gave The Monograph higher
evaluation as a whole. And it may also be that because of this
better and more global understanding that the reviewers’ ex-
pectations or/and perceptions of the accuracy of content in turn
became a bit higher.
Summary and Discussion of the Results
From the above results it is clear that all panel reviewers
evaluated The Monograph positively and highly on all criteria
and for all chapters respectively in terms of the structured for-
mat mean ratings and the average rated comment means for all
chapters. In general, almost all reviewers’ mean ratings were
skewed towards the higher level of assessment. These very
positive and uniform mean ratings indicated the high quality
and consistency of The Monograph that was achieved by em-
ploying the guiding instructional materials development model
which was presented at the beginning of this article; namely,
the appropriate construction, and the effective communication
and representation of the general MKF in The Monograph for
the target audience. This result replicated the results of seven
other recent uses of this instructional materials development
model given at the beginning of this article but for somewhat
more difficult, complex and fuzzy subject-matter in a bilingual
and bi-cultural context which is both a new and very positive
result and finding for this instructional materials development
330

W. XU ET AL.
model as it shows that aspects of skills, knowledge, under-
standings, and metacognitions both transcend and can be repre-
sented and communicated successfully across languages and
cultures and to different professional audiences as well. This
finding we believe is more than a hopeful finding and result.
We have also further synthesized our findings to answer the
previously-mentioned three specific questions of importance
related to the subject-matter of this study. These three ques-
tions are: a) how all reviewers viewed the construction of the
general Metacognitive Knowledge Framework, b) how they
viewed the elaboration of the general MKF, and c) what the
differences between the two panels of reviewers were in evalu-
ating The Monograph.
The construction of the general MKF was presented in
Chapter Two of The Monograph. The means and weighted
combined means for structured format ratings in Tables 2-4 by
all expert reviewers for Chapter Two were all “good plus.” The
rated comments means by the academic teacher educators were
more than positive, and those by the practitioner in-service
reading teachers were somewhat positive. It can thus be rea-
sonably concluded that all reviewers agreed that the general
MKF was appropriately constructed for the purpose of this case
study and its intended audiences even though a more simplified
version of the MFK had to be presented for the reasons outlined
at the beginning of this article than academic psychologist
might prefer. Actually some reviewers made very positive
comments about the MFK presented, saying that the researcher
was quite successful in analyzing reading teachers’ metacogni-
tive knowledge and laying a very solid theoretical foundation
for further discussions at this aspect. Further, all reviewers
shared the opinion that reading teachers should acquire relevant
metacognitive knowledge to conduct reading strategies instruc-
tion effectively. Therefore, one might conclude that for intro-
ductory instructional texts on meta-cognition (and similar con-
tent) the highly nuanced, complex and fully fuzzy representa-
tion of academic psychological theorists and researchers might
not be the most successful approach and that simpler and less
nuanced but accurate representations that may be built upon
later will be more successful with certain audiences and that
this is a point and criterion that should be kept in mind by in-
structional materials developers and publishers as well as those
who select instructional materials for others.
Next, the results of data analyses also revealed that all re-
viewers thought that the general MKF was successfully elabo-
rated in different chapters as to investigating the task and strat-
egy variables of metacognitive knowledge in terms of their
declarative, procedural and conditional metacognitive compo-
nents. The key consensus specific reviewer findings concerning
the elaboration of the general MKF presented were as follow:
First, as to supporting the components of metacognitive
knowledge that were identified with a reading model (Chapter
Three), teacher educators thought that Kintsch’s (1998) Con-
struction-Integration (CI) Model was the best choice and ap-
propriately presented for the purpose of exploring reading
teachers’ metacognitive knowledge concerning the process of
summarizing and summarizing strategies instruction for ex-
pository texts. This CI model was the most useful and valid
theory currently available to explain mature readers’ skilled
reading at the higher-level of information processing. Further,
the use of macrostructural strategies illustrated in the CI model
was crucial to summarizing and generating macropropositions
to represent main statements or topic sentences in a text. Even
reading teachers who rated this section of The Monograph
somewhat lower than teacher educators in open-ended re-
sponses agreed that CI model was highly relevant to and ap-
propriate for the investigation of reading teachers’ metacogni-
tive knowledge of reading strategies instruction.
Second, in terms of integrating the general MKF presented in
The Monograph into the contexts of ESL/EFL reading strate-
gies instruction (Chapter Four), reviewers thought that it was
essential to reconstruct a more effective reading strategies in-
struction approach on the basis of current approaches to reading
strategies instruction, such as those proposed for particular
groups of students respectively by Pressley (1990), Almasi
(2003), and Chamot and O’Malley (1994) rather than introduce
a new model de novo and built everything around that model.
The model of reading strategy instruction synthesized by the
first author of this article includes:
 The CALLA’s framework of a five-stage strategy instruc-
tion procedure: preparation, presentation, practice, evalua-
tion, and expansion (Chamot & O’Malley, 1996),
 Seven cognitive instructional strategies respectively in
CALLA’s each stage of strategies instruction: 1) activating
background knowledge for the preparation stage, 2) ex-
plaining and modeling for the presentation stage, 3) guid-
ing practice, monitoring progress, and providing effective
feedback for the practice stage, 4) evaluating for the evalua-
tion stage, and 5) encouraging transfer for the expansion
stage, and
 Three general principles that teachers should follow: 1)
Organize students’ active involvement, 2) reduce process-
ing demands when needed, and 3) follow the above key
components of the five-stage instruction and principles in a
recursive fashion to help create a “safe and risk-free envi-
ronment that supports and facilitates motivated strategy
use” (Almasi, 2003: p. 50).
The reviewers thought that this synthesized model of reading
strategy instruction was very comprehensive, versatile, and in-
structive, and provided key components and principles for
teaching reading strategies effectively.
Third, with regard to the more specific elaboration of the task
and strategy variables of metacognitive knowledge for teaching
summarizing strategies with expository text (Chapter Five to
Seven), all reviewers were of the same opinion that to teach
summarizing strategies effectively, reading teachers should have
relevant metacognitive knowledge about the following:
1) What is summarizing?
Both panels contended that the definition of summarizing
synthesized by the first author of this article was very complete
and coherent. Summarizing was thus considered the ability to
construct from one’s reading and understanding of the gist of a
text an appropriate summarizing view that conveyed the im-
portant information for a particular reading purpose to different
degrees ranging from disclosing the author’s intention to ful-
filling the reader’s own goals and interests, to goals assigned to
the reader by external sources or demands.
2) What does summarizing strategies instruction with an ex-
pository text entail?
With the above understanding of the concept of summarizing,
reading teachers should know several important key constructs
of summarizing strategies instruction such as teaching student
how to identify important information and how to operate the
three macrorules—deletion rule, generalization rule, and con-
struction rule (van Dijk, 1980; Kintsch & van Dijk, 1978; van

W. XU ET AL.
Dijk & Kintsch, 1983)—at two levels (i.e., microstructure level
and macrostructure level). Three major sub-teaching tasks
should then be considered: a) teaching how to specify the type
of summary to be composed, b) teaching how to identify im-
portant information in the text relative to the nature of the ex-
pository text being studied, and c) teaching how to generate the
gist of a text via macrorules. Each major sub-teaching task can
further be decomposed into certain minor sub-teaching tasks
(discussed in detail in The Monograph). In addition to these key
constructs, reading teachers should also consider such general
teaching demands as the influence of students’ background
knowledge and comprehension proficiency, the different cogni-
tive demands of summarizing strategies, students’ awareness of
expository text structures, the difficulties caused by different
expository text structures for students of different comprehen-
sion proficiencies, different readers’ concepts of importance,
and their abilities and skills in identifying important informa-
tion from a text. All these latter factors, it should be noted, in-
volve the person variables of metacognitive knowledge, which,
as stated previously, are too complex to be explored in this stu-
dy.
3) What cognitive and metacognitive instructional strategies
can be applied in summarizing strategies instruction?
An agreement was reached that reading teachers should make
deliberate decisions on using the aforementioned seven cogni-
tive instructional strategies (detailed in Chapter Two of The
Monograph) to accomplish a particular teaching task, and ad-
just their teaching strategies in time to match students’ devel-
opment. Moreover, all reviewers agreed that the higher-level of
metacognitive strategies at both the domain-specific and gen-
eral level should be the part and parcel of effective teaching.
General metacognitive instructional strategies, such as planning,
monitoring, and evaluating, are ubiquitous in any metacogni-
tive-like processes. However, it is difficult to propose a stan-
dard list of domain specific metacognitive strategies in a par-
ticular cognitive process, such as teaching summarizing strate-
gies with expository text, because the adoption of any domain-
specific metacognitive strategies depends on different teachers’
cognition through years of teaching experience, their familiarity
with different teaching tasks and different students’ motivation
and emotion as well as some other relevant factors.
Overall, the total combined mean ratings indicated that the
teacher educators (professors) were much more positive about
The Monograph than the reading teachers practitioners with the
exception of Chapter Five (Constructs and Teaching Demands)
and Chapter Six (The Sub-teaching Tasks). This result was not
particularly surprising (but was validating) since Chapter Five
and Chapter Six met the reading teachers’ expectations for
specific analyses of instructional issues related to their daily
teaching practice. This particular difference indicates that the
three teacher educators were able to see the big picture and
understand the value of the theoretical background research to
the creation of a scholarly educational document. This outcome
may be due to the factors that all teacher educators have been
trained in doctoral degree programs in either linguistics or
comparative literature and all have comprehensive experience
in pre-service teacher education. By contrast, the reading
teacher practitioners tended to look more at the details of the
usefulness and applicability of the elaboration to their day-by-
day teaching issues. They expressed such expectations to the
senior author of this article from time to time in phone inter-
views. However, all practitioner and academic reviewers ex-
pected to see more practical examples such as the one illus-
trated in Chapter Eight as to the application of teachers’ meta-
cognitive knowledge before, during, and after teaching and they
were quite pleased about the many illustrative examples in the
Monograph.
To sum up, the above comprehensive analyses in terms of
structured format ratings and open-ended responses supports
the conclusion that the general MKF was successfully gener-
ated and elaborated from the perspective of these two panels of
reviewers though the two panels somewhat disagree with each
other in their evaluation of The Monograph, both chapter by
chapter and as a whole, due to their different teaching experi-
ence, academic expertise, and research interest. Such disagree-
ments are not markedly different from their agreements.
Conclusion
One can conclude that this monograph validation study,
which was Phase II of the instructional material development
model used to guide the development of The Monograph, was a
reasonable success. The various analyses done of the detailed
structured format ratings and open-ended responses of the aca-
demic and practitioner expert panels strongly supported the
adequacy, quality and validity of the monograph developed
using the instructional materials development model used. The
various kinds of feedback provided by the different kinds of
reviewers both structured and open-ended also helped guide the
revisions of The Monograph (Phase III) after the validation
study was completed and analyzed. Both Phase I and Phase II
of this work, as well as the revisions done to The Monograph as
a results of the independent expert feedback obtained from the
two panels in Phase II reported here, give new and very nu-
anced meanings and conceptual frameworks for the current
much used phrase of “evidence-based practice,” as The Mono-
graph reflects and is evidence-based and evidence-driven from
the primary research and scholarly literature to the rich and
nuanced knowledge and evidence of various kinds and types of
practitioners who also provide research evidence and even dis-
coveries when they participate in formal reviewer panels such
as those used in this study.
The panel validation process used in this study was a suc-
cessful implementation of an adapted convergent and discrimi-
nant validation design for assessing the quality of developed
instruction materials, which generated a variety of different
kinds of information and results that were externally and ecol-
ogically validating, informative and useful in revisions. The
results also generated a better understanding of the instructional
materials development process and particularly for bi-lingual
and bi-cultural instructional materials. The agreement among
the bilingual and bicultural formal reviewers indicated that the
two different groups of reviewers had “convergent” (similar)
views and opinions about The Monograph, and this conver-
gence of views provided strong evidence for the logical validity
and theoretical or construct validity of The Monograph. The
disagreement between the two review panels revealed how
teaching experience, academic expertise, and research interest
can influence reviewers’ assessments of each chapter of a work
(The Monograph) and the work (The Monograph) as a whole to
some degree but a degree that does not deviant markedly from
the consensus view. The reviews of reviewers, therefore, are
not completely objective and absolute, or completely the re-
verse, and must be contextualized to some degree to be appro-
332

W. XU ET AL.
priately interpreted, which is a metacognition if there was ever
one which should not be forgotten! The pattern of findings in
the present study, as previously stated, cross-validate those of
several other recent studies that used the instructional materials
development model used in this study (i.e., Flores, 2005; Erik-
son, 2006; Kwong, 2008) not only strengthening the findings of
all of these studies and validating the instructional materials
development model used further, but in the case of this study
extending these finding to a bilingual and bi-cultural context
and international audience. As previously stated, this finding is
a very hopeful outcome and particularly so for cross-cultural
scholarship, research, instructional materials development and
instruction itself if all of the aforementioned as model and the-
ory driven as in the case of this study and the similar studies on
which it built.
The validation process used in this study, it should be noted,
also provided the authors of this article and all the reviewers
involved in this validation process and phase of The Mono-
graph’s development a workable platform to openly discuss,
criticize, evaluate, and reflect on The Monograph and metacog-
nition and the metacognitive knowledge needed for summariz-
ing strategies instruction with expository text to EFL under-
graduates. This type of interactional among communities of
professionals is a form of professional development and profes-
sional education and a type of interdisciplinary scholarship of
its own between academics and practitioners that needs to be
encouraged and further developed for both better academic un-
derstanding and better pedagogy and pedagogical approaches
and instructional materials. Both academics and practitioners
have much to learn and gain from each other that is of value to
their work and thus makes better overlapping halves of a whole.
Such types of educational communication and communities are
easier and more feasible today with the rapidly emerging social
media technology available, together with models of the cogni-
tive apprenticeship that are becoming increasing more popular
approaches to instructional and professional development in all
professional areas currently. Such types of educational commu-
nication and communities are still not widely employed, con-
sidering the teacher preparation/training programs going on in
both the USA and China, which do not include metacognition
and developing teachers’ understanding of metacognition of
their own or their students or how it works in instruction. The
instructional materials and educational works development mo-
del presented in this article strongly “blends with” and enables
these emerging approaches improving all those who participate
in various ways.
One of the major findings of this research work, which was a
surprising unanticipated findings to us, was how fuzzy, jumbled
and contradictory the area and differing views of metacognition
are today, and particularly as it applies to instructional and
actual pedagogy and pedagogical strategies for implementing
various exhortations from various academics and researchers to
teach reading and language metacognitively and to develop the
metacognitively sophisticated reader (Hartman, 2001; Pressley,
2002). Key aspects of the fuzzy, jumble and contradictory
views that are present into the literature were summarized at the
beginning of this article (and are presented in more detailed in
The Monograph) but if there is one thing that the instructional
materials developed model used to guide this work did and did
extraordinarily well, it was to both identify and expose this
problem of how fuzzy, jumble and contradictory the area of
metacognition was relative to developing cogent and compre-
hensive instructional materials about it. More subjective and
narrative forms of instructional materials development, which
are the more typical approach, would not have brought all of
these problems to light as they have not for more than twenty
years, and also would not have necessitated that these problems
be reconciled and resolved and so in a manner that was consid-
ered to be adequate and acceptable to an independent panel of
expert reviewers before the work developed to the point of
completed instructional materials ready for independent expert
review. Such a process and formal approach is not how instruc-
tional materials are typically developed and particularly so by
academics and practitioners as opposed to publishing houses.
This is the great benefits of the guiding model used. In fact, it
was the MFK that was revised several times before the writing
of The Monograph commenced and then several more times as
the writing progressed. That was one of the most notable as-
pects and events of this study. This very same point held rela-
tive to the task of trying to analyze and unravel the different
kinds of metacognitive knowledge that were needed to learn
and teach summarizing strategies for expository texts, which
also underwent numerous revisions both before and as the
writing of The Monograph progressed. Therefore, every in-
struction and instructional materials developer needs a guiding
model for carrying out such work as well as a community of
peers and colleagues to help in such work and the production of
such products. Every developer needs to both know and under-
stand as well that they are engaged in a new form of scholarship
that is not well-recognized or well developed yet but needs to
be if we are to better understand how theory and research get
translated into successful practice and successful instructional
materials, particularly relative to teaching metacognitively and
developing metacognitive readers.
REFERENCES
Almasi, J. F. (2003). Teaching strategic processes in reading. New
York: Guilford Press.
Baker, L. (1989). Metacognition, comprehension monitoring, and adult
reader. Educational Psychology Review, 1, 3-38.
doi:10.1007/BF01326548
Baker, L., & Brown, A. L. (1984). Metacognitive skills and reading. In
P. D. Pearson, R. Barr, M. L. Kamil, & P. Mosenthal (Eds.), Hand-
book of reading research (pp. 353-394). New York: Longman.
Brown, A. L. (1985). Metacognition: The development of selective
attention strategies for learning from texts. In H. Singer, & R. B.
Ruddell (Eds.), Theoretical models and processes of reading (pp.
501-526). Newark: International Reading Association.
Brown, A. L., Campione, J. C., & Day, J. (1981). Learning to learn: On
training students to learn from text. Educational Researcher, 10, 14-
21. doi:10.3102/0013189X010002014
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant
validation by the multitrait-multimethod matrix. Psychological Bul-
letin, 56, 81-105. doi:10.1037/h0046016
Carifio, J. (1975). A standard and command-wide model for developing
scientific and technical instructional materials. Alexandria, VA:
United States Office of Naval Research.
Carifio, J. (1977). Toward a macro model of instructional components.
Proceedings of the Annual Meeting of Eastern Educational Re-
search Association. Boston, MA.
Carifio, J. (1990). A model for empirical developing and validating
instructional text and materials. Proceedings of the Annual Confer-
ence of the American Educational Research Association. Boston,
MA.
Carifio, J. (2003). An instructional text evaluation protocol. Annual
Conference of the New England Educational Research Organization.

W. XU ET AL.
334
Portsmouth, NH.
Carifio, J., & Perla, R. (2007). Ten common misunderstandings, mis-
conceptions, persistent myths and urban legends about Likert scales
and Likert response formats and their antidotes. Journal of the Social
Sciences, 3, 106-116. doi:10.3844/jssp.2007.106.116
Carifio, J., & Perla, R. J. (2010). Towards the decline and fall of radical
and educational constructivism. Current Research in Psychology, 1,
1-15. doi:10.3844/crpsp.2010.1.15
Chamot, A. U., & O’Malley, J. M. (1994). The CALLA handbook: Im-
plementing the cognitive academic language learning app roach. White
Plains, MA: Addison Wesley Longman.
Cochran, K. F., De Ruiter, J. A., & King, R. A. (1993). Pedagogical
content knowing: An integrative model for teacher preparation.
Journal of Teacher Educat i o n, 44, 263-272.
doi:10.1177/0022487193044004004
Dagostino, L., & Carifio, J. (1994a). Evaluative reading and literacy: A
cognitive view. Boston, MA: Allyn and Bacon.
Dagostino, L., & Carifio, J. (1994b). Establishing the logical validity of
instructional activities for teaching reading evaluatively. Journal of
Reading Improvemen t , 31, 14-22.
Erikson, L. (2006). An integrated approach to citizenship education for
grades 1-8 in the 21st century. Ed.D. Thesis, Lowell: University of
Massachusetts.
Fenstermatcher, G. D. (1994). The knower and known: The nature of
knowledge in research on teaching. Review of Research on Teaching,
20, 3-56.
Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new
area of cognitive-developmental inquiry. American Psychologist, 34,
906-911. doi:10.1037//0003-066X.34.10.906
Flavell, J. H. (1985). Cognitive development (2nd ed.). Englewood
Cliffs, NJ: Prentice-Hall.
Flores, M. (2005). Instructional strategies, conditions, characteristics,
and contexts for successfully teaching Hispanic Caribbean students.
Ed.D. Thesis, Lowell: University of Massachusetts.
Fradd, S. H., & Lee, O. (1998). Development of a knowledge base for
ESOL teacher Education. Teaching and Teacher Education, 14, 761-
773. doi:10.1016/S0742-051X(98)00023-7
Garner, R. (1987). Metacognition and reading comprehension. Nor-
wood, NJ: Ablex.
Glaser, R., & Chi, M. T. (1988). Overview. In M. Chi, R. Glaser, & M.
Farr (Eds.), The nature of expertise (pp. 15-28). Hillsdale: Erlbaum.
Griffith, P. L., & Ruan, J. (2005). What is metacognition and what
should be its role in literacy instruction? In S. E. Israel, C. C. Block,
K. L. Bauserman, & K. Kinnucan-Welsch (Eds.), Metacognition in
literacy learning: Theory, assessment, instruction, and professional
development (pp. 3-18). Mahwah: Lawrence Erlbaum Associates.
Grossman, P. (1990). The making of a teacher. New York: Teachers
College Press.
Grossman, P., & Richert, R. (1988). Unacknowledged knowledge
growth: A reexamination of the effects of teacher education. Teach-
ing and Teacher Education, 4, 53-62.
doi:10.1016/0742-051X(88)90024-8
Gudmundsdottir, S. (1991). Values in pedagogical content knowledge.
Journal of Teacher Educat i o n, 41, 44-52.
doi:10.1177/002248719004100306
Hartman, H. J. (2001). Teaching metacognitively. In H. J. Hartman
(Ed.), Metacognition in learning and instruction (pp. 149-172). Bos-
ton: Kluwer Academic Publishers.
Kerlinger, F., & Lee, H. (2000). Foundations of behavioral research.
Orlando, FL: Harcourt College Publishers.
Kintsch, W., & van Dijk, T. A. (1978). Toward a model of text com-
prehension and production. Psychological Review, 85, 363-394.
doi:10.1037/0033-295X.85.5.363
Knowles, M., Holton, E., & Swanson, R. (2005). The adult learner:
The definitive classic in adult education and human resource de-
velopment (6th ed.). Burlington, MA: Elsvier Press.
Kwong, B. (2008). The development and validation of a research-based,
tiered new teacher induction program guide for Massachusetts pub-
lic schools. Ed.D. Thesis, Lowell: University of Massachusetts.
Lai, E. R. (2011). Metacognition: A literature review.
http://psychcorp.pearsonassessments.com/hai/images/tmrs/Metacogniti
on_Literature_Review_Final.pdf
López, F., Scanlan, M., & Gundrum, B. (2013). Preparing teachers of
English language learners: Empirical evidence and policy implica-
tions. Education Policy Analysis Archives , 21.
http://epaa.asu.edu/ojs/article/view/1132
Lovett, M. (2008). Teaching metacognition.
http://net.educause.edu/upload/presentations/eli081/fs03/metacogniti
on-eli.pdf
Pellitier, P. (2004). Towards a reader-text-context theoretical model for
reading literary work. Ed.D. Thesis, Lowell: University of Massa-
chusetts.
Perla, R. J. (2006). Use and augmentation of a formal model and theory
to develop instructional material to teach undergraduates about the
nature of science, scientific knowledge and scientific change. Ed.D.
Thesis, Lowell: University of Massachusetts.
Perla, R. J., & Carifio, J. (2011). Theory creation, modification, and
testing: An information-processing model and theory of the antici-
pated and unanticipated consequences of research and development.
Journal of Multidisciplinary Evaluation, 7, 84-110.
Pressley, M. (1990). Cognitive strategy instruction that really improves
children's academic performances. Cambridge, MA: Brookline
Books.
Pressley, M. (2002). Metacognition and self-regulated comprehension.
In A. E. Farstrup, & S. J. Samuel (Eds.), What research has to say
about reading instruction (pp. 291-309). Newark: International Read-
ing Association. doi:10.1598/0872071774.13
Pressley, M., Borkowski, J. G., & Schneider, W. (1987). Cognitive
strategies: Good strategy users coordinate metacognition and knowl-
edge. Annals of Child Development, 4, 89-129.
Pressley, M., & Woloshyn, V. (1995). Cognitive strategy instruction
that really improves children’s academic performance. Cambridge,
MA: Brookline Books.
Purdie, N., & Hattie, H. (1996). Cultural differences in the use of self-
regulated learning. American Educational Research Journal, 33, 845-
871. doi:10.3102/00028312033004845
Reynolds, R. E. (1992). Selective attention and prose learning: Theo-
retical and empirical research. Educational Psychology Review, 4,
345-391. doi:10.1007/BF01332144
Schneider, W., & Pressley, M. (1989). Memory development between 2
and 20. New York: Springer-Verlag.
doi:10.1007/978-1-4613-9717-5
Shulman, L. S. (1986). Those who understand: Knowledge growth in
teaching. Educa t i on a l Researcher, 15, 4-14.
doi:10.3102/0013189X015002004
Shulman, L. S. (1987). Knowledge and teaching: Foundations of the
new reform. Harvard Educational Vie w, 57, 1-22.
Van Dijk, T. A. (1980). Macrostructures: An interdisciplinary study of
global structures in discourse, interaction, and cognition. Hillsdale:
Erlbaum.
Van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse com-
prehension. New York: Academic Press, Inc.
Xu, W., Carifio, J., & Dagostino, L. (2012). Constructing a metacogni-
tive knowledge framework for post-secondary EFL reading teachers’
summrizing strategies instruction with expository text: A case study,
phase I. Creative Education, 3, 829-839.
doi:10.4236/ce.2012.326124
Zimmerman, B. (2008). Investigating self-regulation and motivation:
Historical background, methodological developments, and future
prospects. American Educational Research Journal, 4 5 , 1, 166-183.
doi:10.3102/0002831207312909