2011. Vol.1, No.4, 151-155
Copyright © 2011 SciRes. DOI:10.4236/sm.2011.14019
Consultant Evaluator, Perugia, Italy
Received May 16th, 2011; revised July 24th, 2011; accepted September 7th, 2011.
Among the evaluation techniques based upon group queries (e.g. focus group), brainstorming does not enjoy
particular consideration. This might be the result of its origin and development within organizational and mana-
gerial domains, traditionally focused more on “idea production” (and problem solving) than on idea analysis
within the context of evaluational and social research. This paper presents a development of classical brain-
storming, which is quite useful to evaluation, where the traditional idea-producing step is followed by group
analysis and exploration of the shared evaluand-specific semantic space. This evaluational brainstorming is the
result of a shared understanding of the evaluand by different stakeholders, who can now ascertain their goals and
draw cognitive maps to guide subsequent methodological choices and data gathering requirements.
Keywords: Brainstorming, Participate Evaluation, Social Research, Research Techniques
Brainstorming has yet to become one of the common tech-
niques of evaluators and social researchers alike because of its
significant identification with business consultants, and its lack
of operative impact. Brainstorming is an exploratory technique,
open to new meanings, proposals and ideas, but it seems that it
does not provide operative synthesis, useful to evaluational
Nevertheless, once innovatively upgraded, brainstorming can
become a very powerful tool, allowing:
The evaluand’s goal clarification .
A so-called evaluand’s semantic exploration, in order to
define the correct evaluational path and choose the best
An agreement, among the different stakeholders participat-
ing in the evaluation, on shared meanings, which constitutes
the crucial element needed to share processes and outcomes.
In this article I plan to describe the “evaluational brainstorming”,
i.e. a brainstorming t ailored to support eval uational research. T his
methodological proposal has been successfully experimented
upon in more than ten years of evaluation activities, on different
themes, with groups of different nature and size, experimenting
with different operational modes. This paper thus become a syn-
thesis of the best solutions experimented on so far1.
In almost the same period in which Lazarsfeld and Merton
created the focus group at the Bureau of Applied Social Re-
search, an American business consultant successfully started a
movement to develop creativity within the manufacturing in-
dustries (in addition to arts and sciences), offering innovative
analytical solutions to problem solving. One of these solutions,
the so-called brainstorming, showed an unusual approach to
groups and had autonomous fate and fame, distinct from the
larger body of original proposals. The consultant was Alex
Osborn, and in his major work, Applied Imagination, he dedi-
cated to brainstorming only a few pages at the end of the last
Since then, brainstorming has been implemented by a great
number of managers and consultants, and studied by a few
psychologists, without eliciting much interest among social
researchers. In addition to its origin and non-academic milieu,
brainstorming was seen as lacking a real research application.
After all, this is a group management technique geared to ge-
neric “idea production”. In fact, the term has been adopted by
our common lexicon and associated to the free (and often cha-
otic) expression of one’s thoughts. “To have a brainstorm with
my peers” means “let’s say what comes to mind about our
problem, without formality or pretensions of expressing fully
articulated thoughts from the start.” This approach is useful in
starting design processes (resea rch included), but does not con-
vey information within a structured design of social analysis.
The undeniable revival of constructivist approaches in
evaluational research, and the offerings of techniques based on
a dialogic exchange within a group (e.g. focus groups) has re-
newed the interest on brainstorming as well. With a simple
condition: beyond its function as a problem solving tool, brain-
storming has to be shown to detect information within the
framework of research methodology.
1A much more exhaustive version, including variants, specific cases and a
robust theoretical apparatus supporting the technique, is available in Italian:
Claudio Bezzi and Ilaria Bal dini, Il brainstorming. Pratica e teoria, Franco
Angeli, Milan, Italy 2006.
2Osborn, A. F. (1953). Applied imagination. Principles and procedures o
creative th inki ng, New York, NY: Charles Scribner’s Sons. Although brain-
storming literature is quite vast, normally consists of operator’s manuals,
written for professional corporate consultants and managers, or superficial
“ten-step” quick guides. A minimal bibliography, useful to further the study
of classical brainstorming includes Rawlinson 1986; Cory and Slater 2003;
Rich 2003; Streibel 2003. On electronic brainstorming, all legitimate doubts
aside, see Larey et al. 1996; Kay 1994. In the 90’s an animated debate too
place on t he al leged valu e of g roup int eraction in b rain stormin g, con tras ting
the well known positive strength value usually attributed to it. On this sub-
ect, see Diehl and Stroebe (1987; 1991); Mullen, Johnson and Salas (1991);
Stroebe and Diehl (1991; 1994); and Stroebe, Diehk and Abakoumkin
Such a result could be achieved introducing a new and im-
proved version of this technique, which I call “evaluational
The evaluational brainstorming can find several applications,
but its main feature is its suitability to explore a concept or an
evaluand and build its possible conceptual map. Brainstorming
final product is a set of indicators of the explored concept
(similarly, in evaluation, a set of evaluand’s indicators) built up
by a group of experts, operators, and relevant stakeholders. It is
a group process and, as such, tends to define a shared represent-
tation of that concept or evaluand.
How Does Evaluational Brainstorming Work?
The proposed brainstorming is carried out in the following
Creative phase (Osborn’s classical brainstorming).
Classification phase, an inductive proce s s.
Synthetic phase, a deductive process.
Space considerations force us to overlook important prelimi-
nary remarks on group selection and recruiting, the initial wel-
come of the participants, and the set and setting in which the
brainstorming session should take place. By and large, the in-
dications are well known, similar to those applied to likely
techniques (such as the focus group) and dictated by a modicum
of common sense. Follows a summary of brainstorming man-
agement, phase by phase.
Firstly, the facilitator describes the rules of the creative phase
of brainstorming as follows:
Produce as many ideas as possible.
Do not censor yourself.
Do not censor others.
With minimal modifications, these are the rules set by
Osborn himself, and have remained unchanged for decades. In
reality, Osborn’s fundamental idea is the separation between
the moment of idea production and the critique and discussion
of the ideas thus produced. In fact, the critique can bring the
meeting to a halt, making it sterile with excessive critical fervor
aimed to the ideas which are voiced first, to the detriment of
other proposals to be explored—at a later time—in order to find
the best solution to the problem at hand.
Subsequently, the facilitator explains that “no censorship”
means avoiding both considering trite one’s ideas, and devalu-
ating the ideas of others, maintaining an open attitude towards
different perceptions and interpretations.
Thus, the creative phase consists of an exercise in research-
ing spontaneous ideas, concepts, and expressions pertaining to
the focal issue. The facilitator can offer: “Well, now you can
start saying everything that comes to mind on the brainstorming
subject. What counts is the number of produced ideas and con-
cept, therefore I would urge you to respect the no-censorship
The facil itator immedi ate ly write s each i dea expre ssed by the
participants on a white board. If someone expresses an articu-
lated thought to be retranslated in a few words, the facilitator
should ask to restate and simplify it, reminding the group that
s/he can write on the white board only single words, single
concepts or short expressions, the so-called ‘strings’.
From the linguistic standpoint, the strings are quite a diversi-
fied set of linguistic signs, which can include the following:
Predicative sentences (or simpler still: sentences with a verb,
which are grammatically correct and obvious, i.e. “Pay-
ments are always late”; “The staff is quite tired of it”; “We
need to pay attention to the user”; etc.);
Nominal sentences (without a verb, such as “Enough of it!”
or “Slave drivers!”);
Various rhetorical figures, mainly: metaphors (“That’s an-
other can of worms”; “We are like sheep among wolves”;
etc.), metonymies (“The Man never leaves the upper floor”;
“We do not need brain, just a strong stomach”; etc.) and
synecdoche (“The service works”, to indicate staff, struc-
tures and services of a specific center or unit; “Our bread
and butter”; etc.); allusions, antiphrases (e.g. “We have
everything we need!”, meaning we lack everything), pe-
riphrases, and the vast rhetorical and stylistic spectrum,
which is as diversified as the participants and as under-
standable as their culture and the overall context (when in
doubt, and while s/he writes, the facilitator asks “what does
Deixis and other exophora (or “indexical” sentences, which
can be understood only within their context, such as “Up to
here [with an ostensive gesture]”; “They [the users] keep
Single words (substantives, adjectives, verbs, inflected in
various ways, which acquire a specific meaning according
to their reference context, for example: “Honesty [a team
quality]”; “Training [what is required]”; “Traveling [the
result of a new policy]”; etc.).
This listing suffices to show the extreme heterogeneity of
strings produced during a brainstorming (first phase). Generally,
predicative sentences are rare, and it is hard to imagine a brain-
storming that produce only predicates. The session would be
stifled and boring, lacking the typical creative stage rhythm.
But a brainstorming producing only words, as exemplified last
(the most frequent type) would equally sound a bit too ritual,
especially if is not interleaved with nominal sentences (in gen-
eral, exclamations), metaphors and deixis. Usually, the indi-
viduals who are more creative, restless, expert, and mature (in
other terms and in all probability, the leaders) tend to both
break the order of predicates and simpler lexemes and to pro-
voke the group, explicitly highlighting a concern or indicating a
dimension by means of targeted emphasis, metaphors, me-
tonymies, exophora, and stronger expressions.
Strings thus produced become semantic tiles, the basic units
that describe the object of interest.
Overall, the creative phase has the following objectives: ex-
ploration; agreement on meanings; and interpretational expres-
sion of the object of interest (the central concept). This is
achieved through an untidy and disorderly, albeit exhaustive
The facilitator can think in advance the main areas/dimensions
featured by the central concept, on the basis of his/her mandate
and brainstorming objectives.
Once this set-up is in place, the facilitator can concentrate
his/her attention on some dimensions which are considered
conceptually important. For example, if the research is focused
on “elderly home care”, areas such as “funding”, “organization”
or “home care objective definition” cannot be forgotten or ig-
nored. If the group participants do not address such issues, the
facilitator has the power to discretely suggest them, perhaps at
session’s end or in a moment of rest and group disorientation.
Classific ation Phase
A this stage, “classic” brainstorming is over and the facilita-
tor has to deal with a large amount of strings waiting to be ana-
lyzed. According to classic routines, the activities of analysis,
selection and implementation of the “best” ideas are performed
by the manager, a manager’s designee, a small group, etc. Gen-
erally, the group that has generated those ideas is not involved,
the reason being that in a business environment ideas need to be
adapted to the firm’s overall strategy, the company mission, or
simply the CEO’s whims.
String analysis is the divide bridged by our brainstorming.
C. BEZZI 153
Starting with brainstorming second phase, the analysis is per-
formed by the very group producing the ideas, according to
completely different rules, which must be explained to the par-
Because the goal of evaluational brainstorming is the induc-
tive construction of an indicator set for the explored concept,
the group needs to be guided toward specific cognitive active-
ties, in order to reflect on the possible groupings, by homoge-
neous classes, of the collected strings. Classification—brain-
storming second phase—is implicitly associated to the concept
dimensions, and must be achieved by inductive processes. In-
stead of establishing classes in order to assign strings to them, it
is paramount to work on the strings, massing them together
until the group judges that the emerging classes are coherent. It
is a true work of meaning’s construction, avoiding as much as
possible the burden of past ideas about and upon the concept.
The group is focused on the individual strings (see below) and
mediates their semantic and pragmatic values by pairing them
more and more in larger clades, which only afterward show
their inner coherence as classes (dimensions) both homogene-
ous and characterized.
When the first phase is considered completed, the facilitator
invites the group to review its output and, starting with the first
string, suggests: “Now, our task is to assemble or put together
strings that appear to have some elements in common, no mat-
ter the reason. To this effect, if this is the first string, I’ll read
those following and you’ll tell me which ones have something
in common with the first. I’ll place a marker (e.g.: a red triangle)
next to the strings you select as similar or coherent to the first.
Then we’ll repeat the process with another string, using a dif-
ferent marker (for instance, a blue square), and we’ll continue
until we will analyze all of them.”
This task presents the following facets:
It requires the creation of homogeneous groups (also classes,
sets, categories or families, as long as the participants un-
derstand the concept), containing strings linked by semantic
affinities. He nce the “classification” phase label.
If conceptually more abstract, a string can be marked by
several symbols, because it shares diverse elements with
many other strings, thus belonging to more than one class.
When they are too abstract, strings can produce the so-
called “omnibus” classes, containing a large quantity of
broad-spectrum strings that are both generic and somewhat
useless in terms of further processing. If the facilitator sus-
pects such a classification, s/he should express his/her con-
cern, guiding the analysis toward other classification crite-
Classes should have as much as possible a limited semantic
extension, without assuming too abstract a feature;
The classification phase is based upon an additive method,
because it requires the analysis of one string at a time,
judging whether it can be associated to the reference string.
In this way the group does not immediately perceive the
emerging classes, but simply defines the logical nexus
linking a string to othe r s.
As a result, different classes are formed, each marked by a
These classes can be immediately named. This is much easier
when the meaning of each class is finite and the participants
recognize by themselves that they are dealing with a class as
such. In some cases, though, a strong connotation is absent (e.g.:
because the class is too abstract, see above.)
Nevertheless, to conceptually define the meaning of these
classes is a task reserved to the third, synthetic phase. In the
current phase, the group discussion may focus on whether it is
appropriate to include a given string in one class or another.
This approach encourages the expression of the links between
concepts (or strings). By the same token, because in this second
phase the group builds reality representation criteria (thru clas-
sification), it is quite likely that—at some point—the group
expresses diverging visions. In reality, whether to include or
not a string in a given class is not a purely logical operation, but
impinges upon the string interpretation offered by each indi-
vidual actor. Thus, this interpretation shares to a fair degree the
semantic plane (pertaining to words’ meaning), and is quite
convergent to the pragmatic level (pertaining to words’ usage).
Obviously, the first phase rules (do not censor, do not criti-
cize) do not apply any longer. Even better, it is inevitable and
often useful, to have a discussion clarifying and mediating the
shared framework of meanings.
The third phase is dedicated to indicators.
It is preceded by a break during which the participants can
relax and the facilitator can transcribe all of the strings on
separate sheets, one for the strings with the red triangle (which
by now could have been named by the group), another for the
strings with the blue square, etc.
The group will work class by class (or dimension by dimen-
sion), without the confusing interference of page after page of
strings decked with different symbols, and will have the oppor-
tunity to observe the whole picture and—deductively—catching
its essential elements that lead to the indicators.
The facilitator starts: “Please note that the strings are organ-
ized according to the previous classification. Now we should
revisit the strings order to find further subgroups within them.
Observe the first sheet and tell me if you can find new string
groupings within this class, based on their common elements”.
At this point, the following can happen:
No further groupings emerge within a class. This happens
when the class contains a few strings very coherent and se-
mantically close. In this case, the participants can only at-
tribute a name to the class, if it doesn’t have one already;
Several groupings emerge within a class. These groups
clearly refer to elements, processes, objects, concepts that
can be differentiated even within a class already defined as
homogeneous. In this case, the facilitator marks with the
same symbol, (e.g., a number) the strings that cam be
grouped within that class, asking the participants to name
the sub-class thus formed. The work continues for all
strings within the class.
The third phase goals can be summarized as follows::
Recognition of the classes built during phase 2.
Definition of sub-classes within said classes and their de-
Attribution of meaning to the work done so far, ordering
what was disorganized.
The final result of the third phase is the individuation of in-
dicators of the explored concept. In the same way in which
classes were related to the concept dimensions, its specific ele-
ments are referred to indicators.
Generally speaking, in reality the group is never confronted
with ideas such as concepts, dimensions and indicators. In fact,
this approach generates inductive group thinking, linked to a
communication pragmatics process, fostering the group’s stipu-
lation of reality’s particular meaning within the context of the
From Brainstorming Strings to Indicators
This process is the net opposite of the logic-deductive opera-
tion performed by the researcher and his team while involved in
their arm-chair conceptual exploration, the Top-Down process
typical to Lazarsfeld’s paradigm (Lazarsfeld 1958).
Figure 1 shows a simplified version of Lazarsfeld’s paradigm,
illustrating the role of strings and the differences between the
two approaches. As shown in this figure, Lazarsfeld’s paradigm
(which is basically rationalistic, typically adopted by researches,
including those concerned with evaluation) is a deductive proc-
ess that explores the concept breaking down its dimensions, and
eventually its sub-dimensions (from top to bottom in Figure 1).
From these sub-dimensions, the indicators are finally chosen
and further defined at the operative level (the empirical mode of
information detection). On the contrary, our brainstorming is an
inductive process. The search for indicators is guided by their
logical underpinnings and culturally implicit factors, and it is
by progressive accretion that the concept emerges equivalent to
the research object and—in evaluation—the evaluand (from
bottom to top in Figure 1).
To work inductively with a group is to search latent mean-
ings, sedimentations of professional and corporate culture,
idiomatic expressions. This is the material expressed by the
strings that are generated during the first phase of brainstorm-
ing. They form the raw material from which the indicators are
forged. According to inductive and group logics, the translation
of strings into indicators requires the study of the concept di-
mensions, which takes place during the second phase.
Naturally, the basic approach is constructivist, and it pivots
on the shared construction of a given reality. When brain-
storming is used in evaluation, this framework is very useful,
especially when the participants are also the evaluation future
users. In this case, we have to deal with their corporate culture,
the very way in which they work and represent reality (Bezzi
The following (clearly symbolic) elements of Figure 1 de-
The strings are not the indicators’ further sub-specification
(as the case concept/dimensions/indicators), but autono-
mous linguistic expressions. In most cases, they are linked
to specific indicators (as shown by the dashed arrows con-
necting strings and indicators). It is also easily seen (both in
Figure 1 and brainstorming sessions) that some strings are
directly referred to dimensions or, more rarely, the very
concept. Our goal is to extract indicators from the collected
strings, but each string is per se neither a ready-made indi-
cator nor one of its parts.
Strings can characterize several dimensions and indicators
at once, exactly as each indicator can refer to more than one
dimension or concept at a time.
It is possible to end with oddly placed strings. These strings
appear to refer to dimensions the group cannot isolate and
identify (see the far lower right string, referred to an un-
known and dashed indicator). Last but not least: the sign
and pragmatic context in which the strings are somewhat
freely placed in this or that class is neither arbitrary nor
random. In reality it should be viewed as the analogy to the
Lazarsfeld’s principle of index interchangeability:
Each indicator has a specific character and should never be
considered as fully representative of the searched for classifica-
tion (Lazarsfeld, 1958).
Similarly and a fortiori, due to their proto-indicator nature,
strings represent categories only in an approximate, circum-
stantial and, more importantly, contextual fashion.
Dimension 1 Dimension 2Dimension 3
Subdimension 1 Subdimension 2
Indicatore 7Indicatore 6Indicatore 8
Indicator 7Indicator 6Indicator 8
Concept as evaluand: the
research's object at the
core of brainstorming.
Dimensions or the
evaluand's parts, sectors
or main distinctive
features: they become
evident at the end of
brainstorming second or
classification p h ase.
brainstorming end result:
they surface at the end of
the last or synthetic
dur i ng the first or
creative phase of
brainstorming; they are
linked to specific
only after the
classification phase has
Lazarsfeld’s paradigm and brainstorming logic.
C. BEZZI 155
(Partial) example of evaluational brainstorming rendering about an alcohol rehabilitation service.
In any case, brainstorming rendering, and its subsequent re-
search uses (from the transformation of indicators into
questions of a survey to conceptual mapping to be used in
evaluation) must preserve as much as possible the prag-
matic indicators’ substrate built up during phase 3. To this
effect, each indicator, in addition to its group-selected de-
nomination, is shown with its string set, because of its high
evocative value, especially felt by the very group that gen-
erated them (Figure 2 shows an example of evaluational
Bezzi, C. (2006). Evaluation Pragmatics. Evaluation, 12, 56-76,
Bezzi, C., & Baldini, I. (2006). Il brainstorming. Pratica e teoria. Mi-
lan, Italy: Franco Angeli,.
Cory, T., & Slater T. (2003). Brainstorming. Techniques for new ideas.
Lincoln, NE: iUnive rse Inc.
Diehl, M., & Stroebe, W. (1987). Productivity loss in brainstorming
groups: Toward the solution of a riddle. Journal of Personality and
Social Psychology, 53, 497-509. doi:10.1037/0022-3522.214.171.1247
Diehl, M., & Stroebe, W. (1991). Productivity loss in idea-generating
groups: Tracking down the blocking effect. Journal of Personality
and Social Psychology, 61, 392-403.
Kay, G. (1994). Effective meetings through electronic brainstorming.
Journal of Management Development, 14, 4-25.
Larey, T. S., Leggett, K. L., Paulus, P. B., Putman, V. L., & Evelyn , J.
R. (1996). Social influence processes in computer brainstorming.
Basic and Applied Social Psychology, 18, 3-14.
Lazarsfeld, P. F. (1958). Evidence and inference in social research,
Dedalus, 87, 99-109.
Mullen, B., Johnson, C. & Salas, E. (1991). Productivity loss in brain-
storming: a meta-analytic integration, Basic and Applied Social Psy-
chology, 12, 3-23. doi:10.1207/s15324834basp1201_1
Rawlinson, J. G. (1986). Creative thinking and brainstorming, Alder-
shot: Gower Publishing.
Rich, J. R. (2003). Brain storm. Tap into your creativity to generate
awesome ideas and remarkable results. Franklin Lakes, NJ: Career
Streibel, B. J. (2003). The manager’s guide to effective meetings. New
York, NY: McGraw-Hill.
Stroebe, W., & Diehl, M. (1991). You can’t beat good exp eriments with
correlation evidence: Mullen, Johnson, and Sala’S meta-analytic mis-
interpretations. Basic and Applied Social Psychology, 12, 25-32.
Stroebe, W. & Diehl, M. (1994). Why groups are less effective than
their members: on productivity losses in idea-generating groups,
European Review of Social P sy c h ol ogy, 5, 271-303.
Stroebe, W., Diehl, M., & Abakoumkin, G. (1992). The illusion of
group effectivity, Personality and Social Psychology Bulletin, 18,