Concept coding

When we score texts for their complexity level, we deliberately avoid basing scores on their particular conceptual content. The result is a set of scores that show how texts are related to one another on the hierarchical complexity metric. These scores tell us nothing about the particular conceptual content of the texts (e.g., how meanings change across lectical levels or vary within developmental levels). Whereas scoring involves looking through the content to the structure, concept coding stays at the level of content. This means paying attention to what is said (particular meanings) rather than how it is said (structure). After texts are coded for both conceptual content and complexity level we examine their relation. To assure that concept coding and complexity analyses are independent, we employ different analysts to conduct each type of analysis. Whereas Lectical™ analysts must undergo many hours of training, concept coders can usually be trained to do competent coding (with supervision) in a few hours.

When we score texts for their complexity level, we deliberately avoid basing scores on their particular conceptual content. The result is a set of scores that show how texts are related to one another on the hierarchical complexity metric. These scores tell us nothing about the particular conceptual content of the texts (e.g., how meanings change across complexity levels or vary within developmental levels). Whereas scoring involves looking through the content to the structure, concept coding stays at the level of content. This means paying attention to what is said (particular meanings) rather than how it is said (structure). After texts are coded for both conceptual content and complexity level we examine their relation. To assure that concept coding and complexity analyses are independent, we employ different analysts to conduct each type of analysis. Whereas Lectical™ analysts must undergo many hours of training, concept coders can usually be trained to do competent coding (with supervision) in less than an hour.

Texts are coded at various levels of specificity. We refer to these as levels of analysis. These are typically represented by 3 or 4 conceptual categories, referred to as domains, themes, sub-themes (when they emerge), and concepts. These are conceptualized as follows:

Domains	This is the broadest category. A domain is a very general area of knowledge or collection of skills with some common element. Domains should be circumscribed by the logical or inferential relatedness of their elements (e.g. connections between concepts). Moreover, because domains are defined via inferential connections it should be recognized that domains shade into one another and are often circumscribed purely for heuristic purposes. Nevertheless, domains are not arbitrarily circumscribed. For example, while morality and leadership share many concepts, leadership and physics clearly do not.
Themes	The themes of a given domain are the overarching conceptual categories that can be traced from the representations tier to the principles tier. They represent a relatively broad conceptual strand, subsumed under a more general domain, and circumscribed from the inside by the closeness of inferential relations among thier conceptual elements. Themes often represent the same overarching concepts that experts employ to define a given domain, but this is not always the case. In physics, we identified the energy concept as a theme. In our leadership research, cognition constituted a theme.
Sub-themes	The sub-themes of a domain are conceptual categories that span several developmental levels and are subsumed under themes. We particularly find it useful to identify and work with sub-themes in exceptionally rich and complex domains, in which it can be difficult to "see the forest for the trees.". Because they are subsumed under themes, they represent more narrowly defined conceptual strands. In physics, we identified motion as a sub-theme of energy. In the leadership domain, problem-solving was a sub-theme of cognition.
Concepts	Concepts are the conceptual elements of propositions, assertions, or arguments embedded in texts. For example, in the following statement there are several concepts. "Science is not always right because scientist are people and people make mistakes." These are: science can be wrong, people make mistakes, and scientists are people. These concepts are embedded in more general themes or sub-themes such as doubt concerning scientific truths, and limitations of human ability to know.

Themes, sub-themes and conceptual categories each provide information that can inform our understanding of the development of reasoning in a given domain. Orienting to the themes and sub-themes present in the data provides a general picture of the domain—its major issues, problems, skills, and topics. Orienting to the concepts provides a more fine-grained picture of the domain, offering insight into the variations and trends in respondents' constructions. Either or both levels of analysis can be adopted depending upon your research questions and objectives. We caution, however, that good developmental analyses should ultimately focus on conceptual categories. It is at this level that the sometimes subtle differences in meaning at adjacent developmental levels can best be captured.

Most of our work is phenomenological, in the sense that we most often create our categories from what our respondents tell us is relevant to a domain. But our method can also be employed using conceptual categories that are partially or wholly generated by experts or guided by theory. It is important to makre sure that borrowed categories embody fine enough distinctions in meaning to allow the detection of subtle developmental differences.

The process of concept coding should yield a clear sense of the ideas that comprise the content of a domain. Moreover, by conducting a concept analysis, which involves exploring the results of concept coding in conjunction with a complexity analysis, we can learn a great deal about conceptual development within a domain.

The first step in concept coding is identifying the broad themes that organize concepts. This process varies depending upon the domain under investigation. If the domain has never been studied before then themes must be generated through an iterative bootstrapping process. If the domain has been investigated previously it is often possible to generate preliminary themes based on previous research findings. 

In a new domain, we identify preliminary themes by having two or more researchers identify the most prominent conceptual categories in a set of texts that are to be coded. Ideally, these conceptual categories should be observed in performaces from representations through single principles, but it is often difficult to trace themes through the levels at this stage of the coding process. In fact we rarely accurately identify the major themes in a set of texts during this phase of the coding process. Despite the fact that they are likely to be changed, the themes identified in this way nevertheless serve to organize the early phases of the coding process. The software we use for concept coding, LectiPro™, allows us to add themes and to change existing themes with relative ease.

The figure below shows the theme window in a LectiPro™ database. The themes shown in this window are some of those we idenitified in the early phases of scoring a set of texts about learning. Thus, learning was designated as the domain. Most of the themes listed here were later classified as sub-themes of a smaller subset of themes.

During the initial analysis of a set of texts, analysts also generate a list of the concepts identified in a subset of texts. Generally, we only code responses that are relevant to the domain under investigation and represent judgments, justifications, or assertions. For example, in the learning domain concepts referred to elements of good and bad learning experiences, judgments about what learning ought to be like, and justifications for advocating particular learning experiences or types of learning. In our leadership interviews, concepts were often leader traits or behaviors and justifications for advocating particular traits or behaviors.

Each proposition with an irreducible meaning should be coded. A series of propositions, strung together without a claim that they are part of a single, larger proposition are treated as separate, whereas a proposition that represents a non-additive synthesis of less complex elements, such that it has a meaning distinct from its elements, is treated as a proposition in its own right. The elements of this larger proposition, if explicitly listed by the participant, are also coded as separate propositions. Including all elements in this way ensures that no particular level of conceptualization is privileged in the content analysis.

The list of concepts identified by analysts are employed as initial coding categories. These are entered into the concepts window of LectiPro™ as shown below. Note that each concept code is assigned to a theme and a domain. In this case, the themes include sub-themes. These follow the dash after the theme name. This particular list of themes and concepts was informed by the criterion judgments in Colby and Kohlberg's Standard Issue Scoring System.

The coding process involves both (1) assigning concepts in texts to themes and conceptual categories, and (2) adding new themes and conceptual categories as they appear in texts. Coding is conducted in the concept coding window of LectiPro™, shown below.

Criteria used here are:

1. A proposition should not be assigned an existing code if doing so would require ignoring some aspect of its meaning.

2. A proposition should not be assigned an existing code if it lacks any aspect of the meaning present among the other propositions assigned to that code.

 

 

Once coding is complete, we export the results to a tab delineated file containing themes, complexity levels, case numbers, and concepts, respectively. This document is separated into individual documents by theme and the theme names are removed. We then employ the software, Sequencer, to create the first in a series of data files like the one shown below:

Code	RM % (n=11)	RS % (n=22)	SA % (n=73)	AM % (n=33)
States that a ball will not bounce back up to where it was dropped from	9.09	4.55	17.81	27.27
States that a ball bounces because it is made of a bouncy substance	9.09	45.45	17.81	12.12
States that a ball bounces because it hits the floor/ground hard	18.18	18.18	5.48
Energy is something people have	27.27	4.55
States that a ball bounces because it can only go up (after it hits the floor)	45.45	9.09	2.74
States that a ball bounces because it is bouncy/squishy	45.45	31.82	5.48
States that the energy of a ball is decreased by a bounce		9.09	43.84	33.33
Energy is associated with speed		13.64	54.79	36.36
Energy is associated with pushing		18.18	39.73	45.45
Energy is associated with motion/movement		18.18	42.47	60.61
States that energy is what makes a ball bounce		22.73	23.29	6.06
States that gravity pulls/holds/pushes on a falling ball		40.91	46.58	69.70
Defines kinetic energy			8.22	30.30
States that friction will slow down a ball			8.22	30.30
Defines potential energy			8.22	33.33
States that energy can be stored			9.59	21.21
States that the energy of a rolling ball is decreased by friction			9.59	42.42
States that the energy of a bouncing or rolling ball is decreased by gravity			12.33	12.12
States that energy is released/let go of by a bouncing ball			12.33	15.15
States that the energy of a ball will increase when it bounces			12.33
Mentions kinetic energy			16.44	57.58
States that the energy of a ball decreases during its fall			17.81	6.06
Mentions potential energy			19.18	63.64
General statement that gravity is related to energy			20.55	18.18
States that energy can be transferred (from one object to another)			21.92	42.42
States that the energy of a falling ball increases because of gravity			23.29	39.39
States that energy is absent if there is no movement			30.14	9.09
States that energy can be removed/lost			30.14	15.15
States that the energy of a falling ball increases as it speeds up			30.14	24.24
States that energy is always present			31.51	45.45
Describes energy in terms of force			34.25	21.21
States that the energy of a ball increases during its fall			47.95	36.36
States that energy can't be lost				12.12
States that friction creates heat/thermal energy				12.12
General statement that friction is related to energy				15.15
Provides formula for accelleration				18.18
States that when a ball is dropped, its potential energy changes into kinetic energy				21.21
States that some of a ball's energy is changed to potential energy as it hits the floor.				21.21
States that some of a bouncing ball's energy is changed to heat/thermal energy				24.24

The first tables to emerge from Sequencer are employed to assist the process of refining categories. This is always necessary, because in striving to preserve fine distinctions in meaning by creating concept codes for every concept that is in any way distinct, we inevitably create more concept categories than are actually required to capture important differences in meaning. The Sequencer tables reveal the distribution of concepts across and within developmental levels, allowing us to preserve differences that are due to developmental change as we collapse coding categories. This is a slow and labor intensive process.

The goal of the concept analysis is to describe evaluative reasoning about education at each order of complexity. To accomplish this end, the propositions identified in the content analysis are reintegrated to develop descriptions of reasoning at each complexity order, employing the organizational principles of the complexity orders at which they are found in performances. In a sense, this is the inverse of the process employed in complexity order scoring.

Before the analysis can be conducted, coding categories must be refined. Taking care to preserve subtle nuances of meaning, propositions are collapsed into increasingly general categories. For example, on the play strand at the representational mappings order in our study of evaluative reasoning about education, different children claimed that a good thing about school is that children get to:

1. play outside

2. have recess

3. play with their friends, or

4. play a specific game, such as hide-and-seek or snakes & ladders.

In the original coding scheme, these were given separate proposition codes. After establishing that all of these assertions were made at the representational mappings order, they were collapsed into the category: “At a good school, you get to play.” On the other hand, assertions that at a good school you should

1. partly play and partly work, or

2. do some fun things

were not observed until the representational systems order. Consequently, they were collapsed into a separate category: “At a good school you should work and have fun/play.”

Many propositional categories will be assigned to performances from multiple complexity orders. For example, in our study of evaluative reasoning about education, the propositional category, “At a good school you get to play outside,” appeared at the representational mappings and representational systems orders, while the propositional category, “In a good school you have fun,” appeared at every complexity order represented in the sample. The table above shows some of the final categories in our study of the development of the energy concept.

After making a decision to collapse two categories we return to LectiPro™ and recode the statements assigned to the original coding categories to the new code. LectiPro™ has a find function that makes it easy to locate and recode statements.

We usually conduct a couple of rounds of category refinement before we are satisfied that we have identified optimal final concept categories. Once we have recoded the data into the final categories, we output a final set of Sequencer tables and move on to the concept analysis.