Audit Coding Reliability and Consensus Before ENA
Use the shared synthetic design-talk dataset to practise testing whether the codebook is applied consistently enough for connections among codes to have a stable interpretation, document the decision, test a defensible alternative, and keep every interpretation tied to source evidence.
Проверенный текст урока пока доступен на английском языке.
Источники метода
Полный урок
Audit Coding Reliability and Consensus Before ENA addresses testing whether the codebook is applied consistently enough for connections among codes to have a stable interpretation. This decision belongs in the evidence model, not in a late formatting pass. It affects which coded observations can contribute to a network, what the plotted coordinates represent, and how confidently a comparison can be interpreted. The lesson extends the same eight-team synthetic design-talk case used throughout the Academy so that only the target decision changes while the broader learning context remains stable.
You will work from the source CSV, codebook, and blank decision record. The synthetic file is intentionally small and balanced; it supports transparent inspection but no real-world inference. Your goal is to leave an audit trail that another analyst can rerun: the primary choice, its theoretical rationale, one prespecified sensitivity check, the resulting network evidence, and the claim boundary.
Учебный пример
Audit Coding Reliability and Consensus Before ENA in the design-talk case
Eight fictional teacher-design teams discuss goals, student evidence, strategies, tradeoffs, and revisions. Four are labeled baseline and four scaffolded, with six ordered utterances per team. The recurring question is whether the two conditions organize these ideas differently. For this lesson, the analytic focus is testing whether the codebook is applied consistently enough for connections among codes to have a stable interpretation; the tiny balanced file makes every contributing row inspectable while preventing invented population claims.
Учебные данные
Шаг 1
Restate the relational estimand
Write one sentence naming the unit network, the comparison, and the contextual relation represented by a connection. Then explain how testing whether the codebook is applied consistently enough for connections among codes to have a stable interpretation can change that estimand. Keep the distinction between raw observation, coded evidence, accumulated connection, normalized vector, projected coordinate, and substantive claim visible. If the target choice cannot be linked to the question, pause before opening a model.
Контрольная точкаThe decision record names one unit, one bounded context, one comparison, and the exact relation a weighted edge represents.Шаг 2
Freeze the shared baseline specification
Record the dataset filename and hash; team_id as unit; conversation_id as conversation; line_number as order; condition as metadata; the five binary code columns; the current window, weighting, normalization, and rotation; exclusions; and software version. This frozen baseline prevents an apparent change from being attributed to testing whether the codebook is applied consistently enough for connections among codes to have a stable interpretation when another setting changed at the same time.
Контрольная точкаA second analyst can recreate the baseline model without relying on interface memory or undocumented defaults.Шаг 3
Implement the primary decision
Apply this prespecified procedure: select a representative double-coded subset, calculate agreement at the same segment and code level used by the model, inspect disagreements, revise definitions, and record adjudication without hiding the pre-consensus results. Save the configuration before inspecting the preferred group pattern. Export unit coordinates, accumulated and normalized edge tables, group summaries, and diagnostics rather than keeping only a screenshot. Mark any manual judgment with a date and rationale so later review can distinguish planned configuration from reactive adjustment.
Контрольная точкаThe primary run has a dated configuration, machine-readable outputs, and a rationale written before substantive interpretation.Шаг 4
Inspect units before group averages
Review all eight unit networks, their coded-row exposure, connection magnitudes, and positions. Identify all-zero, sparse, or influential cases and verify that every unit belongs to the intended condition. A mean network can look coherent even when only one team supplies an edge, so record unit coverage for the strongest visible relations and inspect whether within-group heterogeneity contradicts a simple group story.
Контрольная точкаEvery interpreted group edge has a unit-coverage count, and no case was silently dropped because its pattern was inconvenient.Шаг 5
Run the planned sensitivity check
Without changing unrelated settings, compare unit networks built from coder A, coder B, and the adjudicated matrix for the double-coded subset, paying special attention to edges involving rare or frequently confused codes. Save the primary and alternative outputs side by side and compare connection ranks, unit coordinates, group means, diagnostics, and the wording of the conclusion. Sensitivity analysis is not a search for the most separated plot; report stable, weakened, reversed, and ambiguous patterns with equal visibility.
Контрольная точкаThe record states exactly what changed, what stayed fixed, and whether the main interpretation survives the alternative.Шаг 6
Return from edges to evidence
Close the interpretive loop: link every resolved disagreement to the original utterance, both coder judgments, the rule invoked, and the final rationale. Read enough surrounding rows to recover the conversational context and examine counterexamples, not only quotations that fit the network. Preserve team, conversation, and row identifiers in the audit output. If the edge interpretation conflicts with the excerpts, revise the interpretation or investigate the coding and window rather than trusting the graph.
Контрольная точкаThe interpretation cites traceable rows from multiple units and records at least one contradictory or boundary case.Шаг 7
Write the decision and claim boundary
Write three layers in order: the observed source evidence, the modeled pattern under the primary specification, and the explanation the synthetic design cannot establish. Include the sensitivity outcome and this boundary: Agreement coefficients do not prove construct validity, and consensus does not erase ambiguity. A high value can coexist with a narrow codebook, prevalent negative cases, or shared coder bias. End with the exact files needed to reproduce the lesson and a note that the exercise demonstrates workflow logic rather than a finding about real teachers or scaffolds.
Контрольная точкаThe final paragraph separates observation, model output, interpretation, uncertainty, and non-causal limitation in explicit language.