Dependent variables. The first set of hypotheses focuses on active participation in a public process. I break participation into two parts—a willingness to participate among those invited and the level of active engagement among those who participate. The first of these concerns the rate at which a process attracts those who are invited to participate. Response rate of a process varies widely based on variables external to this model, such as providing payment for participation (Crosby & Nethercutt 2005) and intensive recruiting efforts (Fishkin 2009). Some online processes have a cap on the number of persons who can attend (e.g., if it involves a finite resource, such as trained facilitators), so this first variable is measured as the percentage of valid contacts (i.e., subtracting bounced email addresses) yielding an affirmative willingness to participate.2

A second participation measure recognizes that among those who take part in a public event, there are widely varying rates of active engagement. Intensively facilitated deliberative events, such as the Australian Citizens’ Parliament, can increase the equality of participants’ engagement over the course of several days—but come nowhere near equalizing engagement levels (Bonito et al. 2014). Because one can remain cognitively active during a process without speaking as often as others, it is important to augment external measures of verbal participation rates with a psychometric scale measuring ‘deliberation within’ (Weinmann 2018).

Predictor variables. People choose to participate in politics and public affairs for myriad reasons, including time, money, and civic capabilities (Brady, Verba & Schlozman 1995). That said, a public engagement program can attract participants if it constitutes an empowered opportunity. What makes such opportunities most compelling is the chance for real influence, such as through participatory budgeting (Gilman 2016; Wampler 2007), civil and criminal juries (Gastil et al. 2010), or local councils (Barrett, Wyman & Coelho 2012; Coelho, Pozzoni & Montoya 2005). More modestly, Neblo (2015) found that many citizens, particularly millennials, want a more compelling online experience of public engagement, even if only to meet online with a public official. By contrast, forums that provide little chance for influence have equally miniscule odds of sparking interest among disengaged citizens (Dunne 2010).

How does one measure the degree to which an engagement opportunity represents one that empowers citizen participants? A rank-ordered empowerment scale could have at its top those processes that have binding authority over a public good, whether this is budget resources, policy decisions, legislative proposal, or valuable message content, as in the advisory statements the Citizens’ Initiative Reviews write for mass distribution to the electorate (Johnson & Gastil 2015). A secondary level of empowerment would be a process that offers participants a reasonable expectation of influence on such matters, as in Deliberative Polling (Fishkin 2009) and Citizens’ Juries (Crosby & Nethercutt 2005; Smith & Wales 2000). The third level would simply involve meeting with public officials or their agents, as in conventional public hearings and board meetings—but without an expectation of influence, as is common for gatherings that permit only limited discussion and make no pretense of participant influence or equality (Fiorino 1990; Gastil 2000). Finally, the lowest level of empowerment—a score of zero on this scale—convenes citizens without granting any direct access or connection to public representatives, as is common in online forums in which citizens are little more than indirect ‘information providers’ for a government agency (Dunne 2010).

Dependent variables. A surplus of definitions exist for defining deliberation, ranging from precise coding metrics for argument quality (Steenbergen et al. 2003) to subjective observer assessments, such as the ‘free flow’ of ideas (Mansbridge et al. 2006). I do not attempt to resolve conceptual and measurement disputes about deliberation but opt instead to stress what I consider the two primary dimensions—the analytic rigor of a discussion and the democratic quality of the social exchanges that occur during such discussions (Gastil 2008).

In a review of case studies compiled in the Participedia.net archive, these two dimensions were subdivided into the breadth and depth of substantive issue analysis, on the one hand, and the consideration given to others’ ideas and the respect given to other individuals during deliberation on the other hand (Gastil et al. 2017). Neutral third-party coders made those assessments based on the materials available in the case write-ups, and third-party ratings also have proven reliable using transcripts for face-to-face discussions (Gastil, Black & Moscovitz, 2008) and more complex threaded discussions in online work groups (Black et al. 2011).

Predictor variables. I begin with a predictor—level of active participant engagement—that I defined in the previous section. Because this model follows a loop, I will flip dependent variables into predictors in each new section, because their role changes as hypotheses move forward through the chain of cause and effect.

What requires justification in this and all cases, however, is predicting that this variable has a causal effect on a new dependent variable. In this case, active engagement should prove vital to deliberative quality because the deliberative process depends on both cognitive and verbal contributions from participants. Across a wide range of deliberative processes, design elements aim to elicit active participation precisely for this purpose (Gastil & Levine 2005). More than once, however, researchers of hybrid processes have found greater inequality of participation online versus subsequent face-to-face processes (Showers, Tindall & Davies 2015; Sullivan & Hartz-Karp 2013). Widespread participant disengagement decreases the likelihood of hearing a full breadth of perspectives, while also limiting the argumentative depth of discussion. Meanwhile, disengaged participants may be less likely to feel that their perspectives have been heard, or respected—a problem that manifests itself in majoritarian bodies that foreclose opportunities for minority opinion, or even dissent (Gastil 2014; Geenans 2007).

The remaining predictors represent the only set of variables that stands apart from the main flow of cause and effect in the model. I hypothesize that these structural features of an online public consultation system give it the capacity to generate high-quality deliberation, but no other model variables have a reciprocal effect on them. Moreover, this structural set is more illustrative rather than exhaustive. I include in the model only three variables (task clarity, facilitation, and peer assessment), but ongoing testing will clarify which of these—or other—structural features best promote deliberation.

Two structural features—task clarity and facilitation—come as a pair common to deliberative process regardless of whether conducted online or face-to-face. Many of the most longstanding deliberative consultation processes have stressed the importance of a clear ‘charge’ for a body, which gives citizens both a topical focus and a well-defined purpose, and Citizens’ Juries, Planning Cells, and Consensus Conferences all place a premium on professional group facilitation (Crosby & Nethercutt 2005; Hendriks 2005). The Oregon Citizens’ Initiative Review illustrates this most clearly because it asks two dozen citizens to spend their 3 to 5 days together scrutinizing a specific ballot measure. A team of facilitators help the panelists work through complex issues so that the citizens can write a one-page issue analysis for the full electorate to read (Knobloch et al. 2013).

As for measuring these factors, a subjective scale could measure task clarity both before and after deliberation to ascertain whether a complex deliberation functions well enough so long as participants get up to speed during the process itself. Published work on deliberative facilitation suggests using a rank-order approach—strong, weak, or no facilitation (Dillard 2013), although more subtle measures of facilitation style should be used as available.

A predictor that could be crucial in an ongoing consultation system asks participants to assess one another after each deliberation. I call this peer assessment linked to incentives to capture a wide range of potential social-rating methods, so long as those give participants a reason to both desire favorable ratings and give accurate ratings. Online peer rating systems, such as those used in eBay and Uber, have proven themselves as effective accountability mechanisms that build confidence and trust between buyers and sellers (Lee 2015; Thierer et al. 2015). Dystopian fiction, however, cautions us against how such systems could go awry, as in the Black Mirror episode ‘Nosedive,’ in which social rating cascades lead to social isolation. In China’s authoritarian system, this scenario is well underway, with the government licensing ‘eight private companies to come up with systems and algorithms for social credit scores’ (Bostman 2017).

The promise and hazards of such systems suggest experimenting with a variety of social designs. For the purpose of predicting deliberative quality, such a system should ask participants to rate one another on precisely the same dimensions used to assess deliberation itself—breadth, depth, consideration, and respect. Non-obvious questions might touch on underappreciated social roles. For example, a rating question might ask which participants helped spark the most substantive disagreement—a necessary part of deliberation that can make participants uncomfortable (Esterling, Fung & Lee 2015; Zhang & Chang 2014). Or, which participants actively encouraged quieter members to speak up? To encourage more accurate ratings, random third-party spot checks could be compared against participants’ subjective assessments to amplify high-fidelity ratings and discount ones that appear punitive or disingenuous. Beyond these suggestions, possibilities for deliberative peer assessment abound (Black et al. 2011; Chang, Jacobson & Zhang 2013; Steenbergen et al. 2003).

As for transforming those ratings into something that has value within the system, many approaches could prove feasible. A reputation system by itself might be enough, given participants’ desire to receive favorable marks from their peers, but a ‘gamified’ approach could provide more powerful incentives (Chou 2015; Lerner 2014). An online system could offer within-system opportunities for leveling-up toward greater levels of opportunity (and responsibility) within the system (Gastil 2016). Or, there could be external incentives—even monetary ones—for making valuable contributions to deliberation, analogous to the rewards given in ‘forecasting tournaments’ (Tetlock et al. 2014). Whichever form such incentives take, the point is to tie them as directly as possible to peer assessments of contributions to the deliberative process—as opposed to other socially desirable behaviors.

Dependent variables. Aside from the indirect benefit of legitimacy, a key rationale for involving lay citizens in governance is improving public policies themselves (Dzur 2008). Thus, at the center of my model lies the quality of the recommendations, decisions, and other substantive outputs that come from a citizen deliberation (hereafter called simply ‘decisions’). Measuring the quality of decisions, however, presents problems both practical and philosophical. Critics have pointed out that deliberation cannot justify itself in terms of the decisions it yields because there exists no independent ground from which one might judge the public value of any decision (Ingham 2013).

Even so, there are two approximations that should suffice. First, one can measure the level of participant support for decisions/recommendations (e.g., Paul, Haseman & Ramamurthy 2004), because participants in a deliberative process can end up varying in their assessment of their own decisions. Evaluative disagreement occurs even on juries that use a unanimous decision rule (Gastil et al. 2010), and there is reason to be concerned about the quality of decisions groups render that even their members are reluctant to support.

That said, a deliberative body may fall victim to self-congratulatory assessment of its decisions, so a third-party judgment is warranted. After removing the subjective question of whether the group’s decision is desirable, in terms of a public good, criteria suitable for expert judgment are judgments about a decision’s feasibility; the sufficiency of its rationale; and its recognition of drawbacks, tradeoffs, and dissenting viewpoints. The research team studying the Oregon Citizens’ Initiative Review has made similar assessments of the one-page statements written by citizen panelists (Gastil, Knobloch & Richards 2015), but no inter-rater reliability data exist for such a measure.

Predictor variables. Epistemic theories of democracy presume a direct link from deliberative quality to decision quality, marshalling a good deal of logic and evidence supporting that claim (Estlund 2009; Landemore 2013). Thus, I draw a direct connection to decision quality from the foreshortened conception of deliberation I provided earlier—breadth/depth plus consideration/respect.

As to the first part of this hypothesis (breadth/depth), evidence from small group research suggests that more extensive and in-depth information processing yields higher-quality decisions (e.g., Peterson et al. 1998). As to the social dimension (consideration/respect), notice that consideration amounts to weighing conflicting views (i.e., substantive conflict), whereas respect is the absence of personal conflicts. Decision-making groups thrive precisely when they have the necessary substantive clash without the interpersonal drama (Gastil 1993). This is particularly important in an online context, where participants can be less likely to engage in ‘positive conflict management’ (e.g., respectful and substantive analysis of divergent views) than in face-to-face settings (Zornoza, Ripoll & Peiro 2002).

Dependent variables. A government agency or public official may be reluctant to solicit public input for fear that citizens will want something infeasible or undesirable. Such an eventuality puts the policymaker in the awkward position of rejecting a recommendation from constituents convened precisely to exert influence over a government decision. Government responsiveness, however, is not the same as mere acceptance. For example, a genuinely responsive public agency should be able to give a direct and thoughtful explanation for why it cannot act on a recommendation. In a sense, such a reply might count as more responsive than an agency that simply accepts a public statement, without comment, and acts on it.

Measurement of responsiveness has further complications. How does one measure the ‘response’ of a government when the public’s decisions have a legal force all their own—independent of government action? Does it matter how quickly a government responds, or does a government get credit for taking more time to give a precise reply? Do downstream policy implications matter, such as when a promise to act on advice goes unfulfilled, or when hasty action in response to a recommendation yielded a policy outcome worse than the status quo?

Conventional measures of responsiveness get around these problems by measuring whether the people get what they say they want. In one nondeliberative approach, a government scores as responsive if it produces public policies that—when coded by experts—align with survey measures of surface-level public opinion (Erikson, Wright & McIver 1993). That method also has been used to judge whether direct democracy (i.e., ballot initiatives) produce laws more responsive to the public’s values than do electoral democracy (Matsusaka 2004). A similar approach compares public preferences with a government’s policy promises and its budgetary decisions (Hobolt & Klemmensen 2008).

Translating those approaches to the present context, the public’s more deliberative judgments can stand in for any indirect measure of its preferences. That is, whatever the public decides to recommend represents its considered opinion on a matter (Fishkin 2009, 2018; Mathews 1999; Yankelovich 1991). Expert coders can then judge the alignment of the text and policy responses from a government with the public’s decision. Stopping there, however, would deem any disagreement as unresponsive, so a second coding should judge the response on two more dimensions—directness and rationale. A direct response addresses the substance of the issue in question and the public’s decision (vs. a more strategically ambiguous or oblique response). A strong rationale is one that receives a high argument score on a Discourse Quality Index (Steenbergen et al. 2003) for providing reasoning, evidence, and a clear expression of underlying values. To capture the public’s subjective sense of responsiveness, participant surveys can tap into the same two dimensions to discern whether they thought the response constituted a direct and substantive response.

Two special—but common—cases, however, require supplementary measures of responsiveness. First, when a government’s response comes later than promised, or expected, a question of timeliness arises. If citizens recommend establishing a review board for a police department, for example, how does one assess a government response if one is not forthcoming even months after the conclusion of a public consultation? In such a case, the researcher has missing data: There are no government documents to code (other than a budgetary allocation of zero), and surveys have no focal government action for participants to assess. In such a case, one may be forced to enter a score equivalent to a zero to indicate total nonresponsiveness.

The second special case is when the public consultation is empowered fully—with a final decision pulling a policy or budgetary trigger. When this is the case, expert judgments of alignment, directness, and rationale all appear unnecessary, and a survey assessment would have to carefully phrase its questions lest they instead measure broader public attitudes toward government (such as legitimacy, discussed below). One reasonable option would simply score these cases as maximally responsive.

Predictor variables. One might take for granted that better decisions yield a favorable government response. After all, under good conditions this stands as the very purpose for consulting the public (Dzur 2008). Such a direct relationship between decision quality and responsiveness did not appear, however, in a statistical analysis of a large sample of public participation cases (Gastil et al. 2017). Even so, there exist numerous examples of favorable government responses to clear guidance (Barrett et al. 2012; Johnson & Gastil 2015). Deliberative Polling, for example, has elicited rapid and direct responses in contexts as far apart as the Texas public utilities and Chinese municipal governments (Fishkin 2009, 2018; Leib & He 2006).

Moderating this relationship are a set of systemic variables that characterize a government body’s default responsiveness to democratic input. At one extreme, He and Warren (2012) discuss the prospects of deliberation in China, where an authoritarian government welcomes public input on particular issues and with strict limits on the range of alternatives considered. One could not expect a Chinese municipal government to respond favorably to any proposal that contradicted central party dictates, but this counts as an extreme example along a continuum of governments that may be more or less responsive.

In the context of a multiparty political system, a key to responsiveness may be the government’s perception that its political fortunes require such action owing to the presence of an independent media system, a vibrant civic culture, and an engaged public (Besley & Burgess 2002; Cleary 2007). Governments also vary in their capacity to respond, depending on the distribution of authority across different levels of government and the restrictions placed on government by ordinances, charters, constitutions, and other laws (Rodrik & Zeckhauser 1988). Finally, the civic attitudes and habitual practices of public officials will predict responsiveness, because only some public officials view public input as vital to their jobs (Dekker & Bekkers 2015; Dzur 2008).

Dependent variables. As much disagreement as there is about the precise meaning of deliberation, legitimacy has a longer history of conflicting conceptions and measures. One approach that fits with the conception implicit in the discussion thus far distinguishes behavioral from psychological variables as they relate to democratic political systems (Levi, Sacks & Tyler 2009). In this view, such a system has legitimacy if people comply with its laws and decisions. Underlying that willingness to comply are perceptions of procedural justice and trust in the government itself. If one’s leaders are trustworthy (and follow fair procedures), one can rest assured that the public institutions that put those leaders in place is legitimate.

Those psychological variables translate well to the context of an online deliberative consultation. Surveys could ask participants three questions about the government body that sponsored, hosted, or authorized the consultation. Is that body’s interest in public engagement genuine? Did it strive to ensure procedural fairness? Finally, does it aim to render final decisions in the public interest?

Asking those straightforward questions would constitute a departure from the more commonly used measures of legitimacy that ask about government more broadly. The most common tool is a four-item survey measure of ‘perceived system responsiveness,’ or ‘external efficacy’ distinguished from the internal variety (Niemi, Craig & Mattei 1991). Previous research has found that even a rewarding and empowered experience of deliberation can fail to move the needle on this abstract variable (Gastil et al. 2010; Knobloch & Gastil 2012). Part of the problem is that exceedingly high scores on this variable can reflect a kind of complacency, which clashes with the civic spark engendered by public engagement (Gastil & Xenos 2010). Although one might choose to include this variable to probe the long-term effects of ongoing consultation, I eschew this option in the name of parsimony.

In the model shown in Figure 1, I predicted that this legitimacy boost could come not only to government bodies but also to the civic organizations that organize deliberative events with them. Government sometimes contracts with such civic entities, or they cocreate processes (Kelshaw & Gastil 2008; Lee 2014; Nabatchi et al. 2012). Although changes in government legitimacy remain my principal focus, it is plausible that the public perceptions of civic partners in these efforts will move in parallel.

Predictor variables. I include in the model as predictors only the subjective measures of government responsiveness. Although governments might hope to receive direct credit for their responses, as viewed by neutral third parties, the public’s disaffection stems from a felt disconnection from government (Dzur 2008). The key predictor, then, is the subjective one, whereby the sense of receiving a direct response with a strong rationale reinforce citizens’ trust in the officials, public institutions, and civic organizations that welcomed their input.

This limited hypothesis both supports and challenges assumptions implicit in deliberative democratic theory. Many theorists hold that the legitimating function of deliberation comes from the open contestation of arguments and the sense of having one’s views heard (e.g., Gutmann & Thompson 2004). In this view, it should not matter whether a government’s decision aligns with one’s own preferences or with the substance of the recommendations given from an online deliberation. Rather, if the response is ‘genuine,’ it is legitimate. I take the same view by predicting no independent effect on legitimacy beliefs from the alignment of a government response with the public’s recommendations. At the same time, I depart from this most sanguine conception of deliberation in expecting participants’ subjective assessments of directness and rationale as paramount. I predict no independent effect from expert codings thereof.

This discussion of legitimacy might seem incomplete without including as predictors the quality of public deliberation and the decisions/recommendations that citizens reach. The model I summarized in Figure 1 includes these variables but posits that they only have indirect effect on legitimacy perceptions. Rather, deliberative quality influences decision quality, which leads to government responsiveness and consequent changes in legitimacy perceptions. A study fortunate enough to have all those variables measured in sequence can use structural equation modeling to test whether deliberation and decision quality also have direct effects on legitimacy ratings, aside from their effects through the main paths of the model.

At this point, I have specified every variable in the model, but two causal paths require justification. The first of these closes the main loop in the model, whereby creating an opportunity for empowered deliberation leads to more such opportunities in the future. One purpose of drawing out this model was giving a more precise formulation of this notion that public deliberation is a self-reinforcing process (Burkhalter et al. 2002; Richards & Gastil 2015).

The last link in this chain holds that heightened legitimacy perceptions are the proximate cause of government institutions and civic organizations providing more deliberative opportunities in the future. This claim has face validity in that governments turn to deliberative processes partly to bolster their public credibility (Fishkin 2009, 2018). Likewise, civic organizations that partner with governments rely on public trust and can be expected to reinitiate those processes that win them more credibility. Success, in the form of rising legitimacy ratings for the consulting agency, should therefore prompt repeated iterations of this curative process.

There are at least two reasons, however, to doubt the simplicity of that formulation. First, the model I propose dismisses the possibility of other direct links. One that stands out is decision quality. In those instances where government convened the public not to score legitimacy points but to render better public policy (Dzur 2008), decision quality might prove the more powerful direct predictor of whether that same government body convenes empowered deliberative processes in the future. Second, deliberation could be its own undoing if it proves so effective at eliciting public values and policy justifications that it leaves a consulting agency with no further uncertainties about the public’s views. In such a case, if the agency sought the input for its own sake, one or two empowered deliberative processes might suffice.