What can an AI readiness score actually establish?

Construct validity, aggregation and the difference between reported and demonstrated capability

This retrospective review considers source versions available by 11 May 2026. The cutoff defines the evidence included; preparation and source checking took place on 21 October 2026.

Keywords: AI readiness; construct validity; organisational capability; psychometrics; implementation

1. Readiness as a measurement problem

What can an organisational AI readiness score establish about the likelihood of effective implementation? A score can summarise responses to a questionnaire, describe the presence of selected resources or locate an organisation relative to a comparison group. Predicting implementation outcomes is a further claim. It requires evidence that the score measured before implementation is related to a subsequently observed outcome, under conditions relevant to the decision. A plausible set of questions does not establish that predictive relationship on its own.

The object being assessed also needs definition. An organisation may be prepared to introduce a drafting assistant while lacking the controls needed for an autonomous customer-facing service. A marketing team can possess substantial subject expertise while depending on another function for access permissions and integration. “Ready for AI” leaves both the change and the level of analysis unspecified. Readiness for a named task, process and deployment arrangement is a more interpretable proposition.

This review examines how organisational-change theory, AI capability research and production-engineering evidence bear on readiness assessment. Its concern is the validity of the measurement and the claims made from it. It does not introduce a new readiness index, assign weights to assessment dimensions or certify an organisation. The question is whether the available literature supports interpreting a score as a description, a diagnosis or a prediction, and what additional evidence each interpretation requires.

2. Method and source selection

Targeted searches on 21 October 2026 covered organisational readiness for change, AI readiness factors, AI capability measurement, production readiness, data documentation and AI risk management. 20 sources were selected from primary research, author publication records and official frameworks. The review uses a narrative comparison because the sources differ in construct, population, design and purpose. No comprehensive database search, independent screening, common effect-size calculation or original survey was undertaken.

Three central open-access papers were examined in full: Weiner's conceptual treatment of readiness, Shea and colleagues' psychometric assessment, and Jöhnk and colleagues' interview study of AI readiness factors [1] [2] [3]. Other research was consulted through author or publication summaries, with the distinction recorded in the access register. The review separates proposed theory, qualitative factor discovery, instrument validation and engineering guidance. A useful result in one category is not treated as a substitute for evidence in another.

The historical range is deliberate. Organisational-change research supplies measurement distinctions that predate generative AI, while engineering sources identify properties of functioning systems. Transfer to marketing is discussed at the level of constructs and evaluation boundaries. No claim is made that a healthcare readiness instrument has already been validated for marketing teams or that an engineering rubric predicts the financial success of a transformation programme.

3. Readiness, resources and capability

Weiner defines organisational readiness around shared commitment to implement a particular change and shared belief in collective capability [1]. The article distinguishes that psychological state from the resources and circumstances that may influence it. This is a consequential construct boundary. An inventory of servers, staff or budgets describes resources; a set of responses about collective resolve describes a perception of the impending change. Combining them under one label can conceal which quantity has actually been observed.

Jöhnk, Weißert and Wyrtki's qualitative study uses interviews with 25 AI experts, literature triangulation and iterative analysis to identify 18 factors and 58 illustrative indicators across five categories [3]. Its contribution is discovery and organisation of relevant factors. The authors explicitly present a starting point for further validation. A factor's inclusion in this account does not estimate its numerical contribution to implementation success, nor does it provide a universal weight for a composite index.

Mikalef and Gupta conceptualise AI capability, develop a measurement instrument and examine relationships with organisational creativity and performance [4]. This concerns a related but distinct object: the organisation's capability to assemble and use relevant resources. Readiness for a specific change, capability in an ongoing domain and realised performance can be related without being interchangeable. The published abstract supports that distinction; detailed causal identification and measurement robustness were not independently assessed here.

A readiness assessment needs to state which object it is intended to capture. If the purpose is to describe resource gaps, direct evidence about those resources may be more informative than confidence ratings. If the purpose is to understand shared commitment, resource counts alone cannot answer it. If the purpose is to forecast outcomes, both kinds of information may be candidates, but their predictive relevance must be evaluated rather than assumed from face plausibility.

4. Levels of analysis and aggregation

4.1 The respondent and the organisation

A questionnaire is commonly answered by individuals, while its headline claim concerns a team or firm. Moving between those levels requires justification. One senior leader's response can report that leader's perspective, but does not directly observe every team's working conditions. Several respondents can provide broader information, yet their average can still conceal substantial disagreement. An organisational score should therefore identify who responded and which parts of the organisation their observations cover.

The readiness literature treats aggregation as an empirical question. Weiner emphasises within-group agreement for a shared team property, and Shea and colleagues examine reliability and agreement when aggregating individual responses [1] [2]. The issue is more specific than whether the mean is easy to calculate. A mean can be numerically precise while failing to represent a common state. If half a team is confident and half is not, the midpoint describes the arithmetic balance without establishing a shared moderate level of confidence.

The appropriate unit may be a department, project group or implementation team rather than the legal entity. A firm can contain several deployments with different dependencies and decision rights. Combining them can answer a portfolio question, but the result should not be reinterpreted as evidence that each deployment is equally ready. The unit of analysis follows the implementation boundary; organisational size alone does not determine it.

4.2 Coverage and missing observations

Response coverage affects interpretation even before aggregation. People closest to technical implementation may know about data access and integration constraints, while intended users know the tasks and current approval practices. Each can possess information the other lacks. An instrument asking one respondent to answer every domain can convert missing knowledge into apparent certainty. Allowing an explicit unknown category preserves that distinction, although it also reduces the amount of complete data available for a score.

Missing answers should not automatically be scored as failure or silently replaced by the midpoint. Those choices impose different assumptions about what nonresponse means. A missing answer can reflect an absent capability, lack of visibility or an unclear question. The assessment can report missingness and investigate its cause. A single completion percentage cannot determine which explanation applies, and a total score should not hide the resulting uncertainty.

Sampling also matters when comparing organisations. A score based on procurement leaders in one company and hands-on users in another may reflect respondent role as well as organisational differences. Matching the respondent population improves comparability, while collecting multiple perspectives can support diagnosis. These are different design goals. A comparison must explain how its respondents and coverage support the inference it asks readers to make.

5. Instrument validity and what reliability does not prove

Shea and colleagues assessed the Organizational Readiness for Implementing Change measure across four studies, including content assessment, laboratory work and field data. Their analyses examined related commitment and efficacy dimensions and the properties of aggregated responses [2]. This is a materially stronger measurement process than presenting a questionnaire with intuitively appealing labels. It also remains evidence about the instrument and settings studied, rather than automatic validation of any questionnaire using similar words.

Reliability concerns consistency under a specified measurement arrangement. Validity concerns the interpretation supported by the evidence. Questions can produce consistent responses while measuring enthusiasm, general confidence or access to resources instead of readiness for the named change. A reliable score can also fail to predict implementation outcomes. Treating high internal consistency as a complete validation therefore exceeds what that statistic can establish.

Changing an established instrument changes the evidential question. Replacing references to a defined organisational change with a general phrase such as “using AI” can broaden the construct. Combining or deleting items can alter the relationship between dimensions. Translating an item or using it with a different population can change interpretation. These changes may be reasonable, but the resulting measure needs its own assessment rather than inheriting all prior validation by name.

Content coverage is another boundary. An instrument can omit an important implementation dependency while performing well on the items it contains. Conversely, adding many questions can expand coverage while increasing burden and overlap. Instrument length is not a direct measure of validity. The useful account explains why the included domains correspond to the assessment purpose and which relevant domains remain outside it.

No original psychometric test is conducted in this review. The distinctions identify the evidence needed to interpret a readiness result. A vendor's assertion that an instrument is research-based may indicate that concepts were informed by literature. It does not specify whether the particular item set, scoring rule and use case were evaluated. Those are inspectable methodological questions, separate from the presentation of the score.

6. Composite scores, weights and thresholds

6.1 What weighting assumes

A weighted average permits strength in one domain to offset weakness in another. Whether that is appropriate depends on the task. More available infrastructure cannot necessarily compensate for the absence of permission to use the required data. A highly motivated team cannot substitute enthusiasm for a missing integration. These examples illustrate non-substitutable prerequisites; they do not identify an empirically optimal scoring model. The assessment needs to distinguish such prerequisites from dimensions that can sensibly be traded off.

Weights can have several interpretations. They may represent expert judgement about importance, a statistical relation with an outcome or a convenient reporting convention. These interpretations support different claims. A weight chosen to balance presentation does not estimate causal importance. A weight fitted to predict one outcome can change when the outcome or population changes. Publishing weights helps readers inspect the calculation, while leaving their evidential justification as a separate question.

A composite can also conceal the scale of its components. A resource item recorded as present or absent differs from a confidence rating on an ordinal scale. Converting both into percentages creates numerical comparability by convention; it does not demonstrate that a one-point change has equal meaning across them. The final arithmetic may be exact while the interpretation remains limited by those transformations.

6.2 What a threshold claims

A threshold that labels an organisation ready implies a decision rule. Its consequences depend on the errors it permits. A false positive may authorise a deployment that lacks a necessary capability. A false negative may delay a beneficial experiment. The relative consequences differ by use case. A private drafting assistant and an autonomous external action system need not share the same acceptable error balance.

A threshold selected from observed outcomes can be evaluated for the population used to choose it. Testing it on the same observations can overstate generalisation because the rule has already been adapted to them. A prospective or held-out evaluation addresses a different question: how well does the rule perform on cases not used to choose it? Without such evidence, a threshold can still be an explicit management convention, but should not be presented as a scientifically established readiness boundary.

Uncertainty remains even with a validated predictive model. A probability of success describes a distribution over defined cases, not a guarantee for one deployment. It also depends on the meaning of success and the follow-up period. Completing a pilot, sustaining adoption and improving a business outcome are distinct endpoints. A readiness score cannot be interpreted properly unless the endpoint against which it was validated is stated.

7. Production readiness requires operational evidence

Breck and colleagues' production-readiness rubric identifies testing and monitoring needs for machine-learning systems [8]. It concerns the ability to operate and observe a service, rather than staff confidence in a planned change. This suggests an important distinction between reported readiness and demonstrated capability. An interview can establish that a person says a rollback process exists. A documented exercise can establish what happened when the process was used under the test conditions.

Amershi and colleagues' software-engineering study describes challenges in data management, model reuse and AI-component dependencies [7]. An assessment confined to model access would leave these surrounding obligations unexamined. The service can depend on another team's source information, a supplier's release or a downstream approval step. Readiness for one component does not establish readiness of the assembled workflow. The dependencies define the relevant operating boundary.

Data documentation contributes another form of evidence. Datasheets for Datasets proposes recording collection, composition and intended use [10]. For a marketing application, the analogous questions include whether source material covers the required products and markets and whether its status is understood. A file count provides little information about those properties. The existence of a documented source inventory can support inspection; the material still needs evaluation for the actual task.

A readiness assessment can therefore contain several kinds of observation: self-reports, records, demonstrations and measured trial outcomes. They need not be forced into one undifferentiated evidence category. A demonstration is narrow but direct; a self-report can be broad but dependent on the respondent's visibility. Recording the type and scope of evidence makes the resulting diagnosis more interpretable than using the same affirmative tick for both.

8. Governance evidence and the limits of a framework

NIST's AI Risk Management Framework is voluntary guidance for considering trustworthiness across the AI lifecycle, and the Generative AI Profile provides a companion treatment for generative systems [5] [6]. Their role in readiness assessment is to structure questions and responsibilities. They do not provide a universal pass mark or certify that a particular deployment is acceptable. Referencing a framework and demonstrating the relevant controls remain different observations.

A policy can specify who should approve a task while leaving unresolved whether those people have the information and time to do so. A training record can establish attendance without measuring judgement on an unfamiliar case. A risk register can list a failure mode without showing that the mitigation works. These are distinctions between an intended process and its observed operation. They do not imply that policies, training or registers are unhelpful; they delimit the evidence each supplies.

The intended autonomy of the system affects what needs demonstration. A system that proposes text for review has a different action boundary from one that publishes, spends or modifies records. An assessment can describe the permitted actions and the controls around them, rather than treating every use of the same model as the same deployment. The relevant risks arise from the assembled arrangement of data, tools, permissions and review.

Governance also changes over time. A new source of data, a different user population or an additional tool can alter the service boundary without changing its product name. Readiness recorded before that change may remain informative about unchanged components, but cannot automatically cover the new configuration. A dated assessment needs a defined scope and a trigger for reconsideration. The current literature does not establish a universally sufficient reassessment interval.

9. Linking readiness to outcomes

A readiness measure can be useful descriptively even when its predictive validity is incomplete. It can identify uncertainty, disagreement or an observed missing dependency. A stronger claim that improving the score will improve outcomes requires causal evidence. An association between higher scores and successful deployments can reflect resources, selection into easier projects or prior experience as well as the measured factor. The direction of influence can also run from successful adoption to more favourable readiness reports.

Jöhnk and colleagues describe readiness as purpose-specific and evolving through adoption [3]. That makes the measurement date material. Asking an experienced team whether it was ready before implementation can produce a retrospective account shaped by the result. A prospective assessment records the judgement before the outcome is known. The two designs answer different questions, and should not be combined without preserving their timing.

Outcomes themselves need stable definitions. Adoption can mean access to a licence, occasional use or routine dependence on a service. Implementation can mean technical launch or sustained operation. Business value can mean lower resource use, expanded output or improved commercial results. A model predicting one of these outcomes does not thereby predict the others. The assessment purpose should identify which endpoint matters and the period over which it will be observed.

Brynjolfsson, Rock and Syverson's treatment of complementary investment shows why measured productivity can change over a technology's implementation period [11]. For readiness analysis, this makes the observation window a substantive choice. It does not establish that every low-performing project needs more time. A claim about delayed benefits requires evidence about the investment and resulting performance, rather than an assumption that all implementation costs eventually pay off.

The FinOps Framework distinguishes business-value measurement from recording technology usage [12]. A workflow becoming cheaper and a marketing intervention becoming more effective are separate outcomes. A readiness score can organise information about conditions under which work is attempted; measuring the subsequent work is still necessary to establish its consequences. Neither an activity count nor an aggregate score supplies that outcome evidence by itself.

10. Interpreting readiness in marketing practice

Marketing offers a particularly clear example of task-specific readiness. The same team may use AI for summarising internal notes, exploring creative alternatives and preparing product claims. These tasks differ in source requirements, error consequences and review arrangements. A general familiarity score cannot establish that each use is supported. The assessment can instead identify the task population and examine whether the necessary information, permissions and evaluation capability exist for it.

A practical trial also needs a defined comparator. If a new workflow is assessed only on whether users like it, the trial measures preference under the trial conditions. Comparing accepted output, time and error categories against the current process supports additional claims. HELM's multi-metric approach illustrates the general need to preserve dimensions rather than letting a single score represent every objective [9]. The appropriate application metrics still depend on the task.

The choice of trial cases affects the conclusion. Cases selected because they are easy or well documented can establish performance on those cases while leaving coverage gaps. A sample drawn from actual work requests offers a different basis for generalisation. Difficult and unusual cases may deserve separate examination because they can be rare yet consequential. Reporting how cases were chosen permits readers to understand which population the result describes.

Readiness can also differ across the workflow. A team may generate useful drafts but lack a reliable process for checking source claims. Another may have strong checking but insufficient capacity to review the volume automation produces. These are distinct constraints with different implications for scale. A total score that assigns them the same value may be adequate for a high-level summary while remaining insufficient for deciding which constraint to address.

10.1 Changes in the score over time

A repeated assessment introduces a further comparison: does a change in the numerical score reflect a change in readiness, a change in respondents or a change in the meaning of the questions? An initial survey can be answered before staff understand the deployment. After a pilot, the same wording may elicit more informed but less confident responses because participants have discovered difficulties. A falling confidence score can therefore coexist with a better understanding of the task. The interpretation requires evidence about what changed, rather than assuming that every increase is progress and every decline is deterioration.

Comparisons are clearer when the assessment preserves the task description, respondent roles and scoring procedure, while recording departures from them. If a new model or workflow changes the implementation being considered, an unchanged question may refer to a different object. The earlier score remains a valid historical record of its own scope; it should not be treated as an identical measurement merely because the questionnaire file is unchanged. This is particularly relevant when a trial moves from internal assistance to external action, since the capability and review requirements have changed.

10.2 Reporting evidence without hiding the unresolved parts

A scorecard can preserve a domain profile alongside the composite. Readers can then see whether an apparently satisfactory overall result depends on compensation between domains. Evidence status can be reported separately from the assessed level: a capability believed to exist differs from one demonstrated on the required task, even if both receive the same qualitative rating. Keeping these observations separate allows a decision-maker to distinguish a known constraint from a verification gap.

Uncertainty can also be attached to the assessment process. The report can identify which roles were unrepresented, which documents were unavailable and which demonstrations were limited to selected cases. These observations do not need to be converted into invented statistical confidence intervals. Their purpose is to delimit the claim. Where the assessment is qualitative or judgement-based, numerical presentation should not imply a sampling precision that the design cannot supply.

Finally, a readiness report can state the decisions it is capable of informing. A description of team confidence may support discussion of implementation concerns. A production test may support a bounded deployment decision. Neither alone forecasts the commercial return from the programme. Keeping the intended use explicit reduces the risk that a summary prepared for one decision is later reused as evidence for a stronger claim.

Davis studies perceived usefulness and ease of use as constructs in technology acceptance, not as demonstrations of realised AI value. [13] Task–Technology Fit links a technology’s contribution to the requirements of a defined task. [14] Absorptive-capacity theory concerns the ability to recognise, assimilate and apply external knowledge. [15]

Bresnahan and colleagues examine complementarities among information technology, workplace organisation and skilled labour. [16] Rubin’s framework distinguishes the outcome under an intervention from an association between a readiness score and an observed outcome. [17] Allcott’s site-selection analysis makes transport across organisations a separate validity question. [18]

Underspecification research identifies deployment differences that aggregate held-out scores can miss. [19] Shmueli distinguishes predictive and explanatory goals. A readiness measure that forecasts implementation does not automatically identify which capability caused that implementation or which investment would improve it. [20]

11. Limitations and conclusion

The review does not validate a new instrument or compare commercial readiness products. Its central measurement evidence includes healthcare-oriented organisational-change research, an exploratory AI interview study and technical production guidance. Their populations and purposes differ from contemporary marketing deployments. Full-text appraisal was concentrated on three central readiness sources; other papers were inspected through summaries. No universal weighting, threshold or success probability can be estimated from this selected set.

Readiness is interpretable only in relation to a specified change, unit of analysis and evidential purpose. Existing literature distinguishes shared commitment, perceived collective capability, resources and demonstrated production properties. A questionnaire score can describe responses, while a diagnosis additionally needs evidence about the relevant constraints. Prediction requires validation against a defined subsequent outcome. The main scientific requirement is to keep those claims separate: numerical precision in a composite does not establish that it measures the intended construct or forecasts implementation success.

Source access and appraisal register

The additional comparative sources in references 13–20 are used for the specific distinctions cited in the discussion. Access is limited to previously inspected primary-source abstracts, publication records or manuscripts. Their inclusion does not imply a fresh full-text risk-of-bias appraisal or direct validation in the application discussed here.

The full texts of [1], [2] and [3] were inspected, including their construct definitions and methodological sections. Reference [4] was consulted through indexed publication and author summaries; the institutional repository redirected to an unavailable application, so no detailed model appraisal is claimed. Official overview pages were consulted for [5] and [6]. Research summaries or author abstracts were inspected for [7], [8], [9], [10] and [11]. The complete live guidance page [12] was inspected. Independent psychometric replication, instrument-specific validation and prospective marketing studies remain outside this review.

References