Power, governance, and accountability in humanitarian artificial intelligence

In their Comment, Shreenik Kundu and colleagues convincingly argue that ethical artificial intelligence (AI) could help humanitarian organisations respond to worsening aid shortfalls by improving data collection and analysis.¹ Humanitarian systems face expanding needs, reduced resources, and growing demands for data. AI systems can support translation, summarisation, classification, and analysis of qualitative data used to understand context, needs, or response effectiveness. These applications raise technical challenges around data quality, accuracy, and reliability, as well as ethical questions around bias, privacy, and responsible use. The central governance question, however, is whether AI systems strengthen accountability, participation, and recourse for affected people or further concentrate power over data, decisions, and remedies in ways that create new protection risks.

The risks associated with AI in humanitarian settings exist at several levels, each ultimately pointing to governance issues. Kundu and colleagues warn of the risks of hallucination, bias, and inaccuracy, especially in low-resource and Indigenous languages.¹ Large language models might generate outputs that are plausible without being reliable or valid. Our ongoing evaluation of the use of large language models for qualitative data analysis shows that the performance varies depending on how needs are expressed.² AI systems appear to perform reasonably well on needs stated in direct and unambiguous terms, such as hunger or shelter, but less well when experiences are shaped by culture, fear, stigma, or context, including discrimination, protection concerns, and physical safety. Errors in these areas can distort needs assessments, obscure patterns of abuse, or direct attention away from those most at risk.

A second level concerns safeguards that are often treated as more protective than they are. Encryption can secure data while the data are stored or transmitted, but AI models still need to operate on readable data, so sensitive information might remain exposed during processing and can be difficult to detect or remove. Anonymised data remain vulnerable to re-identification, particularly when combined with other datasets, and can be used to make harmful inferences about entire communities. These risks raise questions about who is permitted to process humanitarian data, for what purpose, under what oversight, and with what consequences for misuse.

A third level concerns consent and participation. A scoping review of personal data processing and AI in humanitarian crises identified concerns about privacy, consent, surveillance, bias, data quality, unequal access to assistance, and weak accountability, all of which carry direct protection implications for affected communities.³ People affected by crises often provide information under constrained conditions because registration, assessment, or service access depend on participation. Consent mechanisms lack the conditions for meaningful choice when future data uses, commercial processing, dataset linkage, or security risks are difficult to anticipate. The risk is that participation becomes a condition for being helped, while control over data and decisions remains elsewhere.

In the context of funding cuts, humanitarian AI adoption risks further prioritising cost reduction, faster reporting, automated decision making, and organisational reach over affected people’s ability to understand, challenge, or refuse harmful uses. There is also a growing gap between large international organisations with the resources to deploy advanced AI tools and smaller local organisations that lack the infrastructure and resources to do so, deepening existing inequalities in humanitarian decision making. Taken together, these risks are structural rather than technical or procedural. The fundamental concerns around humanitarian AI are about governance, authority, and remedy.

A central commitment in humanitarian reform is to shift leadership, resources, and decision making towards national and local actors, including organisations, communities, and people most affected by crises. Digital systems can support this shift when local actors define priorities, shape data governance, evaluate outputs, control language resources, and exercise real authority over how systems are deployed. Research on humanitarian data systems warns that such systems can also reduce communities to data providers and reinforce existing hierarchies when international agencies, donors, or vendors retain control over infrastructure, standards, procurement, and funding.⁴ Controversies over data sharing, biometrics, interoperability, and unauthorised access often reflect deeper struggles over authority in the humanitarian space.⁵

Evidence from community-led digital protection initiatives indicates that technology supports locally led action only when local actors have the resources and negotiating power to shape how systems are designed and used.⁶ Local data collection, consultation workshops, or validation exercises are insufficient if decisions about model selection, risk tolerance, procurement, and accountability remain in external hands. Communities affected by crises have a legitimate claim to explicability, auditability, and accountability when automated systems make decisions about humanitarian assistance, protection, or resource allocation that affect them.⁷ Systems that infer preferences or model collective perspectives can create an appearance of participation without the social and institutional relationships that genuine consultation requires. Genuine consultation allows people to disagree, correct institutional assumptions, challenge categories, and demand accountability. AI systems should support this process rather than simulate or replace genuine consultation.

Humanitarian data are produced by converting people's accounts, locations, relationships, and vulnerabilities into institutional evidence. AI adds further layers of interpretation through transcription, translation, summarisation, labelling, and ranking, removing context, obscuring uncertainty, and conferring authority on technical outputs that affected people cannot inspect or contest. Technical safeguards can reduce specific risks, but they do not establish who has authority over humanitarian data, who can approve secondary uses, or what remedies are available when harm occurs. Humanitarian organisations, therefore, need enforceable legal obligations for data minimisation, retention, deletion, access review, incident response, and harm mitigation.

Human rights-based approaches to humanitarian AI argue that ethical commitments need to be translated into governance tools, accountability mechanisms, and safeguards.⁸ Practice also shows that ethical guidance and retrospective oversight provide weak protection when automated systems operate within unequal institutions. For example, a cash and voucher assistance programme in northeast Nigeria showed that digital assistance systems can deepen exclusion when political, gendered, and digital barriers intersect.⁹ Humanitarian applications of AI vary widely, and this anecdotal example is not isolated. Large-scale harm can occur when automated decisions are opaque, difficult to challenge, and treated as technical outputs rather than accountable judgements, and when affected people lack meaningful avenues to understand, contest, or correct these decisions.

Human review is frequently proposed as a safeguard against these concerns, but its protective value depends on how the review is designed and governed. The review needs to be defined operationally rather than invoked as a general assurance. To function as a genuine safeguard, reviewers need authority to intervene, technical and contextual understanding of system limitations, access to records of inputs and outputs, and accessible mechanisms for appeal. When applications of humanitarian AI cannot provide effective and genuine safeguards, they should, in most cases, not be used. AI systems whose outputs are used to determine access to assistance should not proceed unless affected people have access to a meaningful appeal mechanism and human decision makers have clear authority to review, reverse, or override the automated decision. Qualitative analysis involving protection, discrimination, or safety requires systematic review by people with the contextual knowledge to assess outputs. Commercial AI systems should not process sensitive humanitarian data unless their governance, security, auditability, and remedy arrangements meet humanitarian protection standards.

The International Committee of the Red Cross' policy on AI grounds their requirements in humanitarian principles, including precaution, do no harm, transparency, explainability, responsibility, and accountability, and offers a practical framework for the legitimate deployment of AI.¹⁰ Humanitarian organisations also need investment in open, auditable, and locally governable infrastructure that reduces dependence on proprietary systems whose incentives and accountability arrangements might not align with humanitarian mandates.

AI can help humanitarian actors to analyse data more consistently and make better use of information already collected from affected people. Achieving these goals without deepening institutional distance, normalising data extraction, or making accountability harder to locate requires governance arrangements that define who controls data, who can contest outputs, who can refuse particular uses, and who receives remedies when harm occurs. These arrangements need to be treated as preconditions for deployment rather than issues to be addressed once systems are already in use.