Procurement Accountability for Government AI Systems
When a federal agency procures an AI system from a private vendor and uses it to make individualized determinations, who bears liability for system errors — the agency, the vendor, or both?
Existing procurement law assigns accountability to the contracting agency, but current contract structures systematically fail to preserve the agency's ability to explain or override vendor system outputs.
The federal procurement system was designed around a fundamental assumption: the government buys goods and services, but retains responsibility for the decisions those goods and services enable. When an agency procures a weapons system, the agency — not Lockheed Martin — bears responsibility for how it is deployed. When an agency contracts for IT services, the agency — not the vendor — bears responsibility for the data processed through those systems.
This framework maps poorly onto AI procurement. When an agency contracts with a vendor to provide a machine learning model that scores benefit applications, assesses fraud risk, or prioritizes enforcement actions, the agency nominally retains decision-making authority. But in practice, the "decision" is the model's output. Agency personnel lack the technical capacity to evaluate individual predictions, the contract often prohibits access to the model's internal logic, and the operational tempo makes case-by-case human review impractical.
The result is a form of de facto delegation that procurement law was not designed to address. The agency remains legally accountable for each determination, but it has contractually surrendered the capacity to explain or override the system that generates those determinations. This is not merely an academic concern — it has due process implications when the determinations affect individual rights, and it has accountability implications when the system produces systematic errors.
Current FAR provisions for acceptance testing and quality assurance are inadequate because they focus on system-level performance metrics, not individual-determination explainability. An AI system can satisfy its contractual performance requirements while producing unexplainable or unjustifiable outcomes in specific cases.
Professor Okafor correctly identifies a real tension in AI procurement, but his framing overstates the novelty of the problem and understates the adequacy of existing legal tools. The government has long procured complex systems whose internal workings agency personnel do not fully understand — from pharmaceutical efficacy models to financial risk assessment tools. In each case, the procurement framework has adapted through contract design, not statutory reform.
The FAR already provides mechanisms to address the concerns Professor Okafor raises. Agencies can — and increasingly do — include contractual requirements for model documentation, algorithmic impact assessments, and source code escrow. The DFARS supplements include provisions for technical data rights that can be adapted to ensure government access to model architectures and training data specifications. Section 508 compliance requirements already impose accessibility and transparency obligations on procured IT systems.
The real problem is not legal inadequacy but contracting officer capacity. Most government contracting professionals lack the technical background to draft AI-specific requirements, evaluate vendor compliance, or negotiate meaningful transparency provisions. This is a workforce development problem, not a structural legal deficiency.
Moreover, Professor Okafor's call for "individual-determination explainability" sets a standard that even the most sophisticated AI developers cannot currently meet for many model architectures. Requiring contractual guarantees of explainability would either limit agencies to simple, less accurate models or create unenforceable contract terms. Neither outcome serves the public interest.