Making AI operational in constrained public sector environments
The core operational insight is that public agencies do not fail at advanced models because they lack ambition; they fail because standard deployment assumptions clash with government realities. Continuous cloud connectivity, loose data movement, and opaque infrastructure are tolerable in many firms but can be unacceptable where sensitive records, legal duties, and continuity requirements dominate. That is why locally hosted small language models look less like a downgrade than a governance-aligned architecture for constrained environments.
The practical mechanism is to move the model to the data, not the reverse. Smaller models consume far less compute than the largest systems, can run on local servers or even devices, and avoid dependence on scarce GPU capacity that many agencies are not set up to manage. Their usefulness comes from retrieval methods, vector search, and verifiable grounding, which let systems answer from checked sources rather than from stale training alone. For practitioners, that changes deployment from speculative experimentation to operational search and document handling.
The limits matter as much as the promise. Small models are not a universal substitute for broader systems, and the articleโs case depends on disciplined prompt design, curated data, and ongoing monitoring to keep errors, bias, and unsupported output in check. Their advantage is not isolation for its own sake but strategic autonomy that supports auditability, privacy compliance, and resilience. The editorial takeaway is clear: in public service settings, usefulness is measured by control, verification, and continuity, not model size.
The AI boom has hit across industries, and public sector organizations are facing pressure to accelerate adoption. At the same time, government institutions face distinct constraints around security, governance, and operations that set them apart from their business counterparts. For this reason, purpose-built small language models (SLMs) offer a promising path to operationalize AI in these environments.ย ย
A Capgemini study found that 79 percent of public sector executives globally are wary about AIโs data security, an understandable figure given the heightened sensitivity of government data and the legal obligations surrounding its use. As Han Xiao, vice president of AI at Elastic, says, โGovernment agencies must be very restricted about what kind of data they send to the network. This sets a lot of boundaries on how they think about and manage their data.โ
The fundamental need for control over sensitive information is one of many factors complicating AI deployment, particularly when compared against the private sectorโs standard operational assumptions.
Unique operational challenges
When private-sector entities expand AI, they typically assume certain conditions will be in place, including continuous connectivity to the cloud, reliance on centralized infrastructure, acceptance of incomplete model transparency, and limited restrictions on data movement. For many state institutions, however, accepting these conditions could be anything from dangerous to impossible.ย
Government agencies must ensure that their data stays under their control, that information can be checked and verified, and that operational disruptions are kept to an absolute minimum. At the same time, they often have to run their systems in environments where internet connectivity is limited, unreliable, or unavailable. These complexities prevent many promising public sector AI pilots from moving beyond experimentation. โMany people undervalue the operating challenge of AI,โ Xiao says. โThe public sector needs AI to perform reliably on all kinds of data, and then to be able to grow without breaking. Continuity of operations is often underestimated.โ An Elastic survey of public sector leaders found that 65 percent struggle to use data continuously in real time and at scale.ย
Infrastructure constraints compound the problem. Government organizations may also struggle to obtain the graphics processing units (GPUs) used to train and access complex AI models. As Xiao points out, โGovernment doesnโt often purchase GPUs, unlike the private sectorโtheyโre not used to managing GPU infrastructure. So accessing a GPU to run the model is a bottleneck for much of the public sector.โย
A smaller, more practical model
The many nonnegotiable requirements in the public sector make large language models (LLMs) untenable. But SLMs can be housed locally, offering greater security and control. SLMs are specialized AI models that typically use billions rather than hundreds of billions of parameters, making them far less computationally demanding than the largest LLMs.
The public sector does not need to build ever-larger models housed in offsite, centralized locations. An empirical study found that SLMs performed as well or better than LLMs. SLMs allow sensitive information to be used effectively and efficiently while avoiding the operational complexity of maintaining large models. Xiao puts it this way: โIt is easy to use ChatGPT to do proofreading. Itโs very difficult to run your own large language models just as smoothly in an environment with no network access.โย
SLMs are purpose-built for the needs of the department or agency that will use them. The data is stored securely outside the model, and is only accessed when queried. Carefully engineered prompts ensure that only the most relevant information is retrieved, providing more accurate responses. Using methods such as smart retrieval, vector search, and verifiable source grounding, AI systems can be built that cater to public sector needs.ย
Thus, the next phase of AI adoption in the public sector may be to bring the AI tool to the data, rather than sending the data out into the cloud. Gartner predicts that by 2027, small, specialized AI models will be used three times more than LLMs.
Superior search capabilities
โWhen people in the public sector hear AI, they probably think about ChatGPT. But we can be much more ambitious,โ says Xiao. โAI can revolutionize how the government searches and manages the large amounts of data they have.โ
Looking beyond chatbots reveals one of AIโs most immediate opportunities: dramatically improved search. Like many organizations, the public sector has mountains of unstructured dataโincluding technical reports, procurement documents, minutes, and invoices. Todayโs AI, however, can deliver results sourced from mixed media, like readable PDFs, scans, images, spreadsheets, and recordings, and in multiple languages. All of this can be indexed by SLM-powered systems to provide tailored responses and to draft complex texts in any language, while ensuring outputs are legally compliant. โThe public sector has a lot of data, and they donโt always know how to use this data. They donโt know what the possibilities are,โ says Xiao.
Even more powerful, AI can help government employees interpret the data they access. โTodayโs AI can provide you with a completely new view of how to harness that data,โ says Xiao. A well-trained SLM can interpret legal norms, extract insights from public consultations, support data-driven executive decision-making, and improve public access to services and administrative information. This can contribute to dramatic improvements in how the public sector conducts its operations.
The small-language promise
Focusing on SLMs shifts the conversation from how comprehensive the model can be to how efficient it is. LLMs incur significant performance and computational costs and require specialized hardware that many public entities cannot afford. Despite requiring some capital expenses, SLMs are less resource-intensive than LLMs, so they tend to be cheaper and reduce environmental impact.ย
Public sector agencies often face stringent audit requirements, and SLM algorithms can be documented and certified as transparent. Some countries, particularly in Europe, also have privacy regulations such as GDPR that SLMs can be designed to meet.
Tailored training data produces more targeted results, reducing errors, bias, and hallucinations that AI is prone to. As Xiao puts it, โLarge language models generate text based on what they were trained on, so there is a cut-off date when they were trained. If you ask about anything after that, it will hallucinate. We can solve this by forcing the model to work from verified sources.โ
Risks are also minimized by keeping data on local servers, or even on a specific device. This isnโt about isolation but about strategic autonomy to enable trust, resilience, and relevance.
By prioritizing task-specific models designed for environments that process data locally, and by continuously monitoring performance and impact, public sector organizations can build lasting AI capabilities that support real-world decisions. โDo not start with a chatbot; start with search,โ Xiao advises. โMuch of what we think of as AI intelligence is really about finding the right information.โ
To learn more about AI in the public sector, visit Original Postublic-sector?utm_source=publisher-direct&utm_medium=other-mit-tech-review-pubsec&utm_campaign=mit-ps-gc-pubsec" target="_blank">Elastic.
This content was produced by Insights, the custom content arm of MIT Technology Review. It was not written by MIT Technology Reviewโs editorial staff. It was researched, designed, and written by human writers, editors, analysts, and illustrators. This includes the writing of surveys and collection of data for surveys. AI tools that may have been used were limited to secondary production processes that passed thorough human review.
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.