"Starting EBPM from the ‘question’"

Where do the collected voices go afterward?

In local government offices, surveys are conducted each time a plan is drafted, and the public hearing desk receives many voices daily. Public comments, letters to the mayor, awareness surveys conducted independently by each department, posts on social media—opportunities to hear citizens’ voices are actually increasing.

However, when asked how much those voices are actually used to improve policies or inform the next decisions, it is difficult to answer with confidence. The collected voices often remain scattered across departmental files or systems, fall asleep, and are no longer referenced by the time plans change. Many staff members are likely familiar with the situation where the process ends after receiving the input.

A clue to solving this “wastefulness” lies in the increasingly widespread EBPM, i.e., evidence‑based policy‑making.

What the fiscal authorities of the UK and the US have been practicing

In the paper “How UK and US Treasury Departments Engage with EBPM” written by Yohei Kobayashi of the Ministry of Finance’s Policy Research Institute, the development of EBPM in Britain and America is carefully organized.

What is emphasized there is the fact that, in both countries, the fiscal authorities are the main players of EBPM. In the UK, the Treasury and the Cabinet Office lead, while in the US the Office of Management and Budget is at the core, integrating budget formulation and performance‑management mechanisms with EBPM. It is an approach that does not separate the process of deciding how money is spent from the process of checking what actually works.

At the core of this is the concept called the Learning Agenda. It is a medium‑term plan, typically spanning more than four years, that first defines the “questions” that should be clarified in policy formulation, and then systematically builds up the data and evaluations needed to answer those questions. In the United States, the Evidence Act even mandates each agency to develop such a plan, showing how highly it is valued. The detailed annual plans derived from it are called annual evaluation plans and operate in tandem with the Learning Agenda.

The crucial point here is the order. First, the question of what we want to know in order to advance a policy is clarified through dialogue with the field and the responsible department. This represents the demand for evidence. On the other hand, what is already known constitutes the supply of evidence. The gap—the part that is still unknown—is the evidence gap, and a plan is made to fill it over time. This is the backbone of the Learning Agenda. Because there is a question, the information to be collected is defined, and the collected information gains meaning.

However, in the UK and the US, both the setting of the question and the subsequent evaluation have been carried out within the administration—by evaluation officers, evaluation task forces, and other internal experts. Citizens may be the subjects of surveys, but they are not necessarily envisioned as participants who co‑think the questions themselves.

Evaluation exists to prevent spending on weak policies

When you hear “evaluation,” you might think it is simply grading a project and finishing. Yet the cases presented in the reports teach another meaning of evaluation. A home‑visiting program by nurses for low‑income pregnant teenagers, which showed great impact in the United States, was tested in the United Kingdom and found to add only limited benefit because standard care was already comprehensive. Although the result may seem disappointing at first glance, it prevented the allocation of budget to a policy with weak effect. Determining what works and what does not becomes a decision‑making tool for directing limited resources to where they are truly needed.

Four steps based on Liqlid

Our Liqlid starts not by keeping the question inside the administration but by sharing it with citizens. Citizens are welcomed not as mere survey subjects but as partners who think about the question together. Consequently, the voices that come in become responses to the questions the administration truly wants to answer, rather than just impressions. Specifically, we nurture citizens’ voices into policy assets through four steps.

First, we formulate the question. On Liqlid, we set and publish the question of what we want to know from citizens, and solicit new public opinions. The question set at this stage is aligned with, or linked to, a question included in the separately defined Learning Agenda. The difference from traditional public hearings lies in overlapping the question the administration wishes to clarify with the question citizens want to consider as their own, into a single shared question.

Second, we aggregate the voices. Survey results conducted by each department for plan formulation and the public‑hearing data held by the hearing division are imported into Liqlid and centralized. This means gathering voices that were previously separated by department into one place.

Third, we extract from the question. By crossing opinions gathered, departmental surveys, and public‑hearing data—each with different sources—we create a state where citizen voices related to a specific question can be retrieved in an integrated manner.

Fourth, we match with projects. We compare the list of administrative projects with the accumulated citizen voices to understand which project is linked to which voice, when, and how much.

Visualizing the “ripple effect” of policies over multiple years

Among these four steps, the recurring activities each year are the solicitation of new opinions and the import of departmental surveys and public‑hearing data. Voices accumulate inside Liqlid year after year.

In addition, matching with projects is not done annually but every few years, which is an important point. Reactions to a policy in a given year do not necessarily all come together within that same year. Some voices emerge in the same year, while others appear in the following year or later. By observing the voices for a policy implemented in year t as the voices of year t, the next year, and subsequent years, we can track how that policy ripples into citizens' lives as a time‑series change. This overlaps with the idea of the UK/US Learning Agenda that builds evidence over several years.

The insights gained from the matching are then fed back into setting the next question. What has become clear, and what remains unknown—the difference becomes the new question to be posed for the next fiscal year. By continuously cycling from question to insight and back to question, citizens’ voices thicken like tree rings.

Outcome evaluation is conducted as outcome evaluation

Here is something we must not confuse. Understanding obtained from surveys that ask about satisfaction or awareness, and outcome evaluation that verifies what actual results a policy has produced are separate tasks. No matter how many citizen voices are collected, that alone does not prove policy effectiveness. To determine what worked and to what extent, it is essential to conduct impact verification using specialized methods, including comparisons with control groups. The evaluation of the home‑visit program mentioned earlier was precisely such a rigorous outcome evaluation. Not making the premature assumption that results have been achieved merely from the amount of output or reactions is also the starting point of EBPM.

Liqlid’s role is not to replace this evaluation. It is to share the question with citizens, continuously accumulate the voices related to that question, and make visible which responses are linked to which projects. That is Liqlid’s function. Outcome evaluation remains an outcome evaluation, separate from Liqlid, and should be carried out properly with the necessary methods. Then, clues from the perspective of citizen voices are added. The two are not competing; they complement each other.

Turning broad listening from ‘just receiving and ending’ into an ‘asset for answering’.

What the fiscal authorities of the UK and the US have demonstrated is an attitude of embedding evaluation and evidence directly into the decision‑making mechanism itself. However, much of that activity is completed within the administration or among experts. What we aim for is to invite citizens from the stage of formulating the question and to use the region‑specific evidence of citizen voices in decision‑making. It is important not to simply receive the submitted voices and stop, but to store them as assets for answering policy questions, creating a state where they can be used at any time for reviewing projects or setting priorities.

Even without a dedicated analysis unit, you can start by formulating a single question and gathering the existing broad‑listening data in one place. Rather than letting citizen voices end as a one‑off event, turn them into a resource that continuously supports urban‑development decisions. We intend to explore a new form of EBPM.