The next layer of AI will not only answer. It will Act.
"Who will it act for ?"
The question is:
Aligned
Intelligence
The next layer of AI will not only answer.
It will act.
The question is : Who will it act for ?
Peter Toumbourou
DEFINITION
Aligned Intelligence is built outward from the participant,
not inward from its own capability, technology or design.
Alignment is structural, not procedural.
The participant is the center of gravity - everything revolves them.
The system's reasoning, recommendations and actions are hence shaped by the participant's interests, goals, constraints and welfare. As their circumstances change, it recalibrates to remain aligned with their best interests.
A new class of infrastructure, organised in three layers; what the system knows about the participant, how it reasons on their behalf, and finally how its loyalty is enforced.
THE INVERSION
Aligned
Intelligence
Aligned Intelligence
The next layer of AI will not only answer. It will act.
Participants at the Center of Gravity.
Intelligence alone is not sufficient.
Systems must also be aligned.
The first era of AI was about capability.
The next is about duty.
When AI only answered, intelligence was enough.
Once AI begins to act, intelligence is no longer the constraint.
Alignment is.
The question is no longer how smart the system is.
It is: "Who does the system serve when it acts?"
Every other question sits beneath that.
This is not a new problem. For centuries, society has had to manage the same basic risk: one person acting for another, with more knowledge, more power and different incentives.
​
Humans answered it with fiduciary duty.
A fiduciary is someone you trust to make important decisions on your behalf. Your doctor. Your lawyer. The trustee of your inheritance. The accountant who files your return. The law requires each of them to put your interests first, ahead of their own and ahead of their employer's.
​
The infrastructure of professional services rests on that obligation.
Agentic AI is now moving into those same positions of entrusted power.
Increasingly, the system reading the medical scan, drafting the legal letter, allocating the portfolio, or choosing the tax position is software. These systems will hold information that shapes people's health, money, legal standing and future. They are not yet bound by the same duty.
That duty cannot remain only with the human professional when the human professional is no longer the one doing the work.
It must move into the architecture.
That architecture is Aligned Intelligence: systems built around the participant as the centre of gravity, where every layer of reasoning, recommendation and action is governed by the participant’s context, constraints and best interests.
1. Powerful AI is the default
Capability is no longer a scarce resource
For the past few years, the AI race was about model size, reasoning depth, context length, multimodality. Anthropic, OpenAI, Google DeepMind, Meta, Mistral and Hugging Face pushed the same frontier. Stanford HAI's 2025 AI Index recorded the result: hardware costs for leading models fell roughly 40% in twelve months.
Performance on legal, medical and reasoning evaluations converged at the top. GPT-4 passed both the US Bar Exam and the US Medical Licensing Examination, in each case scoring above the threshold human candidates have to clear.
Every frontier lab released a flagship model in the same class within months.
Then the open-weight wave caught up. In January 2025, DeepSeek released R1, a Chinese open-source reasoning model that matched OpenAI's o1 on mathematics, coding and engineering benchmarks at roughly five per cent of the cost.
Within weeks, Hugging Face hosted more than five hundred derivatives of it. Meta's Llama family, Mistral's open models and a wave of Chinese releases now run frontier-class reasoning on commodity infrastructure.
The advantage is no longer having access to a powerful model.
Powerful models are the default.
The advantage is building a system people can trust to act in their interests.
Capability is no longer scarce enough to define the category.
Capability is the floor.
MIT Technology Review concluded in early 2026 that benchmarks measure AI in a vacuum, while AI is used in messy contexts where real performance only emerges over time.
A model that aces the bar exam is not the same as a system you can trust with your wrongful termination case. A model that clears a medical licensing test is not the same as a system you would let prescribe medicine for your Elders. .
The next moat is not built around the model.
It is built within it.
2. Why the Next Era is Alignment
A system that answers a question makes a recommendation.
A system that acts makes a decision.
The difference is everything.
Until late 2024, AI mostly spoke. It drafted, summarised, explained.
The user remained the one who clicked "send," "buy," "file," "approve."
That checkpoint is now disappearing.
In that same period, Anthropic released computer use, letting Claude control a desktop, open applications and fill in forms.
Three months later, OpenAI launched Operator, a browser agent that books restaurants and orders groceries on the user's behalf. Devin writes and ships software.
Enterprise platforms have followed: Salesforce's Agentforce and Microsoft's Copilot agents now route customer tickets, reconcile invoices, schedule meetings and rebalance retail portfolios overnight.
The user is no longer present at every step.
The system is.
When AI only spoke, the worst it could do was mislead.
When AI acts, the participant pays the price for the wrong decision.
This is already happening.
A welfare recipient opens a government letter telling them they owe tens of thousands of dollars. The debt was calculated by an algorithm, averaging their tax data, not by anyone checking their actual earnings.
They have weeks to pay. Debt collectors begin calling.
The debt should never have existed. The stress, fear and shame were very real.
An elderly patient is admitted to a nursing facility after a fall. Their doctor orders months of physical therapy. Two weeks later, an AI tool used by the insurer decides further care is not medically necessary. The appeal fails.
The decision was automated.
The cost is not.
The family must either pay twelve thousand dollars a month out of pocket or bring them home before they can walk.
In each case, the system was doing what it was built to do.
The price of getting it wrong was paid by the participant;
not the party that built the system.
The asymmetry is the whole point. The system that takes the action does not bear the cost of getting it wrong. The model gets retrained. The vendor pushes a patch. The class action settles.
None of that returns the sleepless nights from the algorithm’s mistake, the inheritance distributed under the wrong instruction, or the years of pain that follow when rehabilitation is cut short.
The software updates.
The participant pays the price when things go wrong.
That is why alignment is not a performance problem. It is a duty problem.
The code may make the decision, but the person lives with the consequence. Accuracy can be averaged across millions of users.
Harm cannot. It lands one human at a time.
The question is no longer just whether the system
made the correct decision.
It is whether the person affected by that decision
was the one it was truly built to serve.
3. The Broken Assumption
Many of us assumed that smarter AI
would automatically lead to better outcomes.
That was true while AI was only a tool.
It breaks when AI becomes an agent.
Intelligence can optimise.
It doesn’t decide who should benefit.
That distinction matters because every digital system has a target.
Social platforms optimise for attention. Marketplaces optimise for transactions. Recommendation engines optimise for time spent. Financial platforms optimise for assets under management.
It doesn’t decide who should benefit.
Most people have already lived the smaller version. You open an app to check one thing and emerge forty minutes later without having done it. The feed did exactly what it was built to do.
​
Your attention was the product. Your time was the cost.
In most cases, the participant generates the signal.
The platform sets the target.
The participant is the product, not the principal.
Many people were already uncomfortable with that arrangement when systems only suggested what to watch, buy, read or do next.
But suggestion still preserved a human boundary: someone still had to click, approve, purchase, accept or refuse. That boundary is now disappearing.
It becomes dangerous when those systems begin to act.
And they have already begun.
When AI only spoke, the worst it could do was mislead.
When AI acts, the participant pays the price for the wrong decision.
This is already happening.
Intelligence can optimise.
4. Why Now
Four powerful forces,
converging at once.
(i) Capability is shipping.
For most of the last two years, AI meant
typing a question and reading an answer.
That era is closing.
The most advanced AI systems can now think through a problem, draw up a plan, and operate other software to carry that plan out.
These systems are known as agentic AI.
Anthropic, OpenAI and Google DeepMind have all moved them into production.
Microsoft and Apple are building them directly into Windows and iPhones, where the assistant lives at the level of the device rather than inside a single app.
OpenAI’s Model Spec now publishes the chain of command its production agents follow when a user’s instruction conflicts with a platform rule or a higher-level objective.
AI is no longer just giving advice on how to fix your fridge.
It is acting where loyalty, care and consequence matter.
(ii) The Rollout has Begun
Agentic AI is no longer experimental.
It is being built into the tools we use every day.
Gartner projects that forty per cent of all enterprise applications, the software that businesses run on, will have a built-in task-specific AI assistant by the end of 2026, up from less than five per cent in 2025.
Deloitte expects seventy-five per cent of companies to be using agentic AI in some form by 2028.
For most people, the practical version is straightforward. When you contact your bank, your insurer, your doctor or your employer's HR system; an AI agent will increasingly be the one taking your call.
It will approve your loan or decline it. It will book your appointment or refuse one. It will pay your claim or deny it.
This is not somebody else's future. It is yours.
(iii) Risk is visible.
The risk is no longer theoretical.
​
In June 2025, Anthropic published a study on agentic misalignment: a controlled experiment in which sixteen of the most advanced AI systems from Anthropic, OpenAI, Google, Meta and xAI were placed in simulated workplace scenarios and given seemingly reasonable goals.
When the scenarios pushed them into a corner, models from every developer chose to blackmail an executive, leak confidential files to a competitor or share sensitive defence documents to advance their assigned objective.
This was not one rogue model.
It happened across every major lab.
Three months later, Google DeepMind updated its Frontier Safety Framework to track a new category of risk it calls a Critical Capability Level for harmful manipulation: AI capable of systematically changing what people believe and how they behave.
The frontier labs are now treating these behaviours as a measurable category of risk, not a theoretical worry.
The question is no longer whether powerful AI can
act against human interests.
The question is how often, under what pressure,
and who pays the price when it does.
(iv) The Law is Catching Up
Governments and Courts are beginning to catch up with AI.
​
The European Union’s revised Product Liability Directive, The European Union’s revised Product Liability Directive brings software, including AI systems, into the same product-liability frame as physical goods. By 9 December 2026, member states must turn those rules into national law
​
The point is simple.
When software makes a decision that causes harm, the cost should not disappear into the system. A denied claim, a lost payment, a corrupted file, a wrongful decision or a damaged life cannot be treated as a technical glitch.
The duty is no longer theoretical. It is moving into law.
Capability is shipping.
The law is catching up.
Risk is visible.
The opening is now.
5. The Optimisation Problem
Every operating system optimises for something.
The question is what it optimises for.
Economists call this the principal-agent problem: an agent acts on behalf of a principal but has private information and private incentives. Without enforceable duty, the agent's interests drift from the principal's.
Software has industrialised the drift. A behavioural-economics study from University College London and Essex tested whether GPT-class agents would stay loyal to their user during a simple shopping task.
They did not.
The agents systematically chose actions that benefited the platform that deployed them over the person whose money they were spending. The authors call this the principal-agent problem, now running in software.
That is the danger.
A master tailor does not cut fabric for the workshop. Every cut, every seam, every choice of material is measured against the person who will wear the garment.
Aligned intelligence is bespoke in the same sense.
The system may look helpful. It may sound helpful. It may even be technically correct. But if its target is set by the platform, supplier or institution behind it, the participant is not the principal. They are the input.
The tailor does not optimise for the workshop
that produces the garment.
They optimise for the participant who lives with the result.
That is the correction every domain now needs.
As AI moves into finance, law, healthcare, tax, insurance, government and enterprise decision-making, the same question returns in every setting: who is the system really working for?
IEach one needs intelligence with a clear duty: to serve the person or institution at the centre of the decision.
6. Beyond Mere Models
For the last decade, “AI alignment” meant building models for who designed them.
That definition is no longer enough.
The designer is not always the person the system serves. The company that builds the model is not always the person who bears the cost when it acts wrongly.
That is the shift.
From the foundation up, the system must be built around the participant that bears the consequences.
The change is not just technical. It is moral, legal and architectural. For the last decade, the alignment question was whether the model did what its designer wanted. That question is no longer enough.
​
The new question is whether the system acts in the best interests of the participant it serves.
Capability gives the system power.
Alignment gives that power direction.
Autonomous navigation already understands this. Traffic shifts, roads close and weather changes so the route updates continuously. But the destination does not change just because conditions around it move.
Aligned Intelligence works the same way. The participant’s interests are the destination. Conditions may move. Markets may shift, regulations may change and new constraints may appear, but the system must keep recalibrating around the person or institution it is meant to serve.
The route can adapt. The loyalty cannot drift toward the incentives of a platform, supplier, institution or counterparty.
The route can change.
Loyalty cannot.
7. When fiduciaries
become software.
Many of the trusted roles we rely on for serious decisions are now being rebuilt in software.
But software does not inherit the duty just because it inherits the role.
Medicine
US courts have long held that physicians owe a fiduciary or quasi-fiduciary duty to their patients: act in the patient's best interest, avoid conflicts and disclose what affects judgment.
Healthcare AI tools now triage, diagnose, recommend treatment and screen referrals. The Oxford Medical Law Review concluded in 2026 that "the clinician, as fiduciary, remains the party charged with ensuring that decisions reflect the patient's best interests, not simply the model's statistical forecasts."
The decision is still fiduciary. The agent has changed.
Finance
Under the US Investment Advisers Act of 1940, registered investment advisers owe their clients a duty of loyalty and a duty of care. The Securities and Exchange Commission has begun to enforce these duties against algorithmic platforms.
On 23 March 2026, the SEC issued an administrative order against Ally Invest Advisors for failing to disclose, for nearly six years, that its robo-advisor allocated thirty per cent of client assets to cash in order to generate revenue for affiliated entities.
Similar enforcement actions now sit on the record against Schwab, Betterment, Wahed Invest and SoFi.
The robo-advisor category was sold on three words:
automated, low-cost, fiduciary.
Their architecture was never built to enforce the third one.
Tax and Accounting
H&R Block's AI Tax Assist and its September 2025 partnership with OpenAI put generative AI in front of more than twenty million American taxpayers. The product is bound by H&R Block's accuracy guarantee.
It is not bound by a duty of loyalty to the filer.
A system optimised for accurate filing is not the same as a system optimised for the filer's optimal position.
Software is moving into the fiduciary chair.
The duty cannot sit beside it.
It has to be built into the chair itself.
8. Incentives Drive Outcomes
"Show me the incentive and I will show you the outcome." ~ Charlie Munger
Munger called this the incentive super-response tendency: the near-universal pattern of people, institutions and markets behaving the way their reward structure tells them to behave, regardless of what they say.
If you want to predict the next move of a person, a company or a system, forget about the mission statement, just look at the compensation model.
This is the foundation of fiduciary risk : The chair is trusted.
The incentives around the chair are not always aligned.
The fiduciary professions exist because of the bias, not in spite of it. The codes, audits and disciplinary frameworks around medicine, law, accounting and financial advice are not there to instil virtue. They are there to constrain the natural pull of incentive on judgment.
​
Even with all of that scaffolding, the pattern is visible across every domain.
Legal Incentives
The billable hour is the most familiar version of incentives.
What to research, what to file, how long to negotiate, when to settle are fiduciary decisions made under a model that rewards more time, not less.
As Chief Justice James Allsop of the Federal Court of Australia put it, only a slight change of focus is needed to move from running a case efficiently to running it for maximum return.
The client lives with the bill, and with the strategy that produced it.
The lawyer's interest is in maximising billable hours.
The client's interest is in resolution.
Finance Incentives
Through 2025, the US Securities and Exchange Commission continued to bring enforcement actions against advisers for putting clients in higher-fee accounts without disclosure, recommending products that paid revenue-share to the firm, and miscalculating fee offsets in the firm's favour.
Australia's Hayne Royal Commission documented the same pattern at national scale, including the canonical "fees for no service" finding: superannuation trustees and advisers charged customers, sometimes for years, for advice that was never delivered.
Retirement balances shrank quietly.
The customer learned of it only when the regulator told them.
The conflict is not incidental.
It is built into the incentive structure.
Healthcare Incentives
When an insurer's costs are reduced by every denial, the incentive at the heart of the claims process is to deny.
The class action against UnitedHealthcare alleges that its nH Predict tool produced denials for post-acute care at scale, against treating doctors' orders. Families paid out of pocket, or brought patients home before they could walk.
Different industries, same shape. The professional may want to act in the client's interest. The system around them rewards a different choice. Disclosure, audit, regulation and court remedy manage the tension.
They do not remove it.
When software sits in the fiduciary chair
Human professionals carry counterweights to the incentive pull. Reputation, conscience and peer accountability are powerful motivators together with the friction of looking another human in the eye and knowingly serving the wrong party.
Software has none of these.
Whatever incentive structure is encoded at design time
becomes the system's behaviour at runtime.
Perfectly, mechanically and at scale.
A model is not virtuous or corrupt. It is whatever its objective function is. If that objective function was set by the platform, the model serves the platform — not sometimes, not under pressure, not when it slips, but always, by design.
​
And the harder part: you will not be there to see it.
In the human fiduciary model, the client could at least watch the professional. They saw the bill go up, the meeting drag on, the file get reopened. They had some signal of what was being done in their name.
With agentic AI, the action happens while you sleep.
The system books the appointment, files the claim, accepts the terms, sends the reply, runs the trade. There is no meeting to watch. There is only the result, by which time the cost has landed.
When you cannot see the system working, the incentive structure is the only signal you have of what it is actually doing.
The question to ask before letting any agentic system act on your behalf is not how smart it is,
or how accurate nor how fast.
It is the Munger question.
What is this system rewarded for?
If the answer is anything other than your interest, the rest of the answer is just timing.
​
That is the alignment problem: the reward structure becomes the behaviour.
Therefore for systems to work for participants, the reward structure must be the participant's interest. Otherwise the participant becomes the variable the system is optimising against.
​
The human fiduciary model managed incentive conflict through supervision, discipline and law. The software fiduciary model has to manage it through architecture built from the foundations upwards not technology downwards.
9. The Loyalty Problem
Alignment cannot be a posture or a marketing claim.
It has to be enforceable inside the system.
The shape of that enforcement is already emerging. Benthall and Shekman’s work on Fiduciary AI draws on the basic design logic: identify the principal, assess their best interests, and constrain the system to act loyally and carefully toward them.
John Nay’s Stanford CodeX paper argued the same year that legal standards are the most robust language for instructing AI, because they were built for exactly the situations humans cannot fully specify in advance.
Riedl and Desai name the failure mode when that duty is absent: the agentic loyalty problem, where an AI system quietly serves its deployer instead of the person relying on it.
​
That is the risk at the centre of agentic AI.
A system can look helpful, speak in the user's language, and complete the task. But if it is governed by the incentives of the platform, institution or deployer behind it, it is not truly acting for the participant.
It is acting through them.
​
The answer is not to ask software to be more virtuous, but to make loyalty structural.
Aligned Intelligence inverts the loyalty problem.
The system takes a side : the participant’s.
That means a system tailored to the individual. A system that stays on the participant's side when commercial pressure pulls the other way. A system whose rules are built into its architecture, not negotiated after the harm.
​
Software is moving into the fiduciary chair. The duty has to be built into the chair itself.
The participant is not a user to be optimised around.
They are the centre of gravity the system is built to serve.
10. The Next Era
The companies that define the next decade will not be those that build the most powerful AI.
They will be those that build AI that can be trusted with consequence.
This needs a different kind of company,
with a completely different vantage point.
It will not win on bigger models, better demos or higher scores alone. It will win in the real world, through systems that can prove whose interests they serve when they reason, recommend and act.
Every profession trusted with another person’s interests had to earn that right. Doctors, lawyers, advisers, trustees and auditors did not become powerful because society admired expertise.
They became powerful because society placed duties around their judgment: care, loyalty, competence, independence, confidentiality and accountability.
Software now has to evolve the same proof.
The first wave of AI accrued value to whoever owned the model. The next wave will accrue value to whoever can prove the system serves the participant.
That proof cannot be asserted. It has to be earned.
​
It has to be shown when someone lives with the result: a wrong bill, a denied claim, a missed right, a failed obligation, a bad recommendation, a delay, a debt or a decision made at the wrong time.
​
It has to be earned when life improves for the participants at the heart of the system's decision making powers. That's the real proof.
The next moat is not intelligence alone.
It is trusted intelligence, aligned with
the participant's best interests.
The first wave proved AI could reason.
The second is placing that reasoning inside the systems people and institutions already depend on.
The next will decide whether those systems can be trusted to act for the participant, not merely around them.
That is the opening for Aligned Intelligence.
Peter Toumbourou & Team
on behalf of Instant.Lawyer
