Machines & Machinations
416SPACES · MARKET ESSAY
On AI-assisted reasoning, the verification illusion, and the particular kind of self-deception that feels most like thinking.
I work in a field that runs on high-touch, high-stakes conversation. Buyers in acquisition mode. Sellers navigating disposition. Investors reshuffling portfolios. The discourse is sensitive by nature, and sensitive discourse has tells.
One of the tells I have started noticing, with increasing frequency, is when a new participant enters the conversation. The response in a chat thread or email that is too structured, too comprehensive, too even-handed, or too tone-deaf. The hedging is calibrated in an obvious way. The paragraph breaks are intentional. The voice is there, in a way, but also not, in another. Guilty of all or guilty of some, we’ve all likely been on this side of the conversation.
I can usually spot it. The giveaway is when someone has not primed their machine to their conversational style, their deductive quirks, their locutionary commoners.
When I suspect it, I go to the phone out of curiosity. I have started calling this runaway discursion. The condition in which prior AI prompts have branched the reasoning logic, refed the user conclusions from it, and each subsequent exchange between interlocutors is itemized and passed through a biased chat history that has been quietly steering the whole time. On the phone you ask: can you walk me through your reasoning and how you got here? If the conversation isn’t enriched from there, you have your answer before you ask your question.
The verification illusion
Here is the psychological mechanism worth naming. When you reason with a machine that knows you well enough to emulate you, something interesting happens to the epistemic status of the output. It feels like you verified the proposition independently. The machine is outside of you. It is hooked up to the world and all its data, or so the feeling goes. So when it confirms what you were already thinking, it registers not as a mirror but as a third party, and the conviction it produces approaches something like RCT level certainty.
But you did not get an independent opinion. You prompted yourself, in a way, to believe what you already believed.
Folk psychology and scientific psychology are both convincing on how easy it is to dupe ourselves. That is why we use safeguards, conditionals, meta-reasoning — reasoning about our reasoning — when the stakes are high. But most people do not go up that layer. Because there is an interesting loss of identity that occurs when reasoning with a machine trained enough on the world to sound like a reasonable version of you. The output feels authored. The conclusions feel earned. The assumptions baked in go unexamined because nobody asked them to surface.
There is no literature on psychological bias propagation through machine-assisted reasoning that functions as standard dictum (far as I know). Which means there is no cultural safeguard, no social nudge pattern, reminding us when we are over-relying on pattern recognition machines that simulate natural reasoning and apply it to the most intimate and sensitive parts of our lives.
What you hand over without knowing it
The most instructive thing you can do with a machine, if you want to understand the scale of what you hand over in a standard interaction, is revoke its authority over “study design” and “reasoning structure”. Ask it not to make any assumptions regarding what the optimization function to your answer should be. Ask it not to choose, not to prioritize, not to synthesize. Ask it instead to return the decision about how to frame the problem back to you, and to surface multiple equally plausible paths rather than collapsing them into a single well-argued recommendation.
Standard mode: You dump a question and get a response. The machine selects the optimization function, decides what a good answer looks like, chooses which considerations to weight, and returns a structured conclusion. Fast. Feels thorough. The assumption space corrupted in the process is invisible.
Revoked authority mode: You ask for five equally plausible analyses and make the design decisions yourself. Slower. Considerably more uncomfortable. You see exactly how much was being assumed on your behalf, how much longer the real inquiry takes, and how much of your prior certainty was borrowed from the machine's editorial choices rather than your own reasoning.
Do this once with something you thought you understood clearly; whether it is to wait on a mortgage renewal, or whether certain condos on W King W are projected to appreciate over the next decade, or any of the thousand questions you have asked a machine and received a tidy answer to. See what happens when the machine is asked not to optimize at all, and to hand the optimization function back to you.
What you find, reliably, is that the certainty you felt after the clean response had less to do with the quality of the answer and more to do with the quality of the framing. And the framing was not yours.
None of this is new
The interesting thing is that none of this originates with artificial intelligence. We do it all the time, with ourselves, with our conversational partners, every day and in every direction. We assume shared context. We assume our words carry the same meanings they carry in our own heads. We proceed as though understanding has occurred when it frequently has not, because it is cheap to be misunderstood and not realize it, and to keep the conversation going, because we are in fact the same pattern recognition machines, the same strange little loops.
What AI adds to this is scale. The machine is fast enough and coherent enough that the assumption gap never produces friction. There is no moment of confusion that breaks the spell. The conversation just proceeds, carrying the original bias forward, elaborating on it with increasing confidence, until the person on the other side has a very thorough and well-structured version of what they already thought before they started.
That is not thinking. That is thinking that feels like thinking, which is harder to catch and more expensive when you do not.
What to do with this
Meta-reasoning helps with some of this. Reasoning about your reasoning. Asking not just what the machine returned but what it assumed in order to return it. What optimization function it selected. What it decided a good answer would look like before it started constructing one. These are questions that slow the interaction down considerably and produce a different kind of output — less satisfying in the moment and more useful over time.
This applies beyond real estate, though I try to keep these articles steered close to the decisions I work inside professionally. It applies to how people reason about love, loss, finances, fears. If I had access to a broad sample of chat archives spanning that cross-section of life, I would not dare open it.
The practical version is this. Next time you ask your machine (brain or computer) whether it is advisable to wait on a mortgage renewal, or whether a given neighbourhood is worth the premium, or any of the thousand questions that carry real consequence: ask it to give you five equally likely and equally plausible analyses. Or better, ask it to revoke all authority over the optimization function entirely, surface the assumptions it would otherwise have made silently, and let you make the design decisions yourself.
See what happens. See how much longer it takes. See how much was being decided on your behalf, quietly, every time you asked a clean question and got a clean answer.
The machines are not the problem. The problem is the belief that they are outside of us in any meaningful epistemic sense, that their outputs carry a different weight than our own thoughts because they arrived in a different window. They do not. They carry the weight of what we brought to them, organized and returned with a confidence the original material did not always deserve.
We are pattern recognition machines in different sleeves. The useful move is to remember that when the output feels most like certainty.
P.S — Parts of this article were written with the help of a machine.
