Semantic Coherence and Cognitive Offloading with LLMs
(This is just a one off Sunday write off as a response for a LinkedIn post by a friend; let me know in the comments if you want me to write a more polished version)
The key problem here is that this AI hype is feeding on the shame of people who "identified" as writets or artists but never wanted to put in the hours to learn. Now they are given this tool that removes the mechanical steps and produces a “ready made product”.
However, professional writer or artist often has a vision towards which they want to navigate. And this is where the semantic coherence gap is. The embedding space has been optimized for syntactic coherence by masking and diffusing human generated artifacts. Also the supervised and human feedback enhanced aspects of training data processes are void of narrative structures. In other words they lack information to gain from the aspects of what makes narratively good quality feature.
When artist tries to change the images iteratively with AI, what often happens is the AI is unable to reliably perform the change. This happens because the change requested might not be accessible with the high rank vectors of the embedding. For example, because the mountains in a portrait picture were learned from huge mass of portrait pictures, the machine learning model treats the mountains as noise, because the high rank embedding vectors for mountains might not be active. In other words the mountains are latent features from noise diffusion.
Semantic coherence has a lot to do with being able to distinguish when some words or picture objects are syntactic noise that supports the narrative content and when they are narrative content.
Real artists and writers are masters of human attention. When artist makes an image they try to build a narrative path, which mathematicians might consider as attractors. Your attention first goes to something, then something else and finally lands on something. This is what succesful painting or photograph does: it gives you a story.
Same goes for text. Writers are masters at guiding your attention to something so that you can later be surprised or moved or invoked some other emotions, because your expectations got played.
The same is true for programming also. Junior developers are happy when the “unit tests pass”, while senior programmers try to implement constraints to the code that would make the system robust against unwanted changes done by eager newbies who have not paid the price of cognitive dissonance yet in order to understand the domain first.
Seniors also favor design patterns that make the code easier to read because the narrative emphasis is on correct places. Martin Fowler and Eric Evans are perhaps two of the most senior developers of narrative programming. It is probably impossible to generate robust code without metalanguages for Design Patterns and Domain Language.
I have been trying to use Agentic AI systems for fiction writing help. The problem is that their advices are very superficial and blind to narrative structures. They just parrot advices from writing forums. I have tried to enforce them to use writing theories, but currently fluent usage of them seems to be out of scope.
So my experience as a writer is that the advice given are often unhelpful; it often wants to remove the artistic elements that make the story interesting and individual coherent piece, and following the instructions often directs you towards pulp fiction.
If I use the AI to generate the story from detailed manuscripts there is another problem. I have to read each generated text, which is absolutely too time consuming.
I have been competing with short stories and I tend to re-read my own texts only four times. After four generations with advanced Agentic AI system specifically using same writing theory elements as I do, four iterations is still pulp fiction. Even worse; it seems that the iterations are recycling same pieces of content if it stays on same theme, which suggests I might be recycling sentences from someone else’s work.
What AI actually offers is cognitive offloading. Trump is a good example of what happens to democracy, when citizens engage in cognitive offloading of politics to media and politicians instead of trying to push through the hard emotins invoked by cognitive dissonance.
Cognitive dissonance is the emotion you have to go through everyday in order to learn to be better at something. Don’t be “American” and try to get away with by pushing forward something that looks correct but is hollow. Socrates called these snakeoil. Western civilization started from Socrates, who believed that going through the Cognitive Dissonance is the guiding principle on our journey towards robustness.
If something looks like a story for uneducated mind, they might think it is and that there are no better stories. Quality stories are a scarce resource. Capitalism benefits from selling pulp fiction by making it abundant and making it harder to discover authentic content. Adolescense was not created from Netflix algorithms, but by a man whose mother worked in an industry that is very relevant to understanding the content.
Capitalism wants you to pay by representations instead of content, because representations allows it to take your money and run. This is why professional content curstion is important, but it will become impossible now that anyone can skip learning the craft of narration and push their mechanically generated representations forward.
In optimistic scenario, during next years we will pay more attention to content curation than production. In pessimistic scenario everything will become flimsy, but people will keep on consuming it anyway, because they perceive there is no choice. Though there is always The choice to unplug from the Matrix… if you can still find an analog line from a telephone booth. I would recommend Adolescense by Stephen Graham.
