Directing Intelligence Digest | November 17, 2026
Much of the recent bad news about AI and learning came from studies that handed students a generic chatbot or a tool designed centrally. A new trial tried the opposite. Across 55 applied STEM classes at 19 universities, instructors built AI for their own courses, and their students scored higher on exams taken without AI. At work, OpenAI’s study of more than 1,800 scientists shows the same habit: people choosing the question and pushing back on the answer.
One question to carry through today’s brief: in your organization, who decides what the AI does, and who checks what it gives back?
The short version
Instructor-built AI raised exam scores: in a preregistered randomized trial with about 7,200 students in 55 applied STEM classes at 19 universities, scores on exams taken without AI rose about a third of a standard deviation (0.32 at the student level). (arXiv 2611.19099)
Scientists keep directing the work: two in three of the 1,800-plus scientists OpenAI studied used ChatGPT to explore research directions, and about once every three messages they came back for an explanation, a check or a revision. (OpenAI)
The saved time goes somewhere: Gallup finds about half of frequent AI users spend less time gathering information and writing, and about one in five spends more time reviewing and correcting work. (Gallup, Nov 16)
Here’s AI at work: scientists pick the question and push back on the answer
The research. Chris Liao, Carl Shan and Ronnie Chatterji of OpenAI’s economic research team released “Science at the Frontier: How Scientists Use ChatGPT” on November 16. They analyzed more than 580,000 scientific messages from over 1,800 scientists on the ChatGPT for Academic Researchers plan between August 20 and September 22, 2026, using an internal tool that sorted messages into categories so the researchers never saw the messages themselves. They also surveyed 251 of those scientists. OpenAI report.
What it finds. Two in three scientists (68%) used ChatGPT to explore possible research directions, such as brainstorming or tightening how a problem is framed. After a scientific request, scientists followed up within a week to ask for an explanation, a check or a revision about once every three messages, including in coding and data analysis. Surveyed scientists estimated AI makes research tasks nearly twice as fast at the same quality. The limits they named most were time or staffing (51.8%) and funding (43.8%).
How it fits. Gallup’s survey of U.S. employees, published the same day, shows the shift from the worker’s side. Among frequent AI users, 52% spend less time gathering information and 47% less time writing, while 82% spend about the same time with customers. About one in five spends more time reviewing and correcting work (21%), and about as many spend less (23%). Gallup expects more weight on more emphasis on “reviewing AI-assisted work, applying information in context and working well with others.” Gallup.
Limits. This is OpenAI studying its own product, on scientists who chose a dedicated plan. The speed figures are the scientists’ own estimates, and the report doesn’t claim cause and effect. Gallup’s numbers are self-reports too.
What to watch: whether outside researchers can test the “nearly twice as fast” estimate against measured output.
Read it: OpenAI, Science at the Frontier | Gallup | On X: Ronnie Chatterji on the paper
Who’s Directing? I run my own team of AI agents, so I read this as a question about where the checking happens. If I ran an operations team, I’d mark the steps in each AI-assisted process where a person has to question the output before it moves on, and I’d put that review time on the schedule. That keeps people doing the thinking the work depends on, which is what we mean by Cognitive Coverage.
Now, here’s what’s happening in learning: the instructors designed the AI
The research. A team from 20 institutions, led by Jun Li of the University of Michigan, posted “Instructor-designed AI improves learning in a randomized trial across 19 universities” to arXiv on November 13. The trial was preregistered and ran from spring 2025 to spring 2026 in 55 applied STEM classes (statistics, optimization, analytics, forecasting and similar) with about 7,200 students. Classes were randomly assigned to start their AI right away or one term later. Instructors designed their own tools with technical support from the research team, including Socratic coaching agents (19 classes), practice tutors (18) and role-plays with AI characters (8). arXiv 2611.19099.
What it finds. Learning was measured on assessments where AI wasn’t allowed, usually proctored, in-person, pen-and-paper exams, and compared against the same classes in earlier years. Scores rose 0.32 standard deviations at the student level and 0.26 at the class level. A standard deviation is a common yardstick for test gains; the authors note that the median effect in a study of pre-K to 12 randomized programs was 0.10. Engagement and course ratings showed no detectable change, so the gain was in learning, not in how much students liked the class. One student wrote: “I like how this AI learning tool doesn’t give answers to my hw questions right away.”
How it fits. The authors write that their estimates “contradict the negative estimates found in the other major studies.” Their author list includes Hamsa Bastani and Alp Sungu, names on the 2025 PNAS study of nearly 1,000 Turkish high school students, where a plain chatbot raised practice grades but left students 17% lower than never-AI peers once access was removed. Their explanation is design: earlier studies imposed one tool from the top; this one let instructors build for their own students. Bastani et al., PNAS.
Limits. Applied STEM classes only. Instructors who joined got training and technical help from the research team, which most campuses won’t have. Few classes ran the tools for a second or third term, so it’s unclear whether the gain lasts. It’s a preprint, not yet peer reviewed.
What to watch: whether the gain holds outside STEM and in a course’s second year.
Read it: arXiv 2611.19099 | Bastani et al., PNAS (2025)
Who’s Directing? If I were a community college dean, I’d give two or three faculty members a term of release time and technical help to build AI for one course they know well, then judge it by exams taken without AI, the way this trial did. The researchers credit the gain to instructors who knew their students, so that’s who should design the tool.
Quick hits
Agents break rules they’ve already read. NAVER AI Lab’s Unfinished-Bench (arXiv, November 12) put AI agents in 97 work scenarios across 7 simulated organizations where finishing the task meant breaking a rule. Pushing the agent to finish raised violations of a rule kept in an internal document from 32.2% to 43.6%, five times the 8.5% rate when the user stated the same rule. In 98.3% of those violations, the agent had already been shown the rule. If you hand work to an agent, put the rule in the request. arXiv 2611.17460
Triangulation: signal and noise
Signal. A 19-university randomized trial, OpenAI’s usage data and Gallup’s survey meet at one point: AI helps most where people decide what it does and keep checking what it gives back. That’s Directing Intelligence, and it’s how Cognitive Coverage holds, keeping people doing the thinking the work depends on.
Noise. A preprint in applied STEM, a vendor studying its own product, self-reported time savings.
What I’d do. Mark the steps where a person must question the output. Give faculty time to build AI for their own course and test it without AI.
Until tomorrow, that’s the intelligence. How it’s directed is up to you.
Todd






