Foundry DST is a complicated AI tool with a simple goal: help people, companies and campaigns make better decisions, specifically in Oklahoma.
Hart Brown, president of artificial intelligence and transformation at Saxum, an Oklahoma City-based consulting firm, said Foundry DST stands for "decision support tool." It uses a proprietary dataset to predict how Oklahomans in a certain county might react to a product or ad.
"In effect, we had to build the brains of Oklahomans around the state," he said.
And lately, this "brain" has shown promise somewhere else predictions come in handy: politics.
"So you have a new piece of legislation or an executive order, or in this case, elections, you actually can put this into the system and see how receptive Oklahomans are to whatever that may be, to include candidates," Brown said.
In August, the AI tool predicted 50.7% of Republicans would vote for businessman Mike Mazzei as their gubernatorial nominee for the November General Election, leaving 49.3% to vote for Attorney General Gentner Drummond. The confidence for both calls was over 80%.
State-certified results show 50.28% of those voters cast their ballots for Mazzei, while 49.72% did so for Drummond. Foundry DST's prediction was off by less than half a percent.
On the face, that level of accuracy borders on the uncanny. But the tool is nascent and hasn't always been right, which raises questions for computer science and polling experts skeptical about what the tool does versus how it's marketed.
Additionally, a recent Pew Research study shows such AI prediction tools are still not as reliable as human surveys when it comes to predicting human behavior, especially for certain subgroups including Republicans, Democrats and several racial demographics.
What do the latest polls say about Oklahoma's headlining November races?
Mazzei won the Republican runoff in August by a tight margin, so he's poised to face Democrat Cyndi Munson and Independents Orlando Lynn Bush, Robert Brooks and Jerry Griffin in the general election.
Mazzei is the frontrunner as Oklahoma hasn't elected a Democrat to statewide office in 20 years.
Polling has been limited. Still, prediction markets Kalshi and Polymarket have Mazzei at a more than 90% chance of winning.
Brown said Saxum can use its AI tools to simulate a poll without the need for actual respondents, and has done so several times since the runoffs.
As of Oct. 9, Foundry DST had Mazzei winning the governorship statewide in November with a 17-point lead at 79% confidence.
KOSU also asked Saxum to run the standings for the frontrunners in the race for Oklahoma's 5th Congressional District: former teacher of the year and Democrat Jena Nelson and incumbent Republican Rep. Stephanie Bice.
The AI gives Bice a near-20-point lead with 81% confidence across the six counties that make up the 5th District. Those are Lincoln, Pottawatomie, Seminole, Oklahoma, Canadian and Logan counties.
Brown also prompted a prediction of the governor's race within the 5th Congressional District. The results have Munson leading over Mazzei in CD5 by 9.2 points at 81% confidence.
The AI tool's prediction comes with some caveats. It assumes all votes go to either Bice or Nelson; in real life, voters could choose either of two independent candidates or write in a vote. And the prediction is based on turnout for the 2022 gubernatorial race.
Before Saxum's AI tool was eerily close in predicting the Republican gubernatorial runoffs in August, it was wrong about the first round of primaries in June, according to a report by The Oklahoman.
Foundry DST had Mazzei and candidate Chip Keating neck-and-neck in June, with 22% of simulated voter approval for each. It had Attorney General Gentner Drummond in third place at nearly 19%.
In reality, Republican voters chose Mazzei and Drummond as their top contenders, with Drummond leading by a small margin.
Brown said the AI tool was tweaked after the failed June prediction, which resulted in a better August prediction and, he says, should lead to better predictions heading into November.
How is Saxum's AI prediction tool different from traditional polling?
Pat McFerron is a longtime Oklahoma pollster and president of the consulting firm Cole Hargrave Snodgrass & Associates, which regularly predicts Oklahoma elections using surveys and polls.
He said understanding how Foundry DST can make such close predictions requires understanding the data it uses and how it treats that data. He said that isn't uncommon in market research, but it raises questions about what people see when the AI spits out an answer in response to a prompt.
"This sounds like it's a new iteration of modeling which has been going on for quite some time and it's something that we've used as part of our efforts as well," McFerron said. "But it always seems to have its limitations because individuals are individuals."
The AI tool runs on weighted probability functions using the data Saxum is inputting, and it's unlikely the system accounts for human unpredictability. McFerron said part of the reason is that humans produce too many data points for AI to consider.
Bill Shapard runs Sooner Poll. He said that dilemma is why pollsters have turned population sampling into a science, with established ways to categorize unwieldy data.
"Market research falls into two categories," Shapard said. "There's primary market research and secondary market research."
Primary research means creating your own data, identifying a large enough random sample of real people and asking them a question directly to determine the likelihood of something; the more people you ask, the more likely you are to predict what they might do as a group.
The practice directly implements the large number theorem, a fundamental rule in statistical research that Shapard said is essential to the kind of political polling he and others conduct.
"Secondary research is, like, a firm who works for McDonald's, and McDonald's comes to them and says, here is the demographic profile of the people that are most likely to eat at a McDonald's," he said. "Now go find that in Oklahoma City and tell us where our next location should be."
It matters whether the data is created, Shapard said, because if it's not, an AI can only ever make a highly educated guess as to what a prompter asks based on the pre-existing data it has. AI agents that work that way are known as Large Language Models, or LLMs, and aren't different from ChatGPT, Claude and other leading models designed by AI-devoted companies.
"The AI is just a souped-up analysis with the ability to interact back as a human," Shapard said. "But it does not replace collecting data and making sure the data is a probability sample of the population of interest. And no AI is going to be able to solve that problem."
But Brown said that while the experts' analogy is a good one, Saxum's AI tool is unique for a couple key reasons: because the data is tailored to Oklahoma and because it takes the extra steps to simulate those random population samples traditional polls rely on.
How Saxum says Foundry DST works
Brown said it's important to know the tool is actually using three different AI models running different sets of data against each other. He said it starts with what he called the "Oklahoma Factbook," a set of 15,000 public domain data points including information about all 77 counties.
"It is demographic data," Brown said. "It's economic data. It's coming from places like the Federal Reserve Bank. It's coming from places like the Bureau of Labor Statistics. It's…statistics that are coming from the state."
That's coupled with near-real-time updates of live news and social media trends, the weather, and fluctuations in gas prices, among other items, to create the living context the tool needs to make a decision, Brown said.
But that alone still doesn't really help it make a prediction, he said.
There is one more crucial part that brings the "Brain of Oklahomans" to life: Psychographics and psychometrics, which refer to market research studies meant to determine human behavior based on things like our attitudes, fears and values.
"That is…the component that takes all of this information, along with what we know about the communities themselves and the people within those communities, to actually make a statement around sentiment and ultimately intent," Brown said.
Consulting firms already use psychographic studies to analyze consumer behavior. But Brown said Saxum is inputting those studies in a way that allows the AIs to simulate human behavior before it happens.
"Once we build the context around that brain and give it all of the data around the state, we can then ask it a question that maybe we've never asked anybody in the state before," he said.
The questions come in phases, intending to eventually get to the heart of what someone may really want to know. For example, you might wonder how likely a 20-year-old male, registered as an independent in Logan County, is to vote for Republican Mike Mazzei over Democrat Cyndi Munson for governor in the upcoming election.
"So what we end up doing is asking, in this case … 'are you willing to go and vote?' And that's one part of the sentiment analysis," Brown said. "The second part of that is, 'if you're willing to go and vote, are you willing to go and vote for this specific candidate?' And when we ask that question, we have to pull in all of the dynamic data for that candidate for the system to be able to make that assessment and then we populate that on a map."
The map shows the probability of such a person voting for Mazzei in every county, including Logan. The charts above are big picture summaries of what the maps show for each county.
Also included in the response is a 15-page report of the combined analysis of the AIs, detailing how they arrived at their prediction. This part, Brown says, is reviewed by a team of human strategists at Saxum with expertise in areas like AI modeling, public policy and market research. It's also what clients see and pay the company to interpret for them.
"Our strategists step in and are now able to authenticate or help to validate or confirm that what the system is saying is generally what we know and believe based on our experience on human behavior as well," he said. "And that is to protect against this idea that a system can predict everybody, which it cannot."
Brown said the tool shouldn't be considered a replacement for traditional polling.
"It is not a 1-to-1 comparison," Brown said. "There's value potentially in both. The value that we see, certainly, in the work that we're doing here is we can get results in minutes. We can track things day to day. If something shifts in the news overnight, we can actually test that the next morning and see what that looks like."
'Be like Jesus…'
Steven Skiena is a computer scientist and AI modeling professor at Stony Brook University in New York. He said the great thing about this kind of AI agent is you can ask them anything, and they will give you a response. It's also one of their biggest weaknesses.
He used the example about the hypothetical voter in Logan County to explain:
"That particular question…is a pretty hard question to verify," Skiena said. "You have to go to a sample, a bunch of people in Logan County, and you've got to ask them."But if you replace Logan County with something like a few hundred – or thousand – data points about Logan County, he said, like the number of people under 21, their jobs and income, and their habits and values, then arriving at an accurate and confident answer could be plausible.
But giving an answer is not the same as knowing it's right, he said. And trusting the tool, in that way, requires trusting the company and the people selling it.
"There's a saying among modelers that, 'You should be like Jesus and make sure your miracles are well publicized,'" Skiena said.
"There is always someone who has something, and it's hidden behind a fence, and they claim they get great predictions and they have voodoo that they won't tell you what it is. There's always reason to be somewhat skeptical."
Brown chuckled when asked if he'd heard the adage before, saying he thinks it points to a possible change in the mindset about how market research is conducted going forward.
"In most cases, we've designed campaigns, and then we put information out, and then we start learning what the reaction of people is," Brown said. "And so turning that around and using more and more data in the beginning is a bit of a shift. It is something that we have to talk to, and work with, all of our clients on."
To that point, Skiena said, Saxum's tool, like any AI tool that's sold, is a product, and whether it's a good one depends on proven results and customer satisfaction.
And that track record remains to be seen.