Someone at Fireworks changed how the team’s AI requests were routed. According to Benny Chen, his colleagues kept working without noticing the change.
That small incident inspired me to write this after our conversation on Inside the Silicon Mind. If the model underneath a product can change without the person using it noticing, what gives that person a reason to keep choosing your company?
Benny, co-founder of Fireworks AI, has spent years working on the infrastructure that makes models useful. When I asked where Fireworks had found its strongest product-market fit, his answer was two words:
“Specialized intelligence.”
He described helping companies customize open models for their work, run them at a competitive cost, and build a process for improving them with their own data.
What interested me was the continuity in that process. A better base model can arrive. The company still has what it has learned about its customers and the work they need done.
Start with demand you can already see
Before Fireworks began in September 2022, Benny and his co-founders had seen GPU demand growing inside large technology companies. Recommendation systems, driving simulations, and other applications were placing new demands on infrastructure.
They could see the pressure before they knew exactly which application would define the next wave.
The team initially explored recommendation systems. Then ChatGPT arrived, and Benny described it as a hard reset for the industry. They tried different ideas while holding onto two convictions: open models would improve, and their experience running models could help other companies.
That experience included operating production services. As Benny put it, “We know how to hold the pager.”
There is a useful distinction here for founders. You can have strong evidence that a problem is growing while still needing to discover the product customers will buy.
The work is to keep those two judgments separate. Which part of your thesis comes from repeated demand you have observed? Which part is still a proposal about how to serve it?
Fireworks’ early direction changed. The team’s understanding of the infrastructure problem gave it something concrete to work from as it made those decisions.
Define the job your customer needs the model to do
In our August conversation, Benny described work with companies including Cursor and Harvey, helping adapt models to coding and legal tasks. His explanation of Fireworks’ fit centered on making those models useful for a particular business and economical to run.
A model’s broad capabilities only tell a customer so much. The buying decision depends on whether it can do the work that matters to them, under the conditions in which they need to use it.
The routing story brings that into focus. Benny said colleagues often could not tell the difference between models during ordinary use. That was his observation inside Fireworks, rather than a claim that every model performs equally on every task.
For a founder, I would turn it into a practical question: where does your customer still notice the product falling short?
Find the task they have to redo, the answer they must check, or the step that makes the workflow too expensive. Ask them to show you recent examples. Those examples give you a much clearer target than a general request for better AI.
You also need to understand what a successful result would change for them. Saving a few seconds on an occasional task may matter less than reliably completing the one job that keeps bringing them back.
Make customer feedback part of the product
Benny kept returning to what he called the data flywheel: helping companies improve their models over time and change the base model as better options appear.
The phrase can sound automatic. In practice, somebody has to decide what counts as useful feedback and make sure it leads to a better product.
My takeaway is that a founder should be able to follow one customer correction through the business. Where is it captured? Who decides what it means? How does it change the next version? How do you check whether the change helped?
Consider a customer who repeatedly rewrites the same part of an AI-generated answer. That could point to missing context, a poorly defined task, or a capability the model does not yet handle. Storing the interaction alone will not tell you which one it is.
You have to understand the correction, use data you have permission to use, and test the resulting change against examples that matter to the customer.
This also gives you a way to judge model upgrades. Keep a set of real tasks and known failures. When you try a new model, check whether it improves those outcomes and what it costs to deliver them.
Over time, that record becomes valuable knowledge about the work your customers need done. It can guide product decisions even when the technology underneath changes.
Keep the ability to change
Benny is concerned about businesses whose continued operation depends on access to a single model provider. Fireworks’ mission, as he described it, is to help companies become more self-sufficient through models they can customize and control.
For an early company, the practical question is how much freedom you need to keep delivering the customer’s result.
A hosted model can be a sensible way to start. As the product becomes part of a customer’s daily work, it is worth understanding what you would need to change providers, adjust deployment, or respond to different costs.
That does not require building every layer yourself. It does require knowing which parts of your product depend on a particular supplier, and whether your evaluation process and accumulated customer knowledge can move with you.
The more clearly you understand the customer’s task, the better placed you are to make that decision. You can judge a replacement against the result you promised.
Trace one improvement all the way through
Before your next product meeting, choose one recurring customer task. Bring examples of where the current product succeeds and where someone has to correct it.
Pick one failure to work on. Agree on the improvement the customer should notice, make the change, and return to those examples to check it. Then see whether the customer uses the improved workflow again.
A feedback process does not establish product-market fit by itself. You still need customers who choose the product, keep using it, and pay for the value it delivers. But it gives each customer relationship a chance to make the next version more useful.
That is what I took from Benny’s account of Fireworks. The models will keep changing. A company that understands its customers’ work, and turns that understanding into repeated improvement, has something it can carry forward.
What did your product learn from its customers this week, and where will they feel the difference?