Personalization on Demand

I’m working on making the whole machine learning stack something we can spin up for a client in hours. Someone tells us what they want, we build it, and they can try it on their own site. Search and recommendations are a big part of that, but the requests get much more specific once you start talking to people.

A lighting store wants to show similar chandeliers below the one a shopper is looking at. A retailer wants to help people choose a size. Someone selling cooking classes wants to suggest the next class a visitor might enjoy. Each needs different data and different logic, and I want us to be able to build all of them without starting over each time.

That’s what we’re building at Coframe. Our agents already make changes to websites. My part is building the backend and the models that decide what each visitor sees.

1What the shopper sees

Say you arrive at a store from a coffee newsletter and look at two espresso machines. Another person arrives from a lighting ad and looks at a brass pendant. You end up on the same collection page, but you’ve given the store very different reasons for being there.

We train a model on that store’s own visitors and use it to reorder the products for each person. The coffee shopper should find the coffee equipment near the top. The person shopping for lighting should see the lamps. On a phone, where only a few products fit on the screen, that ordering is a large part of the experience.

The model can use what brought someone to the site, what they just searched for, and which products they’ve viewed or put in their cart. As they keep browsing, it has more to work with. We don’t have to wait for someone to become a regular customer before paying attention to what they’re doing.

Search has the same problem. Someone types “pants I can wear on a long flight,” and the store needs to find pants even if none of the product titles say that. We combine word search, including typo tolerance, with retrieval based on meaning. The ranker then uses those results along with the shopper’s recent activity to choose the order.

Fig. 2

A search no title answers

a place to sleep

a
~~place~~
to
sleepin no product

By meaning 10 nearest of 202

Ravenmoor Wood Four-Poster Bedsleep ≈ bed 0.66
Dovemere Carved Wood Day Bedsleep ≈ bed 0.66
Ashstead White Metal Bed Framesleep ≈ bed 0.66
Yunlan Painted Wood Room Divider Screensleep ≈ room 0.59

The made-up catalogue sells home goods, so the flight becomes a night’s sleep. No product’s words include “sleep”; the search finds beds because the two words lie close together in meaning (the number is how close, out of 1). A misspelt word is forgiven, and a word the catalogue uses is simply matched. The whole search runs on the front page, where you can type anything.

I like these examples because the request is easy to understand. You can look at the page and have an opinion about whether it makes sense. Then you can test it.

2What we have to build behind it

A client asking for recommendations also needs us to find their catalog, keep the prices current, collect the events, train the model, and serve the answer quickly. All of that has to keep working after the first version ships.

Our backend agent starts by writing down what the client wants, where it will appear, and how we’ll judge it. We also give it a simple baseline to beat. For recommendations, that might be best sellers or products people often view together. If a simple rule works better, I want to know that before we put the model in front of shoppers.

Then the agent looks for the data. It starts with the store’s own API and works through the other available sources. It writes the jobs that keep the catalog and prices fresh, prepares the training data, and checks the recipe on a small sample.

That sample check came from losing hours to training jobs that failed near the end. Sometimes the data was in the wrong shape. Sometimes a file was too large for the scoring step. Now the run has to demonstrate that its steps work on a small sample within ten minutes before we pay for the full training job.

Once the model passes its evaluation, we build a preview on the client’s page and check that it actually displays the model’s choices. The person responsible for the launch decides when the experiment goes live.

Client request leads to data and model preparation, a sample check and training, evaluation and preview, human approval, a live A/B test, and retraining. Fresh models return through evaluation and preview. — The model reaches a preview before launch. A person approves the live test, and fresh data feeds subsequent training.

A sketch of the idea, not our production checks. Choose what is wrong with the data, switch the sample check on or off, and run it. The check here really runs: twelve of the made-up catalogue’s rows go through every step that follows the training (read the rows, build each one’s inputs, score them, size the scoring file), which takes a moment in a browser; the rule in the essay is ten minutes. With the check off, the same fault waits until the training is done. Nothing trains on this page: the 80 trees were trained ahead of time on simulated shoppers. The two bars are the essay’s own numbers, drawn to scale.

Our best recent internal drill took 9.5 hours from the request to a serving model. The goal is three hours. Agents already handle much of the work, and we’re still connecting the remaining steps so the whole process is reliable.

3Teaching the model what matters

Someone clicking on a product is useful information, but it doesn’t necessarily mean they want to buy it. We give more weight to an addition to the cart, and more again to a purchase. Where we have the item prices, we weight purchases by revenue, with a cap so one unusually large order doesn’t dominate the training data.

Fig. 4

What a label weighs

Click1

Add to cart5

Purchase20

Cap on one order$500

click < cart < purchasekept

1
Steamwick Metal Gooseneck Kettle547 clicks · 110 carts · 62 orders · $3,479
2,489
2
Cornwick Burl Wood Bowl646 clicks · 97 carts · 50 orders · $3,245
2,429
3
Rivetson Metal Pipe Table Lamp451 clicks · 73 carts · 35 orders · $3,471
2,204
4
Duskmere Brass Lantern Chandelier323 clicks · 23 carts · 10 orders · $3,490
1,834
5
Saltwick Brass Nautical Compass848 clicks · 39 carts · 9 orders · $531
1,255
6
Haybrook White Painted Wood Dining Chair234 clicks · 23 carts · 9 orders · $7,152 one order of $5,960
1,026

A toy with made-up events for six products; the weights are not ours, and they are yours to move. Each bar is what a product’s events weigh in the training data: its clicks, its additions to a cart and its purchases. Weighted by revenue, a purchase counts its weight once for every $50 of the order, up to the cap. One product has a single order of forty chairs. Take the cap away and that one order puts it first; make a click weigh as much as a purchase and the most-clicked product wins.

We also have to make sure the model sees the information we already have. A few of our early models missed useful signals, including browsing history and the campaign that brought the visitor in. We made those inputs part of the standard setup.

The most recent actions come from a small journal in the shopper’s browser. It keeps their recent product interactions and searches and sends them with the request. Longer-term history comes from our feature store. That means the model can see what someone looked at a moment ago while also using information from earlier visits when it’s available.

Fig. 5

What the model is sent

Journal in the browser

Feature storeNothing

1Hearthby Brass Pan with Wood Handle
2Hearthby Brass Cooking Pot with Lid
3Hearthby Brass Pot with Wood Handles
4Duskmere Brass Lantern Chandelier

The journal here is the first visitor’s from Fig. 1, so whatever you pressed there is in it (in this tab’s memory; this page stores nothing). The earlier visits are made up: an oil lamp and a lantern, some weeks ago. The four products are the top of the same collection, ordered by the same trees, for what was sent.

The catalog matters just as much. For a lighting store, shape and finish can make two products good alternatives. For clothing, the useful attributes are different. We use product embeddings alongside the store’s own attributes, so the model has a way to compare products beyond their names.

There are basic things we still have to get right around all of this. An item that’s sold out shouldn’t make the recommendation. A price needs to match the store’s source. We’re working through the remaining gaps in those checks as we move surfaces onto the new system. They matter directly to the person trying to buy something.

4Getting the answer there in time

We’ve had tests where the model was running, but shoppers never saw its recommendations because the first request took too long. The page had already fallen back to its original order. That changed how we check a launch: we open a fresh visit and verify that the model’s choices actually appear.

The serving engine is written in Rust and runs on Cloudflare Workers. Training happens separately on Google Vertex AI. For models we can serve through precomputed tables, much of the expensive work is done before the shopper asks for anything. The request then needs a lookup and a sort.

We also moved shopper history closer to the traffic using it. Our first store of that history was in the US, and requests from Europe often timed out. It now lives in Cloudflare Durable Objects, with a separate object for each shopper.

There’s still work to do on latency, especially just after a new model ships and its tables are cold. We put deadlines on lookups so a slow history request doesn’t hold up the page. If our answer is missing or too late, the store can show its own ordering.

Fig. 6

The answer against the page

Simulation

Request36
History24
Model522
Answer37

ServerAnswered at 619 ms

Shopper sawThe store’s own order

The model answered. Nobody saw it.

A simulation. The milliseconds are made up to show the shape of the problem and measure nothing of ours. What is real is the two orders the shopper can end up seeing: the model’s, from the same trees as Fig. 1, and the store’s own best sellers. As it first stands, the server did its work and answered, and the page had already painted without it.

5Finding out whether it helps

Before launch, we evaluate on shoppers held out of training and compare the model with both the store’s current ordering and a simple rule. That catches a lot of mistakes, but those purchases happened under the old experience. An offline score can’t tell us what the same shoppers would have done with a different page.

So we run the live version as an A/B test against what the client already has, including their current vendor where applicable. We measure revenue per visitor. That’s the part that tells us whether changing the experience helped the business.

It also takes longer than building the first version. We can generate ideas and ship code much faster than shoppers can give us enough evidence to judge them. Some of the estimates look encouraging, but we still have to let the experiment answer the question. A positive number with a wide interval doesn’t settle it.

Fig. 7

A positive number, and its interval

Simulation

Visitors in each arm: 200,000
A, per visitor: $14.42
B, per visitor: $14.71
B − A: +$0.29 between −$0.49 and +$1.06

Not settled

A simulation with made-up visitors to a made-up store; none of these numbers is a result of ours or of anyone’s. Each visitor buys one product with a small chance and pays its price. The line is the difference in revenue per visitor, B minus A, and the band is its 95% interval, narrowing as visitors arrive (left to right, from 200 in each arm to 200,000). The dashed line is the truth the simulation was built with. As it first stands the truth is no difference at all, and after 200 visitors B looked nearly eleven dollars a visitor ahead.

I like having that constraint. We have clients with actual problems and enough activity to keep learning from what we build. We can have a strong opinion about an idea, put it in front of people, and find out where we were wrong.

6Building it for the next client too

Each client has its own data, models, and custom logic. We run that logic in a separate sandbox on the same engine, so a new request can use the platform we already have. When we learn something useful, we improve the shared templates and checks. The client’s data stays with that client.

Fig. 8

One engine, a sandbox each

A lighting shop

26 products of its own

Its logicSame category

Hawkmoor Brass Hurricane Lantern, Small

A kitchen shop

16 products of its own

Its logicNo dearer than this one

Cornwick Hardwood Cutting Board, Large

A furniture shop

81 products of its own

Its logicAnother category

Kelsby Wood Dining Table, 72-Inch

One engine

Templates
Serving
In stock
One per family

A sketch. Three pretend shops are three departments of the made-up catalogue. Each asks the same question, what to show below this product, of the same shared function, with its own products and its own rule. Ship the check and every shop gets it at once: two sizes of one lantern stop appearing together. Ask across a wall and there is nothing to be had: a sandbox holds only its own shop’s rows.

That lets the work extend beyond product grids. We already use the engine for things like choosing relevant reviews for a service page and recommending cooking classes. I’m interested in what happens when a shopper answers a quiz or asks a question, and the rest of the site can use what they just told us.

A small team and a lot of agents are building this. The agents write code, run training, and review each other’s work. We set the goals, check the results, and decide what goes live. I haven’t written a single PRD or ERD since joining. We talk about what a client needs, make something they can try, and keep working on it as we get feedback.

It’s pretty incredible how much we can build this way. I think about the work it would take for each client to put together and maintain all of this themselves. I want them to be able to ask for the experience they have in mind and try it while the conversation is still fresh. That’s the part I’m excited to keep working on.

We’re hiring at Coframe. If you’d enjoy building this with us, take a look at the open roles.