The problem
A large consumer brand's Facebook, Instagram and LinkedIn pages receive hundreds to thousands of comments a day. Most are the same fifteen questions: is this in stock, why is my bill wrong, where is the nearest branch, does this offer still apply.
Almost none of them get answered. One industry study puts the share of brand social comments that receive no reply at 97%. Every unanswered comment is a customer who asked in public and was ignored in public.
Two things make this hard to automate in this region specifically. The volume is real, and the language is not standard. People comment in Algerian, Moroccan or Gulf dialect, in French, or in a mix of both inside one sentence. Generic multilingual models handle formal Arabic reasonably and local dialect badly.
What we did
We worked with the product team building this capability into their existing platform. Our scope was the part that decides whether a system like this can safely go live:
- System improvement. Reworking how the model produces replies in local Arabic and French, so the output reads like a person from that market rather than a translation.
- Evaluation design. Building the framework that judges whether a generated reply is actually usable — the thing most teams skip, and the reason most of these systems quietly underperform after launch.
- Monitoring. Continuous measurement in production, so quality drift is visible rather than discovered through complaints.
Results
Human reviewers scored a sample of 125 generated replies, comparing the first version of the system against the current one. Same sample, same reviewers, same criteria.
- •Three quarters of generated replies now go out as written. The share needing a human rewrite fell by roughly two thirds.
- •The number that matters operationally is the first row. At a few hundred comments a day, moving usable-as-written from half to three quarters is the difference between a team reviewing everything and a team reviewing exceptions.
| Reviewer verdict | First version | Current version | Change |
|---|---|---|---|
| Usable as written | 53% | 76% | +23 pts |
| Usable after editing | 20% | 7% | -13 pts |
| Not usable | 27% | 17% | -10 pts |
What we learned
The model was never the hard part. The hard part was defining what a good reply is in a dialect, agreeing that definition with the client, and building something that measures it continuously. Evaluation was the deliverable.