Experts are training AI systems on the work they know best

Experts training AI jobs are putting doctors, lawyers, writers, engineers and other specialists in front of chatbots to judge their work. The assignment is not to hand over an entire profession. It is to turn pieces of professional knowledge into ratings, examples and tests that developers say can guide an AI system’s future answers.

Mercor, a company that recruits contractors for AI projects, lists work across fields including software engineering, finance, medicine, law and cybersecurity. Its site advertises an average contracted rate of $108 an hour, a company figure, while visible listings show different rates by role.

Other platforms are pitching similar gigs. Braintrust says its projects can involve evaluating answers, labeling data, writing prompts and benchmarks, checking code, and probing systems for weaknesses. The platform advertises pay ranging from $20 to $200 an hour, which is marketing information rather than a measure of typical earnings.

What do experts training AI jobs involve?

The work often relies on reinforcement learning from human feedback, or RLHF. In plain terms, people compare or score a model’s responses, and those judgments are used in training intended to steer later responses. AWS describes the process as adding human feedback to the system that rewards certain model behavior.

CBS News reported that screenwriter Robin Palmer has assessed chatbot writing for elements such as structure, characterization and transitions through work with Mercor. It also reported that an anesthesiologist, Mike Prokop, saw expert review as a way to flag unsupported AI answers, a view he expressed to the network.

In medicine, CNN reported that emergency physician Alice Chiao draws on scenarios from her career and steps in when a model response looks misleading, alarmist or unsafe. CNN said Mercor’s experts use rubrics developed after consultation with other specialists, then feed the evaluations into further training.

That process is meant to improve systems on particular tasks. It does not show that a model can perform a whole profession, or substitute for the judgment and accountability that come with it.

Does teaching AI put experts out of work?

The evidence points to a live argument, not a settled answer. Mercor CEO Brendan Foody told CBS that AI could take repetitive work off people’s plates, leaving more engaging work. Workers interviewed by CBS, including Palmer and Prokop, said they saw reasons for qualified people to help shape tools already entering their fields.

At the Columbus law firm Vorys, Sater, Seymour and Pease, CBS reported that AI personas were built from the experience and philosophy of senior lawyers. Partner Kim Herlihy said the tools can offer perspectives while lawyers prepare, but said AI would not walk into a courtroom to deliver an opening statement and that clients want a lawyer’s advice.

Others see a harder trade-off. Economist Daron Acemoglu told CBS that AI-driven automation could displace workers and that training roles would be small compared with jobs lost, an assessment rather than a forecast established by the reporting. Clara Shih told CBS that AI agents had enabled smaller teams to do product-development work she said once required dozens of people.

Even expert judgment has limits. A 2026 study presented at ACM FAccT found three psychiatrists reviewing AI mental-health responses frequently disagreed, especially on safety-critical material involving suicide and self-harm. The researchers said the differences reflected competing clinical approaches, warning that averaging expert scores can blur meaningful disagreement.

For now, the practical divide is between AI assistance on bounded tasks and replacing the professional judgment behind them. The people training the systems are helping define that boundary, even as they debate where it should sit.

This story draws on original reporting from CBS News.