Auditing the notes
your scribe drafts

A note drafted by an AI scribe is still the clinician's record. Registration standards and record-keeping obligations attach to the clinician who signs the note, and they apply with the same force whether it was typed, dictated or drafted by software. That makes the quality of AI-generated notes a clinical governance question, and clinical audit is the established way Australian practices answer governance questions about their own work.

An audit of scribe-drafted notes is really an audit of a system with two parts: the drafts the software produces and the review the clinician performs before signing. A weak draft that is corrected before signing never becomes a problem, so the audit looks at signed notes, because the signed note is the record, and it treats every finding as information about the whole workflow.

There is also a practical payoff. A structured audit turns vague impressions into specific, fixable findings. It gives the practice documented quality improvement evidence for accreditation, and for many clinicians the activity can be recorded against continuing professional development requirements.

Keep the sample small enough to finish. Ten signed notes per clinician per audit cycle is a workable starting point for most practices. A team of five clinicians then reviews fifty notes, which two scorers can work through in an afternoon. A small sample repeated every quarter tells you far more than a large sample audited once, because the repeat cycles show whether your changes worked.

Select notes at random within a few deliberate groups. Include the consult types your practice actually runs: standard consults, longer consults, chronic disease and care plan visits, mental health consults, and telehealth. Weak drafting tends to cluster in particular consult shapes, for example fast multi-problem consults or three-way conversations with a carer present, and a stratified sample will surface that clustering where a purely random one can miss it.

Audit notes that were signed at least a week before the audit, so the content is read cold and nobody is scoring work they can still recite from memory. Where your scribe lets you view the original draft alongside the signed note, include both. The differences between them show what the review step is already catching, which is some of the most useful information the audit produces.

Score every note against the same short list of criteria. The list below covers the failure modes that matter clinically for ambient documentation, and it is deliberately shorter than a full note-quality framework; you are auditing the scribe and the review habit around it, and a broader note-quality audit can run separately.

Attribution deserves particular attention because it is a distinctive failure mode of ambient tools. A transcribing system hears several voices and must decide whose words become clinical content. A note that records a carer's account as the patient's own report, or that renders the clinician's provisional thinking aloud as a settled diagnosis, can be accurate word by word and still misleading as a record.

• Accuracy: everything stated in the note happened in the consult. Look for invented examination findings, embellished histories and statements the patient never made.
• Completeness: significant symptoms, examination findings, decisions, safety-netting advice and follow-up arrangements all made it into the note.
• Attribution: the note distinguishes what the patient reported from what the clinician observed, and identifies statements made by a carer, interpreter or family member as theirs.
• Medication detail: drug names, doses, frequencies, routes and any changes to existing therapy are exactly right. Errors here carry the highest clinical risk, so score this criterion on its own.
• Plan fidelity: the management plan in the note matches what was agreed with the patient, including referrals, investigations and recall intervals.

Resist the urge to build a sophisticated instrument. For each criterion, score each note one of three ways: met, minor issue, or significant issue. Define significant before you start. A significant issue is an error that could plausibly affect care if it went unnoticed, such as a wrong dose, a wrong side, a missed red flag or a plan that contradicts what was agreed. Minor issues are matters of wording, ordering or verbosity that a careful reader would work around.

Have clinicians score one another's notes rather than only their own. Self-review is worth keeping as a habit, but scorers are consistently gentler on their own work, and a colleague reads the note the way its future readers will, without memory of the consult filling in gaps. Before the first cycle, ask every scorer to rate the same two or three notes and compare results, so the group calibrates on what counts as significant.

Record results in a simple register: the date, a de-identified reference to the note, the score for each criterion, and a one-line comment for anything other than met. Keep clinical content out of the register. The findings should be readable by a practice manager without exposing any patient's record, which also keeps the register easy to store and to table at a practice meeting.

Decide before the audit what will trigger action. A sensible default: any significant medication finding is acted on immediately, a criterion that shows minor issues across several notes gets a workflow response, and an isolated minor finding is noted and watched at the next cycle. Setting thresholds in advance keeps the discussion about the system and stops individual results being explained away.

Findings map to three destinations. Patterns in the review habit go back to clinicians as training; if signed notes repeatedly carry attribution slips, teach reviewers to check the history section specifically before signing. Patterns in draft quality go to the vendor as a specific, evidenced report, which is far more likely to produce a fix than a general complaint. Structural findings go into the practice's documentation policy: which consult types use the scribe, what the mandatory checks before signing are, and how corrections to signed notes are handled.

Then close the loop. Re-audit after any change, using the same criteria and the same register, so cycles are comparable. Over time the register becomes dated, documented evidence that the practice measured a new tool, found specific issues, acted on them and confirmed the fix. That is precisely the record you want to hold if the quality of your documentation is ever questioned.