AI over care notes: what it can honestly do, and where it must stop

This is for coordinators and quality leads weighing up AI in a care record — for an NDIS, aged-care or community service — who need to know what the technology actually reads, what it’s allowed to write, and where the privacy obligations sit.

In care software, “AI” usually means one thing: a language model reading text and producing more text. It is not machine learning trained on your outcomes, and it is not predictive analytics. It has no clinical judgement. It cannot diagnose, triage in the medical sense, or decide anything on a participant’s behalf. Anyone selling you an AI that builds your roster, or one that makes clinical decisions, should be able to tell you exactly what the model does mechanically — and in a care record a human decides, always; if they can’t say that plainly, be sceptical.

The safest framing, and the one worth holding a vendor to, is that AI in a care record should be assistive and office-only: it reads what staff have already written, it produces its own separate output, and a human decides what happens next. It should never edit, block or author the actual care documentation — the progress note, the incident report, the support plan — because those are legal and clinical records that a person is accountable for.

The useful version of this technology runs a periodic pass over progress notes — new ones and edited ones, because an edit can change the meaning of what was flagged — and surfaces things a busy coordinator might miss: a mention of a fall, a note that reads like distress, a shift that went sideways. Done properly, each signal should carry:

• a category (health, behavioural, shift issue, client distress, safeguarding, or something else)
• a confidence score, not a certainty
• a verbatim excerpt from the actual note, so you can check it yourself in seconds
• a severity and a record of where it came from

That last point matters more than it sounds. A signal with no traceable excerpt is just a black-box alert you either trust blindly or ignore. A signal you can click through to the exact sentence is something you can act on.

From there the workflow should look like ordinary triage: open, follow-up, acknowledged, cleared, or escalated to a formal incident with the detail already carried across. Nothing should close itself. A well-built system also lets an organisation tune the sensitivity — how far back it looks, which categories it cares about, which note types it scans, how confident it needs to be before it bothers anyone — because what counts as noise for one service is a real flag for another.

A related but separate idea is a documentation quality check: comparing what’s been written against an organisation’s own minimum-note rules, plus a judgement on whether the note actually says enough, not just whether it’s long enough. That’s a genuinely useful nudge for supervision conversations. It is not a performance review tool, and it shouldn’t be treated as one.

First: what it reads. Signal-reading AI of this kind reads progress notes — the text your staff have already written — so ask any vendor directly which record types its model opens, and hold them to the answer.

Second: any org-wide summary or “clients needing attention” view that’s built from your data is built from named records, not de-identified ones. If a rollup tells a manager which participants need attention this week, it is doing that by name. Never accept “de-identified” or “anonymised” as a description of that kind of view — it isn’t, and claiming it is creates a false sense of privacy.

The same logic applies to two newer features worth understanding: voice dictation of notes, and recording a meeting to get a summary.

Dictation should only ever append to whatever a carer has already typed — it shouldn’t replace or silently edit anything — and the record should show that speech was involved, so a reviewer isn’t reading a transcript thinking it’s typed prose. A meeting summary tool should work the same way in spirit: it can separate speakers, pull out decisions and actions with owners, but it should be fully editable, and only a deliberate human click should file it into anyone’s record. Nothing should land in a care file automatically because a microphone was on.

This is the one place to be precise, because it’s where a provider’s own privacy obligations kick in.

Speech-to-text is often not processed on Australian soil — many products send the audio to a transcription service offshore to be converted to text, so ask where (Corella’s is transcribed in the European Union). Do not assume the audio is then thrown away. For the asynchronous transcription APIs these products actually use, the vendor’s documented default is usually to retain both the audio and the transcript — zero-retention tends to be a separate product tier covering live streaming, not the batch path a care app uses. So the sharper question to put to a vendor is not whether the audio is stored, it is whether their software issues an explicit delete once the text is back, and on which paths. Ask about the one where a recording was abandoned part-way, because that is the path that leaks quietly. The text that comes back is what gets stored, and it should sit in an Australian-resident database that belongs to the organisation, not to whichever transcription vendor did the conversion.

Two separate facts, and they should never be merged into one reassuring sentence:

1. The recording leaves Australia, briefly, to be transcribed.
2. The resulting text is stored in Australia, in the provider’s own instance.

“Your notes are stored in Sydney” is true. “Your audio never leaves Australia” is not, and no provider should let a vendor’s marketing collapse the two. Under APP 8, the disclosure obligation attaches to the fact that data crosses a border for processing — not to where it ends up sitting afterwards. That means this is genuinely your compliance responsibility as the provider, not just the software’s.

Because that obligation is yours, it’s worth having ready-made wording rather than drafting it from scratch under time pressure: a clause for your privacy policy, a clause for your service agreements, and a plain sentence a staff member can say out loud before they hit record on a meeting. All three should name the actual region the processing happens in and your organisation by name — generic wording that could apply to anyone is not a real disclosure.

Be wary of anyone claiming otherwise. This category of feature is new across the sector, not proven over years, and no serious vendor should be quoting accuracy figures for it — a hit rate quoted off a handful of notes is marketing, not evidence, because the sample sizes involved are never large enough to mean anything. Treat every signal, every summary and every dictated note as a first draft that a person checks, not a verdict.

Corella runs this model of AI — office-only, note-reading, human-in-the-loop — as switchable modules included in the one price, not a separate AI tier. Care Signals reads your progress notes and gives every signal its verbatim excerpt, severity and provenance, so each one can be checked in seconds. Corella states plainly that dictation and meeting summaries send audio to a transcription step in the European Union before the text lands in your Australian instance, and that the residency detail is two facts rather than one.