# CASE STUDY REFERENCE PACK

# GUIDANCE

The format for these is a case study, with 6 parts: What happened, Who was involved, Where things went wrong, Why it mattered, The lesson, What could have been done differently

These short summaries are provided so you can make light, contextual references during the simulation. Do not present them in full to the participant unless needed. Use them only to inspire brief, humorous nods (“smooth flying…”, “some lawyers wish they’d checked…”, etc.).

## 1. Air Canada’s Chatbot Gave Wrong Policy Advice

### What happened

Jake Moffatt used Air Canada’s website chatbot to understand bereavement fare rules after his grandmother died. The bot told him he could apply for a refund within 90 days after completing travel.

### Who was involved

Air Canada; British Columbia Civil Resolution Tribunal; passenger Jake Moffatt.

### Where things went wrong

The advice was incorrect. Air Canada refused the refund and claimed the chatbot was a “separate legal entity,” not their responsibility.

### Why it mattered

The tribunal ruled that the airline is responsible for information on its own site, bot included. Air Canada was ordered to pay about $700 CAD. The reputational damage far exceeded the cost of the refund.

### The lesson

If an organisation publishes AI-generated information, it owns the outcome. You can’t outsource accountability to a tool.

### What could have been done differently

Air Canada likely saved millions in service costs through automation. A better stance would have been: “Most of the time this tool helps us serve customers faster; occasionally it will get things wrong, and we’ll fix those mistakes.” Practical, honest, and far less damaging.

## 2. Chevrolet Watsonville’s Chatbot Agreed to Sell a Car for $1

### What happened

A Chevy dealership used a ChatGPT-powered chatbot on its website. Users quickly discovered it would answer anything — from writing code to recommending a Tesla. One user tricked it into agreeing to sell a 2024 Chevy Tahoe for $1.

### Who was involved

Chevrolet of Watsonville; GM; customers experimenting with the bot.

### Where things went wrong

The bot had no guardrails. It wasn’t restricted to the dealership’s domain, and it responded to prompts as if any answer were acceptable.

### Why it mattered

Screenshots went viral. The dealership effectively gave away a free ChatGPT interface, and GM had to respond publicly. The bot was removed.

### The lesson

If you deploy AI in customer-facing services, constrain it. Otherwise, the internet will stress test it for you.

### What could have been done differently

Internal chatbots for staff can be flexible. Public chatbots cannot. The dealership moved too far, too fast. A narrow-scope bot limited to actual dealership tasks would have stayed out of trouble.

## 3. California Lawyer Fined $10,000 for Submitting Fake AI-Generated Citations

### What happened

Attorney Amir Mostafavi filed an appeal full of fabricated case citations generated by ChatGPT — 21 of the 23 quotes were fake.

### Who was involved

California’s 2nd District Court of Appeal; attorney Mostafavi.

### Where things went wrong

He didn’t verify any of the AI-generated material before submitting it. Courts treated it as a serious breach of professional responsibility.

### Why it mattered

It led to the largest AI-related fine issued in California to date. Courts across multiple countries are now encountering similar issues, and incidents are increasing rapidly.

### The lesson

AI can draft, but humans must verify. Especially where professional judgement carries legal or organisational weight.

### What could have been done differently

If a junior law intern handed over citations, you’d check them. AI deserves the same treatment. The mistake wasn’t using AI; it was assuming it couldn’t be wrong.

## 4. CarCover and the Privacy Breach Triggered by an Employee Using ChatGPT

### What happened

A CarCover employee uploaded a customer’s financial hardship application — including health and family details — into ChatGPT to generate a summary.

### Who was involved

CarCover (fictional case based on real OAIC notifications); affected customer; OAIC.

### Where things went wrong

Sensitive personal information was put into a public AI tool against policy. The summary produced by the tool downplayed key risk factors, contributing to the customer’s application being wrongly refused.

### Why it mattered

It caused emotional and financial harm to the customer, created a data breach, and raised regulatory consequences.

### The lesson

Never put personal or sensitive information into a public AI tool. Even strong policies fail if staff aren’t trained and supported to follow them.

### What could have been done differently

People either didn’t know or didn’t care — and ambiguity creates risk. Clear training, simple decision checks, and technical blocks on uploading personal data would have changed the outcome.

## 5. Massive Spike in AI-Related Academic Misconduct at WA Universities

### What happened

Murdoch University and Edith Cowan University reported huge increases in academic misconduct involving AI misuse — including unacknowledged AI writing and fabricated references.

### Who was involved

Students at ECU and Murdoch; academic integrity offices; academic researchers such as Guy Curtis.

### Where things went wrong

Students used AI without disclosure or copied AI-generated content directly into assessments. Many cases involved fake references created by AI.

### Why it mattered

Misconduct cases more than doubled. Universities had to overhaul assessment design, introduce more supervised tasks, and strengthen security.

### The lesson

AI doesn’t increase dishonesty by itself — but it makes sloppy or naïve misuse more visible. Assessment design must adapt, and staff must understand AI’s limits.

### What could have been done differently

If AI is a black box that can’t explain its reasoning, humans must double-check key outputs — especially in official processes like misconduct decisions, where evidence, not hunches, counts.

## 6. Victorian Child Protection Worker Entered Sensitive Case Data into ChatGPT

### What happened

A Child Protection worker used ChatGPT to help draft a Protection Application Report for court.

### Who was involved

Department of Families, Fairness and Housing (DFFH); Office of the Victorian Information Commissioner.

### Where things went wrong

They entered names and delicate case details into a public AI system. The generated text included inaccurate information that softened the assessed risks to the child.

### Why it mattered

Personal data was disclosed overseas, and a court document was affected by inaccurate AI-generated content. DFFH was required to block such tools for child protection workers.

### The lesson

In high-stakes environments, public AI tools are off-limits. Accuracy and privacy risks are too great.

### What could have been done differently

A clear rule: public AI tools and sensitive casework don’t mix. Approved, secure, internal tools — with training — would have avoided both the privacy breach and the content errors.

## 7. Australia’s National Security Chief Used AI to Draft Speeches — and the FOI Logs Were Obtainable

### What happened

Hamish Hansford, head of national security in Home Affairs, used Microsoft Copilot to draft speeches and internal communications. FOI requests later revealed the prompts and outputs.

### Who was involved

Department of Home Affairs; Hansford; journalists using FOI.

### Where things went wrong

Nothing unlawful occurred — but the existence of logs revealed how much of his communication had been AI-assisted. Sensitive themes, references, and early drafts became visible to the public through FOI.

### Why it mattered

It exposed a new transparency risk: AI usage logs themselves can become public documents. It also raised questions about over-reliance and originality in high-profile communication.

### The lesson

Assume anything generated or drafted with AI may be discoverable. Use organisationally approved tools with appropriate safeguards.

### What could have been done differently

Treat AI tools like email: assume input and output may be stored or discoverable. Use approved systems, and be mindful that drafts can become public.

## 8. NSW Planning Official Used Her Husband’s Unauthorised AI Tool for Development Assessments

### What happened

A senior NSW Housing Delivery Authority official used an unapproved AI system — built by her husband — to assess housing development proposals.

### Who was involved

NSW Housing Delivery Authority; state government; the official and her husband.

### Where things went wrong

The tool wasn’t authorised, monitored, or documented. Hundreds of assessments relied on it without transparency or oversight.

### Why it mattered

All assessments had to be independently reviewed. It raised major administrative law concerns: decisions must be made by authorised people, not outsourced to untested tools.

### The lesson

AI used in decision-making must be properly piloted, approved, monitored, and visible. No hidden tools, no off-record systems.

### What could have been done differently

If AI supports an official decision, the method must be approved, visible, and monitored. A transparent pilot would have allowed the tool to evolve legally and safely.

## 9. Deloitte Used AI in Government Reports — Then Denied It

### What happened

Deloitte produced major government reports containing AI-generated hallucinations, including fake academic references and fabricated quotes. The firm initially denied AI use, then admitted it.

### Who was involved

Deloitte Australia; federal government; academic Dr Christopher Rudge; Deloitte Canada in a similar case.

### Where things went wrong

Staff used generative AI without approval or adequate checking. Fake citations slipped into official documents. The firm failed to disclose AI use to its client.

### Why it mattered

Deloitte agreed to a partial refund, and its reputation took a hit. The Canadian case added pressure: multiple fabricated references discovered in a $1.6M government report.

### The lesson

If AI is used for research or drafting, the output must be verified and disclosed. Tools don’t replace expertise, and denial only worsens the damage.

### What could have been done differently

Transparency from the start would have helped: “We used AI tools to produce early drafts; all content has been checked and verified.” If AI speeds up quality work, everyone benefits. Cutting corners and hiding the method only damages credibility.

##10 Grammarly’s “Expert Review” feature used real experts’ names without permission

### What happened

In August 2025, Grammarly launched an “Expert Review” feature as part of its new AI agents. The tool gave users writing and editing feedback presented as if it came from named experts, including well-known writers, journalists, and academics. Some of those people, such as Stephen King, Neil deGrasse Tyson, and Carl Sagan, had not agreed to take part

### Who was involved

Grammarly, its parent company Superhuman, Grammarly users, and a group of writers, journalists, and academics whose names and reputations were used by the feature were all involved.

### Where things went wrong

The feature blurred the line between AI-generated feedback and real human expertise. By attaching real names and credentials to AI output without consent, Grammarly created the impression that these experts had personally reviewed the writing.

### Why it mattered

This raised serious issues around consent, false endorsement, and commercial use of identity. It also damaged trust, because users could reasonably believe the feedback came from a real person rather than a system imitating their voice or authority.

### The lesson

AI systems should not present synthetic output as if it came from a real named person unless that person has clearly agreed. When organisations use AI to simulate expertise, transparency and consent are not optional extras, they are core safeguards.

### What could have been done differently

Grammarly could have designed the feature around clearly labelled fictional personas, generic expert categories, or opted-in contributors with explicit permission and control over how their name and style were used. It also could have tested the feature more carefully for legal, ethical, and reputational risk before launch.