Data Availability Audit
Overview
RAG, fine-tuning, and eval harnesses all assume data exists, is legal to use, and is clean enough to work. A data availability audit inspects real samples before you commit to an approach — schemas lie; rows do not.
When to Use
- Before RAG (need retrievable corpus)
- Before fine-tuning (need labeled examples)
- Before promising accuracy from "we have logs somewhere"
- Skip for pure prompt engineering with no external data
Quick Reference
| Approach | Verify |
|---|
| RAG | Corpus size, chunkability, access rights, update cadence |
| Fine-tuning | Label quality, class balance, held-out set |
| Eval harness | Ground truth or calibrated judge exists |
| Tool calling | APIs authenticated and rate limits understood |
Implementation
- Pull 50–100 real records — not ER diagrams.
- Check volume — enough for retrieval recall or training generalization?
- Check quality — duplicates, OCR garbage, wrong language, broken HTML.
- Check PII and licensing — can this train or be shown to a model vendor?
- Check freshness — stale corpus → confident wrong answers.
- Write go/no-go — if data fails, change approach before build.
On This Portfolio
Assistant retrieval only indexes published knowledge — admin draft/archived status is the data gate. Job-fit analytics assumes recruiter feedback exists; without labels, the eval loop cannot close.
Common Mistakes
- Trusting a schema export without reading documents
- Discovering GDPR or contract blocks mid-sprint
- Embedding admin-only drafts into a public RAG index
Related Skills
- RAG vs fine-tuning decision
- Chunking strategy selection
- Instruction-tuning dataset curation