AI Tool Review Checklist
What to check before you adopt an AI tool
A practical framework for evaluating AI tools on privacy, security, transparency, and maturity — before they reach your stack.
How to use this checklist
Follow these five steps in order for a systematic evaluation.
Start with public docs
Read the homepage, Privacy Policy, and Terms of Service before anything else. Missing documentation is itself a signal.
Check privacy & legal basics
Look for data retention, training rights, and subprocessor disclosures. Vague language warrants follow-up.
Inspect repo maturity
Review commit activity, issue response time, and changelog regularity. A dormant repo often means a dormant security process.
Assess data flows
Understand where your data goes and who processes it. Cloud-only tools require more scrutiny for sensitive workloads.
Make your adoption call
Decide: personal use only, team use with safeguards, or not yet. Document what you checked and when you checked it.
The six areas to evaluate
Every AI tool evaluation should cover these six areas before an adoption decision.
Transparency
Can you identify who is accountable for the tool, how to contact them, and what commitments they make publicly?
- Is the team named and identifiable?
- Is there a public security contact?
- Do docs and marketing stay consistent?
- Is the legal entity clear?
Privacy & Data Handling
How is your data — and your users' data — collected, retained, and used?
- Is there a clear Privacy Policy?
- What are the data retention rules?
- Are training rights opt-in or opt-out?
- Are subprocessors listed?
Security Posture
Are defaults safe? Is there evidence the team responds to security issues?
- Is there a vulnerability disclosure process?
- Are permissions scoped minimally by default?
- Are known CVEs documented and resolved?
- How does the team handle incidents?
Product Maturity
Is the codebase actively maintained? Does the tool do what it claims to do?
- Is the GitHub repo actively maintained?
- Are issues resolved in reasonable time?
- Is there a public changelog?
- Are dependencies kept current?
Legal & Commercial Clarity
What do the contracts actually say about your rights, your data, and the vendor's obligations?
- Are IP and licence terms clear?
- Are data processing agreements available?
- Is there a clear deletion or export path?
- Are SLAs or uptime commitments documented?
Operational Fit
Does the deployment model and integration scope match your environment and risk tolerance?
- Does it support local or self-hosted deployment?
- What file, shell, or network access does it request?
- Are auth and access controls available?
- Is it suitable for sensitive or regulated data?
The full checklist
12 items to check before adopting any AI tool. Core items are non-negotiable; Recommended items depend on your use case.
Privacy Policy
Explains how prompts, data, and account information may be collected, used, retained, or shared with third parties.
Terms of Service
Defines IP ownership, acceptable use, and the rights the vendor claims over your inputs and outputs.
Security Contact
A public path to report vulnerabilities signals the team takes security responsibly and has a process.
Named / Identifiable Team
Anonymous teams have no accountable legal entity when things go wrong with your data.
Data Retention Policy
Clarifies how long your data stays in the system, under what conditions, and whether deletion is possible.
Training on User Data
Some tools use your inputs to improve models by default. Opting out may require configuration or a paid plan.
GitHub / Repo Health
Commit frequency, issue response time, and contributor activity indicate whether the tool is actively maintained.
Hosting Model
Cloud vs local vs self-hosted significantly changes your exposure surface and data residency options.
Subprocessors
Third-party services that process your data are often undisclosed — each one is an additional risk surface.
Auth & Access Control
Tools with file, shell, or browser access need clear permission scoping and audit mechanisms.
Permissions Scope & Defaults
Overly broad defaults are a common source of unintentional data exposure in AI agent tools.
Marketing vs Reality
Claims about security, compliance certifications, and capabilities are often aspirational rather than current.
Quick red flags
If you spot any of these, slow down before adopting.
High risk — stop and investigate
- No Privacy Policy anywhere on the site
- No named team, company, or legal entity
- No security contact or vulnerability disclosure path
- Terms that claim broad ownership of your outputs or code
Caution — dig deeper before proceeding
- "May use your data to improve our services" with no opt-out
- File, shell, browser, or credential access with no explanation
- Dormant or barely maintained repository
- No data deletion or export mechanism described
- Subprocessors unlisted or undisclosed
- Marketing claims that outpace actual documentation
- Compliance certifications claimed but not verifiable
Limpid reviews
See this checklist in practice
Every Limpid review applies this framework to a real AI tool — with evidence, scored dimensions, and plain-language recommendations for developers and teams.