Evaluating AI Agents in Fintech: A Compliance and Privacy Checklist

Most AI agent demos skip the questions your compliance team will ask first. Here is a practical checklist for evaluating agents in a fintech before the pilot starts.

Most AI agent demos are built to answer one question: can it do the task? In a fintech, that is the easy question. The hard ones come from compliance, security, and your partner bank, and they usually show up after the pilot has already started.

Where does customer data go? Who approved that message? What happens when the agent is wrong? Can you show an examiner what it did last Tuesday?

If those answers are vague, the pilot stalls. Not because the agent failed, but because nobody designed it to be reviewed. Below is the checklist I want answered before an agent touches a real customer record. It is general guidance, not legal advice. Your counsel and compliance team decide what applies to you.

Start with the job, not the agent

"We want AI agents" is not something a risk committee can review. "We want an agent to draft first replies to card dispute emails, using the dispute record and our policy library, for a person on the team to review and send" is.

Write the job down in one sentence, then answer five things:

• What starts it
• What sources it reads
• What it produces
• Who approves the output
• How you will know it worked

A named job gives compliance something concrete to evaluate. It also keeps the pilot small enough to finish.

1. Data access: what can the agent see?

This is usually the first question, and it should be.

• List every system and field the agent reads. Account numbers, Social Security numbers, transaction history, and credit data belong on that list only if the job truly needs them.
• Use an approved list, not "connect it to everything." An agent that can search the whole data warehouse will eventually pull something it should not.
• Mask sensitive fields before they reach a model when the task does not need the raw value. Drafting a reply about a dispute rarely requires the full card number.
• Know where the model runs and what the provider does with your inputs. Get it in writing: how long inputs are kept, whether they are used for training, and where they are processed.

Many fintechs already operate under rules like the Gramm-Leach-Bliley Act and the FTC Safeguards Rule, which shape how you protect customer information and oversee service providers. An AI vendor is a service provider. Review it like one.

2. Actions: what can the agent do on its own?

Reading data and taking action are different levels of risk. Sort every possible action into three groups before you build:

• The agent can do it alone: summarize, classify, draft, look up, flag.
• The agent prepares, a person approves: customer messages, account notes, refund requests, anything sent to a regulator or partner.
• The agent never touches it: moving money, changing limits, closing accounts, credit decisions.

Then enforce those groups in the system, not just in the prompt. A prompt that says "never issue refunds" is a suggestion. An agent with no permission to call the refund system is a control.

3. Audit trail: can you show what happened?

If an examiner, auditor, or partner bank asks what the agent did on a given day, you want an answer in minutes, not a research project. For every run, record:

• The input and the sources it pulled
• The model and prompt version used
• The output it produced
• Who approved, edited, or rejected it, and when
• Any action taken in another system

Keep those records where your other records live, under the same retention rules. Logs that only exist inside a vendor's dashboard are hard to produce on request, and harder to keep if you change vendors.

4. Credit decisions need a higher bar

If an agent touches anything related to lending, slow down. The CFPB has made clear that lenders using complex models still owe applicants specific, accurate reasons when they deny or change credit. "The model said no" is not a reason.

The practical move for most teams is to keep agents out of the decision itself at first. Use them to gather documents, summarize files, and flag missing information for an underwriter. Widen the scope only when compliance is comfortable with how the reasoning is captured and reviewed.

5. Vendor review: ask before you sign

Whether you buy a platform or hire someone to build, ask:

• Where is our data stored and processed, and for how long?
• Are our inputs used to train models? Can the contract turn that off?
• What security reports can you share, such as a SOC 2 report?
• Which other companies touch our data on your behalf?
• Can we export our prompts, logs, and settings if we leave?
• What happens when the underlying model changes?

That last question gets skipped. Model providers update their models, and output can shift when they do. You want a pinned version or advance notice, plus a way to retest before a new version handles live work.

If you work with a sponsor bank, expect its third-party risk team to ask similar questions about your vendors. Having the answers ready shortens that review.

6. Failure handling: what happens when it is wrong?

Agents will be wrong sometimes. The question is whether you find out before your customer does.

• Route unclear cases to a person instead of letting the agent guess.
• Give reviewers a one-click way to reject and flag an output, and look at those flags every week.
• Test on masked or sample data before live data, and rerun the same test set after every prompt or model change.
• Name an owner. Someone on your team should own how the agent performs, the same way someone owns your fraud rules.

A pilot compliance can actually approve

Put together, a reviewable pilot looks like this:

1. One written job with a clear success measure.
2. An approved list of sources and fields.
3. Actions sorted into alone, approve, and never.
4. Logging turned on from the first run.
5. A fixed test window on masked or sample data, then a limited live run with a person approving every output.
6. A go or no-go review with compliance, using the logs.

None of this requires a big platform decision. It requires a clear design before the build. It also makes the second and third agent faster, because the review pattern is already approved.

Where Casey Key Advisors fits

I install and run multi-agent systems for fintech, medtech, and other regulated teams. Every engagement starts with the work above: one named job, clear data limits, approvals where they matter, and logs your compliance team can read. If you are evaluating agents and want a second set of eyes on the design, explore AI Agents & Automation or start a conversation.

Common questions

Can fintechs use AI agents with customer data?

Yes, with limits. Give the agent access only to the systems and fields the job needs, mask sensitive values where you can, and get the model provider's retention and training terms in writing.

What should an AI agent audit log include?

The input and sources pulled, the model and prompt version, the output, who approved, edited, or rejected it and when, and any action taken in another system.

Should AI agents make credit decisions?

Not at first. Lenders still owe applicants specific reasons when they deny or change credit. Start with agents that gather documents and summarize files for an underwriter, and widen the scope only with compliance sign-off.

What should we ask an AI vendor before signing?

Where data is stored and for how long, whether inputs are used to train models, what security reports they can share, who else touches the data, whether you can export logs and settings, and how model changes are handled.