Advance Planning for AI Project Evaluation

What’s Happening

So basically The work to do before the work begins The post Advance Planning for AI Project Evaluation appeared first on Towards Data Science.

It’s a common situation to find in businesses right now — there is a proposed product or feature that would involve using AI, such as an LLM-based agent, and discussions begin about how to scope the project and build it. Product and Engineering will have solid ideas for how this tool might be useful, and how much excitement it can generate for the business. (shocking, we know)

But, if I’m in that room, the first thing I want to know after the project is proposed is “how are we going to evaluate this?

The Details

” Sometimes this will result in questions about whether AI evaluation is fr important or necessary, or whether this can wait until later (or never). Here’s the truth: you only need AI evaluations if you want to know if it works.

If you’re comfortable building and shipping without knowing the impact on your business or your users, then you can skip assessment — but, most businesses would not actually be okay with that. Nobody wants to think of themselves as building things without being sure whether they work.

Why This Matters

So, let’s talk about what you need before you start building AI, so that you’re ready to evaluate it. The Objective This may sound obvious, but what is your AI supposed to do? What is the purpose of it, and what will it look like when it is working?

The AI space continues to evolve at a wild pace, with developments like this becoming more common.

The Bottom Line

It is also important to spend time on this question before you begin, because you may discover that you and your colleagues/leaders do not actually agree about the answer. Too often organizations decide to add AI to their product in some fashion, without clearly defining the scope of the project, because AI is perceived as valuable on its own terms.

Are you here for this or nah?