Enterprise AutoML Workflow Redesign

The primary users were business analysts — people who understood their domain well but had limited hands-on experience with machine learning. Most of them were introduced to dotData by their organization, not by choice. Before the platform, many had relied on Excel or leaned on their data science team to run analyses.

That dependency didn't go away when dotData arrived. During configuration, analysts would typically work through settings in regular check-ins with a dotData data scientist — and in many cases, the DS would end up configuring the model on their behalf. The tool existed to give business users independence, but in practice, it still required expert support to operate.

The cost of a misconfiguration was hours. Model training could take one to two hours depending on data size. If a user set the wrong data type, they'd have to re-upload the dataset and start over from scratch. If the model ran but accuracy came back low, they often had no idea why. Most of dotData's customers didn't have an internal data science team — that's exactly why they were using the platform. But every time something went wrong, they were filing it into our DS team's calendar.We were absorbing the cost of a UX problem as customer support hours.

Going into this project, I assumed the problem was mostly terminology. The interface used data science language, and business analysts weren't data scientists — so the fix should be a language problem, right?

An engineer pushed back on that early. He told me to take a real dataset and try to configure a task myself. See if I could get it right without help.

I couldn't.

That changed how I thought about the problem. I started reading documentation, asking engineers to walk me through the pipeline logic, and using ChatGPT to translate technical concepts into plain language with concrete examples. Slowly I built up a mental model of what the system was actually doing. And that's when I realized the terminology wasn't the real issue. The issue was that the interface gave users no way to understand the consequences of their decisions before it was too late to fix them.

Time column → Prediction Target Time → Lead Time → Feature Window

These four parameters are tightly coupled. Each one determines how the system slices data for training. The interface never communicated that — and without understanding it, designing something helpful was impossible.

Our first instinct wasn't to scrap the wizard. We spent time exploring how to make it better — clearer section labels, easier navigation between steps, more explicit definitions for each parameter. But the more we dug in, the more something felt off.

At some point I realized: what users were actually doing, across all these steps, was defining a single flat table. That table was the training dataset. Every parameter — time column, prediction target time, lead time, feature windows — was just determining what that table would look like. The wizard was hiding that. It was breaking one coherent decision into a sequence of isolated steps, so users never had a mental model of what they were building toward.

That's when the canvas idea came in. If the real task was constructing a dataset, the interface should make that visible — not walk users through it one field at a time.

We tested with three external participants — master's students in business analytics — given a real dataset and a concrete problem to solve: predict which servers were likely to fail in the next seven days. A PM briefed them on the scenario beforehand. During the sessions, I sat back and observed, tracking behavior rather than what they said.

One thing stood out immediately. All three skipped every tooltip and description text without reading it — not because they were rushing, but because they were focused on getting through the task. They weren't learning the system; they were trying to use it. That confirmed that any solution relying on user education would fail in practice.

One tester flagged something that directly shaped a later design decision: she found herself going back and forth between the description text and the configuration panel trying to understand prediction target time, and suggested the concept should be interactive rather than explained in static text. That observation led directly to the timeline visualization.

When I first looked at this system, it felt like a black box. Users put data in, waited an hour or two, and got a result they couldn't interpret. My assumption was that the fix was simple — just show people more.

That turned out to be wrong. The question I kept coming back to was: does the user actually need to know this? Or is it just going to make them feel like they're doing something wrong? Some information helped. Some just made people second-guess themselves. Most of the real design work was figuring out which was which — and I'm still not sure we drew the line in exactly the right place.

One thing I wanted to push further but didn't get to: closing the loop between configuration and model performance. Right now users set things up, wait, and get a result — but they can't connect what they configured to what came out. That gap is still there, and I think it's the most important thing to go after next.