“Wait, how do you do that?”
We heard that question more than once during our workshop on A/B testing in CRM and email marketing. We had invited beta testers who help us shape new features, as well as people who use SEINō regularly. Around the table were people from Funnelboost, Happy Horizon, PI Marketing, Termeer Groep, Doktor Online and Mepal: agencies and brands comparing how they actually work.
We came to listen
We wanted to understand how teams design and run tests today. Why do they test, and where does the work get difficult? We started with those questions instead of asking people to come up with solutions. As everyone mapped their workflows in FigJam, the board filled with sticky notes. Some described steps that worked well. The red ones marked the friction: this takes too much time, this gets complicated, this is where we lose track.
Same goal, different workflows
Everyone was trying to do the same thing: run experiments that help them make better decisions. But the route there differed from team to team. Some used a separate calculator to estimate whether a send had enough volume and how long a test should run. Ideas and results lived in spreadsheets or presentations. When several tests ran at once, making sense of the outcomes became harder.
We were struck by how openly people talked about what did and didn’t work. They offered each other tips as they compared their approaches. We learned a lot, but the conversations were useful for the participants too: they could take ideas from each other back to their own teams.
When the learning gets lost
One participant put the documentation problem plainly:
“I have a lot of A/B tests running at the same time. I track them all in Excel, but I still need to find a moment to write them up. As a result, the tests feel a bit lost to me.”
Excel kept track of what was running. Turning those results into something the team could use for the next campaign still took time.
Why a hypothesis matters
One participant gave a simple example of why it helps to write down what you expect before running a test:
“Adding ‘Sign up’ actually caused the conversion rate to drop. We thought being clearer would help.”
The gap between the team’s expectation and the result was the learning. Recording both makes it easier to explain the outcome and use it when planning the next campaign.
Three dots made the priorities visible
Once everyone had mapped their workflows, we brought the red sticky notes together. Which problems belonged together? Where were people running into the same difficulty? The group helped cluster them around choosing test ideas, working out a hypothesis and potential impact, analysing results, and documenting and sharing what they had learned.
Then we gave everyone three dots to place on the problems that mattered most to them. Watching those dots collect was one of our favourite moments of the afternoon. We could see the group’s priorities taking shape on the board.
The strongest focus landed on planning a good A/B test, assessing its potential impact and documenting what happened. Analysing results also drew votes. We were struck by how much attention went to planning the experiment itself. The hard questions were often upstream of the result: What is our hypothesis? Why is this experiment worth running? Is there enough volume, and when can we reasonably judge the outcome?
What this could mean for SEINō
The workshop pointed us toward three areas to explore: planning tests, interpreting results and making past findings easier to find. Today, some participants use a separate calculator to estimate volume and test duration before a test, then Excel to track tests and results afterward. Those are the tools they use now; the workshop did not settle which steps should move into SEINō.
We’re exploring whether SEINō could keep a test’s plan and results in one place. That would help a team see why it ran the test, what happened and what it decided. We’ll show this idea to users before deciding what to build.
What participants said afterward
The notes people left after the session told us that comparing processes had been valuable to them too:
“Good session, short and focused. It was good to learn from other processes and see how everyone struggles with the same things.”
“Very interesting to hear what others are running into as well. I'd love to hear about the next steps!”
It was encouraging to hear that people wanted to see what happens next and keep thinking along with us.
What happens next
We will turn the workshop input into a few early designs and bring the first screens back to the group. We hope to share them within a couple of weeks. We want to see which parts of planning and documenting a test would be easier in SEINō, and which steps teams prefer to keep in the tools they already use. Their feedback will shape the first beta.
To everyone who joined us: thank you for showing us how you work, including the parts that are still messy. That honesty gives us something useful to take into the next round.