How do you A/B test cold outreach campaigns?
To A/B test cold outreach, send two or three versions to random shares of one list, hold part of it back, compare replies once each version has enough sends, then send the rest to the winner. Ragworks AI SDR experiments do this with two or three playbooks, a holdback and a score with a confidence rating.
By Kishore Kannan, Co-Founder & CEO. Last updated .
What is an experiment in Ragworks AI SDR?
An experiment tries two or three playbooks on the same list of leads, shows which gets better results, then sends the rest of the list to the winner. A normal campaign runs one playbook on the whole list, so you only learn afterwards whether it worked, and every lead has already been contacted.
| Term | Meaning |
|---|---|
| Experiment | One test. A source campaign split across two or three playbooks. |
| Source campaign | The campaign whose leads you split. It's left holding the held-back leads. |
| Arm | One playbook and its share of the leads. Each arm runs as its own campaign. |
| Holdback | Leads you don't contact during the test. They're your control group and what you promote to the winner later. |
| Score | A rating from 0 to 100 for one arm. |
| Confidence | Low, Medium or High. How much you can trust a score. |
| Minimum sample | How many sends an arm needs before its score starts to count. The default is 50. |
| Promote | Send held-back leads to the winning arm. |
Read more: Campaign Experiments
How do you run an A/B test, step by step?
- Pick a campaign that has your leads loaded but hasn't been launched yet. This is the source campaign.
- Split its leads between two or three playbooks. Each playbook with its share of leads is an arm. Any share you don't give to an arm is held back.
- Create the experiment. It's saved as a draft and nothing is sent.
- Approve and launch it. Ragworks creates one real campaign for each arm and divides the leads between them.
- Score the arms to see how each one is doing.
- Promote the winner, which sends the held-back leads to it.
- Complete the experiment when you're finished.
A new experiment starts with two arms at 40% each and 20% held back. An even split of three arms is 26%, 26% and 28%, with 20% held back. Each arm needs at least 1%, and the arms together can't go over 100%.
What keeps the test fair?
- A random split that never changes. Each lead is placed by its ID and a random value stored with the experiment, so it lands in the same arm for the whole test. Arm sizes are approximate: on a list of hundreds of leads they come close to your percentages, and on a handful an arm can be well off or empty.
- Everything but the messaging is the same. The schedule, sending accounts, AI model, CRM mapping and channels are copied from the source campaign for every arm, so differences in results come from the playbooks.
- A holdback that stays untouched. Held-back leads are not enrolled in any campaign in your organization while the test runs, so they stay a clean control group and are what you promote to the winner later.
- The same targeting for every arm. With lead scoring on, it is on for every arm, because an arm that contacts its best-fit leads first can't fairly be compared with one that contacts leads in no particular order. Use the same criteria for every arm to compare messaging only.
- No changes mid-test. You can't edit an experiment once it's created. Pausing an arm, editing its messages or adding leads to it after launch skews the comparison, so leave the arms alone.
- A fair share of the sending limit. Arms that share a sending account normally share its daily limit by how many leads each has left, so an arm that does well would get a smaller share. Sender quota lets you pin each arm's share. It takes effect for LinkedIn connection requests, LinkedIn profile visits and email.
Read more: Lead Scoring
How does Ragworks AI SDR score each arm?
Each arm gets a combined score from 0 to 100: 0.6 × the results score plus 0.4 × the messaging score. If the AI review can't be produced, the combined score is the results score alone.
| Results signal | Weight | What it measures |
|---|---|---|
| Reply rate | +0.40 | Leads who replied, divided by messages sent. |
| Positive replies | +0.30 | The share of replies that were interested, a booked meeting, a referral or "follow up later". |
| Acceptance rate | +0.20 | Connection requests accepted, divided by sent. Only counts for arms that send LinkedIn connection requests. For email-only arms it adds nothing. |
| Rejections | −0.10 | The share of replies that were not interested, a vendor pitch or "already have a solution". More of these lowers the score. |
A results score of 50 means the arm is exactly at the benchmark. The benchmarks are a reply rate of 8%, a positive share of replies of 33%, acceptance of 30% and a rejection share of 25%. Each rate is adjusted for how much data there is, so a few lucky early replies don't swing the score.
An AI reviewer reads the arm's playbook copy and message sequence, your positioning and ideal customer, the campaign goal, the arm's numbers and a sample of real replies. It scores five things from 0 to 100, and the messaging score is their average.
- Message and ICP fit. Whether the opener and value proposition suit the target person and their pain.
- Sequence coherence. Whether the steps build on each other or just repeat.
- Positioning alignment. Whether the copy matches your positioning and ideal customer.
- Goal alignment. Whether it serves the campaign's goal, rather than just being good writing.
- Observed reply quality. What the sampled replies suggest: real interest, a soft maybe later, or the wrong person.
Scores update when you ask: Score now refreshes every arm at once, and you can score an experiment once every 6 hours.
When can you trust a result?
Confidence says how much to trust a score. It isn't a measure of how good the arm is. An arm is Low if it has sent fewer than the minimum sample (50 by default), has no results yet, or any of the rates behind its score is still too uncertain. Only Medium and High arms can be promoted.
| Level | Meaning | Can be promoted? |
|---|---|---|
| Low | Not enough data to rely on. | No |
| Medium | A fairly solid read. | Yes |
| High | A very solid read. | Yes |
Replies matter more than sends. The positive and rejection shares are worked out from replies, so an arm can have plenty of sends and still be Low because it doesn't have many replies yet. As a rough guide:
| Reply rate | When an arm reaches Medium |
|---|---|
| About 8% | Between 800 and 1,600 sends (roughly 65 to 130 replies). Still Low at 800. |
| About 20% | Between 400 and 800 sends (roughly 80 to 160 replies). Still Low at 400. |
These are estimates for typical cold outreach and yours will vary. In practice, expect to need somewhere between several hundred and a couple of thousand sends per arm before one can be promoted.
High confidence doesn't mean a good result. An arm with a very low score and no replies after many sends can show High, because Ragworks is confident it isn't working. Check the score as well as the badge before you promote.
How do you promote the winner?
Promoting sends held-back leads to the arm or arms you choose. It becomes available once at least one arm has Medium or High confidence, and only those arms are offered.
| Compared on | Manual | Auto |
|---|---|---|
| What you enter | A percentage for each arm you pick, taken from the held-back pool. | A weight from 1 to 1000 for each arm you pick. |
| What's left | Anything you don't allocate stays held back. | Nothing. The whole pool is sent out. |
| Leads added later | Not routed. They stay on the source campaign and aren't contacted. | Sent to the winners using the same weights, for as long as the policy stands. |
| Use it when | You want to roll out in stages. | You've picked the winner and want everything to go there. |
- Leads who haven't engaged can follow the winner. Tick the option to migrate non-engaged leads from the losing arms. Only leads with no reply, no positive label and no human waiting are moved, and they start the winning arm's sequence from the first step.
- Promote before you complete. Completing is final. The arms you promoted keep running as normal campaigns, and every other arm is closed and stops sending.
- If you complete without promoting, everything stops. Every arm is closed. The held-back leads stay on the source campaign and aren't contacted.
- Check the winning campaign is running. Leads meant for a paused or completed arm are skipped and stay where they are.
What do you need, and what are the limits?
| Item | What it takes |
|---|---|
| Source campaign | One with your leads loaded that hasn't been launched yet. It shows as Pending. A launched campaign's leads have already been enrolled or contacted. |
| Playbooks | A different Ready or Completed playbook for each arm, each with a published sequence and a message at every step. |
| Arms | 2 to 3. |
| Share per arm | 1 to 100%, in whole numbers, and all arms together up to 100%. |
| Default split | Two arms at 40% and 40%, or three at 26%, 26% and 28%, with 20% held back. |
| Minimum sample per arm | 50 by default, and at least 1. |
| Manual scoring | Once every 6 hours per experiment. |
| Experiments per source campaign | One draft or running at a time. |
| Leads split at approval | Up to 50,000. |
Common questions about A/B testing cold outreach
How do you A/B test cold outreach campaigns?
Split one list across two or three versions of your outreach, keep part of the list back as a control, score each version on replies once it has enough sends, then send the rest of the list to the winner. In Ragworks AI SDR you do this with an experiment: each version is a playbook, each playbook and its share of leads is an arm, and the arms run as real campaigns.
What should you change between the two versions?
One thing. Two playbooks that differ only in the opening line teach you more than two that differ in everything. In Ragworks AI SDR an arm is a whole playbook, so to test a deeper research level, for example, clone the playbook, set a step to Deep in the copy and test the two against each other.
How many leads do you need to A/B test cold outreach?
It depends on your reply rate, so there is no single number. Ragworks AI SDR counts an arm's score only after its minimum sample of 50 sends, but promoting needs Medium confidence, which depends mostly on replies. At a reply rate of about 8% that takes between 800 and 1,600 sends per arm, and at about 20% between 400 and 800. Expect several hundred to a couple of thousand sends per arm.
How long should an A/B test run?
Long enough to collect replies, because confidence depends on replies, not just sends. Ragworks AI SDR has no fixed duration: score the arms when you like, at most once every 6 hours, and promote when one arm reaches Medium confidence and the gap in score is clear. Early scores are noisy, so don't promote too early.
What is a holdback and why keep one?
A holdback is the share of the list you don't contact during the test. In Ragworks AI SDR it is your control group and the pool you promote to the winner later. If the arms use all 100%, nothing is held back, so there is no control group and nothing left to promote.
Can you test more than two versions, and on which channels?
Yes, up to three: a Ragworks AI SDR experiment has two or three arms. It uses whichever channels the source campaign uses, such as email, LinkedIn, Instagram and WhatsApp. The acceptance rate only counts toward the score for arms that send LinkedIn connection requests.
What happens to the losing arms?
When you complete a Ragworks AI SDR experiment, the arms you promoted keep running as normal campaigns and every other arm is closed and stops sending. The held-back leads stay on the source campaign and aren't contacted. Completing can't be undone, so promote first.
Can I run an A/B test from Claude or ChatGPT?
Yes. The Ragworks AI SDR connector lets an AI assistant create an experiment on a draft campaign, start it, score it, promote an arm and complete it. It asks you before it starts or completes one, because both contact people, and scoring is an AI review that needs a positive balance.
Sources
The Ragworks docs describe each setting, score and limit. They were read on 7 November 2026.
Ready to put your pipeline on autopilot?
Talk to our team. We'll help you pick the right plan and build your outreach playbook.
Book a Demo