About the Author

Ché Macy

Marketing Director

in Business 2 min read

A/B Testing, Bracketology Lessons Learned

With the upcoming webinar reveal of A/B testing within the Broadleaf eCommerce platform and the excitement of March Madness, we couldn't resist launching our own 'Bracketology' challenge.


We implemented a series of 7 A/B tests on 8 different Broadleaf homepage design concepts, using the bracket format. The tests were implemented using a third-party source, Zipinion. Zipinion provides crowd-sourced feedback for concept testing, completing one test takes just a few short hours - and allowed for us to maintain the integrity of our current Broadleaf homepage for the live site. Each test had 100 respondents. Our test metrics were limited to content and design.


As a team, we hypothesized that the most popular concepts based on content and design would be:

Click here to see larger versions of the homepage concepts.


What happened?!

Well. Not at all what we expected. The majority of the team favorites were out of the running within the first round of testing. For each test set, we uploaded two images into Zipinion, and watched as the polls came in. Following the bracket format, we first tested Version A vs Version B, Version C vs Version D, etc. - here are the results along with comments from some of the respondents:

Round 1:




From a design standpoint, testing two images of equal size would work well with the tool we chose, however we found fairly consistently that voters were not opening/expanding their screen to see the images in full view - with feedback from voters reading, "larger font, easier to read" - even if the fonts were in fact the same size between the two concepts in the fully expanded view - longer homepage concepts consistently underperformed. For our purposes, it emphasized the importance of being able to conduct A/B testing for webpages within our own live environment. The ability to display content in its true context should naturally improve the relevance of testing results. As Broadleaf begins to roll out A/B and multivariate testing within the platform, I look forward to testing page content within the broadleafcommerce.com site - and managing tests from within the Broadleaf Admin Tool! Overall, the polled audience was partial to the green and blue/gray color scales, and loved the "Top 10" Broadleaf list.

I would like to give a special thanks to Zipinion for this fun experiment - I envision using the tool in the future for testing info sheets, slogans, and booth banner concepts for upcoming events!

The A/B Testing Functionality within Broadleaf will be revealed via the April 15th web event, join us!

Register Here

Hope you enjoyed the A/B 'Bracketology' experiment!