Recently, we had the opportunity to interview Ariel Weil, Yottaa’s VP of Product Marketing to get his insight on the evolving content delivery landscape. The conversation was exciting and educational, as the innovation-driven Yottaa continues to develop and deliver next generation features to its customers. Yottaa is investing a great deal of resources to ensure they create innovative services to help clients overcome their challenges in the online world. At the same time, Yottaa is focused on disrupting the top incumbents in the space, by out-innovating them. Ariel is well spoken, and has incredible grasp of IT, CDN, dev/ops and marketing. A big thanks to Yottaa for their insight.
Bizety Team: The topic is ValidateIT, please tell us what is it, what kind of metrics does it provide, and what are the benefits?
Ariel: ValidateIT is a methodology that allows enterprises to measure, analyze, and verify the business, technical and organizational value they realize from IT investments like CDNs through inspection of key performance indicators (KPIs). Because every enterprise is different, which means that there can be no set number of metrics or magic formulas to show business impact, we meet with the product owners and key decision makers, assess their desired goals, their needs, and the technologies in use across their web and mobile applications before establishing the KPIs and goal targets for a POC. ValidateIT is then used during our POC and onboarding processes to show incontrovertible proof of the value each player in the POC is providing.
ValidateIT is important when you look at the web performance optimization proof points and decision-making criteria in the market. For example, 500 milliseconds of latency showed a 1% decrease in eCommerce revenues for Amazon.com. When you look at a company like Amazon, you’re seeing a global enterprise that invests $3bn per year in R&D to maximize eCommerce revenues on a massive scale. They pioneer and provide many of the user engagement and conversion rate optimization tactics that have spawned successful SaaS and consulting businesses to fuel the sea of eCommerce websites competing for your attention and dollars. For Amazon, “all things are equal” when they test the impact of latency on conversions – traffic is consistently high, most pages have been hyper optimized for performance, engagement and ultimately conversions, and so reaching statistical significance for an A/B test is simpler for them to achieve than the average eCommerce enterprise.
The challenges most businesses face (businesses not in the 90th percentile of online business) stem from the technologies that are available to them, and the less predictable and consistent engagement they see from their users. For those businesses, performance can drive engagement, but that engagement must subsequently be tuned to realize business impact. It’s the very reason that A/B testing solutions are typically constrained to line of business users and applied on the client using javascript: IT provides a platform for eCommerce (or media, financial services, healthcare, etc) but to drive and tune engagement the Marketing and eCommerce teams need a fast means of dialing in content, layout, 3rd party tools, and social integration to, for example, decrease bounce and abandonment rates and increase pages per visit, time on site, and ultimately goal completions and revenue.
When Yottaa implements ValidateIT for a prospect or customer of ours, we focus on two things to ensure a credible test. First, we leverage the business analytic solution(s) already in place. Enterprises are comfortable with and trust these tools, and have established means of reviewing KPIs and reports to understand business performance. So when we interview the product owners to understand their goals, we ensure they also identify how they are currently tracking those goals using their business analytic platform. We want to snap into that process.
Second, we use a simultaneous A/B test to split traffic to each of the technologies participating in the POC. For example, if we are competing against an incumbent CDN like Akamai or Limelight, we will split traffic 50/50: sending a randomized half of users through the incumbent technology, and the other half through Yottaa. This is very telling for many of our customers and prospects because this can expose severe limitations in the other technologies, specifically where a lack of flexibility and open standards makes a simultaneous A/B test impossible.
Why is a simultaneous A/B test important? If I have a test that runs Monday through Friday this week, and another test that runs Monday through Friday next week, can you tell me beyond the shadow of a doubt that the information that you’re going to get about the resulting engagement and business impact of your site wasn’t due to other factors? In order to stay competitive online businesses are constantly evolving their content, rolling out new campaigns, and reacting to events in the media. For example, if we perform a POC in February with a company that is advertising during the Superbowl, the proximity to the big game will have a dramatic impact on site traffic, online offers, and ultimately user engagement.
Timing is critical because unless you’re at Amazon’s scale, events far smaller than the Superbowl will result in a wide variety of changes in visitor dynamics and buying behavior. For this reason we leverage real-time traffic splitting in our POCs – in most cases we will run a 50/50 split versus the incumbent technology, but we can and have split traffic 33/33/33 when there are other competitors vying for that company’s business. By splitting traffic, and leveraging the 3rd party business analytics platform’s tools for instrumenting and segmenting visitor traffic for reporting, we can show a clear winner in terms of business impact. In addition, because IT-centric performance metrics are critical, Yottaa uses our own (and can also use 3rd party application performance monitoring (APM)) synthetic monitors to track metrics for performance, asset count, page weight, etc. In that way we can show how business impact correlates to performance (speed), and also the importance of tuning user engagement to reach the enterprise’s goals.
BT: It seems that ValidateIT is a component of a broader solution. Is it sold as a feature, or is it included by default in the Yottaa service?
Ariel: It’s better than both of those things. We don’t sell it and it’s not proprietary to Yottaa. What we are bringing to the market is a methodology that anybody – any enterprise and any technology vendor – can apply. The point is to democratize the POC process so that enterprises can make educated decisions instead of resorting to a brand name; our aim is to enable businesses to choose the solution that’s fit for purpose, but also fit for the use case they are looking to optimize. For example, Akamai is the largest brand in the CDN space, and so most any Fortune 2000 company will already have or will be in the process of evaluating an Akamai contract.
Everyone has heard of Akamai and the technical lore is that “nobody gets fired for buying Akamai” because they chose the blue chip company. Akamai is a strong company with powerful technology that’s fit for certain use cases – global static asset caching, media streaming, and security being the main focus areas. But if you have a dynamic web application where your goals are to optimize personalized user experience across a multitude of different devices, operating systems, locations, browsers etc, then Akamai may not offer the best solution. Akamai acquired two technologies to compete in that arena, and we’ve seen evidence that, predictably, the bolt-on technologies do not perform like purpose-built solutions including Yottaa. We created ValidateIT in order to empower enterprises to challenge my assertion using real-world conditions and trusted business analytic data and prove which solution works better for them.
BT: Do different lines of business, such as marketing and dev/ops work closely together in implementing ValidateIT?
Ariel: If I were to answer this with my marketing hat on, I’d say that this is as close to true DevOps as possible. ValidateIT necessarily brings together cross-functional teams to make a purchasing decision for a technology that will impact each of their groups. To configure and enable a ValidateIT POC, the enterprise will have IT/Operations involved to make the DNS change and evaluate security and networking configurations and performance; AppDev will be involved to evaluate the impact on the web and mobile application code and it’s performance, as well as to evaluate the ability to tune that performance in real time; Marketing/eCommerce will be involved as the primary users of the business analytics data as well as to understand and tune the impact on user engagement and goal attainment throughout the POC.
The primary goal of DevOps is to enable businesses to move forward faster, and to ensure they can recover quickly from unexpected events and anomalies by eliminating functional silos, integrating cross-functional processes and applying automation to any repeatable process. So in that way, ValidateIT aligns perfectly to that DevOps mindset and can be used with any toolkit an enterprise has in place.
BT: A/B testing is very common among Fortune 1000 and larger enterprises, but are smaller online businesses using A/B testing as well? Also, do they have their own homegrown systems?
Ariel: The great thing about companies like Google is that they have really democratized A/B testing and made it something that anyone can leverage. So if you use Google Analytics you can run as many A/B tests as you like, for free. Now Goggle doesn’t give you the in-depth reporting that for example Optimizely or Test and Target may, but it’s an affordable solution that any marketer can use. But A/B testing is most often applied to front-end use cases and therefore uses client-side technology – A/B tests are most often enabled through the use of JavaScript to transform the client-side presentation of a web application once the content has been delivered. Very few businesses use simultaneous A/B testing to validate IT investments.
That’s because it’s just too hard to achieve and most people are unaware conceptually how they would get this done. Plus, what we’ve found since we started using ValidateIT in 2013, there are actually very few IT solutions that are able to participate in a simultaneous A/B test because their solutions require a very specific or closed implementation architecture. That particular limitation is critical for an enterprise to be aware of, because DevOps and really any modern web or mobile application architecture requires integration with an evolving set of packaged 3rd party solutions. If your CDN, for example, cannot function together with any CMS, marketing automation platform, eCommerce platform, personalization engine, etc. then you will have significant ongoing challenges with delivery cadence and ultimately your ability to compete and win online.
BT: Does Yottaa have any customers that have used ValidateIT and what is their feedback?
Ariel: Every customer listed on the Yottaa website has gone through this process. ValidateIT has allowed us to record testimonials that identify Yottaa as a no-brainer because the ValidateIT POC showed measurable business improvement that would result in ROI and payback figures that, in Venture Capital parlance, are unicorns for web and mobile applications. There is nothing more rewarding than ringing the closed-won gong in the office and hearing the sales rep read a customer email saying, for example, “from 48 hours into this POC and I could see that the 50% of my traffic not running through Yottaa is losing me money; put 100% of my traffic on your system.” What better motivation for Sales and Marketing to carry our message out to the market?!
ValidateIT keeps us honest and also helps us to stay focused. Because web and mobile applications have become highly specialized, and because there are distinct use cases enterprises must optimize for, we will not focus on OTT-streaming use cases because we know there are specialty products in the market that cater to that use case and Yottaa isn’t one of them; we wouldn’t enter into a POC with a company that is looking for a solution that we can’t provide. The enterprises we work with using ValidateIT find it refreshing that a vendor allows them to evaluate and judge the efficacy of our solution based on their own tools and instrumentation. It’s the IT equivalent of the Pepsi challenge – when all things are equal, which one works better for you?
BT: How can businesses tell if ValidateIT is a winning proposition? Is there an actual metric that tells them if they’re going to lose money or not?
Ariel: Our enterprise prospect or customer will already have an established set of KPIs they track, as well as dashboards and reports they typically review to evaluate how their web and mobile applications are performing from a business perspective. So once we configure ValidateIT they will go into those dashboards and reports and see the “With Yottaa” and “Without Yottaa” traffic segmented side-by-side, with clear indications of how the various KPIs they care about are trending for each set of traffic. The Performance-related metrics are pretty standard across verticals: time to first byte, time to start render, time to fully displayed, time to interactivity.
For eCommerce use cases, our customers typically track bounce and abandonment rates, pages per session, session duration, conversion rate, and revenue per visitor. For Media companies bounce and abandonment rate are still tracked, but there is more of a focus on page views, time on site, goal completions and advertising impressions. Yottaa works with our customers to help them understand how individual dimensions, like specific geographies, device types, browsers, etc. are performing so tuning can be done to maximize the outcome (and also highlight how that tuning can be accomplished or not with the technology we’re competing against).
BT: How many A/B test does a large company run over a period of time? Is it just one or more?
Ariel: The typical scenario is a two-week POC, but online businesses and any their analysts are aware of the notion of statistical significance. Based on what it takes to reach statistical significance – for example with a seasonal business like a Halloween costume supplier, POCs will necessarily take longer because traffic volumes for off-peak periods will be lower – we will work together to identify whether one more multiple tests are required. Some enterprises will run ValidateIT several times over the course of a year as traffic patterns change, because they want to make sure that they’re fully informed.
There are other factors that would justify running ValidateIT as well, including a major redesign, restructuring, or other architectural changes to their web or mobile applications. In those cases ValidateIT can prove that those initiatives resulted in a business improvement – or not. Because Yottaa applies automation as well as machine learning through our SaaS platform, customers do not have to undergo the constant A/B testing that client-side solutions require because we are integrated into the entire end-to-end workflow, from the datacenter, through the cloud, to the end user and back again. With Yottaa’s feedback loop, our system learns from every interaction and only becomes more efficient over time, which ongoing business metric review proves.
BT: Why is Yottaa offering this service for free? Why not charge for it?
Ariel: We don’t charge for ValidateIT because it goes against the very nature of an objective A/B test. I’m not suggesting we’re altruistic, we are still running a business and are motivated to win in engagements, but we are confident that our technology is superior and significantly differentiated so we challenge ourselves to prove that to ourselves and our prospects in every deal. Also, ValidateIT is a methodology that uses generally available technology and processes. So there is no IP to protect, and again, it would call into question the validity of the outcomes if we were to charge for its use. Imagine a company saying, “Want to see if our solution works? Ok well, pay me and I’ll show you.” That’s high-pressure sales and reeks of snake oil. Yottaa is working to educate the market to make everybody smarter about how to use and how to evaluate the benefit they receive from their technology solutions. Otherwise everyone gets pressured into long-term contracts and predatory pricing without any understanding of value.