James Wilby 0:00:04.0:

I'm James Wilby, I am a director at Keyrus. I oversee our workforce planning, our FP&A, and our SPM practices. Today, I'm very excited to bring Tatiana to the stage. Tatiana, I'll let you introduce yourself.

 

Tatiana Goff-Morrell 0:00:19.6:

I'm Tatiana Goff-Morrell, and I am the director of workforce solutions at YouTube.

 

James Wilby 0:00:25.7:

Amazing. All right, so let's see what we're going to get into today. We're going to try to keep about, let's say five to ten minutes at the end for an open Q&A. There may be a couple of places where we ask for some audience participation, but we want to start with introductions. I'll say a few words on Keyrus, Tatiana will speak to YouTube and where she fits in the organization. We'll go into her team, which is called support ops and trust and safety. We'll talk about some of the initial business challenges that YouTube faced before Anaplan. We'll go into what the actual solution is that YouTube is using. What's the actual value? We've done a lot of value realization. What is the value of implementing Anaplan? Then going into what's next, what's on the roadmap, and after that roadmap, we should have ten minutes or so for Q&A. Appreciate everyone being here, really excited for this session.

 

James Wilby 0:01:23.7:

So, the two cents on Keyrus. Keyrus has been a partner of Anaplan for ten-plus years. We've got about 3300 employees. We've had about 5500 clients in our 25 to 30 years of existence. There's no French accent here, but we are headquartered out of France, in Paris, and then we have a New York headquarters as well on the east coast. We actually got introduced to Tatiana and YouTube through Anaplan, but also through another workforce client of Keyrus. That's how we got in touch maybe two or three years ago, and it's crazy now just to sit up here with Tatiana, two years of partnership, two-plus years. I'll let you introduce yourself, or maybe the support ops and trust and safety team here, Tatiana.

 

Tatiana Goff-Morrell 0:02:11.7:

Yes, so I work for a small video sharing company that some of you might have heard of called YouTube. Actually, we've just celebrated our 20-year anniversary of YouTube, so pretty exciting stuff there. We actually support four different business units across what we call global vendor operations at YouTube. That is we have our support ops business, so think about consumer. That's subscription services like YouTube TV, NFL Sunday Ticket, YouTube Music and Premium. That's more traditional contact center. You would have email, chat, social media, a little bit of phone. Creator support, so let's say you're a creator that has gotten your account hijacked, you're having some trouble trying to monetize on the platform. As well as tech service delivery, so that would be monitoring video content for YouTube TV to make sure there's no outages, or doing the copyright, making sure as we upload music onto the platform that there's no infringement, and then our trust and safety, which is the content moderation. Really, the biggest thing that we look at is we're across the globe. We support 80 languages in about 100 different countries, and we actually have to do forecasts and planning for all of those different verticals and languages. You can imagine it gets really, really complex to be able to do all of that.

 

James Wilby 0:03:38.7:

Awesome. I may ask a little bit or maybe highlight YouTube for a second, and that is we're here talking about workforce planning and contact center planning and vendor management. I think about the history of Anaplan, how it got started as really a core finance use case. You see a lot of FP&A, which is awesome, in the Anaplan space, and YouTube has been able to use the same platform for its contact center planning and its vendor management, so all of its outsourced resources. If we've got any Anaplan current customers or Anaplan prospects here, I would say the support ops team on the slide is much more around just your customer service. That's how I think about it. What are examples of a support ops? Why would someone contact the support ops team? What's an example of what they're actually trying to do?

 

Tatiana Goff-Morrell 0:04:30.5:

Yes, so for YouTube TV, signing up for the service. If you're a creator, we have both what we call reactive support, which means that you're having a problem and you need to reach out to us, as opposed to proactive support, which means we're reaching out to you to try to help you make your channel better, monetize, that sort of thing. Also, huge social media. We get tweeted at a lot, or is it X'ed now? We get X'ed a lot, and so making sure that things that are coming up as trends, that we're monitoring that and responding to it. It's not traditional contact center in the fact that we don't actually take that many things that are calls. It's more casework, that sort of thing.

 

James Wilby 0:05:15.1:

I think a lot of people can resonate with the back office work. Maybe give us an example of what is an example case for trust and safety, so the third column here?

 

Tatiana Goff-Morrell 0:05:23.0:

Trust and safety is by far our largest and most complex. We support, I think it's 18 different workflows. Anything from egregious content such as harmful violence, adult, to copyright, general misinformation. We support that in all the so many different languages, so being able to understand what that looks like. It's basically while our machines catch most of the content, some of it does go for human review, and so it's a back office in the fact that it's a video that's being shared and watched, and making sure that there are no policy violations or any reason for it to be stricken off the platform.

 

James Wilby 0:06:04.9:

Awesome. Thank you, so that's some background. Let's talk a little bit, Tatiana, about, let's take 2023, we're first connecting. What were some of the challenges that you faced that really brought us to even start a project?

 

Tatiana Goff-Morrell 0:06:19.5:

Between trust and safety and support ops, basically, I was trying to manage about $600 million to $700 million worth of labor off of spreadsheets, which sounds great, right? We're a data sharing company, what could go wrong with that? The other thing too is like, my team was the first team that was brought together to centralize these functions. These would all sit within separate teams before. As you can imagine, everybody had a different set of spreadsheets that they were looking at, or trixes, as we call it at Google or YouTube. We would spend a lot of time arguing over whose numbers were right? How much labor did we actually have? The other thing too when I started at YouTube is everything was done on a headcount model. The vast majority of our business was going to vendors, and we were paying for each headcount as opposed to getting more precise. I could see very easily that as we started to try to move towards a more precise model that requires a lot more detail in your planning, which I was not ready or given the budget to quadruple the size of my team to be able to do, and that's not a scalable solution. I knew that having a single source of truth that we all had access to, really meant that I had to find some sort of a platform that would help us.

 

James Wilby 0:07:35.6:

I like it, and maybe just to double click on the headcount piece, because when we get to the value of what we did, that's a big driver of the value that we'll get to. Tatiana, can you be honest that the example would be, hey vendor, we're going to sign up for 300 headcount, and that's the contract. We're going to pay you for 300 headcount, as opposed to what we're doing now, which we'll get to in some of the solution. Actually vendor, we're going to give you 20 hours - I'm going to keep it simple - for questions on your password. We're going to give you 40 hours for some sort of content questions. We're going to give you another 60 hours, and you can see we're not just agreeing to a headcount number. You're now agreeing to hours by type of activity, which allows for improvement in precision and accuracy.

 

Tatiana Goff-Morrell 0:08:20.3:

Yes, absolutely, and if you think about when I first joined YouTube back in 2022, there had been hypergrowth on the platform, and so no one was worried about efficiency. The platform was growing like crazy, and I could see that it was going to be very important to us to curb what the spend was. When I started, for instance, trust and safety did exactly that. They'd say hire 300, don't touch it again until we tell you different, and we basically would have a governance process that would happen every six months. Vendors would keep the same headcount for six months, and then they would get a new headcount number. That was not going to work with what we were trying to do and be a lot more nimble on the platform, so we needed something that was going to allow us to change to evolving and changing business needs, but still maintain our need for safety, for quality, and meeting all of the service levels that we had.

 

James Wilby 0:09:18.6:

I like it. Let's go into, we had all these problems. What's the actual solution that we went into? I alluded to it a little bit, but maybe within your support ops and trust and safety teams, I'll call it let's say five areas of work that your team accomplishes. Can you talk a little bit about that?

 

Tatiana Goff-Morrell 0:09:38.8:

Yes, so I have a data science team that does forecasting and analytics. They're taking basically whatever the business forecast is, whether it be subscribers on the consumer side or video uploads on the trust and safety side, and trying to determine what is going to result in contacts, so pretty typical. They're going to actually forecast that at the macro level, so how many creators are we actually going to support? Then that actually goes over to my workforce team, and they're going to look at, okay, well, what labor do I actually need to plan for that? How are we going to allocate that between our vendors? That's a big piece for us, because we definitely want to be competitive, and we want to reward vendors or BPO outsourcers based off the quality of their work. While we might at the macro level have, say, 10,000 contacts in a week, we might want to reward vendor A with 60 per cent of that because of their performance, or possibly their cost. The workforce management team, the resource planning actually works very closely with our agent enablement team, which is quality, and our vendor management, to understand, how do they want to allocate that?

 

Tatiana Goff-Morrell 0:10:47.5:

Almost think of it like a pie, and how do you want to allocate those pieces of the pie? We have real time management, which is definitely more for chat and phone. Most of our cases are handled within 24 hours, but trying to do an offload of what happens. We had a major outage, may have affected some of you with global operations, where Portugal went down. We had a huge impact to our contact centers in Lisbon, so being able to offload that work from Lisbon to other locations. Reporting and analytics, so what does this mean in terms of, do we see that some suppliers are outperforming in other places? What does it mean to our spend? Do we see impacts in service level? Finally, I have an invoice team in cost management that centralizes and does all of the invoice reconciliation across the vendor operations.

 

James Wilby 0:11:42.4:

We'll take these five groups, multiply them by two because we've got support ops and trust and safety, so it's really fair to say ten groups, ten different teams all working together. Now, to take what Tatiana said and put it into the typical Anaplan honeycomb, the one major call out here is we actually do not do real time analytics on the last slide in Anaplan. If you're thinking about, where do we draw the line of where Anaplan is really good and where it's not as strong? We actually let the core contact center tools, you might think of nice or a variant and do a lot of the real time work, but we do a lot of forecasting. We'll have some vendor site forecasting, we'll have some resource team which is basically doing long-term supply plans, vendor allocations. That feeds into some reporting in blue, and then we do a lot of invoice reconciliation, PO tracking, and we'll actually show you we're generating invoices in Anaplan. I'm going to pause, hey, that's really cool. We're generating invoices in Anaplan off of the hours that our vendors are working, so that's a huge time saver.

 

Tatiana Goff-Morrell 0:12:52.4:

Can I just do a plug on this one too? Since we've been able to do this in our trust and safety, we have been able to reduce the amount of time it takes us to get an invoice into our tool, which is Ariba. We went from being 21 to 22 days out, and now we've been able to do that within ten days. That's since implementing Anaplan in trust and safety in January, so our vendors are extremely happy with us.

 

James Wilby 0:13:18.0:

Yes, and that's basically what this slide is calling out. We have a 90-day, it's kind of small here on the slide, but there's planning that has to take 90 days before execution. I think the fun one for you, Tatiana, is the NFL Sunday Ticket. We've got NFL Sunday Ticket is going to be launching in, or I guess it's going to have viewers in September, I think. Is it Thursday kickoff, Cowboys?

 

Tatiana Goff-Morrell 0:13:40.8:

Well, Sunday Ticket. 

 

James Wilby 0:13:42.3:

There you go, Sunday Ticket.

 

Tatiana Goff-Morrell 0:13:44.1:

Yes, the first Sunday after Labor Day.

 

James Wilby 0:13:46.6:

So there's a plan 90 days before that kickoff where all the capacity planning takes place. There is an alignment that takes place, let's say 60 days out of, okay, we've got all this macro workload, like you said. Who's going to work this? That's the blue, and then there's always changes to the plan, and that's the light blue. Hey, we locked the plan at 60 days where you're going to have 3000 hours. Stuff happens, and that's the blue, and then we get into the invoicing. Anything you'd add here on either the adjustments or the invoicing?

 

Tatiana Goff-Morrell 0:14:20.7:

Yes, so one of the things is that we used to do the invoicing with each of the vendor management teams, and as you could imagine, it was decentralized, everybody had their own set of templates. I actually, when I first started, had the vendors reach out to me and they were like, it's really confusing because even though we're one company, we're having to do this in seven different ways, and it's causing a lot of labor on our side, we can't reconcile it. By putting this into Anaplan, one, we're taking all that administrative burden off of our vendor managers. Do I want my vendor manager driving strategy within the business or do I want them worrying about an invoice? Clearly I don't want them worrying about an invoice, so putting it into this other team not only saves labor on our end for the people that are more strategic, but also it made our vendors extremely happy, because now they have one way to do it. It's all in the tool, there's no guessing. We had problems in the past where we would send out a sheet, and somebody would change a number and nobody knew where it came from or why it was. That's gone away now, so it has made both sides very happy, and they're getting paid a lot faster than they used to be.

 

James Wilby 0:15:26.8:

I think explicitly in this blue, rather than Gmail or Outlook back and forth, any change to the plan is actually marginally planned in Anaplan. Rather than, hey guys, you're clear for 20 more hours, hit send on my email, it's, no, you have to put 20 hours into Anaplan, and then that is then tracked everywhere.

 

Tatiana Goff-Morrell 0:15:47.7:

Yes, and not only obviously from the vendor side, but also our side, we're finding that we're not spending where we didn't actually order hours, which we could never get to that level of precision before.

 

James Wilby 0:15:57.5:

Awesome. I wanted to go through a few screenshots. Now, we can't share YouTube's screenshots, so these are the Keyrus app. Shout out Keyrus model builders. This is really a sample of what this actually does. This is a view where we've got our product that we're supporting. We've got our different BPOs, or business process outsource, or you could call it a vendor, you could call it external labor. We've got some KPIs of, okay, we've got 207,000 units of volume. It could be case reviews, it could be chats. How many hours does that convert to? What is the OpEx cost? I think this last piece is huge value right now, because previously it was, hey, 20 more hours, you're good. Hit send on my email, but there was never an implication on what does that actually cost if we're agreeing to 20 more hours from vendor A? Now it's all recalculated. It's an insane table of probably a million cells of just the rates to drive an OpEx cost, so it's really powerful, really excited about this. This is one page.

 

James Wilby 0:17:07.5:

Here's another one where we can actually allocate. Let's say we've got a vendor that supports five different products. Where do we want their time going? Maybe they're more efficient on product one versus product five, so just another way to make this a little more real using our Anaplan screenshots. In this, I mean, this is magic. This is the invoice, so as we have vendor one say, okay, we're going to work 1000 hours. We can track where those hours are worked. We load those hours into Anaplan, and now we have the vendors log in, see a management report, so management product team shout out. We're using management reporting to drive an invoice, and the invoice is what the vendor has to approve. Think about that, instead of YouTube approving an invoice. I'm going to say that one more time. The vendors now log in to Anaplan to view a management report, to view an invoice, rather than Tatiana's invoice team being like, okay, they just invoiced us for 1000 hours, but our plan only said 800 hours. What happened? Just another driver of value. Anything you'd add here?

 

Tatiana Goff-Morrell 0:18:17.5:

Yes, we were actually just recently moved to being able to do weekly invoice reconciliation, which was really unheard of. What used to take a long time for an invoice dispute, exactly what you were saying. Let's say we ordered 800 hours, but there's 1000 hours. Now we can actually reconcile that within the same week. We can also look at any sort of invoice leakage in a way that we couldn't before, so absolutely unheard of, would not have been able to do that without the platform.

 

James Wilby 0:18:44.0:

Awesome. All right, so what's the value? How are we doing on time in the back? Okay, I think I can read that. What's the value? Tatiana, I'll let you speak to this a little bit.

 

Tatiana Goff-Morrell 0:18:56.4:

All right, so since we've been able to implement Anaplan, we've actually been able to drive out 12 per cent of our spend for labor across both the support ops and trust and safety, so absolutely unheard of. I was a little surprised by this big number. Actually, when we first started working together, you had me talk to one of your other clients, and he had mentioned pretty big savings. I remember being a little nervous about putting this in the ROI. We were much more conservative.

 

James Wilby 0:19:26.2:

We were. We were at like, two per cent. That's how we ended up getting the project leadership to agree to it.

 

Tatiana Goff-Morrell 0:19:31.2:

Yes, so really the biggest thing is getting to a level of precision in our planning that we just couldn't do before. In fact, when I had started, there had been this big project to try to get some sort of workforce management tools to the trust and safety platform. There just wasn't anything out on the market. They were talking about potentially using engineers to actually build something, and ultimately decided not to do that. We were actually able to use Anaplan to fill the void of what they had recognized even before I started, so just really huge, more accurate forecasts. Now that we have moved to more of an outcome-based billing, forecasts matter. When you're doing a headcount, I can put a lot of fluff in if I'm just ordering 300 headcount. If I'm trying to drive out the cost and I have to give you hours and I have to deliver on those hours, it matters if I'm one or two or three per cent off. We actually build it into our contracts so that we pay out if we don't get our forecasts accurate, so there's a lot riding on us doing this correctly. Again, getting off of the Google Sheets. The company is getting more and more nimble and making changes. YouTube TV, since I've been with YouTube, has had so many iterations and so many changes, and trying to keep up with multi-view and all the things that it's going to do. Our old planning just wasn't working, so we needed to be able to really have one centralized place, one way of having taxonomy, and again, having a scalable solution.

 

Tatiana Goff-Morrell 0:21:07.8:

One of the things that I'm sure you've heard me say is whatever I'm building today, it has to work for me tomorrow. I can't just build a band aid right now, and I knew just adding headcount to put more and more spreadsheets, yes, I could have done that, but that's not going to solve for today's complexity or tomorrow's future complexity. What's really been great is just having the vendors be in the same tool that we're in, right? I'm even at the point where we're no longer sending them manual forecasts. We're not exporting, they can go directly into the tool and actually see what they're getting. What's amazing to me is we're only in phase one, so we've been able to accomplish all of this. It was mostly a lift and shift just to get to this point, and so there's so much more that's coming. What-if scenario planning, there's a lot of different spend that we have across the platform, and being able to understand if I make a small change here, how does that cascade down? I'm excited because 2026 is going to be the first year that we've been fully on the platform between trust and safety and support ops, and I'm really expecting it to change the amount of planning that we do to plan for 2026 using the tool.

 

James Wilby 0:22:25.7:

Yes, I like it. Maybe a couple things I'll double click into. Tatiana, you kind of breezed over this. Google, which creates Google Sheets, had hundreds of their own Google Sheets, and then the decision was made 1) to not go hire more people, 2) to not go build more Google Sheets, but 3) instead to avoid option one and two and go with Anaplan. Can you speak maybe a minute or two on that? Why not go hire more people? Why not go build more Google Sheets?

 

Tatiana Goff-Morrell 0:23:01.0:

It just wasn't a scalable solution. I could see that the business was getting more and more complex, and we're actually the first product area in Google for YouTube that actually moved away from the headcount billing model and onto outcome based billing. I knew that it just wasn't possible with the workforce team that I had. Not that they're not capable, but just not enough labor to be able to solve this problem, and frankly, it wasn't the right solution. I always look at it as, are there tools? Is there automation? Anaplan really solved that for us, so I've been able to grow to a much more complex environment, without actually increasing the staff size.

 

James Wilby 0:23:41.5:

Awesome, thanks. You alluded to it as this was phase one, getting hundreds of Google Sheets into Anaplan. We moved the onus of the invoicing, like we talked about. Vendors are now logging in to approve their own invoice. Well, what's next? I showed this slide earlier. This is what's been built, and the green here is where we're going. On the bottom left here we've got handle time forecasting, so think of, how long does it take to perform a case review? How long does it take to handle some sort of request? That's more of a forecasting component. Specifically it's handle time, so it's not like I'm forecasting three things. I'm forecasting a time interval, a time stamp for a particular activity, so just a little more complex. I don't know if you want to add anything.

 

Tatiana Goff-Morrell 0:24:30.6:

Yes, and I'm sure as with many of your companies, we're looking at, what can you automate? What can be done with AI? What can be done with chatbots? What does that mean for what actually goes to agent human reviews? If you take the less complex work out and you leave the more complex work out, how does it not only change your volumes, but how does that actually change your handle time? You need to understand it's not linear, and so Anaplan is going to help us solve for that.

 

James Wilby 0:24:56.8:

I like it. A couple others. There's already short-term forecasting and a separate tool that we actually load in. Might surprise you, might not surprise you, but Google has got all these PhD stat forecasters that create a very accurate demand forecast for the short term. We load that into Anaplan and then we plan off of that, because we're going to let the PhD folks continue to do their Python scripts. What's nice with Anaplan is we can actually generate a pretty low effort but high accuracy, long range forecast. If we let the PhD folks do all the short term, hey, we've got this new event coming out, we have to go do, this is going to be the volume in the next week, the next day, we can at least cover that monthly forecast and take that off of their plate. We're really excited about the long range forecasting. Couple others. Vendor invoicing we talked about, and then the end-to-end scenario modeling. I think to Tatiana's point, if handle time goes up, if we have a marketing event, if we have a change in some sort of vendor attrits and they no longer want to work with us and they're out, what does that do to our OpEx and essentially our customer support? I don't know if you want to add anything on the scenario planning.

 

Tatiana Goff-Morrell 0:26:12.1:

The what-if scenario modeling is actually probably one of the biggest drivers of time for my team. We see that a lot, even in say trust and safety, where we try to get the tools to understand what needs to be stricken off the platform. If we can improve what we call classifiers, what's left? How does it impact? Are we going to see the same impact in Burmese as we will in Swedish or Romanian? What's left? Can we cover the hours of operation with the staff we have? This actually takes a lot of time. It's usually not real time, but occasionally there will be something that's real time. The Ukraine war, for instance, had a huge impact to the business. Things that are more predictable, like last year I think over half the world voted, and being able to understand, what does that actually mean? These are the sort of questions that we have to answer all the time, and Anaplan is going to - we're excited about this because we're going to have a module that my team can use to get really intricate.

 

Tatiana Goff-Morrell 0:27:18.5:

We actually pay different rates depending on if you're a tier one agent, a tier two agent, if you're based in India versus Malaysia versus Europe. However, for our stakeholders, maybe they don't need all that precision. They just need to have a high level, what if I took 10 per cent of the business out? What would that do? We're going to actually have a lite version that they can play in the sandbox on their own to make some quick decisions, whereas where we need precision, I need to actually understand that the OpEx is going to drive $152,607 out of the business. Believe it or not, we have to get to that level of precision and get VP approval. I can have that, but I can also have the, I just need to know directionally what this might do. I'm really excited about this. This is probably one of the biggest drivers of time for my team to be able to model all of the things that get thrown at the wall.

 

James Wilby 0:28:09.5:

I love it. Yes, makes total sense, and that's pretty much all we've got today. Awesome. Well, thank you guys so much, you've been great. I really appreciate the time.