| Jim |
Hi, my name is Jim Compton. I am Technology Partner Manager at RWS and I’m here today with Kirill Soloviev. He’s the founder and CEO of ContentQuo and today we’re going to be talking about language quality and language quality as an investment. Kirill, thanks for being here today. Could you maybe tell us a little bit about ContentQuo and how that got started? |
| Kirill |
Absolutely. Hello, everyone. It’s my pleasure to be on the Globally Speaking podcast today. Thanks for having me, Jim. Yeah, ContentQuo got started a number of years ago. Before starting the company, I was in the localization business for 13 years. So at the points, I was actually a director of a global localization program at a software company and when I left that position, I started to think, “Okay, what are the difficult challenges that were really bugging me when I was on the buyer’s side?” So what are the things, what are the problems around localization that are still largely unsolved?
And then I met my co-founder and business partner, Alexander, and together, we basically decided on this idea, hey, that, you know, language quality has been largely unchanged and the way that teams manage it has been unchanged from the early 2000s. So people were still throwing Excel spreadsheets and sending them by email in 2015. So we set out to solve that and to make the process easier, less painful, more efficient and also more enjoyable. And here we are, 2021, our technology is now used by some of the largest and smartest translation organizations, localization teams and global top 10 LSPs around the world. So that’s ContentQuo. |
| Jim |
Cool. So I actually wanted to throw you kind of a challenging question today and this is based on an experience that I had a few years back when I was at a QBR with this big customer who is spending a lot of money on localization and a lot of money on LQA. And they were pretty frank and they said, “We actually don’t like spending money on LQA. They described it as a mistrust tax and said essentially every dollar that we spend focusing on sort of this mistrust tax is a dollar that we can’t use to bring content to our customers. I thought you might have an opinion about that and wanted to see what you think about that. |
| Kirill |
Absolutely. And thanks for sharing the story, Jim. I probably have even more than one opinion, which we can definitely explore. So first, I might live in Estonia, but I’m an ethnical Russian and in Russia, we have the saying that goes like this. Trust, but verify. So this is basically our company’s motto and we feel that it very, very nicely sums up everybody’s approach to this kind of question. And I think it rhymes very nicely with this mistrust tax, right? So, or trust but verify.
There is the other angle that I wanted to throw in here, is, yeah, you might think about it as an investment you’re making into your localization program and you typically have several alternatives where you can invest this money. So one of the best paradigms that I’ve found while building ContentQuo and deploying it in different localization teams all over the world is thinking about quality management as an investment game, as a way to put the money in strategic areas to reduce the risk of inadequate quality as much as possible with a fixed budget.
So this is very much akin to how, I don’t know, venture capitalists would invest their money into technology companies or into startups, right? They’re looking for a 100X return on their investment. So the quality management science and the quality management are just kind of doing the same. So how do we find that one area of our localization program where we would invest the quality dollar to get some amazing returns in terms of reducing the risk, the potential for quality problems?
So this is how we prefer to think about, yeah, on one hand, this is a bit of a necessity in the modern localization, world of complex supply chains that span several organizations easily, and yet at the same time it’s also a strategic investment. And actually, actually one of the main drivers why companies engage with us, why companies engage with ContentQuo, they want to reduce the amount of LQA or of quality assessment or review they’re doing. And data that we help them derive from this process is actually crucial to drive that reduction. So it’s a bit of a snake that’s eating its own tail in a funny and weird way. Go figure. |
| Jim |
So I want to ask you about that. Like, I like this idea of, you need to think about it as an investment. How do you approach that? Like, if you were to have a customer come to you and ask you, you know, “Help me invest my money to maximize my investment and, you know, invest in quality in the right way.” What does that look like? How would you advise them? |
| Kirill |
Yeah. So, to be honest, Jim, and especially lately in the past quarter, our conversations are a little bit less about investment and actually more about fundraising. So we actually help to have our customers raise the internal funding in order to sponsor any kind of a quality program before they can actually invest, right? So that’s been a major, major focus for us lately, just helping people understand the opportunity and build the business case internally to start doing something around quality, start doing something around data that they can use to reduce the risk, right? So this is usually the first investment problem or challenge to solve. Once we’re through that, once we’ve actually helped our customers raise this internal quality fund, then we can get into the really fun stuff. Then we can get into how we allocate this investment and so on and so forth across their localization program, different content types, different language barriers, different supply chains, different quality levels they want to hit and so on and so forth. So these are the two stages we could potentially talk about.
And, well, the last stage of course is, well, exiting. So getting some fabulous returns from your quality program, be it in terms of, you know, internal promotions they can get, or in terms of reducing the overall budget for their review program, or speeding up the time to market for high quality localized content. So this would be the kind of free turns that our customers usually get from a data-driven approach to quality. |
| Jim |
It’s interesting to me that you have to work with your customers sometimes to raise the funds internally to support quality. Why aren’t those funds already available as part of, let’s say, the localization program budget for these companies? |
| Kirill |
Great question, Jim. They actually might be available and I think this is exactly the situation which you started from our conversation today, right? So, the company that was saying every dollar spent on quality is actually not spent on something else, on crafting our global content with high impact. I would argue this is the case. Yeah, this is still part of the overall localization budget, but in fact, what we found especially was, let’s say, lower maturity organizations and sometimes smaller teams, is it’s incredibly hard for them to justify moving a part of the localization program budget and allocating this to quality.
This is actually very counter intuitive that it has to be this way, because let’s say if you’re a technology company, if you’re building software, for example, any person, any executive managing software product development would have a certain portion of their engineering budget set aside for QA, all right? People test software all the time. They report bugs, they fix them, they improve the software quality. The process is very well established, there’s usually no problem whatsoever allocating budget for a software QA program.
Now doing the same in exactly the same organization but for the localization program is sometimes incredibly and counter-intuitively hard. So in some cases, we even have to pitch together with our customers to their executives to explain how exactly localization quality investments is very much like software quality investment or, you know, content quality or source content quality investment. People just don’t get the analogy. Localization is still a black box and, you know, still the key role for any localization manager is to evangelize and explain and make sure that people understand why and how they will be investing that localization far from budget. |
| Jim |
I’ve heard other localization buyers, when you try to get into a conversation about quality, the sort of data that is returned to them isn’t really understandable or relevant in terms of like what the business performance would be of the content globally. What do you think about that? Do you think the data that’s traditionally produced today can be correlated back to the business performance? |
| Kirill |
Great question, Jim. I would say this is the holy grail of all quality management programs. And let’s try and unpack this maybe for a couple of minutes. Now, obviously, when we localize, we want to drive some business outcomes for our organization. We want to increase sales, we want to influence marketing KPIs, we want to reduce the support costs, you know, whatever the individual departments or businesses objectives might be. And, of course, the type of content that you create has an influence in terms of how and how much of those business outcomes can be achieved through localization.
Now the really big and hairy question is, okay, can we predict what will happen after we publish this content without actually having to wait and see how our customers, our users, our players, our readers will react? So, in other words, I think the ultimate allure of quality metrics that we can gather in the localization process is that they are actually leading metrics. You can have an idea about them before the content is out the door. Whereas any kind of business outcomes that localization is driving, they are inevitably lagging indicators. It takes a long time to see the results, right?
So ideally, in an ideal world, we would have a straight connection line between those leading indicators like quality and lagging indicators like outcomes. But yeah, there is a bit of a problem in many organizations we’ve seen; it’s incredibly hard to breach the silos or the walls between the compartments inside the organization and have this kind of straight line visibility between the two. So sadly, most organizations we’ve seen have not yet gotten to this holy grail. They can’t predict, although they’re really trying hard to align the internal, you know, localization view of global content quality towards something that might be more relevant towards the business outcomes.
And I think one big mistake that many teams make, actually both on the buyer side and on the LSP side, they’re trying to approach quality with a cookie cutter approach. So what worked for organization A surely must work for an organization B because on the surface their business is very similar. Now, this cannot be further from truth. And I think one of the major ideas that has really shaped our current approach to localization quality in the past decade, I would say, is the idea that quality is dynamic, that there is no one size fits all. And in fact, the best quality programs we’ve seen are acutely aware of that and they tailor the very way they define the right level of quality towards the content type, language barrier and even individual stakeholders they have to work with in order to get this localized content out the door.
So, yeah, holy grail not achievable, but moving in this direction is really, really desirable. And this dynamic view of quality as a set of moving targets is probably one of the best ideas we had from the 2010s. That’s where we are. |
| Jim |
Yeah, this is interesting. The even idea of what quality means has changed. The metrics have to do with language, right? Things that would be linguistic attributes. And do you think we’re moving more into a phase where quality is more a description? Could it mean something more like how the content is received by the intended audience? Like, how they respond to it? |
| Kirill |
Absolutely. Absolutely. So this is an aspect of what I was talking about before, when we talked about this outcome-driven notion of quality, right? So the end user’s or the reader’s perception, the actions they take when they consume your localized content, of course, these are the ultimate quality measure, right? Each piece of content is generally brought into this world for a reason. And if, after you localize it, it continues to perform and deliver towards that reason, towards that outcome, then the quality is adequate, right? And then you don’t need to care about downstream linguistic aspects. If it does its job, it means it’s high quality enough.
Now, the big challenge that we’ve noticed is in many cases, it’s actually really hard to get to the outcomes driven by content or even to the reaction of the readers and the users to that content. I’ll give you an example. We work a lot with some governmental organizations here in the EU, or with the European Parliament for example, they have an amazing quality program. Now, what kind of content do they produce? They actually localize laws. Okay? They localize parliament session transcripts and many other things that don’t have any direct way of collecting user satisfaction information. There are no marketing KPIs driving and it’s basically something that has to exist. Maybe there’s a certain type of a compliance component there, but there is no performance measurements attached.
Does it mean that they cannot run a quality program or should not? Quite the contrary. This is actually the only option they have in order to keep the results in check, because they know that for them and for their content, outcome-based metrics are never coming, not in a month, not in a year, not in 10 years. There’s simply no way to collect them for this type of content. So yeah, this is basically what’s keeping quality quite alive. In some cases, you just simply cannot get to how the readers will react. There’s no channel to collect this information. In other cases, it takes so much time that the metrics are incredibly lagging. And in yet many other cases, you only get surface-level feedback.
So that’s maybe the biggest problem was this kind of outcome-driven notion of quality. It tells you where you are and how people react to your localized content, but you have no clue why this is happening. And this is where the good old linguistic quality approach really shines, doing the root cause analysis and trying to understand, okay, but why did they react this way? Can we change something in our copy, in our content, in our localized software to have them react differently? And it turns out that for that matter, a formal linguistic quality program is incredibly useful. It gets you from the what, from the outcome high level results, from marketing KPIs, to ideas about why and then you can act on those ideas. |
| Jim |
So this translation of law I think is a really interesting example through the investment model, right? In that sense, it’s almost a form of insurance, right? You’re protecting against the possible disastrous situation that your law, through the process of translation, just made something, you know, go from illegal to being, you know, mandatory or something like that. |
| Kirill |
Right. |
| Jim |
So when you are working with teams, localization teams, let’s say, and they understand the importance of having this quality program built into that localization program and they need to, let’s say, convince stakeholders internally to raise funds, how do you coach them to make that argument inside their companies? |
| Kirill |
Yeah, it depends a little bit on what types of content they need to localize. Let’s take an example of marketing or digital marketing, ‘cause usually that has a very direct path to, you know, metrics important for the business. So I really like that. Usually, when it comes to global marketing content, most localization teams, at at one point or another, would institute some kind of an in-country review process. They have local marketers who would use the centrally localized content as part of their campaigns on the websites and their emails and so on and so forth. And this process is usually set up before any kind of notion of quality management comes into play. And usually what we find when we engage with such teams is that their in-country review process has gotten really out of hand, the in-country teams hate localization team because localization doesn’t deliver what they feel is right or good quality.
They are really pissed about having to make corrections to the localized copy over and over and the localization team and its suppliers seem to ignore of their feedback all the time. Right? So they really have nowhere to go. They basically, they have to continue cutting the proverbial tree with a very, very blunt axe, okay? And they don’t have a minute to stop that and sharpen the axe. So this is where we come in. We give them a way to sharpen the axe, cut down the tree once and for all and stop getting this endless stream of complaints from in-country teams.
Also reducing the time it takes to take the content to market from your source language, copied to your localized copy. So this is one very easy way to advocate for a quality program. If you’re struggling with in-country review, it’s taking you a long time and getting lots of pushback, internal stakeholders are unhappy or, you know, external partners start to complain. Put a quality program in place in order to gradually minimize the effort it takes the in-country marketers to improve your content and significantly reduce the time to market for globalized content. When you combine those two, less effort and faster time to market, a quality program is usually an easy sell. So that’s one good example. |
| Jim |
You mentioned your impetus for starting ContentQuo is that you felt that there were these problems unsolved. What were those problems and do you feel that they’re solved now? |
| Kirill |
Great question. I wish I could take credit for solving them, but this is actually not the case. I want to say not the case yet, but it, even that is a stretch. You see, machine translation has really, really advanced into the industry and back in 2015, when we started, even neural was not really a major player. I think just the very first releases of neural engine technology happened in 2015.
Now it’s mainstream and it’s driving a whole new set of quality challenges that we didn’t even know will exist back in 2015. So I kind of like to think about it as a never, never ending story. You solve one aspect of the quality challenges, like we like to think we solve the collaboration aspects of a quality program. We solved the exchange of feedback between the people who review the translation and the people whose translation is reviewed. Right? So our technology is great at solving that.
But now how do you do the same with machine translation engines? They cannot really understand your feedback. You need to do it in an entirely different way, okay? You can do much more. Translated content was an empty engine, right? Because your relative cost is so very low. Now you need completely new methods to keep your empty output quality in check, right? So this is basically the era of machine translation, with its own very, very intricate and special challenges. And so our job is never done. That’s kind of how I like to think about it now. |
| Jim |
I wanna ask about that because absolutely we are in the era of machine translation and I love this idea of when you solve a problem, the new one pops up, right? That just is how life works. Right? There’s a- |
| Kirill |
Right. |
| Jim |
Thank goodness for that, right? There’s an infinite amount of problems to solve. What do you think for the next six years is gonna be the focus of quality assessment, quality management? |
| Kirill |
Yeah. Surprisingly, what we see and we have been helping some of the more advanced machine translation teams run their own quality problems for MT, right? So I’m speaking from experience. What we see is this trend of converging disciplines; as the amount and the diversity of machine translation offerings increases in growth, we see this huge, huge trend of machine translation management becoming similar to vendor management in localization.
Let’s just think about it for a split second. When you start to have lots of MT engines and MT engine providers, you can almost see them as your linguists or as your suppliers or as your LSPs. You give them content to translate in one or two source languages; they give it back to you in a certain timeframe for a certain cost and with a certain quality level. And was this plethora of new MT players on the market was the much, much simplified process of training customer engines for different types of content and, you know, tone of voice and language variance. There’s basically an infinite amount of MT engines coming up and they can be managed using the same good old methods that vendor management teams and localization have been practicing for the past two decades.
So this is maybe one area that always keeps striking me as unusual. Yeah, they are machines, but we kind of need to herd them, right, almost in the same way that we have sometimes to herd our human suppliers. We need to help them do a better job. We need to monitor how they perform; we need to give them feedback; we need to select the good ones and give them more work and we need to stop using the bad ones, right? And this process never ends. And I think it will get bigger and bigger and bigger over the years. |
| Jim |
That is really interesting analogy. I envisioned like a HR department that’s on a bunch of conference calls with actual robots. |
| Kirill |
Absolutely. Honestly wish I never live to see the day, Jim, when a machine translation engine can actually respond to the feedback you’re providing it’s way. So I honestly hope will not be there when that happens. So for now, one major distinction between machine translation suppliers and human suppliers is that machines don’t talk, at least not yet, or at least not in the same sense as human suppliers sometimes might when you give them feedback on the quality of their transactions. |
| Jim |
That’s interesting. The idea of an actual argument or not an argument but, like, a healthy debate about like the output with a machine translator. Yeah. Will be interesting to see how this evolves.
Is there anything in terms of like interesting new technology in the quality management space that we should know about? |
| Kirill |
Hmm. Yeah, maybe there are a couple of different levels that I think people should be aware of. One is, again, for machine translation, there’s been some nice developments in the area of what they call quality estimation, which is basically building a machine learning engine that can assess the quality of other machine learning engines, right? So MT can be evaluated by something that’s not MT itself, but can get us some insights on how good or bad the output is.
And I feel that when that becomes widespread and it’s probably still several years away, that this will have a major impact on how we procure and deploy and publish MT. And right now, I think this is really, really advanced cutting-edge frontier that only the best and the smartest teams can afford to experiment with and trying to find their way around this new technology that’s still nascent, that’s still emerging and trying to figure out how to best use it.
But this will surely be interesting and I think it will even increase the need for human involvement because there’s been so much expertise in the traditional quality management community inside localization that can be applied to MT and to training better algorithms. But I personally believe quality experts will never be out of a job. Okay? So that’s one.
And number two, I think this is going back to the same idea that was mentioned in the beginning, right? So treating quality management and localization as an investment game. So we have been quietly working on some advanced solutions to actually build out something like that in the product, a tool or app or a feature, whatever you call it, that will help you become this kind of investor for your quality program and manage your portfolio risk in a smart way and run it on autopilot without having to deal with the daily chores that usually are parts of any quality management person’s or expert’s job, right?
So that might be on a smaller tactical scale, just making this, the life of those quality investors a little bit easier, less routine and more focused on analysis and decision-making than, you know, pushing the buttons or responding to comments. So this is probably a shorter term perspective that can also make certain things easier on a tactical level for many teams. |
| Jim |
That sounds awesome. Kirill, was there any topic that you would have wanted to discuss that we didn’t bring up or any question you wish I would’ve asked? |
| Kirill |
Maybe there is one, actually. Over the years, I was always surprised by how many quality experts actually don’t know much about the previous work that has been done in the industry to make their lives a little bit easier. I’m talking about standards here, Jim, and I know that you also personally are involved into quite a few of those. So we also have our own standards developments and the quality community and there’s one exciting thing that’s probably going out this year. There used to be a really nice effort called MQM, multi-dimensional quality metrics, spearheaded a number of years ago and finally published in 2015.
Now over the past years, there has been a very nice committee. Some of the best quality managers are now quietly working on the next version of that, basically trying to collect the best of the best experiences from around the industry on how language quality programs are set up or designed, managed and run. And I know that there’s some amazing people from RWS actually part of that effort too. So this new effort should culminate this year when the standards will be hopefully approved and published under ASTM. So I invite everyone to watch that space and once it’s out, really check what it has to offer.
It’s an incredible amount of work and thinking and knowledge exchange that went on to preparing the standard. And I really feel this represents the best easily available body of knowledge that anybody thinking to start their own quality program or improve their own quality program, whatever they might be doing in the industry, I invite everyone to take a look at that and try to figure out, “Okay, what can I borrow? What are the elements that I can reuse in order to avoid reinventing the proverbial wheel and try to actually get more alignment in how the industry treats quality?” I do foresee long-reaching consequences for the relationship between buyers of localization services and vendors that can potentially come out of the standard being adopted. So fingers crossed, when people listen to that, they might already see that available. |
| Jim |
Kirill, thank you so much. I think this was a great conversation. It’s super interesting to think about how quality has been evolving and where it’s going. |
| Kirill |
Thank you for asking smart questions, Jim. And I do really hope that was interesting and hopefully eye opening for all the people listening to Globally Speaking podcast. It was a pleasure to be here. Thanks very much. |
