Transcript: Building a GenAI Product From Scratch for a Traditional Sports-Betting Industry — and the Pivot That Followed

Eyal David: [00:06] Hey Itzik, how's it going? — Great, it's been fun being here. What a nice intro we had. And there was good news too — I don't even know what to call it now: weekend, holidays, good news. Yeah, totally good. Finally we can get a little optimism in this period. So this is an episode that went a bit 'green,' but that's how we opened it. We've been talking for a long time about you coming on, and you finally made it. Itzik, tell us a bit about what the company does. And tell us a bit about yourself first, so we get to know you. — Sure, with pleasure. So I'm Itzik Adziashvili from Rishon LeZion, married with three kids — challenging and fun. Twenty years in software. I started in the army, in Mamram, as a developer, doing development there for eight years, and for the last three years I've been in product-management roles. I started as a product manager and moved up to lead the product team. I went through a big adventure with my own startup, which was hard, and then I joined a startup — LSports, where I am today. which was eventually sold. Most of my years were in technology products — cloud services, products for DevOps people. And for almost the last three years I've been at LSports in a range of roles. — What actually drew you there, to sports? Yeah, the combination of sports and technology. Being a product manager for many years means being in a space where you can solve problems through technology, and then you combine that with your personal passion, which is sports. It's something that always fascinated me, and when I found this connection it was a perfect fit. — So tell us a bit about the company — who does it serve, what does it do, how does it bring joy to the world? — So LSports serves sites in the sports-betting and media space. Basically sites that offer huge volumes of matches across a variety of sports, whose main goal is to get all the data. There are huge volumes of data at very, very low latency, and everything has to work in a data pipeline that's pretty much automatic. Their main need is to consume all this data, which comes in very large volumes, so that they can focus on their consumers. We're B2B; they have consumers and they want to focus on their consumers. They need the data pipeline delivered down to the millisecond. We work at very, very high rates, and we have an SLA of under a second for every update. And just so you get the scale — we offer 3 million matches a year across 60 different sports, almost every sport you can think of, and many of these games run live, with odds that behave like stocks, meaning they update at very, very high rates. And because we can give the client this pipeline automatically and accurately, while maintaining low latency and very high accuracy, our client can focus on optimizing their own users. Great, and you already started telling me — keep telling us about the company. Tell us also that it's from Ashkelon, tell us everything. Yeah, you told me, but also that this world is changing slowly. You just talked about sports matches, right? Tell us a bit about the trends. — Gladly. So broadly, in this world of sports betting there are the big, old players who focus more on the commercial side. They buy the data rights, pay a lot of money for them, offer expensive products accordingly, but their focus is more commercial — they work more with the teams and the leagues, whereas our angle — and this is also the market shift we're seeing and pushing toward — is to bring much more innovation and much more value to our clients from the technology itself. So on one hand it's a challenging market, because the clients are a bit old-fashioned — it's hard for them to adopt technology; these are very hesitant processes that take a long time. Turning on a new customer with us can take at least six months, but their stickiness is high accordingly, because when we show them the value of our robust product, they don't go anywhere. But that's part of the market's challenge, and we do see a certain trend, which really started during COVID, when there were no games and suddenly the market had to reinvent itself — they started with virtual games and began looking more at e-sports and offering many more updates and marketing that are much faster. So there's a trend that started then, and it's steadily developing in recent years: investing more in technology and reshaping these products for the new world, where they're much more social and go to the users, who have much lower attention spans, need it much faster, and don't want to wait until the end of the match for something to happen — they want it during the single play. I want to bet on the next goal, on the next assist, to be much more engaged during the game. Or for example, there's a lot more demand today — I'd say pretty crazy demand — for e-sports: games like Counter-Strike and League of Legends have crazy demand. You can see huge numbers of users on Twitch whose whole activity is watching some streamer playing, with hundreds of thousands of people watching, and around that there are already bets, a lot of content, a lot of engagement running there. So in our market too we see it heading there, and we work very hard on our technology so we can stay ahead of them and already give them products that meet this need. Wow, great. Tell me — one sec, I'll ask — what are you measured on against them? Because you mentioned latency, you talked about data volumes. What are you actually measured on? So we have two main axes we're chosen on: coverage — how many matches and how many odds we can give them at any given moment; and the second axis is latency. We have what's agreed with them, an SLA with them, of up to a second per update. We're talking about very, very fast updates, with sports odds behaving like stocks, and they reward us based on that — the wider our coverage, the more we can supply it at very, very low latency without hurting quality, which is always a balance and very important — because if I tilted toward one of the axes, my life would be much easier: to give as much as possible at the expense of accuracy, or to give it as fast as possible. When I have to balance between them, that's exactly the world that forces us to work much harder. And that's part of the fun of it. — So those are basically your two metrics. Do you have some kind of dashboard in the company that someone watches all the time and jumps at? What does it look like? — So yes, we do, a few more dashboards than that, because we also have several different product lines in the company, each with its own metrics, but broadly, yes, it all revolves around this balance of how much I can broaden my coverage versus how accurate the latency is, and from there it starts to break down by sport, by client type, by region. For American clients, say, who have special regulation, they'll also put more emphasis on accuracy and ease up a bit on latency, versus Korean clients, say, who'll put more emphasis on latency. Yeah, the differences are really in the per-mille range, but in systems that move so many odds at any given moment, every change in the decimal point can significantly move the game. So, you talked about them being more traditional — so you're also offering them concepts? Is that also part of what you do? Yes, it's part of our balance. We have the main product lines that are really our cash cow, that sustain us, but we also go out for innovation and do a lot of things like that. First, since we're from Ashkelon, I'll expand on that a bit: the company is almost 400 people, with headquarters based in Ashkelon. The company's HQ sits in Ashkelon, which is both fun and a bit of a challenge, but also great — we have new offices there now, a small office elsewhere, and another office in Poland. We're a bootstrapped company that's been around for years now — it took shape and grew from its own performance, so to speak; it never raised money, but it truly invests a lot in technology, and we're built accordingly — almost 70% of the company is technology people. R&D, product, data scientists, and all the various technical roles make up most of the company, because that's part of our philosophy — we invest heavily in creating value for the client through technology. And in parallel we have go-to-market teams — super-professional, super-strong commercial teams — but they don't have to be huge; they know how to do the job and take clients on themselves, while we try to bring as much innovation to the market as possible. So ultimately we invest in our portfolio such that there's the basic technology, the data feed itself, which we invest a lot in so it's the most reliable there is, and on top of that we add more products and new models that are more innovative — a model based on GenAI, some LLM we trained ourselves, engagement models, optimization models that help the operator improve their users' conversion. — Tell me, as GM, what are your main challenges at LSports? So I'll tell you a bit about my time at LSports. I've been at LSports almost three years. Half the time I was VP Product, managing the product teams, with everything that means — managing a product team, a product department that grew to almost 30 people: product managers, designers, analysts, and everything you need. At the end of 2024 we moved to what's called product-oriented squads, where product and R&D split by product lines and came together under the same group. And so I became the group manager of one of our product lines in the company. I took the data-collection product line — basically the first point in the chain, which collects all the data from across the internet and makes it available to the other products in the company. What that means in practice is that it created new opportunities for me — now I have a group made up of both the product people I know well, and a dev team and a data-science team, and that involves challenges that aren't always simple. I was always close to technology, and on the dev side too — I started as a developer, always close to it; it's also a personal love of mine. But now that you're a group manager like this, you're also responsible for production and system health — especially in systems like these, where when latency goes up from 200 ms to 350 ms we jump at any hour of the day. So there are challenges of connecting all these teams that are, so to speak, different disciplines — connecting them together — challenges of increasing collaboration between the groups, where each group is ostensibly an independent product, but ultimately our value to the client is created only when all the groups work together, because the whole company, as you know, is one big pipeline. So each group is chained one after another, and for the client to get the right update at the right time, everyone has to work together, so to speak. Now I thought you were first in the chain — that's a bit stressful. On one hand it's stressful, on the other it's fun for anyone who loves the action, and it also gives you a cool level of influence. Before we continue — click the link to the Product Builder WhatsApp group in the episode description and join the community, where you can continue the discussion, ask questions, and be the first to get updates on new episodes of Tachles. We're waiting for you there. And now, back to the episode. Tell me — okay, so given these challenges, what product skills do you think are the most important for a product manager working with you? So there are two that are very, very important. First is the connection to the technology itself and the love of detail — I deliberately put them together, because a product manager's ability to dive in and really know what runs under the hood, when the product is mainly a data pipeline — we have everything the clients need, but ultimately the core business is the data we send them, so to know a product like this you have to know the technology running underneath. What a data stream means, what it means to pass messages at low latency, how you do recovery, how you make changes — you can't just wake up in the morning and change the structure of a message that runs on a very serious back end; you have to manage it very carefully. So intimacy with the technology and attention to detail is super important. I mentioned earlier — 60 sports, 3 million matches a year — a huge matrix of content that we offer. So no, you don't have to know it all; there's a content team that's super-professional and knows all these things, but you need to know how to work within this crazy matrix, where every tiny detail can either add significant value to the client or break it entirely. So you have to be very, very sensitive to these details. — So let's open a parenthesis for a sec: what actually is significant value to the client? From their perspective, as you say, it's that it just ticks along, that things work. And what is significant value beyond that? — So the fact that it ticks — you can take that in many directions. The world of sports and a sporting event has several phases. What happens before the match — that usually starts a week or two before the game. What happens during the match, where the emphasis on latency is much more important, but there you also have many more odds, many more events — everything has to flow, everything has to tick. And there's also the post-game — after the match you want to close out, send an event summary, close all the markets, give all the surface all the gaps, reconcile the operation and close them. So in each of these stages you have significant improvements and a lot of ways to give the client value, whether that's in the pre-match phase for example — we created a product that generates tips intelligently for their users, so they have ready insights. If I'm a fan and want to place a bet on the next match and I don't fully know the teams playing, I'll get some insight that tells me the history. We're a technology provider; we don't know which bet you should place, but we know how to look at data and say what the statistics say and what the history says. Whether that's during the match — giving them an engagement tool. For example, one of our products is a very nice, visual, very engaging tracker that gives our clients something to engage their own users, like the trackers we see on sports sites like bet365. So we have products like these too, even better. So at every point in the chain you have many more innovations derived from the basic data we transmit, and once you understand this, there are a lot of opportunities. That you hunt for — is that thinking you do, or does the client come and in a joint conversation these things come up, that you actually arrive at? I'd say in most cases it's listening to the client and leveraging their desire. So yeah, it usually won't be exactly something they ask for, but we're very customer-centric, we believe in being close to the clients — I mentioned we're a bootstrapped company, they're the ones who sustain us, so we work very smoothly with them, and it's not just the customer-facing side — also the technology teams, product people, even a bit of the dev team are in very close contact with the clients, because it's super important to us; we're there for them. And that's also where the best insights about the clients come from, and about how they use things. As I said earlier, it's an old-fashioned market, but that also gives the advantage that these folks have lived it for decades — they know it really well. So I don't know if I'd say they say 'build me an LLM that gives me insights,' but they know how to point you to the area, and from there we have a product team that knows how to take the problems, break them down, do the evaluation with them, and bring the innovations. But yes, I'll go back to the start — we're very, very close to the clients, because it's very important to us. So there you go, you've already brought up more skills along the way. Do you have more skills to add? — Right, so that's attention to detail, super important. The second skill I see as acute for a product manager in our domain is the ability to manage focus like an accordion — knowing when to zoom in and when to zoom out. We talked about the details, so in certain meetings or certain initiatives you need to drill down, down to the level of the single sporting event — what's the difference between a given match in the US, in leagues in the US and in Europe. There are differences in both, by the way. On one hand you need to know how to do that — to ask the right questions, to know the differences; on the other hand, still constantly zoom out, step out of our rich content world and look at the big picture, know how to look at the whole pipeline that LSports ultimately offers clients. As I said, we're groups chained one after another, so I can't know all the company's products deeply, but I must know the basic dependencies — what each product line offers and how it affects me. So the ability to work like an accordion isn't simple, it's challenging, but it's a skill that's very important, and it also saves us a lot. As I hear you talk, I sense you have no patience for bullshit. Totally, totally. By the way, I can say that's one of the things that captivated me from the first moment I arrived at the company. I'd been at quite a few companies, including very good ones I really enjoyed, but what struck me when I came to this company is the directness — maybe it's part of the vibe of us being in Ashkelon, and that it's a family company where everyone is genuinely very close to one another. By the way, it's a thing — we have very, very high percentages of employees who've been with the company over 8 years, which is very hard to see today. So this closeness and directness — we have a lot of teams like a regular company, but the work is much more direct. And sometimes it's challenging, because when you're a bit new to the scene you wonder how there isn't a bit of that distance, but once you learn it you understand that everyone is around the same goal, and yes, there's much more of the agenda — which I also love — of 'cut the bullshit, let's get to the service, to what's needed, tell me how I can help you with fewer stories.' How do you do it, how do you do it? I'm like that too, but I'm curious — to keep it very... 'thank you for asking, kind sir'... how do you do it? So it's all built on the trust we have for one another, and we do align that we're working toward the same goal. So it does start with trust, and we invest a lot in building trusting relationships with each other, with all the activities that aren't just for fun but also to connect and create this trust. So I do believe in trust. I believe that when I work with someone, I'll build the kind of relationships where they trust me that we're both coming to succeed together. Maximum, we won't agree — we'll open it up, but... And we'll open it up, but it comes from very, very clear trust. Second thing — cut to the chase. Tell me directly what you need. As product people we often love to go back to the full story, what the client needs, how it came about — a lot of stories that are important to revisit, like we teach the new product managers: go back like a broken record to your goal and what you want to achieve and how you want to achieve it. But that's not always needed. Often when you work with people who are super-professional, they need the endpoint, they don't need the whole story. Give them service. If you're talking with a developer who's already experienced enough — now, we have dev people who've been at the company a long time, they know the business — I can say, I've been here three years, they know the business at least as well as me, if not more, and with the dev people. So when you see you have someone experienced, use them and give them the service of what you need. You want to do an evaluation of some development — it doesn't need a whole grooming session or a big meeting; that too can be much shorter. Just say what you need. For instance, if you did the evaluation of some new feature and concluded you need it a certain way, or that it's not worth doing because otherwise the timing is off — that's fine too, put it on the table. Don't try to waste time or just try to understand — put it on the table and see whether it converges or not. Great — doesn't converge? That's also great. That's the second thing. The third thing is much more direct communication — this too comes from the CEO and passes through the company: cutting down meetings, working in much shorter cycles. And it's also something I've noticed: usually when you're in a conversation with more than three people, it becomes less effective, because someone in the conversation is no longer active and loses interest. So the short forums of 1-on-1 or three people are the max you need to make a decision, because ultimately what matters is to make a decision and move forward. We're in a world with very high frequencies, and often it's more important to make the decision — when it's justified and reasoned — but to make it and move forward rather than keep drilling. So these practices, I'm a fan of them; they push us much more toward the delivery itself. And yes, I do see it too — I can tell you, you talked about CEOs, so I work with a lot of CEOs, so there's that aspect too, a lot of it. circles like these that CEOs or management members run, and you want to bring them into some work rhythm, so there too you sometimes have to bang the table and say okay — maybe there sometimes more... than with the developer, where maybe it's more delicate, or where they'll take you for a spin, and maybe there you don't have that kind of story, maybe. Yeah, it's open upward too, totally. First, I can add that to get to the point more effectively, it's also about keeping the ego low, which is easier said than done, but we have to keep reminding ourselves, and luckily we have an environment that reminds us of it all the time, and that's super important. By the way, I often do this at the end — I tell myself, okay, if I'd had a bit less ego, it would have been much easier for me to reach the right decision. So it's something I do a kind of post-mortem on and I call them on it. Yes, it also meets you when you work with management, management's choice and with the CEO's choice, as we said, which is much more open. So quite a few products were created this way — who knew you'd have a hallway chat with the CEO who comes with an idea, and boom — we have a CEO who's very visionary with lots of good ideas. Usually the challenge is to slow him down a bit, but that's part of the fun — to brainstorm a lot of things and really distill what we think will give us value. And last year too we had such a product that I got to give a talk about at the last Product-conference, so... This was an opportunity where we charged ahead last year, when it started from the hype of GPT and the whole world went crazy and everyone needed an AI product. So let's start from the end — we said we need an AI product, don't know what, don't know how — I'm deliberately exaggerating, but we said we want to build one as fast as possible, and we want it for the annual conference. Tell that part for a sec before you dive in, because you talked about a very traditional industry, right — so why exactly did you go after the North Star of AI? We wanted to leverage the trend in the market: on one hand the industry is very traditional, on the other the cutting edge of technology, and we understand that we — unlike many companies — can't afford not to, not to get into AI. By the way, we've had quite a bit of AI for many years, but specifically GenAI, which is on one hand at least the frontier of technology — because there's a lot of buzz, we told ourselves we want to stay ahead of the market, let's test it out. Maybe something good comes out of it, or maybe we'll drop it, but first, it's to check whether there's real buzz here — is it just buzz or is there really some real magic here. But we also leveraged the opportunity of the annual conference, where we said if we want to bring an innovation, this is the time. We talked about it half a year before the conference. But let's factor this for a second in terms of a product-development cycle from zero to one, especially a GenAI product in a new technology — that's not a lot of time. And because the industry is old, the annual conference is like the place to bring innovations — it's the time when everyone comes with real attention. At any other time it's hard to capture people's real attention for innovation; they talk to you more about their bugs, less about innovation. So we said we want to bring such a product, and we took the opportunity we'd had in mind for many years. As we said earlier, we have a product that generates insights for you automatically toward the match and during the match. So we took this concept and gave it a GPT-like experience, where I as a user can talk with a chat that prepares me for the match, and just like I watch a commentator on TV mediating the game, I can engage and talk with this commentator. So basically we went all-in, and that's also the unique point in this process — that unlike a regular product where you build a small, meticulous MVP and go out with it fast, we went all-in and stood up a full product. I have to understand — so he basically defined a very big goal: an AI product, two words, and we also set a target, right — and then what, and you went to see? Because you jumped really fast to 'okay, I'd probably also want an agent like that,' but I don't know how far it went and what to tell — but yes, you jumped to it very fast; so what was the process you did in the meantime, until you got there? Did you do market research, did you check — how did you actually arrive at why it'd be this AI product? — So yes, we did the whole process, but we did it much more condensed, much more focused. We took it as another exercise — basically to present. It sounds like a nice presentation, by the way, but did you actually go to present this now to blow their minds at the conference, or already to start actually selling this thing — like, is this something your clients can really implement and profit from? What did it look like? — So the intention was to come to the conference with a ready product you can already sell — not just sell a presentation, but sell the product, so that the morning after they can start trying it and implementing it on their site. So we set out on a process that was thorough but relatively short, of market research, to see if there's such a product in the market. Yes, we focused very quickly on the opportunity of match preparation, because it's something we've had in mind for many years — that no one does better preparation beyond tips, which are super useful but ultimately capped in how much users need them; it's hard to measure that, hard to show the value there, it requires a lot of hoops until our client sees the user's conversion. So we wanted to focus around this opportunity, and we ran two studies — not simple ones, but focused. So one was user research to see whether anyone offers something like this, or something similar in that direction at all. We saw there's no such product — no one offers any interactive experience with the user to prepare for the match, in the sports-betting world. And I emphasize this because unlike the media world, preparation in the sports-betting world is a very different context. And second, there was a lot of technological prep work of what it means to deal with GenAI technology — can I use services like GPT, Claude, and so on, or do we, heaven forbid, build our own? These two tracks met after about two weeks. And basically we understood we're going for a product that's a widget for our clients — a chat widget they can embed on their site, and it gives their user the whole experience with all the data — which, by the way, is also why we focused on the match-prep opportunity, because we have years of sports-event data that we store, and now suddenly there's technology that can help us leverage it significantly. So whereas before we'd tell ourselves, if I now train my own model on all ten years back of data I have to work very, very hard — suddenly I have some LLM that can also index it much better and make it accessible to users. So you went for quite a goal here — you're basically saying, again, this is a product that's sale-ready, is that what you're saying? And it's also operational — not just that it's operational and we show it in a demo; I can actually already embed it on their site, right, with the clients. You did research, understood what exists and doesn't, defined some very high-AI-level scope, right — and then what, like, you reached the conclusion that you're betting... You said it earlier — because it's not an MVP, you're betting on their product, right. That mainly expressed itself in the quality of the product. We trained it, aimed it, built the product, started using it both as sports fans and as potential users. And it turned out, for instance, that half the time it answers me well and half the time it doesn't. Normally, when a product is at fifty percent accuracy and it's version one, I as a product manager would go for it — I'd launch and start rolling it out. There's a user learning curve, there are a lot more processes that come until you reach the stage where you need accuracy at much higher percentages. In our case, because it's a product that's customer-facing to our clients, it's basically a B2B2C product. And our clients — from the initial conversations with them, which by the way were part of the initial research too, talking with them and developing this idea — so it was also in that focus to concentrate on football. How was that? As usual we focused on football, which is the number-one sport in the world, so we focused on football, we even focused on specific leagues, we went for the big leagues where there's much more interest and also quite a few users. So at least that way you narrowed... yes, yes, so there we narrowed the scope, but when we started testing the accuracy of the product and we were around fifty percent, we understood it's not good enough — also from our own personal use. As I said, if you and I want to talk with a chat and half the time it gives me hallucinated answers, I get frustrated very fast. And the tests also showed us that we need version one to have much higher accuracy — the performance of a much more mature product than its version one. I have to ask you one more question before you continue — tell me, isn't this basically public information, you mentioned it a lot? Since it's public and today a regular model — that a model company knows how to pull it and bring a good result — tell us for a sec about the quality, this thing you know how to bring. So first we have the competitive advantage that we've been doing this data for many years — there's no database you can consume, 3 million sports matches a year going back years. The context of the data — to segment and index and filter and slice the information the way we do, per sport, league, location, team, player, individual sports, team sports, races, Olympics, and so on — there's so much metadata you need to index it, which takes many years, so we're what you'd call ready to go. So because we've been indexing it for so many years, it's much easier for us — whereas if you started this from scratch, you'd have to work very, very hard to generate it. By the way, part of the user research showed — it's not that there's no competition; I said there's no competition, but I'll be more precise: the competition is scattered — there are scattered players who focused on some specific sport and offer it there. But we, from the start, do want to give much broader coverage, so that's also why we understood we have some advantage that only we can bring — something innovative with very broad coverage. Okay, so there was surely someone in the room who said we can do something much more shallow — let's do a quick one-two-three. What did that discussion look like, and how does it develop from there? — First, we're much more direct and it's much simpler, plain. We open a note, talk with it, see the answers, and we have a few people in the room, and we said it's not good enough. And that's also part of our trust — since we're already sports fans ourselves, if we see that users suddenly... I wouldn't use this myself. So yes, metrics are good and valuable, and sometimes a person's intuition is super important, and we understood that with ours too, this wouldn't hold up. Obviously when you present something like this at a conference, the first thing people do is pull out their phones and start playing with it, right, right. So the goal was exactly to tell them, come, open it — we had such a booth, a huge booth, a huge tablet designed for decision-makers to come and talk with the chat toward real upcoming matches — really, not some demo. High fidelity, a really real system — because we know our clients are super-professional, and if we can't show them something that works, and not just in a slideshow, it won't speak to them, and we didn't want to just nod along. So we really did it so that the morning after, with the first pilots, they could start implementing it. So, like, really — we're saying, okay, five months before, that was the thinking, we bit off one month, so there are four months to the conference — what does it look like from here? So with a lot of investment — how do you even, sorry, one sec — how do you even cascade this to the team, that we're now going for such a big MVP? What does it look like? So with a very open approach — I gathered all the teams, and there are about three teams there, which is not insignificant relative to the product's scale for the year: dev teams and a data-science team and a product manager and designers — all the roles you need. I set the goal — we were very open. I understood it's challenging, but we were also glad to have super-professional people there who love the challenge and got excited about it and led us there. So I put the goal in front of our eyes and said we want to reach this quality in this timeframe. Now let's see what's needed and start cutting backward. So yes — figuring out how to get there also gave them the tools to do the tradeoffs, for instance, to open up their algorithm from the back. We also swapped it several times until we found the right LLM that fits us, and we kept working with it, did a lot of optimizations and tweaks on all kinds of logic we could, to help the model perform much better, or changes that work much faster. Another thing that was part of the tailwind was that it was also during the summer break — we're a seasonal business, and over the summer break there are fewer matches, the number of matches drops by 60%, so we leveraged that too to invest much more in these required changes. But yes, I can say the main magic was putting the final hand on it — everyone talking about it very openly and working in very short cycles. There are always conflicts and tradeoffs and more that come up, but we knew how to work in very, very short cycles. We were also helped by the fact that we have a very developed and very strong technological core, so that also gave us the confidence to go for such a move in the first place. And tell me, what does it look like in terms of QA? Like, how do you go for such a move, how do you — is it worth telling about that a bit too? You also gave a talk on this topic, right? Yes, right — I had a talk for the army's product team on this topic. So I can say that QA in systems like these based on GenAI is something that, in my view, is still not solved, still not clear — you still have to work out the practice and the... It's something your product team defined. I hear all kinds of stories and all kinds of companies, how it looks for them. So in our case it was more defined by the technological side. The data-science people and the CTO put more emphasis there, because it really requires more understanding of what an LLM is and how it works and how you measure it. And then we started with experimenting and tuning parameters of accuracy or precision/recall, trying to play with them to see, against the comparison of the manual tests for instance, and to see which of the changes really gives us that. I don't think we found the right answer — I can say it's very, very hard to distill the AI-evals you need to know whether my product works properly; this gets built over time. We started from pretty simple accuracy of scientific tagging at first, and afterward also building metrics of how much value it generates for the user — for instance, we understood it's hard for us to estimate our model, so we started measuring how many conversions the user does, on the assumption that if the user converted, our client got the value they need. — What does 'conversion' mean here — basically closed a bet? Yes, right — because if I give my client a widget, they put it on their site with their user — probably I brought them a good experience. So we actually started at an end point to show there's value for the client, and we said we'll advance from there. Afterward the product changed, but that was also part of the challenge of — if I can't measure the model, let's know the variable — which is an insight we got along the way, also a byproduct along the way. And you set a target to present at the conference, so the bottom line was great for us in such and such a way. It's a very good bottom line, totally — especially since it's also a very strong driver for the client themselves, who wants to engage their users, so they understand where it meets them even before I start explaining the magic that happens behind the scenes. Exactly. Good, so you probably also understand your clients. So tell me — you mentioned you also swapped the model a few times and optimized it, so basically around the models, around these basic metrics — or was there also some kind of evolution? So when I say model, there are several models behind it — we also swapped our LLM and also added a few more models, some more classic, but supporting models, so that by the time it reaches the LLM it already has much clearer context. But that's part of the development, so... Even when we reached the final stage, where the product was ready, there was the LLM and three more models before it that first generate all the context it needs — every data point, relevant data — and then it's much easier to give the right answer. There's more here in measuring the user and in measuring the models themselves, which are more classic-looking, and we knew how to measure them, for instance. A model that's simple linear regression — I know how to measure it, I have precision and recall I know how to work with — but yes, there we put more emphasis, because the LLM models, I think to this day it's hard for us to put the real AI-evals in place, but I think we're getting more sophisticated about it. Okay, so bringing us really to the end — how was it, how was the launch at the conference? — So it was amazing. I skipped over all the KPIs and all that, yeah, it's so much time, but yeah, totally. So it was amazing — a product came out, clients were very excited, a lot of hype was created, and if I take us a few months forward, we also decided to make significant changes. With all the hype created, what was hard was to persist on the client's side, where it reached a stage where they saw it works nicely and then they need to raise it to their users, and we're working with clients who are B2B, they have tens of thousands and hundreds of thousands of users a day, so they too work at very high rates and have a very big fear of making changes on their sites. So for some of them there's no A/B-testing capability, say, to expose it only to a portion of users, so suddenly they didn't want to test it, some of them were afraid — it's an interesting thing. There were fears — bigger than that — that our chat would curse their users, like they saw on Twitter that GPT or Gemini, Google's model, once became antisemitic, and GPT started cursing users, so complaints started about that. It was also hard to show them the guardrails there. On top of that we reached areas of real challenges of languages and translations and users' slang — the fact that the model supports English doesn't mean the user in Ireland won't use their own slang. So from there it opened quite a few challenges, and to keep the long story short, I can say that in the end we did a pivot on the product, and today we're taking it more toward being API-based, toward the social world, and less as a widget — because this area of injecting our product into the users' feeds, we understand that in our market it has many more barriers, and it's hard to deliver value there — because we know the product delivers value, but it's hard for the client to adopt it. So we're taking it to areas that are easier to consume from us — the more programmatic, API-based and service-based areas — so that they can get the same data and control the presentation to the world. users, think — wow, think, if you had today's structure, would you have approached this MVP differently, knowing this? Actually we designed it very much to create a wow-impact at the conference, right — it was both the wow-impact and a matter of timing in the market. I'll jump a year and a half forward — but in that time the market also advanced, and the market too... So at no point in time do I think it's the wrong direction, because in the process we did the process and also saw the learnings. Relatively, if you look at building a product from 0 to 1, it was in very short cycles, even though we invested a ton in building this MVP. And there's also a matter of maturity in the market and this insight, and also the ability to take new products and test them in the water — I think that's very instructive. By the way, it also opened a new business for us that we hadn't even thought of. But yes, the process itself also had a lot of insights, it also let us experiment with this technology, and for it to be very, very controlled — which is what we call, ultimately, the combinations we like: to take risks but know how to manage them well. It also gave you another insight — I think you'll never in your life make a product that's basically in your clients' wheelhouse; you understand it's a play that's not worth it, probably, right — it's much harder. I don't know if we won't do it, but so to speak, we... understand much more, understand the meaning also for our clients of entering this area, and that's fine — when we know what we're up against, we'll do it if there's a good opportunity, and we can seize it much more effectively. What an amazing story, well presented. So we actually learned another thing about skills you hold dear — that's think big; you basically thought big here, you went for it, you said... It's been a really interesting conversation. I want — maybe now we really said it's a good time, after a bit — I don't know, a war ended, I'm a bit drained. So, Ashkelon — yes, are you hiring? Yes, maybe tell us a bit what you're looking for, which roles, maybe we'll put a link afterward. Gladly, I'll share the link. We're hiring in almost every team today — developers and data engineers and data scientists and product managers, and business people and sales people and marketing people and customer-success people — super-interesting roles, some in Ashkelon, some in Tel Aviv, so it's worth checking out. It's a fit for anyone who loves the sports domain; second, it's a fit for anyone who loves a company that runs very fast, and what I generally think... And it's not just a product team that reports to product — it's a mindset that starts from the CEO all the way down: 'let's think big, take risks but know how to manage them,' but also do it in a more close-knit, family atmosphere without bullshit — and yes, totally, bullshit is also... It's also fun to see the investment there is in the southern region — I live in Rishon, but originally I'm from Ashdod, I grew up in this area — and to see a company that also ideologically invests in this. And you said earlier the war ended, thank God, but this period was also very challenging for us as a company in Ashkelon — many of our employees live in the border communities and many live in Ashkelon, and people were away from home for a very long time, and we have quite a few reservists. We really love reservists — it's also something we make a lot of effort to support them; we have a company where truly, since October 7th, we barely saw them for months and months, on reserve duty. So knowing how to manage all this, in a company that's also outside the central region we're used to, has its challenges, but I think it gives a lot of advantages. And thirdly, the fact that I'm constantly beating the traffic — that's the biggest advantage, that's a very significant saving in my life. Great, good — I'm also totally for the south, but let's give one more nugget of info: you told me about bootcamps you teach — for how long? In short — whoever wants to follow you and hear more about this — basically LinkedIn, right? Yes, LinkedIn, and LSports' page. You see a lot of innovations, posts from meetups and lots of events, and yes, we also have a very strong content team, so it's hard to miss everything we post, so it's worth following. Great, so look — bootcamps for them too in the product domain, which until now there weren't, but there was the good idea and the whole thing came together — I'd be glad to also find talent everywhere, and especially if there's talent in the south you can develop, I think that's a good idea. And yes, we also often take on product people from the army, whom we really love — both military graduates, and we have a lot of friends there, so yes, we try to foster and build circles here and also give back, because we believe in that too. Great, wonderful, glad we found a moment for that. One sec — it was important to me, let me just close one more thing to wrap the episode: what do you wish for yourself a year from now? I can say first, I wish all of us a bit of quiet in this country, so that we can also really focus on thinking ahead, in a calmer time. So I wish that for myself and for everyone; I think we deserve it after this crazy period. And the thing I wish for myself is to be in a place that's always moving forward, advancing and improving all the time. I really believe now in LSports — I really believe in building this different thing, not just another Tel Aviv startup. So I do wish myself to at least simply be a good product manager with what I have today, and to keep improving all the time and thinking big, to be in this place that seeks challenges and chases after them — it doesn't always work, but it always keeps you in the game. So yes, that's what gives me a lot of fun. Great, Itzik — so you'll come back to us in a year? It was fun. For sure, friends. Thanks for listening. If you found this podcast valuable, you can subscribe and follow us, of course, for more episodes on Spotify, Apple Podcasts, or any other app. And of course, if you didn't find us on some app, I'd be glad if you'd write to us. We'd love five stars on every platform, and for you to follow us so more listeners can discover us and find the podcast. You can also find the previous episodes on any app or on our YouTube channel — we have links in the description. Until next time — come on, effective people, and bye bye.