How Incredibuild accelerates game development | The DeanBeat

A lot of companies say that their tools can help offload work from game developers so the devs can focus on making games.

One of these companies is Incredibuild. I’ve always had a tough time grasping and understanding exactly what these companies do. So I sat down with Tami Mazel Shachar, CEO of Israel’s Incredibuild, to get a better understanding of the work. They basically tackle chores like trying to spread cloud-based game code across a lot of servers to fully utilize the processors in those servers.

Incredibuild takes game code and breaks it down into pieces and determines the best way to execute it across distributed computing platforms, whether that’s high-end AMD Threadripper systems with 64 cores or simpler CPUs with 16 cores.

Bandai Namco uses Incredibuild to get rid of nightly build runs, where rendering and other kinds of processing work has to be done in a time-consuming batch. Now the company can accomplish some of the same tasks in just 40 minutes.

I interviewed Mazel Shachar at the Devcom event last week in Cologne, Germany.

Here’s an edited transcript of our interview.

GamesBeat: Tell me about Incredibuild, including what you do and who is the real competition.

Tami Mazel Shachar: If you look at Forrester or any of the analysts, we don’t have a rubric of our own. We would probably touch four different ones, which would be build acceleration, cloud enablement, dev tools, and dev ops in general. You don’t have one competitor, because what our technology does is accelerate workloads by using idle CPUs or the cloud elasticity in order to have a supercomputer using all of the CPUs around. We also do reduction of time, but we do reduction of cost. We improve the workflow of the developers.

If I look at competition, I can find competition for each of those different areas. If you look at distribution or parallelization, you could find open source like Taskbuild, or even SMDBS by Sony. On their SDK they have something that helps developers build or compile better. If I look at cloud enablement, I have many players that are helping with auto-scaling of the machines and how to use cloud better. In orchestration you can look at Bazel as a build system that does distributed in Linux, although not in Windows. So we don’t have direct competition. I don’t think you’d find a market report. That’s the beauty of Incredibuild.

In a way we’re creating a category. In my vision the idea is to get an end to end dev acceleration. How we can take the whole dev process, not only build and compilation, but later on the testing and everything in between. The code signing, the static code analysis, everything like that. All of that gets accelerated. The reason gaming is our world is because the pain in gaming is the biggest. All of the graphics, the shader compilations, all of that. It’s a huge pain.

This is how Incredibuild grew. We started in gaming, and all of a sudden we have 600 game companies using us, like Epic and EA and Sony and Bandai Namco and Konami and Square Enix. But also many studios, even smaller ones.

GamesBeat: Something like Playfab that Microsoft owns, is that touching on some of those areas? Or are they doing different things with the back end?

Shachar: They’re very different. Microsoft, by the way, is a very strong partner of ours. They have put us in their Visual Studio. Not commercially, but in Visual Studio. When you choose to do a build, you can do it with Incredibuild. Microsoft is a good example where they see the value that we bring to their developers. The game studios at Microsoft, like The Coalition and others, all use Incredibuild. But also we’re working with the Visual Studio team and the dev box team to make tools that improve the experience of the Microsoft developers.

We’ve even introduced caching lately, because our technology is based on parallelization and then distribution. We take the workload, like C++ or Unreal code with millions of lines, and the first thing we do is break it down into smaller processes that can run in parallel. Then we scan the network. If it’s on-prem, every idle CPU, and if it’s in the cloud we spread it across smaller machines. We run it in parallel, and then you bring it back. This is how the technology works.

We got a patent in the U.S. in June on our caching. It means that before you do the distribution, builds are done incrementally. You don’t always change all the code. Everything that is cached, you don’t rebuild it. You only start with the parallelization and distribution of what was not cached. This is something that all the developers are excited about, because it’s such a bottleneck. It’s so frustrating to wait so long for those processes to happen. The idea is to find more technologies and more ways to improve that experience, because graphics are only getting more complicated.

People talk about the metaverse, or maybe not now. It’s a question of trends. But the quantity of compute is just continuing to soar. The compute power isn’t following suit. Look at everything that’s happening with Intel and AMD. What happened to Moore’s Law? We should have been here with an average computing power of, I don’t know, 128 cores per machine. Most developers have 16-core machines. Maybe the rich ones have the 64-core Threadrippers from AMD. But all of them, they don’t have enough compute power. Without software, there’s something it doesn’t meet. It’s a very important space.

GamesBeat: Are you multithreading?

Shachar: Yes, we support multithread as well.

GamesBeat: It’s still slightly amorphous to me, exactly what you might do. Which piece would be competitive? If Playfab isn’t competitive, is there some other piece at Microsoft that’s more directly competitive?

Shachar: Microsoft has a lot of tools for developers. They haven’t invested in distributed computing. Right now I don’t think there is one piece of software that plays the part that we do. But as I said, the competition comes from getting better in other places. If you have tools that are doing great orchestration of the pipeline, it works very well. But they’re still confined to the power of the machine they’re running on. The secret sauce of not being confined to that machine, but being able to use the full power of the network or the cloud, that’s very unique to us.

Our patent, by the way, is going to protect that in the future. But it doesn’t mean we’re not concerned about hardware getting stronger at some point. At least for now that seems far away. It’s not happening. Or having better–Bazel, for example. Bazel by Google is an open source option–it’s a build system. As part of the builder, doing some sort of distribution, that’s a threat. We’re looking at that.

GamesBeat: What about something like Improbable, what they do? They do talk about a distributed OS to do something like–the Yuga Labs folks, the Bored Ape Yacht Club, they’ve been doing about 20,000 concurrent players in the same 3D space, where you could only get 100 players in, say, a Fortnite game. They’ve been doing collective play events or things like virtual concerts.

Shachar: I think the difference is that we’re almost on the CPU level. We’re really low-level in terms of what we do in the processing of the development, not in the production or the front end. On the production level you have a lot of systems doing it online and they have all kinds of ways to optimize. We’re in the back end, at the bottom of the funnel. In the development area, it’s an area that’s a real bottleneck. There are not many solutions addressing that. All of the multiplayer, the improvements they’re doing to make online games quicker, they’re on the upper level. This is an area where you have a lot of players. They’re looking at how to improve the play experience in real time. We’re in the dev area.

GamesBeat: At the assembly level?

Shachar: It’s low level, CPU level, and going forward also GPUs. Trying to get that processing power that is needed in order to do the development. What happens today, people need to launch games more quickly. They’re expected to launch with more features and high quality. Nobody wants to have any glitches in their game. Then you get to a situation where games have more advanced graphics, more features and so forth, and the company needs to decide. Do I launch it with bugs? Do I launch it with fewer features? How can I shorten the time of development? That’s the big difference between the environment of the game, which is real time, and how you get there.

A good game takes how many years to develop now? Five or six years? Just imagine how much pressure is on the economic side, on the development, on the hardware and so forth. It’s important to make it much quicker. We’re a bit different.

GamesBeat: Is it a matter of developers knowing how to write single-threaded code, and then you can take them to 16 or 64 threads more easily?

Shachar: We have to look at the areas we support, which are basically all of the C++ and graphics that are based on it. Take the Unreal Engine. The code they write is the base of the game. If they can run multithreaded or use VM containers or any other way, that’s wonderful. We support all of that. We’re one level below that.

You write your code and tell me which machines you want to run on – if you’re multithreaded, if you have VM, if you have containers, if you’re doing it in the cloud or on-prem – we’re going to take that, look at the code, and see if we can break it down. That’s the first thing. Are there dependencies or linkages? Once we take the code and break it down–this is why gaming is our biggest area. In gaming the code itself is very heavy. They’re looking for all the solutions you mentioned, how to run it on the machines, the virtual machines. But the code itself remains millions of lines. Unless you can process them in parallel, you’ll always have this long line of waiting for the processor to manage it, with all the power you have.

Because of the way we run it, some companies–I think Bandai Namco is a good example. We have a case study. They took build farms in order to run Incredibuild, because they don’t have nighttime builds anymore. They used to run builds at night, come in the next morning, see the bugs, and then test again. Now they do everything on a daily basis, because they get it done within 40 minutes with the power they have. They have more iterations, more quality, more features. This is how they do it.

The mindset is to understand that we’re not–if you run it any way you want it, whether multithreaded, containers, VMs, Kubernetes, whatever, in cloud, that’s just what we use in order to make it quicker. We adapt to the way the customer or the game company is doing that.

GamesBeat: Is Coherence anywhere in the same space? Making multiplayer easier for game companies?

Shachar: All of the companies–some of them, as you see, I don’t really know them. But they’re in the space around how to play in a better way in real time. They’re less in the stage where they’re actually coding. Games sometimes have 30 million square kilometers of terrain. We’re helping Mainframe. They work with us and AMD, because AMD provides them with Threadrippers. They were still stuck. They needed our help, because this is the type of terrain they needed to compile and make into the world. It doesn’t matter what you do. This is the quantity of code we’re talking about. And it’s only growing. Every game we look at, the immense requirements of the 3D and the terrain and so forth, it’s always getting stuck on the most simple thing. How long is it going to take me to code it, to make the game?

The player experience, that’s where most of the other companies are working. This is why we’re a bit different in this space. Our buyers, our customers, they’re the R&Ds, the architecture, the CTOs, the release managers, and the developers themselves. We work with small studios of four or five people, or with the Epics and EAs of the world. We have a solution for very small companies. We have a free solution for smaller studios. We even now have the Megagrants tied in with Epic, so they can use that to start working with Incredibuild for free. It’s on Visual Studio. The idea is to help smaller studios keep from having to buy too much hardware or invest too much in the cloud. They can start working and reduce what they need to invest in the power, and then we walk them all the way to being much bigger companies.

GamesBeat: If the metaverse were to come back in fashion, would you be able to make that happen better?

Shachar: Definitely. Our last round of investment–we’ve gone through two rounds. Hiro Capital, they invest a lot in gaming. One reason they came in, they said that Incredibuild is essential for the metaverse world. What does metaverse mean? 3D. Huge 3D models that are going to require intense compute power. The compute world is at a standstill. There’s no way.

We work, by the way, with Intel and AMD. You would assume they are competitors, but they came to us and said, “Look. We can’t meet the compute requirements of developers right now.” Developers are moving to the cloud. Maybe in the cloud they’ll have more elasticity. But the cloud is trying to develop their own hardware. The hardware vendors have commercial issues. They want to keep control and give the best experience to developers so they don’t all run to the cloud quickly. Part of that is saying, “Please use software like Incredibuild. You’ll get much more than we can do for you right now with our hardware.”

We have a great collaboration both with the cloud vendors, like AWS and Microsoft and GCP, and on the other side we have a great relationship with the Intels and AMDs of the world. All of them see how we can help them mitigate the constraints of their customers. Cloud processing is really expensive. Almost no dev is run in the cloud. Only the production. Developers look at these huge costs and they don’t know how long they can keep that up. It takes a lot of time and you pay as you go. But if you do it with Incredibuild, you pay maybe 50 percent less, because instead of trying to book those 64-core machines, 128-core machines for so long, you take the same workload and use four-core machines, or even spot instances, and you finish within minutes.

They pay much less, and the cloud vendors, although they make less money, they’re still happy. We create utilization of the low-core machines, and we also get people to move to the cloud when they thought they wouldn’t. Once they’re in the cloud they’re in the cloud. Same with the hardware. Those who want to stay on-prem and think they’re never going to be able to run everything they need to do, they use Incredibuild, and suddenly even their old hardware, those small machines that are still there, they give so much value that they don’t need to buy more hardware. It’s also a green effect. Think about the carbon footprint and throwing out old hardware because it’s not useful anymore.

GamesBeat: On that front, do you think that collectively, computing is going to be more efficient? I don’t know whether the demands of something like the metaverse or gaming are going to drive us into more difficulty.

Shachar: The GPUs are going to play a stronger part in the market. There’s a big compute problem. It’s getting bigger. Even AI is showing that. We’re talking about the metaverse, but let’s talk about AI too. The need for compute is only growing. There’s no answer. I met someone in the semiconductor industry recently. He said that there is a physical limitation to the nanometers in the silicon right now. We’re stuck on that. If they don’t overcome it and find a different technology–quantum computers are not relevant. They’re for different uses. The costs are different. But there’s a technology barrier. The answer needs to be software.

Incredibuild plays an important role here in the ability to drive efficiency through software on hardware. But it’s going to call for innovation in general. You’ll find innovation in that area. The world is moving so quickly in its requirements for compute, and the compute world is not following.

GamesBeat: The area you attack is an inefficient area, it seems like. Other companies that were trying to deliver the metaverse, people like Subspace–I don’t know if you remember them. They went out of business. But they were trying to find alternative routes around the hardware bottlenecks of the internet. They were trying to set up a parallel internet with separate hardware networks wherever there was a bottleneck and route traffic through that, both for online games and eventually to enable the metaverse. It’s interesting to see that it was such a big job that you couldn’t do it with a company like that. They just ran out of money.

Shachar: It’s true. I think the metaverse is not only about compute. You’re right that it’s also about bandwidth. One of the limitations, by the way, especially when COVID was here and everyone was working from home–you could have really strong compute in some areas, but if the bandwidth didn’t allow it, you couldn’t use cloud and other things properly. That’s another issue that the world has not solved yet. Another area to look at is the file transfer technologies. How do you transfer huge amounts of data from one place to another?

All the companies working on transfer–I think there’s a company called Resilio doing that, and others. This is another important area. It opens up a completely new market for innovating in how to deal with heavy compute, with huge files, with graphics beyond the power of what the cloud and the current on-prem world can do.

One thing we see in the recession that’s happening right now, or whatever you want to call it, is that on-prem is gaining traction again on the development side. We see it from a lot of customers. They say to themselves, “You know what? Cloud is way too expensive for development. Let’s try to use our hardware as best we can.” Costs are still very high. That’s a big issue. But I definitely think you’re going to see a lot of innovation around everything that can support those huge compute power requirements for all of the futuristic worlds like metaverse and AI and IOT and anything that comes around.

GamesBeat: Do you feel like you’re succeeding in communicating to all the right people?

Shachar: Incredibuild has gone through an interesting journey. The company was very low-touch and under the radar for a few years. It was founded by two developers. The way that we gained a bit of our brand in the gaming world was just because gamers used it. They loved it. The developers themselves. This is how we grew. We have around 600 customers in gaming. When I meet people they say, “Oh, I used to use Incredibuild when I worked at this place before, and then I moved here and I’m using Incredibuild again.”

The company grew after we got funded by Insight Partners and then by Hiro Capital. Suddenly we’re trying to build a go-to-market and accelerate growth by bringing more value, developing more features, but also growing out of gaming into areas where this technology is also relevant. If you look at automotive, for example, they’re using the game engines today to design cars. Unreal, for example, is used in automotive for simulations, in architecture, in finance, in different areas.

We’re building the company, and it’s hard to build a company. It’s always hard. We have around 200 people today. We’ve opened offices in Tokyo, where we have more than 100 customers. We have offices in Chengdu. We’ve opened our U.S. office. We’re trying to build a footprint and be close to the customers so we can understand their needs. We’re trying to create communities where we can pick up the voice of the customer. It’s a hard job to, on the one hand, be a company and be successful, but stay close all the time to the needs of developers.

CTOs immediately get it. They understand the challenges. CEOs, if they come from the technical side, they probably already know as well. In some cases we reach the CEO level, the decision-making around buying Incredibuild, at our larger gaming customers because it becomes a strategic tool. But the CTOs are usually in the process. It’s part of the technical chain, understanding the technical challenges and how to solve them. The reason why CEOs are involved, or even CFOs, is because of the economic savings. When you look at the ROI and you realize, “Wait, I’ll spend $200,000, or even a million on Incredibuild at a large company, but what am I getting for that?” Well, you’re saving 40 percent on your cloud costs. You’re saving 10 percent on the productivity of your developers. You save on your hardware. Then you see that the ROI is huge. That’s an economic, strategic benefit.

At the bottom of that you have the developers who are saying, “I’m just getting my time and my focus back. I don’t need to switch contexts all the time, moving from one thing to another because I’m waiting for things. I’m not frustrated.” All of those parts. The experience of the developers. It’s hard to get developers today, everywhere.

I think a lot of people don’t see the main issues that are blocking innovation, where a lot of the ideas should come from. It’s a very interesting area. It might be less shiny and beautiful than the games themselves, but it’s under the hood. We’re under the hood. I came from semiconductor. My first job was in a fab. You see how the advances in the silicon and semiconductors were like this, and now they’re like this. It’s amazing to see through our lifetime. Software was basic and now software is soaring. We’ll have to see where the innovations in software will come.

GamesBeat: Consumers are wondering where the slim PS5 is. The mid-life smaller consoles. They’re not here.

Shachar: Even the Oculus, the VR hardware, it’s not going anywhere. It’s not going anywhere because it just doesn’t work. It’s tech. There are real technology barriers to getting there. The quantity of data you expect to run in order to get that seamless experience, there’s not enough compute power. There are a lot of issues. This is part of what’s blocking VR and AR as well.