Posted by TitanFall 2
After wrapping up our first weekend of the MP Tech Test, I wanted to sit and talk about what we did, what we learned, and how this helps us for launch.
The goal for the Tech Test is to stress our server infrastructure to make sure it’ll scale and handle common events that are seen during a game launch. Despite our best attempts at writing massive simulations, there are gaps and assumptions that will come out when you have real players. We are very focused on having a smooth launch on October 28th and only a real Tech Test could answer these questions.
We are doing these two weekends of the Tech Test because we knew we would hit problems that we’d never seen before. While a massive number of people were hard at work getting wins this weekend, we were busy finding and fixing at least a dozen separate issues. Those are all problems that we won’t have to find and fix at launch. Many issues that we identified and fixed were things that users never noticed, which was one of our goals.
Given the way game certification works and the timelines involved, the Tech Test was from a June snapshot of our game, and some of the feedback and issues fans noticed this weekend have been fixed in our game for weeks or months now. In addition, the full game has a lot more Tactical abilities, weapons, unlocks, customization, Titans, and maps than the Tech Test build does, and that slimmed-down view of the game for a Tech Test can be its own challenge for balancing and learning curve reasons. More on that later – the designers are going next with their own blog update – this is focused on the engineering goal of the Tech Test.
Let me go through a few critical systems that we tested this past weekend:
First – Dedicated Servers
For Titanfall 2, we are running on a new server infrastructure with a company called Multiplay, who is responsible for running our fleet of dedicated servers to host games. And we run a LOT of servers – many thousands of servers all over the world. We also run them dynamically, so we boot them up before people need them, and shut them down when they aren’t needed anymore. We learned a lot this weekend about how fast players will join matches, and we’re using that knowledge to verify that we can spin up servers really quickly so players aren’t left waiting.
We made sure to have a lot of datacenters for Titanfall 2, to give us the ability to scale very rapidly as well as to move games to another datacenter if we hit issues. We actually tested datacenters going offline completely and we learned how quickly we could move 100% of users from one datacenter into another datacenter. We weathered network attacks as well using this same system. After having done it a few times, it gives us a process to follow so that we can move players around even faster in the future. And for launch, we’ll have even more datacenters available!
We collected useful information about how long it takes each datacenter to spin up new servers – some datacenters are faster than others, and since we are spinning up servers in advance of players needing them, speed is important.
Second – “Stryder”
Aside from servers, we also have our own “Stryder” collection of backend services.
As players saw, our new matchmaker is FAST. We learned that we could reliably make up to about 24,000 matches every 2 seconds, which thankfully allows us to handle player counts that no game sees. We did see some problems with matches not being full on startup, so we both improved our Join In Progress path, and we dug deeper to find the root cause of the non-full match starts. Our matchmaker had a trial by fire this weekend, and we’re really happy with the results.
Networks were a big unknown for us – we didn’t know how people would use them, or which parts they would use most. We had over 200,000 networks created by Sunday night, and we saw thousands of users in chatrooms creating parties and going into matches. We were amazed by the creativity and fun that gamers brought to the game through their own Networks.
We had some times where we were fixing issues with the brand new Invite Network functionality, and we could actually see the impact of those in the player counts in real-time! We saw a huge amount of players who ONLY joined games using the Invite Network popup in lobbies. With the Network Invites, Happy Hour, clan tags, and Inbox functionality, it seems like people really embraced our new Networks system. This was a huge success for us, and they were something that was hardest to verify before this weekend. Networks were created for real humans to use, and no simulation can prove that it would be successful like real people can.
Thank You!
You guys and gals played so much of the game this weekend, and you gave us so much data and feedback. It’s invaluable as we go towards the home stretch of getting the game ready for October 28. Thank you for all of your hours played, feedback on our forums and on social networks, and disgustingly great skill displayed in matches against us. We are all trying to deliver the best, most fun game we can, and the Tech Test is a huge opportunity to learn from you.
We got a ton of feedback from players this weekend. I want to thank some really great people who told us about technical issues they spotted, which helped us find issues and verify when they were fixed. We also received a lot of useful feedback not only on technical issues but also game design & balance. While the content available in the tech test constitutes only a slim portion of the full Titanfall 2 experience (so many more maps, modes, weapons, tacticals and Titans to come, plus an entire single player campaign!), we take that feedback very seriously. Our multiplayer team is interested in going into detail about their decision making process and design goals for Titanfall 2, to provide more context on why things are how they are and addressing feedback. Stay tuned for that update soon.
Thanks everyone for all of your effort, patience, and feedback. It means a lot to us.
Weekend Two – August 26 to August 28:
We’re going to do some fun things for weekend two of the Tech Test. With the learnings from weekend one, our next goal is to really spike the number of players playing online at the same time. We’re going to try to do an event this weekend to try to get everyone online at once, which will be hugely valuable to the engineers here at Respawn as we continue to prove that we are prepared for a big launch this fall. Stay tuned to the Titanfall social channels for more details on this event. Until then, we have a list of things to look at before the Tech Test kicks off again on Friday morning at 9am PDT / 4PM GMT. We’re very excited to see you back online again in just a few days!
XOXO,
Jon “Slothy” Shiring
Lead Engineer
Titanfall 2
