Yesterday, The New Screen Savers aired an episode featuring a Pokemon Go map (PoGoMap) developer Michael Christopher and npm (*) co-founder and Chief Technology Officer Laurie Voss who went on to explain Pokemon GO Server issues and the way PoGO Map is made.
npm is a distributed package repository for Node.js, a huge hosting site for all Node packages with thousands of concurrent users, so having Voss explain problems with scaling software products was eyeopening.
Video link: THE NEW SCREEN SAVERS 63 MAPPING POKÉMON GO
Key points from the video:
- Server issues explained – 17:30
- Scaling challenges explained – 24:00
- Pokemon GO tech stack explained – 26:00
- Shots fired at Nintendo and Niantic – 26:30
- Niantic has no opinion on the PoGO Map – 27:30
- Niantic doesn’t know what app is using the API, but they know what is the account that’s requesting – 28:30
- Bans are comming – 29:00
- /r/pokemongodev goals – 30:00
Written Summary
The interview starts around 17:00 and the first interesting question about Pokemon GO server issues and how distributed systems like Pokemon GO work comes up around 17:40 minutes:
Host
“Why couldn’t Niantic just throw in a bunch of servers and fix the issues?”
Voss
“People who work at Niantic are ex-googlers, they know how to build things that scale. What’s more interesting is the way how the game is built – it’s built with specific choke points, the login server for example.
The login server is typically not the busiest part of your game so you don’t expect it to fail. But when it fails, people are continually trying to get back in, making the issue even worse!
It’s a very common problem – if the login server is fine, everything is fine, but if the login server is flaky, everything goes down.”
Host
“So these are cascading failures?”
Voss
“Absolutely, that’s how a big system like this fails – one thing that shouldn’t have happened happens and then all the other things fall down, even though you thought it would never happen!”
Next up was a question about scale and reliability at scale and Voss went on to defend Niantic saying that “no body could have known how big it will get”. The interesting fact that came up was this question:
Host
“Can’t they just buy more servers?”
Voss
“It depends on how you architectured the thing. If you made it so it can scale horizontally, then it should be easy to make it twice as big by spinning twice as many servers. But making an app that can really scale horizontally and has no single point of failure is really hard. Login and authentication is often the problem as you can not discrete them, right, like the definition what your password can not live in two places at once, it must live at one point of truth – the login server. This is where you usually hit the point of failure in distributed systems, especially if you were thinking you’re going to get away with it.
And that’s not a problem you can fix in two weeks by buying new servers – you have to re-architect.”
Michael jumped in after Voss and confirmed that reverse engineering showed that PoGO backend is still operated by Nintendo and continued to say that “Nintendo was not well equipped”.
Michael continued to explain that Niantic has not issued a statement yet about his map, but he admits he is breaking Terms of Service as they did reverse engineer the API and are currently using the public API that the official app also uses.
He also went on to explain that Niantic doesn’t know who’s pulling data but he does expect them to eventually implement mechanisms for banning players that do.
The interview went on a bit more with discussions about various Pokemon GO server issues and ways developers are currently using the API.