Hey you. It’s been a while since you read something nice and heartwarming on GO Hub. We got so caught up in the news cycle, that we lost touch with the humane between us. Let’s catch up, dear reader.
You’re right – we should have written more about the psyche and the deep problems that you faced in 2020. Not just you, mind me, everyone. I don’t often share the editorial stories that happen behind the scenes – in fact, I never do. Well, today is different.
GO Hub nearly died in 2020
You don’t know this, but on July 21st, 2020, GO Hub nearly died. Since life is not a Marvel movie, our death would probably be permanent. Sucks, right?
JellyBean, our resident “Hub mom”, messaged me that she and Alistar can’t do this anymore. Jelly and Alistar have been with GO Hub since forever. You rarely see them in the wild, but they are there, always watching, always judging.
I can’t run the Hub without them. Hell, I wouldn’t know what time of day it is without them. You see, much like the rest of GO Hub’s staff, they’re incredible people. The GO Hub staff are my sanity check, my Hail Mary pass, and the people whose opinion I value more than my own.
Honestly, without its staff, GO Hub would be just another crapsite on the interwebs. Like vinegar makes salad taste like “not garbage”, it’s our writers and moderators that make GO Hub palpable. Not me, and certainly not the spaghetti code that runs the database. Which I also wrote, for the better or for worse.
As you have probably gathered in 2020, people are not replaceable. Everyone is someone’s special person. Yes, even your annoying nephew that slurps mash potato on family dinners (remember those?).
I’ve known this idea for a while now. Little did I know that Jelly’s message would kick the shit out of me as much as it did.
Setting your priorities right
Truth be told, when Jelly’s message arrived I was in a middle of something else. Another emotional horror – personal by nature, but equally as scary. The deadly combination of these two situations – the standstill – was terrifying.
People often neglect how difficult decision making can be. We undervalue ourselves and we forget what are our priorities. That day, Jelly’s message woke me up. Like a slap to the face, the virtual Hydra of bad management and my low engagement at GO Hub was rearing its ugly face at me again.
What do I do? Fight or flight? I chose to fight – but not for everything. I chose to fight for GO Hub. Not for myself, not for anything else in my life, but for GO Hub. For Jelly and for Ali, I wanted to make this place a good place for them. For the writers, whom I wanted to promote and celebrate to you – the readers. For the mods, who lacked the proper moderation tools for far too long.
Over the next few days, I’ve gone in-depth with most folks at the Hub at what’s wrong, what’s the issue and how I can help. Coronavirus, general anxiety, feeling of uselessness left and right. It was a bloody nightmare to tell you the truth. I hate talking with people in that capacity.
Somehow, we fought through those issues. I have no idea how. All I remember is talking, working and talking some more. But that’s fine, I don’t understand Inception either.
What I’ve learned is that it works. Talking works. Working hard works. People are not replaceable, let them know that. People – humans – matter. Cherish that, for fracks sake, cherish them. Yes, even on the days when you hate yourself.
And so I did, day by day.
For months we steadily rebuilt and improved the Hub, culminating in our recent redesign. Hard battles were won, even if the foes were strange and unfamiliar: depression, lack of enthusiasm, lack of ideas, lack of motivation.
Our method? Talk. Motivate. Others. Not yourself – we’re all mostly an annoying bag of problems anyway – but other people. And allow yourself a pat on the back now and then. Just don’t give up.
Look at us. Somehow, in this entire mess of 2020, we managed to get our virtual house in order, and have a fully seated family table. Jelly is still here, so is Alistar. So am I. We pulled through. And so did all the writers and all the mods, and even a few returning folks who were re-attracted by a reinvigorated GO Hub.
You can do it as well mate. It’s going to be fine.
But it’s not over yet: almost losing our finances
As if 2020 was not challenging enough, we almost lost our only revenue source: advertisements. The following excerpt was posted on GO Hub’s internal staff channel (aptly called town-hall) on Nov 22, 2020:
@everyone This is the story of the past 10 or so days at the Hub, and what really took place behind the scenes.
It all started on Wednesday, November 11 th. We are usually able to convert around 0.5% of page views into ad clicks in a 24 hour window, give or take 0.1% more or less. This has been a stable metric over the course of past few years for the Hub. However, on Nov 11, I observed something strange: more than 20% of page views coming from Netherlands were converted into ad clicks.
Not only is this impossible in real world scenarios (the best optimized websites have around 2% conversion), the strange narrowness of visitors location raised a lot of red flags. Even more strangely, these clicks were registered against ad slots that haven’t been used since 2018 (at least).
So, as a good soldier boy, I went ahead and pulled down the offending ad slots from the website. However, the clicks didn’t stop. How is this possible? How are we getting clicks on ads that are not physically available on the website? I was perplexed. Went ahead and checked the Adsense forums and noticed that we were not the only ones with this problem.
Folks were reporting ghost clicks from Netherlands, UK and Taiwan. One of the posters even took down their entire website but the clicks continued.
This went on until Sunday, November 15th, with AdSense reporting sometimes higher, sometimes lower clicks on ads that were non existent. Strange as it is, this was either a bot attack against Adsense or against a random group of publishers.
On Monday, November 16th, Google AdSense limited the number of ads we can display on GO Hub to zero due to invalid traffic concerns. Rightfully so, given the ghost clicks, but this was quite a scare. AdSense is our primary source of revenue and without it, paying for server and writing costs becomes impossible. Something had to be done.
And so I went ahead and did the following:
– update and harden our Ubuntu server
– update and harden our runtime environment (PHP version updates and extension updates)
– converted all of our legacy pages into posts
– updated all of our plugins and themes
– replaced legacy shortcodes as best as I could, but there was just so much of it
– made sure that none of our currently served pages had the offending ad code
But the clicks didn’t stop. We were reaching almost middle of the week, and the problem wasn’t getting solved despite all the efforts made to improve the Hub’s infrastructure and security. Once again, I was at a loss. Obviously, some random bot is clicking ads somewhere and it’s not in our domain of influence.
So I sat down and started thinking. Hell, I thought about it for hours and there was only one place where our content could be stored and accessed outside of GO HUB:
Google’s fracking AMP Cache.
For those unfamiliar with it, AMP pages are “trimmed down super fast versions of an actual page”. For example:
Google cached AMP:
And this is where the bot was doing its damage. They were hammering Google’s AMP cache that saves cached versions of AMP posts for up to 6 months. And so I went ahead, followed their dreadful guide and wrote a script that manually purges their AMP cache, page by page, URL by URL.
This happened on Wednesday, just as our theme migration bugs started cropping up. So at one point, we were fighting malicious bots, dealing with content disparity between old and new theme, going forward with a redesign and trying to keep the server afloat. I won’t even go into random CPU spikes that happened throughout the week, but it was terrible seeing our CPU go from our usual 20-30% to 100% on all 6 cores at random times in a day.
So… where are we at now?
Well, it seems that Google is going to be lenient on us and ad revenue is slowly being restored, day by day. We’re not at 100% yet, but last night and today I’m seeing strong improvements and an upward trend. Our migration process has been fully completed and we have a brand new, up-to-date, top of the notch website. I think we did a generational leap with the new design, structure and plugin infrastructure.
Our rendering and network performance has improved quite a bit with these changes and it seems that everything is slowly getting back into place. We went down from ~80000 404 daily errors on Wednesday to around ~21000 today, and this will improve with time and effort as well.
We’ve also updated our core infra across the board, brought Forum back into the herd, promoted writers, fixed the GO bot and cleaned up the server. Fuck, a lot has happened.
I just want to say thank you – this was one of the most challenging weeks since I started GO Hub and fuck me if I’d manage to do anything without your support, dedication, moderation, writing and just plain company.
I love this place. I love our Hub family.
Love you all,
What I want you to do
Make a list. Hide it from everyone. A list of things you did right this year. A list with people who made your life better this year. A list of priorities for yourself. A list of things that were there for you throughout this year.
And stick to it. Don’t remove anything from this list – this is you. This is your core. Add to it in 2021. And don’t give up. You got this.
I also want you to make another list. List of people that didn’t treat you fairly and things that you hated in this year. Get rid of those. ASAP.
Do this, and it’s going to be fine. Trust me – an internet stranger – it’s going to be fine.