Running Screaming Frog on Google’s Own Servers

Screaming Frog is the industry standard SEO crawler, but it can slow down your computer and monopolise your internet connection. By running it, instead, on Google’s Compute Cloud it can run significantly faster and you’re free to get on with other jobs whilst it runs. Moreover, if you can run one Google Compute Cloud Instance, you can run many.

In this part 1 of 2, I’ll explain how to setup Screaming Frog to run in the cloud and then, in part 2, I’ll explain how to take it to the next level and use it to crawl potentially hundreds of websites, simultaneously, faster than you’ve ever done before.

Before you start this part 1, you’ll need:

1. a Google account;
2. a card that Google can charge for use of their servers (you’ll be able to use them for free for a while, but have to enter card details immediately in any case); and
3. a Screaming Frog license. If you’re planning to simultaneously use Screaming Frog on your computer and the cloud you’ll need two licenses. Screaming Frog is inexpensive – support its continued development.

Now, here’s how to get it running:

1. Go to the Google Compute Cloud Console and sign up for the free trial.
2. Create a new project, calling it whatever you’d like.
3. Click on ‘Compute Engine’ ‘VM instances’ and create a new instance. Here are the setting you’ll need:
4. Once entered, you’ll be back on the virtual machine page and your shiny new virtual machine will begin booting. Click on ‘SSH’ ‘Open in Browser Window’ and you’ll have a box like this appear:
5. enter ‘sudo add-apt-repository ppa:webupd8team/java’
6. enter ‘sudo apt-get update’
7. enter ‘sudo apt-get upgrade’
8. enter ‘sudo apt-get install xfce xfce-goodies autocutsel tightvncserver’
9. enter ‘sudo apt-get install libxss1 libappindicator1 libindicator7 fonts-liberation’
10. enter ‘sudo apt-get install cabextract enchant fonts-wqy-zenhei gstreamer1.0-plugins-good gstreamer1.0-x hunspell-en-us libaa1 libavc1394-0 libcaca0 libdv4 libenchant1c2a libgstreamer-plugins-good1.0-0 libharfbuzz-icu0 libhunspell-1.6-0 libhyphen0 libiec61883-0 libjavascriptcoregtk-4.0-18 libraw1394-11 libshout3 libspeex1 libv4l-0 libv4lconvert0 libvpx4 libwavpack1 libwebkit2gtk-4.0.37 libwebp6 ttf-mscorefonts-installer zenity zenity-common’
11. enter ‘sudo apt-get install oracle-java8-installer openjdk-8-jre-headless libatk-wrapper-java-jni libgif7 fonts-dejavu-extra java-common gsfonts-x11 oracle-java8-set-default openjdk-8-jre gsfonts libatk-wrapper-java ca-certificates-java libpcsclite1’
12. enter ‘touch ~/.Xresources’
13. enter ‘wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb’
14. enter ‘sudo dpkg -i google-chrome*.deb’
15. enter ‘wget https://download.screamingfrog.co.uk/products/seo-spider/screamingfrogseospider_9.1_all.deb’
16. enter ‘sudo dpkg -i screamingfrogseospider_9.1_all.deb’
17. enter ‘vncserver’ and set whatever password you like – I recommend not bothering with a view-only password.

You now have everything setup in the cloud – you just can’t access it. To do that you need to  create an SSH connection and install a VNCViewer.

Creating an SSH Connection

We first need to install GCloud SDK.

On Mac OS X

1. Open Terminal and enter ‘curl https://sdk.cloud.google.com | bash’
2. enter ‘exec -l $SHELL’
3. enter ‘gcloud init’
4. enter ‘gcloud login’

On Windows

1. Download and run the Cloud SDK Installer.
2. Choose ‘start Cloud SDK Shell’ and ‘Run gcloud init’
3. In command prompt, run ‘gcloud login’

Once the above is complete, enter ‘gcloud compute ssh [the name of your virtual machine] –project [the unique name of your project] –zone us-central1-f –ssh-flag “-L 5901:localhost:5901”

It will ask you to set a password and then the connection will be made.

Installing & Setting up VNCViewer

1. I like RealVNC’s VNC Viewer, though it doesn’t really matter which you use. You can download it here.
2. Once installed/opened, click on ‘File’ ‘New’ and enter ‘127.0.01:5901’ as the VNC Server – name it whatever you like.
3. Once saved, click on the server’s icon and type in the password you set when you typed ‘vncserver’ earlier. A window should then open connecting you to your virtual machine:

And there you have it – you’re all set to run your first crawl. One important thing to remember is that you’re paying (or using your free trial credits) whilst the computer is running (and only a tiny amount when it’s not) so, when you’re done with the VM, shut down the terminal window, VNC viewer and in the Google Cloud Compute Console, stop the machine.

You’re ready to crawl hundreds of thousands of URLs without having to worry about it slowing down your internet connection or computer. BUT I wouldn’t recommend stopping here. Ultimately the above is a lot of work if all you’re trying to do is stop colleagues complaining about you bogging down the connection. The real power of this doesn’t come from running a single one – it happens when you automate the whole thing, running multiple instances at the same time, giving you a huge speed boost that’ll allow you to crawl entire industries faster than you’ve previously crawled single sites. Luckily, that’s a lot easier than it sounds. So jump over to part 2 here to turn your basic setup into something that’ll change the way you work.