Secret values

Scraper code on is public and visible to anyone. So, how can you use an authenticated API, say the Twitter API, in a scraper without letting everyone know your authentication credentials?

This is where secret values come in. These are environment variables you can set. Only you and your scraper can see them.

Setting secret values

Let's say you have a secret value sdf8sdbw9Hihf you want to use in your scraper. We will now assign that value to an environment variable. We'll call it MORPH_MYSECRET. The name needs to start with MORPH_.

  1. Go to your scraper and click the "Settings" button
  2. Click "Add variable" and fill in the environment variable name and value
  3. Click "Update"

Before you clicked "Update" it should have looked something like

You can of course set as many different secret values as you want. You just need to click "Add variable" again and give each secret value its own unique name.

Reading secret values in your scraper

  # Use my secret value
  my_secret_value = ENV['MORPH_MYSECRET']
  // Use my secret value
  $my_secret_value = getenv('MORPH_MYSECRET');
  import os
  # Use my secret variable
  my_secret_value = os.environ['MORPH_MYSECRET']
  # Use my secret value
  my $my_secret_value = $ENV{'MORPH_MYSECRET'};
  // Use my secret value
  my_secret_value = process.env.MORPH_MYSECRET;


Sometimes there's nothing like seeing a real-life example. If you have a scraper you would like to add to this list, please let us know.

Cat Photos - Example Python Scraper with CKAN API upload
Scraper for Australian food product information with barcodes