lowndsy / PositionAnalyser

A record of where key brands are ranked


For educational use only, this is an example of a simple scanner that checks a site to see where specific brands are placed in a pricing table. The code is fully commented and has been designed to be easy to deconstruct.

This resource is a real world example of how to make a simple price scanner that gives data integrity through redundancy. Each piece of info is stored more than once in one form or another so everything can be verified and checked against itself.

It does the same job as a person and takes the same server load as one person visiting a few pages every morning. As soon as you start gathering info for more than one brand it probably even saves the retailer bandwidth.

The principles can be extended to any site which displays structured or tabular data, and the code demonstrates how to use the simplehtmldom.php library to work with P and SPAN tags as well as tables.

This is built in PHP for the Morph.io platform, but it could be used in almost any automation tool, or even run locally. The single line of code which needs altering to be platform independent is commented and there is only one dependency.

Contributors lowndsy

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... Injecting scraper and running... Check Complete: 07-09-2017 13:42

Data

Downloaded 1080 times by DataSparrow lowndsy MikeRalphson Mitch1983 IDatiD

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (39.9 MB) Use the API

rows 10 / 38734

id datestamp page position dealname dealdesc contractdesc monthlycost totalcost network brand buylink fullcode prikey notes
1
30-05-2017 14:04
1
Samsung Galaxy S7 32GB Gold £25.00 upfront Double speed 4G
Unlimited minutes Unlimited texts 5GB data
24 month contract - 4G
27.99
EE
Mobiles.co.uk
24 month contract - 4G Samsung Galaxy S7 32GB Gold £25.00 upfront Double speed 4G 24 month contract - 4G Unlimited minutes Unlimited texts 5GB data </p> £27.99 per month £696.76 total cost Double speed 4G £27.99 per month £696.76 total cost See deal
e0b148b15c9a37d5db09ac09be6a9520
2
30-05-2017 14:04
2
Samsung Galaxy S7 32GB Gold FREE
Unlimited minutes Unlimited texts 4GB data
24 month contract - 4G
25.00
Three Mobile
Buy Mobile Phones
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE 24 month contract - 4G Unlimited minutes Unlimited texts 4GB data </p> £25.00 per month £600.00 total cost £25.00 per month £600.00 total cost See deal 0800 144 8349
91d6d9f91aacc94b03c1d232b1081aa6
3
30-05-2017 14:04
3
Samsung Galaxy S7 32GB Gold FREE £15 cashback
Unlimited minutes Unlimited texts 5GB data
24 month contract - 4G
27.99
EE
Mobile Phones Direct
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE £15 cashback 24 month contract - 4G Unlimited minutes Unlimited texts 5GB data </p> £27.99 per month £656.76 total cost £15 cashback £27.99 per month £656.76 total cost See deal 0800 144 8618
c72f58b2b6698f369218b81026c057d2
4
30-05-2017 14:04
4
Samsung Galaxy S7 32GB Gold FREE
Unlimited minutes Unlimited texts 30GB data
24 month contract - 4G
29.00
Three Mobile
Affordable Mobiles
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE 24 month contract - 4G Unlimited minutes Unlimited texts 30GB data </p> £29.00 per month £696.00 total cost £29.00 per month £696.00 total cost See deal 0800 691 1667
6c70bcb5a29ce046a3ee1ff70a48da9c
5
30-05-2017 14:04
5
Samsung Galaxy S7 32GB Gold FREE Double speed 4G
Unlimited minutes Unlimited texts 5GB data
24 month contract - 4G
27.99
EE
Buy Mobile Phones
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE Double speed 4G 24 month contract - 4G Unlimited minutes Unlimited texts 5GB data </p> £27.99 per month £671.76 total cost Double speed 4G £27.99 per month £671.76 total cost See deal 0800 144 8349
ddcbf805939738642589342338af9592
6
30-05-2017 14:04
6
Samsung Galaxy S7 32GB Gold FREE £25 cashback
Unlimited minutes Unlimited texts 8GB data
24 month contract - 4G
28.00
Three Mobile
Affordable Mobiles
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE £25 cashback 24 month contract - 4G Unlimited minutes Unlimited texts 8GB data </p> £28.00 per month £647.00 total cost £25 cashback £28.00 per month £647.00 total cost See deal 0800 691 1667
c5561ad3ffa59c27e8eeb1907a218ee5
7
30-05-2017 14:04
7
Samsung Galaxy S7 32GB Gold £50.00 upfront Double speed 4G
Unlimited minutes Unlimited texts 2GB data
24 month contract - 4G
22.99
EE
Mobiles.co.uk
24 month contract - 4G Samsung Galaxy S7 32GB Gold £50.00 upfront Double speed 4G 24 month contract - 4G Unlimited minutes Unlimited texts 2GB data </p> £22.99 per month £601.76 total cost Double speed 4G £22.99 per month £601.76 total cost See deal
21ef6f9485d1b39a953871e396b02c91
8
30-05-2017 14:04
8
Samsung Galaxy S7 32GB Gold FREE
Unlimited minutes Unlimited texts 30GB data
24 month contract - 4G
29.00
Three Mobile
Buy Mobile Phones
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE 24 month contract - 4G Unlimited minutes Unlimited texts 30GB data </p> £29.00 per month £696.00 total cost £29.00 per month £696.00 total cost See deal 0800 144 8349
bf904b63f604d4e52ddc0cc676d5a2fe
9
30-05-2017 14:04
9
Samsung Galaxy S7 32GB Gold FREE £15 cashback Double speed 4G
Unlimited minutes Unlimited texts 5GB data
24 month contract - 4G
27.99
EE
Affordable Mobiles
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE £15 cashback Double speed 4G 24 month contract - 4G Unlimited minutes Unlimited texts 5GB data </p> £27.99 per month £656.76 total cost £15 cashback Double speed 4G £27.99 per month £656.76 total cost See deal 0800 691 1667
11c3d7be83b001500d6623be35fc7368
10
30-05-2017 14:04
10
Samsung Galaxy S7 32GB Gold FREE
Unlimited minutes Unlimited texts 8GB data
24 month contract - 4G
27.00
Three Mobile
Buy Mobile Phones
24 month contract - 4G Samsung Galaxy S7 32GB Gold FREE 24 month contract - 4G Unlimited minutes Unlimited texts 8GB data </p> £27.00 per month £648.00 total cost £27.00 per month £648.00 total cost See deal 0800 144 8349
b093a99f463b789161260e1dbf7f2001

Statistics

Average successful run time: 12 minutes

Total run time: 6 days

Total cpu time used: about 5 hours

Total disk space used: 40 MB

History

  • Manually ran revision b12bbe18 and completed successfully .
    13 records added, 1 record updated in the database
  • Auto ran revision b12bbe18 and failed .
    nothing changed in the database
  • Auto ran revision b12bbe18 and completed successfully .
    13 records added, 1 record updated in the database
  • Auto ran revision b12bbe18 and failed .
    nothing changed in the database
  • Auto ran revision b12bbe18 and completed successfully .
    13 records added, 1 record updated in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

PHP

PositionAnalyser / scraper.php