chadskelton / course_scraper

enrolment data


Contributors chadskelton

Last run completed successfully .

Console output of last run

Injecting configuration and compiling... [1G [1G-----> Python app detected [1G ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported). [1G ! We recommend upgrading by specifying the latest version (python-2.7.14). [1G Learn More: https://devcenter.heroku.com/articles/python-runtimes [1G-----> Installing python-2.7.9 [1G-----> Installing pip [1G-----> Installing requirements with pip [1G DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support [1G Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 2)) [1G Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki [1G Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki [1G Running command git checkout -b morph_defaults --track origin/morph_defaults [1G Switched to a new branch 'morph_defaults' [1G Branch morph_defaults set up to track remote branch morph_defaults from origin. [1G Collecting BeautifulSoup==3.2.0 [1G Downloading BeautifulSoup-3.2.0.tar.gz (31 kB) [1G Collecting beautifulsoup4==4.1.3 [1G Downloading beautifulsoup4-4.1.3.tar.gz (58 kB) [1G Collecting mechanize==0.2.5 [1G Downloading mechanize-0.2.5.tar.gz (383 kB) [1G Collecting requests==1.0.4 [1G Downloading requests-1.0.4.tar.gz (336 kB) [1G Collecting lxml==2.3.3 [1G Downloading lxml-2.3.3.tar.gz (3.1 MB) [1G Collecting pyOpenSSL==0.13 [1G Downloading pyOpenSSL-0.13.tar.gz (250 kB) [1G Collecting pyasn1==0.4.5 [1G Downloading pyasn1-0.4.5-py2.py3-none-any.whl (73 kB) [1G Collecting dumptruck>=0.1.2 [1G Downloading dumptruck-0.1.6.tar.gz (15 kB) [1G Building wheels for collected packages: BeautifulSoup, beautifulsoup4, mechanize, requests, lxml, pyOpenSSL, dumptruck [1G Building wheel for BeautifulSoup (setup.py): started [1G Building wheel for BeautifulSoup (setup.py): finished with status 'done' [1G Created wheel for BeautifulSoup: filename=BeautifulSoup-3.2.0-py2-none-any.whl size=31888 sha256=3126bd1b175e60d1e0c74d3717143015974dc203470e38fd7199dbccb3917c21 [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/07/ef/84/b06e7bf99ec3061d9a1d31ddfae45eee089c0cc6cb7741578b [1G Building wheel for beautifulsoup4 (setup.py): started [1G Building wheel for beautifulsoup4 (setup.py): finished with status 'done' [1G Created wheel for beautifulsoup4: filename=beautifulsoup4-4.1.3-py2-none-any.whl size=66446 sha256=066ce5d8d9945d30573c3954a45e50767fe80f053060a59a5c8006b7cc25142b [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/d3/19/bb/b9d3a3ed20aa640d4424e824bf9664ca88da65578a8e0aa7dc [1G Building wheel for mechanize (setup.py): started [1G Building wheel for mechanize (setup.py): finished with status 'done' [1G Created wheel for mechanize: filename=mechanize-0.2.5-py2-none-any.whl size=150110 sha256=f1476d4f483af714134f1abf62f6c27e714c31a7878391e9fd48ebacf1d5bbc3 [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/fe/20/00/f8e9ca5a8b8c75eec22f5d63730b66e95be06274f04e9babf6 [1G Building wheel for requests (setup.py): started [1G Building wheel for requests (setup.py): finished with status 'done' [1G Created wheel for requests: filename=requests-1.0.4-py2-none-any.whl size=361328 sha256=1e9916ba06107d0fd1ff6df921ebc4ad9089f9c2be706e7eb847701dfb87dc4e [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/ad/80/ae/0057447a08dd9fb3970f6c95ca2f27c4ee95fbcffdccff4adf [1G Building wheel for lxml (setup.py): started [1G Building wheel for lxml (setup.py): finished with status 'done' [1G Created wheel for lxml: filename=lxml-2.3.3-cp27-cp27m-linux_x86_64.whl size=2164338 sha256=5780f128bbd7f9d0f739fae2a083e56f494a218d5b6c22f4d03da0b9341d42fd [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/03/52/93/c8b3b7b1337aab5a449bf53c8efecfc57755c391e5d20e5dd8 [1G Building wheel for pyOpenSSL (setup.py): started [1G Building wheel for pyOpenSSL (setup.py): finished with status 'done' [1G Created wheel for pyOpenSSL: filename=pyOpenSSL-0.13-cp27-cp27m-linux_x86_64.whl size=293428 sha256=8cd14bec195818d9c2fa47a8de9090dcde3df42e4efb23d159672a9036e12f6f [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/bb/19/ca/ded396f6fe2b146e971146ec72757bb38cc4e384638c6034a2 [1G Building wheel for dumptruck (setup.py): started [1G Building wheel for dumptruck (setup.py): finished with status 'done' [1G Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11842 sha256=16d1fd67ad82d55f3f1507baad3238fb9ceb23ef14b09db4fff54c2f7b97146f [1G Stored in directory: /tmp/pip-ephem-wheel-cache-_O4YRH/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62 [1G Successfully built BeautifulSoup beautifulsoup4 mechanize requests lxml pyOpenSSL dumptruck [1G Installing collected packages: dumptruck, requests, scraperwiki, BeautifulSoup, beautifulsoup4, mechanize, lxml, pyOpenSSL, pyasn1 [1G Running setup.py develop for scraperwiki [1G Successfully installed BeautifulSoup-3.2.0 beautifulsoup4-4.1.3 dumptruck-0.1.6 lxml-2.3.3 mechanize-0.2.5 pyOpenSSL-0.13 pyasn1-0.4.5 requests-1.0.4 scraperwiki [91mDEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support [0m [1G [1G [1G-----> Discovering process types [1G Procfile declares types -> scraper Injecting scraper and running... *** Didn't work for Department: ACCT *** *** Didn't work for Department: ACUP *** *** Didn't work for Department: AGRI *** *** Didn't work for Department: ANTH *** *** Didn't work for Department: CMNS *** *** Didn't work for Department: APSC *** *** Didn't work for Department: ARTH *** *** Didn't work for Department: ARTS *** *** Didn't work for Department: ASIA *** *** Didn't work for Department: ASTR *** *** Didn't work for Department: BIOL *** *** Didn't work for Department: BIOQ *** *** Didn't work for Department: HOPS *** *** Didn't work for Department: BUSI *** *** Didn't work for Department: BUSM *** *** Didn't work for Department: BUQU *** *** Didn't work for Department: CHEM *** *** Didn't work for Department: CHEQ *** *** Didn't work for Department: CADA *** *** Didn't work for Department: CADM *** *** Didn't work for Department: COMM *** *** Didn't work for Department: CBSY *** *** Didn't work for Department: CPSC *** *** Didn't work for Department: CADD *** *** Didn't work for Department: COOP *** *** Didn't work for Department: CNPS *** *** Didn't work for Department: CRWR *** *** Didn't work for Department: CRIM *** *** Didn't work for Department: ECON *** *** Didn't work for Department: EDAS *** *** Didn't work for Department: EDUC *** *** Didn't work for Department: ENGL *** *** Didn't work for Department: ELST *** *** Didn't work for Department: ELSQ *** *** Didn't work for Department: ENGQ *** *** Didn't work for Department: ENTR *** *** Didn't work for Department: ENVI *** *** Didn't work for Department: EXCH *** *** Didn't work for Department: FASN *** *** Didn't work for Department: FMRK *** *** Didn't work for Department: FNSR *** *** Didn't work for Department: FINA *** *** Didn't work for Department: FIND *** *** Didn't work for Department: FREN *** *** Didn't work for Department: GEOG *** *** Didn't work for Department: GNIE *** *** Didn't work for Department: GNQU *** *** Didn't work for Department: GDMA *** *** Didn't work for Department: GRMT *** *** Didn't work for Department: HCAP *** *** Didn't work for Department: HSCI *** *** Didn't work for Department: HEAL *** *** Didn't work for Department: HIST *** *** Didn't work for Department: HORT *** *** Didn't work for Department: HRMT *** *** Didn't work for Department: INDG *** *** Didn't work for Department: INFO *** *** Didn't work for Department: IDEA *** *** Didn't work for Department: IDSN *** *** Didn't work for Department: IBUS *** *** Didn't work for Department: JAPN *** *** Didn't work for Department: JRNL *** *** Didn't work for Department: LANC *** *** Didn't work for Department: LGLA *** *** Didn't work for Department: LING *** *** Didn't work for Department: MAND *** *** Didn't work for Department: MRKT *** *** Didn't work for Department: MATQ *** *** Didn't work for Department: MATH *** *** Didn't work for Department: MAMT *** *** Didn't work for Department: MUSI *** *** Didn't work for Department: NRSG *** *** Didn't work for Department: OSCM *** *** Didn't work for Department: PHIL *** *** Didn't work for Department: PHYS *** *** Didn't work for Department: PHYQ *** *** Didn't work for Department: POST *** *** Didn't work for Department: POLI *** *** Didn't work for Department: DEPD *** *** Didn't work for Department: PSYN *** *** Didn't work for Department: PSYC *** *** Didn't work for Department: PRLN *** *** Didn't work for Department: PUNJ *** *** Didn't work for Department: WRTG *** *** Didn't work for Department: SOCI *** *** Didn't work for Department: SPAN *** *** Didn't work for Department: DETA *** *** Didn't work for Department: TMAS *** *** Didn't work for Department: ZZZZ *** *** Didn't work for Department: WELD ***

Data

Downloaded 49 times by chadskelton

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.66 MB) Use the API

rows 10 / 1806

enrolled course uniqueid max instructor department waitlisted CRN link available credits campus title
0
ECON 1250 S15
2020-06-26 17:43:16.197282
35
 
ECON
0
33092
 
35
3
Online
Principles of Macroeconomics
0
HEAL 1100 L10
2020-06-26 17:43:45.720189
35
 
HEAL
0
33294
 
35
3
Online
Mental Wellness and Communication
0
HEAL 1150 L10
2020-06-26 17:43:45.729992
20
 
HEAL
0
33296
 
20
3
Online
Personal Care Skills: Lab 1
0
HEAL 1150 L11
2020-06-26 17:43:45.738772
20
 
HEAL
0
33888
 
20
3
Online
Personal Care Skills: Lab 1
0
HEAL 1180 L10
2020-06-26 17:43:45.746710
35
 
HEAL
0
33300
 
35
3
Online
Introduction to Health Research
0
BUQU 1130 S11
2020-06-28 21:42:16.193279
35
Meena Bhagat
BUQU
0
33709
 
35
3
Online
Business Mathematics
0
BUQU 1230 S50
2020-06-28 21:42:16.534455
35
 
BUQU
0
33478
 
35
3
Online
Business Statistics
0
ENVI 1106 L02
2020-06-30 09:41:21.786839
15
Astrid Opsetmoen
ENVI
0
34862
5F
15
0
Langley
Environmental Chemistry I
0
CHEM 1110 S10
2020-07-02 13:54:58.048386
35
Catherine Chow
CHEM
0
30321
 
35
4
Surrey
The Structure of Matter
0
CHEM 1110 S11
2020-07-02 13:54:58.059960
35
Jennifer Au
CHEM
0
30322
 
35
4
Surrey
The Structure of Matter

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3.66 MB) Use the API

rows 10 / 19754

enrolled course uniqueid max instructor department waitlisted CRN link available credits campus title
0
ACCT 1110 L10
2020-06-26 17:42:36.110833
35
Catriona Eigenfeldt
ACCT
0
32075
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 L11
2020-06-26 17:42:36.175702
35
Carol Stewart
ACCT
0
35520
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 R10
2020-06-26 17:42:36.184084
35
Bo Zhang
ACCT
0
30792
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 R11
2020-06-26 17:42:36.193321
35
Sarah Lee
ACCT
0
30710
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 R12
2020-06-26 17:42:36.200372
35
Bo Zhang
ACCT
0
30711
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 R13
2020-06-26 17:42:36.207646
35
Sabrina Rai
ACCT
0
32743
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 R15
2020-06-26 17:42:36.215751
35
Elaine Chan
ACCT
0
30850
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 R50
2020-06-26 17:42:36.224048
35
Sarah Lee
ACCT
0
35522
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 S10
2020-06-26 17:42:36.233311
35
Ella Bohinska
ACCT
0
32542
 
35
3
Online
Introductory Financial Accounting I
0
ACCT 1110 S11
2020-06-26 17:42:36.241992
35
Ella Bohinska
ACCT
0
32710
 
35
3
Online
Introductory Financial Accounting I

Statistics

Average successful run time: 5 minutes

Total run time: 1 day

Total cpu time used: about 4 hours

Total disk space used: 3.77 MB

History

  • Auto ran revision ae0eafab and completed successfully .
    nothing changed in the database
  • Auto ran revision ae0eafab and completed successfully .
    nothing changed in the database
  • Auto ran revision ae0eafab and completed successfully .
    nothing changed in the database
  • Auto ran revision ae0eafab and completed successfully .
    3582 records added, 1791 records removed in the database
  • Auto ran revision ae0eafab and completed successfully .
    3582 records added, 1791 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

course_scraper