New York City Housing Authority (NYCHA) Temperature Sensor Data Scraping

NYCHA regularly updates a web page that displays information about indoor temperature sensor data by each development.

Morph.io scraper

This web scraper runs daily and extracts the data from all of these tables and saves the results in a SQLite database. The scraper is hosted on morph.io.

See temp_scraper on morph.io.

Credit to @austensen for inspiring this scraper. Thank you @julia-guo @ryanywchen for contributing to this scraper

Contributors ybchen00 julia-guo

Last run completed successfully .

Console output of last run

Injecting configuration and compiling...  -----> Python app detected  ! The latest version of Python 2 is python-2.7.14 (you are using python-2.7.9, which is unsupported).  ! We recommend upgrading by specifying the latest version (python-2.7.14).  Learn More: https://devcenter.heroku.com/articles/python-runtimes -----> Installing python-2.7.9 -----> Installing pip -----> Installing requirements with pip  DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.  Obtaining scraperwiki from git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki (from -r /tmp/build/requirements.txt (line 6))  Cloning http://github.com/openaustralia/scraperwiki-python.git (to revision morph_defaults) to /app/.heroku/src/scraperwiki  Running command git clone -q http://github.com/openaustralia/scraperwiki-python.git /app/.heroku/src/scraperwiki  Running command git checkout -b morph_defaults --track origin/morph_defaults  Switched to a new branch 'morph_defaults'  Branch morph_defaults set up to track remote branch morph_defaults from origin.  Collecting numpy  Downloading numpy-1.16.6-cp27-cp27m-manylinux1_x86_64.whl (17.0 MB)  Collecting beautifulsoup4  Downloading beautifulsoup4-4.9.3-py2-none-any.whl (115 kB)  Collecting requests  Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB)  Collecting html5lib  Downloading html5lib-1.1-py2.py3-none-any.whl (112 kB)  Collecting pandas  Downloading pandas-0.24.2.tar.gz (11.8 MB)  Collecting dumptruck>=0.1.2  Downloading dumptruck-0.1.6.tar.gz (15 kB)  Collecting soupsieve<2.0,>1.2; python_version < "3.0"  Downloading soupsieve-1.9.6-py2.py3-none-any.whl (33 kB)  Collecting idna<3,>=2.5; python_version < "3"  Downloading idna-2.10-py2.py3-none-any.whl (58 kB)  Collecting certifi>=2017.4.17  Downloading certifi-2021.10.8-py2.py3-none-any.whl (149 kB)  Collecting chardet<5,>=3.0.2; python_version < "3"  Downloading chardet-4.0.0-py2.py3-none-any.whl (178 kB)  Collecting urllib3<1.27,>=1.21.1  Downloading urllib3-1.26.9-py2.py3-none-any.whl (138 kB)  Collecting six>=1.9  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)  Collecting webencodings  Downloading webencodings-0.5.1-py2.py3-none-any.whl (11 kB)  Collecting python-dateutil>=2.5.0  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)  Collecting pytz>=2011k  Downloading pytz-2022.1-py2.py3-none-any.whl (503 kB)  Collecting backports.functools-lru-cache; python_version < "3"  Downloading backports.functools_lru_cache-1.6.4-py2.py3-none-any.whl (5.9 kB)  Building wheels for collected packages: pandas, dumptruck  Building wheel for pandas (setup.py): started  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): still running...  Building wheel for pandas (setup.py): finished with status 'done'  Created wheel for pandas: filename=pandas-0.24.2-cp27-cp27m-linux_x86_64.whl size=27470757 sha256=025faca7d5b208b52bd86c36560f6acfac9cf00d37ff8b64df11675a9b3ca995  Stored in directory: /tmp/pip-ephem-wheel-cache-1Ha0Xy/wheels/5f/b4/0b/b1670f56250abdd8f7f200c1a63fbb96e2132f2ed7ee1f8b71  Building wheel for dumptruck (setup.py): started  Building wheel for dumptruck (setup.py): finished with status 'done'  Created wheel for dumptruck: filename=dumptruck-0.1.6-py2-none-any.whl size=11844 sha256=dad4f04a1a03daeed745c5f2e80f1fe47fb2be17459215a60f3b14f571b05af0  Stored in directory: /tmp/pip-ephem-wheel-cache-1Ha0Xy/wheels/dc/75/e9/1e61c4080c73e7bda99614549591f83b53bcc2d682f26fce62  Successfully built pandas dumptruck  Installing collected packages: dumptruck, idna, certifi, chardet, urllib3, requests, scraperwiki, numpy, backports.functools-lru-cache, soupsieve, beautifulsoup4, six, webencodings, html5lib, python-dateutil, pytz, pandas  Running setup.py develop for scraperwiki  Successfully installed backports.functools-lru-cache-1.6.4 beautifulsoup4-4.9.3 certifi-2021.10.8 chardet-4.0.0 dumptruck-0.1.6 html5lib-1.1 idna-2.10 numpy-1.16.6 pandas-0.24.2 python-dateutil-2.8.2 pytz-2022.1 requests-2.27.1 scraperwiki six-1.16.0 soupsieve-1.9.6 urllib3-1.26.9 webencodings-0.5.1 DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.    -----> Discovering process types  Procfile declares types -> scraper Injecting scraper and running...

Data

Downloaded 0 times

To download data sign in with GitHub

Download table (as CSV) Download SQLite database (3 KB) Use the API

rows 10 / 13

Online Sensors Nighttime Exceptions (10:00pm to 6:00am) Total Apartments Daytime Exceptions (6:00am to 10:00pm) Borough Development
0
0
217
0
Manhattan
344 EAST 28TH STREET
143
0
162
0
Brooklyn
CONEY ISLAND
330
0
1,945
0
Bronx
CASTLE HILL
0
0
236
0
Bronx
EAST 180TH STREET-MONTEREY AVENUE
0
0
103
0
Bronx
STEBBINS AVENUE-HEWITT PLACE
107
0
131
0
Brooklyn
CAREY GARDENS
56
0
100
0
Brooklyn
CONEY ISLAND I (SITE 1B)
0
0
96
0
Manhattan
131 SAINT NICHOLAS AVENUE
397
0
433
0
Manhattan
LEXINGTON
59
0
79
0
Manhattan
TWO BRIDGES URA (SITE 7)

Statistics

Average successful run time: 11 minutes

Total run time: 5 days

Total cpu time used: 20 minutes

Total disk space used: 52.2 KB

History

  • Auto ran revision 14f70e5b and completed successfully .
    11 records added, 11 records removed in the database
  • Auto ran revision 14f70e5b and completed successfully .
    11 records added, 11 records removed in the database
  • Auto ran revision 14f70e5b and completed successfully .
    11 records added, 11 records removed in the database
  • Auto ran revision 14f70e5b and completed successfully .
    11 records added, 11 records removed in the database
  • Auto ran revision 14f70e5b and completed successfully .
    11 records added, 11 records removed in the database
  • ...
  • Created on morph.io

Show complete history

Scraper code

Python

temp_parser / scraper.py