------------------
* First release on PyPI.
* Using Jupyter Notebook spikes to explore ideas
[2016-03-30]
Working on way to display the scraped meta data for Miami.
Have a rough working prototype to output an html table that contains a row of meta data for one of the resources.
.. code:: python
git clone https://github.com/dm-wyncode/sopiva
cd ./sopiva
pip install .
cd ./notebooks
python3 ./html_table.py
Runing the above creates a file called ``table.html`` is written inside of ``./notebooks``
If you open ``./notebooks/table.html`` in a browser you can get an idea of what I am aiming for.
I was working with only one source (one row of data) in the table today.
[2016-03-29]
An example script is inside the ``example_scripts`` directory.
.. code:: python
git clone https://github.com/dm-wyncode/sopiva
cd ./sopiva
pip install .
cd ./example_scripts
python3 ./get_responses_from_miami.py
Here is the output from running the above code:
.. code::
Available cities:
{'albuquerque',
'anchorage',
'ann-arbor',
'arlington',
'asheville',
'athensga',
'atlanta',
'austin',
'bangor',
'batonrouge',
'bellca',
'birmingham',
'blacksburg',
'boise',
'boston',
'boulder',
'burbankca',
'burlington',
'calabasas',
'cary',
'charlotte',
'chattanooga',
'chesapeakeva',
'chicago',
'cincinnati',
'cleveland',
'culvercity',
'dallas',
'dayton',
'denver',
'detroit',
'durham',
'elsegundo',
'fallschurch',
'glendaleca',
'grandrapids',
'hamptonva',
'hartford',
'honolulu',
'houston',
'jerseycity',
'kansascitymo',
'lexington',
'lincolnne',
'longbeach',
'longbeachca',
'losangeles',
'louisville',
'madisonwi',
'manhattanbeach',
'mesa',
'miami',
'minneapolis',
'nashville',
'newportnews',
'newyorkcity',
'norfolk',
'oakland',
'oklahoma city',
'pasadena',
'philly',
'pittsburgh',
'portlandor',
'portsmouthva',
'providence',
'raleigh',
'richmond',
'sacramento',
'sanantonio',
'sandiego',
'sanfrancisco',
'sanjose',
'santaclarita',
'santamonica',
'seattle',
'slc',
'spartanburgsc',
'stcloud',
'stlouis',
'suffolkva',
'toledo',
'tulsa',
'tuscaloosa',
'vabeach',
'vegas',
'washingtondc',
'westcovina',
'westhollywood',
'williamsburg'}
Sample info on Miami
zoning http://gis.mdc.opendata.arcgis.com/
spending https://cognoswww.miamidade.gov/cognos/cgi-bin/cognosisapi.dll?b_action=cognosViewer&ui.action=run&ui.object=%2fcontent%2ffolder%5b%40name%3d%27Financial%20Transparency%20Reports%27%5d%2ffolder%5b%40name%3d%27Production%20Reports%27%5d%2freport%5b%40name%3d%27Disbursements%2fPayments%27%5d&ui.name=Disbursements%2fPayments&run.outputFormat=&run.prompt=true
assessment http://www.miamidade.gov/propertysearch//
contracts https://www.miamidade.gov/DPMww/SearchContracts.aspx
business-listings http://www.miamidade.gov/licenses/
budget http://www.miamidade.gov/budget/current.asp
campaign-finance http://www.miamidade.gov/elections/cand-comm.asp
code-enforcement http://gisweb.miamidade.gov/CodeViolations/
lobbyist http://www.miamidade.gov/govaction/lbViewMain.asp
food-safety http://www.floridahealth.gov///reports-and-data/eh-tracking-and-reporting/food-hygiene.html?appSession=890302517583696
deeds https://www2.miami-dadeclerk.com/public-records/Search.aspx
Getting responses from all urls asynchronously.
GET http://www.miamidade.gov/propertysearch// . Returned 200.
GET http://www.miamidade.gov/licenses/ . Returned 200.
GET http://www.miamidade.gov/budget/current.asp . Returned 200.
GET http://www.miamidade.gov/elections/cand-comm.asp . Returned 404.
GET http://gisweb.miamidade.gov/CodeViolations/ . Returned 200.
GET https://www.miamidade.gov/DPMww/SearchContracts.aspx . Returned 200.
GET http://www.miamidade.gov/govaction/lbViewMain.asp . Returned 200.
GET https://cognoswww.miamidade.gov/cognos/cgi-bin/cognosisapi.dll?b_action=cognosViewer&ui.action=run&ui.object=%2fcontent%2ffolder%5b%40name%3d%27Financial%20Transparency%20Reports%27%5d%2ffolder%5b%40name%3d%27Production%20Reports%27%5d%2freport%5b%40name%3d%27Disbursements%2fPayments%27%5d&ui.name=Disbursements%2fPayments&run.outputFormat=&run.prompt=true . Returned 200.
GET https://www2.miami-dadeclerk.com/public-records/Search.aspx . Returned 200.
GET http://www.floridahealth.gov///reports-and-data/eh-tracking-and-reporting/food-hygiene.html?appSession=890302517583696 . Returned 200.
GET http://gis.mdc.opendata.arcgis.com/ . Returned 200.
Psuudocode: `if response.ok then examine the HTML for forms, actions, inputs, etc.`
[2016-03-28]
Spikes are found in the notebook directory.
Both Jupyter notebooks and their conversions into Python scripts are provided. (The exceptional white space in the \*.py scripts is the result of the conversion.)