This project involved creating a Python web application using a Flask server with REST API to perform CRUD operations on several database tables using AJAX calls. The application links to a third party API - the Irish open data portal at https://data.gov.ie using the CKAN API.
This post contains an overview of a data representation project which was completed for the Data Representation and Querying module at GMIT as part of the Higher Diploma in Computing and Data Analytics.
The Data Representation module focused on investigating and operating the protocols, standards and architectures used in representing data, with a focus on interacting with data services across the internet with practical experience in developing applications that interact with such data.
The project involved writing a Flask server program that consumes a REST API and creating a web interface. The project links to the Ireland’s open data portal at https://data.gov.ie, populates a MySql database with some simple tables, then allows a user to interact with the data stored in the database through a web interface. It also allows the user to request additional data from the Irish open data portal using additional API actions.
The project repository is on GitHub and contains several files including:
requirements.txtfile containing the Python packages required to run the application
initdb.sqlfile containing the SQL code for creating the database.
An overview of the instructions for downloading and running the project is included on this website here.
There were two choices in the project instructions for this project with some overlap.
I chose to use the Irish Government’s Open data portal at https://data.gov.ie as I wanted to use real data and to explore the Irish Open-Data portal more. This portal is built using CKAN. Using the CKAN API you can get (JSON-formatted) lists of all the datasets that are hosted on the open data portal. You can also get lists of the organisations that publish the datasets as well as lists of tags used. (Note The term
package is used for datasets).
Some notes on the Irish Open data portal are included here.
Once you have retrieved the list of packages, tags or organizations you can then programmatically retrieve the lists of actual dataset resources using further API actions.
From the list of dataset resources you can extract the URLs to the actual datasets and retrieve them using Python, Curl or by going to the URLs.
Through this project I gained experience with the following:
JQuery methods for Ajax functionality.
Writing Data Application Objects (DAO).
The CKAN API