News of developments
Content
Accès to codes
All the codes produced during the PORTIC project are accessible online on the humanum gitlab here. The codes are licensed under AGPL v3.
The ETL (Python3 / SQL) which allows to retrieve the flat data from Navigocorpus to build a Postgres spatial base is here.
The API code (python3 / Flask) is here.
A gazetteer delivers a description of the ports and various places of maritime commerce in the 18th century online on a Rest API here.
Accès aux données Navigocorpus
Like the API of the PORTIC project, the data in the Navigocorpus database can be viewed in open access on the project’s Gitlab Humanum.
Data is entered and corrected on an ongoing basis in FileMaker on the Navigocorpus database which is hosted on humanum, and can be consulted online. But regularly (every 3 to 6 months), a numbered snapshot of this database is:
- uploaded to GITLAB in flat files
- made available on the web through a RESTful API.
Indeed, in order to be able to create visualizations and propose new analyzes, this snapshot is imported on a Postgres database. In the ETL folder are the source code allowing the import in python of flat data to Postgres (version 11).
The flatdata folder includes flat data, which are exports from the database in interoperable csv, as well as the geographic data of the project. Find all the documentation related to the database here.
Current version of online data: v5 dated July 8, 2020
Comment marche l'API Restful http://data.portic.fr/api ?
An API is a programming interface allowing the consultation of the data of the online database or their recovery in a csv format. This interface is questioned using a query language.
The data related to the API of the PORTIC program are in open access here.
This git lab repository (online and open access file repository) provides access to the API source code in Python as well as flat files, data in text format, and installation procedures to do run the API locally. This will allow you to use the tool on your computer without having access to a web server.
To view the API user file, click here.
How to create a request on the API?
The infographic opposite allows you to visualize the different stages of query creation.
- The data exposed is a compilation of data from navigocorpus for 1787, from the Marseille health register and G5 holidays, as well as the small cabotage notebooks in Marseille. They are retrieved from the navigocorpus database (Filemaker software) online and downloaded in postgres with the code of an ETL available in the humanum gitlab.
- The request consists above all of a root URL: http://data.portic.fr/api
- You must add to this URL the request that interests you: travels, pointcalls, flows and detail. This step is required for the API to return data.
- In order to complete your request, there are several possibilities:
- Formatting the query:
As explained below, different parameters can be added to specify your request: the format, the fields searched etc. These elements can be placed in the URL in any order after the request (step 3).
The first parameter added must be introduced by a “?”, They are then separated by “&”
Example:
http://data.portic.fr/api/travels/?format=csv¶ms=id,departure
http://data.portic.fr/api is the root URL. We try to display all the trips, we then add the request / travels /. The first parameter we add is the format, so it is introduced with a “?”. The second parameter is separated by an “&” and allows to display the fields that interest us in the database, here, id and departure.
- Which format to use?
The default format is json, allowing online consultation of data. If you want to download an export from the database, you must use the csv. In your request you must add this part: format = json or format = csv.
- Which fields to display?
By default, all the fields are displayed, in order to select specific fields from the database, you must use the parameter function: params = and put the fields you want to display after, separated by commas.
Examples of fields:
Id, departure, destination, captain_id, payment_date etc.
- Fielnames, shortenfields and field encoding
In order to make the data easier to read, the shortenfields parameter can be added to the query. Thus the names of the fields are encoded (eg: id: T01). This feature is disabled by default. To activate it, add: shortenfields = true after the URL.
Example:
- Particularity of “details” and “agg”requests
Created for visualization, the details request allows you to know the number of departures from a geographical area determined by: its latitude, longitude and its radius.
On the same principle, /agg/ departures allows to display the number of departures according to each port in the determined geographical area. /agg/ destinations will display the count of destinations for each different admiralty, for points located in the determined geographic area.
Example:
http://data.portic.fr/api/details/departures/?lat=45.2333&lon=-1.5&radius=10
This query will display the starting number in the geographic area with a latitude of 45.2333, a longitude of -1.5 and a radius of 10km.
http://data.portic.fr/api/agg/departures/?lat=46&lon=-1&radius=20
This query will display the number of ships left per port in the defined area (latitude 46, longitude -1 and within a radius of 20km.)
http://data.portic.fr/api/agg/destinations/?lat=45.2333&lon=-1&radius=50
This query will display the count of destinations for ships departing within a radius of 50km around latitude 45.2333 and longitude -1.
Navigocorpus update history
Current version of online data: v5 dated July 8, 2020
Update history
| Version 1 | october 2018 | Sanitation board of Marseille and G5, 1787 and 1789 |
| Version 2 | december 25, 2019 | Sanitation board and G5, 1787 and 1789 – typing and modalities corrections of categorical variables |
| Version 3 | april 2020 | entries and corrections in progress |
| Version 4 | june 2020 | entries and corrections in progress |
| Version 5 | july 8, 2020 | Sanitation board and G5, 1787 and 1789 + cahiers du petit cabotage + corrections completed on 1787 |