Webapps with python

Webapps with python

For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together.

This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

Demonstrations

Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like Plotly dash and web2py to do the frontend. I use docker containers based on dokku -- a PaaS (Platform as a Service) --to host these apps.

Geohacking publications with Python Natural Language Processing and Friends

Google Scholar results on the map LINK

What is this and how it works

Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map.

1. 1. Background

Python provides all the tools needed to do Natural Language Processing, including

• Web scraping e.g. [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)
• Parsing and identifying entities e.g. [NLTK toolkit](https://www.nltk.org/)
• Flag geographical locations mentioned in the text and geolocating them (Geoparsing)

e.g. [Mordecai](https://github.com/openeventdata/mordecai), [geography3](https://pypi.org/project/geograpy3/) and (simple)[geotext] (https://pypi.org/project/geotext/).

What it does

• Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)
• Store them in a NoSQL database (MongoDB)
• Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract.
• Feed the data to this app, so that the user can interactively look at them.

How to use

• (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map.
• Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country.
• It is possible to select more than one keyword (simply select from the dropdown list)
• It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right).
• Click on the link below each record to see it on google scholar.

What is missing

• Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish).

NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names.

• 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).

Next step?

This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.

With large scale applications some of the possibilities are:

• Identify temporal trends in publications.
• Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)

Seasonal levels in waterbodies using Sentinel-2 data

End-to-end automated system for calculating seasonal water availability in practically any waterbody.

Interactive map, analysis and plotting tool for precipitation. LINK

Background

Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.

Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.

This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.

What it does

It uses Sentinal-2 images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)

How it works

User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button. An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server. The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below. The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph. Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).

Current limitations

While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.

So at the moment, I am running this system in a limited mode.

User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources. While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.) Currently I am using a simple water detection method (Otsu threshold method). Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.

What you can do (currently)

Feel free to experiment with the job submission system. Examine already processed result sets.

Technologies used

The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service. The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.

Next step?

These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification. For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.

(After reading, click on the cross (top right) to get this message out of the way.)

Precipitation records of Europe

Interactive map, analysis and plotting tool for precipitation. LINK

European Climate Assessment & Dataset project managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see here for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations.

We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).

LINK

Life-cycle Costing Tool

Life-cycle cost calculator. LINK

One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the present value (PV) and aggregated.

This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.

Orographic lift of wind fields - atmospheric quantities calculator

Atmospheric quantities with orographic uplift LINK

This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).

Urban population of the world

Urban population of the world. Selected 13000 urban centres from around the world. LINK

Using the curated dataset provided by simplemaps website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.

Concrete crack detection with CNN

In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[1]

In this simnple demo example, I have trained a CNN with 40000 concrete images provided by .

To test this app, first find some images of concrete with and without cracks. For example:

• 

  1

• 

  2

• 

  3

• 

  4

Crack detection in concrete with tensorflow LINK

If you don't have such images, just search the web and download a few. Then go to the app and upload them. (You can drag and drop them onto the app as well.

A simple front-end to a urban drainage/flood model

Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore (a) Response (runoff) under pristine conditions (b) under built-up conditions (of a given percentage) (c) built-up conditions together with some SUDS interventions. for various urban catchments.

This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public.

SWMM model results under different catchment conditions LINK

Calculating Rainwater Harvesting Potential for Small Islands

The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability.

Rainwater harvesting calculator results.LINK

LINK

Groundwater recharge calculator]

The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow.

Groundwater recharge calculator resultsLINK

LINK

Flood Hazard and Risk Simulator

Flood Hazard and Risk Simulator Results. LINK

LINK