https://assela.pathirana.net/api.php?action=feedcontributions&user=Root&feedformat=atomassela Pathirana - User contributions [en]2024-03-28T21:59:56ZUser contributionsMediaWiki 1.37.0https://assela.pathirana.net/index.php?title=MediaWiki:Common.css&diff=4693MediaWiki:Common.css2021-12-27T10:51:54Z<p>Root: </p>
<hr />
<div><!-- No [view source] tab for anonymous users --></div>Roothttps://assela.pathirana.net/index.php?title=MediaWiki:Common.js&diff=4692MediaWiki:Common.js2021-12-27T10:51:37Z<p>Root: Created page with "/* Any JavaScript here will be loaded for all users on every page load. */ <!-- No [view source] tab for anonymous users --> <?php global $wgUser; if( $wgUser->isAnon() ) { ?> <style type="text/css"> #ca-viewsource { display: none !important; } </style> <?php } ?>"</p>
<hr />
<div>/* Any JavaScript here will be loaded for all users on every page load. */<br />
<!-- No [view source] tab for anonymous users --><br />
<?php global $wgUser; if( $wgUser->isAnon() ) { ?><br />
<style type="text/css"><br />
#ca-viewsource { display: none !important; }<br />
</style> <br />
<?php } ?></div>Roothttps://assela.pathirana.net/index.php?title=MediaWiki:Common.css&diff=4691MediaWiki:Common.css2021-12-27T10:47:56Z<p>Root: </p>
<hr />
<div><!-- No [view source] tab for anonymous users --><br />
<?php global $wgUser; if( $wgUser->isAnon() ) { ?><br />
<style type="text/css"><br />
#ca-viewsource { display: none !important; }<br />
</style> <br />
<?php } ?></div>Roothttps://assela.pathirana.net/index.php?title=MediaWiki:Common.css&diff=4690MediaWiki:Common.css2021-12-27T10:46:41Z<p>Root: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
#ca-nstab-main { display:none!important; }<br />
#ca-nstab-user { display:none!important; }<br />
#ca-talk { display:none!important; } <br />
#ca-view { display:none!important; } <br />
#ca-edit { display:none!important; } <br />
#ca-history { display:none!important; } <br />
#ca-watch { display:none!important; } <br />
#ca-unwatch { display:none!important; } <br />
#ca-delete { display:none!important; } <br />
#ca-move { display:none!important; } <br />
#ca-protect { display:none!important; } <br />
#ca-viewsource { display:none!important; }</div>Roothttps://assela.pathirana.net/index.php?title=MediaWiki:Common.css&diff=4689MediaWiki:Common.css2021-12-27T10:45:25Z<p>Root: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */</div>Roothttps://assela.pathirana.net/index.php?title=MediaWiki:Common.css&diff=4688MediaWiki:Common.css2021-12-27T10:44:43Z<p>Root: </p>
<hr />
<div>/* CSS placed here will be applied to all skins */<br />
<!-- No [view source] tab for anonymous users --><br />
<?php global $wgUser; if( $wgUser->isAnon() ) { ?><br />
<style type="text/css"><br />
#ca-viewsource { display: none !important; }<br />
</style> <br />
<?php } ?></div>Roothttps://assela.pathirana.net/index.php?title=Sozi&diff=4687Sozi2021-12-15T08:57:01Z<p>Root: </p>
<hr />
<div>[http://sozi.baierouge.fr/wiki/en:welcome Sozi]] is an [[wikipedia:Inkscape|Inkscape]] plugin that makes the vector editing program into a powerful presentation maker. The tool makes non-linear (a bit like [[wikipedia:Prezi|Prezi]]) in SVG format. A main advantage is that the user does not have to have any special software to play them -- A good web browser will do. I have had great success with Firefox and Google Chrome (or Chromium). <br />
<br />
==Sozi on Steroids==<br />
<br />
Due to limitations imposed by Inkscape plugin mechanism (?) registering frames with sozi need to be done one at a time. This makes the user impatient. Following is a script that helps to automate adding a bunch of Sozi frames to a presentation. See the header for information on usage.<br />
<source lang=python><br />
#!/usr/bin/python<br />
<br />
#######################################################################<br />
### Assela Pathirana 2012 <br />
#######################################################################<br />
### Automates adding number of frames to sozi<br />
### How to: <br />
### 1. Make a frame in inkscape, give it an easy id, and add it to sozi. <br />
### 2. Make copies of that frame to all other places where needed. <br />
### 3. Run this script with the svg file and that id as arguments. It will <br />
### add all the copied frames to sozi<br />
### 4. Do the fine tuning of labels with sozi. <br />
######################################################################<br />
import sys<br />
<br />
if (len(sys.argv) > 2):<br />
rectid=sys.argv[2]<br />
file=sys.argv[1]<br />
else:<br />
print "Usage: $0 <svg file> <\"id\">\n"<br />
exit(5)<br />
print "Using parent id: ", rectid , "\n"<br />
<br />
SVG_NAMESPACE = "http://www.w3.org/2000/svg"<br />
SVG= "{%s}" % SVG_NAMESPACE<br />
ns1= "{%s}" % "http://sozi.baierouge.fr" <br />
from lxml import etree<br />
tree=etree.parse(file)<br />
rects=tree.findall("//%srect" % SVG)<br />
sozis=tree.findall("//%sframe" % ns1 )<br />
import re<br />
def choose(x,y):<br />
return re.search(y,x)<br />
parent=filter(lambda x: rectid==x.attrib['id'] , rects)<br />
if len(parent)!=1 : <br />
print "Can't find the parent id!!"<br />
exit(0)<br />
<br />
target=filter(lambda x: rectid==x.attrib['%srefid' % ns1 ] , sozis)<br />
if len(target)!=1 : <br />
print "Can't find the parent in sozi!!"<br />
exit(0)<br />
root=tree.getroot()<br />
myrects=filter(lambda x: re.search(rectid,x.attrib['id'])!=None , rects) <br />
#myrects=list(set(myrects)-set(parent))<br />
myrects=myrects[1:]<br />
# now deepcopy target with attribute changed to id of myrects<br />
from copy import deepcopy<br />
for item in myrects:<br />
print "Doing: " , item.attrib['id']<br />
newitem=deepcopy(target[0])<br />
newitem.attrib['%srefid' % ns1]=item.attrib['id']<br />
root.append(newitem) <br />
tree.write(file+"_.svg", pretty_print=True)<br />
<br />
</source></div>Roothttps://assela.pathirana.net/index.php?title=Sozi&diff=4686Sozi2021-12-15T08:56:46Z<p>Root: </p>
<hr />
<div>[http://sozi.baierouge.fr/wiki/en:welcome Sozi]] is an [[wikipedia:Inkscape|Inkscape]] plugin that makes the vector editing program into a powerful presentation maker. The tool makes non-linear (a bit like [[wikipedia:Prezi|Prezi]]) in SVG format. A main advantage is that the user does not have to have any special software to play them -- A good web browser will do. I have had great success with Firefox and Google Chrome (or Chromium). <br />
<br />
==Sozi on Steroids==<br />
<br />
Due to limitations imposed by Inkscape plugin mechanism (?) registering frames with sozi need to be done one at a time. This makes the user impatient. Following is a script that helps to automate adding a bunch of Sozi frames to a presentation. See the header for information on usage.</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4683Webapps with python2021-04-29T11:25:23Z<p>Root: /* Groundwater recharge calculator] */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
====Background====<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]<br />
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing). e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and <br />
(simple) [https://pypi.org/project/geotext/ geotext].<br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4682Webapps with python2021-04-29T11:24:47Z<p>Root: /* Background */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
====Background====<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]<br />
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing). e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and <br />
(simple) [https://pypi.org/project/geotext/ geotext].<br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4681Webapps with python2021-04-29T11:24:30Z<p>Root: /* Background */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
====Background====<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]<br />
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing) <br />
e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and <br />
(simple) [https://pypi.org/project/geotext/ geotext].<br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4680Webapps with python2021-04-29T11:23:03Z<p>Root: /* What is this and how it works */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
====Background====<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)<br />
* Parsing and identifying entities e.g. [NLTK toolkit](https://www.nltk.org/)<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing) <br />
e.g. [Mordecai](https://github.com/openeventdata/mordecai), [geography3](https://pypi.org/project/geograpy3/) and <br />
(simple)[geotext] (https://pypi.org/project/geotext/).<br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4679Webapps with python2021-04-29T11:22:24Z<p>Root: /* Geohacking publications with Python Natural Language Processing and Friends */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
## Background<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)<br />
* Parsing and identifying entities e.g. [NLTK toolkit](https://www.nltk.org/)<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing) <br />
e.g. [Mordecai](https://github.com/openeventdata/mordecai), [geography3](https://pypi.org/project/geograpy3/) and <br />
(simple)[geotext] (https://pypi.org/project/geotext/). <br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=File:Nlpgeohacking_publications.png&diff=4678File:Nlpgeohacking publications.png2021-04-29T11:21:41Z<p>Root: </p>
<hr />
<div></div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4677Webapps with python2021-04-29T11:19:25Z<p>Root: /* Demonstrations */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|350px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
## Background<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)<br />
* Parsing and identifying entities e.g. [NLTK toolkit](https://www.nltk.org/)<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing) <br />
e.g. [Mordecai](https://github.com/openeventdata/mordecai), [geography3](https://pypi.org/project/geograpy3/) and <br />
(simple)[geotext] (https://pypi.org/project/geotext/). <br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4676Webapps with python2021-04-29T11:18:55Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends===<br />
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|350px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div><br />
==== What is this and how it works====<br />
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map. <br />
## Background<br />
Python provides all the tools needed to do Natural Language Processing, including<br />
* Web scraping e.g. [BeautifulSoup](https://pypi.org/project/beautifulsoup4/)<br />
* Parsing and identifying entities e.g. [NLTK toolkit](https://www.nltk.org/)<br />
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing) <br />
e.g. [Mordecai](https://github.com/openeventdata/mordecai), [geography3](https://pypi.org/project/geograpy3/) and <br />
(simple)[geotext] (https://pypi.org/project/geotext/). <br />
<br />
====What it does====<br />
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)<br />
* Store them in a NoSQL database (MongoDB)<br />
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract. <br />
* Feed the data to this app, so that the user can interactively look at them. <br />
<br />
====How to use====<br />
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map. <br />
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country. <br />
* It is possible to select more than one keyword (simply select from the dropdown list)<br />
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right). <br />
* Click on the link below each record to see it on google scholar. <br />
<br />
====What is missing====<br />
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish). <br />
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names. <br />
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).<br />
<br />
====Next step?====<br />
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing). <br />
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks). <br />
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.<br />
<br />
With large scale applications some of the possibilities are:<br />
<br />
* Identify temporal trends in publications. <br />
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4675Webapps with python2021-04-09T08:45:32Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div><br />
[https://rwh1.srv.pathirana.net/ LINK]<br />
<br />
===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[https://infil1.srv.pathirana.net/ LINK]<br />
<br />
===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div><br />
[http://fg.srv.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Assela_Pathirana&diff=4674Assela Pathirana2021-04-09T08:14:01Z<p>Root: </p>
<hr />
<div><!-- ''Average teachers become good ones by developing a strong passion for their act. Some, then become excellent ones by learning to control this passion, so that students get more space to learn''. --><br />
-----<br />
<br />
{{MainPageIntro}} <br />
{| cellspacing="3"<br />
|- valign="top"<br />
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|<br />
<div style="padding: .4em .9em .9em" id="MainPageBox1"><br />
==Climate Security of Small Islands==<br />
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]<br />
<br />
The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]<br />
<br />
<br />
==Urban adaptation in rapidly growing cities need a paradigm shift==<br />
[[File:kuznet_framework_for_scgs.png|thumb|500px]]<br />
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others. <br />
<br />
In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]<br />
<br />
==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==<br />
This paper puts forth a systematic approach to select the governmental actions that can ensure<br />
water quality based on the connection between governmental actions at present and in the future<br />
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and<br />
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems<br />
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework<br />
to quantify the connections using impact based indicators for connections between actions. The<br />
methodology has been tested using six governmental actions across four plausible future<br />
scenarios in Luzhi Town, a unique water village within Suzhou city, China.<br />
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]<br />
<br />
==The book on Climate Change, Extreme Rainfall and Urban Drainage==<br />
[[File:igur_cc_rain_book.gif|thumb|400px]]<br />
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems. <br />
<br />
In addition to a state-of-the-art overview of<br />
existing methodologies and relevant results related to the<br />
assessment of the climate change impacts on urban rainfall<br />
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]<br />
<br />
==SWMM5-Python==<br />
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]<br />
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]<br />
<br />
==IMHEN Report on Climte Change in Vietnam==<br />
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam. <br />
<br />
IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]<br />
<br />
<br />
</div><br />
----<br />
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|<br />
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2"><br />
<br />
==Webapps with python==<br />
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.<br />
<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
[[Webapps with python|Read more..]]<br />
<br />
<br />
==Crowdsourcing groundwater quality in small islands==<br />
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]<br />
<br />
The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring. <br />
<br />
But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]<br />
<br />
==Urbanization not only heats up the cities, but could increase extreme rainfall==<br />
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]<br />
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city. <br />
<br />
With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]<br />
<br />
<br />
==Using Wiki in Higher-Education: Application in organizing research groups==<br />
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]<br />
<br />
This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]<br />
<br />
==Improving Drinking Water Quality without Compromising Long-term Safety==<br />
<br />
Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.<br />
<br />
We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]<br />
<br />
==Ecosystem value of SuDS==<br />
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better <br />
manage storm water runoffs and reduce urban flooding, and at the same time provide significant <br />
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water <br />
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]<br />
<br />
</div><br />
|}<br />
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc"><br />
<h3 id="lang_h3" class="lang"></h3><br />
</div><br />
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;"><br />
===<span id="sister_heading" class="sister"></span>===<br />
<br />
<div style="clear:left"></div><br />
</div> __NOTOC__ __NOEDITSECTION__<br />
{{main links}}<br />
[[Category:Meta]]</div>Roothttps://assela.pathirana.net/index.php?title=Assela_Pathirana&diff=4673Assela Pathirana2021-04-09T08:13:29Z<p>Root: </p>
<hr />
<div><!-- ''Average teachers become good ones by developing a strong passion for their act. Some, then become excellent ones by learning to control this passion, so that students get more space to learn''. --><br />
-----<br />
<br />
{{MainPageIntro}} <br />
{| cellspacing="3"<br />
|- valign="top"<br />
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|<br />
<div style="padding: .4em .9em .9em" id="MainPageBox1"><br />
==Climate Security of Small Islands==<br />
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]<br />
<br />
The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]<br />
<br />
<br />
==Urban adaptation in rapidly growing cities need a paradigm shift==<br />
[[File:kuznet_framework_for_scgs.png|thumb|500px]]<br />
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others. <br />
<br />
In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]<br />
<br />
==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==<br />
This paper puts forth a systematic approach to select the governmental actions that can ensure<br />
water quality based on the connection between governmental actions at present and in the future<br />
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and<br />
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems<br />
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework<br />
to quantify the connections using impact based indicators for connections between actions. The<br />
methodology has been tested using six governmental actions across four plausible future<br />
scenarios in Luzhi Town, a unique water village within Suzhou city, China.<br />
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]<br />
<br />
==The book on Climate Change, Extreme Rainfall and Urban Drainage==<br />
[[File:igur_cc_rain_book.gif|thumb|400px]]<br />
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems. <br />
<br />
In addition to a state-of-the-art overview of<br />
existing methodologies and relevant results related to the<br />
assessment of the climate change impacts on urban rainfall<br />
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]<br />
<br />
==SWMM5-Python==<br />
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]<br />
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]<br />
<br />
==IMHEN Report on Climte Change in Vietnam==<br />
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam. <br />
<br />
IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]<br />
<br />
<br />
</div><br />
----<br />
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|<br />
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2"><br />
<br />
==Webapps with python==<br />
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
[[File:raingauge_stations_europe.png|thumb|450px|Interactive map, analysis and plotting tool for precipitation]]<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.<br />
<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
[[Webapps with python|Read more..]]<br />
<br />
<br />
==Crowdsourcing groundwater quality in small islands==<br />
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]<br />
<br />
The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring. <br />
<br />
But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]<br />
<br />
==Urbanization not only heats up the cities, but could increase extreme rainfall==<br />
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]<br />
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city. <br />
<br />
With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]<br />
<br />
<br />
==Using Wiki in Higher-Education: Application in organizing research groups==<br />
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]<br />
<br />
This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]<br />
<br />
==Improving Drinking Water Quality without Compromising Long-term Safety==<br />
<br />
Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.<br />
<br />
We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]<br />
<br />
==Ecosystem value of SuDS==<br />
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better <br />
manage storm water runoffs and reduce urban flooding, and at the same time provide significant <br />
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water <br />
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]<br />
<br />
</div><br />
|}<br />
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc"><br />
<h3 id="lang_h3" class="lang"></h3><br />
</div><br />
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;"><br />
===<span id="sister_heading" class="sister"></span>===<br />
<br />
<div style="clear:left"></div><br />
</div> __NOTOC__ __NOEDITSECTION__<br />
{{main links}}<br />
[[Category:Meta]]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4672Webapps with python2021-04-09T08:10:24Z<p>Root: /* Seasonal levels in waterbodies using Sentinel-2 data */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=File:Water-detect-webapp.png&diff=4671File:Water-detect-webapp.png2021-04-09T08:09:00Z<p>Root: </p>
<hr />
<div></div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4670Webapps with python2021-04-09T08:08:46Z<p>Root: /* Seasonal levels in waterbodies using Sentinel-2 data */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4669Webapps with python2021-04-09T08:08:19Z<p>Root: /* Demonstrations */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4668Webapps with python2021-04-09T08:00:30Z<p>Root: /* [Seasonal levels in waterbodies using Sentinel-2 data https://waterdetect.srv.pathirana.net/] */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4667Webapps with python2021-04-09T07:59:44Z<p>Root: /* Demonstrations */</p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[Seasonal levels in waterbodies using Sentinel-2 data https://waterdetect.srv.pathirana.net/]===<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
====Background====<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
====What it does====<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
====How it works====<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
====Current limitations====<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
====What you can do (currently)====<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
====Technologies used====<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
====Next step?====<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4666Webapps with python2021-04-09T07:59:07Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.<br />
<br />
===[Seasonal levels in waterbodies using Sentinel-2 data https://waterdetect.srv.pathirana.net/]<br />
<br />
End-to-end automated system for calculating seasonal water availability in practically any waterbody.<br />
<br />
Background<br />
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.<br />
<br />
Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.<br />
<br />
This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.<br />
<br />
===What it does===<br />
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)<br />
<br />
===How it works===<br />
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.<br />
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.<br />
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.<br />
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.<br />
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).<br />
===Current limitations===<br />
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.<br />
<br />
So at the moment, I am running this system in a limited mode.<br />
<br />
User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.<br />
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)<br />
Currently I am using a simple water detection method (Otsu threshold method).<br />
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.<br />
===What you can do (currently)===<br />
Feel free to experiment with the job submission system.<br />
Examine already processed result sets.<br />
===Technologies used===<br />
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.<br />
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.<br />
<br />
===Next step?===<br />
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.<br />
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.<br />
<br />
(After reading, click on the cross (top right) to get this message out of the way.)<br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4665Webapps with python2021-04-09T07:54:45Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4664Webapps with python2021-04-09T07:54:37Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
<br />
For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together. <br />
<br />
This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.<br />
<br />
One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion. <br />
<br />
<br />
==Demonstrations==<br />
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4663Webapps with python2021-03-11T15:49:50Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.apps.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.apps.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4662Webapps with python2021-03-11T14:56:43Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.srv.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.srv.pathirana.net/ LINK]<br />
<br />
===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.srv.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Test&diff=4661Test2020-10-30T14:55:41Z<p>Root: Blanked the page</p>
<hr />
<div></div>Roothttps://assela.pathirana.net/index.php?title=Test&diff=4660Test2020-10-30T14:52:53Z<p>Root: </p>
<hr />
<div><html><br />
<iframe<br />
src='https://api.mapbox.com/styles/v1/mapbox/streets-v11.html?title=true&zoomwheel=false&access_token=pk.eyJ1IjoiYXNzZWxhIiwiYSI6ImNqMmFlNThndjAwMTkycXFndGs4bG42Z2UifQ.ahJsIac4Y87yP4C3dHgHKg#15/37.771/-122.436' width='100%' height='400px'><br />
</iframe><br />
</html></div>Roothttps://assela.pathirana.net/index.php?title=Test&diff=4659Test2020-10-30T14:49:49Z<p>Root: Created page with "<iframe src='https://api.mapbox.com/styles/v1/mapbox/streets-v11.html?title=true&zoomwheel=false&access_token=pk.eyJ1IjoiYXNzZWxhIiwiYSI6ImNqMmFlNThndjAwMTkycXFndGs4bG42Z2Ui..."</p>
<hr />
<div><iframe<br />
src='https://api.mapbox.com/styles/v1/mapbox/streets-v11.html?title=true&zoomwheel=false&access_token=pk.eyJ1IjoiYXNzZWxhIiwiYSI6ImNqMmFlNThndjAwMTkycXFndGs4bG42Z2UifQ.ahJsIac4Y87yP4C3dHgHKg#15/37.771/-122.436' width='100%' height='400px'><br />
</iframe></div>Roothttps://assela.pathirana.net/index.php?title=File:Flood_calculator_results.png&diff=4658File:Flood calculator results.png2020-10-26T09:11:59Z<p>Root: Root uploaded a new version of File:Flood calculator results.png</p>
<hr />
<div></div>Roothttps://assela.pathirana.net/index.php?title=File:Flood_calculator_results.png&diff=4657File:Flood calculator results.png2020-10-26T09:10:22Z<p>Root: </p>
<hr />
<div></div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4656Webapps with python2020-10-26T09:08:40Z<p>Root: /* Groundwater recharge calculator] */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.wa.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.wa.pathirana.net/ LINK]<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]<br />
<br />
===[http://fg.apps.pathirana.net/ Flood Hazard and Risk Simulator]===<br />
<br />
<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[http://fg.apps.pathirana.net// LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Blogs&diff=4655Blogs2020-10-23T13:59:23Z<p>Root: </p>
<hr />
<div>==Recent Blogs==<br />
* [https://www.mv.undp.org/content/maldives/en/home/blog/2020/Thousandsensorstowatchoverourworld.html Thousand sensors to watch over our world]<br />
* [https://undpmv.exposure.co/post-393429 Our water is telling us something by UNDP Maldives]<br />
* [https://soundcloud.com/user-539321192/fd-episode-9-undp-maldives-chief-technical-advisor-water-engineering-with-dr-assela-pathirana Interview on Water Security on Frontier Dialogues]</div>Roothttps://assela.pathirana.net/index.php?title=Blogs&diff=4654Blogs2020-10-23T13:59:14Z<p>Root: Created page with "==Recent Blogs== * [https://www.mv.undp.org/content/maldives/en/home/blog/2020/Thousandsensorstowatchoverourworld.html Thousand sensors to watch over our world] * [https://und..."</p>
<hr />
<div>==Recent Blogs==<br />
* [https://www.mv.undp.org/content/maldives/en/home/blog/2020/Thousandsensorstowatchoverourworld.html Thousand sensors to watch over our world]<br />
* [https://undpmv.exposure.co/post-393429 <br />
Our water is telling us something by UNDP Maldives]<br />
* [https://soundcloud.com/user-539321192/fd-episode-9-undp-maldives-chief-technical-advisor-water-engineering-with-dr-assela-pathirana Interview on Water Security on Frontier Dialogues]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4653Webapps with python2020-10-23T09:18:36Z<p>Root: /* Precipitation records of Europe */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.wa.pathirana.net/ LINK]]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
[http://er.wa.pathirana.net/ LINK]<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4652Webapps with python2020-10-23T09:17:39Z<p>Root: /* Concrete crack detection with CNN */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden><br />
<gallery class="center"><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4651Webapps with python2020-10-23T09:16:40Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden, margin: auto"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4650Webapps with python2020-10-23T09:14:53Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4649Webapps with python2020-10-23T09:13:24Z<p>Root: </p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4648Webapps with python2020-10-23T09:12:33Z<p>Root: /* Precipitation records of Europe */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).<br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4647Webapps with python2020-10-23T09:12:09Z<p>Root: /* Webapps with python */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
==Demonstration==<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly). <br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4646Webapps with python2020-10-23T09:11:24Z<p>Root: /* Concrete crack detection with CNN */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly). <br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|550px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4645Webapps with python2020-10-23T09:10:48Z<p>Root: /* Webapps with python */</p>
<hr />
<div>==Webapps with python==<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly). <br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]</div><br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]</div><br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<div style="overflow: hidden"><gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|350px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
</div><br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div><br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] </div><br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
<div style="overflow: hidden">[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] </div><br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4644Webapps with python2020-10-23T09:09:00Z<p>Root: /* Webapps with python */</p>
<hr />
<div>==Webapps with python==<br />
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly). <br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]<br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]<br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|350px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]<br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] <br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] <br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4643Webapps with python2020-10-23T09:08:07Z<p>Root: /* Life-cycle Costing Tool */</p>
<hr />
<div>==Webapps with python==<br />
[[File:raingauge_stations_europe.png|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly). <br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]</div><br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]<br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]<br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|350px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]<br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] <br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] <br />
[https://infil1.apps.pathirana.net/ LINK]</div>Roothttps://assela.pathirana.net/index.php?title=Webapps_with_python&diff=4642Webapps with python2020-10-22T15:00:52Z<p>Root: /* Calculating Rainwater Harvesting Potential for Small Islands */</p>
<hr />
<div>==Webapps with python==<br />
[[File:raingauge_stations_europe.png|thumb|350px|Interactive map, analysis and plotting tool for precipitation]]<br />
Python has a number of libraries that makes creating graphics based on data. Some of these tools can create interactive graphics and also web applications so that one can allow non-programmers to explore, analyse and visualize data. [https://plot.ly/dash/ Ploty Dash] is such a library with a particularly easy learning curve. <br />
<br />
Here are some prototype applications that were created using this library. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps. <br />
<br />
===[http://er.wa.pathirana.net/ Precipitation records of Europe]===<br />
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of september 2019, this database includes observations from 57312 stations. <br />
<br />
We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly). <br />
<br />
===[http://lcc.wa.pathirana.net/ Life-cycle Costing Tool]===<br />
[[File:life_cycle_costing_tool_python.png|thumb|350px|Life-cycle cost calculator. [https://lcc.wa.pathirana.net/ LINK]]]<br />
<br />
One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated. <br />
<br />
This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.<br />
<br />
===[http://oro.wa.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===<br />
[[File:orographic_lifting_tool_python.png|thumb|350px|Atmospheric quantities with orographic uplift [http://oro.wa.pathirana.net/ LINK]]]<br />
<br />
This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).<br />
<br />
===[http://citypop.wa.pathirana.net/ Urban population of the world]===<br />
[[File:citypop_tool_python.png|thumb|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.wa.pathirana.net/ LINK]]]<br />
<br />
Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.<br />
<br />
===[https://cp.wa.pathirana.net/ Concrete crack detection with CNN]===<br />
<br />
In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]<br />
<br />
In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>. <br />
<br />
To test this app, first find some images of concrete with and without cracks. For example: <br />
<gallery><br />
File:Concrete_crack.jpg|1<br />
File:Concrete_crack_2.jpg|2<br />
File:Concrete_crack_3.jpg|3<br />
File:concrete_crack_4.jpg|4<br />
</gallery><br />
<br />
[[File:concrete_crack_results.png|thumb|right|350px|Crack detection in concrete with tensorflow [https://cp.wa.pathirana.net/ LINK]]]<br />
<br />
If you don't have such images, just search the web and download a few. Then [https://cp.wa.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.<br />
<br />
===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===<br />
<br />
Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore<br />
(a) Response (runoff) under pristine conditions<br />
(b) under built-up conditions (of a given percentage)<br />
(c) built-up conditions together with some SUDS interventions. <br />
for various urban catchments. <br />
<br />
This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public. <br />
<br />
[[File:web2py swmm interface.png|thumb|right|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]<br />
<br />
===[https://rwh1.apps.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===<br />
<br />
The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability. <br />
<br />
<br />
[[File:rwh calculator results.png|thumb|right|350px|Rainwater harvesting calculator results.[https://rwh1.apps.pathirana.net/ LINK]]] <br />
[https://rwh1.apps.pathirana.net/ LINK]<br />
<br />
===[https://infil1.apps.pathirana.net/ Groundwater recharge calculator]]===<br />
<br />
The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow. <br />
<br />
[[File:infil calculator results.png|thumb|right|350px|Groundwater recharge calculator results[https://infil1.apps.pathirana.net/ LINK]]] <br />
[https://infil1.apps.pathirana.net/ LINK]</div>Root