assela Pathirana - User contributions [en]

Webapps with python

2023-02-01T05:23:28Z

Assela.pathirana.admin: /* Concrete crack detection with CNN */

==Webapps with python==

For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together.

This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

==Demonstrations==
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.

===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div>
==== What is this and how it works====
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map.
====Background====
Python provides all the tools needed to do Natural Language Processing, including
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing). e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and
(simple) [https://pypi.org/project/geotext/ geotext].

====What it does====
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)
* Store them in a NoSQL database (MongoDB)
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract.
* Feed the data to this app, so that the user can interactively look at them.

====How to use====
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map.
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country.
* It is possible to select more than one keyword (simply select from the dropdown list)
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right).
* Click on the link below each record to see it on google scholar.

====What is missing====
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish).
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names.
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).

====Next step?====
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing).
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks).
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.

With large scale applications some of the possibilities are:

* Identify temporal trends in publications.
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)

===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===

End-to-end automated system for calculating seasonal water availability in practically any waterbody.

<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

====Background====
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.

Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.

This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.

====What it does====
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)

====How it works====
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).
====Current limitations====
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.

So at the moment, I am running this system in a limited mode.

User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)
Currently I am using a simple water detection method (Otsu threshold method).
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.
====What you can do (currently)====
Feel free to experiment with the job submission system.
Examine already processed result sets.
====Technologies used====
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.

====Next step?====
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.

(After reading, click on the cross (top right) to get this message out of the way.)

===[http://er.srv.pathirana.net/ Precipitation records of Europe]===
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations.

We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).

[http://er.srv.pathirana.net/ LINK]

===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div>

One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated.

This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.

===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div>

This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).

===[http://citypop.srv.pathirana.net/ Urban population of the world]===
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div>

Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.

===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===

In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]

In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>.

To test this app, first find some images of concrete with and without cracks. For example:
<div style="overflow: hidden>
<gallery class="center">
File:Concrete_crack.jpg|1
File:Concrete_crack_2.jpg|2
File:Concrete_crack_3.jpg|3
File:concrete_crack_4.jpg|4
</gallery>

[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]
</div>

If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.
Or here is a sample (I just harvested from the web) to start with. Download the zip file, extract and drag and drop all the image files to the app! ([[file:webapps_with_python_concrete_images_sample_12.zip|zipfile]])

===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===

Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore
(a) Response (runoff) under pristine conditions
(b) under built-up conditions (of a given percentage)
(c) built-up conditions together with some SUDS interventions.
for various urban catchments.

This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public.

<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div>

===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===

The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability.

<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div>
[https://rwh1.srv.pathirana.net/ LINK]

===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]===

The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow.

<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div>
[https://infil1.srv.pathirana.net/ LINK]

===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===

<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div>
[http://fg.srv.pathirana.net// LINK]

Webapps with python

2023-02-01T05:23:02Z

Assela.pathirana.admin: /* Concrete crack detection with CNN */

==Webapps with python==

For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together.

This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

==Demonstrations==
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.

===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div>
==== What is this and how it works====
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map.
====Background====
Python provides all the tools needed to do Natural Language Processing, including
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing). e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and
(simple) [https://pypi.org/project/geotext/ geotext].

====What it does====
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)
* Store them in a NoSQL database (MongoDB)
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract.
* Feed the data to this app, so that the user can interactively look at them.

====How to use====
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map.
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country.
* It is possible to select more than one keyword (simply select from the dropdown list)
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right).
* Click on the link below each record to see it on google scholar.

====What is missing====
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish).
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names.
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).

====Next step?====
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing).
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks).
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.

With large scale applications some of the possibilities are:

* Identify temporal trends in publications.
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)

===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===

End-to-end automated system for calculating seasonal water availability in practically any waterbody.

<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

====Background====
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.

Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.

This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.

====What it does====
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)

====How it works====
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).
====Current limitations====
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.

So at the moment, I am running this system in a limited mode.

User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)
Currently I am using a simple water detection method (Otsu threshold method).
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.
====What you can do (currently)====
Feel free to experiment with the job submission system.
Examine already processed result sets.
====Technologies used====
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.

====Next step?====
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.

(After reading, click on the cross (top right) to get this message out of the way.)

===[http://er.srv.pathirana.net/ Precipitation records of Europe]===
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations.

We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).

[http://er.srv.pathirana.net/ LINK]

===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div>

One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated.

This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.

===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div>

This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).

===[http://citypop.srv.pathirana.net/ Urban population of the world]===
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div>

Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.

===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===

In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]

In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>.

To test this app, first find some images of concrete with and without cracks. For example:
<div style="overflow: hidden>
<gallery class="center">
File:Concrete_crack.jpg|1
File:Concrete_crack_2.jpg|2
File:Concrete_crack_3.jpg|3
File:concrete_crack_4.jpg|4
</gallery>

[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]
</div>

If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.
Or here is a sample (I just harvested from the web) to start with. Download the zip file, extract and drag and drop all the image files to the app! [[file:webapps_with_python_concrete_images_sample_12.zip|Zip_File]]

===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===

Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore
(a) Response (runoff) under pristine conditions
(b) under built-up conditions (of a given percentage)
(c) built-up conditions together with some SUDS interventions.
for various urban catchments.

This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public.

<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div>

===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===

The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability.

<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div>
[https://rwh1.srv.pathirana.net/ LINK]

===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]===

The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow.

<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div>
[https://infil1.srv.pathirana.net/ LINK]

===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===

<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div>
[http://fg.srv.pathirana.net// LINK]

Webapps with python

2023-02-01T05:22:45Z

Assela.pathirana.admin: /* Concrete crack detection with CNN */

==Webapps with python==

For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together.

This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

==Demonstrations==
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.

===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div>
==== What is this and how it works====
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map.
====Background====
Python provides all the tools needed to do Natural Language Processing, including
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing). e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and
(simple) [https://pypi.org/project/geotext/ geotext].

====What it does====
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)
* Store them in a NoSQL database (MongoDB)
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract.
* Feed the data to this app, so that the user can interactively look at them.

====How to use====
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map.
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country.
* It is possible to select more than one keyword (simply select from the dropdown list)
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right).
* Click on the link below each record to see it on google scholar.

====What is missing====
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish).
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names.
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).

====Next step?====
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing).
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks).
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.

With large scale applications some of the possibilities are:

* Identify temporal trends in publications.
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)

===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===

End-to-end automated system for calculating seasonal water availability in practically any waterbody.

<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

====Background====
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.

Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.

This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.

====What it does====
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)

====How it works====
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).
====Current limitations====
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.

So at the moment, I am running this system in a limited mode.

User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)
Currently I am using a simple water detection method (Otsu threshold method).
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.
====What you can do (currently)====
Feel free to experiment with the job submission system.
Examine already processed result sets.
====Technologies used====
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.

====Next step?====
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.

(After reading, click on the cross (top right) to get this message out of the way.)

===[http://er.srv.pathirana.net/ Precipitation records of Europe]===
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations.

We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).

[http://er.srv.pathirana.net/ LINK]

===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div>

One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated.

This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.

===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div>

This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).

===[http://citypop.srv.pathirana.net/ Urban population of the world]===
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div>

Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.

===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===

In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]

In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>.

To test this app, first find some images of concrete with and without cracks. For example:
<div style="overflow: hidden>
<gallery class="center">
File:Concrete_crack.jpg|1
File:Concrete_crack_2.jpg|2
File:Concrete_crack_3.jpg|3
File:concrete_crack_4.jpg|4
</gallery>

[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]
</div>

If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.
Or here is a sample (I just harvested from the web) to start with. Download the zip file, extract and drag and drop all the image files to the app! [[file:webapps_with_python_concrete_images_sample_12.zip|Zip File]]

===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===

Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore
(a) Response (runoff) under pristine conditions
(b) under built-up conditions (of a given percentage)
(c) built-up conditions together with some SUDS interventions.
for various urban catchments.

This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public.

<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div>

===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===

The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability.

<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div>
[https://rwh1.srv.pathirana.net/ LINK]

===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]===

The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow.

<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div>
[https://infil1.srv.pathirana.net/ LINK]

===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===

<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div>
[http://fg.srv.pathirana.net// LINK]

File:Webapps with python concrete images sample 12.zip

2023-02-01T05:22:20Z

Assela.pathirana.admin:

Webapps with python

2023-02-01T05:21:06Z

Assela.pathirana.admin: /* Concrete crack detection with CNN */

==Webapps with python==

For a long time, the tradition of providing technical solutions was the experts to do the designs and implement the designs for the benefit of the society. At a later stage, we started practising ‘awareness raising’. While this was a step in the right direction, by attempting to describe the why’s what’s and how’s of a technical solution to the societal stakeholders, awareness-raising often worked as an afterthought. A large body of evidence has shown that the best outcomes can be achieved by involving the community from day one of a technical solution. This day one is the planning and design stage. Whether it is money management in a family, a piece of policy in a government or an institution or any type of technical solution, best stakeholder support is obtained when people co-own the product. The journey to co-ownership starts with co-discovery (of knowledge), co-design and implementing together.

This applies to any type of technical solution. However, it becomes a non-negotiable requirement for success in climate and nature-based solutions. By their very nature, both the problem and its solutions are distributed in nature. Delivery of electricity from a customer from a thermal power plant also involves dealing with the ‘users’ – we call this customer management. But when we go for household level, grid-connected, solar electricity generation, that customer must become a business partner! That is the transformation we are witnessing in many sectors addressing problems with climate and nature-based solutions. This is how people are empowered to do designs. The pandemic is a portal, to make the wrongs right, and to build back better and greener.

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

==Demonstrations==
Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.

===[https://gs.srv.pathirana.net/ Geohacking publications with Python Natural Language Processing and Friends]===
<div style="overflow: hidden">[[file:nlpgeohacking_publications.png|thumb|center|700px|Google Scholar results on the map [https://gs.srv.pathirana.net/ LINK]]]</div>
==== What is this and how it works====
Using Natural Language Processing to 'Geoparsing' google scholar search results. Select a keyword (or more) and see where the publications refer to. Click on a bubble of a country to see its publications listed below the map.
====Background====
Python provides all the tools needed to do Natural Language Processing, including
* Web scraping e.g. [https://pypi.org/project/beautifulsoup4/ BeautifulSoup]
* Parsing and identifying entities e.g. [https://www.nltk.org/ NLTK toolkit]
* Flag geographical locations mentioned in the text and geolocating them (Geoparsing). e.g. [https://github.com/openeventdata/mordecai Mordecai], [https://pypi.org/project/geograpy3/ geography3] and
(simple) [https://pypi.org/project/geotext/ geotext].

====What it does====
* Downloads Google Scholar search hits for each keyword (In this demo, I have limited each to 500 top hits, to keep things simple)
* Store them in a NoSQL database (MongoDB)
* Run a geoparser (geotext in this case) to locate mentions of countries in the title or the abstract.
* Feed the data to this app, so that the user can interactively look at them.

====How to use====
* (After closing these instructions) Select a keyword. The locations of the publications will be shown on the map. A list of all the publications will be shown below the map.
* Click on the bubbles on the map to filter by country. Then the list will be updated to cover only that country.
* It is possible to select more than one keyword (simply select from the dropdown list)
* It's also possible to select several countries. Either SHIFT+Click on the map or use the select tools (top-right).
* Click on the link below each record to see it on google scholar.

====What is missing====
* Many publications concerning the United States of America, typically does not write the country name (e.g. A statewide assessment of mercury dynamics in North Carolina water bodies and fish).
NLP tools are usually smart enough to detect these (North Carolina is in the USA so tag as 'USA'), but the current (demo) implementation misses some obscure names.
* 'The United Kingdom vs. England' tagging is complicated. This issue has to be fixed (That's why no articles are tagged for England).

====Next step?====
This demo provides a framework for Natural Language Processing of online material to make sense of information (e.g. geoparsing).
It combines several Big-data constructs (Unstructured data, NoSQL (Jason) data lakes, NLP tricks).
While a web app is not the right place to scale up these to the big-data level, the framework presented here can easily be implemented to do large-scale processing using a decent cluster computer system.

With large scale applications some of the possibilities are:

* Identify temporal trends in publications.
* Locate 'hotspots' as well as locations with few (or no) studies (geographical gaps)

===[https://waterdetect.srv.pathirana.net/ Seasonal levels in waterbodies using Sentinel-2 data ]===

End-to-end automated system for calculating seasonal water availability in practically any waterbody.

<div style="overflow: hidden">[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

====Background====
Sentinel-2 s an Earth observation mission from the Copernicus that systematically acquires multispectral, optical imagery at high spatial resolution (10 m to 60 m) over land and coastal waters.

Sentinelhub (a partner of Euro Data Cube consortium), operated by a private IT company, provides cloud services for sentinel-2 data. This provides an excellent way to access (petabytes of) data using a Python-based API. Eo-learn is a collection of open-source Python packages that have been developed to access and process Spatio-temporal satellite image sequences.

This web application shows how these technologies can be integrated to provide cloud-based data services in a manner that can be used by non-experts.

====What it does====
It uses [https://sentinel.esa.int/web/sentinel/missions/sentinel-2 Sentinal-2] images (specifically RGB and near-infrared bands - or B02, B03, B04 and B08) to identify water. It goes through a series of images (taken at different times) of the location of interest and calculate the fraction of area covered by water. This provides an easy way to estimate the water amount in reservoirs, lakes and other water bodies. (Note: However, what this application calculates is the fraction (Area covered with water)/(a nominal area). So to really convert it to water volumes, we need the land contours of the reservoir.)

====How it works====
User provides a water body by marking it with a polygon by clicking on the map. Gives the location a name and submit it for processing by pressing the 'Request' button.
An automated processing engine is waiting in the background. (However, see the limitation below for a caveat!). Once the user submits the request, it will queue it with many other requests (made by multiple users) and process them. This is a numerically expensive (and costing a lot of network bandwidth) task for the server.
The user visits the site later (and press 'Refresh' button) to check if the submitted job is finished. If it is it will appear in the table below.
The user can select a result set by clicking on a tick-box on the left. Then the time-series of the water area fraction will appear at the bottom graph.
Click on the graph points to see the satellite images overlayed on the map. (use Gain and Opacity to finetune how it looks).
====Current limitations====
While the web-application system can do this whole process automatically and upscale really well (responding many user requests at the same time for example), it cost processing power, network bandwidth and sentinel-hub processing units (which have to be paid). Currently my server (which runs more than 15 web applications) has only 8GB of RAM. For the backend to work well it needs about 32GB. Network bandwidth is not a problem. However, sentinel hub processing units are. Currently I am using their 'trial' account to demonstrate this. However, for the system to scale-up a paid account is needed.

So at the moment, I am running this system in a limited mode.

User submitted jobs do not directly go to the processing engine. Instead, an admin (currently me) have to 'approve' them. (I can do this by clicking on the same table after authenticating). Then it will automatically be processed by the backend system. Why: My account at the sentinel hub has very limited resources.
While the system processes all satellite images available covering a period, only the first 10 are saved as images. (Try clicking on points on the time-series graph, only the first 10 will show the corresponding image on the map.)
Currently I am using a simple water detection method (Otsu threshold method).
Due to the limited resources of the server the system is not very responsive at the moment. The client application needs optimising too.
====What you can do (currently)====
Feel free to experiment with the job submission system.
Examine already processed result sets.
====Technologies used====
The frontend (what you see) is implemented on Flask and Plotly-dash systems. Maps are provided by OpenStreetMap with Leaflet interactive map library. Plotly was used to draw the graphs. The system runs on a Linux Virtual Private Server with two vCores and 8GB of RAM. The processing engine (the system that talks with the sentinel hub and processes satellite data, does the calculations and convert the results to something that can be understood by the front end) is implemented as a systemd service.
The administration system is secured with two-factor authentication, along with a time-based OTP (onetime password) system.

====Next step?====
These type of end-to-end automated data processing systems provide great opportunities to apply machine learning techniques like the convolutional neural networks to generate higher-level knowledge based on what we calculate. For example, I have used Tensorflow to demonstrate how to detect cracks in concrete by automatic image identification.
For example, the same can be used to do land use classification at a much better accuracy level and higher scalability. (See this, this and this But that's for later.

(After reading, click on the cross (top right) to get this message out of the way.)

===[http://er.srv.pathirana.net/ Precipitation records of Europe]===
<div style="overflow: hidden">[[File:raingauge_stations_europe.png|center|thumb|center|350px|Interactive map, analysis and plotting tool for precipitation. [http://er.srv.pathirana.net/ LINK]]]
[https://www.ecad.eu/ European Climate Assessment & Dataset project] managed by Royal Netherlands Meteorological Institute (KNMI), collects meteorological data (pressure, humidity, wind speed, cloud cover and precipitation - see [https://www.ecad.eu/dailydata/datadictionaryelement.php here] for the complete description) from thousands of observation stations from (at the time of writing) 63 countries. As of September 2019, this database includes observations from 57312 stations.

We extracted the precipitation data from this dataset and provide it with a web application where the user can explore, do some simple trend analyses and download, downsampled data (Annual and monthly).

[http://er.srv.pathirana.net/ LINK]

===[http://lcc.srv.pathirana.net/ Life-cycle Costing Tool]===
<div style="overflow: hidden">[[File:life_cycle_costing_tool_python.png|thumb|center|350px|Life-cycle cost calculator. [https://lcc.srv.pathirana.net/ LINK]]]</div>

One of the routine tasks of Infrastructure Asset Management is to calculate the 'total cost of ownership' of an asset, for example, a building, a bridge or barrage. This involves consideration of the cost of purchase or construction, annual operation and maintenance, periodic renewal and sometimes the ultimate cost of disposal. These costs are all brought to the [https://en.wikipedia.org/wiki/Net_present_value present value (PV)] and aggregated.

This app provides a convenient way to play with different cost components and interest rates (that is needed to calculate PV) and understand how that affects the whole life cost.

===[http://oro.srv.pathirana.net Orographic lift of wind fields - atmospheric quantities calculator]===
<div style="overflow: hidden">[[File:orographic_lifting_tool_python.png|thumb|center|350px|Atmospheric quantities with orographic uplift [http://oro.srv.pathirana.net/ LINK]]]</div>

This is an educational tool to demonstrate the interaction of wind fields with mountains. The user can change the mountain height, humidity and temperature of the air and observe how they contribute to the formation of precipitation (liquid or sometimes ice/snow).

===[http://citypop.srv.pathirana.net/ Urban population of the world]===
<div style="overflow: hidden">[[File:citypop_tool_python.png|thumb|center|350px|Urban population of the world. Selected 13000 urban centres from around the world. [https://citypop.srv.pathirana.net/ LINK]]]</div>

Using the curated dataset provided by [https://simplemaps.com/data/world-cities simplemaps] website, this plot shows the urban population of the world. Note: This dataset does not cover all the populated places. It covers almost all major cities and towns, but the coverage of smaller places could be uneven.

===[https://cp.srv.pathirana.net/ Concrete crack detection with CNN]===

In deep learning, a convolutional neural network (CNN) is a class of deep neural networks, most commonly applied to analyzing visual imagery.[https://en.wikipedia.org/wiki/Convolutional_neural_network]

In this simnple demo example, I have trained a CNN with 40000 concrete images provided by <ref>Lei Zhang , Fan Yang , Yimin Daniel Zhang, and Y. J. Z., Zhang, L., Yang, F., Zhang, Y. D., & Zhu, Y. J. (2016). Road Crack Detection Using Deep Convolutional Neural Network. In 2016 IEEE International Conference on Image Processing (ICIP). http://doi.org/10.1109/ICIP.2016.7533052</ref>.

To test this app, first find some images of concrete with and without cracks. For example:
<div style="overflow: hidden>
<gallery class="center">
File:Concrete_crack.jpg|1
File:Concrete_crack_2.jpg|2
File:Concrete_crack_3.jpg|3
File:concrete_crack_4.jpg|4
</gallery>

[[File:concrete_crack_results.png|thumb|center|550px|Crack detection in concrete with tensorflow [https://cp.srv.pathirana.net/ LINK]]]
</div>

If you don't have such images, just search the web and download a few. Then [https://cp.srv.pathirana.net/ go to the app] and upload them. (You can drag and drop them onto the app as well.
Or here is a sample (I just harvested from the web) to start with. Download the zip file, extract and drag and drop all the image files to the app! [[file:webapps_with_python_concrete_images_sample_12.zip|files]]

===[https://web2py.pathirana.net/urbangreenblue/default/index A simple front-end to a urban drainage/flood model]===

Increasing built-up area causes runoff to increase. Sustainable drainage systems (SUDS) like rain-gardens, vegetated swales, urban wetland in-turn reduces runoff by mimicking the pre-urbanized natual conditions. This application is a proof of concept of running a dynamic urban drainage model (EPA SWMM 5.0) in the backend to explore
(a) Response (runoff) under pristine conditions
(b) under built-up conditions (of a given percentage)
(c) built-up conditions together with some SUDS interventions.
for various urban catchments.

This application has an administrative interface that can be used to add more examples. Obviously that interface is password protected and not open to the public.

<div style="overflow: hidden">[[File:web2py swmm interface.png|thumb|center|350px|SWMM model results under different catchment conditions [https://web2py.pathirana.net/urbangreenblue/default/index LINK]]]</div>

===[https://rwh1.srv.pathirana.net/ Calculating Rainwater Harvesting Potential for Small Islands]===

The user provides several input parameters: roof area over which the rainwater is collected, the size of tank planned and the demand – how much water is planned to be used for how many months (e.g. during dry period). Based on user’s location the tool will automatically select historical rainfall data from the nearest meteorological station. Then it simulates the harvesting and usage performance for several decades. Such calculations involving multiple years (for the Maldives this can be about 20-30 years in most cases) gives much better indication of the working of a rainwater harvesting system than just assessing it against one or two years of data. Like many other climates of the world the climate of the Maldives also shows great deal of ‘inter-annual’ variability. Simulations involving a longer period therefore gives a more realistic picture of the performance of the system in the backdrop of such variability.

<div style="overflow: hidden">[[File:rwh calculator results.png|thumb|center|350px|Rainwater harvesting calculator results.[https://rwh1.srv.pathirana.net/ LINK]]] </div>
[https://rwh1.srv.pathirana.net/ LINK]

===[https://infil1.srv.pathirana.net/ Groundwater recharge calculator]===

The tool allows the user to choose this by providing the hourly rainfall rate and number of hours over which that intensity will continue. After providing the size of the infiltration pit, whether it is filled with gravel or just empty and the surrounding soil type, the user can calculate the performance of the system. The results are provided as a graphic as well as graph of water level in the structure and any overflow.

<div style="overflow: hidden">[[File:infil calculator results.png|thumb|center|350px|Groundwater recharge calculator results[https://infil1.srv.pathirana.net/ LINK]]] </div>
[https://infil1.srv.pathirana.net/ LINK]

===[http://fg.srv.pathirana.net/ Flood Hazard and Risk Simulator]===

<div style="overflow: hidden">[[File:flood calculator results.png|thumb|center|350px|Flood Hazard and Risk Simulator Results. [https://infil1.srv.pathirana.net/ LINK]]] </div>
[http://fg.srv.pathirana.net// LINK]

Assela Pathirana

2022-03-21T08:39:06Z

Assela.pathirana.admin:

-----

{{MainPageIntro}}
{| cellspacing="3"
|- valign="top"
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|
<div style="padding: .4em .9em .9em" id="MainPageBox1">

==[[Systems_thinking_for_Urban_Water_Systems|Systems thinking for Urban Water Systems]]==
[[image:Systems_thinking_cartoon.jpg|thumb|400px|Systems thinking for water systems]]

By Dr. [https://www.linkedin.com/in/mohanasundar-radhakrishnan-27757b1/?originalSubdomain=nl Mohanasundar Radhakrishnan]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses. [[Systems_thinking_for_Urban_Water_Systems| Read more ...]]

==Climate Security of Small Islands==
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]

The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]

==Urban adaptation in rapidly growing cities need a paradigm shift==
[[File:kuznet_framework_for_scgs.png|thumb|500px]]
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others.

In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]

==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==
This paper puts forth a systematic approach to select the governmental actions that can ensure
water quality based on the connection between governmental actions at present and in the future
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework
to quantify the connections using impact based indicators for connections between actions. The
methodology has been tested using six governmental actions across four plausible future
scenarios in Luzhi Town, a unique water village within Suzhou city, China.
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]

==The book on Climate Change, Extreme Rainfall and Urban Drainage==
[[File:igur_cc_rain_book.gif|thumb|400px]]
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems.

In addition to a state-of-the-art overview of
existing methodologies and relevant results related to the
assessment of the climate change impacts on urban rainfall
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]

==SWMM5-Python==
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]

==IMHEN Report on Climte Change in Vietnam==
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam.

IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]

</div>
----
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2">

==Webapps with python==
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.
[[Webapps with python|Read more..]]

==Crowdsourcing groundwater quality in small islands==
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]

The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring.

But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]

==Urbanization not only heats up the cities, but could increase extreme rainfall==
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city.

With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]

==Using Wiki in Higher-Education: Application in organizing research groups==
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]

This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]

==Improving Drinking Water Quality without Compromising Long-term Safety==

Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.

We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]

==Ecosystem value of SuDS==
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better
manage storm water runoffs and reduce urban flooding, and at the same time provide significant
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]

</div>
|}
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc">
<h3 id="lang_h3" class="lang"></h3>
</div>
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;">
===<span id="sister_heading" class="sister"></span>===

<div style="clear:left"></div>
</div> __NOTOC__ __NOEDITSECTION__
{{main links}}
[[Category:Meta]]

Assela Pathirana

2022-03-21T08:38:44Z

Assela.pathirana.admin:

-----

{{MainPageIntro}}
{| cellspacing="3"
|- valign="top"
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|
<div style="padding: .4em .9em .9em" id="MainPageBox1">

==[[Systems_thinking_for_Urban_Water_Systems|Systems_thinking_for_Urban_Water_Systems]]==
[[image:Systems_thinking_cartoon.jpg|thumb|400px|Systems thinking for water systems]]

By Dr. [https://www.linkedin.com/in/mohanasundar-radhakrishnan-27757b1/?originalSubdomain=nl Mohanasundar Radhakrishnan]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses. [[Systems_thinking_for_Urban_Water_Systems| Read more ...]]

==Climate Security of Small Islands==
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]

The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]

==Urban adaptation in rapidly growing cities need a paradigm shift==
[[File:kuznet_framework_for_scgs.png|thumb|500px]]
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others.

In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]

==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==
This paper puts forth a systematic approach to select the governmental actions that can ensure
water quality based on the connection between governmental actions at present and in the future
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework
to quantify the connections using impact based indicators for connections between actions. The
methodology has been tested using six governmental actions across four plausible future
scenarios in Luzhi Town, a unique water village within Suzhou city, China.
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]

==The book on Climate Change, Extreme Rainfall and Urban Drainage==
[[File:igur_cc_rain_book.gif|thumb|400px]]
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems.

In addition to a state-of-the-art overview of
existing methodologies and relevant results related to the
assessment of the climate change impacts on urban rainfall
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]

==SWMM5-Python==
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]

==IMHEN Report on Climte Change in Vietnam==
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam.

IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]

</div>
----
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2">

==Webapps with python==
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.
[[Webapps with python|Read more..]]

==Crowdsourcing groundwater quality in small islands==
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]

The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring.

But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]

==Urbanization not only heats up the cities, but could increase extreme rainfall==
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city.

With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]

==Using Wiki in Higher-Education: Application in organizing research groups==
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]

This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]

==Improving Drinking Water Quality without Compromising Long-term Safety==

Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.

We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]

==Ecosystem value of SuDS==
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better
manage storm water runoffs and reduce urban flooding, and at the same time provide significant
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]

</div>
|}
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc">
<h3 id="lang_h3" class="lang"></h3>
</div>
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;">
===<span id="sister_heading" class="sister"></span>===

<div style="clear:left"></div>
</div> __NOTOC__ __NOEDITSECTION__
{{main links}}
[[Category:Meta]]

Assela Pathirana

2022-03-21T08:38:34Z

Assela.pathirana.admin:

-----

{{MainPageIntro}}
{| cellspacing="3"
|- valign="top"
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|
<div style="padding: .4em .9em .9em" id="MainPageBox1">

==[[Systems_thinking_for_Urban_Water_Systems]]==
[[image:Systems_thinking_cartoon.jpg|thumb|400px|Systems thinking for water systems]]

By Dr. [https://www.linkedin.com/in/mohanasundar-radhakrishnan-27757b1/?originalSubdomain=nl Mohanasundar Radhakrishnan]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses. [[Systems_thinking_for_Urban_Water_Systems| Read more ...]]

==Climate Security of Small Islands==
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]

The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]

==Urban adaptation in rapidly growing cities need a paradigm shift==
[[File:kuznet_framework_for_scgs.png|thumb|500px]]
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others.

In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]

==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==
This paper puts forth a systematic approach to select the governmental actions that can ensure
water quality based on the connection between governmental actions at present and in the future
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework
to quantify the connections using impact based indicators for connections between actions. The
methodology has been tested using six governmental actions across four plausible future
scenarios in Luzhi Town, a unique water village within Suzhou city, China.
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]

==The book on Climate Change, Extreme Rainfall and Urban Drainage==
[[File:igur_cc_rain_book.gif|thumb|400px]]
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems.

In addition to a state-of-the-art overview of
existing methodologies and relevant results related to the
assessment of the climate change impacts on urban rainfall
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]

==SWMM5-Python==
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]

==IMHEN Report on Climte Change in Vietnam==
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam.

IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]

</div>
----
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2">

==Webapps with python==
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.
[[Webapps with python|Read more..]]

==Crowdsourcing groundwater quality in small islands==
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]

The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring.

But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]

==Urbanization not only heats up the cities, but could increase extreme rainfall==
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city.

With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]

==Using Wiki in Higher-Education: Application in organizing research groups==
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]

This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]

==Improving Drinking Water Quality without Compromising Long-term Safety==

Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.

We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]

==Ecosystem value of SuDS==
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better
manage storm water runoffs and reduce urban flooding, and at the same time provide significant
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]

</div>
|}
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc">
<h3 id="lang_h3" class="lang"></h3>
</div>
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;">
===<span id="sister_heading" class="sister"></span>===

<div style="clear:left"></div>
</div> __NOTOC__ __NOEDITSECTION__
{{main links}}
[[Category:Meta]]

Assela Pathirana

2022-03-21T08:38:10Z

Assela.pathirana.admin:

-----

{{MainPageIntro}}
{| cellspacing="3"
|- valign="top"
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|
<div style="padding: .4em .9em .9em" id="MainPageBox1">

==Systems Thinking for urban water systems==
[[image:Systems_thinking_cartoon.jpg|thumb|400px|Systems thinking for water systems]]

By Dr. [https://www.linkedin.com/in/mohanasundar-radhakrishnan-27757b1/?originalSubdomain=nl Mohanasundar Radhakrishnan]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses. [[Systems_thinking_for_Urban_Water_Systems| Read more ...]]

==Climate Security of Small Islands==
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]

The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]

==Urban adaptation in rapidly growing cities need a paradigm shift==
[[File:kuznet_framework_for_scgs.png|thumb|500px]]
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others.

In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]

==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==
This paper puts forth a systematic approach to select the governmental actions that can ensure
water quality based on the connection between governmental actions at present and in the future
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework
to quantify the connections using impact based indicators for connections between actions. The
methodology has been tested using six governmental actions across four plausible future
scenarios in Luzhi Town, a unique water village within Suzhou city, China.
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]

==The book on Climate Change, Extreme Rainfall and Urban Drainage==
[[File:igur_cc_rain_book.gif|thumb|400px]]
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems.

In addition to a state-of-the-art overview of
existing methodologies and relevant results related to the
assessment of the climate change impacts on urban rainfall
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]

==SWMM5-Python==
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]

==IMHEN Report on Climte Change in Vietnam==
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam.

IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]

</div>
----
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2">

==Webapps with python==
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.
[[Webapps with python|Read more..]]

==Crowdsourcing groundwater quality in small islands==
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]

The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring.

But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]

==Urbanization not only heats up the cities, but could increase extreme rainfall==
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city.

With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]

==Using Wiki in Higher-Education: Application in organizing research groups==
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]

This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]

==Improving Drinking Water Quality without Compromising Long-term Safety==

Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.

We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]

==Ecosystem value of SuDS==
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better
manage storm water runoffs and reduce urban flooding, and at the same time provide significant
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]

</div>
|}
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc">
<h3 id="lang_h3" class="lang"></h3>
</div>
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;">
===<span id="sister_heading" class="sister"></span>===

<div style="clear:left"></div>
</div> __NOTOC__ __NOEDITSECTION__
{{main links}}
[[Category:Meta]]

Assela Pathirana

2022-03-21T08:37:09Z

Assela.pathirana.admin:

-----

{{MainPageIntro}}
{| cellspacing="3"
|- valign="top"
|width="45%" class="MainPageBG" style="border: 1px solid #F0F0FF; color: #000; background-color: #F0F0FF"|
<div style="padding: .4em .9em .9em" id="MainPageBox1">

==Systems Thinking for urban water systems==
[[image:Systems_thinking_cartoon.jpg|thumb|400px|Systems thinking for water systems]]

By Dr. [https://www.linkedin.com/in/mohanasundar-radhakrishnan-27757b1/?originalSubdomain=nl Mohanasundar Radhakrishnan]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

==Climate Security of Small Islands==
Climate change presents unique challenges to Small Island Developing States (SIDS). The difficulties that all countries face in effectively coping with climate change impacts are exacerbated in SIDS because of their small geographical area, isolation and exposure. The associated development challenges from sea-level rise, altered rainfall patterns, and storm-surges threaten to reverse progress made towards the Millennium Development Goals now and in the future. The United Nations Development Programme (UNDP) provides on-the-ground support for small island and low-lying countries at the global, regional and national scales. - [https://sustainabledevelopment.un.org/index.php?page=view&type=400&nr=960&menu=1515 UNDP]

The Maldives is already facing the brunt of Climate Change in the guise of water scarcity. Small atoll islands' water security depend on the important, but highly climate-sensitive, fresh water lenses. [[Climate and Water Security in the Maldives| Read more ...]]

==Urban adaptation in rapidly growing cities need a paradigm shift==
[[File:kuznet_framework_for_scgs.png|thumb|500px]]
We have been working with rapid developing cities like [https://en.wikipedia.org/wiki/C%E1%BA%A7n_Th%C6%A1 Can Tho], vietnam ([https://scholar.google.nl/scholar?as_q=can+tho&as_epq=&as_oq=&as_eq=&as_occt=any&as_sauthors=pathirana&as_publication=&as_ylo=&as_yhi=&hl=en&as_sdt=0%2C5 see our work]) regarding adaptation. Here we intentionally use the term "Urban Adaptation" instead of more common "[https://en.wikipedia.org/wiki/Climate_change_adaptation Climate Adaptation]". The reason for that is, in cities like Can Tho (or any other Secondary City in the Global South - SCGS), climate change -- while being extremely relevant and important -- is only one of many pressures that they have to deal with in adaptation; Land-cover change, rapid population increase, increasing pollution are some others.

In these two part series, we argue the importance of changing the way we look at urban adaptation in SCGS. Simply put, we are in a context that is a combination of high urgency and high uncertainty. But, we still use 'predictive' planning approaches that works well only in contexts of good predictability and less uncertainty. While there are multiple reasons (historical, cultural, economic) for this, it is important to start the discussion on how to change that to a more 'adaptive' -- or as we call it 'agile' approach. That is the story of these two articles. [[Urban_adaptation_in_rapidly_growing_cities_need_a_paradigm_shift|read more...]]

==Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems==
This paper puts forth a systematic approach to select the governmental actions that can ensure
water quality based on the connection between governmental actions at present and in the future
using Systems approach (Von Bertalanffy, 1968), SWOT analysis (Houben et al., 1999) and
DPSIR framework (OECD, 2003). The proposed methodology under the ambit of systems
approach employs: (i) SWOT analysis to identify the connections; and, (ii) DPSIR framework
to quantify the connections using impact based indicators for connections between actions. The
methodology has been tested using six governmental actions across four plausible future
scenarios in Luzhi Town, a unique water village within Suzhou city, China.
[[Capturing the changing dynamics between governmental actions across plausible future scenarios in urban water systems|read more...]]

==The book on Climate Change, Extreme Rainfall and Urban Drainage==
[[File:igur_cc_rain_book.gif|thumb|400px]]
Some members of International Group on Urban Rainfall ([http://www.kuleuven.be/hydr/gurweb/index.html IGUR]) of IWA/IAHR Joint Committee on Urban Drainage[http://www.jcud.org/] has produced a book on the topic on the influence on climate and other forms of future change on extreme rainfall and the implications on urban drainage systems.

In addition to a state-of-the-art overview of
existing methodologies and relevant results related to the
assessment of the climate change impacts on urban rainfall
extremes as well as on urban hydrology and hydraulics, it contains a number of tutorials on how to use the described techniques in practice. One example is an easy-to-use tutorial on how to use [Category:WRF_Model|WRF model] on personal computers. This is a similar to -- but very much improved (therefore easier to use) -- the framework described in the article "[[Running WRF Model on Windows]]". [[The book on Climate Change, Extreme Rainfall and Urban Drainage|read more...]]

==SWMM5-Python==
[[file:QT4Agg_Swmm5_python.png|300px|right|thumb]]
This is a python module for running SWMM5 model via python and extract results as python objects. [[SWMM5-Python|Read more..]]

==IMHEN Report on Climte Change in Vietnam==
(C) Institute of Meteorology, Hydrology and Environment (IMHEN), Vietnam.

IMHEN conducted the project "Impact of climate change on water resources and adaptation measures" sponsored by the Government of the Kingdom of Denmark with the participation of consulting experts from the Danish Hydraulic Institute (DHI) and the participation of many Vietnamese agencies. [[Vietnam: Impact of climate change on water resources and adaptation measures|read more]]

</div>
----
|width="55%" class="MainPageBG" style="border: 1px solid #F0FFF0; color: #000; background-color: #F0FFF0"|
<div style="clear: right; text-align: left; float: right; padding: .4em .9em .9em" id="MainPageBox2">

==Webapps with python==
[[File:water-detect-webapp.png|center|thumb|center|500px|Interactive map, analysis and plotting tool for precipitation. [https://waterdetect.srv.pathirana.net/ LINK]]]

One of the surefire ways of creating co-ownership is to encourage co-discovery and co-design. For this, we face the challenge of bringing modern technological knowledge to the stakeholders, including communities, in an understandable way but yet allowing for them to interact and contribute in a meaningful sense. One of the modern tools that contribute to this mission is interactive web applications. They allow water managers and scientists to bring complex data analysis solutions, big-data technologies, dynamic water models closer to the non-specialist stakeholders in an appealing and simple-to-interact fashion.

Here are some prototype web applications that were created for the water management, agriculture and asset management sectors. They are written in python using libraries like [https://plotly.com/dash/ Plotly dash] and [http://www.web2py.com/ web2py] to do the frontend. I use [https://auth0.com/blog/hosting-applications-using-digitalocean-and-dokku/ docker containers] based on [http://dokku.viewdocs.io/dokku/ dokku] -- [https://en.wikipedia.org/wiki/Platform_as_a_service a PaaS (Platform as a Service)] --to host these apps.
[[Webapps with python|Read more..]]

==Crowdsourcing groundwater quality in small islands==
[[file:fresh_water_lens_xsection.jpg|thumb|500px]]

The Maldives has more than 180 inhabited islands with many with populations less than 1000 inhabitants. Less populations do not mean they are facing lesser challenges to the sustainability or lower rate of change. In fact, these are among the most climate-vulnerable places on the earth and undergoing rapid change that need to be continuously monitored. But the challenge is every rufiyaa allocated for monitoring the vulnerability is money taken away from addressing the same vulnerability – a zero sum game. Therefore, in the Maldives, as well as other small islands, we need to continuously innovate on cost-effective, but also EFFECTIVE, ways of doing environmental monitoring.

But can we enlist the help of the inhabitants of these islands to do the monitoring of the groundwater resources. This is exactly what we did with 45 inhabited islands. A simple idea: Every household that has a groundwater well (nearly 100% in many islands) is a potential sensor of the health of the freshwater lens. We just have to enlist their support and suddenly we have thousand sensors to monitor our resources! This is exactly what we did. We surveyed more than 2100 households from the 45 islands and inquired them how the perceive the groundwater they obtain from the family wells. Does it smell? Looks clear? Do they use it for drinking and cooking? Washing, bathing and toilet flushing? We went on. The result was a dataset that revealed plenty of useful information. [[Crowdsourcing groundwater quality in small islands|Read more ...]]

==Urbanization not only heats up the cities, but could increase extreme rainfall==
[[File:heat_maps.png|thumb|450px|Latent heat and sensible heat flux maps for Mumbai (Simulation)]]
On 21 July 1999, Nerima-ku region of Tokyo received an unprecedented 111 mm of rainfall in one hour! Later studies lead to the conclusion that this heavy rainfall development was aided by urban heating due to the [[wikipedia:urban heat island|urban heat island]] effect of the Tokyo city.

With the cities in the world increasing in size at a phenomenal rate, the question whether larger urban footprints could enhance extreme rainfall is no longer solely an academic one. [[Urbanization not only heats up the cities, but could increase extreme rainfall.|Read more..]]

==Using Wiki in Higher-Education: Application in organizing research groups==
[[File:Wiki_in_edu_fig_2.png|thumb|400px|left|]]

This is a natural extension of the story of this web site. I started this web site in 2006 [[How this site works|hacking MediaWiki software]]. That was just before I moved from Japan to Netherlands and started working at UNESCO-IHE. From the end of 2006, I used the same contraption to support my research group at UNESCO-IHE (which consisted of five six masters students every year, myself and one or two other faculty members). The experiment completed its fifth year in 2011. I was thinking this is the time to document the experience when I came to know that [http://www.hydrol-earth-syst-sci.net/| Hydrology and Earth System Sciences Journal] is publishing [http://www.hydrol-earth-syst-sci-discuss.net/special_issue72.html special issue on Hydrology education in a changing world]]. Recently the article finished the review cycle and was published in HESS. [[Using Wiki in Higher-Education: Application in organizing research groups|Read More ...]]

==Improving Drinking Water Quality without Compromising Long-term Safety==

Treating water with chlorine is a time-tested way of ensuring biological safety of drinking water. However, excessive chlorination creates chlorination by-products, that are known to cause long-term risk of cancer. We attempted to make a safe compromise.

We integrated EPANET2.0 (a steady-stage, demand-driven water distribution network model), a particle back-tracking algorithm (can trace the origin of water delivered at any demand point in the network), chlorine and disinfection by product model and a multi-objective optimization algorithm to enable computing the optimal water treatment, minimizing long-term chemical risk (represented as cancer treatment cost.) [[Improving Drinking Water Quality without Compromising Long-term Safety|Read more...]]

==Ecosystem value of SuDS==
We analyse the implementation of Sustainable Drainage Systems (SuDS) as a solution to better
manage storm water runoffs and reduce urban flooding, and at the same time provide significant
Ecosystem Services (ES). ES vary from temperature control at urban and building scale to ma in water
savings, depending on the type of SuDS considered. [[Ecosystem benefits of SuDS|read more]]

</div>
|}
<div class="MainPageBG" style="padding: .5em 1em 0; margin: 0 3px 3px; border-bottom: 2px solid #ccc">
<h3 id="lang_h3" class="lang"></h3>
</div>
<div class="MainPageBG" style="padding: .5em 1em 1em; margin: 3px;">
===<span id="sister_heading" class="sister"></span>===

<div style="clear:left"></div>
</div> __NOTOC__ __NOEDITSECTION__
{{main links}}
[[Category:Meta]]

Systems thinking

2022-03-21T08:35:44Z

Assela.pathirana.admin: Assela.pathirana.admin moved page Systems thinking to Systems thinking for Urban Water Systems

#REDIRECT [[Systems thinking for Urban Water Systems]]

Systems thinking for Urban Water Systems

2022-03-21T08:35:44Z

Assela.pathirana.admin: Assela.pathirana.admin moved page Systems thinking to Systems thinking for Urban Water Systems

[[image:Systems_thinking_cartoon.jpg|thumb|800px|Systems thinking for water systems]]

By Dr. [https://www.linkedin.com/in/mohanasundar-radhakrishnan-27757b1/?originalSubdomain=nl Mohanasundar Radhakrishnan]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

Managing the water losses and reducing the risks in the urban system are the outcomes of water system understanding. Understanding the systems leads to the identification of various risks based on impacts and probability of failure. The participants will learn the basics of risk management, resulting in the prioritisation of assets that subsequently leads to asset management of urban water systems and water loss reduction. Certain losses are inevitable from an urban water system and have to be accepted. This acceptance can only happen when the interplay of elements in the urban systems is understood through social, political and economic perspectives and by looking at the urban water systems from various urban water stakeholders’ points of view.

An urban water strategist understands and embraces the socio-political turbulence that is omnipresent in the urban system and makes use of it to manage the losses. Such an understanding of the urban system and urban water system can be achieved through systems of thinking, essential for strategic planning. Systems thinking will enable the participants to understand the characteristics of urban systems; the traps and opportunities in the urban systems; and, the identification of the levers of the system for the change to reduce losses from the systems, not just the water loss. The participant will learn that is not just water that is lost from the urban systems and will understand the other elements that are flowing through the pipes and more importantly start exploring why are these losses happening before starting to quantify them.

[[:file:Systems_thinking_radhakrishnan.pdf|Download the full report here]]

Systems thinking for Urban Water Systems

2022-03-21T08:34:35Z

Assela.pathirana.admin:

Systems thinking for Urban Water Systems

2022-03-21T08:32:52Z

Assela.pathirana.admin:

[[image:Systems_thinking_cartoon.jpg|thumb|800px|Systems thinking for water systems]]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

Managing the water losses and reducing the risks in the urban system are the outcomes of water system understanding. Understanding the systems leads to the identification of various risks based on impacts and probability of failure. The participants will learn the basics of risk management, resulting in the prioritisation of assets that subsequently leads to asset management of urban water systems and water loss reduction. Certain losses are inevitable from an urban water system and have to be accepted. This acceptance can only happen when the interplay of elements in the urban systems is understood through social, political and economic perspectives and by looking at the urban water systems from various urban water stakeholders’ points of view.

An urban water strategist understands and embraces the socio-political turbulence that is omnipresent in the urban system and makes use of it to manage the losses. Such an understanding of the urban system and urban water system can be achieved through systems of thinking, essential for strategic planning. Systems thinking will enable the participants to understand the characteristics of urban systems; the traps and opportunities in the urban systems; and, the identification of the levers of the system for the change to reduce losses from the systems, not just the water loss. The participant will learn that is not just water that is lost from the urban systems and will understand the other elements that are flowing through the pipes and more importantly start exploring why are these losses happening before starting to quantify them.

[[:file:Systems_thinking_radhakrishnan.pdf|Download the file here]]

Systems thinking for Urban Water Systems

2022-03-21T08:32:25Z

Assela.pathirana.admin:

[[image:Systems_thinking_cartoon.jpg|thumb|800px|Systems thinking for water systems]]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

Managing the water losses and reducing the risks in the urban system are the outcomes of water system understanding. Understanding the systems leads to the identification of various risks based on impacts and probability of failure. The participants will learn the basics of risk management, resulting in the prioritisation of assets that subsequently leads to asset management of urban water systems and water loss reduction. Certain losses are inevitable from an urban water system and have to be accepted. This acceptance can only happen when the interplay of elements in the urban systems is understood through social, political and economic perspectives and by looking at the urban water systems from various urban water stakeholders’ points of view.

An urban water strategist understands and embraces the socio-political turbulence that is omnipresent in the urban system and makes use of it to manage the losses. Such an understanding of the urban system and urban water system can be achieved through systems of thinking, essential for strategic planning. Systems thinking will enable the participants to understand the characteristics of urban systems; the traps and opportunities in the urban systems; and, the identification of the levers of the system for the change to reduce losses from the systems, not just the water loss. The participant will learn that is not just water that is lost from the urban systems and will understand the other elements that are flowing through the pipes and more importantly start exploring why are these losses happening before starting to quantify them.

[[file:Systems_thinking_radhakrishnan.pdf|Download the file here]]

Systems thinking for Urban Water Systems

2022-03-21T08:32:11Z

Assela.pathirana.admin:

File:Systems thinking radhakrishnan.pdf

2022-03-21T08:31:33Z

Assela.pathirana.admin:

Systems thinking for Urban Water Systems

2022-03-21T08:26:28Z

Assela.pathirana.admin:

File:Systems thinking cartoon.jpg

2022-03-21T08:25:36Z

Assela.pathirana.admin:

Systems thinking for Urban Water Systems

2022-03-21T08:25:13Z

Assela.pathirana.admin:

[[image:systems_thinking_cartoon.jpg|thumb|800px|Systems thinking for water systems]]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

Managing the water losses and reducing the risks in the urban system are the outcomes of water system understanding. Understanding the systems leads to the identification of various risks based on impacts and probability of failure. The participants will learn the basics of risk management, resulting in the prioritisation of assets that subsequently leads to asset management of urban water systems and water loss reduction. Certain losses are inevitable from an urban water system and have to be accepted. This acceptance can only happen when the interplay of elements in the urban systems is understood through social, political and economic perspectives and by looking at the urban water systems from various urban water stakeholders’ points of view.

An urban water strategist understands and embraces the socio-political turbulence that is omnipresent in the urban system and makes use of it to manage the losses. Such an understanding of the urban system and urban water system can be achieved through systems of thinking, essential for strategic planning. Systems thinking will enable the participants to understand the characteristics of urban systems; the traps and opportunities in the urban systems; and, the identification of the levers of the system for the change to reduce losses from the systems, not just the water loss. The participant will learn that is not just water that is lost from the urban systems and will understand the other elements that are flowing through the pipes and more importantly start exploring why are these losses happening before starting to quantify them.

[[file:systems_thinking_radhakrishnan.pdf]]

File:Systems thinking cartoon.png

2022-03-21T08:12:24Z

Assela.pathirana.admin:

Systems thinking for Urban Water Systems

2022-03-21T08:11:10Z

Assela.pathirana.admin:

[[image:systems_thinking_cartoon.png|thumb|800px|Systems thinking for water systems]]

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

Managing the water losses and reducing the risks in the urban system are the outcomes of water system understanding. Understanding the systems leads to the identification of various risks based on impacts and probability of failure. The participants will learn the basics of risk management, resulting in the prioritisation of assets that subsequently leads to asset management of urban water systems and water loss reduction. Certain losses are inevitable from an urban water system and have to be accepted. This acceptance can only happen when the interplay of elements in the urban systems is understood through social, political and economic perspectives and by looking at the urban water systems from various urban water stakeholders’ points of view.

An urban water strategist understands and embraces the socio-political turbulence that is omnipresent in the urban system and makes use of it to manage the losses. Such an understanding of the urban system and urban water system can be achieved through systems of thinking, essential for strategic planning. Systems thinking will enable the participants to understand the characteristics of urban systems; the traps and opportunities in the urban systems; and, the identification of the levers of the system for the change to reduce losses from the systems, not just the water loss. The participant will learn that is not just water that is lost from the urban systems and will understand the other elements that are flowing through the pipes and more importantly start exploring why are these losses happening before starting to quantify them.

[[file:systems_thinking_radhakrishnan.pdf]]

Systems thinking for Urban Water Systems

2022-03-21T08:08:42Z

Assela.pathirana.admin: Created page with "What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three p..."

What is lost in an urban water system? The answer is obvious when you look at what flows in and out of these urban water networks, it is water. Some might say money and some might say energy. Yes, this is true but is not the whole truth. This training is about finding out what is lost from an urban water system by looking beyond water and looking in the urban systems to track the lost water and the other elements that are being lost. The training touches upon the three prime movers of the urban water utility, namely engineering, management and strategic planning. The place, proportion and play of this trinity in an urban water utility can be ascertained only by a thorough understanding of the urban system that is served by the water utility. This can shed light on what is actually gained and lost through urban water systems.

We the engineers and managers of urban water utilities are trained to ascertain the problem, quantify the impact, find out why it happened and devise ways to minimise or eliminate the problem. Quantification has always been the guiding principle, the means and the end in this endeavour. The water balance methods in their original or modified form used all over the world follow this logic and has proved to be a success in estimating the quantity of non-revenue water and formulating actions to reduce the losses in terms of water and revenue. The training starts by introducing the conventional water loss assessment methods to the urban water engineers and managers; the various elements of a water distribution system comprising physical, conceptual and design aspects; and, water loss management measures. Understanding and analysing these engineering aspects of the water supply system would create a comfort zone from which the engineer can start the process of identifying and minimising losses.

Managing the water losses and reducing the risks in the urban system are the outcomes of water system understanding. Understanding the systems leads to the identification of various risks based on impacts and probability of failure. The participants will learn the basics of risk management, resulting in the prioritisation of assets that subsequently leads to asset management of urban water systems and water loss reduction. Certain losses are inevitable from an urban water system and have to be accepted. This acceptance can only happen when the interplay of elements in the urban systems is understood through social, political and economic perspectives and by looking at the urban water systems from various urban water stakeholders’ points of view.

An urban water strategist understands and embraces the socio-political turbulence that is omnipresent in the urban system and makes use of it to manage the losses. Such an understanding of the urban system and urban water system can be achieved through systems of thinking, essential for strategic planning. Systems thinking will enable the participants to understand the characteristics of urban systems; the traps and opportunities in the urban systems; and, the identification of the levers of the system for the change to reduce losses from the systems, not just the water loss. The participant will learn that is not just water that is lost from the urban systems and will understand the other elements that are flowing through the pipes and more importantly start exploring why are these losses happening before starting to quantify them.

[[file:systems_thinking_radhakrishnan.pdf]]