Monday, February 22, 2010

Setup a http based SVN repository with TortoiseSVN and Apache http server

It is very useful to setup a http based SVN repository on your local machine just for daily check in and out of source code. I am pretty sure there are plenty of materials over the web that explain how to do it for different combinations. Here is the one which I went through and it works pretty good for me.
1. Install and configure TortoiseSVN
2. Install Apache HTTP Server
    • Download Apache HTTP Server from its' website at http://httpd.apache.org/download.cgi#apache22, and the version I’m using is 2.2.14 (either the one with or without ssl); try avoid using 1.x or 3.x which either doesn’t work well or hasn’t been tried by many people. So apache 2.2 is the best pick for now;
    • Install Apache http server by following the wizards;
    • Change the Apache HTTP Server listening port if necessary in conf.
    • After installation, just test it by launching http://localhost:port/ in browser and you should see a “It works!” page; (you can also monitor the apache http service in Apache Service Monitor)
3. Install Subversion
    • Download the latest version of the Subversion Win32 binaries for Apache. Be sure to get the right version to integrate with your version of Apache, otherwise you will get an obscure error message when you try to restart. If you have Apache 2.2.x go to http://subversion.tigris.org/servlets/ProjectDocumentList?folderID=8100;
    • !!!DON'T!!! use the msi version of subversion (it doesn't work somehow), instead use the zip version; (after unzip the subversion folder to your local disk, you also need to add the bin path to 'path' system environment variable)
    • Using the windows explorer, go to the installation directory of Subversion and find the files /httpd/mod_dav_svn.so and mod_authz_svn.so. Copy these files to the Apache modules directory;
    • Copy the file /bin/libdb*.dll and /bin/intl3_svn.dll from the Subversion installation directory to the Apache bin directory;
4. Configure Subversion and Apache HTTP Server
Edit Apache's configuration file (usually C:\Program Files\Apache Group\Apache2\conf\httpd.conf) with a text editor such as Notepad and make the following changes:
Uncomment (remove the '#' mark) the following lines:
    #LoadModule dav_fs_module modules/mod_dav_fs.so
    #LoadModule dav_module modules/mod_dav.so
Add the following two lines to the end of the LoadModule section.
        LoadModule dav_svn_module modules/mod_dav_svn.so
    LoadModule authz_svn_module modules/mod_authz_svn.so
b. At the end of the config file add the following lines:
    <Location /svn>
         DAV svn
         SVNListParentPath on
         SVNParentPath D:\SVN
         #SVNIndexXSLT "/svnindex.xsl"
         AuthType Basic
         AuthName "Subversion repositories"
         AuthUserFile passwd
         #AuthzSVNAccessFile svnaccessfile
         Require valid-user
     </Location>

c. This configures Apache so that all your Subversion repositories are physically located below D:\SVN. The repositories are served to the outside world from the URL: http://MyServer/svn/ . Access is restricted to known users/passwords listed in the passwd file.

d. To create the passwd file, open the command prompt (DOS-Box) again, change to the apache2 folder (usually c:\program files\apache group\apache2) and create the file by entering

           bin\htpasswd -c passwd <username>

This will create a file with the name passwd which is used for authentication. Additional users can be added with

          bin\htpasswd passwd <username>

e. Restart the Apache service again;

f. Point your browser to http://MyServer/svn/MyNewRepository (where MyNewRepository is the name of the Subversion repository you created before). If all went well you should be prompted for a username and password, then you can see the contents of your repository.
5. Create SVN repository and access it through HTTP
    • Create a folder on your local disk (under the folder where you specify in Apache http server to be the root folder of your SVN);
    • Right click on the folder and use the TortoiseSVN context menu “Create Repository Here”;
    • Access your SVN repository through HTTP as http://server:port/svn/repositoryName/ (you need to type in username and password).

Wednesday, February 3, 2010

Google Latitude finally locates me

I joined Google Latitude more than half year ago but no matter whether I’m at home (Rancho Cucamonga, CA) or at my office (Redlands, CA), it always tells me that I’m in Hague Netherlands, which I assume is what my friends will see too. Recently it finally corrects the location to Rancho Cucamonga or Redlands. Since I am not using Google Latitude on my mobile device, the only way Google Latitude can rely on should be the IP address and associated ISP. Just wonder why there could be such a big mistake.

Friday, January 22, 2010

First experience of armchair mapping using Mapzen and JSOM WMS plug-in

In previous post I shared my first experience of making changes to OSM database with Potlatch and JOSM. As I mentioned there both of them are good tools but neither is perfect so I’m still looking for some alternatives.

Mapzen caught my eye recently probably because it becomes quite a popular buzz word in blogs and news feedings so I decided to give it a try. At first glance, Mapzen looks quite similar to Potlatch (CloudMade who hosts Mapzen is founded by the same person who started OSM):

1. It requires an account (you can register it for free) to work with;

2. A peek at source of Mapzen front page shows it’s based on Flex too;

3. It uses OSM map as base layer in view mode;

4. It uses Yahoo Imagery as based layer in edit mode; (by the way, Yahoo Aerial Imagery is probably the only imagery available for free which has decent globe coverage. I never see other options, do you?)

Mapzen in view mode:

1

Mapzen in edit mode:

2

I like the way it categorize those pre-defined features in Points, Lines and Shapes and each one of the feature types has a meaningful icon associated. Of course it lacks some of the advanced editing functionalities that are available in Potlatch (it seems to have a tool to line up points) and JOSM, but it has more than enough for majority of the users. Another thing I like Mapzen over Potlatch is the look and feel when you draw the geometry on map. With Potlatch I always feel like using a big brush for wall painting, where all I actually need is pencil for sketch. The only weakness though is that I didn’t find a way to upload data as input for my editing, either through an osm file or a gpx track while Potlatch provides that option. Everything must be hand drawn. But anyway you can always combine the power of both.

After editing simply click “Save Map” button at corner (you might want to input some notes for this particular editing). The picture above shows the hand-draw building and foot way of my house using Mapzen, as well as a BBQ island garden “August Town” (named after my boy).

JOSM with its WMS plugin (with the Yahoo Aerial Imagery extension) is another option I tried after Mapzen, which I probably rank the highest among all three of them (Potlatch, Mapzen and JOSM). Like I mentioned before, the lack of a decent base imagery is a huge bottleneck which makes it almost impossible for armchair mapping, but the WMS plug-in with Yahoo Aerial Imagery extension overcomes it like magic. So let’s equip with it first by following the instructions here. A brief list of steps:

Install WMS plugin itself:

With the plugin manager

You can easily install plugins from within JOSM as follows

  1. Start JOSM, open the preferences window (Edit->Preferences or use the toolbar icon) and select the plugins tab.
  2. Click on "Download List" to download the list of available plugins.
  3. Check the plugins you want installed.
  4. Click the accept button. All new plugins should start downloading and installing.
  5. Restart JOSM.
Manually

With older versions (up until 277), you have to install the plugins manually.

  1. Download the plugin file from wherever the plugin is hosted. Look in the plugin page or the 'Plugins' page on the JOSM wiki site for the location of this file.
  2. The file should have an ".jar" extension. If it doesn't, rename the downloaded file so that it ends with ".jar". Internet Explorer, for instance, may rename some files to ".zip".
  3. Move the .jar file to the JOSM preferences directory ("%APPDATA%/JOSM" in windows, "~/.josm/plugins/" in Unix/MacOS.)
  4. Start JOSM, open the preferences window (Edit->Preferences) and select the plugins tab.
  5. Activate the plugin in the plugins tab.
  6. Restart JOSM.

Install Yahoo! Aerial Imagery Downloader

On Windows use the WebKit based downloader called webkit-image as follows

  • Download webkit-image.zip
  • Unzip it.
  • Move the contents so that the DLL files and the EXE file are somewhere "on your system path" (eg. c:\windows ). The best way to achieve this might be to place them alongside josm-latest.jar Keep the 'imageformats' subfolder alongside too (so all the contents of the zip).
  • Restart JOSM
  • Do 'WMS' menu -> YAHOO (Webkit)

You should start to get Yahoo! imagery (may take up to 30 seconds to start showing). If not, it may not be finding the DLL files correctly.

Note: If you don't want to place webkit-image in your system path or the JOSM directory you don't have to. By editing the download program you can specify an absolute or relative path to the webkit-image executable. Examples:

  • webkit/webkit-image {0} - loads webkit-image.exe from the subdirectory webkit relative to the JOSM installation directory
  • D:/webkit/webkit-image {0} - uses webkit-image D:\webkit\webkit-image.exe

After a successful install, restart JOSM and you should a new menu called “WMS” like below:

3

Now download your data from OSM database as usual first, and after select the “WMS” menu and select “Yahoo Sat”. Wait for about 5, 6 seconds you will see a nice aerial imagery base layer underneath your OSM data.

4

Well done! And this is perfect for armchair mapping. Now enjoy your smooth mapping experience with powerful tools in JOSM. I didn’t try out the WMS layer and rectified image yet which can definitely be better base layer options for specific area.

After the editing simply click upload button (right next to download button) and upload the changes. (you may need to set your OSM account username and password)

6

5

So my conclusion is that JOSM with WMS plugin is an obvious winner due to the smooth and rich desktop UI experience if you’re doing armchair mapping at home. But Potlatch and Mapzen have their own advantages of being browser based and easy to use for non-professional users.

Wednesday, January 20, 2010

Install Apache HTTP Server + PHP5 on Windows 7 (64bit)

Recent installation of dokuwiki (a simple web-based wiki engine) requires a combination of Apache HTTP Server + PHP5 on my 64-bit Windows 7 machine, but setup process didn’t seem to be as simple as “following the wizards”. So I decide to jot a few lines down such that I don’t have to search on the web again when I need to repeat the installation.

1. Install Apache HTTP Server; (just go to the official site and pick up .msi, and install in default path C:\Program Files(x86)\Apache Software Foundation\ seems to be ok)
2. Install PHP5;

This worth a short paragraph. My understanding of the install steps on PHP website is that it should be automatically taken care of by the installation wizards if it is for a popular http server like apache, and the install wizard UI does indicate that too. But unfortunately I didn’t get it to work in which I either got an general error message or no error message but just not working. So I switch to the zip version of PHP5 which is suggested by some article online and following the steps below makes it work properly:
2.1 Unzip the the zip version of PHP5 to your local disk; (unzip it to C:\Program Files(x86)\PHP5 seems to be working too)
2.2 Open httpd.conf file of Apache HTTP server in a text editor;
2.3 At the end of the http.conf file add following lines to make Apache HTTP Server be aware of PHP5; (adjust you path accordingly)
1: # Begin enable php5
2: LoadModule php5_module "C:/Program Files (x86)/PHP5/php5apache2_2.dll"
3: AddType application/x-httpd-php .php
4: PHPIniDir "C:/Program Files (x86)/PHP5"
5: # End enable php5
2.4 Save the change and restart Apache HTTP Server, and if successful you should see
“Apache/2.2.14(Win32) PHP/5.3.1” in the status bar of Apache Service Monitor.
Now your Apache HTTP Server should be able to display php content, at least my php based dokuwiki is running perfect.

Tuesday, January 19, 2010

Google Maps Navigation failed me again

As an Android G1 mobile phone user and a fan of Google products and services, I was really excited when the GPS killer app Google Maps Navigation was ported to Android 1.6. But I don’t usually drive to a lot of new places that I have no idea how to get to, even when I do I will do my homework on desktop first, which leaves Google Maps Navigation on my Android phone very few chances to prove its value. But the only time I was trying to rely on it to help me out, it failed. That was about one month ago, I am on my way to metro station in Riverside to pick up my wife after work. Due to the heavy traffic on CA-91 I decided to switch to the station right after the original one (Riverside-La Sierra station), which as I recall later was the first time Google Maps Navigation on my G1 phone coming to rescue in a real situation rather than a fake drill. It actually did a very good job for the most part, which I even see the metro station I was looking for (the red place marker). But amazing thing happened when I passed by the destination (I should have followed the green dash line). First Google Maps Navigation didn’t tell me that I had arrived (I had a second thought that I should just turn into the parking lot, but I hesitated because Google never failed me before), instead it told me to get onto the highway that I just got off again in other direction (the solid green line). I followed although with a little doubt (still believe Google was right) but was totally pissed off after a few seconds when right in the middle of the ramp (where the red cross is) it said that I’ve arrived.

I followed the solid green line which Google Maps Navigation told me to, but the red cross is the place where it said that I’ve arrived.

4

Today the fancy free Google Maps Navigation failed me again. I was looking for Bed Bath & Beyond in Mira Loma, CA, which seems to be pretty new because my 3-year old Magellan GPS wasn’t able to find it at all. Searching in Google Maps Navigation (by typing “Bed Bath and Beyond” and click search) gave me a matching Bed Bath & Beyond in Mira Loma, which has the accurate telephone number and address. But obviously it put the spin at a wrong place which is about half mile north west of the actual location in a lake. Using the same searching pattern in Google Maps generates the same error, but if you type the result address back in both Google Maps and Navigation points to the right place.

Bed Bath & Beyond in Swan Lake

2

The red place marker is where Google Maps Navigation directed me to, while the green spin is the correct place:

1

Another thing I notice is that the streets around wrong Bed Bath & Beyond location in Google Maps hasn’t been updated at all, or at least not recent enough to reflect current name. An example is that the “Peachtree Dr” in red circle doesn’t exist any more. Looking at OpenStreetMap for the same area gives me the accurate result as below:

3

Friday, January 15, 2010

Customize WMS GetFeatureInfo response :: GeoServer 2.0 versus MapServer 5.4.2

OGC Web Map Service (WMS) specification is a simple and popular protocol for serving maps over the web nowadays. But everything has two sides and the weakness of WMS is that it doesn’t define a standard format for its GetFeatureInfo operation, neither does it reference any existing standard like GML or so. The direct consequence of that is the truth that most of today’s WMS server implementations speaks different “languages” in their query results which results in the opposite of interoperability between servers and clients from different vendors.

Using HTML for WMS GetFeatureInfo somehow alleviate the situation, because without parsing clients can just throw HTML to browser. Some other implementations support GML to be more interoperable, but again those are just workarounds and the real issue should be solved in spec itself.
Anyway while waiting for the next version of spec (1.4.0?, 2.0? or else) recently, I spent some time with GeoServer 2.0 and MapServer 5.4.2 to customize the WMS GetFeatureInfo response in HTML, which is a feature available in both popular open source WMS server implementations.

What is the goal?

I have a simple vector dataset for San Francisco area in shape file format, which I published as WMS in both GeoServer and MapServer. Below is the map I will get when I send a WMS GetMap request:

http://<wms_service_url>?REQUEST=GetMap&SERVICE=WMS&VERSION=1.1.1&LAYERS=blockgroups,highways,pizzastores&STYLES=&FORMAT=image/png&BGCOLOR=0xEEEEEE&TRANSPARENT=false&SRS=EPSG:4326&BBOX=-122.545074509804,37.6736653056517,-122.35457254902,37.8428758708189&WIDTH=1020&HEIGHT=906

sf
Now if I send a GetFeatureInfo request to query features on the map:

http://<wms_service_url>?REQUEST=GetFeatureInfo&SERVICE=WMS&VERSION=1.1.1&LAYERS=pizzastores,highways,blockgroups&STYLES=&FORMAT=image/png&BGCOLOR=0xFFFFFF&TRANSPARENT=TRUE&SRS=EPSG:4326&BBOX=-122.545074509804,37.6736653056517,-122.35457254902,37.8428758708189&WIDTH=1020&HEIGHT=906&QUERY_LAYERS=pizzastores,highways,blockgroups&X=652&Y=368&INFO_FORMAT=text/html

Zero or more features records will be returned (Note: this particular request actually returns records from all three layers: ‘pizzastores’, ‘highways’, and ‘blockgroups’). So what I am trying to achieve here is to have both GeoServer and MapServer output GetFeatureInfo results in html with styles like below:
sf2

It’s a very simple customization but it proves the ability in GeoServer and MapServer, and of course more rich HTML elements like charts & pies, audios and videos can be easily added.

MapServer 5.4.2
For MapServer (the ms4w.exe that contains MapServer 5.4.2) I finally made what I planned, but I have to say that the whole user experience is far less smooth than I expected. A non-guru user like me mostly replies on the doc and samples as a start point, but the information regarding to this topic is scattered all over piece by piece without links pointing to each other. So compared to this I liked the new GeoServer 2.0 user manual much better.

To make shape file layer queryable in MapServer WMS, I started from a sample map file that I modified from online sample, and here is a section for one of my layers
1: # ===============================================================================
2: # highways layer
3: # ===============================================================================
4: LAYER
5:     NAME "highways"    
6:     METADATA
7:         "ows_title" "highways"
8:         "ows_srs"   "EPSG:4326"
9:     END
10:     TYPE LINE
11:     STATUS ON    
12:     DATA "./sanfrancisco/shp/highways"    
13:     PROJECTION
14:         "init=epsg:4326"
15:     END    
16:     CLASS    
17:         NAME "highways"            
18:         STYLE        
19:             WIDTH 1
20:             COLOR 0 0 255
21:         END
22:     END      
23: END
But unfortunately, all WMS layers are defined unqueryable (you will see <Layer …queryable=”0”…> in WMS capabilities files) by default, and online documentation doesn’t say anything clear on how to enable query on WMS layers. I figured out in the end by searching through the forum, in which some others people are asking the similar questions. Basically you have to add following line in each layer definition in map file:
1: LAYER
2:     ...
3:     TEMPLATE "blank.html"
4:     ...
5: END
“TEMPLATE” is suppose to point to a html file used as a template for WMS query result, and it makes WMS layer queryable even though the html file you’re pointing to doesn’t exists (I just feel a little awkward about this)

Now I can actually get GetFeatureInfo response from WMS, but I encountered three more problems right way:

1. GetFeatureInfo response doesn’t support GML as it claims; by reading the MapServer WMS doc “Reference Section” it can be solved by adding “DUMP TRUE” into layer definition:
1: LAYER
2:     ...
3:     DUMP TRUE
4:     ...
5: END
2. Either GML response or plain text response of GetFeatureInfo doesn’t include any attribute of the result feature; by reading the doc, you can solve that for GML by adding “gml_include_items    all” in metadata of layer definition. But I didn’t complete get rid of the problem until I searched through the forum again and found another undocumented “wms_include_items”. So what you need is:
1: LAYER
2:     ...
3:     METADATA
4:         ...
5:         "gml_include_items"   "all"
6:         "wms_include_items"   "all"
7:         ...
8:     END
9:     ...
10: END
3. “text/html” is not supported in GetFeatureInfo response; this is also documented but in a very obvious place. “wms_feature_info_mime_type    text/html” in the web section of the map files fix the problem:
1: Map
2:     ...
3:     WEB
4:         ...
5:         "wms_feature_info_mime_type" "text/html"
6:         ...
7:     END
8:     ...
9: END
Until now I can finally start creating the html template for GetFeatureInfo response. Although not specific to WMS GetFeatureInfo response, there is a detailed documentation page on MapServer template. As expected, you can define a header template, a footer template and another template for the content.
1: LAYER
2:     ...
3:     HEADER "../template/getfeatureinfo_header.html" # header html template
4:     TEMPLATE "../template/getfeatureinfo_content.html" # content html template
5:     FOOTER "../template/getfeatureinfo_footer.html" # footer html template
6:     ...
7: END
All three html templates are specified at layer level which allows you to customize GetFeatureInfo response differently for individual layer. Two pros of MapServer template are (1) a lot of server related information are exposed in template which you can reference by operation “[]”, e.g. server host, port, map and layer metadata etc instead of just limited to query results of GetFeatureInfo;  (2) since there is no other template language involved, I always feel easier to embed javascript code in template which is a big plus. But there are also many cons in this work flow too. If you have more than one included in WMS “query_layers” and the query result has multiple features then the the html content in header and footer template will be repeated for every feature in query result. It’s probably not too difficult to tweak the template to achieve what you want, but I don’t see a clean solution if I just want one header and footer template for all layers. Another thing I didn’t figure out is how to loop through each feature in query result in a more generic way instead of hard code the attribute name in each layer. I think it’s a very simple and typical work flow but I just don’t know how to do it in MapServer. Currently what I did for my “pizzastores” point layer (other two layers have similar but separate templates too) is like below:

header template for “pizzastores” layer:
1: <!-- MapServer Template -->
2: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/transitional.dtd">
3: <html>
4:   <head>
5:     <!-- enforce the client to display result html as UTF-8 encoding -->  
6:     <meta http-equiv="content-type" content="text/html; charset=UTF-8"></meta>
7:     <style type="text/css">
8:       table, th, td {
9:         border:1px solid #e5e5e5;
10:         border-collapse:collapse;
11:         font-family: arial;          
12:         font-size: 80%;            
13:         color: #333333
14:       }             
15:       th, td {
16:         valign: top;
17:         text-align: center;
18:       }          
19:       th {
20:         background-color: #aed7ff
21:       }
22:       caption {
23:         border:1px solid #e5e5e5;
24:         border-collapse:collapse;
25:         font-family: arial;          
26:         font-weight: bold;
27:         font-size: 80%;      
28:         text-align: left;      
29:         color: #333333;        
30:       }
31:     </style>
32:     <title>GetFeatureInfo Response</title>
33:     
34:   </head>
35:   <body>
36:     <table>
37:       <caption>layer names: pizzastores</caption>
38:       <tbody>
39:         <th>Layer Name</th>
40:         <th>NAME</th>
41:         <th>ADDRESS</th>
42:         <th>TYPE</th>  
content template for “pizzastores” layer
1: <!-- MapServer Template -->
2:     <tr>
3:       <td>Pizzastores</td>
4:       <td>[item name=NAME format=$value escape=none]</td>
5:       <td>[item name=ADDRESS format=$value escape=none]</td>
6:       <td>[item name=TYPE format=$value escape=none]</td>
7:     </tr>
footer layer for “pizzastores” layer
1: <!-- MapServer Template -->    
2:       </tbody>
3:     </table>
4:     <br/>
5:   </body>
6: </html>
You notice that in content template for the layer I can only access the current single query result which makes me think the templates will be repeatedly called for each feature in query results. I got the result below in browser but I have multiple <html> tags in the source:
r
The native GML format for WMS GetFeatureInfo response doesn’t have such issue though.

GeoServer 2.0

For GeoServer (latest released version 2.0.0), I found the work flow of customizing WMS GetFeatureInfo is very straight forward and smooth. All related information and samples are described in “GetFeatureInfo Templates” and “Freemaker Templates” sections of the online user manual, which is neat. Similar to MapServer, GeoServer also uses the concept of header, footer and content html template (templates are with suffix .ftl which is just html with freemaker engine tags). There are two things I really like about GeoServer: (1) templates can be set at different levels like global, workspace (not tested though), datastore, layer so common header and footer template can be shared; (2) the content template is repeatedly applied for each feature collection (meaning all the query results from one layer) instead of each feature such that I can loop through each feature in a generic way which in the end reduces the number of templates I need.

The only place I got trapped is that the online documentation is up to date enough the reflect the data folder structure change introduced in GeoServer 2.0.0. The old “featuretypes” folder is gone (“workspaces” folder is replacing it) but the online manual has a lot of places pointing to it.

Here is what I did for geoserver:

I created header.ftl and footer.ftl templates at global level and copy them to GEOSERVER_DATA_DIR\templates\ (create templates folder if it doesn’t exist):
1: <html>
2:   <head>
3:     <title>Geoserver GetFeatureInfo output</title>
4:   </head>
5:   <style type="text/css">
6:         table, th, td {
7:       border:1px solid #e5e5e5;
8:       border-collapse:collapse;
9:       font-family: arial;          
10:       font-size: 80%;            
11:       color: #333333
12:     }             
13:     th, td {
14:       valign: top;
15:       text-align: center;
16:     }          
17:     th {
18:       background-color: #aed7ff
19:     }
20:     caption {
21:       border:1px solid #e5e5e5;
22:       border-collapse:collapse;
23:       font-family: arial;          
24:       font-weight: bold;
25:       font-size: 80%;      
26:       text-align: left;      
27:       color: #333333;
28:       
29:     }
30:   </style>
31:   <body>

1:   </body>
2: </html>
3: 
I created content.ftl template at datastore level which is copied to GEOSERVER_DATA_DIR\workspaces\<my_workspace>\<my_datastore>\
1: <table>
2:   <caption>layer names: ${type.name}</caption>
3:   <tr>
4:   <th>fid</th>
5:     <#list type.attributes as attribute>
6:       <#if !attribute.isGeometry>
7:       <th >${attribute.name}</th>
8:       </#if>
9:     </#list>
10:   </tr>
11: 
12:   <#assign odd=false>
13:   <#list features as feature>
14:     <#if odd>
15:       <tr class="odd">
16:     <#else>
17:       <tr>
18:     </#if>
19:     <#assign odd=!odd>
20:     <td>${feature.fid}</td>    
21:     <#list feature.attributes as attribute>
22:       <#if !attribute.isGeometry>
23:         <td>${attribute.value?string}</td>
24:       </#if>
25:     </#list>
26:     </tr>
27:   </#list>
28: </table>
29: <br/>
New templates seem to require a restart of GeoServer and after that I get GetFeatureInfo response displayed in browser like below:
r


Tuesday, January 12, 2010

Extend GeoServer with customized OWS service :: part 5

In previous post of this series, I summarize the basic work flow of GeoServer OWS dispatcher dispatching OWS requests based on request parameters “service”, “version”, and “request”. We already know that in “ags-ows” service project, the “export” method will finally be called to process the request, and it takes a request bean (AgsOwsExportRequest) instance as input parameter. So in this post let’s pick up from that point and have a look at how response is produced.

We already know ows dispatcher dispatches requests, but after it finds and calls the appropriate method in appropriate service class with request bean it doesn’t stop it job. Instead it keeps doing its job to produce the response.
1: ...
2: // this is where OWS dispatcher finds the service and method and execute it
3: // execute it
4: Object result = execute(request, operation);
5: 
6: // this is where OWS dispatcher keeps its job to produce response
7: //write the response
8: if (result != null) {
9:     response(result, request, operation);
10: }
11: ...
The code snippet above is from OWS dispatcher, the “execute” method actually calls the service operation method (in case of “ags-ows” service, it is the export method in AarcGISServerOWSService class) and return whatever it returns, which will be assigned to “result”. Look at export method again
1: // export method in ArcGISServerOWSService class
2: ...
3: public AgsOwsExportResponse export(AgsOwsExportRequest request) {
4:     return new AgsOwsExportResponse(this.geoServer);
5: }
6: ...
the result is actually in type of AgsOwsExportResponse (you saw that in part 2 but I told you not to worry about then). AgsOwsExportResponse is a subclass of org.geoserver.ows.Response, which is the core element for an OWS service to produce responses. Before we jump into the implementation of AgsOwsExportResponse, let’s first look at how ows dispatcher uses response class. Go back to dispatcher code again and step into the “response(result, request, operation);” method we saw before:
1: ...
2: // response method in org.geoserver.ows.Dispatcher class
3: void response(Object result, Request req, Operation opDescriptor)
4:     throws Throwable {
5: 
6:     // loop through all registered subclass of org.geoserver.ows.Response 
7:     //   in application context and try to match a unique response type based on
8:     //   the class type of result and the binding class of Response subclass   
9: 
10:     ...
11:     ... 
12:     Response response = (Response) responses.get(0);
13:     ...
14:     // you need to implement getMimeType() in Response subclass 
15:     req.httpResponse.setContentType(response.getMimeType(result, opDescriptor));
16:     ...
17:     // you need to implement write() in Response subclass
18:     response.write(result, output, opDescriptor);
19:     ...
20:     // finally flush out the response to clients
21:     req.httpResponse.getOutputStream().flush();
22: }
23: ...
Dispatcher will loop through all the subclasses of org.geoserver.ows.Response registered in application context and find a unique response class whose binding class matches the class type of result object returned by service operation method. Take AgsOwsExportResponse as an example, since it’s binding class is itself which exactly matches the result object returned by export method of ArcGISServerOWSService class, it will be picked up by dispatcher.

After dispatcher locates the appropriate response class, two methods actually matters and thus must be implemented. (1) getMimeType(), which returns mime type string of your response, and it will be set directly in the header of http response back to clients; (2) write(), which writes response body in output stream of http response back to clients. Below is a sample implementation of AgsOwsExportResponse class and how it is registered in application context:
1: public class AgsOwsExportResponse extends Response {
2: 
3:     public AgsOwsExportResponse() {
4:         super(AgsOwsExportResponse.class);
5:     }
6: 
7:     @Override
8:     public String getMimeType(Object value, Operation operation)
9:         throws ServiceException {  
10:         // to simplify the problem, always return image/png
11:         // but in your own ows service, you can decided based on what you support 
12:         //    and what clients request
13:         return "image/png";
14:     }
15:     ...
16:     @Override
17:     public void write(Object value, OutputStream output, Operation operation)
18:         throws IOException, ServiceException {  
19:         ...
20:         // write response map image back to clients 
21:         ...
22:     }
23: }

1: <!-- service operation response -->
2: <bean id="agsOwsCapabilitiesResponse" 
3:     class="org.geoserver.ows.arcgisserver.responses.AgsOwsCapabilitiesResponse"
4:     singleton="false">     
5:     <constructor-arg ref="geoServer"/>   
6: </bean>
Notice the binding class is AgsOwsExportResponse itself but it could definitely be something else if the service operation method export chooses to return something else (org.geoserver.wcs.responses.GetCapabilitiesResponse is a good example of that). As I mentioned in the beginning, “image/png” is hard coded in getMimeType() to simplify the problem. And finally in write() I need to produce a map image and write it in output stream. The implementation of write() method deserves another separate paragraph of explanation which I will do right after, and so far ows dispatcher has finished the whole life cycle for an ows service request dispatching.

Implement Write() in AgsOwsExportResponse and AgsOwsMapProducer

Implementing Write() in AgsOwsExportResponse can be as simple as less than 10 lines of code if you only want to produce and output some simple static content back to clients.
1: private void write(Object value, OutputStream output, Operation operation) 
2:     throws IOException, ServiceException {
3:     String responseStr = "Static content";
4:     OutputStreamWriter writer = new OutputStreamWriter(output);
5:     writer.write(responseStr);
6:     writer.flush();
7: }
But if it is creating a map image, it can definitely become way more complicated (org.vfny.geoserver.wms.responses.GetMapResponse in wms project is a good example), which sometimes needs one or more helper classes (those map producer classes under package org.vfny.geoserver.wms.responses.map are very good example). In AgsOwsExportResponse of “ags-ows” service project I sort of borrowed the idea from GeoServer WMS GetMapResponse class and PNGMapProducer class, in which a helper class similar to PNGMapProducer called AgsOwsMapProducer is created to render map. Of course I omitted most of the details and keeps just enough code to create an map image in png format and write out to client.

In the write() method of AgsOwsExportResponse, I create an instance of GraphicEnhancedMapContext class (from GeoTools library) and pass it to AgsOwsMapProducer. The GraphicEnhancedMapContext instance basically stores all information (e.g. map projection, background color, transparent, layers, styles etc.) needed to produce a map image using GeoTools.
1: private void write(Object value, OutputStream output, Operation operation)
2:       throws IOException, ServiceException {    
3:     
4:     AgsOwsExportResponse exportResponse = (AgsOwsExportResponse)value;
5:     AgsOwsExportRequest exportRequest 
6:         = (AgsOwsExportRequest)OwsUtils.parameter(operation.getParameters(), AgsOwsExportRequest.class);
7:     
8:     /*
9:      * the main purpose here is to create mapContext {GraphicEnhancedMapContext} -
10:      * - and pass it to map producer to generate the map
11:      * - borrow the idea from GetMapResponse but omitted a lot details
12:      */    
13:     // requested image crs
14:     final CoordinateReferenceSystem imageSR = exportRequest.getImageSR();
15:     // requested layers
16:     final LayerInfo[] layers = exportRequest.getLayers();          
17:     // initialize mapContext
18:     this.mapContext = new GraphicEnhancedMapContext();    
19:     try {
20:       this.mapContext.setCoordinateReferenceSystem(imageSR);
21:     } catch(FactoryException e) {
22:       e.printStackTrace();
23:     } catch(TransformException e) {
24:       e.printStackTrace();
25:     }        
26:     // bbox and bbox crs
27:     final CoordinateReferenceSystem bboxSR = exportRequest.getBboxSR();
28:     final Envelope bbox = exportRequest.getBbox();
29:     if(bboxSR != null) {
30:       this.mapContext.setAreaOfInterest(bbox, bboxSR);
31:     } else {
32:       // TODO: should throw exception, no?
33:       this.mapContext.setAreaOfInterest(bbox, DefaultGeographicCRS.WGS84);
34:     }
35:     
36:     this.mapContext.setMapWidth(exportRequest.getWidth());
37:     this.mapContext.setMapHeight(exportRequest.getHeight());
38:     this.mapContext.setBgColor(exportRequest.getBgColor());
39:     this.mapContext.setTransparent(exportRequest.isTransparent());
40:     
41:     try {
42:       for (int i=0; i<layers.length; i++) {      
43:         final Style layerStyle = layers[i].getDefaultStyle().getStyle();        
44:         final MapLayer layer;                
45:         if(layers[i].getType().getCode() == LayerInfo.Type.VECTOR.getCode()) {
46:           FeatureSource<? extends FeatureType, ? extends Feature> featureSource;
47:           FeatureTypeInfo resource = (FeatureTypeInfo)layers[i].getResource();
48:           if(resource.getStore() == null || resource.getStore().getDataStore(null) == null) {
49:                   throw new IOException("");
50:               }
51:           Hints hints = new Hints(ResourcePool.REPROJECT, Boolean.valueOf(false));
52:           featureSource = resource.getFeatureSource(null, hints);
53:           
54:           layer = new FeatureSourceMapLayer(featureSource, layerStyle);
55:                     layer.setTitle(layers[i].getResource().getName());                    
56:                     // use default filter and version in DefaultQuery
57:                     final DefaultQuery definitionQuery = new DefaultQuery(featureSource.getSchema().getName().getLocalPart());                                      
58:                     layer.setQuery(definitionQuery);
59:                     mapContext.addLayer(layer);                    
60:         } else if(layers[i].getType().getCode() == LayerInfo.Type.RASTER.getCode()) {
61:           //
62:         }
63:         
64:         // default outputFormat to 'png' and mimeType to 'image/png'
65:         this.mapProducer = new AgsOwsMapProducer();
66:         this.mapProducer.setMapContext(this.mapContext);
67:         
68:         this.mapProducer.produceMap();
69:         this.mapProducer.writeTo(output);
70:       }
71:     } catch(Exception e) {
72:       e.printStackTrace();
73:     }
74:   }
75: 
And AgsOwsMapProducer, which takes the map context, finally renders a map using GeoTools StreamingRenderer by calling produceMap() and write out image stream by calling writeTo().
1: ...
2: public void produceMap() throws Exception {  
3:     ...
4:     Rectangle paintArea 
5:         = new Rectangle(0, 0, mapContext.getMapWidth(), mapContext.getMapHeight());
6:     String antialias = "none";
7:     IndexColorModel palette = null;
8:     ...
9:     boolean useAlpha = true;
10:     final boolean transparent = mapContext.isTransparent();
11:     ...
12:     final Color bgColor = mapContext.getBgColor();
13:     ...
14:     final RenderedImage preparedImage 
15:         = ImageUtils.createImage(paintArea.width, paintArea.height, palette, useAlpha);
16:     ...
17:     final Map<RenderingHints.Key, Object> hintsMap 
18:         = new HashMap<RenderingHints.Key, Object>();      
19:     final Graphics2D graphic 
20:         = ImageUtils.prepareTransparency(transparent, bgColor, preparedImage, hintsMap);
21:     ...
22:     graphic.setRenderingHints(hintsMap);
23:     ...
24:     StreamingRenderer renderer = new StreamingRenderer();      
25:     renderer = new StreamingRenderer();        
26:     renderer.setContext(mapContext);
27:       
28:     RenderingHints hints = new RenderingHints(hintsMap);
29:     renderer.setJava2DHints(hints);
30:               
31:     Map<Object, Object> rendererParams = new HashMap<Object, Object>();
32:     rendererParams.put("optimizedDataLoadingEnabled", new Boolean(true));    
33:     rendererParams.put("maxFiltersToSendToDatastore",  DefaultWebMapService.getMaxFilterRules());
34:     rendererParams.put(ShapefileRenderer.SCALE_COMPUTATION_METHOD_KEY, ShapefileRenderer.SCALE_OGC);    
35:     ...
36:     rendererParams.put(StreamingRenderer.ADVANCED_PROJECTION_HANDLING_KEY, true);
37:     ...
38:     renderer.setRendererHints(rendererParams);
39:     try {
40:         final ReferencedEnvelope dataArea = mapContext.getAreaOfInterest();
41:         renderer.paint(graphic, paintArea, dataArea);        
42:     } finally {
43:         graphic.dispose();
44:     }
45:     ...
46:     this.image = preparedImage;
47: }
48: ...
49: public void writeTo(OutputStream out) throws ServiceException, java.io.IOException {
50:     ...
51:     final String format = getOutputFormat();
52:     ...
53:     new ImageWorker(image).writePNG(
54:         outStream, 
55:         "FILTERED", 
56:         100.0f,  
57:         true,
58:         image.getColorModel() instanceof IndexColorModel
59:     );
60:     ...
61: }
Note: the sample above for both produceMap() and writeTo() omitted a lot of details which can not be simply copied and paste to be compiled. But I will provide the link to download the sample code in the end of the series.

Wrap it up

So far I’ve finished everything I planned for this little “ags-ows” project. So just restart GeoServer and the customized OWS service “ags-ows” will be ready to serve out map. Here are two sample requests and result maps:

http://localhost:8080/geoserver/ows?service=agsows&request=export&version=1.0.0&bbox=-180,0,0,90&transparent=true&size=512,256&bboxSR=4326&imageSR=4326&layers=show:states
wgs84
http://localhost:8080/geoserver/ows?service=agsows&request=export&version=1.0.0&bbox=-20037508.34,-20037508.34,20037508.34,20037508.34&transparent=true&size=512,512&bboxSR=900913&imageSR=900913&layers=show:states
mercator

This is the final part of the whole series and the source code covered in my little "ags-ows" project is packaged as geoserver-playground-ags-ows.zip, which can be downloaded from here. Feel free to leave any feedback and comments.