Friday, January 15, 2010

Customize WMS GetFeatureInfo response :: GeoServer 2.0 versus MapServer 5.4.2

OGC Web Map Service (WMS) specification is a simple and popular protocol for serving maps over the web nowadays. But everything has two sides and the weakness of WMS is that it doesn’t define a standard format for its GetFeatureInfo operation, neither does it reference any existing standard like GML or so. The direct consequence of that is the truth that most of today’s WMS server implementations speaks different “languages” in their query results which results in the opposite of interoperability between servers and clients from different vendors.

Using HTML for WMS GetFeatureInfo somehow alleviate the situation, because without parsing clients can just throw HTML to browser. Some other implementations support GML to be more interoperable, but again those are just workarounds and the real issue should be solved in spec itself.
Anyway while waiting for the next version of spec (1.4.0?, 2.0? or else) recently, I spent some time with GeoServer 2.0 and MapServer 5.4.2 to customize the WMS GetFeatureInfo response in HTML, which is a feature available in both popular open source WMS server implementations.

What is the goal?

I have a simple vector dataset for San Francisco area in shape file format, which I published as WMS in both GeoServer and MapServer. Below is the map I will get when I send a WMS GetMap request:


Now if I send a GetFeatureInfo request to query features on the map:


Zero or more features records will be returned (Note: this particular request actually returns records from all three layers: ‘pizzastores’, ‘highways’, and ‘blockgroups’). So what I am trying to achieve here is to have both GeoServer and MapServer output GetFeatureInfo results in html with styles like below:

It’s a very simple customization but it proves the ability in GeoServer and MapServer, and of course more rich HTML elements like charts & pies, audios and videos can be easily added.

MapServer 5.4.2
For MapServer (the ms4w.exe that contains MapServer 5.4.2) I finally made what I planned, but I have to say that the whole user experience is far less smooth than I expected. A non-guru user like me mostly replies on the doc and samples as a start point, but the information regarding to this topic is scattered all over piece by piece without links pointing to each other. So compared to this I liked the new GeoServer 2.0 user manual much better.

To make shape file layer queryable in MapServer WMS, I started from a sample map file that I modified from online sample, and here is a section for one of my layers
1: # ===============================================================================
2: # highways layer
3: # ===============================================================================
5:     NAME "highways"    
7:         "ows_title" "highways"
8:         "ows_srs"   "EPSG:4326"
9:     END
10:     TYPE LINE
11:     STATUS ON    
12:     DATA "./sanfrancisco/shp/highways"    
14:         "init=epsg:4326"
15:     END    
16:     CLASS    
17:         NAME "highways"            
18:         STYLE        
19:             WIDTH 1
20:             COLOR 0 0 255
21:         END
22:     END      
23: END
But unfortunately, all WMS layers are defined unqueryable (you will see <Layer …queryable=”0”…> in WMS capabilities files) by default, and online documentation doesn’t say anything clear on how to enable query on WMS layers. I figured out in the end by searching through the forum, in which some others people are asking the similar questions. Basically you have to add following line in each layer definition in map file:
2:     ...
3:     TEMPLATE "blank.html"
4:     ...
5: END
“TEMPLATE” is suppose to point to a html file used as a template for WMS query result, and it makes WMS layer queryable even though the html file you’re pointing to doesn’t exists (I just feel a little awkward about this)

Now I can actually get GetFeatureInfo response from WMS, but I encountered three more problems right way:

1. GetFeatureInfo response doesn’t support GML as it claims; by reading the MapServer WMS doc “Reference Section” it can be solved by adding “DUMP TRUE” into layer definition:
2:     ...
3:     DUMP TRUE
4:     ...
5: END
2. Either GML response or plain text response of GetFeatureInfo doesn’t include any attribute of the result feature; by reading the doc, you can solve that for GML by adding “gml_include_items    all” in metadata of layer definition. But I didn’t complete get rid of the problem until I searched through the forum again and found another undocumented “wms_include_items”. So what you need is:
2:     ...
4:         ...
5:         "gml_include_items"   "all"
6:         "wms_include_items"   "all"
7:         ...
8:     END
9:     ...
10: END
3. “text/html” is not supported in GetFeatureInfo response; this is also documented but in a very obvious place. “wms_feature_info_mime_type    text/html” in the web section of the map files fix the problem:
1: Map
2:     ...
3:     WEB
4:         ...
5:         "wms_feature_info_mime_type" "text/html"
6:         ...
7:     END
8:     ...
9: END
Until now I can finally start creating the html template for GetFeatureInfo response. Although not specific to WMS GetFeatureInfo response, there is a detailed documentation page on MapServer template. As expected, you can define a header template, a footer template and another template for the content.
2:     ...
3:     HEADER "../template/getfeatureinfo_header.html" # header html template
4:     TEMPLATE "../template/getfeatureinfo_content.html" # content html template
5:     FOOTER "../template/getfeatureinfo_footer.html" # footer html template
6:     ...
7: END
All three html templates are specified at layer level which allows you to customize GetFeatureInfo response differently for individual layer. Two pros of MapServer template are (1) a lot of server related information are exposed in template which you can reference by operation “[]”, e.g. server host, port, map and layer metadata etc instead of just limited to query results of GetFeatureInfo;  (2) since there is no other template language involved, I always feel easier to embed javascript code in template which is a big plus. But there are also many cons in this work flow too. If you have more than one included in WMS “query_layers” and the query result has multiple features then the the html content in header and footer template will be repeated for every feature in query result. It’s probably not too difficult to tweak the template to achieve what you want, but I don’t see a clean solution if I just want one header and footer template for all layers. Another thing I didn’t figure out is how to loop through each feature in query result in a more generic way instead of hard code the attribute name in each layer. I think it’s a very simple and typical work flow but I just don’t know how to do it in MapServer. Currently what I did for my “pizzastores” point layer (other two layers have similar but separate templates too) is like below:

header template for “pizzastores” layer:
1: <!-- MapServer Template -->
2: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "">
3: <html>
4:   <head>
5:     <!-- enforce the client to display result html as UTF-8 encoding -->  
6:     <meta http-equiv="content-type" content="text/html; charset=UTF-8"></meta>
7:     <style type="text/css">
8:       table, th, td {
9:         border:1px solid #e5e5e5;
10:         border-collapse:collapse;
11:         font-family: arial;          
12:         font-size: 80%;            
13:         color: #333333
14:       }             
15:       th, td {
16:         valign: top;
17:         text-align: center;
18:       }          
19:       th {
20:         background-color: #aed7ff
21:       }
22:       caption {
23:         border:1px solid #e5e5e5;
24:         border-collapse:collapse;
25:         font-family: arial;          
26:         font-weight: bold;
27:         font-size: 80%;      
28:         text-align: left;      
29:         color: #333333;        
30:       }
31:     </style>
32:     <title>GetFeatureInfo Response</title>
34:   </head>
35:   <body>
36:     <table>
37:       <caption>layer names: pizzastores</caption>
38:       <tbody>
39:         <th>Layer Name</th>
40:         <th>NAME</th>
41:         <th>ADDRESS</th>
42:         <th>TYPE</th>  
content template for “pizzastores” layer
1: <!-- MapServer Template -->
2:     <tr>
3:       <td>Pizzastores</td>
4:       <td>[item name=NAME format=$value escape=none]</td>
5:       <td>[item name=ADDRESS format=$value escape=none]</td>
6:       <td>[item name=TYPE format=$value escape=none]</td>
7:     </tr>
footer layer for “pizzastores” layer
1: <!-- MapServer Template -->    
2:       </tbody>
3:     </table>
4:     <br/>
5:   </body>
6: </html>
You notice that in content template for the layer I can only access the current single query result which makes me think the templates will be repeatedly called for each feature in query results. I got the result below in browser but I have multiple <html> tags in the source:
The native GML format for WMS GetFeatureInfo response doesn’t have such issue though.

GeoServer 2.0

For GeoServer (latest released version 2.0.0), I found the work flow of customizing WMS GetFeatureInfo is very straight forward and smooth. All related information and samples are described in “GetFeatureInfo Templates” and “Freemaker Templates” sections of the online user manual, which is neat. Similar to MapServer, GeoServer also uses the concept of header, footer and content html template (templates are with suffix .ftl which is just html with freemaker engine tags). There are two things I really like about GeoServer: (1) templates can be set at different levels like global, workspace (not tested though), datastore, layer so common header and footer template can be shared; (2) the content template is repeatedly applied for each feature collection (meaning all the query results from one layer) instead of each feature such that I can loop through each feature in a generic way which in the end reduces the number of templates I need.

The only place I got trapped is that the online documentation is up to date enough the reflect the data folder structure change introduced in GeoServer 2.0.0. The old “featuretypes” folder is gone (“workspaces” folder is replacing it) but the online manual has a lot of places pointing to it.

Here is what I did for geoserver:

I created header.ftl and footer.ftl templates at global level and copy them to GEOSERVER_DATA_DIR\templates\ (create templates folder if it doesn’t exist):
1: <html>
2:   <head>
3:     <title>Geoserver GetFeatureInfo output</title>
4:   </head>
5:   <style type="text/css">
6:         table, th, td {
7:       border:1px solid #e5e5e5;
8:       border-collapse:collapse;
9:       font-family: arial;          
10:       font-size: 80%;            
11:       color: #333333
12:     }             
13:     th, td {
14:       valign: top;
15:       text-align: center;
16:     }          
17:     th {
18:       background-color: #aed7ff
19:     }
20:     caption {
21:       border:1px solid #e5e5e5;
22:       border-collapse:collapse;
23:       font-family: arial;          
24:       font-weight: bold;
25:       font-size: 80%;      
26:       text-align: left;      
27:       color: #333333;
29:     }
30:   </style>
31:   <body>

1:   </body>
2: </html>
I created content.ftl template at datastore level which is copied to GEOSERVER_DATA_DIR\workspaces\<my_workspace>\<my_datastore>\
1: <table>
2:   <caption>layer names: ${}</caption>
3:   <tr>
4:   <th>fid</th>
5:     <#list type.attributes as attribute>
6:       <#if !attribute.isGeometry>
7:       <th >${}</th>
8:       </#if>
9:     </#list>
10:   </tr>
12:   <#assign odd=false>
13:   <#list features as feature>
14:     <#if odd>
15:       <tr class="odd">
16:     <#else>
17:       <tr>
18:     </#if>
19:     <#assign odd=!odd>
20:     <td>${feature.fid}</td>    
21:     <#list feature.attributes as attribute>
22:       <#if !attribute.isGeometry>
23:         <td>${attribute.value?string}</td>
24:       </#if>
25:     </#list>
26:     </tr>
27:   </#list>
28: </table>
29: <br/>
New templates seem to require a restart of GeoServer and after that I get GetFeatureInfo response displayed in browser like below:

Tuesday, January 12, 2010

Extend GeoServer with customized OWS service :: part 5

In previous post of this series, I summarize the basic work flow of GeoServer OWS dispatcher dispatching OWS requests based on request parameters “service”, “version”, and “request”. We already know that in “ags-ows” service project, the “export” method will finally be called to process the request, and it takes a request bean (AgsOwsExportRequest) instance as input parameter. So in this post let’s pick up from that point and have a look at how response is produced.

We already know ows dispatcher dispatches requests, but after it finds and calls the appropriate method in appropriate service class with request bean it doesn’t stop it job. Instead it keeps doing its job to produce the response.
1: ...
2: // this is where OWS dispatcher finds the service and method and execute it
3: // execute it
4: Object result = execute(request, operation);
6: // this is where OWS dispatcher keeps its job to produce response
7: //write the response
8: if (result != null) {
9:     response(result, request, operation);
10: }
11: ...
The code snippet above is from OWS dispatcher, the “execute” method actually calls the service operation method (in case of “ags-ows” service, it is the export method in AarcGISServerOWSService class) and return whatever it returns, which will be assigned to “result”. Look at export method again
1: // export method in ArcGISServerOWSService class
2: ...
3: public AgsOwsExportResponse export(AgsOwsExportRequest request) {
4:     return new AgsOwsExportResponse(this.geoServer);
5: }
6: ...
the result is actually in type of AgsOwsExportResponse (you saw that in part 2 but I told you not to worry about then). AgsOwsExportResponse is a subclass of org.geoserver.ows.Response, which is the core element for an OWS service to produce responses. Before we jump into the implementation of AgsOwsExportResponse, let’s first look at how ows dispatcher uses response class. Go back to dispatcher code again and step into the “response(result, request, operation);” method we saw before:
1: ...
2: // response method in org.geoserver.ows.Dispatcher class
3: void response(Object result, Request req, Operation opDescriptor)
4:     throws Throwable {
6:     // loop through all registered subclass of org.geoserver.ows.Response 
7:     //   in application context and try to match a unique response type based on
8:     //   the class type of result and the binding class of Response subclass   
10:     ...
11:     ... 
12:     Response response = (Response) responses.get(0);
13:     ...
14:     // you need to implement getMimeType() in Response subclass 
15:     req.httpResponse.setContentType(response.getMimeType(result, opDescriptor));
16:     ...
17:     // you need to implement write() in Response subclass
18:     response.write(result, output, opDescriptor);
19:     ...
20:     // finally flush out the response to clients
21:     req.httpResponse.getOutputStream().flush();
22: }
23: ...
Dispatcher will loop through all the subclasses of org.geoserver.ows.Response registered in application context and find a unique response class whose binding class matches the class type of result object returned by service operation method. Take AgsOwsExportResponse as an example, since it’s binding class is itself which exactly matches the result object returned by export method of ArcGISServerOWSService class, it will be picked up by dispatcher.

After dispatcher locates the appropriate response class, two methods actually matters and thus must be implemented. (1) getMimeType(), which returns mime type string of your response, and it will be set directly in the header of http response back to clients; (2) write(), which writes response body in output stream of http response back to clients. Below is a sample implementation of AgsOwsExportResponse class and how it is registered in application context:
1: public class AgsOwsExportResponse extends Response {
3:     public AgsOwsExportResponse() {
4:         super(AgsOwsExportResponse.class);
5:     }
7:     @Override
8:     public String getMimeType(Object value, Operation operation)
9:         throws ServiceException {  
10:         // to simplify the problem, always return image/png
11:         // but in your own ows service, you can decided based on what you support 
12:         //    and what clients request
13:         return "image/png";
14:     }
15:     ...
16:     @Override
17:     public void write(Object value, OutputStream output, Operation operation)
18:         throws IOException, ServiceException {  
19:         ...
20:         // write response map image back to clients 
21:         ...
22:     }
23: }

1: <!-- service operation response -->
2: <bean id="agsOwsCapabilitiesResponse" 
3:     class="org.geoserver.ows.arcgisserver.responses.AgsOwsCapabilitiesResponse"
4:     singleton="false">     
5:     <constructor-arg ref="geoServer"/>   
6: </bean>
Notice the binding class is AgsOwsExportResponse itself but it could definitely be something else if the service operation method export chooses to return something else (org.geoserver.wcs.responses.GetCapabilitiesResponse is a good example of that). As I mentioned in the beginning, “image/png” is hard coded in getMimeType() to simplify the problem. And finally in write() I need to produce a map image and write it in output stream. The implementation of write() method deserves another separate paragraph of explanation which I will do right after, and so far ows dispatcher has finished the whole life cycle for an ows service request dispatching.

Implement Write() in AgsOwsExportResponse and AgsOwsMapProducer

Implementing Write() in AgsOwsExportResponse can be as simple as less than 10 lines of code if you only want to produce and output some simple static content back to clients.
1: private void write(Object value, OutputStream output, Operation operation) 
2:     throws IOException, ServiceException {
3:     String responseStr = "Static content";
4:     OutputStreamWriter writer = new OutputStreamWriter(output);
5:     writer.write(responseStr);
6:     writer.flush();
7: }
But if it is creating a map image, it can definitely become way more complicated (org.vfny.geoserver.wms.responses.GetMapResponse in wms project is a good example), which sometimes needs one or more helper classes (those map producer classes under package are very good example). In AgsOwsExportResponse of “ags-ows” service project I sort of borrowed the idea from GeoServer WMS GetMapResponse class and PNGMapProducer class, in which a helper class similar to PNGMapProducer called AgsOwsMapProducer is created to render map. Of course I omitted most of the details and keeps just enough code to create an map image in png format and write out to client.

In the write() method of AgsOwsExportResponse, I create an instance of GraphicEnhancedMapContext class (from GeoTools library) and pass it to AgsOwsMapProducer. The GraphicEnhancedMapContext instance basically stores all information (e.g. map projection, background color, transparent, layers, styles etc.) needed to produce a map image using GeoTools.
1: private void write(Object value, OutputStream output, Operation operation)
2:       throws IOException, ServiceException {    
4:     AgsOwsExportResponse exportResponse = (AgsOwsExportResponse)value;
5:     AgsOwsExportRequest exportRequest 
6:         = (AgsOwsExportRequest)OwsUtils.parameter(operation.getParameters(), AgsOwsExportRequest.class);
8:     /*
9:      * the main purpose here is to create mapContext {GraphicEnhancedMapContext} -
10:      * - and pass it to map producer to generate the map
11:      * - borrow the idea from GetMapResponse but omitted a lot details
12:      */    
13:     // requested image crs
14:     final CoordinateReferenceSystem imageSR = exportRequest.getImageSR();
15:     // requested layers
16:     final LayerInfo[] layers = exportRequest.getLayers();          
17:     // initialize mapContext
18:     this.mapContext = new GraphicEnhancedMapContext();    
19:     try {
20:       this.mapContext.setCoordinateReferenceSystem(imageSR);
21:     } catch(FactoryException e) {
22:       e.printStackTrace();
23:     } catch(TransformException e) {
24:       e.printStackTrace();
25:     }        
26:     // bbox and bbox crs
27:     final CoordinateReferenceSystem bboxSR = exportRequest.getBboxSR();
28:     final Envelope bbox = exportRequest.getBbox();
29:     if(bboxSR != null) {
30:       this.mapContext.setAreaOfInterest(bbox, bboxSR);
31:     } else {
32:       // TODO: should throw exception, no?
33:       this.mapContext.setAreaOfInterest(bbox, DefaultGeographicCRS.WGS84);
34:     }
36:     this.mapContext.setMapWidth(exportRequest.getWidth());
37:     this.mapContext.setMapHeight(exportRequest.getHeight());
38:     this.mapContext.setBgColor(exportRequest.getBgColor());
39:     this.mapContext.setTransparent(exportRequest.isTransparent());
41:     try {
42:       for (int i=0; i<layers.length; i++) {      
43:         final Style layerStyle = layers[i].getDefaultStyle().getStyle();        
44:         final MapLayer layer;                
45:         if(layers[i].getType().getCode() == LayerInfo.Type.VECTOR.getCode()) {
46:           FeatureSource<? extends FeatureType, ? extends Feature> featureSource;
47:           FeatureTypeInfo resource = (FeatureTypeInfo)layers[i].getResource();
48:           if(resource.getStore() == null || resource.getStore().getDataStore(null) == null) {
49:                   throw new IOException("");
50:               }
51:           Hints hints = new Hints(ResourcePool.REPROJECT, Boolean.valueOf(false));
52:           featureSource = resource.getFeatureSource(null, hints);
54:           layer = new FeatureSourceMapLayer(featureSource, layerStyle);
55:                     layer.setTitle(layers[i].getResource().getName());                    
56:                     // use default filter and version in DefaultQuery
57:                     final DefaultQuery definitionQuery = new DefaultQuery(featureSource.getSchema().getName().getLocalPart());                                      
58:                     layer.setQuery(definitionQuery);
59:                     mapContext.addLayer(layer);                    
60:         } else if(layers[i].getType().getCode() == LayerInfo.Type.RASTER.getCode()) {
61:           //
62:         }
64:         // default outputFormat to 'png' and mimeType to 'image/png'
65:         this.mapProducer = new AgsOwsMapProducer();
66:         this.mapProducer.setMapContext(this.mapContext);
68:         this.mapProducer.produceMap();
69:         this.mapProducer.writeTo(output);
70:       }
71:     } catch(Exception e) {
72:       e.printStackTrace();
73:     }
74:   }
And AgsOwsMapProducer, which takes the map context, finally renders a map using GeoTools StreamingRenderer by calling produceMap() and write out image stream by calling writeTo().
1: ...
2: public void produceMap() throws Exception {  
3:     ...
4:     Rectangle paintArea 
5:         = new Rectangle(0, 0, mapContext.getMapWidth(), mapContext.getMapHeight());
6:     String antialias = "none";
7:     IndexColorModel palette = null;
8:     ...
9:     boolean useAlpha = true;
10:     final boolean transparent = mapContext.isTransparent();
11:     ...
12:     final Color bgColor = mapContext.getBgColor();
13:     ...
14:     final RenderedImage preparedImage 
15:         = ImageUtils.createImage(paintArea.width, paintArea.height, palette, useAlpha);
16:     ...
17:     final Map<RenderingHints.Key, Object> hintsMap 
18:         = new HashMap<RenderingHints.Key, Object>();      
19:     final Graphics2D graphic 
20:         = ImageUtils.prepareTransparency(transparent, bgColor, preparedImage, hintsMap);
21:     ...
22:     graphic.setRenderingHints(hintsMap);
23:     ...
24:     StreamingRenderer renderer = new StreamingRenderer();      
25:     renderer = new StreamingRenderer();        
26:     renderer.setContext(mapContext);
28:     RenderingHints hints = new RenderingHints(hintsMap);
29:     renderer.setJava2DHints(hints);
31:     Map<Object, Object> rendererParams = new HashMap<Object, Object>();
32:     rendererParams.put("optimizedDataLoadingEnabled", new Boolean(true));    
33:     rendererParams.put("maxFiltersToSendToDatastore",  DefaultWebMapService.getMaxFilterRules());
34:     rendererParams.put(ShapefileRenderer.SCALE_COMPUTATION_METHOD_KEY, ShapefileRenderer.SCALE_OGC);    
35:     ...
36:     rendererParams.put(StreamingRenderer.ADVANCED_PROJECTION_HANDLING_KEY, true);
37:     ...
38:     renderer.setRendererHints(rendererParams);
39:     try {
40:         final ReferencedEnvelope dataArea = mapContext.getAreaOfInterest();
41:         renderer.paint(graphic, paintArea, dataArea);        
42:     } finally {
43:         graphic.dispose();
44:     }
45:     ...
46:     this.image = preparedImage;
47: }
48: ...
49: public void writeTo(OutputStream out) throws ServiceException, {
50:     ...
51:     final String format = getOutputFormat();
52:     ...
53:     new ImageWorker(image).writePNG(
54:         outStream, 
55:         "FILTERED", 
56:         100.0f,  
57:         true,
58:         image.getColorModel() instanceof IndexColorModel
59:     );
60:     ...
61: }
Note: the sample above for both produceMap() and writeTo() omitted a lot of details which can not be simply copied and paste to be compiled. But I will provide the link to download the sample code in the end of the series.

Wrap it up

So far I’ve finished everything I planned for this little “ags-ows” project. So just restart GeoServer and the customized OWS service “ags-ows” will be ready to serve out map. Here are two sample requests and result maps:


This is the final part of the whole series and the source code covered in my little "ags-ows" project is packaged as, which can be downloaded from here. Feel free to leave any feedback and comments.