Thursday, January 7, 2010

Extend GeoServer with customized OWS service :: part 1

Recently I have been playing with GeoServer by following the online developer manual. Everything went smoothly until I found the OWS service section of programming guide still remains empty. Just out of curiosity and the believe that the best way to learn GeoServer is to extend it with either a new or a similar existing functionality, I decided to create a fake OWS service end point (I call it 'ags-ows' service, which I will explain later) in GeoServer.

This "ags-ows" service implements a minimum set of request parameters from ArcGIS Server RESTAPI for Map Service. To make it have more practical sense and also simple enough to fit in a sample a lot of optional parameters and details are omitted. The only function of "ags-ows" service is simply responding map images to an export request (in syntax of ArcGIS REST API) as below:

http://localhost:8080/geoserver/ows/agsows?request=export&bbox=-115.8,30.4,-85.5,50.5&layers=show:0,1,2&size=800,600&transparent=true
Or
http://localhost:8080/geoserver/ows?service=agsows&request=export&bbox=-115.8,30.4,-85.5,50.5&layers=show:0,1,2&size=800,600&transparent=true

To simplify the problem further, only following request parameters will be recognized and accepted:
“bbox”, “layers”, “size”, “imageSR”, and “transparent”
And default values will be hard coded for optional request parameters listed below:
“format” as “image/png”
“bboxSR” as “4326”
“dpi” as 96
A sample response image from "ags-ows" service looks like:

Extending GeoServer with such an "ags-ows" service involves following steps:
  1. Checkout and build GeoServer existing source code from trunk;
  2. Create Eclipse project for "ags-ows" service and integrate it into existing GeoServer source code tree and build system;
  3. Create "ags-ows" service's application context (applicationContext.xml);
  4. Reuse org.geoserver.ows.Dispatcher to dispatch the request to "ags-ows" service;
  5. Extend org.geoserver.ows.KvpParser to parse request parameter;
  6. Extend org.geoserver.ows.KvpRequestReader to deal with "ags-ows" service requests;
  7. Extend org.geoserver.ows.Response to handle "ags-ows" service responses.
Now begin with this post and a few more following up, details will be explained for each step above.

Checkout and build GeoServer existing source code from trunk

All my works are based on GeoServer source code from trunk, but if you choose to work on a GeoServer code branch equal or higher than 2.0.0, it's also fine. As for building GeoServer from source code and run/debug it in built-in jetty web server, there are already more than enough materials covering the topic over the Internet so it won't be repeated here again. I myself though followed the instructions from GeoServer website's developer manual which works perfect.
Before jump into the next step, make sure you've successfully built the GeoServer source code from trunk, and also check if it can run in debug mode with jetty (run org.geoserver.web.Start class in web-app project as Java Application) web server so that you have a valid develop and debug platform to start with. The best way to validate it is to set a break point in GeoServer source (e.g. in org.geoserver.ows.Dispatcher) and send an OGC request (e.g. http://localhost:8080/geoserver/ows?service=WMS&request=GetCapabilities) to see if it stops at the break point as expected. If it is not working you'd better fix it before continuing.
Now if you think you've set up a decent develop/debug environment from GeoServer source code, you can move on.

Create Eclipse project and integrate into existing GeoServer source code tree and build system

To avoid modifying anything in existing GeoServer code base if possible (either GeoServer core modules or community/extension modules), I decided to create a new "groupId" called "playground" where my project for "ags-ows" service will go under it. "playground" is just a similar structure to the "extension" and "community" concept in current GeoServer source code tree.
  • Under \trunk\src\, create a folder called "playground".
  • Add a pom.xml (maven project file) in "playground" folder for GeoServer maven build process; (the pom.xml should look similar to the one under "extension" folder) Here is the pom.xml I use:
  • 1: <?xml version="1.0" encoding="ISO-8859-1"?>
    
    2: <project xmlns="http://maven.apache.org/POM/4.0.0" 
    
    3:     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    
    4:     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
    
    5:                         http://maven.apache.org/maven-v4_0_0.xsd">
    
    6:   <modelVersion>4.0.0</modelVersion>
    
    7:   <parent>
    
    8:     <groupId>org.geoserver</groupId>
    
    9:     <artifactId>geoserver</artifactId>
    
    10:     <version>2.1-SNAPSHOT</version>
    
    11:   </parent>
    
    12:   <groupId>org.geoserver</groupId>
    
    13:   <artifactId>playground</artifactId>
    
    14:   <packaging>pom</packaging>
    
    15:   <name>GeoServer Playground Extensions</name>
    
    16:   <dependencies>
    
    17:     <dependency>
    
    18:       <groupId>org.geoserver</groupId>
    
    19:       <artifactId>platform</artifactId>
    
    20:     </dependency>
    
    21:   </dependencies>
    
    22:   <profiles>  
    
    23:     <profile>
    
    24:       <id>ags-ows</id>
    
    25:       <modules>
    
    26:         <module>ags-ows</module>
    
    27:       </modules>
    
    28:     </profile>    
    
    29:     <profile>
    
    30:       <id>allPlaygroundExtensions</id>
    
    31:       <modules>        
    
    32:         <module>ags-ows</module>    
    
    33:       </modules>
    
    34:     </profile>
    
    35:     <!--  
    
    36:     <profile>
    
    37:       <id>release</id>
    
    38:       <modules>
    
    39:         <module>ags-ows</module>    
    
    40:       </modules>
    
    41:     </profile>
    
    42:     -->
    
    43:   </profiles>
    
    44: </project>
  • Every project under "playground" should be listed as a "profile" so that when building GeoServer with maven those projects can be referenced. By default "mvn clean install" doesn't include any extension or community projects as well as those in "playground" folder. So it's very useful to add another profile just for maven to include all "playground" modules in building process, which I call "allPlaygroundExtensions" (similar to profile "allExtensions" in extension module). Optionally you can create another profile called "release". 
  • Modify the pom.xml under \trunk\src of GeoServer source code to includes modules defined under "playground" folder (that's probably the only thing I need to touch on existing GeoServer code). Open pom.xml under \trunk\src and locate following section, and add "<module>playgound</module>" to include all "playground" projects defined in the pom.xml under playground folder.
  • 1:  <modules>
    
    2:   <module>maven</module>
    
    3:   <module>platform</module>
    
    4:   <module>main</module>
    
    5:   <module>wcs</module>
    
    6:   <module>wcs1_0</module>
    
    7:   <module>wcs1_1</module>
    
    8:   <module>wfs</module>
    
    9:   <module>wms</module>
    
    10:   <module>ows</module>
    
    11:   <module>gwc</module>
    
    12:   <module>rest</module>
    
    13:   <module>web</module>
    
    14:   <module>community</module>
    
    15:   <module>extension</module>
    
    16:   <module>playground</module>
    
    17:  </modules>
  • Under "playground" folder, create a folder structure for "ags-ows" service project like below (since "ags-ows" is the name I gave for "ags-ows" service project in playgroud group's pom.xml, I will name the root folder after it):
            ags-ows


    src
                    main 
                        java 
                    test 
                        java 
                        resources 
                test



  • Add a pom.xml in ags-ows folder like below: (be careful on the "version" to reflect your specific environment, as well as highlighted places; also list all your dependencies)
  • 1: <?xml version="1.0" encoding="UTF-8"?>
    
    2: <project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    
    3:     xmlns="http://maven.apache.org/POM/4.0.0" 
    
    4:     xsi:schemalocation="http://maven.apache.org/POM/4.0.0 
    
    5:                         http://maven.apache.org/maven-v4_0_0.xsd">
    
    6:   <modelversion>4.0.0</modelversion>
    
    7:   <parent>
    
    8:     <groupid>org.geoserver</groupid>
    
    9:     <artifactid>playground</artifactid>
    
    10:     <version>2.1-SNAPSHOT</version>
    
    11:   </parent>
    
    12:   <groupid>org.geoserver.playground</groupid>
    
    13:   <artifactid>ags-ows</artifactid>
    
    14:   <packaging>jar</packaging>
    
    15:   <name>ArcGIS Server OWS Service Module</name>
    
    16:   <dependencies>
    
    17:     <dependency>
    
    18:       <groupid>org.geoserver</groupid>
    
    19:       <artifactid>main</artifactid>
    
    20:     </dependency>
    
    21:     <dependency>
    
    22:       <groupid>org.geoserver</groupid>
    
    23:       <artifactid>ows</artifactid>
    
    24:     </dependency>
    
    25:     <dependency>
    
    26:       <groupid>org.geoserver</groupid>
    
    27:       <artifactid>wms</artifactid>
    
    28:     </dependency>
    
    29:     <dependency>
    
    30:       <groupid>org.geoserver</groupid>
    
    31:       <artifactid>wfs</artifactid>
    
    32:     </dependency>
    
    33:     <dependency>
    
    34:       <groupid>org.geotools</groupid>
    
    35:       <artifactid>gt-render</artifactid>
    
    36:     </dependency>
    
    37:     <dependency>
    
    38:       <groupid>org.geotools</groupid>
    
    39:       <artifactid>gt-shapefile-renderer</artifactid>
    
    40:     </dependency>
    
    41:   </dependencies>
    
    42: </project>
  • Rebuild GeoServer with "-P allPlaygroundExtensions" option; (go to \trunk\src and execute "mvn clean install -P allPlaygroundExtensions" to build GeoServer from source with all projects in playground group)
  • Create Eclipse project file after a successful build, still under \trunk\src execute "mvn eclipse:eclipse -P allPlaygroundExtensions" to generate eclipse project files. Don't forget -P allPlaygroundExtensions otherwise projects in playground won't be imported into eclipse.
  • Import all projects under \trunk\src\ into Eclipse;

    Other than projects for normal GeoServer modules (e.g. web-app, wms, wfs, etc), you should have one more project (in this case, it should be called "ags-ows"). Check the "ags-ows" project to make sure all 3rd party library jar files are there (if accessible through remote Maven repository, otherwise you will get build failure until manually copied). If you list any other GeoServer modules (e.g. "main"), they should be added and listed under the "Projects" tab of "Java Build Path" of the project.
  • To use "ags-ows" service in GeoServer, you can either do:
    • 1. Drop built jar (e.g. xxx-snapshot-2.1.jar) into GeoServer's exploded war's WEB-INF/lib folder and restart GeoServer;
    • 2. Include "ags-ows" project in GeoServer's web-app project's java build path, and restart GeoServer.
Up to this point if you follow the steps above without any problem, you should already have a functional GeoServer "ags-ows" service extension, it just doesn't do anything but provide you a develop and debug environment within GeoServer codebase. Anyway it's a good start point and I will provide more details on the concrete functionality soon in next blog entry.

Wednesday, January 6, 2010

The very first experience of uploading OSM data with Potlatch and JOSM

This is the first entry of 2010, and I already have a confession to make. I've been bluffing about Open Street Map to people as if I am an expert or enthusiastic, but I never made any update to OSM ever myself. What a shame! But anyway let me make up for it in the first week of new year.
The data I've collected and  will upload to OSM is an unclassified parking driveway in ESRI Inc. Redlands campus, and the whole process involves using Android based application "Every Ttrail", and two most popular OSM edtiors JOSM (Java based desktop application) and Potlatch (the one comes with official OSM website).
1. Collect raw GPS data using Every Trail on Android G1
I'm using my Android GI phone to collect raw GPS data, and it's much fun and can not be easier. So just start Every Trail on your phone, touch "Reccord" and then walk or drive as you want. After you've finished the whole trip, simply click "Stop" and the GPS data that has been collected while you are moving is saved in your phone. The only inconvenient thing about Every Trail is that you can not directly export the GPS data to my computer (by looking through Android SD card, it seems the data is not saved in GPX format on disk). But you can easily upload the trails onto Every Trail's website (it requires an account, so just register one), and then from there you can easily export/save them in either KML or GPX format to your desktop machine. In this case I saved it as GPX (.gpx files on disk) which then can be piped to OSM.
2. Upload GPS data (.gpx file) to OSM website as GPS traces
Basically you need to upload your GPS data to OSM website as GPS traces before you can upload data into main OSM database. Both OSM wiki page and FAQ page has a very comprehensive explanation on how to upload GPS data as traces onto OSM, so I am not repeating it here.
http://wiki.openstreetmap.org/wiki/Editing
http://wiki.openstreetmap.org/index.php/FAQ
You should receive an email confirmation about the successful upload.
3. Upload GPS traces data to OSM database using Potlatch
Once raw GPS data is uploaded in OSM website as GPS traces, you are ready to make changes to OSM database. Simply find the particular GPS trace you are interested in and click "edit" (not the Edit tab in OSM website main menu, but the little on the right side of each GPS trace) like below:

and when dialog with "Edit with Save" and "Edit Live" pops up, make sure you check "Convert GPS track to ways". After your GPS trace shows up in Potlatch user interface, make necessary changes and finally click "save" to upload data. A successful save meaning that the data is already uploaded in OSM database. But it may take some time for the map tiles to be updated to include the change.

Update or upload GPS data in JOSM
Alternative to the web based editor Potlatch, I also used desktop based JOSM editor to do the same thing. In JOSM you basically follow:
(a) Download OSM data for a specific small area (if the area is too big, it won't allow you to download the data for local editing);

(b) Overlay your own GPS data and then modify either OSM baselayer data or your overlay;

(c) upload or update your changes back to OSM database. Again for the map tiles to be updated, it takes some time.

4. Very first experience: Potlatch vs. JOSM
Potlatch and JOSM are basically functional equal, but carry their own characteristics inherited from the IT technology they based on. Potlatch doesn't require you to install anything (probably flash plugin, those who doesn't have that must really be out of date), and it's coming together along with OSM website which provides a more integrated work flow for people to work with their GPS data etc. But just like most of other web based application, you have to use Potlatch online, and GUI is not that fancy and responsive. JOSM on the other hand has richer GUI and friendly UI experience, and it also provides you a lot more tools to edit or correct your GPS data. In JOSM you work in a checkout, editing and checkin work flow which doesn't seem to bother me at all. But the biggest disadvantage for JOSM is that it lacks of a tiled image based background layer (Google Maps, Bing Maps or OSM map itself), which makes it impossible for me to adjust some of my GPS data. I am quite surprised not being able to find this feature in JOSM, other than that I would even prefer JOSM over Potlatch.
In the following blog, I will collect more GPS data for ESRI Redlands campus and upload it onto OSM database.