Surveying, Mapping and GIS

After a great meeting this afternoon, I am looking to rehash some of my previous MetaCarta work. Looks like some of the previous kinks that I was running into with using a SOAP call to the HeadNode service originally developed on MetaCarta 3.0.1 have been worked out in newer versions (we are currently standing up two boxes with 3.7), and further, it may be a good opportunity to look at the JSON API that MetaCarta has developed.

I am also anxiously awaiting some of the document focus and other enhancements that MetaCarta has coming down the pike later this year. Other things I'd like to look at are streamlined, enhanced and more-focused document crawling.

It is my position that the W3C should extricate itself from further direct work on SOAP, WDSL, or any other WS-* specifications and redirect its resources into evangelizing and standardizing identifiers, formats, and protocols that exemplify Web architectural principles. This includes educating enterprise application architects how to design "applications" that are "native" web applications.

It appeared that there are also a few drawing a line between "World Wide" and "Enterprise"... Or suggesting that W3C should abandon aspects of their pursuits and leave it up to industry.

Certainly the geospatial community will need to sit up and take note of where this leads... Having written my own WFS and WMS services and clients from scratch, though I am still no Web Services guru, I still had to wonder about the wisdom of the approaches being used in the whole paradigm. Are the current OGC standards really in line technologically with the vision that was originally expressed by Tim Berners-Lee? I think not. Are they compatible with some of the security and other needs of the community? I think not. Not the end of the world, but certainly much iterative refinement to get us where we ultimately need to be.

Stefan Tilkov at InfoQ provides a good roundup of many of the position papers. Will we see some simplification and resolution to the ever-emergent convoluted forest of Web Services and Standards?

Though I have been playing with code for close to 30 of my 40 years, I have always considered myself far more of a hack than any kind of disciplined programmer with a formal background, as my focus has always been one of the science and engineering professional, oriented toward solving specific domain issues, such as surveying, hydrology, environmental issues, transportation, and the like - and most recently I have been thrust into Enterprise Architecture mainly due to my understanding of both the business and IT side of geobusiness - yet in terms of coding I have always managed to get some quick and often unique results in my forays into the programming world, and I am once again quite pleased tonight. One of the things I was ruminating on a few days ago is the possibility of using ASPMap as a web service, and calling the service via JavaScript and populating the view via AJAX. Better yet, extending ASPMap to serve WMS requests, but I am getting ahead of myself...

Yesterday I sat down and managed to build a web service around ASPMap (MS Visual Studio 2005 ASP.NET and .NET Framework 2.0), and as its primary functionality, it currently accepts bounding box and image size as parameters - layers and other settings will be forthcoming. Essentially the service renders the image and then dumps it out as a file - I used a milliseconds function in JavaScript to provide the unique identifier (no cookies, no viewstate or other serverside issues - all these are managed by JavaScript in the browser)

I then tried Matteo Cassati's Soap Client for JavaScript and then used the DOM to insert the image (with the known URL) into the view. I still want to better understand what I can and can't do with the return types, but that will come...

I then built some JavaScript functions to pan, zoom, et cetera, as well as a one to resize the image (with a new call) whenever the browser is resized (note the views here are full-screen, although with the banner blurred out to protect potential customer). The map view will be dynamic, trying to maximize screen real-estate. I also want to do some nice AJAX magic with collapsing/resizable map controls and query panels (the navigation/toolbar shown in the upper right is already enabled in this fashion)...

The next thought actually ties back to my original thought in a way... Having already built all the infrastructure, what would be involved in hitting an OGC WMS service and bringing in the image? Not much at all, just a little magic trying to navigate how JavaScript deals with ampersands in constructing strings. An hour and a half tonight, and my WMS client is also done - you see here a classic image from NASA JPL (MODIS Mosaic) - there is still much more tweaking and functionality to be developed, but for a mere weekend's work for a hack like me, I am quite pleased with the results.

Considering I managed to fit all of this in during a hectic weekend, in the midst of preparing for a move (with painting and carpeting the new house, boxing things up at the old house, et cetera) and driving 4 hours to Washington, DC tonight for meetings tomorrow, I am frankly amazed that I managed to get this done so quickly.

I am very much looking forward to what the next few months will bring. My intent is to build some very robust and reusable infrastructure. What I have started here will become universal throughout future applications, and the beauty of it is that I will be able to snap in whatever mapping infrastructure is available, whether ArcIMS or ArcGIS Server, ASPMap, WMS Servers, or anything else I can either build a Web Service wrapper around or convert to WMS outright.

The scrappad blog posts about a matrix of ArcWeb APIs that was recently published by ESRI - unfortunately the original link posted on scrappad is broken, but I was able to locate the document here:

http://support.esri.com/index.cfm?fa=knowledgebase.whitepapers.viewPaper&PID=20&MetaID=1183

The table lists functionalities, and if they are currently supported by SOAP, REST, OpenLS, J2ME, and JavaScript APIs. It is interesting to note that virtually everything is supported by SOAP, followed by JavaScript, with REST, OpenLS and J2ME bringing up the rear.

As things move forward, it will be interesting to see how these geospatial web services play with regard to BPEL, orchestration and other emerging SOA efforts.

I have several interesting things cooking in the next few weeks –

One we just had a kickoff for - Agency Best Practices, which follows much of what I have already been doing in Geospatial Enterprise Architecture, but which is all the more serendipitous in light of the Geospatial Line of Business effort by OMB. I have already been working on a concept of operations for transitioning to and implementation of reusable, distributed service components for geospatial business (SOA and other enterprise-level solutions). My previous effort will be expanded and merged with efforts by other Domain Experts in other arenas, such as Portal/SSO, Service-Oriented Architecture, ETL, and other technologies. I’m putting a draft together for May 15th.

The other is to perform another MetaCarta integration, this one to geo-enable a repository containing 50 million documents, for MetaCarta-powered geocoding and mapping. I will likely follow a similar model as the previous integration project, using ArcIMS and existing enterprise mapping as the base with a query interface, perform the SOAP call to the appliance, parse the response and plot them on an acetate layer. This one will be a new J2EE implementation from scratch, as opposed to the previous one, which was an integration into a legacy ASP/IIS-based application.

For the past couple of months, one of the projects I have been working on is integrating MetaCarta into Window to My Environment (WME)- The deep integration is achieved through the use of SOAP calls from within the WME application to the search appliance. Writing the code, parsing the results, and plotting them in ArcIMS was fairly trivial (WME is a legacy ASP / ArcIMS ActiveX Connector implementation).


On the other hand, the task that has been far more daunting is the enterprise security aspect. EPA, like many agencies and enterprises, has guidelines and testing parameters for applications. In the instance of appliance-based technology and this era where hardware and software begin to blur, such as routers with embedded linux, the standard practice of the appliance industry is to build a box that has its own OS, typically Linux-based, that has then been stripped of unnecessary functionality wherever possible, locked down for security, and tweaked to optimize device performance, wherever possible. In the process, they typically go through their own security audits as well, and get various security certifications from their target markets.

With Metacarta, we ended up going through several months of tests and probes by EPA IT security staff- finally the unit emerged, passing with flying colors. We got the unit back in our development environment and finished developing the application in a few weeks. Now, we're again waiting more weeks for the unit to be stood up again at EPA so that the app can be deployed.

It seems to me that this process is overly cumbersome. Perhaps part of the issue, not to blame Linux, is that Linux is so customizable, to the point of being a very different OS from one implementation to the next. The other part that leads to things being cumbersome is that it seems every agency and organization has differing security criteria and concerns. The appliance in question has already been used in DoD and intel applications, which one would think would have extremely stringent security requirements, as opposed to a civilian agency. I would think there should be a certain core set of criteria which go across the board between all customers- this type of common certification would certainly go far, as would acceptance of other agencies' certifications.

Just my humble $0.02...

We have recently been working with MetaCarta- they provide search engine technology with a twist- as documents are ingested and indexed, the MetaCarta ingest engine also contextually extracts any geographic references and geocodes them- and then stores the geocoded locations with the other indexed data.

What this does is allow the search engine to do a number of additional things that ordinary search engines cannot: for instance restrict its search results to a bounding box or map window, and retrieve only the documents pertaining to that geographic extent- or you could plot the location of all the matching documents on a map as pushpins…

A nice feature of MetaCarta is that they have had the foresight to provide a Web Services interface to their appliance- this allows deep integration into existing applications by using SOAP calls to perform the query- it then is a simple matter to parse the XML response and retrieve the returned document titles, URLs, lat/longs, abstracts, or other parameters.

MetaCarta – http://www.metacarta.com/