Surveying, Mapping and GIS

One of those key philosophical notions that circulates among many stewardship and advocacy groups is to think globally, act locally - The same notion can be applied in best practices in solutions architecture, requirements development, and coding principles, where it applies to so many things - in terms of true enterprise architecture and integration approaches, in terms of data stewardship, and for public sector agencies, in terms of transparency and open government.

Each and every day, folks craft queries for accessing complex data, they write business logic for analysis and so on - and more often than not, they then wrap it up with a nice glossy user interface.  MVC and many other best practices applied, check...  But here is where another key opportunity to really open things up is lost.

Let's take the example of state or federal agencies - many of them are doing great things, in terms of publishing datasets (Open Government Intitiative, Data.gov, Geospatial One-Stop, State Cartographers' Offices and Clearinghouses, and so on).  They are doing great things in terms of reporting tools, web queries and lookups, and mapping and visualization tools.  But...  As wonderful as all of these efforts are, imagine how powerful it would be if these could be linked and integrated, from agency to agency?  Imagine, for example, applications pulling in data from NOAA, NASA, EPA and other agencies?  Imagine state environmental agencies pulling in information from federal environmental agencies - the power of open government and transparency begins to multiply and become more and more powerful by orders of magnitude.

So what are we talking about?  Web services and APIs.  Not even anything complex, it could be as simple as REST and JSON, as in the space of public-facing applications, for a vast majority of data-centric or analytical applications, we are generally talking about read-only, non-transactional, nonsensitive data.  Why web services or APIs?   Isn't something like publishing datasets to Data.gov enough?  Publishing datasets to someplace like Data.gov is fine - but consider this - if someone downloads your data and integrates it into their own standalone application, what control do you then have over it?  Precious little.  When you refresh your database afterward and post a new dataset, is there any assurance that they will come back and get a fresh copy and keep up to date, or do you begin to incur a liability of your data being out of sync as it is represented in someone else's public-facing application?  Absolutely.

If, on the other hand, you post an easy-to-integrate RESTful service or API, you make it far easier to have your data current.

Criticisms?  Yes, there are many, but we can look at those as well - some might push back and say, "is there sufficient bandwidth and infrastructure for hosting those web services?"  The mere fact of web services in and of themselves generally incur no more new bandwidth or infrastructure demand than the public-facing web applications that you are already building.  So if that's really a concern, then maybe you shouldn't be building ANY applications.  The argument is generally a red herring.

Security concerns?  Same deal - you are already exposing the data and business logic via your web applications - if you think the web service poses new security concerns, then perhaps you haven't adequately thought through your EXISTING security.  So another red herring.

"They will come and take all our data!" - Again, why would they, if they know they can grab it dynamically and that you will be assured of their reliance on them by continually posting data refreshes?

What about poorly-behaved queries against your web service that might bring your database to its' knees?  Same thing - those are things that really should be handled in the logic for your existing application.  Better yet, think ahead a bit more and provide options - query filters, such as by a data attribute or domain, bounding boxes for geodata, and so on.

So - in conclusion, this type of thinking can up developers' approaches to how they deal with security, queries and filters, and so on, and generally tighten things up - but in the meantime, also provide a tremendous opportunity for data sharing and integration, whether within an agency, company or office, fostering collaboration across departments and programs, or outside of one's own organization, fostering collaboration across agencies and levels of government, across industry, academia and so on.

An individual organization can try all they like to anticipate all of the public's thoughts, needs, interests, they can try to walk in their stakeholders' footsteps - but ultimately, one can never fully anticipate all possibilities, and there are a lot of bright people out there with many untapped ideas - so why not open it up and facilitate that larger collective brainstorm of ideas by providing services ready to integrate?  Let's take things like the Open Government Initiative and other good things ongoing in government and elsewhere to the next level - dynamic, services and resources oriented architecture...

I got to spend a little time in the last week or two examining integration of web mapping with BusinessObjects XI R2 Web Intelligence, for a little turbo proof-of-concept application - I hadn't seen many folks talking about integrating web mapping and BO, so I figured I would share my own experiences. Though I'm certainly no BusinessObjects guru, I was able to successfully do some basic integration, with the help of the BusinessObjects "Masher" application available on their "Labs" site: http://labs.businessobjects.com/mashup/default.asp.

What BusinessObjects presents via the Masher is a dynamically-named div tag, which is accessible via container in JavaScript, which is where your rendered DHTML content needs to be targeted. They also then present three basic functions for housing and executing all user-developed JavaScript, init(), mash() and dispose() - the intent of which is self-evident. They also provide some JavaScript objects, pivot and parameters - pivot contains the BO report table contents, and parameters contains values that can be preset and/or passed when initializing or formatting the template.

So, essentially just a matter of figuring out what's available to latch on to and use - and how to leverage what BO provides to permit building the application. For this proof-of-concept, there had been some discussion and interest in Google Maps from the customer, so - for the sake of simplicity, I chose to just stick with Google Maps. In reality, I'd be fairly confident in suggesting that virtually any web mapping framework that supports DHTML and JavaScript should be usable with this same approach, e.g. OpenLayers, Microsoft Virtual Earth, or others.

So, ultimately - the results:

"Turn table into..." (the intent being as simple as "click a button to turn this table or graph into a map")

Being a table, the application has to be able to understand and use designations in the underlying table, from it's "geography" dimension such as "AZ" to latch on to the appropriate mapping geographies. So... Turn map into...


Here, I have my newly published template showing up among the choices as "Google Maps Equal Interval Polygonal Choropleth Map", and click on "OK" - Et voila!


For this quick-and-dirty proof of concept, I just wrote some JavaScript that examines the array of pivot data and the range of values, and then just does some quick calculations to create a lookup table and legend. One trick was to dynamically rewrite BO's generic container div tag to embed two new div elements inside of it, one which I use to hold the map, and a second, which I use to hold my legend - other elements as well as a hidden control panel could be embedded here also.

For generating the classification and legend, a similar type of logic and approach could of course also be applied to a variety of classification schemes such as standard deviations, natural breaks, and so on. I then examine the pivot table to see what geographies are represented, display them on the map rendered with their appropriate hex color values matching their respective measures from the lookup table and add a Google Maps openWindowHtml popup for the click event handler for each location. Within the pivot object, labels are also provided - and given a table with multiple columns of measures, these could either be managed with code to run analysis on both, just thematically map the first, and display the remainder in the popup, and so on - a variety of possibilities for the adventurous coder.

Want to change the number of classification breaks? Didn't like the color scheme?

Within the template, there is opportunity to define parameters - here I set it up to allow me to change the number of classification breaks, starting color, ending color, and so on. And the BO folks nicely even include a color picker. So, for example, I can then retrieve parameters.startColor as a hex value, split it into RGB values and compute even jumps for my color ramp to parameters.endColor, based on the parameters.numBreaks value that the user specifies.

Like so... (deliberately odd colors selected to show the difference...)


Additionally, mash() fires every time the underlying table's filters or results are changed, which is nice, as the map updates accordingly.

Some additional development insights - I decided to keep a lot of my functions in an external .js library - one issue I encountered is that, given everything was encapsulated in the three BO Masher functions init(), mash() and dispose(), whenever I needed to create and use global variables, I needed to do so explicitly, e.g. window.myVarName. Additionally, I noted that BusinessObjects XI R2 uses the dojo JavaScript library... so for fans of dojo, that also provides a great deal of functionality and capability to leverage. As with anything, potential collision with other pieces of code or cross-browser compatibility issues are also a question, but through judicious naming conventions and error handling, these can generally be addressed.

So, while perhaps my turbo solution may not the most elegant, it's nonetheless illustrative that web mapping integration into a BO platform is definitely doable and definitely functional, in this case developed in just a few days' time with a little quick hackery, even for someone who may have web mapping expertise but little prior integration or development experience with a platform like BusinessObjects (such as myself in this instance) - hopefully this will be an encouragement to others who are looking to merge BI tools with geospatial technology, and likewise, hopefully the BI folks will continue to increasingly see the value of geospatial visualization in addition to their charts, graphs and dashboards. I have already seen some very interesting visual geographic trends popping out of the data, which is definitely not anywhere near as evident in some of the more standard BI views. Other possibilities include drilldown, however I found that the masher does not handle hyperlinks embedded in tables well - but within BO, URL-driven access to reports to allow drilldown is definitely a possibility as well.

What the future of the BO Masher is in the future is uncertain, but the potential use case and benefit of such integration is substantial.

In the long run, there will definitely be an ever-emergent intersect between these technologies and tools. All in all, another fun little proof-of-concept. Next, I may want to take on BusinessObjects Xcelsius 2008 and Flex-based integration with web mapping frameworks - will remain to be seen, depending on how various other projects go over the next few months...

I don't always post about some of the things that we are working on, and perhaps I should do so more often.

I had some mixed feelings about recently seeing the Google Maps mashup that TerraIMS recently put up, showing EPA Superfund sites in proximity to a given address.
http://www.terraims.com/webservices/superfund.php



EPA has, for many years (in some instances, dating back to 1999), already had a number of community-oriented web mapping applications which deliver a great deal of information on EPA regulated sites, EPA cleanup activities, and so on - For example, one flagship of USEPA Office of Environmental Information is the venerable Window to My Environment - which, among other things, provides a great deal of information on EPA regulated facilities, watersheds, local and state resources, and the like at a community level. Window to My Environment currently gets well in excess of 50,000 hits per day.

Another excellent application which provides a lot of detailed information on environmental cleanup activity at a local level is the USEPA's Office of Solid Waste and Emergency Response (OSWER) Cleanups In My Community application.

These are just some of the web mapping projects that I have been involved in lately - there are actually several more EPA EnviroMapper applications available at the EnviroMapper StoreFront - and we are currently in the process of overhauling the core infrastructure of these, to migrate from legacy ASP/VBScript platforms to reusable component-based design, web services, and the ability to host some of these applications in an Oracle Portal environment, and integrating MetaCarta searches, among other things. We also have been doing some preliminary explorations in making EPA data available as KML for Google Earth, presenting some of our own Google Maps mashups (similar to TerraIMS), all of which are working wonderfully thus far. Unfortunately some of these enhancements are still in prototype, and others are only available on the EPA intranet.

So what of TerraIMS? Great work they did - but my concern isn't constrained, nor in any way specific to this particular mashup. The concern I have is with the description of the effort:

The U.S. Environmental Protection Agency recently released its National Priority list of Superfund Sites in XML format. We converted the XML data and integrated it with a database and then mashed things up a bit. Users enter an address and it is geocoded on the fly, enabling a distance query to be processed against the EPA data in the database. This mashup allows anyone to quickly and easily find the nearest Superfund Sites to their home address or a location of interest.

From this description, it sounds like they essentially have performed an extract of the EPA data, have massaged and processed it, and then do their mashup. The business case isn't tremendously compelling, as these maps and data are already long available, so it appears to be a mashup just for the sake of doing a mashup.

Fortunately Superfund data isn't tremendously volatile, however it nonetheless raises the question of other potential applications, of currency, completeness and accuracy of data utilized in mashups. Herein lies potential liability. If one uses a third-party mashup as a decision support tool, they would need to know to track back to the original source to ensure that the data they are viewing in the mashup is current, complete, and accurate...

After a great meeting this afternoon, I am looking to rehash some of my previous MetaCarta work. Looks like some of the previous kinks that I was running into with using a SOAP call to the HeadNode service originally developed on MetaCarta 3.0.1 have been worked out in newer versions (we are currently standing up two boxes with 3.7), and further, it may be a good opportunity to look at the JSON API that MetaCarta has developed.

I am also anxiously awaiting some of the document focus and other enhancements that MetaCarta has coming down the pike later this year. Other things I'd like to look at are streamlined, enhanced and more-focused document crawling.

It is my position that the W3C should extricate itself from further direct work on SOAP, WDSL, or any other WS-* specifications and redirect its resources into evangelizing and standardizing identifiers, formats, and protocols that exemplify Web architectural principles. This includes educating enterprise application architects how to design "applications" that are "native" web applications.

It appeared that there are also a few drawing a line between "World Wide" and "Enterprise"... Or suggesting that W3C should abandon aspects of their pursuits and leave it up to industry.

Certainly the geospatial community will need to sit up and take note of where this leads... Having written my own WFS and WMS services and clients from scratch, though I am still no Web Services guru, I still had to wonder about the wisdom of the approaches being used in the whole paradigm. Are the current OGC standards really in line technologically with the vision that was originally expressed by Tim Berners-Lee? I think not. Are they compatible with some of the security and other needs of the community? I think not. Not the end of the world, but certainly much iterative refinement to get us where we ultimately need to be.

Stefan Tilkov at InfoQ provides a good roundup of many of the position papers. Will we see some simplification and resolution to the ever-emergent convoluted forest of Web Services and Standards?