MetaCarta is holding a users group meeting next week, October 8th - from their site:

It's been a very busy few weeks, so I haven't had much chance to post... things going on with Virtual Earth, ArcIMS Route Server and GDT/TeleAtlas, survey-grade GPS data collection, emergency response, logistics, modeling and simulation, and plenty of other things flying around.

It's all fun stuff, but I always enjoy rolling the sleeves up and getting dirty... doing the AJAX thing, mashing various web services with Virtual Earth's V5 API via pure JavaScript clients as various proof-of-concept applications.

Here's one quick app: Virtual Earth and the NASA MODIS WMS server:

Here I'm showing the VE and MODIS side-by-side, both views refresh dynamically. The next step will be to explore the "Roll Your Own Tile Server" approach to seamlessly getting custom WMS content directly into VE.

The next one was even more fun: Virtual Earth and the MetaCarta JSON API:All pure Web 2.0, neogeo, slippy AJAXIAN goodness, "Look, Ma! No "SUBMIT" button!!" Type in your query, (searching for documents about toxic substances here...) and it fetches results from the MetaCarta appliance and sprays them back into the VE map view. Pan, zoom, and instantly you get new stuff popping up. All self-contained in a few k of DHTML, CSS and JavaScript, no Java, .NET, Ruby, Python or other infrastructure needed. Both written in a total of about 2.5 hours, just noodling around in the APIs without any real thought ahead of time.

I just discovered a great Linux distribution called HostGIS - it comes ready to run MapServer and a whole suite of other goodies, right out of the box.

I went to the HostGIS site, and within about 20 minutes, had downloaded it, loaded it up on MS VirtualPC, and had it up and running:





It comes complete with OpenLayers and TileCache from our friends at MetaCarta, MySQL, PostgreSQL, and a few other add-ons and goodies that I am not as familiar with, such as Map-Fu.

While I am usually spoiled with plenty of ESRI tools, this certainly provides a refreshing alternative for projects without an ESRI budget...

I don't always post about some of the things that we are working on, and perhaps I should do so more often.

I had some mixed feelings about recently seeing the Google Maps mashup that TerraIMS recently put up, showing EPA Superfund sites in proximity to a given address.
http://www.terraims.com/webservices/superfund.php



EPA has, for many years (in some instances, dating back to 1999), already had a number of community-oriented web mapping applications which deliver a great deal of information on EPA regulated sites, EPA cleanup activities, and so on - For example, one flagship of USEPA Office of Environmental Information is the venerable Window to My Environment - which, among other things, provides a great deal of information on EPA regulated facilities, watersheds, local and state resources, and the like at a community level. Window to My Environment currently gets well in excess of 50,000 hits per day.

Another excellent application which provides a lot of detailed information on environmental cleanup activity at a local level is the USEPA's Office of Solid Waste and Emergency Response (OSWER) Cleanups In My Community application.

These are just some of the web mapping projects that I have been involved in lately - there are actually several more EPA EnviroMapper applications available at the EnviroMapper StoreFront - and we are currently in the process of overhauling the core infrastructure of these, to migrate from legacy ASP/VBScript platforms to reusable component-based design, web services, and the ability to host some of these applications in an Oracle Portal environment, and integrating MetaCarta searches, among other things. We also have been doing some preliminary explorations in making EPA data available as KML for Google Earth, presenting some of our own Google Maps mashups (similar to TerraIMS), all of which are working wonderfully thus far. Unfortunately some of these enhancements are still in prototype, and others are only available on the EPA intranet.

So what of TerraIMS? Great work they did - but my concern isn't constrained, nor in any way specific to this particular mashup. The concern I have is with the description of the effort:

The U.S. Environmental Protection Agency recently released its National Priority list of Superfund Sites in XML format. We converted the XML data and integrated it with a database and then mashed things up a bit. Users enter an address and it is geocoded on the fly, enabling a distance query to be processed against the EPA data in the database. This mashup allows anyone to quickly and easily find the nearest Superfund Sites to their home address or a location of interest.

From this description, it sounds like they essentially have performed an extract of the EPA data, have massaged and processed it, and then do their mashup. The business case isn't tremendously compelling, as these maps and data are already long available, so it appears to be a mashup just for the sake of doing a mashup.

Fortunately Superfund data isn't tremendously volatile, however it nonetheless raises the question of other potential applications, of currency, completeness and accuracy of data utilized in mashups. Herein lies potential liability. If one uses a third-party mashup as a decision support tool, they would need to know to track back to the original source to ensure that the data they are viewing in the mashup is current, complete, and accurate...

After a great meeting this afternoon, I am looking to rehash some of my previous MetaCarta work. Looks like some of the previous kinks that I was running into with using a SOAP call to the HeadNode service originally developed on MetaCarta 3.0.1 have been worked out in newer versions (we are currently standing up two boxes with 3.7), and further, it may be a good opportunity to look at the JSON API that MetaCarta has developed.

I am also anxiously awaiting some of the document focus and other enhancements that MetaCarta has coming down the pike later this year. Other things I'd like to look at are streamlined, enhanced and more-focused document crawling.

Gratefully, things are finally starting to sort themselves out on the move...

Currently I am gearing up to head south to the Savannah International Trade and Convention Center in Georgia, to attend the EPA Environmental Information Symposium from December 5th to December 7th. This afternoon, I just finished putting together a PowerPoint for a brief talk on ingestion, analysis and mapping of unstructured data during one of the sessions.

We are also hoping to put on a live demo - using MetaCarta web services integrated with ArcIMS and ASP-based Window to My Environment - we are also currently in the midst of a redesign for the infrastructure of Window to My Environment and EnviroMapper applications, to look at good stuff like integrating the application into the Oracle Portal environment, ESRI Geospatial Portal Toolkit, Service-Oriented Architecture, and Reusable Components.

Along the MetaCarta lines, there are a few other interesting tools emerging, with potential for geotagging - Aerotext, which I discussed previously, and SRA's NetOwl, developed for the CIA - which I may get an opportunity to see next week.

I had a great time in Las Vegas last time, so I am looking forward to this event as well...

Still unwinding from the ESRI User Conference - I had meant to post during the conference but there just wasn't enough time, from the daily activities and parties after.

Now, I still have a few hours to kill in San Diego before my flight, and am finally taking it easy. The Conference was a lot of fun - Though being in that mode of hundreds of intense 5-minute conversations over the course of the day can tend to put one in an odd state of mind - by the end of that day, you can scarcely remember who you talked to, or about what... And too much going on, at a frenetic pace. I had to step out of quite a few sessions to support a few ongoing things with our federal customers, so it was definitely a working vacation...

At any rate, a lot of old friends, colleagues and adversaries at USDA, EPA, other agencies, as well as CSC, Perot Systems, Lockheed, Jim Knudson, my MetaCarta buddies, Chris Cappelli from ESRI and others from PA and elsewhere, as well as many excellent and interesting new contacts made, and as always a lot of excellent intel and takeaways - I did get a brief chance to talk to keynote speaker, Senator Bob Kerrey, now of the New School such as Kim Ollivier, who is the father of the New Zealand national Transverse Mercator grid, and some who I've known for years but never actually met, like Mike Binge - he and I have been on the same side, preaching the Surveying and GIS message for years, and often diametric opposites, busting each others' chops on political issues...

The biggest thing for me - some of my main customers were there... And though there's plenty of excellent showcasing and eyepopping demos, sometimes one has to put the brakes on and use a cautious and critical eye. We are currently in the midst of going from just ArcIMS in our shop to Geospatial Portal Toolkit and ArcGIS Server for Web development, (yay, great new toys!), but there are still some questions and concerns that linger, with regard to placing too much dependency on some of the more granular out-of-the-box ESRI pieces that these ship with, for map viewers, for instance - with developer ADFs not yet completely standardized and harmonized, not all versions of map viewers standardized, and the very distinct possibility in some instances of an ESRI update rollout breaking our apps where we have plugged them into our customizations.

The good news is that the new architecture I have been developing decouples components anyways, so I will be more able to plug in some robust ESRI pieces rather than reinvent the wheel, and this strategy also reduces potential breakage and/or makes the breaks much simpler to fix, and the second half-full glass is that a more standardized and harmonized ADF is coming coming, and not necessarily terribly far off. I think the ESRI team has come quite a ways.

So... on to ArcGIS Server...

Tomorrow morning I'm boarding my flight to head out to San Diego for the 2006 ESRI User Conference - I am looking forward to getting away for a few days, and am looking forward to the conference. Quite a few friends and colleagues will be there once again... It will probably mostly be EPA-centric for me (you may find me helping out at their booth at several points), as a majority of my work is still focused on their projects. 85 people representing EPA, from program offices, to regions, to some of their contractors like CSC will be attending.

It's been a few years since I last went... I usually end up swamped with too many things on my plate, or with other competing events going on. So this time I end up hitting both FedUC and the regular UC in the same year. I am also expecting to see several of my other friends from MetaCarta and other companies and organizations there - and am also hoping to make some new friends and contacts.

A little over a week ago, I was invited to a classified NGA facility to discuss geospatial technologies and capabilities.

One of the things that came out of the conversation afterward, which I can discuss is AeroText, which is a product developed by Lockheed-Martin. AeroText shares some similarities to MetaCarta, in that it performs natural-language parsing and indexing of unstructured data.

Where AeroText particularly excells is in identifying things like people, things, places, events, temporal references, and co-references, for example if the subject of the discussion is Bill Gates, a subsequent reference to "he" or "his" may then be joined back to Bill Gates, as would other aliases, such as William Gates, Bill, or Mr. Gates.

AeroText appears to be a very powerful tool for intelligent extraction of data, and even supports multilingual applications.

On the geospatial side, the application uses a gazetteer of places of interest for resolving placenames. It appears to be very effective, allowing similar aliases to be utilized, however the content will be limited to locations contained in the gazetteer.

Here MetaCarta has the upper hand, as, in addition to a gazetteer, it also utilizes the Sagent geocoding engine to allow any street address to be resolved.

AeroText nonetheless appears to be a very robust and intriguing tool, with the main driving force being gurus who are linguists, as opposed to geospatial folks. However, I do see potential for tremendous growth for AeroText as it matures.

I have several interesting things cooking in the next few weeks –

One we just had a kickoff for - Agency Best Practices, which follows much of what I have already been doing in Geospatial Enterprise Architecture, but which is all the more serendipitous in light of the Geospatial Line of Business effort by OMB. I have already been working on a concept of operations for transitioning to and implementation of reusable, distributed service components for geospatial business (SOA and other enterprise-level solutions). My previous effort will be expanded and merged with efforts by other Domain Experts in other arenas, such as Portal/SSO, Service-Oriented Architecture, ETL, and other technologies. I’m putting a draft together for May 15th.

The other is to perform another MetaCarta integration, this one to geo-enable a repository containing 50 million documents, for MetaCarta-powered geocoding and mapping. I will likely follow a similar model as the previous integration project, using ArcIMS and existing enterprise mapping as the base with a query interface, perform the SOAP call to the appliance, parse the response and plot them on an acetate layer. This one will be a new J2EE implementation from scratch, as opposed to the previous one, which was an integration into a legacy ASP/IIS-based application.

The announcement went out yesterday morning - President Bush has nominated a new CIO for EPA, to replace Kim Nelson, who left late last year. The new CIO is Molly O'Neill, who has been heading EPA's Environmental Council of States (ECOS) efforts within the Exchange Network for bringing data in from the states. To date, Linda Travers had been filling the post as acting AA. Hopefully O'Neill will bring her experience in what's involved in integration of data from external sources toward further supporting ongoing efforts for internal integration and SOA.

I have been working extensively on the Geospatial side, toward mapping and modelling as-is and to-be architectures for the entire horizontal sector of Geobusiness across EPA. "Line of Business" is becoming a major issue within OMB these days, and Geospatial processes may be part of that equation. I also has been preparing for a presentation on MetaCarta to Linda Travers - this may now become my introduction to the new CIO instead.

Our EPA GIS Center of Excellence team did manage to get a couple more people aboard, they are starting in two weeks, but we still have room for one more GIS developer.

The main thrust of the work involves GIS web applications (such as ArcIMS-based apps), along with GIS web services (OGC WMS and WFS) and GIS database development (Oracle 9i/10g / Oracle Spatial and SDE 9.1 environment)...

A full description is available here.

I've been delinquent on updating my blog- lots of stuff going on, fortunately most of it good. A little while back I attended the MetaCarta Users Group meeting with Dave Catlin of the EPA Geoservices team, we presented the Window To My Environment prototype that features deep integration of MetaCarta search technology. We're finally also working through some of the agency bureaucracy to move the MetaCarta appliance toward a production environment and allowing it to be populated with real data, as opposed to the months-old canned data we had been using to date. We did a trial crawl with the unit, and it behaved quite well- EPA National Computing Center staff had evidently been worried about it running rampant and consuming all available bandwidth, crashing servers, et cetera. We can tweak its crawling behavior quite well, though...

One of the comments at the MetaCarta Users Group was that it was an interesting and unusal thing to have EPA and CIA in the same venue presenting together. At any rate, some creative seat-of-the-pants flying involving wi-fi to VPN to remote desktop connection and we were able to demo the application live, in addition to the PowerPoint material (aside from applications live in MetaCarta's shop, everyone else's presentation was canned...)

The MetaCarta team had a number of great things they shared at the Users Group meeting. One of the things I thought was pretty exciting was the new ArcGIS functionality, which can provide temporal views of the data, when used with Tracking Analyst. There were a few presentations which show DoD and Intel applications of the technology in analyzing message traffic, to provide near-realtime spatial and temporal data mining and display capabilities. Great stuff. What became a running bit from one presentation to the next was the obligatory example of IEDs in Iraq...

They also discussed MetaCarta labs - some public betas of various MetaCarta experiments. More on those later, but they are doing a lot of fun stuff with Google-style AJAX. I hadn't realized it but Schuyler Erle, coauthor of Google Maps Hacks is now working with MetaCarta as well...

I apparently also missed a chance to meet face to face with GeoMullah, of the Fantom Planet blog... Have had some discussions with him since, on Open Source solutions.

Some previous MetaCarta-related posts:

• Architecture, MetaCarta and other stuff....
• Back from FedUC
• ESRI Federal Users Conference Tomorrow...
• Feedback wanted
• ESRI Federal User Conference
• Pitfalls of Appliance-based Technology...
• Map Symbology
• Viva Las Vegas!
• EPA Environmental Symposium in Las Vegas

Haven't posted in a few days, things have been very hectic. We serendipitously have synergy of many things coming together at once - an excellent opportunity to rearchitect a number of legacy GIS databases and applications for 2006, at the same time that new Federal Enterprise Architecture guidance and other technology items are arriving on the scene, such as enhanced support for OGC standards in the pipeline from Dr. Sharma and the Oracle Spatial team and from ESRI. I've been spending a lot of time over the last two weeks trying to go from high-level enterprise standpoint to solution architecture across over a dozen systems, identifying the gaps, overlaps, and opportunities for standardization, FEA realignment and reusability. This also ties in well with work I'm doing for the USEPA Geospatial Metadata/Data/Services Architecture project I've been involved in since the fall.

Apart from that, plenty of exposure on a few other things- the 2006 MetaCarta Public Sector User's Group meeting at Tyson's Corner on the 16th is going to highlight one of our recent successes in integrating MetaCarta technology into the USEPA "Window To My Environment" application. I'm hoping to attend for at least part of it.

For anyone interested in attending the 2006 MetaCarta Public Sector UG, I think they may still have some (very limited) space remaining... Details are available on the MetaCarta website, or you can contact my good friend John-Henry Gross, 703-629-0972. I note that one of the highlights is that John-Henry will be discussing the new GTS Analyst feature, as well as some enhancements in the GIS Connector and other additions and improvements.

This WME/MetaCarta application is also going to be featured at the upcoming USEPA Contractor Forum as well...

Just got home from Washington DC, but it was an extremely busy week with one meeting after another all around the beltway- Rumor is still circulating about us being asked to attend at the State of the Union address... Well, one sad truth is that I ultimately only got to spend about three hours at the ESRI FedUC conference yesterday afternoon. One business development opportunity after another this week - many interesting things brewing.

Got to see a bunch of familiar faces- my friends from the EPA geospatial world were present- Dave Wolf and Dave Catlin of the EPA Internet Geoservices team, Kevin Kirby, point man for the Locational Data Improvement and other initiatives, and Wendy Blake-Coleman, who I've been working with on next-generation architecture for EPA geospatial business. I understand GIO Brenda Smith was slated to speak this morning- unfortunately I was tied up in a meeting to discuss a couple of upcoming DOD projects.

My friends from the Metacarta crew had a nice hospitality room, they want me to get together with their marketing folks to promote our recent successess with integrating their products in EPA. They have a few new tools in the works for their DOD and Intel customers, including an analyst function that includes customizable alert features via e-mail and other notifications, as well as RSS feeds, to provide instant updates when a match of specific geographic and keyword parameters is made- they also have several other new things out and in the process of coming out.

There was a contingent of fellow Pennsylvanians present- PA GIS coordinator Jim Knudson and his deputy Stacy White were there- although I unfortunately didn't get to see Jim, I did say hello to Stacy. I also saw GeoDecisions had a booth- and found out later, over a few beers with a friend from Lockheed-Martin that we have some mutual friends and potential collaborative projects brewing between our company, Lockheed and GeoDecisions. PA firm Michael Baker was also present - who I did business with years back, though manned by the Virginia contingent. Here was Bob Austin, who has also recently been brought into our Integrated Project Team for the EPA Geospatial Data, Metadata and Services Architecture project I have been working on with Wendy Blake-Coleman. It's exciting to see so much synergy between different pieces of different projects, customers and agencies, and past and present partners of ours suddenly converging in many interesting ways.

Well, I've got much to catch up on and much to digest- was a very interesting week, even though most of it was outside of the FedUC context. Certainly there will be more bits and pieces to post as I get caught up tomorrow and over the weekend.

I'm registered for the ESRI Federal Users Conference tomorrow through Thursday... I'm looking forward to seeing the speakers, vendors, exhibits and old cameraderies, particularly our friends, partners and folks that we have done work with over the years, like Metacarta, SRA International Inc., Topcon Positioning Systems, Inc., BAE Systems ADR, i-cubed, Michael Baker, GeoDecisions, Trimble, Thales Navigation, Safe, and others... however things are going to be tough.

I have quite a few other meetings around the DC area booked throughout the week as well. I'm hoping to do a couple of hit-and-run attendances throughout the week but as far as any substantial block of time, it looks rough.

For anyone else who is thinking about attending but who doesn't have the info, I'd highly recommend it...

http://www.esri.com/events/feduc/index.html

Hope to see you there!

Looking for thoughts and input to improve the EPA's Window To My Environment application-

http://www.epa.gov/enviro/wme/

Enter your City/State or ZIP code, and the application will render a map window, with the ability to query environmental, demographic, and facilities data.

Aside from the Metacarta pilot we are working on (still in staging) our team will have a further opportunity to update and enhance this legacy EPA application with regard to usability, functionality, user interface, performance, et cetera. If anyone has any thoughts, suggestions, recommendations, please contact me: dsmith@synergist-tech.com

I'm thinking about going to the ESRI Federal Users Conference in Washington, DC, January 31-February 2 2006.

Hopefully I will actually be able to attend for a reasonable part of it, but my schedule might be tough- just prior to it is the EPA's Hurricane Katrina: Lessons Learned session down in Research Triangle Park- I am probably going to attend that event as part of the work we have ongoing with EPA's Geospatial Enterprise Architecture along with the work we did in support of Katrina for the EPA Emergency Operations Center, as well as Geodata services and the Enviromapper for Katrina deployments our team rolled out.

I tend to find that trade shows can come with mixed blessings, as they tend to be everyone selling to everyone else. I have presented at a few shows, and usually that results in getting a lot of people coming up who want to give me their business card, telling me about how wonderful their own company is- "buy our products and services"... Oh, yay. Yet when I ask a technical question about the product or services they are selling, often not much more than a glazed look in return. The more technical types of conferences are the ones I like far better, where the techies can meet and talk- that and ones where they are internal to the agency, typically invitation-only.

I guess the upshot will be, as always, seeing a few old friends (I see Metacarta and a few other familiar faces will be manning booths), perhaps an interesting presentation or two, and some new contacts made. And finally, worst case, what better opportunity to remind some of these agency people and large prime contractors of Public Law 106-50, and subsequently Executive Order 13360, requiring that 3% of Federal contracting dollars to go to Service Disabled Veteran Owned Small Businesses, and that there are indeed qualified, competent and capable SDVOSB firms that do geospatial - like ours :).

For the past couple of months, one of the projects I have been working on is integrating MetaCarta into Window to My Environment (WME)- The deep integration is achieved through the use of SOAP calls from within the WME application to the search appliance. Writing the code, parsing the results, and plotting them in ArcIMS was fairly trivial (WME is a legacy ASP / ArcIMS ActiveX Connector implementation).


On the other hand, the task that has been far more daunting is the enterprise security aspect. EPA, like many agencies and enterprises, has guidelines and testing parameters for applications. In the instance of appliance-based technology and this era where hardware and software begin to blur, such as routers with embedded linux, the standard practice of the appliance industry is to build a box that has its own OS, typically Linux-based, that has then been stripped of unnecessary functionality wherever possible, locked down for security, and tweaked to optimize device performance, wherever possible. In the process, they typically go through their own security audits as well, and get various security certifications from their target markets.

With Metacarta, we ended up going through several months of tests and probes by EPA IT security staff- finally the unit emerged, passing with flying colors. We got the unit back in our development environment and finished developing the application in a few weeks. Now, we're again waiting more weeks for the unit to be stood up again at EPA so that the app can be deployed.

It seems to me that this process is overly cumbersome. Perhaps part of the issue, not to blame Linux, is that Linux is so customizable, to the point of being a very different OS from one implementation to the next. The other part that leads to things being cumbersome is that it seems every agency and organization has differing security criteria and concerns. The appliance in question has already been used in DoD and intel applications, which one would think would have extremely stringent security requirements, as opposed to a civilian agency. I would think there should be a certain core set of criteria which go across the board between all customers- this type of common certification would certainly go far, as would acceptance of other agencies' certifications.

Just my humble $0.02...