I started this blog back in 2006 during a time when I wasn’t doing much geospatial work at all. I was working on building a human resources system for a federal government customer who was falling under the then-new and now-defunct National Security Personnel System. Because it was new and sufficiently different from the GS system, there were no off-the-shelf products to acquire. So I found myself deep in the development of logic to model workflows for personnel reviews, tracking accomplishments, and other minutiae of managing different types of personnel. There was no room for anything geospatial and I felt it, probably incorrectly, slipping away so I started doing personal projects at home. This blog started out as the means for documenting those diversions, which included my first dabblings with PostGIS among many other things.
I find myself in a similar period now. I’ve been mostly occupied the past few months with migrating to a new billing system. It’s not sexy and it’s certainly not geospatial, but billing is a necessary engine of any business. When people talk about “growing pains” as businesses scale up, billing is one of the biggest.
Migrating any key business system from one platform to another invites a series of related considerations. Some of the most pressing I’m dealing with right now are data integration, business intelligence, and reporting. Data that we used to get from one place is now coming from another, behind a different API. Like many organizations, our “backoffice” business system are all SaaS. SaaS platforms, to state it charitably, don’t place an emphasis on openness. I look back fondly on the debates that raged within the geospatial community about open formats and interoperability.
The larger SaaS and information systems world doesn’t see the need or value in such things. Openness, as the geospatial community has debated it, runs counter to the SaaS value proposition of getting users to lock their data into a platform. Openness, to the extent that it is pursued at all, typically comes in the form of blessed integrations between “partner” platforms. APIs tend to be REST in the sense that there’s an HTTP interface with some form of OAuth or token-based authentication that will return arbitrary JSON payloads. Arbitrary in the sense that an “account” from one CRM platform looks nothing like an “account” from another CRM platform.
This leads me to the concept of the “Integration Platform as a Service” (IPaaS). This unwieldy name is the label given to a bevy of 80% solutions that amount to hosted ETL. The best, those which approach that 80% threshold, usually have Apache Airflow under the hood somewhere. The rest diminish in utility from there. None have any meaningful understanding of spatial data objects or formats unless in the exceedingly rare case that Esri’s ArcGIS Online is one of the supported source or destination platforms.
The SaaS business systems I am working with are not exotic. Many are the de facto leaders in their particular segments. Even still, I have yet to find an IPaaS platform that adequately supports all of them as sources and writes to my intended destination, BigQuery. One of my goals is to do as little custom integration as possible. Many platforms have prioritized the “ease” of no-code to the point where no meaningful customization is possible. This can result in transferring too much data or it can result in failure when an internal integration within a source system introduces unexpected objects or data types. (This is all too common in the Salesforce ecosystem.)
My most reliable fallback continues to be FME. FME was in use when our former data service product was in existence and made the transition to support the ETL/ELT needs of our internal analytics. It has deep spatial roots, though that is less relevant for current needs. What continues to be valuable about FME is its robust reader/writer model that tends to offer far more configuration options for supported sources and formats than is typical with IPaaS offerings. It also offers the ability to massage transformations with Python if needed. Finally, it can be extended to support to arbitrary JSON payloads from various proprietary APIs. This last part is useful but can also be fragile as underlying APIs change. (Which is not the fault of FME.)
FME has a native reader and writer for Salesforce, which is incredibly useful and one of the better ones I have found. If Safe Software is open to suggestions, I have a list of other common, non-spatial business platforms I’d be happy to see FME support in a similar manner.
That said, until FME brings an IPaaS-like offering to market, the openness and interoperability that we luxuriate in (and often complain about) in the geospatial market continues to be elusive in the world of business-oriented SaaS platforms.
