cloud

Analyzing Location Change Over Time in BigQuery

by

I’ve recently spent a lot of time doing various forms of business analytics in BigQuery. As discussed in a previous post, I’ve been using BigQuery as the data integration environment for several business systems. I’ve found integration at the data level via an ETL/ELT/IPaaS pipeline to be a lot more stable than system-level integrations that involve chaining together dependencies on fairly volatile SaaS APIs.

The original premise was fairly straightforward: Given a table of user-level statistics over time, identify only those points in time where one or more of the statistics changed value. In our case, we had several million rows of user-level data captured on a daily cadence. Manually inspecting this data for changes in individual values by customer was simply not a viable plan. The BigQuery LAG function came to the rescue.

Read more

Building a Simple Geodata Service With Node, PostGIS, and Amazon RDS

by

tl;dr

This post describes the construction of a simple, lightweight geospatial data service using Node.JS, PostGIS and Amazon RDS. It is somewhat lengthy and includes a number of code snippets. The post is primarily targeted at users who may be interested in alternative strategies for publishing geospatial data but may not be familiar with the tools discussed here. This effort is ongoing and follow-up posts can be expected.

Read more

Arkansas Evaluating Geospatial Cloud Infrastructures

by

My company has recently begun working on a project for the Arkansas Geographic Information Office (AGIO) to evaluate options for potentially migrating the GeoStor geospatial infrastructure to a cloud computing environment. If you are unfamiliar with GeoStor, it is the official geospatial platform for the State of Arkansas and is maintained by AGIO. I realize, as does AGIO, that “the cloud” has become a buzzword that has lost any meaningful context. So, for purposes of discussion, we’ll go with the NIST draft definition available here: http://csrc.nist.gov/publications/drafts/800-145/Draft-SP-800-145_cloud-definition.pdf.

For this effort, we are open to considering any service model but we have strict guidance that the only deployment model of interest is the public cloud model.

Read more

Clouds

by

I spent the vast majority of my time at the 2010 ESRI User Conference working the Zekiah/Arc2Earth booth. That was fun as I got meet/reconnect with a lot of people but I didn’t see much of the conference itself. As a result, I haven’t really blogged it.

ESRI continued with the “cloud ready” theme that was rolled out at the Federal User Conference but with more details about how they are moving to “the cloud.” This generated a lot of buzz amongst many of the attendees from what I could tell. One of the big new features of Arc2Earth v3 (disclaimer: my company is an Arc2Earth reseller) is Cloud Services. As a result, we had a banner in our booth that had the word “cloud” on it, prompting lots of people to stop.

Read more

Importing Data From GeoCommons Into ArcMap

by

UPDATE: The code for this post is available at the bottom of the page.

I have been doing a lot of development with the ESRI Silverlight API recently. One of the requirements of my project is to be able to dynamically add KML data at runtime. The incorporation of KML was handled for us through one of the ESRI samples on the resource center so we pretty much just had to integrate that code and test against our use cases. For testing, I typically reached out to GeoCommons since any data set available there can be streamed as KML.

Obviously, this is not my first exposure to GeoCommons but, when discussing it, I found that many of the analysts I spoke with were not aware of it and did not use it much. So I decided to tackle developing a simple ArcMap extension to allow a user to search GeoCommons and then download/add data to ArcMap without the need to manually download, unzip and add the data themselves.

Read more