Photo by Eric McCowan

It is often still necessary to run certain applications on an EC2 instance, particularly if it is not easily containerised and you do not want to pay extra for a managed service. In a non-production environment, you will be able to reduce costs by shutting down these instances when they are not needed by your developers or users.

Off-the-shelf scheduling with an AWS template

The team at AWS has already put together a CloudFormation template you can deploy. The documentation and deployment guide is available here:

It uses a new command-line interface to initialise and configure a schedule. This metadata is stored in DynamoDB, and a…


Geoscience data has typically been harder to obtain as it comes from disparate source systems and physical media, and the file sizes — particularly for seismic surveys— can be massive. The Geological Survey of Queensland (GSQ) has recently undergone a data modernisation transformation, and terabytes of geoscience data is now available on their new Open Data Portal (ODP) at https://geoscience.data.qld.gov.au. Work is still underway to bring online vast archives of newly digitised data from literal warehouses — totalling over 120TB — some of which will also make it into the portal.

The data management system GSQ chose for their new…


Spiders can be used to capture website content (Photo © Eric McCowan)

Accessing rich datasets from public web platforms is a common scenario for clients and the trend looks set to continue. The number of social media users is growing by the millions as more people get access to faster and cheaper smart devices. For one of our clients, getting data from some of these online platforms and matching them with their internal databases seemed like it would be a time consuming process. Their requirements were based on compliance checks, so each post needed to be captured and logged in its entirety so that they could be cross-checked and analysed.

An ideal…

Eric McCowan

Data Consultant @ Servian

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store