It is often still necessary to run certain applications on an EC2 instance, particularly if it is not easily containerised and you do not want to pay extra for a managed service. In a non-production environment, you will be able to reduce costs by shutting down these instances when they are not needed by your developers or users.
The team at AWS has already put together a CloudFormation template you can deploy. The documentation and deployment guide is available here:
It uses a new command-line interface to initialise and configure a schedule. This metadata is stored in DynamoDB, and a…
Geoscience data has typically been harder to obtain as it comes from disparate source systems and physical media, and the file sizes — particularly for seismic surveys— can be massive. The Geological Survey of Queensland (GSQ) has recently undergone a data modernisation transformation, and terabytes of geoscience data is now available on their new Open Data Portal (ODP) at https://geoscience.data.qld.gov.au. Work is still underway to bring online vast archives of newly digitised data from literal warehouses — totalling over 120TB — some of which will also make it into the portal.
Accessing rich datasets from public web platforms is a common scenario for clients and the trend looks set to continue. The number of social media users is growing by the millions as more people get access to faster and cheaper smart devices. For one of our clients, getting data from some of these online platforms and matching them with their internal databases seemed like it would be a time consuming process. Their requirements were based on compliance checks, so each post needed to be captured and logged in its entirety so that they could be cross-checked and analysed.
Data Consultant @ Servian