Reading time:

5 minutes

Word count:

1025

This are my verbatim notes to the PEAT UK podcast:

Hello there once again to another hot shot. My name is Peter Pilgrim, DevOps specialist, Java enterprise and platform engineer and Java Champion.

How do microservices M/SOA deal with data?

Software developers certainly write code that processes data. We can become blind and less caring to the data that we are asked to process. I certainly understand this way of working when building a service. All we care about is where does that data arrive? Where do we need to push that data? Do we need to save this very important and critical data into a persistent store? Do we need to transform the data from one form to another form in the middle of the operation?

Coding data process is essentially the life of engineer. Information arrives on a message gateway, queue or channel and generally we do something useful with it. Well actually, we apply business logic to it. The same happens to the data that arrives from UI, know a days it might be a JSON payload from some sexy JavaScript framework. It is still data, and ordinarily, we might have to filter, reduce and map that data to another form, namely an object representation in our favourite programming language. Of course, Java has all of this boilerplate with getters and setters, we might be lucky enough to use a framework to transform JSON into objects and back again. If we are extremely lucky then we write our microservice code in Groovy, Scala or even Kotlin since those modern JVM language support case classes (or data classes).

Even then, we are programming with data, do understand we actually the data that we are working with?

Data at Rest

Data at rest in information technology means inactive data that is stored physically in any digital form (e.g. databases, data warehouses, spreadsheets, archives, tapes, off-site backups, mobile devices etc.).

This information that developers of microservices have to know. What happens to your customer’s data in that microservice? After the microservice has processed the information, this data is further published onwards to another microservice? Where does it flow to? What is the conduit? Is it Rabbit MQ or Kafka or something else? What happens if the data is saved to a database? Is it persisted to AWS RDS e.g Aurora in the cloud inside your production environment?

So data at rest for a engineers is the storage state of the data. You can get think of this as long term storage. Your business might have some concerns of where the data is stored. For instance, in the cloud environment, can you be sure that the data is fixed to geographical region of the world.

AWS has the concept of regions of most of its services. I can develop a microservice application in the UK and ensure that my VPC and availability zone are fixed to eu-west-2 (EU London), I am pretty certain that my data at REST is fixed to UK. I know this because I can launch EC2 instances in this region only. As long as I configure the security groups and ensure that the applications and service running on EC2 cannot see outside of this region, I can be reasonably confident that my Data at Rest inside the UK, and if it is is not, I would take this up with Amazon. Well not me, but this billion dollar company wouldn’t have a leg to stand on, if it was proven that data can leak across regions with its infrastructure.

So data at rest is clearly important for customer data. Especially, in the European Union, including the UK, until March 2019, we need to think about GDPR.

Data at Use

Data in use has also been taken to mean “active data” in the context of being in a database or being manipulated by an application. For example, some enterprise encryption gateway solutions for the cloud claim to encrypt data at rest, data in transit and data in use.

This is where we as software engineers have a great deal or responsibility. We write the service / MSOA that transform customer data into real outcomes, costs and benefits.

For platform engineers and DevOps understanding where Data is use and at Rest is also very important. For example, AWS have a Elastic Kubernetes Service (EKS), which is available to regions only in the USA / North America art the moment. Therefore it precludes any work where the application and data must be kept only Europe or the UK at the moment.
Another example, is AWS Simple Storage Service (S3) and you can configure a bucket to serve static web pages from an EC2 instance. If your application and friends use S3 as a very simple file transfer mechanism, then you have to protect and continuously review S3 mechanisms and permission in order to ensure only authenticated and authorised users and applications can see only the Data at Rest that they are entitled to.

Salient Notes

GDPR – General Data Protection Regulation which came into force on 25th May 2018 for citizens of the European Union

Safe harbour – customer data privacy issues. Essentially sharing data across The Pond (The Atlantic Ocean) concerning global conglomerates and international institutions.

Data at Rest – definition from Wikipedia and also includes Data in Use.

Amazon Relational Database Service (RDS) – official documentation on a bespoke MySQL implementation and extension in the cloud. See also Amazon Aurora DB – Wikipedia

Amazon Elastic Container Service for Kubernetes Service (EKS)

Amazon Simple Storage Service (S3)

Security breach #1 – Tech Republic of an S3 leaked data breach with FedEx customers

Security breach #2 – 7% of All Amazon S3 Servers Are Exposed, Explaining Recent Surge of Data Leaks
(Bleeping Computer)

Security breach #3 – Data on 123 Million US Households Exposed Due to Misconfigured AWS S3 Bucket and also Info Security Every Single American Household

That’s it for this hotshot. I hope you liked it.

Bye

+PP+
May 2018

By the way, all your Shares, Likes, Comments are always more than welcomed!