Using distributed databases across various microservices will be explained based on a research project example. The presentation elaborates of how to achieve data consistency accross microservices, how to communicate using message brokers, how to scale the microservices and achieve application high availability. Container virtualisation and orchestration are technology basis for the solution. The project example show of how to build an Artificial Intelligence (AI) solution - as a service!
Target Audience: Data Engineer, Data Scientist, Project Leader, Architects
Developing microservices has various advantages compared to traditional monolith approach. Loose coupled microservices enables smaller teams to develop independently and using CI/CD on a daily basis. Microservices can be scaled independently using different database technologies optimised for particular use cases. The microservices are fault isolated so particular failures will not result in an overall outage.
But how to ensure data consistency for all distributed databases accross the microservices? How to react on particular failures? And how to interact and communicate between services?
A research project for intelligent energy analysis will be presented. The solution realizes an Artificial Intelligence (AI) solution analyzing streaming data near real time to ensure energy savings in a production environment. The presentation will explain the steps necessary to establish a microservice environment for Artificial Intelligence and Machine Learning. Central logging guarantees operations monitoring accross all microservices. Dashboards presents the results to technical staff monitoring the Machine Learning libraries as well as to the process owner of the domain, e.g. the operations manager or insurance agent. The solution is based on Docker as container virtualisation technology and Kubernetes for container orchestration. In the research project, the solution is realized on premise, but can be easily deployed in the cloud.