Automated Machine Learning (AutoML) and Big Data Platforms

Although machine learning exists already since decades, the typical data scientist – as you would call it today – would still have to go through a manual labor-intensive process of extracting the data, cleaning, feature extraction, regularization, training, finding the right model, testing, selecting and deploying it. Furthermore, for most machine learning scenarios you do not use one model/algorithms but evaluate a plethora of different algorithms to find one suitable for the given data and and use case. Nowadays a lot of data is available under the so-called Big Data paradigm introducing additional challenges of mastering machine learning on distributed computing platforms.

This blog post investigates on how to ease the burden on the data scientists of manual labor-intensive model evaluation by presenting insights on the recent concept of automated machine learning (AutoML) and if it can be adapted to Big data platforms.

What is AutoML

Machine learning is about learning and making prediction from data. This can be useful in many contexts, such as autonomous robots, smart homes, agriculture or financial markets. Machine Learning – despite the word „machine“ in its name – is mostly a manual process that requires a highly skilled person to execute. Although the learning part is rather automated, this person needs to extract data, transform it in potentially different alternative ways, feed it into many alternative machine learning algorithms as well as using different parameters for the same algorithm, evaluate the quality of the generated prediction model and finally deploy this model for others so that they can make their own predictions without going through this labor-intensive process themselves. This is illustrated in the following figure:
AutoML Flow Diagram(1)

Given this background, it comes at no surprise that huge market places have been created where people sell and buy trained machine learning models. For example, the Azure Machine Learning Market place, the Amazon AWS Artificial Intelligence market place, the Algorithmia Marketplace, the caffe model zoo, deeplearning4j models, mxnet model zoo, tensor flow models or the Acumos Marketplace (based on the open source software).

Nevertheless, most organizations that wants to keep up with the competition in these market places or that do not want to rely on market places due to unique problems in their organizations have still the issue to find skilled persons that create prediction models. Furthermore, machine learning models become so complex that they are not the outcome of a single person, but a team that needs to ensure consistent quality.

Here AutoML comes into play. It can support highly skilled machine learning persons, but also non-skilled machine learning persons to create their own prediction models in a transparent manner. Ideally this is done in an automated fashion for the whole machine learning process, but contemporary technology focuses mostly on the evaluation of different models, a set of suitable parameters for these models („hyperparameter optimization“) and automated selection of the „winning“ (or best) model(s) (this is highlighted in the previous figure in green). In case of several models a ranking is created on a so-called “leaderboard”. This has to be done in a given time budget, ie one has only limited time and resources. Potentially several models could be combined (aka deep learning) in more advanced solutions, but this is currently in its infancy.

Some observations here for this specific focus:

  • AutoML does not invent new models. It relies on an exisiting repository of algorithms that it iterates and tries out with different parameters. In itself AutoML can be seen as a “meta-algorithm” over these models.
  • Similarly it relies on an existing repository of tests that determine suitability of a model for a given problem.
  • It requires clever sampling (size of sample, randomness) of the underlying training data, otherwise it may take very long to evaluate the models or simply the wrong data is used. A preselection of the data by a person is still needed, although the person does not require as much machine-learning specific skills.
  • A person still needs to determine for the winning model if it makes sense what it predicts. For this the person does not need machine learning skills, but domain specific skills. For instance, a financial analyst can determine if from a dataset of financial transaction attributes can predict fraud or not.

Big Data platforms

The emergence of the open source Hadoop platform in 2006 introduced Big Data platforms on commodity hardware and networkl clusters to the world. Few years later Hadoop was adopted for data analytics in several data analysis organizations. The main focus of Hadoop was to enable analytic tasks on large dataset in a reliable manner it was not possible before. Meanwhile further improved platforms have been created, such as Apache Flink or Apache Spark, that focus not only on processing large data volumes, but processing them also faster by employing various optimization techniques.

Those platforms employ several nodes that communciate over a network to execute a task in a distributed manner. This imposes some challenges for machine learning, such as:

  • Standard machine learning algorithms are not designed to be executed over a distributed network of nodes.
  • Not all machine learning algorithms can be converted into a distributed one. For instance, if you need to estimate the parameters of a model then gradient descent might require a lot of memory on a single node. Hence, other estimation methods need to be used to work in parallel on different nodes.

This led to specific machine learning libraries, such as Flink ML or Spark Mlib, for those platforms that supported only a dedicated subset of algorithms that can be executed efficiently and effectively over a given set of nodes communicating via the network.

Tool support for AutoML

AutoML can be very useful for your organization. Amongst others the following tools exist.

Tool Description Supported Models Supported Hyperparameter optimization
Auto-SKLearn Automated machine-learning toolkit to be used in lieu of the non-automed scikit-learn. Budget can be defined by time and memory as well as search space. Automated preprocessing can be defined. Multicore processing is supported for some of the algorithms. Multiple classifiers and regressors as well as combinations (ensemble construction) Bayesian Optimization
TPOT Automated machine-learning toolkit offering various automated preprocessors (e.g. Standard Scaler, Principal Component Analysis). Multiple classifiers and regressors as well as combinations Genetic Programing
Auto-Weka Automated machine-learning toolkit. Budget can be defined by time and memory as well as search space. Multiple classifiers and regressors as well as combinations Bayesian Optimization
Devol Automated machine-learning toolkit for deep neural network architectures. Expects the data to be prepared and encoded. Sees itself more as support for experienced data scientists. Neural Networks and combinations. Genetic Programing
Machine.js/Auto-ML Automated machine learning kit based on auto-ml. Expects the data to be prepared and encoded. Various classifiers and regressor as well as Neural networks based on the Keras library. Supports combinations. Genetic Programing / Gridsearch

Most of these tools support only one method for hyperparameter optimization. However there are several methods. Some models do not require hyperparameter optimization, because they can derive optimal hyperparameter from the trained data. Unfortunately, this is currently integrated in none of the tools.

Those tools might not always be very end user friendly and you still need to deploy them in the whole organization as fat clients instead of light-weight HTML5 browser applications. As an alternative popular cloud provider integrating more assistants in the cloud that help you with your machine learning process.

AutoML on Big Data Platforms

The aforementioned tools have not been primarily designed for Big Data platforms. They usually are based on Python or Java, so one could use them with the Python or Java-bindings of those platforms (cf. Flink or Spark). One could use the available data sources (and efficient data formats such as ORC/Parquet) and sampling functionality of those platforms (e.g. sampling in Flink or sampling in Spark) and feed it into the tools that could even run on the cluster. However, they would only use one node and the rest of the cluster would not be utilized. Furthermore, the generated models are not necessarily compatible with the models provided by the Big Data platforms. Hence, one has to write a manual wrapper around those models so they can be distributed over the cluster. Nevertheless, also these models would only use one node.

This is not necessarily bad, because usually data scientists will not run one dataset to evaluate with AutoML but multiple datasets so you can utilize the whole cluster by running several AutoML processes. However, it also means data size as well as budget for time and memory is limited to one node, which might not be sufficient for all machine learning problems. Another interesting aspect could be to run one or more winning models over a much larger dataset to evaluate it in more detail or to optimize it even more. This would again require a more deep integration of the AutoML tools and Big Data platforms.

H2O AutoML is a recent (March 2018) example on how to provide AutoML on Big Data platforms, but this has currently similar limitations as described before with respect to the Big Data platform. The only advantage currently is that the models can be generated are compatible with the machine learning APIs of the Big Data platforms.

Apache Spark has some methods for hyperparameter tuning, but they are limited to a pipeline or model and do not cover aspects of other AutoML solutions, such as comparing different models. Furthermore, it only evaluates out a list of given sets of parameters and no time or cost budget definition is possible. This would have to be implemented by the user.

One can see that AutoML and Big Data platforms can benefit from a more tighter integration in the future. It would then also be more easy to leverage all the data in your data lake without extracting it and process it locally. At the moment, although some machine learning models are supported by Big Data platforms (with some caveats related to distributed processing) not all functionality of AutoML is supported (e.g. hyperparameter tuning or genetic programing).

Google AutoML provides an automated machine learning solution in the cloud. It augments it with pre-trained models (e.g. on image data), so not everyone has to spend time again to train models.


AutoML is a promising tool to facilitate the work of less skilled and very skilled machine learning persons to enable them to have more time focusing on their work instead of the manual error-prone labour-intensive machine learning process. You can make your organisation on-board on machine-learning even if you are in your early machine learning stages and facilitate learning of many people on this topic.

However, it does not mean that people without knowledge can use it out of the box. They should have a least a basic knowledge on how machine learning works in general. Furthermore, they should be domain experts in your organizations domain, e.g. a financial analyst in banks or an engineer in the car industry. They still need to decide which data to input to AutoML and if the model learned by AutoML is useful for your business purposes. For example, it does not make sense to put all the data of the world related to problematic situations for autonomous cars into an AutoML and expect that it can solve all problematic situation as best as possible. Moreover, it is more likely to be country/region-specific so it may make more sense to have several AutoML runs with different countries/regions to develop specific models for countries/regions. Other datasets, such as blockchains/cryptoledgers are more complex and require currently a manual preprocessing and selection.

Another example is known from spurious correlations, ie correlations that exists, but do not imply causality. In all this case you still need a domain expert that can judge if the model is a useful contribution for the organization.

All these things are related to the no free lunch theorem.

Even for highly-skilled machine learning persons AutoML can make sense. No-one can know all particularities of machine learning models and it simply takes a lot of time to evaluate them all manually. Some may even have their favorite models, which may mean other models are not evaluated although they may fit better. Or they simply do not have time for manual evaluation, so a preselection of candidate models can also be very useful.

One open issue is still how much time and resources you should let AutoML spend on a specific dataset. This is not easy to answer and here you may still need to experiment if the results are bad then you need to spend probably more.

Nevertheless, AutoML as a recent field still has a lot of room for improvements, such as full integration in Big Data machine learning platforms or support of more hyperparameter tuning algorithms as well as more user-friendly use interfaces, as pioneered by some cloud services.

Then, those models should be part of a continuous delivery pipeline. This requires unit testing and integration testing to avoid that the model has obvious errors (e.g. always returns 0) or that it does not fit into the environment in which it is used for prediction (e.g. web service cannot be called or model cannot be opened in R). Integration machine learning models into continuous delivery pipelines for productive use has not recently drawn much attention, because usually the data scientists push them directly into the production environment with all the drawbacks this approach may have, such as no proper unit and integration testing.

Spending Time on Quality in Your Big Data Open Source Projects

Open source libraries are nowadays a critical part of the economy. They are used in commercial and non-commercial applications directly or indirectly affecting virtually any human being. Ensuring quality should be at the heart of each open source project. Verifying that an open source project ensures quality is mandatory for each stakeholder of this project, especially its users. I describe in this post how software quality can be measured and ensured when integrating them in your application. Hence, we focus only on evaluating the technical quality of the library, because it suitability for business purposes has to be evaluated within the regular quality assurance process of your business.

The motivation for this post is that other open source projects will employ also transparent software quality measures so that eventually all software becomes better. Ideally, this information can be pulled by application developers to have a quality overview on their software, but also its dependencies. The key message is that you need to deploy solutions to efficiently measure software quality progress of your applications including dependencies and make the information transparently available in a dashboard (see screenshot) for everyone in near realtime.


Although I write here about open source software, you should push your commercial software providers to at least provide the same level of quality assurance and in particular transparency. This also includes the external commercial and open source libraries they depend on.

Software quality is “The totality of functionality and features of a software product that bears on its ability to satisfy stated or implied needs” (cf. ISTQB Glossary). This definition is based on the ISO9216 standard (succeeded by ISO/IEC 25000), which describes various quality classes, such as functionality, reliability, usability, efficiency, maintainability and portability.

I chose two open source libraries in which I am deeply involved to demonstrate measures to increase their quality:

  • HadoopCryptoLedger: A library for processing cryptoledgers, such as the Bitcoin Blockchain, on Big Data platforms, such as Hadoop, Hive and Spark

  • HadoopOffice: A library for processing office documents, such as Excel, on Big Data platforms, such as Hadoop and Spark

Before you think about software quality, one important task is to think about what functionality and which business process that is supported by the application brings the maximum value to the customer. Depending on the customer needs, the same application can bring different value to different customers. When looking at the supported business process you also must consider that not all potential execution variants of it should be evaluated, but only the most valuable for the business.

At the same time, you should evaluate the risk that the application fails and how it impacts the business process. If the risk is low or you have equally mitigation measures (e.g. insurance) then you can put there less emphasis compared to a business process where the risk is high and no mitigation measures exist. Additionally, I also recommend to look at value of functionality. If a functionality provides a lot of value then you should test it much more. Sometimes this change of view from risk to value uncovers more important areas of quality of a software product.

Steps for measuring quality

One of the core requirements for increasing software quality is to measure quality for third party libraries. It implies that you have mapped and automated your software development and deployment process (this is also sometimes referred to as DevOps). This can be done in a few steps:

  1. Have a defined repeatable build process and execute them in an automated fashion via a continuous integration tool.

  2. Version your build tool

  3. Software repository not only for source code

  4. Integrate automated unit and integration tests into the build process.

  5. Download dependencies, such as third party libraries, from an artifact management server (repository) that manage also their versions

  6. Evaluate quality of test cases

  7. Code documentation

  8. Code Analytics: Integrate static code analysis into the build process.

  9. Security and license analysis of third party libraries

  10. Store build and tested libraries in a versioned manner into an artifact management server (repository)

  11. Define a Security policy (

  12. Define a contribution policy (

  13. Ensure proper amount of logging

  14. Transparency about the project activity

Of course, these are only technical means for measuring quality. Out of scope here, but equally important is to define and version requirements/use cases/business process models, system tests, acceptance tests, performance tests, reliability tests and other non-functional tests. Additionally you should take into account automated deployment and deployment testing, which is out of scope of this article.

Evaluate quality

Given the aforementioned points, we describe now how they have been realized for the hadoopoffice and hadoopcryptoledger. Of course, they can be realized also with different tools or methods, but they provide you some indicators on what to look for.

1. Have a defined repeatable automated continuous integration process

We use for the hadoopoffice and hadoopcryptoledger library the Continuous Integration Platform Travis CI, which is offered as a free service for open source projects in the cloud. The aforementioned projects include in the source code repository root folder a file named .travis.yml (example), which describes the build process that is executed by travis. Everytime new code is committed the library is build again including the examples and automated unit and integration tests take place.

Travis CI provides a container infrastructure that allows executing the build process in so called software containers, which can integrate flexible different types of software without the overhead of virtual machines. Container can be pulled from repositories and flexible composed together (e.g. different Java Development Kits, operating systems etc.). Additionally, one wastes the costs of virtual machines being idle and capacity not used. Furthermore, you can track previous execution of the build process (see example here) to measure its stability.

Gradle is used for nearly every step in the process to build the application, generate docuemntation, execute unit and integration tests and collect reports.

Shell scripts are used to upload the HTML reports and documentation to a branch in Github (gh-pages) so that it can be easily found in search engines by the users and viewed in the browser.

2. Version your build tool

As said, our main build tool is Gradle. Gradle is based on the popular Java Virtual Machine scripting language Groovy and thus it scales flexible to your build process as well as make build scripts more readable in comparison to XML based tools, which are inflexible and a nightmare to manage for larger projects. Furthermore, plugins can be easily implemented using Groovy.

Modern build tools are continuously evolving. Hence, one want to test new versions of the build tool or upgrade to new versions rather smoothly. Gradle allows this using Gradle Wrapper – you simply define the desired version and it downloads it (if not already done) to be executed for your current build process. You do not need any manual/error prone installation.

3. Software Version Repository not only for source code

One key important ingredient of the software development process is the software version repository. Most of the m now are based on GIT. It is extremely flexible and scale from single person projects to projects with thousands of developers, such as the Linux kernel. Furthermore, it allows you to define your own source code governance process. For example, the source code of open source libraries that I introduced in the beginning are hosted on a GIT server in the cloud: Github. Alternatives are Gitlab or, but also many more. These cloud solutions also allow you having private repositories fully backed up for a small fee.

As I said, these repositories are not only for your software source code, but also your build scripts and continuous integration scripts. For example, for the aforementioned software libraries it stores the Travis scripts and the build files as well as test data. Large test data files (e.g. full blockchains) should not be stored in the version management system, but in a dedicated cloud object storage, such as Amazon S3 or Azure Storage. Here you have also the possibility to store them on ultra-low cost storage which might take longer to access the files, but this can be alright for a daily CI process execution.

4. Integrate automated unit and integration tests into the build process.

Another important aspect of software delivery is the software quality management process. As a minimum you should have not only automated unit and integration tests, but also a daily reporting on the coverage.

The reporting for the aforementioned library is done via and Codacy. Both not only report results on static code analysis, but also on unit/integration test coverage. Two have been chosen, because Codacy supports Scala (a programing language used in part of the aforementioned libraries).

More interesting, you find on both platforms also result for popular open source libraries. Here you can check for the open source libraries that you use in your product how good they are and how they improve quality over time.

Of course there are many more cloud platforms for code coverage and quality analysis (e.g. SourceClear).

Another important method for testing beyond unit testing for developers is integration testing. The reason is that complex platforms for Big Data, such as Hadoop, Flink or Spark, have emerged over the past years. These tests are necessary, because unit tests cannot reflect properly the complexity of these platforms and system tests are not focusing on technical aspects, such as code coverage. Furthermore, the execution of integration tests is very close to the final execution on platforms, but requires less resources and virtually no installation effort (an example is Spring Boot for web applications). However, still unit tests are mandatory, because they are very suitable to verify quickly that calculations with many different inputs are done correctly automatically.

Nevertheless, these integration tests require additional libraries. For example, for the aforementioned library, which are supporting all major Big Data platforms, the Hadoop integration tests suites are used as a base. They provide small miniclusters that can be even executed on our CI infrastructure based on containers with very little configuration and no installation efforts. Of course we use also the testing features of the platforms on top of this, such as the one for Spark or Flink.

5. Download dependencies, such as third party libraries, from an artifact management server (repository) that manage also their versions

It should be a no-brainer, but for all of your third party libraries that you include in your application you should manage them an artifact management server. Popular choices are Nexus or Artifactory. Both offer also cloud solutions for open source and private libraries (e.g. Maven Central or Bintray). The public cloud solutions should be analysed with care, for example the NPM.js gate security affair for Javascript packages or the PyPi issue for Python packages, where attackers replaced popular packages or uploaded packages with similar names to popular packages to trick developers in using malicious libraries in their application code. Having a local artifact management server quickly indicates you which of your developed applications are exposed and you can initiate mitigation measures (e.g. directly communciate to the project which library is affected).

These repositories are important, because you do not need to manage access to third party libraries in your software project. You can simply define in your build tool the third party library as well as the version and it will simply download it during the build process. This allows you simply changing and testing new versions.

Furthermore, having an artifact management server allows you to restrict access to third party libraries, e.g. ones with known security issues or problematic licenses. This means you can easily make your organisations more safe and you can detect where in your software products are libraries with issues.

Especially with the recent trend of self-service by end users in the Big Data world it is crucial that you track which libraries they are using to avoid vulnerabilities and poor quality.

6. Evaluate quality of test cases

As I wrote before, quality of test cases is crucial. One key important aspect of quality is coverage, ie to which extend your tests cover the source code of your software. Luckily, this can easily be measured with tools, such as Jacoco. They give you important information on statement and condition coverage.

There is no silver bullet how much coverage you need. Some famous software engineers say you should have at least 100% statement coverage, because it does not make sense that someone uses/tests a piece of software if not even the developers themselves have tested it. 100% might be difficult especially in cases of rapidly changing software.

In the end this is also a risk/value estimation, but having high statement coverage is NOT costly and saves you later a lot of costs with bugs/generally better maintainable software. It often also uncovers aspects that someone has not thought about before and require clarification, which should happen as early as possible.

For the aforementioned libraries we made the decision that we should have at least 80% statement coverage with the aim to have 100% coverage. Condition coverage is another topic. There are some guidelines, such as DO-178B/DO-178C for avionics or ISO 26262 for automotive that describe a certain type of condition coverage for critical software.

For the aforementioned libraries, there was no such guideline or standard that could be applied. However, both libraries are supporting Big Data applications and these applications, depending on their criticality will need to be tested accordingly.

Finally, another measure for quality of test cases is mutation testing. This type of evaluation basically modifies the software under test to see if the test cases still cover a mutated software. This is very useful in case a method changes frequently, is used in a lot of parts of the application and thus is an important piece. I found it rather useful for the utility functions of the HadoopCryptoledger library that parse binary blockchain data to get some meaningful information of it (cf. based on pitest). In this way you can cover more of the software with higher quality test cases.

7. Code documentation

Documentation of code is important. Of course, ideally the code is self-documenting, but with comments one can easier navigate in code and gets the key concepts of the underlying software. Everybody who has some more in-depth development exposure to popular big data platforms appreciates the documentation (but also of course the source code) shipped with it.

Additionally, one can automatically generate a manual (e.g. by using Javadoc or Scaladoc) that is very useful for developers to leverage the functionality of your library (e.g. find an example of the HadoopCryptoLedger library here). For the aforementioned libraries the documentation is automatically generated and stored in the version management branch “gh-pages”. From there Github pages takes the documentation and makes them available as web pages. Then, it can be found by search engines and viewed in the browser by the users.

The latter point is very important. Usually the problem is not that the developers do not write documentation, but publishing the documentation in time. The documentation for the aforementioned library is published every build in the most recent version. Usually in company it is not the case, it is a manual process that is infrequently or not done at all. However, solutions exist here, for example Confluence and the Docs plugin or the Jenkins Javadoc plugin, but also many others exist.

Furthermore, one should look beyond standard programing language documentation. Nowadays you can integrate UML diagrams (example) and markdown language (example) in your source code comments, which means that your technical specification can be updated every build eg at least once a day!

Finally, tools such as Sonarqube or Openhub can provide you information on the percentage of documentation in comparison to other software.

8. Code Analytics: Integrate static code analysis into the build process.

Besides testing, static code analysis can illustrate areas of improvements for your code. From my experience, if you start right away with it then it is a useful tool with very few false positive.

Even if you have a project that has grown over 10 years and did not start with code analysis, it can be a useful tool, because it shows also for newly added or modified code the quality and this one you can improve right away. Of course code that has never been touched and seems to work already since many years should be carefully evaluated if it makes sense to adapt to recommendation by a static code analysis tool.

Static code analysis is very useful in many areas, such as security or maintainability. It can show you code duplication, which should be avoided as much as possible. However, some code may always have some duplications. For example, for the aforementioned libraries two Hadoop APIs are supported that do essentially the same thing: mapred and mapreduce. Nevertheless, some Big Data tools, such as Hive, Flink or Spark use one API, but not the other one. Since, they are very similar some duplications exist and have to maintained in any case. Nevertheless, in most of the cases duplications should be avoided and remvoed.

One interesting aspect is that popular static code analysis tools in the cloud, such as Codacy, SourceClear or Sonarqube shows you also the quality or even better the evolution of quality of many popular open source projects, so that you can easily evaluate if you want to include them or not.

9. Security & License analysis of third party libraries

Many open source libraries use other open source libraries. The aforementioned libraries are no exception. Since they provide services to Big Data applications, they need to integrate with the libraries of popular Big Data platforms.

Here, it is crucial for the developers and security managers to get an overview of the libraries the application dependent on including the transitive dependencies (dependencies of dependencies). Then, ideally the dependencies and their versions are identified as well as matched to security databases, such as the NIST National Vulnerability Database (NVD).

For the aforementioned libraries the OWASP dependency check is used. After each build the dependencies are checked and problematic ones are highlighted to the developer. The dependency check is free, but it identifies dependencies and matches them to vulnerability based on the library name. This matching is not perfect and requires that you use a build tool with dependency management, such as Gradle, where there are clear names and versions for libaries. Furthermore, the NVD entries are not always of good quality, which makes matching even more challenging.

Commercial tools, such as Jfrog Xray or Nexus IQ have their own databases with enriched security vulnerability information based on various sources (NVD, Github etc.). Matching is done based on hashes and not names. This makes it much more difficult to include a library with wrong version information (or even with manipulated content) in your application.

Additionally, they check the libraries for open source licenses that may have certain legal implications on your organization. Licensing is a critical issue and you should take care that all source code files have a license header (this is the case for the libraries introduced in the beginning).

Finally, they can indicate good versions of a library (e.g. not snapshots, popular by other users, few known security vulnerabilities).

Since the aforementioned libraries introduced in the beginning are just used by Big Data applications, only the free OWASP dependency check is used. Additionally, of course the key important libraries are kept in sync with proper versions. Nevertheless, the libraries itself do not contain other libraries. When a Big Data application is developed with these libraries the developer has to choose the dependent libraries for the Big Data platform.

The development process of Big Data application that use these libraries can employ a more sophisticated process based on their needs or risk exposure.

10. Store build and tested libraries in a versioned manner into an artifact management server (repository)

Since you retrieve already all your libraries in the right version for the artifact management server, you should also make your own libraries or application modules available there.

Our libraries introduced in the beginning are stored on Maven central in a versioned fashion where the developer can pull them in the right version for their platform.

You can also think for your Big Data application to make it available on an on-premise or in the cloud artifact management server (private or publicly). This is very useful for large projects where an application is developed by several software teams and you want to, for example, test integration of new versions of a module of an application and easily switch back if they do not work.

Furthermore, you can use it as part of your continuous deployment process to provide the right versions of the application module for automated deployment up to the production environment.

11. Define a Security policy (

A security policy (example) is a shared understanding between developers and users on security of the software. It should be stored in the root folder of the source code so that it can be easily found/identified.

As a minimum it defines the security reporting process of issues to the developers. For instance, it should be not public – in the beginning – as issues that everyone can see. The reason is that developers and users need to have some time to upgrade to the new software. After some grace period after fixing the issue the issue of course should be published (depending on how you define it in your security policy).

Depending on your software you have different security policies – there is no unique template. However certain elements can be applied to many different types of software, such as security reporting process, usage of credentials, encryption/signing algorithms, ports used by software or anything that might affect security.

Although many open source software has nowadays such a policy it is by far not common or easy to find. Furthermore, there is clearly a lack of standardization and awareness of security policies. Finally, what is completely missing is automatically combining different security policies of the libraries your application depends on to make a more sophisticated security analysis.

12. Define a contribution policy (

Open source libraries live from voluntary contributions of people from all over the world. However, one needs to define a proper process that defines minimum quality expectations of contributions and description on how people can contribute. This is defined in a policy that is stored in the root folder of the source code (example).

This also avoids conflicts if the rules of contribution are clear. Source code repositories, such as Github, show the contribution policy when creating issues or when creating pull requests.

13. Ensure proper amount of logging

Logging is necessary for various purposes. One important purpose is log analytics:

  • Troubleshooting of issues

  • Security analysis

  • Forensics

  • Improving user interactions

  • Proactive proposal for users what to do next based on other users behavior

Finding the right amount of logging is crucial. Too much logging decreases system performance, too little logging gives not the right information. Different configurable log levels supports tuning of this effort, however, usually the standard log levels (error, warn, info, debug etc.) are used and they do not distinguish the use case mentioned before. Even worse, there is usually no standard across an enterprise. Thus they do not take into account the integration of development and operation. For instance, if there is a warning then operation should know if the warning needs to be taken care of immediately or if it has time to prioritize their efforts.

This also implies that there is a need to have standards for logging in open source libraries, which do not exist. As a first step they would need to document their logging approach to get this discussion in the right direction.

Finally, although certain programing languages such as Java with log4j have rather sophisticated logging support this is not necessarily the case for other programing languages, especially high level languages, such as R or Pig. Here you need to define standards and libraries across your enterprise.

14. Transparency about the project activity

This is the last point (in this blog), but also one of the more important ones. Currently, organizations do not collect any information of the open source libraries they are using. This is very risky, because some open source projects may show less and less activity and are abandoned eventually. If an organizations puts a high valuable business process on them then this can cause a serious issue. In fact many applications (even commercial ones) consists nowadays to a large extent of open source libraries.

Additionally, you may want to know how much effort is put into an open source library, how frequently it changes, the degree of documentation etc. Of course, all these should be shown in context with other libraries.

OpenHub is an example for such a platform that exactly does this (find here an example based on one of the aforementioned libraries). Here you can see the activity and the community of a project based on data it pulls out of the source code repository Github. It provides cost estimates (what would it cost to develop the software yourself) and security information.

Several popular open source projects have been scanned by OpenHub and you can track the activity over years to see if it is valuable to include a library or not.


Most of the aspects presented here are not new. They should been part of your organization since several years. Here, we put some emphasis on how open source developers can increase quality and its transparency using free cloud services. This may lead to better open source projects. Furthermore, open source projects have been always the driver for better custom development projects in company by employing best practices in the area of continuous integration and deployment. Hence, the points mentioned here should also serve as a checklist to improve your organizations software development quality.

Software quality increases the transparency on your software product which is the key of successful software development. This itself is a significant benefit, because if you do not know the quality then you do not have quality.

There are a lot of tools supporting this and even if you have to develop unit/integration tests for 100% coverage this usually does not take more than 5-10% of your developers time. The benefits are, however, enormous. Not only that the developer and users better understand the product, they are also more motivated to see the coverage and keep a high standard of quality in the product. If you software provider tells you that unit/integration tests are not worth the effort or that they do not have time for it – get rid of your software provider – there is no excuse. They must make quality information available in real time: This is the standard.

Given the increase of automation and documentation of software products, I expect a significant increase of productivity of developers. Furthermore, given the converge of Big Data platforms, automated assistants (bots) will guide users to develop themselves more easily and quicker high quality analysis depending on their data. Hence, they will be more empowered and less dependent on external developers – they will become developers themselves with the guidance of artificial intelligence doing most of their work to focus more on valuable activities.

Nevertheless, this depends also on their discipline and the employment of scientific methods in their analysis and execution of business processes.

Lambda, Kappa, Microservice and Enterprise Architecture for Big Data

A few years after the emergence of the Lambda-Architecture several new architectures for Big Data have emerged. I will present and illustrate their use case scenarios. These architectures describe IT architectures, but I will describe towards the end of this blog the corresponding Enterprise Architecture artefacts, which are sometimes referred to as Zeta architecture.

Lambda Architecture

I have blogged before about the Lambda-Architecture. Basically this architecture consists of three layers:

  • Batch-Layer: This layer executes long-living batch-processes to do analyses on larger amounts of historical data. The scope is data from several hours to weeks up to years. Here, usually Hadoop MapReduce, Hive, Pig, Spark or Flink are used together with orchestration tools, such as Oozie or Falcon.

  • Speed-Layer/Stream Processing Layer: This layer executes (small/”mini”) batch-processes on data according to a time window (e.g. 1 minute) to do analyses on larger amounts of current data. The scope is data from several seconds up to several hours. Here one may use, for example, Flink, Spark or Storm.

  • Serving Layer: This layer combines the results from the batch and stream processing layer to enable fast interactive analyses by users. This layer leverages usually relational databases, but also NoSQL databases, such as Graph databases (e.g. TitanDB or Neo4J), Document databases (e.g. MongoDB, CouchDB), Column-Databases (e.g. Hbase), Key-Value Stores (e.g. Redis) or Search technologies (e.g. Solr). NoSQL databases provide for certain use cases more adequate and better performing data structures, such as graphs/trees, hash maps or inverse indexes.

In addition, I proposed the long-term storage layer to have an even cheaper storage for data that is hardly accessed, but may be accessed eventually. All layers are supported by a distributed file system, such as HDFS, to store and retrieve data. A core concept is that computation is brought to data (cf. here). On the analysis side, usually standard machine learning algorithms, but also on-line machine learning algorithms, are used.

As you can see, the Lambda-Architecture can be realized using many different software components and combinations thereof.

While the Lambda architecture is a viable approach to tackle Big Data challenges different other architectures have emerged especially to focus only on certain aspects, such as data stream processing, or on integrating it with cloud concepts.

Kappa Architecture

The Kappa Architecture focus solely on data stream processing or “real-time” processing of “live” discrete events. Examples are events emitted by devices from the Internet of Things (IoT), social networks, log files or transaction processing systems. The original motivation was that the Lambda Architecture is too complex if you only need to do event processing.

The following assumptions exists for this architecture:

  • You have a distributed ordered event log persisted to a distributed file system, where stream processing platforms can pick up the events

  • Stream processing platforms can (re-)request events from the event log at any position. This is needed in case of failures or upgrades to the stream processing platform.

  • The event log is potentially large (several Terabytes of data / hour)

  • Mostly online machine learning algorithms are applied due to the constant delivery of new data, which is more relevant than the old already processed data

Technically, the Kappa architecture can be realized using Apache Kafka for managing the data-streams, i.e. providing the distributed ordered event log. Apache Samza enables Kafka to store the event log on HDFS for fault-tolerance and scalability. Examples for stream processing platforms are Apache Flink, Apache Spark Streaming or Apache Storm. The serving layer can in principle use the same technologies as I described for the serving layer in the Lambda Architecture.

There are some pitfalls with the Kappa architecture that you need to be aware of:

  • End to end ordering of events: While technologies, such as Kafka can provide the events in an ordered fashion it relies on the source system that these events are indeed delivered in an ordered fashion. For instance, I had the case that a system in normal operations was sending the events in order, but in case of errors of communication this was not the case, because it stored the events it could not send and retransmitted them at a certain point later. Meanwhile if the communication was established again it send the new events. The source system had to be adapted to handle these situations correctly. Alternatively, you can only ensure a partial ordering using vector clocks or similar implemented at the event log or stream processing level.

  • Delivery paradigms on how the events are delivered (or fetched from) to the stream processing platform

    • At least once: The same event is guaranteed to be delivered once, but the same events might be delivered twice or more due to processing errors or communication/operation errors within Kafka. For instance, the stream processing platform might crash before it can marked events as processed although it has processed them before. This might have undesired side effects, e.g. the same event that “user A liked website W” is counted several times.

    • At most once: The event will be delivered at most once (this is the default Kafka behavior). However, it might also get lost and not be delivered. This could have undesired side effects, e.g. the event “user A liked website W” is not taken into account.

    • Once and only once: The event is guaranteed to be delivered once and only once. This means it will not get lost or delivered twice or more times. However, this is not simply a combination of the above scenarios. Technically you need to make sure in a multi-threaded distributed environment that an event is processed exactly once. This means the same event needs to be (1) only be processed by one sequential process in the stream processing platforms (2) all other processes related to the events need to be made aware that one of them already processes the event. Both features can be implemented using distributed system techniques, such as semaphores or monitors. They can be realized using distributed cache systems, such as Ignite, Redis or to a limited extent ZooKeeper. Another simple possibility would be a relational database, but this would quickly not scale with large volumes.
      • Needles to say: The source system must also make sure that it delivers the same event once and only once to the ordered event log.

  • Online machine learning algorithms constantly change the underlying model to adapt it to new data. This model is used by other applications to make predictions (e.g. predicting when a machine has to go into maintenance). This also means that in case of failure we may temporary have an outdated or simply a wrong model (e.g. in case of at least once or at most once delivery). Hence, the applications need to incorporate some business logic to handle this (e.g do not register a machine twice for maintenance or avoid permanently registering/unregistering a machine for maintenance)

Although technologies, such as Kafka can help you with this, it requires a lot of thinking and experienced developers as well as architects to implement such a solution. The batch-processing layer of the Lambda architecture can somehow mitigate the aforementioned pitfalls, but it can be also affected by them.

Last but not least, although the Kappa Architecture seems to be intriguing due to its simplification in comparison to the Lambda architecture, not everything can be conceptualized as events. For example, company balance sheets, end of the month reports, quarterly publications etc. should not be forced to be represented as events.

Microservice Architecture for Big Data

The Microservice Architecture did not originate in the Big Data movement, but is slowly picked up by it. It is not a precisely defined style, but several common aspects exist. Basically it is driven by the following technology advancements:

  • The implementation of applications as services instead of big monoliths
  • The emergence of software containers to deploy each of those services in isolation of each other. Isolation means that they are put in virtual environments sharing the same operating systems (i.e. they are NOT in different virtual machines), they are connected to each other via virtualized networks and virtualized storage. These containers leverage much better the available resources than virtual machines.
    • Additionally the definition of repositories for software containers, such as the Docker registry, to quickly version, deploy, upgrade dependent containers and test upgraded containers.
  • The deployment of container operating systems, such as CoreOS, Kubernetes or Apache Mesos, to efficiently manage software containers, manage their resources, schedule them to physical hosts and dynamically scale applications according to needs.
  • The development of object stores, such as OpenStack Swift, Amazon S3 or Google Cloud Storage. These object stores are needed to store data beyond the lifecycle of a software container in a highly dynamic cloud or scaling on-premise environment.
  • The DevOps paradigm – especially the implementation of continuous integration and delivery processes with automated testing and static code analysis to improve software quality. This also includes quick deliveries of individual services at any time independently of each other into production.

An example of the Microservice architecture is the Amazon Lambda platform (not to be confused with Lambda architecture) and related services provided by Amazon AWS.

Nevertheless, the Microservice Architecture poses some challenges for Big Data architectures:

  • What should be a service: For instance, you have Apache Spark or Apache Flink that form a cluster to run your application. Should you have for each application on them a dedicated cluster out of software container or should you provide a shared cluster of software containers. It can make sense to have the first solution, e.g. a dedicated cluster per application due to different scaling and performance needs of the application.
  • The usage of object stores. Object stores are needed as a large scale dynamically scalable storage that is shared among containers. However, currently there are some issues, such as performance and consistency models (“eventually consistent”). Here, the paradigm of “Bring Computation to Data” (cf. here) is violated. Nevertheless, this can be mitigated either by using HDFS as a temporal file system on the containers and fetching the data beforehand from the object store or use an in-memory caching solution, such as provided by Apache Ignite or to some extend Apache Spark or Apache Flink.

I see that in these environments the role of software defined networking (SDN) will become crucial not only in cloud data centers, but also on-premise data centers. SDN (which should NOT be confused with virtualized networks) enables centrally controlled intelligent routing of network flows as it is needed in dynamically scaling platforms as required by the Microservice architecture. The old decentralized definition of the network, e.g. in form of decentralized routing, does simply not scale here to enable optimal performance.


I presented here several architectures for Big Data that emerged recently. Although they are based on technologies that are already several years old, I observe that many organizations are overwhelmed with these new technologies and have some issues to adapt and fully leverage them. This has several reasons.

One tool to manage this could be a proper Enterprise Architecture Management. While there are many benefits of Enterprise Architecture Management, I want to highlight the benefit of managed of managed evolution. This paradigm enables to align business and IT, although there is a constant independent (and dependent) change of both with not necessarily aligned goals as illustrated in the following picture.


As you can see from the picture both are constantly diverging and Enterprise Architecture Management is needed to unite them again.

However, reaching managed evolution of Enteprise Architecture requires usually many years and business as well as IT commitment to it. Enterprise Architecture for Big Data is a relatively new concept, which is still subject to change. Nevertheless some common concepts can be identifed. Some people refer to Enterprise Architecture for Big Data also as Zeta Architecture and it does not only encompass Big Data processing, but in context of Microservice architecture also web servers providing the user interface for analytics (e.g. Apache Zeppelin) and further management workflows, such as backup or configuration, deployed in form of containers.

This enterprise architecture for Big Data describes some integrated patterns for Big Data and Microservices so that you can consistently document and implement your Lambda, Kappa, Microservice architecture or a mixture of them. Examples for artefacts of such an enterprise architecture are (cf. also here):

  • Global Resource Management to manage the physical and virtualized resources as well as scaling them (e.g. Apache Mesos and Software Defined Networking)

  • Container Management to deploy and isolate containers (e.g. Apache Mesos)

  • Execution engines to manage different processing engines, such as Hadoop MapReduce, Apache Spark , Apache Flink or Oozie

  • Storage Management to provide Object Storage (e.g. Openstack Swift), Cache Storage (e.g. Ignite HDFS Cache), Distributed Filesystem (e.g. HDFS) and Distributed Ordered Event Log (e.g. Kafka)

  • Solution architecture for one or more services that address one or more business problems. It should be separated from the enterprise architecture, which is focusing more on the strategic global picture. It can articulate a Lambda, Kappa, Microservice architecture or mixture of them.

  • Enterprise applications describe a set of services (including user interfaces)/containers to solve a business problem including appropriate patterns for Polyglot Persistence to provide the right data structure, such as graph, columnar or hash map, for enabling interactive and fast analytics for the users based on SQL and NoSQL databases (see above)

  • Continuous Delivery that describe how Enterprise applications are delivered to production ensuring the quality of them (e.g. Jenkins, Sonarqube, Gradle, Puppet etc).