Splunk and Eureqa; IOT Manufacturing Demo


I have recently been working with the Splunk universal machine data platform. It is a very interesting and useful technology with a wide range of application. I have also been working more with Eureqa and decided to explore how the two technologies could be used together. I decided to use a hypothetical Internet of Things manufacturing scenario to setup a demo using Splunk and Eureqa.

This is a simplified version of a scenario becoming more common in Manufacturing and other industries today. The idea is data is collected about various aspects of the manufacturing process as well as completing quality testing at various stages of the process. The results of the quality testing is used in conjunction with the data collected during the manufacturing process as a training data set to build predictive algorithms/models. These models are then used in real time during the manufacturing process to measure quality and identify defects. This allows for the real time actions to correct quality issues or to stop the manufacturing process to reduce costs incurred by defective product.

Splunk and Eureqa fit in this process like peanut butter and jelly. Splunk is a universal machine data platform that will literally ingest any type of data. It indexes the data and provides a search language to produce valuable information in real time. Eureqa is a robotic data scientist and can find meaningful predictive models automatically from your data. These tools can be used to easily solve the real time IOT manufacturing quality use case. Here is the demonstration scenario:

  • Chemical Manufacturing – (PolyVinylHydroSilica)
  • Problem: Manufacturing data is not timely enough
    • Decisions cannot be made during manufacturing process
    • Data reporting is only looking historically and lags 24 to 48 hours
  • Objective: Improve quality and meet the following goals
    • Improve data collection on manufacturing process
    • Utilize data to accurately predict and identify failures
    • Identify failures as early as possible to reduce costs
  • Identified high level requirements
    • Track manufacturing process data
    • Identify quality issues quickly to reduce costs
    • Provide near real time reporting


Legacy Process Flow

Legacy Process


Updated Process Flow with Splunk and Eureqa

Updated Process

Output Details and Operation

The initial benefit of this process and the low hanging fruit that will allow for a positive return on investment is the savings from halting production early on lots identified as inferior product. The second valuable benefit is the real time reporting of manufacturing process metrics. Splunk has real time reporting capabilities which are used to drive dashboards with valuable information. Here is an example below.

Splunk Dashboard

The above dashboard shows key performance indicators on the manufacturing process such as; speed, volume, and cost of manufacturing as well as the cost saved by stopping failed lots. This information is driven by the manufacturing data from each phase as well as the model results from Eureqa. The models in Eureqa are generated from the training data using the search functionality. The screenshot below shows the candidate models produced from the search.

Eureqa Models

We select models for each phase based on variables included and the performance. Also note Eureqa suggests the best model based on best tradeoff of error vs complexity. I chose to use model 2 for phase I, model 5 for phase II and model 11 for phase III. Phase I is more tolerant and only requires a value of 10 or above to be moved to the next phase. Phase II requires a value of 20 or above to be along to phase III. Finally phase III requires a value of 30 or above to be a passing lot. To demonstrate this process I wrote an application to simulate the manufacturing process. This simulation environment is shown in the diagram below.

Demo Environment

The demo for this process is shown below. I also talk about the various integration points between Splunk and Eureqa and show a couple of examples.