|
Data mining looks for connections in data. Although it sounds scientific – and it largely is, data mining is also an art. Data mining is a method whereby a large volume of data is processed through one or more specialist algorithms to look for connections in the data. So far, so scientific. The art comes in choosing the correct algorithms to use, and in interpreting the results. Algorithms go by exotic names such as naïve Beyes, sequence clustering, neural nets, and choosing the correct ones, with appropriate parameters is a real challenge, but one which needs to be understood to get meaningful results; but this is only the beginning!
Data mining is a very powerful technique when used correctly, helping to discover previously unknown connections in the data. However, and this is really important, just because an algorithm says two pieces of data are related the relationship is not necessarily meaningful or useful. The most important phrase to remember is that correlation does not imply causation – that is if data mining shows two pieces of data to be statistically correlated it does not mean that they are definitively related.
The real art in data mining is to determine which relationships are meaningful, and then from those which are useful, that can be used to change behaviour: increase sales, predict faults, forecast production requirements and so on.
The difference between data mining and classical Business Intelligence is that in Business Intelligence someone suggests a relationship in the data, which is then investigated – ‘I think margins are lower in the South East’, ‘I think product X has a higher rate of quality problems’ – and the proposer of the issue or an analyst will slice and dice the data to determine if the hypothesis has any basis in fact.
Data mining works the other way round, the algorithms suggest the possible issues, and it is then down to the analyst or business person to see if there is a genuine cause, and then if so what to do about it.
Altimus worked with a set of our customer’s data looking for patterns. We discovered that there was a small but statistically significant drop in productivity at certain times, not necessarily the same week on week, but often Saturday afternoons. A bit of digging into the cause showed that the productivity drops coincided with the radio commentaries of the local football team. In this case once the cause was understood it was possible to make an informed decision. To do nothing in this case was right, since the impact on morale of stopping it would have outweighed any potential gains. What data mining did in this circumstance was show a relationship, led to a cause being determined and appropriate action taken.
Another example was an analysis we did on comparing performance of operators and the machines they operated. We discovered that there was no significant difference in the performance of a machine over time, or of an operator over time. However when the two factors were combined we discovered that operator 1 on machine A was more efficient than operator 1 on machine B; and operator 2 on machine B was more efficient than operator 2 on machine A. The machines were outwardly identical, the workers were excellent at their jobs, yet certain combinations worked better. In this case no underlying cause could be found, despite investigation. Yet experiments confirmed the findings, so job assignments are made on this basis, yielding potential efficiency gains estimated to be worth more than £50k.
Finally we’ll give an example of bad use of data mining. It can be shown (see graph below) that there is a statistical correlation between the number of pirates and global warming. So to solve global warming all we need to do is get more pirates. A clear fallacy, but it does illustrate the dangers of trusting data without a clear and logical understanding of the underlying business leading to a critical assessment of why there is a link and what to do about it.
To find out more about data mining, call Altimus on 0800 804 6442 and speak to one of our experts, or contact us through Information Request. |