New research into data volumes and workloads highlights the challenges firms are facing as regards the management and analysis of ever-increasing volumes of data.

Independent research commissioned by KX highlights how businesses across all industry sectors have seen a significant increase in the volume, velocity and variety of data across 2020 – driven by both the ongoing pace of digital transformation across all sectors and the additional challenges of the global pandemic. 

Firms in the manufacturing sector however appear to have been impacted more than most with over 50% reporting a significant increase in multiple data types including eCommerce, customer and data from sensors.

We know that the world of Smart Manufacturing and IoT is grappling with the enormous challenges of fast data and automation required in today’s changing environment. To help with these dual challenges, manufacturing firms reported significant increases in investment in the following technologies: 

  • Cloud computing (50%  of firms significantly increased investment across 2020)
  • Data analytics (45% of firms Significantly increased investment across 2020)
  • Automation (43% of firms Significantly increased investment across 2020)
  • Machine learning and AI (42% of firms significantly increased investment across 2020)

It’s clear from our research that manufacturers understand that faster access to better quality data is critical in helping them remain competitive by improving processes and lowering costs, the more competitive they can be. By implementing a process of ‘Continuous Intelligence’ – where real-time and historical data are continuously combined for real-time analysis – manufacturers can benefit from rapid, accurate, machine-driven decision making.

But challenges exist in getting technology and processes aligned to enable the operational shift to continuous intelligence and as a result, many organizations are failing to exploit the benefits real-time data analytics can bring.

Legacy technologies

Ingesting data fast enough to support analysis and decision making is a critical requirement for continuous intelligence. Legacy technologies however can be pushed to their limit. For example, where in the past periodic samples were taken to detect quality issues, manufacturers are now looking to monitor processes and tools 24×7, Simply being able to capture and store measurement data is a sizeable task in itself. Add to this the analysis piece and existing systems soon struggle.

Another challenge around legacy technologies is access to data. One of the largest complaints and challenge from engineers, data scientists and technicians is getting access to the raw unfiltered data from tools. Legacy tools place restrictions on data access so as not to impact operations of the system, and if access is provided, it is often many hours to days after the fact. Again, for an operating model of continuous intelligence to be successful, access to data must be near instantaneous and constant.

Analysis and insights at the edge

Once the data is collected, the challenge is deriving meaningful value to detect anomalies, make predictions, and make recommendations for improving real-time operations, quality, etc. Even if an organization has the data, if analytics on the data take many hours to days to execute, then the analytics are likely not to be performed or performed only on a small sample. 

With manufacturing processing accelerating and the need to make decisions in milliseconds – such as determining whether a part has to be rejected – making assessments and taking action has to be done at the speed of the process. At those speeds, data processing and analytics has to occur close to where the action is, often at the edge. Moving data to the cloud and back adds too much latency to the process and presents risks to production facilities if access to the internet were disrupted. 

Finding the right analytics platform

Given the challenges outlined above, making the right choice in real time analytics partner is critical. While existing technologies for data capture, management and analysis may present problems in themselves, it’s unlikely that many firms will rip and replace their data software stack. Therefore any real time analytics solution must come with the interoperability and ease of use required to be integrated swiftly and easily into existing systems.

One of the key benefits of is that KX has been designed to address this challenge. KX is a single integrated software platform that can be work with almost any existing software architecture. Moreover, it can be deployed at the edge, close to where data is generated near the tool or in the field, as well as on-premises and in cloud infrastructure to support workloads and use cases where latency matters, while simultaneously providing real-time consolidated views across assets, processes, factories and locations.

KX can deliver improvements to critical processes and systems at the levels of performance, scalability, and fault resilience demanded in today’s world of 24×7 automated operations for delivering fast insights and actions. Moreover, KX can deliver them with no loss of data in extreme conditions and low cost of ownership.

We have customers in a diverse range of industry sectors that are using KX to make considerable savings operationally while also realising significant commercial benefits.

One global leader in materials engineering solutions produces virtually every new chip and advanced display in the world. It saves over 10,000 engineering hours a year by dramatically reducing the time it takes to query real time and historic data. Not only that but the firm also significantly reduced its hardware footprint due to the incredibly efficient design and operation of KX.

In Formula 1 motorsports, we’re helping a number of racing teams improve designs of their vehicles and ultimately be more successful on the track with real-time analysis. Data captured during a race is dynamically compared to real-time wind tunnel data allowing engineers and designers to make adjustments during the testing and analysis sessions rather than having to wait until the testing is over and the data prepared for analysis offline. This is a game changer for teams who now have the capabilities to constantly adapt their design and set-up to respond to the data they’re seeing in real-time.

These are just two example of how real-time analytics enables continuous operational intelligence and the transformative benefits that can be realised. More can be found on our website.

Experts at KX have also created a free eBook called “The Microsecond Mindset” that provides a five-step guide to approaching real-time and streaming analytics with a view to thinking and acting faster. It’s available to download here.