SAS Institute and Intel have asked more than 300 IT-managers from the largest companies in Denmark, Finland, Norway and Sweden about the adoption of Big Data analytics and Hadoop. This report presents conclusions on key findings and some future expectations.
Big Data has become a hyped and misused concept, giving rise to confusion among people. At a high level, however, we are talking about the three Vs (volume, variety and velocity) in general.
This survey found that about 50 percent of the three hundred companies polled do not have a data infrastructure capable of meeting the demands tied to new types of data. Are you one of them?
The three V’s:
Volume: many factors contribute to the increase in data volume; transaction-based data stored through the years, unstructured data streaming in from eg. text, images, video, logs and increasing quantities of sensor and machine-to-machine data.
Velocity: data is streaming in at an unprecedented speed, and must be dealt with in a timely manner. Reacting quickly enough to deal with data velocity is a challenge for most organizations.
Variety: managing, merging and governing different varieties of data, both structured and unstructured, is something many organizations still grapple with.
The interesting part is the new insight obtained from analysing these data. In the past, organisations were constrained in how much data could be stored and what types of analytics could be applied to them. Analysts were often limited to analysing just a sample subset of the data in an attempt to simulate a larger data population, even when using all the information would have yielded a more accurate result.
That little elephant icon is to be seen all over the place. The distributed open-source framework has caught the attention of many organisations searching for better ways to store and process large volumes and varieties of data. From executives and business analysts to data stewards, information scientists and analytic professionals, Hadoop seems to be the most popular kid on the block.
Hadoop delivers enormous processing power – the ability to handle virtually limitless concurrent tasks and jobs. That makes it a remarkably low-cost complement to a traditional enterprise data infrastructure. It can overcome the bandwidth and coordination issues associated with processing billions of records that previously might not have been saved. Hadoop handles Big Data. It does it fast. It redefines the possible when it comes to analyzing large volumes of data, particularly semi-structured and unstructured data (text).
Hadoop distribution vendors have invested a great deal making it mature and enterprise-ready, and a proven data platform is now available. In many cases, however, information is valueless unless it can be used in a business context. We at SAS Institute have seen a big interest from companies wanting to process and analyse these new kinds of Hadoop data.
Watch this video to learn the basics about Hadoop and analytics:
Intel (NASDAQ: INTC) is a world leader in computing innovation. The company designs and builds the essential technologies that serve as the foundation for the world’s computing devices. As a leader in corporate responsibility and sustainability, Intel also manufactures the world’s first commercially available “conflict-free” microprocessors.
SAS is the leader in business analytics software and services, and the largest independent vendor in the business intelligence market. Through innovative solutions, SAS helps customers at more than 70,000 sites improve performance and deliver value by making better decisions faster. Since 1976 SAS has been giving customers around the world THE POWER TO KNOW®.
Do you want to learn about how to get set up with Hadoop? Watch this video: Predictive Analytics & Real-time Processing with the Intel Xeon Processor E7 v2 and read this Enterprise Analytics with Hadoop and Data Preparation and Deployment with Hadoop
This survey has been conducted through a collaboration between SAS Institute, Intel and the research company Norstat. IT managers from the biggest companies in the Nordic region were interviewed on the following subjects:
More than 300 Nordic companies responded through phone interviews (Norstat) and an online questionnaire (SAS Institute).
When asked whether the respondent’s company had an increasing need to collect and analyse data, more than eight in ten responded positively. Norway had the lowest score, at 75 percent, and Finland the highest at 95 percent.
This comes as no surprise to SAS Institute. These numbers show that Nordic businesses have understood the advantage of data analytics and the power of knowledge.
Do you see an increasing need to collect and analyse more data in your company?
Data has no value unless companies begin to do something clever with them. In many cases, applying analytics in a timely manner will allow more companies to explore new ways of transforming their business model, to create new products and services, to increase their operational efficiency and to improve their understanding of their customer’s behaviour and how to enhance the customer experience.
One of the biggest challenges companies face with Big Data, is unstructured information. When people think about “data”, they normally imagine a database whose content resides in a well-defined structure. That is not the case with unstructured data. By definition, no structure, data model or even organisation exists. The information resides in many locations (eg, e-mail, files, websites, etc) and is normally widely dispersed across an organisation.
When asked about their requirements for collecting new types of data (such as unstructured), almost two in three of the organisations needed to store new data that cannot be held in traditional databases and systems.
Do you see a need to collect new types of data (such as unstructured) that cannot be stored in traditional databases and systems?
With new data comes the opportunity to acquire new knowledge and intelligence. Access to the unstructured data uncovers innovative ways to find competitive advantage by getting to know how your customers behave, your own operational excellence or the way your money flows. We asked respondents if they thought new data used for analytics could give their company a competitive advantage. As many as 92 percent agreed that it could.
Would you agree that more and new data used for analytics could give your company a competitive advantage?
The survey’s first three questions were all related to the importance of Big Data, new data and analytics. When correlating respondents who answered yes to all three questions, it is clear that telecommunications, media and insurance companies placed the greatest value on Big Data and analytics:
How are the largest companies in the Nordic region planning to use Hadoop? When we asked them what they saw as the primary reason for considering Hadoop, almost 60 percent of respondents wanted to use it for advanced analytics, data discovery or as an analytical lab. Many would like to speed up processing. Replacing data warehouse or low cost storage were lower down on the ranking.
What do you see as the primary reason for considering Hadoop?
The results show that data discovery and advanced analytics accounted for 59 percent of the reasons for leveraging Hadoop. All the countries and industries agreed on this. A surprisingly low 13 percent saw Hadoop as low-cost storage. Twenty-two percent of respondents identified speeding up processing as the primary application, with telecommunications, banks and the public sector in the lead. Only five percent saw replacing data warehouse as the primary reason.
Most companies were not expecting to replace their current data warehouse with Hadoop. That applied to all industries except for shipping, where 50 percent wanted to do this.
More than one in three (37 percent) of IT business professionals regard customer and market intelligence as the primary case or scenario for Hadoop in their company. Industries known to be customer- and consumer-centric are more mature in their use of data. Banking, insurance, media, retail and telecommunications view the use of Hadoop as important for acquiring customer intelligence.
Companies in these sectors are in a highly competitive business environment, where the consumer is in the driving seat and will choose another supplier if they fail to receive the attention, products and services they need at the right time. These companies therefore want to process and analyse more information and new types of data in order to respond better to market conditions, and to explore the data for innovation purposes and for staying agile in an ever-changing market.
What do you see as the primary business case or scenario for Hadoop in your company? (by country and industry)
Even though Hadoop could be the solution to some of their Big Data processing challenges, most companies need an infrastructure able to handle such volumes of information.
Almost half of the companies lacked a data centre infrastructure (server/network/storage) capable of meeting the demands of new data types (volume, variety and velocity). Of those without such a capability, two in three needed to upgrade or buy new infrastructure to manage new data. Less than one in three would look to cloud technology to meet these new requirements.
Is your current data centre infrastructure (server/network/storage) capable of meeting the demands placed on it by the new types of data (volume, variety, velocity, etc)?
The value of advanced analytics is highly valuable when data are analysed rapidly. But information volume, its velocity and its unstructured nature, put a big strain on existing data warehouses and software. Only 13 percent of respondents believed that their facilities meet the requirements for timely data analysis. To achieve the speed they needed, 44 percent would have to either upgrade their existing infrastructure or add more computing power. Only 15 percent believed that cloud technology would improve analysis.
Swedish respondents were much more confident of their own ability to analyse data in a timely manner (25 percent), while Norway had the lowest share (13 percent) of respondents who were satisfied with the present situation.
What are you doing to improve timely analysis of your data?
Although the Hadoop technology was launched back in 2011, some uncertainty and confusion still prevails in the market about its use and the opportunities it offers – but many signs indicate that it is on the verge of maturing in the Nordic market. We asked the 300 IT professionals where they were in the Hadoop adoption process, and the results support our view.
Twenty per cent of respondents either had installed or were about to install Hadoop, or would make such a decision in 2015. Finland had as many as 10 percent of the respondents already using Hadoop. The Finns appeared throughout the survey to be in the lead for exploiting Big Data and Hadoop among the Nordic countries.
Hadoop is a widely adopted technology for storage and analysis of this new data. Do you have – or are you considering using – Hadoop as part of your future architecture?
Hadoop is a disruptive technology, that can have a great impact on the way users do business and work with information management. But half the respondents would either not consider Hadoop or only consider it in a longer time frame.
More than one in three of the respondents who would not consider Hadoop said that lack of resources and competence was the biggest challenge. Almost half identified uncertainty about the technology as the primary obstacle.
Although the Hadoop technology has matured a great deal over the past couple of years, companies were not sure what kind of resources and competence they needed in their organisation to manage a Hadoop environment. They seemed to have a perception that Hadoop skills and resources were hard to find and would require extensive staff education.
What could be the primary obstacle (if any) for not considering Hadoop?
Hadoop is now a mature and proven technology that can complement existing enterprise data warehouses (EDWs). Companies have an increasing need to analyse more information and new types of data for different business and IT applications, and to process these data at a higher speed in order to make timely decisions. Hadoop provides the best support for this, as well as a data storage and processing platform.
Among the respondents saying they both see an increasing need for using more data and that analytics on the new data will increase the competitive advantage, more than one of three say they already have, will implement or will make a decision on whether to implement or not during 2015.
Despite that two out of three states that they have a longer time frame on their Hadoop adoption or will not consider it at all, we expect technology strategies to mature over the next 12-18 months and that this picture may change. However it is evident that a large part of the respondents currently also have a complementary strategy to Hadoop on how they approach Big Data analytics.
Some Nordic companies still have improvement areas on their use of analytics in the more traditional areas utilizing structured data and where it is necessary to mature on those areas, prior starting a Big Data analytical journey.
Analyst firms like Gartner and IDC expects the adoption of Hadoop to mature in the Nordic region during 2015 and the years beyond. We also expect the need for data management and analytics with Hadoop data (in combination with existing warehouse data) to grow. Companies need to get an insight into the data they possess in order to take action which can realise good return on investment (ROIs) on their applications.
Companies across different industries will soon find more and more business applications for Hadoop, which will drive adoption, and they will quickly discover that skills and resources might not be an obstacle.
Companies which have started the Big Data journey in order to analyse more information and new types of data and thereby discover new patterns, trends and insights, will reach a point where decisions have to be made on architecture, data, analytics, people and applications.
In this discussion, Hadoop and potential alternatives are being considered for data storage and processing. People looking to adopt Hadoop ask themselves “Where do I begin?” and “How do I get the insight?”.
As a market leader in analytics, SAS Institute has a clear strategy of helping companies to process and analyse the data held inside Hadoop. We have a growing number of customer assignments where our data management and analytic capabilities are utilized as important elements in supporting Hadoop applications – without increasing the complexity. Based on this experience, we can advise companies on how to get started, and on the architecture and analytic capabilities required to deliver a certain application.
As more and more companies plan to expand their current Hadoop environment to cover new use cases, we can advise on the next steps in their Big Data journey – reusing or building on previous investment.
Contact SAS Institute
Director, Nordic Center of Excellence
Telephone: +47 92026377
Head of Strategy & Architecture, Nordic Center of Excellence
Telephone: +45 70282830
Alan Priestley, Intel GmbH
Cloud & Big Data Analytics Solutions Director