Success Factors of Big Data to Achieve Organisational Performance: Qualitative Research

KeywordsBig Data Organisational Performance success factors value

JEL Classification O30, L26

Full Article

Success Factors of Big Data to Achieve Organisational Performance: Qualitative Research

1. Introduction

As the subsequent part of the paper, this part continues the preceding literature review (presented in the previous paper of this journal, namely Success Factors of Big Data to Achieve Organisational Performance: Theoretical Perspectives), addresses the previously listed research questions, and contains the empirical findings.

2. Research Methodology

This paper outlines the chosen research methodology chosen for this research study. Information attained from the literature review, outlined in the previous paper of this journal, namely Success Factors of Big Data to Achieve Organisational Performance: Theoretical Perspectives, was used as the basis for the chosen methodology as well as the interview guide used during the semi-structured interviews. The approach adopted was exploratory in nature with the research method, research design, sampling and analysis supporting the qualitative approach.

2.1. Choice of Methodology

Philosophy and Approach

The research philosophy contains important assumptions about how the world in which the project is being conducted is viewed and this has an important impact on how we interpret what is being investigated (Saunders, Lewis, and Thornhill, 2009). The research philosophy adopted for this study was interpretivism because it is important to understand what was occurring in the BDAC context within the organisations and since interpretivism relates to the study of phenomena in their natural environment, this approach provided the right mindset for this understanding to occur (Saunders and Lewis, 2012). Theories described in the previous paper of this journal, namely Success Factors of Big Data to Achieve Organisational Performance: Theoretical Perspectives were used as a lens through which to study the effects of BDAC on organisational performance in each of the companies. Limitations of interpretivism are subjectivity and research legitimisation. Since interpretivism lends itself to the main influence being the researchers particular view of the context, subjectivity and researcher perception bias could have influenced the outcome of the study (Saunders and Lewis, 2012). Research legitimisation is compromised in the validity, reliability and generalisability of the research due to interpretivism often providing an outcome too unique to the context (Kelliher, 2011).

The research approach was a combined approach between deduction and induction. This approach is justified through Saunders and Lewis (2012) and Bloomberg and Volpe (2012) who both state that a combined approach in the same paper can be a good idea. Initially a deductive approach was used to identify themes within the literature that emerged within the BDAC space. These themes were then used to determine context for interpreting the effectiveness of BDAC on organisational performance. Relevant theory with which to compare these capabilities was also identified from the literature review. This context was used to provide an outline to the semi-structured interviews as well as a framework for interpretation of the collected data from these interviews. The inductive approach began when new constructs occurred based on observations during the interviews. These new constructs were used to improve or adapt the initial theory framework in order to develop a model to be tested in future research. This combined approach is consistent with the literature being in its early stages with little commonality and consensus amongst researchers. Further development and convergence of theory was required through the combined deductive and inductive approach. The advantage of using the deductive approach first is to provide some structure to the results to ensure relevance, and alignment to current literature trends. A disadvantage is that outcomes outside of the framework may have been missed if too much emphasis was placed on the known with insufficient room for unknowns to have surfaced during the interviews. An awareness not to lead the interviewees into the defined framework was made.

2.1.1. Type of Study and Strategy

Information was collected at a specific point in time over a three week interview period, leading to a cross-sectional study having occurred (Saunders, Lewis, and Thornhill, 2009). This was appropriate as the research was aimed at understanding what organisations are doing now to be successful with BDAC initiatives and was not to measure how the organisations actually perform over time. Interpretivism interprets social roles to attain knowledge rather than objectively determining the findings. This philosophy lent itself well to a more flexible qualitative research method where there was greater personal interaction to attain more meaningful and in-depth insights (Carson, Gilmore, Perry, and Gronhaug, 2001). Qualitative research does not require the use of numerical techniques to answer proposed research objectives and therefore left room for interpretations and elaborations (Zikmund, Babin, Carr, and Griffin, 2013). The topic was allowed to naturally develop through the process without a predefined path (Patton, 2002).

Qualitative research allowed for greater insights and richer content to be generated since more interaction with the interviewees was accomplished (Carson et al., 2001). This allowed probing and follow up questions to be asked, facilitating a conversation rather than a one-way quantitative interaction. The emphasis of qualitative research was on exploration and discovery (Bloomberg and Volpe, 2012). The aim of the research was to determine what the organisations are doing themselves with regards to BDAC and this required a more open-ended qualitative approach as quantitative approaches have a fixed set of variables and further exploration outside those predetermined variables cannot take place (Guba, Lincoln, and others, 1994). The literature on the effects of BDAC on organisational performance was scarce and in its early stages of development. This situation was particularly suited to qualitative methods as the increased depth in knowledge attained can add to and help to shape the growing and inconsistent literature. There has been a call for more qualitative research to take place within the BDAC literature in order to explore what organisations do practically with regards to strategic management and implementation of big data initiatives in order to achieve improved organisational performance (Vidgen et al., 2017; Y. Wang and Byrd, 2017) and this methodology was therefore able to answer this call and began to fill this gap in the BDAC literature.

An exploratory study type took place as it discovered general information about the BDAC topic not yet known in the literature and gained necessary insights to provide answers to the initial research propositions (Saunders and Lewis, 2012). Saunders and Lewis (2012) and Zikmund (2000) explain that exploratory research is required when there is a relatively new and unexplained area in the literature, which was the case in this study. Interviewing is a well-known qualitative method used within exploratory studies (Saunders, Lewis, and Thornhill, 2009). A semi-structured interview strategy was chosen because this enabled an exploration of BDAC more deeply and was suited to the scenario of wanting a contextual understanding of the current reality of organisations (Vidgen, 2014). This allowed for deeper insights into the emerging BDAC research and it was therefore widely applicable in research conducted with an interpretivist philosophy (Y. Wang, Kung, Wang, et al., 2018). Literature precedent for qualitative, exploratory, semi-structured interviews is provided by Liu et al. (2017); Noblet et al. (2011) and Ram, Zhang, and Koronios (2016) within the BDAC literature. Respondents were a combination of big data consultants and specialists involved in the BDAC projects or strategic processes. Interviewees were purposefully chosen based on their expertise, roles or familiarity with BDAC process and outcomes. Liu et al. (2017) explain that purposive sampling is encouraged in this situation.

2.2. Proposed Research Methodology and Design

2.2.1. Target population

A population is defined by Saunders and Lewis (2012) as the complete set of cases/group members from which the sample can be taken. Zikmund (2003) defines a population as “a complete group of entities sharing some common set of characteristics”. The target population from which the interviewees were chosen was defined as big data consultants and specialists working with big data or on big data projects within South African organisations. But, Zikmund (2003) goes further with the definition to define the target population as “the complete group of specific population elements relevant to the research problem”. The target population was therefore big data consultants and specialists or directors and managers working with big data or on big data projects within South African organisations that have access to sufficient volume and variety of data to require BDAC rather than traditional analytical methods. Assessment of the large volume of data access was done by looking at organisational size and customer base.

2.2.2. Unit of analysis

The unit of analysis was the big data professionals from owners and managing directors to data scientists and architects. Organisational capabilities and their effect on organisational firm performance was being investigated through the semi-structured interviews and the individual opinions of these professionals provided the necessary insights required to obtain the objective as outlined in the previous paper of this journal, namely Success Factors of Big Data to Achieve Organisational Performance: Theoretical Perspectives.

Sampling method and size

The sample was selected using non-probability purposive (judgement) sampling in order to have purposefully selected the professionals at various organisations that were able to provide the necessary insight while being cognisant of remaining representative of the population (Saunders and Lewis 2012; Patton, 2002). In qualitative research, purposive sampling is often utilised. People with the right access and understanding of the big data initiatives and outcomes were required to gain the necessary insights from the interviews. Criteria for the big data professionals were; level of experience must be a minimum of three years working in the big data field, and the magnitude and complexity of the projects must constitute big data and be a significant project in the organisation. The complexity was such that it required big data analytics that traditional analytics tools could not achieve.

According to Patton (2002), the size of the sample should be applicable to the context of the study and is therefore determined on a per case basis. Based on previous literature precedent, the sample of individuals was ideally determined based on theoretical saturation with an aim of at least fifteen interviews. Sixteen interviews were conducted over a period of three weeks consisting of Heads of Departments, Big Data Project Management Office Executives, Big Data Managers, Data Scientists, Big Data Specialists, Big Data Strategists, Big Data Analytics Architects, Big Data Consultancy Owners and Team Leads. Each participant had a minimum of three years’ experience with data analysis and only three of the interviewees were female.

It must be noted that industry was not a criterion for selection and sampling was not done according to any particular industry. Participants were sourced through big data industry leaders, consultancies and social media, such as LinkedIn, which contained groups for big data specialists and professionals with many members from South Africa and Gauteng. Interviewees were from the following industries; Insurance, Banking, Telecommunications, IT and Consulting (see Table 1).

Table 1. Industry and Position of Interviewee Sample

2.2.3. Measurement Instrument

Saunders and Lewis (2012) suggest that one of the best methods for conducting exploratory research is with the use of in-depth interviews after a thorough academic literature review. The chosen measurement instrument was therefore semi-structured interviews where there was a list of pre-determined themes but the order of the themes and questions covered varied. During the interview additional questions were asked and some were also left out if it made sense to do so (Saunders and Lewis, 2012). Flick (2014) points out the advantage to semi-structured interviews is that the similar question structure helps make analysis of the collected data easier. Face-to-face and telephonic interviews were conducted to cover each of the research questions through a set of pre-prepared questions together with probing. This allowed for a balance between consistency and flexibility.

McCracken, (1988) introduced the systematic and practical concept of the long interview approach, which was used during the interviews. A combination of exploratory questions, planned prompts and unplanned prompts were employed throughout the sixteen interviews. Interviews were between 20 and 70 minutes long but on average the interviews lasted 50 minutes. Interviews took place at the venue most suitable to the participant where nine interviews were telephonic and seven interviews were conducted face-to-face. Face-to-face interviews were either conducted at the participants place of work or within a booked room at the Gordon Institute of Business Science. Interviewees were invited to participate via emails or messages within the linked-in social media platform. The messages contained pertinent information on what the research was about, the time required from them as well as the consent form detailing the voluntary nature of the participation and the lack of identifiers when reporting on the information. Once the individual had agreed to participate, a time and date was discussed and meeting request sent to each individual with the interview guide attached for their perusal prior to the interview.

An interview guide was pre-prepared to ensure the topics of culture, value, core competency, technology and challenges were covered with each participant. These were informed by the literature review and a prior questionnaire by Vidgen (2014) (see Appendix Error: Reference source not found for draft discussion guide with the informed consent front page). The interview questions as presented in the interview guide were used to provide insight into the three research questions outlined in the previous paper of this journal, namely Success Factors of Big Data to Achieve Organisational Performance: Theoretical Perspectives. The interview questions can be mapped to the three research questions (see  below) to ensure consistency from the objective, through the literature and research questions to the analysis and discussion of the findings from the interviews.

Table 2. Mapping of Interview Questions to Research Questions

2.2.4. Ethical Considerations

Ethical Clearance was obtained through the Gordon Institute of Business Science as per the guidelines before participants were contacted. Before the interview began, each individual was again taken through the consent paragraph sent prior to the interview and re-informed of the voluntary nature, reporting without identifiers and recording of the conversation to ensure the data was collected in an ethical manner (Saunders and Lewis, 2012). The front page of the interview guide was also a declaration of confidentiality.

All respondents could converse in English and as such, no translator was required. Also, no information about age or race was recorded as this was not pertinent to the analysis and therefore not necessary to ascertain. Company sensitive information was provided during the interviews and therefore no identifier as to which insights come from which company are given. Interview transcripts have been anonymised to ensure confidentiality of the interviewee and their company. In the report respondents are referred to as interviewee 1 to 16 (see Table 3 below) with only contextual information provided when it was important to add context for the understanding of the interpretation or analysis.

2.2.5. Data collection

Prior to the interview, adequate information on the participant and the company was gathered through publicly available online information to facilitate a smooth interview and prevent time wasting during the interview. Data was collected through semi-structured, in-depth interviews, either face-to-face or telephonically (Saunders and Lewis, 2012; Zikmund et al., 2013). The design of the interview guide was purposefully structured to ensure an open dialogue occurred around key aspects as identified through the literature review process as well as left room for other specific topics to be spoken about by the participants. Adequate preparation occurred to minimise the effects interviewer experience may have on qualitative data collection (Agee, 2009).

The interviews began with a background section to establish rapport and attain some general information about each participant and their job description. Different types of prompts; floating, planned, category question and contrast, as described by McCracken (1988), were used when the interview required. During some of the interviews a common understanding of what Big Data actually meant was established with the participant and further clarification on any of the questions or terms occurred throughout the interviews as necessary. Open-ended, non-leading questions within five different sections; culture, value, core competency, technology and challenges occurred in each interview and each participant was asked to answer based on their current and past experiences with their own perceptions and opinions. The confidential nature of the reporting allowed the participants to answer openly and mention company information freely, which aided in the exploratory methodology design. Each participant was given the chance to add anything they felt was important and had not been covered or to ask further questions at the end of each interview (Saunders and Lewis, 2012). During each interview, notes were taken and all interviews were recorded using the Microsoft voice recorder. Seven face-to-face interviews were conducted in private rooms to avoid disturbances (Saunders and Lewis, 2012) and nine telephonic interviews took place since face-to-face was not possible.

2.2.6. Data analysis

The interviews were transcribed fully from the audio recording and the recordings backed up on an external hard drive. The transcribed interviews were loaded into Atlas.ti, a tool for coding of information. Excel was also utilised in the coding process as well as to group information for thematic analysis. The analysis of the first interviews took place on an ongoing basis throughout the data collection for a more focused approach but most of the analysis took place post the interview process by finding common themes throughout the interviews. A combination of content and thematic analysis was utilised. Content analysis assigned codes to the interview guide constructs and groupings, derived from the literature (Patton, 2002). Searches for common themes within the data took place for the thematic analysis and these emerging themes were coded to look for insights and new information. The thematic analysis followed the procedure outlined by Braun and Clarke (2006) who describe thematic analysis as the process whereby patterns or themes are identified in qualitative data through six main phases; becoming familiar with the data, initial code generation, theme searching, theme reviewing, theme defining and naming, and reporting.

The recordings, transcripts and written notes taken during the interview were consulted many times during the analysis in order to determine the codes and these codes were grouped together to form themes (Zikmund et al., 2013). These themes represented constructs that occurred commonly throughout the interviews and added value towards answering the research questions (Saunders and Lewis, 2012). Analysis took place per question in each of the interviews and frequency analysis determined the number of times each construct was mentioned throughout the sixteen interviews. A list of success factor constructs that can be used in a conceptual model was developed based on the thematic findings.

2.2.7. Reliability and Validity

Saunders and Lewis (2012) define reliability as the extent that methods for data collection and analysis produce consistent findings. This is extended for qualitative research to refer to dependability, confirmability and consistency. Validity is defined as “the extent to which data collection and methods accurately measure what they are intended to and whether or not the findings are really about what they appear” (Saunders and Lewis, 2012). Qualitative research has a subjective nature which could lead to interviewer bias, interpreter bias and respondent bias occurring (Zikmund et al., 2013). Since qualitative research in the interpretivism philosophy relies heavily on your interpretation of what you are observing or hearing, researcher perception bias or preconceptions decrease the reliability of the research findings. Awareness of the potential biases throughout the process helped to minimise their effect as a concerted effort was made to mitigate the biases. Internal validity was improved by conducting pilot studies and ensuring the interview guide was appropriate. After three interviews it was determined that the interview guide was appropriate for the research and these three interviews were then included for analysis and interpretation. Reliability was improved by following the semi-structured interview guide within each interview and ensuring the interviewees understood the questions. Reliability and validity are also improved by no previous relationship being present between the interviewer and any of the interviewees. This ensured no perception bias could occur or inferences based on any previous encounters or discussions on the BDAC topic.

2.2.8. Research Limitations

· Based on the scope of organisations covered during the semi-structured interviews, the results may not be generalisable to other industries or geographies

· The scope of constructs defining the semi-structured questionnaire only covered the literature reviewed and this may have missed the effects of other mediators or moderators of the relationship between BDAC and organisational performance. Integrating more variables may be an avenue for future research

· The use of the framework for the semi-structured interviews may have constrained and biased the data collection and interpretation through interpreter bias

· Interviewer bias may have occurred as interpretation takes place through the lens of the interviewer. Researcher perception bias could have been reduced by following the advice of Bloomberg and Volpe (2012) and ensuring some of the participants evaluated the findings

· Respondent bias could have occurred due to their personal opinions and interpretations, for example, their understanding of what big data actually means as well as their main role in the company not having a holistic view of the company’s BDAC

· Interviewer inexperience could have affected the information collected during the interview process

· The research did not actually measure the improved organisational performance and therefore cannot prove any causal relationship between BDAC constructs and improved organisational performance

3. Results

This section presents the results attained from the one-on-one semi-structured interviews. The results are shown as per the research questions in section 3 to ensure consistency throughout the report.

3.1. Overview of Sample

A total of sixteen semi-structured, one-on-one interviews were conducted over a three-week period from 14 September 2018 to 01 October 2018. A total of 13 and a half hours of recordings occurred, which translated into 178 number of pages transcribed. The average interview time was 50 minutes long. The interviews were thirteen males and three females, all from Gauteng, that spanned thirteen different companies in five different industries; Telecommunications, Banking, Insurance, IT and Consulting (see Table 3 below). Gauteng is the economic hub of South Africa and therefore many more companies are based in Gauteng rather than other provinces. Interviewees were from a range of job levels with varying job focuses and the interviews covered from owners and managing directors to data scientists and software engineers in order to get a broad view on BDAC from multiple perspectives. These industries are generally quite large companies and although there were some smaller consulting companies, large business in Gauteng in South Africa dominated the interviews and will be the most relevant area to benefit from the findings.

Table 3. Summary of Participants and Interviews

Through purposive sampling and using mostly Linked-in, the interviewees were selected based on the criteria set ou. Participants were big data consultants and specialists or directors and managers working with big data or on big data projects within South African organisations that have access to sufficient volume and variety of data to require BDAC rather than traditional analytical methods. The interviewees were also previously unknown to the interviewer and this helped avoid any bias that may have occurred due to familiarity or prior conversations on BDAC. Interpretation of the interview findings are then also more objective as no prior perception based on familiarity of the interviewer can be used to skew the analysis or provide context unsaid in the interview. All respondents were alone when interviewed to ensure candid and honest responses and all interviewees had time before the interview to review the interview guide and prepare for a more informative discussion.

3.2. Presentation of Results

All results are presented per research question as posed in previous section. Under each research question are the presentation of constructs that emerged from the codes generated throughout the data analysis.

3.2.1. Test for Saturation

Figure 1. Graph of the code creation over the course of the data analysis

To test for data saturation, new codes were documented and recorded in the order of the interviews conducted. The number of new codes being generated decreased, in general, as the interviews continued. It was seen that by the 16^th interview no new codes were generated and from the 13^th interview no new codes added any significant constructs and themes to those already formed from the remainder of the interviews. This means that theoretical saturation had been reached and the interviews conducted cover the range of knowledge required for thorough analysis and interpretation.

3.3. Results for Research Question 1

Research Question 1: What is the general understanding of big data and its current situation?

The aim of research question 1 was to identify the perceived understanding and current view of big data. This included the understanding of what big data means, what it adds to the organisation and how companies go about utilising big data. It was also established what organisations view as the main uses and application of big data. It was important to understand this context in order to then know what success factors (research question 2) are required to achieve utilising big data successfully in this way and what challenges (research question 3) are inhibiting organisations from utilising big data successfully.

3.3.1. Big Data Understanding and Applicable View

Understanding

Participants views varied on what big data means, with some participants taking a more academic definition while others a more practical definition. Participants in leadership roles tended to look at big data in terms of what it meant for the business while more technical interviewees saw big data more in terms of the data and technology itself.

Many participants, when asked about their understanding of big data would go straight into tasks and uses of big data rather than talk through a holistic understanding of the concept, showing that many people only understand big data from the perspective of where they have seen it applied. One participant answered, “So I think big data is at the heart of a lot of our solutions, a lot of the products that we sell to our clients.”

The most common view of big data across the interviews was the sense that big data provided a greater volume of information on a subject, such as an individual or a business. One participant stated that, “Big data is quite broad, if I start from as many sets of information that you can put together that are tied to one individual or one thing is what I would consider big data” and another stated “the volume of data that is coming through in terms of best motion of data and how do we manage that data”. This point was again emphasised by another interviewee stating that; “we collect quiet a sizeable amount of data… so what big data means to us is being able to prep all that information and show sides that we were not previously able to determine or see”. Participants from a business or consulting viewpoint also then added that big data is equally about more accurate information and the ability to build insight from data knowledge for better decision-making. One participant stated that, “typically we would say it refers to a larger volume of data… but I think the most important thing is what do organisations do with their data”, and interviewee 2 gave an example to represent this fact; “Eventually you are taking away all sense of ‘what if’ you know exactly who it is. You are not saying what if Stephen would like to buy a bicycle; big data is now saying to me Stephen is in the market to buy a bicycle it is just about getting Stephen to the right shop to buy the bicycle he wants”.

Some participants took a more academic view of the definition of big data, showing their knowledge that big data can be explained in terms of the 5 V’s (Braganza et al., 2017; Sheng et al., 2017;Erevelles et al., 2016; Gupta and George, 2016; Sheng et al., 2017;Wamba et al., 2015). One participant could only recall two of these V’s, volume and variety, but this interviewee introduced the fact that data then comes from a variety of sources and is handled through big data by stating; “data is coming from different sources…when it comes to big data the advantage is big data is able to accommodate this, it has a variety which means any type of data, you can be able to handle it”. Another interviewee stated that, “the traditional definition of big data for our organisation holds it’s the reams of data, the variety, velocity, variability and so forth of data”.

Importance

No matter whether an organisation was progressing well with utilising big data or not, there was an overwhelming view of the importance of incorporating big data successfully in order to be competitive (see  below). There was an emphasis that this is for all types of organisations and especially customer focused businesses.

Table 4. Quotes substantiating the majority view on big data and competitiveness

There is also a drive for companies to make BDAC a core competency and this is believed to be the way forward to ensure competitive advantage. “This is what the company is planning towards, it is going to be the core competency of the company without a doubt”, is what interviewee 1 said in their interview. When asking another participant if they believe big data should be a core competency they answered, “Definitely you’ve got to your know customers, you’ve got to know your employees, you’ve got to know your directors, you’ve got to know your business, you’ve got to know what the market thinks of your business. You’ve got to know behavioural patterns of your customers and your employees. You’ve got to know who is dealing with your entity”.

A participant from an organisation that is considered one of the leaders in BDAC in South Africa perceived competition and the importance of big data competency at a greater level than the other interviewees. They expanded on the fact that immediate industry competitors are not their main competition anymore. This is because of the consolidation and convergence of industries that is occurring due to big data analytics becoming a core competency. Companies with a BDAC core competency that collect and utilise big data well are the companies where competition is going to emanate from in the near future.A very impactful quote from interviewee 7 was; “We need to become Google before Google becomes us”. They continued by saying; “Our competition is data companies. We are looking at who is going to disrupt us”.

3.3.2. Data Capturing

In general, most organisations use their business transactions and internal systems to capture data. The predominant sentiment was that companies do not utilise all the data that they generate. One participant stated that; “we generate up to five terabytes of data every day and we are not through with that data yet, some of that data we use but most of it we are not using”.

Mostly organisations are capturing and using structured data for insights but there does exist a lot of untapped potential in unstructured data. Knowing how to utilise the unstructured data is more limited and more big data mature organisations are starting to drive towards utilising more unstructured data. “There is a certain amount of unstructured data that we do capture but I think mostly structured” was a comment from one participant while another stated simply that, “the bulk of our system data is structured but there is a massive drive to gather unstructured data”. Another participant said; “we have a lot of customers, so we have a lot unstructured as well. I would say we naturally tend to focus on the structured, but there is unstructured stuff”. Interviewee 3 clearly stated the gap in unstructured data availability and utilisation; “what they are currently using, the unstructured would be about 20%. What they have got available for them to use unstructured would be 80%”.

The move to utilising unstructured data is new, which is supported by this participant who said; “So we are moving to unstructured, but that is still very much R&D level, it is not operational level” and interviewee 13 echoed the sentiment by saying; “A lot of unstructured data is in almost all organisations, how many are deriving insights from that unstructured information or doing something with it? Very few”. One of the big data consultants summed up the situation nicely by saying; “I think the percentage of unstructured data in the data lake is still very low, it is probably less than ten percent of the data that we are bringing in, mainly because it is difficult, it is difficult to tag the data, it’s difficult to get the data out, so people are still analogue, the structured data is still huge, unstructured data in the outside world vastly supersedes the structured data but even that structured data is so much that we spend more than enough time just analysing structured data without getting unstructured data in, so we should be doing it but it’s not as pervasive as we would have thought”.

3.3.3. Politics

There is mixed feeling around politics within organisations due to big data or switching to utilising big data. Organisations that are new to big data implementation feel that more politics may arise as big data becomes more prevalent but more big data mature organisations report the opposite in that politics has either been resolved or the strong leadership drive leaves no room for politics to interfere. Interviewee 1 stated that; “There is not much politics around it, all we are focusing on now is to make it function as best as we can so that we can deliver to the business, so personally politics not much but as it grows and people realise its value then definitely there will be politics around it”. Interviewee 7 from a more mature big data company said; “whenever there is something that is big like this, it is getting a lot of investment, a lot of focus, and there is opportunity to make a name for yourself, there is always politics involved. There are always various stakeholders in different areas that want to have control of it, they want to take over, and they want to benefit from the success of it. So yes, there has been quite a lot of politics around it. The people that are in charge of it now seem to be quite adroit at side stepping the politics. It also helps though that the group CEO is on top of this thing”. Interviewee 9 from a top big data organisation revealed politics to exist but not be an inhibitor by saying; “there are always politics with certain things, but the business case always wins and it wins quite easily, and it is also again because we are all striving for the common thing”. This participant introduced a common goal and having a good business case process which is respected as a reason for politics not interfering with big data success.

Some participants revealed that politics around systems exist. Interviewee 11 stating that the politics is around legacy systems; “as much as they want the result, they are going to stay with whatever software we have done, so you know there is always politics, people still want their own tools.” and the other around which new system to adopt and relationships with current technology providers; “the only real politics that we have are we have got some relationships with Microsoft and then with Oracle and getting set up and then I am pushing for AWS, so there is a lot of ‘why are you not using this’ and ‘why are you not using that’”.

Interviewee 14 stated; “I think you know the information in itself is a political game, um you know what I have, what I control, what I know is very difficult to let go if you kind of have got the old school of thinking around hoarding of information and so forth” and this brought to light the politics around information sharing and how this can affect power positions and reduce key man dependencies, This can become political if people want to protect their own interests and positions of power above the benefits to the organisation as a whole.

3.3.4. Process

Most organisations initiate big data projects by an identified business problem. Participants stated that’ “you start with a problem in mind”, “anyone who has got a problem they would like to solve with data, puts their data set out there” and “there is a business drive and if we need to understand the problem better then go into the data to understand it”. Some more mature organisations also look at business needs, objectives or targets of business units to initiate big data projects or big data project leads.

Interviewee 1 described their process as starting with the identification of a business problem, which is then assigned a product owner. The data scientists then engage with the big data team to find a solution to the business problem. Data for the solutioning then needs to be secured and a basic solution or minimal viable product (MVP) is prototyped to evaluate the solution before a full solution is developed.

Two other participants diverged after the business problem identification stating that a business case is then provided and vetted by a board as well as a data and analytics exco. Business cases are scrutinised from many different aspects, such as implementation, privacy and operations, and prioritised. Virtual teams consisting of volunteers can then be created to solve the problems presented, which empowers people to build and develop their own models, reducing the demand on the big data team and data scientists. Interviewee 7 stated; “anyone can come with a problem that they believe data will solve. They need to business case it. And then volunteers from multiple business units would get together and we will create a sort of a virtual team that will solve that problem”. Interviewee 8 stated that their process is similar to the above process and they view this as a structured project management process for IT projects; “It is structured in terms of the project portfolio where projects come up; they get prioritised, they get scaled, they get approved, a business case is put forward, the project is sponsored, the owner then perhaps signs their commitment prior to getting the resources to work on a project. So, it comes via a very structured IT project management process.”

Another initiator of a big data project is innovation. The freedom to use big data to innovate rather than solve presented problems was rare amoungst the interviewees organisations. Interviewee 9 stated how innovation within business units themselves will drive big data projects rather than just project management processes. These projects then have a greater amount of freedom at the idea generation phase but caution is taken to ensure an idea has merit and substance. More rigour is applied throughout the process with more checks and balances at the implementation phase. Another big data innovation initiative was revealed by interviewee 10 as a conference; “which is all the actuaries and data scientists and anyone in similar fields, where you have to come up with some idea to save the company or make the company money, and then do like a feasibility study and show some results from that.”

An interviewee from a more immature big data organisation revealed that they do not have a process to follow when it comes to big data projects stating; “when it comes to projects, we let them try and solve things their own way and hope that senior people will guide them in their process”.

3.3.5. Value

One of the consulting interviewees described the value of the data itself and stated; “data is like the black market”, which demonstrates this value. He establishes that data is truly sought after by saying; “It is very touch and go at the moment as to you don’t want to know that a big corporate is buying millions of records at a time to update their database to increase sales”. Building data sets and doing data analytics has a positive impact on the economy in two ways; firstly through selling data; “We do put our data together, but we do definitely sell data, if we don’t sell data we don’t have a job and we don’t make money and the economy gets even worse because it puts people out of jobs” and secondly by providing businesses with valuable information that increases their efficiency, thereby positively contributing to the economy.

Value is also seen in terms of the business value that data can bring to a company. This was the more common view of value in data and BDAC. Some participants perceive that big data has already delivered plenty of value within their organisation by stating; “the amount we have spent for what we have made is nothing; that’s why we have been given almost unlimited budget – not unlimited, but ridiculous budget”, “everyone is on board with it; they see the value”, “data is a big part of what we do… big data is at the heart of a lot of our solutions” and “realising slowly but surely the value of the data”. There is also a common sentiment amoungst the respondents that big data can do more and provide more value. This is seen through the responses; “we are doing really well but we are not doing well enough”, “I think there is huge scope probably everywhere for us to do more, right? It’s definitely under-utilised, and it is an opportunity for a lot of organisations to basically be more successful”, “the company is at a point of knowing that they still can improve”, “in fact there is more coming, there is more coming” and “A lot more. I would go as far as saying… I am so excited about stuff that we can do”.

Some participants revealed that companies have struggled to derive value from big data as evidenced by statements such as; “I think it was a case of lack of vision of where big data could actually help”, “currently I don’t think they are using it enough to be able to say it is of any value”, “it’s like everybody want to do something with it, but nobody kind of knows what they want to do with it” and “So for now it is very very low, very very low because we are still coming up with the proper strategy and more or less use cases for it… so currently I can’t say it is adding that much value yet”.

3.3.6. Training

Most participants eluded to the self-study nature of a lot of the training required to develop big data skills. Interviewee 7 said; “Literally some of the stuff we want, there is nowhere that you can study it. You go online, play with the blasted thing, download it on your machine and use it”. Interviewee 9 echoes this sentiment by saying; “you can actually teach yourself quickly, if you have the attitude and interest and all that stuff”. Interviewee 6 mentions that training is through on the job experience; “We don’t have a formal thing for that, it is on the job training, as you go you adapt” and interviewee 12 says that training is; “more of a self-study”.

Some organisations enhance training through online courses, third party training, programs and seminars. Interviewee 7 stated that; “there are companies and courses all over the place. I mean you can go to Coursera on line and do stuff”. Interviewee 1 said; “it is an external company that does that so it is not only about self-training” and interviewee 15 said; “we have got a lot of user group training sessions… that is basically based around specific platforms”. Companies that are serious about developing their BDAC skills have a training budget and resources that are allocated to big data skills enhancement and development. One participant mentioned; “there is a budget available as well on a yearly basis so you certify whoever you need to certify”.

Interviewee 9 from a more mature big data organisation mentioned how they take training to the next level by having combined sessions, like grade sessions and hackathons, where everybody comes together to find big data solutions and can then learn skills from each other. This is a particularly useful method to utilise for more complex problems that require more complex solutions and techniques.

3.3.7. Organisation

Interviews with participants from companies at varying big data maturity levels were conducted. The general trend was that less mature organisations have tended to not restructure business units for better BDAC incorporation. Interviewee 14 said, “so I haven’t seen reorganisation or redesign, I have rather seen that they have added to the organisational structure”. Some respondents thought it may happen as they mature and others believed it is unlikely the business units will restructure. Interviewee 15 stated, “it’s not a necessity that it needs to change because the aim of big data should be to facilitate what they already have in place”. More mature and leading big data companies have gone through some organisational restructuring to allow big data to be the centre for decision making. Both centralised teams and deploying big data professionals within business units are utilised in more mature organisations. Interviewee 7 said; “two years ago the division I am in was created from scratch; it was an amalgamation of two different areas, so one was looking at analytics, so data science and all of that, and the other one was what was our data warehouse. So, we put this together under a new leadership team, which I am part of, with a specific purpose of having a centralised data repository and platform that we could use to service all the data scientists across the bank” and Interviewee 9 stated “there is now a whole new restructure or different structure in the whole system technology space to accommodate where we are going. At one point they actually put together what they call cross company analytics”.

Interviewee 1 mentioned how they have two division of big data, a business and an IT division, where IT provides the support and business makes the decisions and sets the direction for IT to follow. This is supported through the statement; “it has got two divisions, it has got the IT side and it has got the business side, both of those sides is very well led but it’s the business side that has got the most power and the most decision making the IT side is there to support the business side”.

3.3.8. Application and Uses

Many applications and uses of big data were mentioned throughout the interviews. Some of the main uses mentioned were that of marketing opportunities, customer satisfaction, operational efficiencies, innovation, predictions and forecasts, and increasing revenue. Table 5 below lists the main uses of big data found throughout the interviews with key quotes to substantiate each use.

Table 5. Main Applications and Uses of Big Data

3.3.9. Advanced Big Data

Participants mainly believed that advanced big data was about the technology and achieving automation, AI and machine learning. Interviewee 2 stated; “right now we are not near as sophisticated. We are building a super hi-tech system that is going to be unbelievable”. Interviewee 7 said; “that requires a lot of automation… your AI component of it, so it learns and its ability to decide is a lot better”, while Interviewee 3 and 6 concurred by stating respectively; “I would say advanced big data would be getting to the machine learning and cognitive space. So, where you are starting seeing value out of unstructured data that you weren’t getting beforehand” and “. I think all of the advanced stuff is people trying to crack in essence artificial intelligence, or some form of artificial intelligence to communicate with your customer – and that is what I would consider advanced”.

Interviewee 1 mentioned how they believed advanced big data was also about pervasive big data where big data is a predominant source and part of revenue generation, improving customer experience and resolving business problems. Interviewee 2 agreed saying that although advanced big data is about advanced systems and backends, it is also about being advanced in terms of results, outcomes and value. Advanced big data is seen as accurate and effective big data and the ability to provide up to date information.

Interviewee 7 also believed advanced big data is when the organisation has the ability to compete against other data companies and not just industry competitors. Interviewee 16 gave their dream of big data as advanced big data stating; “Its where we don’t even need to ask a question, all we need to do is wake up in the morning check your phone and you have got a message telling you yesterday this is what happened and this is why it happened and potentially this is what you can do to improve this going forward”.

Interviewee 9 gave an integrated view of advanced big data across the organisation where big data competency has the ability to exploit data and derive exceptional value from it, big data implementation has the ability to apply and use advanced big data tools and techniques and big data reporting has the ability to get insights to the clients or departments in user-friendly ways for them to action. “Practically combining the data… and then it is about the intellectual ability to look for the right things, and then it is about implementing”

3.4. Results for Research Question 2

Research question 2: What capabilities are required in order to be successful at translating big data insights into organisational performance?

Research question 1 established which organisations were more mature with regards to big data and more advanced with their ability to successfully use big data and develop BDAC. The questions in the interview guide for research question 2 were asked to establish the perceived capabilities and reasons as to what enables an organisation to successfully use their BDAC for better organisational performance. The question also sought to confirm whether the factors identified in the literature of the previous paper were confirmed to be the main drivers of BDAC success and this will be expanded on in the next section.

During the interviews, participants mentioned what they believed to be positive factors driving big data success. These factors were grouped into constructs and then further grouped into nine main themes that emerged from the sixteen interviews conducted. Table 6 below shows the frequency count of how many participants mentioned each theme throughout their interview. The higher frequency does not necessarily mean that that theme contributes more to BDAC success but rather that it is a more prevalent or known aspect that is required for big data success amoungst the participants. This section will now outline each of the themes and their constructs in the order of their frequency.

Table 6. Frequency of Big Data Success Factors

3.4.1. OrganisationalStrategy

Mature organisations and big data consulting participants agreed that companies have a strategy and vision for big data and big data implementation with a good idea of the opportunities in big data and the value than can be derived from it. Interviewee 2 stated; “Yes so we do, we know where we want to be in about three- or four-years’ time”

In order for big data to be successful it needs to form part of the overall, long-term strategy as well as the digital strategy. This fact was mentioned by the majority of the respondents with interviewee 1 stating; “It is very very strategic, in fact big data is part of what they call vision 2020”, interviewee 7 said; “they identified I think between ten and twelve strategic things that the bank wants to build and underpinning that is the data strategy” and interviewee 9 mentioned that; “data is critical to absolutely everything we do”. Interviewee 3 mentioned; “they view it as a very strategic play as to how organisations become a cognitive organisation; so, without having big data mechanisms in place, an organisation can’t move to become a cognitive organisation” Interviewee 10 made the interesting point that big data projects that align with the strategic objectives of the organization easily get buy-in and support.

When making big data a part of the strategy, organisations need to make decisions based on data whenever possible and ensure there is a clear goal and vision within the organisation to help define what big data value and value generation looks like. Interviewee 9 stated; “It doesn’t say that there is not “argy bargy” and you know, very stressful situations that happen and what’s the best way to implement this, but it is almost like the business case wins.” And interviewee 4 said; “its core to our business, all our strategies revolve around big data, 2025 digital strategy, it is part of our innovation and we rely on it heavily to make strategic decisions”. It is important for organisations to not only have a plan for big data technology and insights determination but also for big data implementation as well.

Part of a strategy process is benchmarking and best practice and this no different for BDAC. The top big data companies are benchmarking internationally and against top data companies for their goals when it comes to BDAC. Interviewee 7 stated; “our exco actually flew over and visited Amazon because what they are doing is quite fascinating”.

Leadership

A key organisational factor that drives big data success is the support and drive of big data from senior leadership at an executive level and this is then disseminated down the organisation. If big data is seen as a vision of leadership, the rest of the organisation will make it a priority to implement. The leadership believing in and driving big data from a place of personal passion also helps to create buy-in and lead others to be passionate about big data success. Interviewee 7 stated; “So our group CEO, I think he sold his car a year or two ago; he only uses Uber. Because his thing is he believes someone is going to disrupt the industry, like Uber disrupted the taxi industry… it is very centrally driven. So, ja, there is massive buy in, and again, it comes back to our CEO”. Interviewee 8 was very clear that; “it is supported at top level management downwards”. Interviewee 14 revealed that the emergence of top leadership for big data focus (chief data officer and chief analytics officer) has helped propel big data forward in the organisation. Interviewee 1 stated; “driving from the top down, CE is driving it, the CT, the CIO is driving it, so there is quite a lot of interest in big data to make sure that it succeeds”.

Some companies require a change in leadership to fully reap the benefits of big data. Leadership may need to have different skillsets and different cultures in order to embrace big data rather than the legacy systems and processes. Interviewee 5 mentioned the importance of leadership making big data a priority so it does not get overlooked ignored for other tasks or current day-to-day operations.

Forming a centralised BDAC department with top leadership is important and something that sets the two top big data organisations interviewed from the rest of the interviewee’s organisations. This centralised team also becomes the leader of BDAC in the organisation and drives the forward progressing uptake and implementation of big data throughout the organisation. Interviewee 7 stated; “we put this together under a new leadership team with a specific purpose of having a centralised data repository and platform that we could use to service all the data scientists across the bank”.

It is important that leadership takes place on each big data project itself where someone takes ownership of the project and holds the accountability for its success. This was demonstrated by interviewee 7 who stated; “because it needs to have a sponsor so if someone is not going to drive it, we won’t use it”. It is also important that leadership ensures the core structures are correct for fully realising benefit from big data and are willing to make any necessary structure changes to ensure they make sense for successful big data implementation. Interviewee 9 said; “making sure the core structures are supported… making sure that structurally everything makes sense”.

Commitment

Commitment to utilising big data and implementing big data solutions is of utmost importance to show the employees that the organisation is serious about big data and making big data a central part of decision making and solution driving. The attitude of, failure not being an option and everyone needs to make big data work, is important to drive positive and determined employee behaviour and shape the correct culture for big data success. Interviewee 1 stated; “there is a lot of money and resources we are investing in it… it is a big deal so there is no turning back” and “there is no margin for failure, no one can invest so much in big data and then it’s a failure so failure is not an option”. Commitment needs to be shown by spending money on systems and investing in resources. Interviewee 14 said; “a lot of the organisations are spending a lot of money on big data implementations”. Interviewee 10 talks about the investment in data collection systems; “since I started there has been a big push, we need access to this data, and we need all of these different features”. These resources need to be skilled in big data and visible within the business units.

Investment into the big data agenda is key to its success and growth in maturity. Hiring the correct human resources with the skillsets required for each aspect of big data is important to show the commitment to seeing big data from systems to capture data through to implementation and realised improved organisational performance. Interviewee 5 gave an example of the difference between organisations that dedicate resources to big data properly and those that do not by saying; “here’s a good example I can give you: you will go to some companies where there is one person that is responsible for all of this, and you’ll go to other companies where there is an entire team and structure dedicated…and that in itself says something, so how much resources are being dedicated to it makes a difference”. The next step to commitment is to develop and enforce new processes and procedures that will enable big data implementation success. Structural reorganisation may be required to optimise these processes.

Commitment can be measured through the growth in the number of big data projects being completed successfully by the business units and the centralised big data team themselves. An increase in the amount of investment that is required as well as the big data resource skills required will also indicate an increased commitment by the organisation to utilise big data. Interviewee 7 stated; “the demand in terms of what we are building is massive, so I am starting to have to juggle resources… I am currently going through a massive planning phase with everyone and some of the teams have got up to 24 months of work in their current capacity – booked up! Which is quite crazy!... I have literally doubled the team in terms of running the projects”.

Monitoring, Measurement and Incentivising

It is important to monitor the progress of big data and big data implementations. Putting measurements in place with clear goals and then ensuring the organisation is tracking the performance. Interviewee 7 revealed that they set targets on the number of data projects successfully completed for the year. Interviewee 9 stated; “So we track and monitor pretty much everything so as we implement new stuff, we get the feedback as to whether it is working or not and make changes based on that”. Interviewee 4 mentioned that performance evaluation and remuneration and business units are aligned with big data. Interviewee 7 also highlighted the importance of feedback on big data and told us that; “we have a data and analytics EXCO which spans the entire group and so in that EXCO we get feedback on what is happening, challenges and issues are also raised”.

Incentivising successful and top achievement in big data performance from data gathering to implementation is important. Both interviewee 7 and 9 are from top big data organisations and they differentiate themselves from the rest of the organisations sampled by stating that they both run internal competitions where data related problems and posed and there are rewards for the best presented solution to the problem. Interviewee 5 also said; “there has to be an incentive for everybody else in the organisation to work towards”.

A common method in most larger organisations for measurement is through the use of Key Performance Indicators (KPIs). KPIs can be used for big data monitoring, measuring and incentivising. One participant said; “your KPI’s are quite clearly set and you always do your day job. We are very good at rewarding and incentivising, and it is very clear that you need to deliver, you get paid to do a job, so they make quite sure that it’s measured”. Data itself can be used to measure and evaluate KPI’s. Interviewee 9 revealed the importance of refining the KPIs and ensuring they are clearly understood in terms of contribution.

Human Resources

Not only is it important to hire resources with big data skills but it is also important to hire the correct type of people with the correct aptitudes in all business units going forward. This will allow people to adapt to big data implementations and have the correct competencies to view situations in ways that can use big data as a method to solve problems or be innovative. Interviewee 9 believes employing many highly educated people who are accustomed to problem solving and data analytics contributes to the success of utilising BDAC for improved organisational performance. Interviewee 15 concurs with this sentiment by stating; “you need people with that core critical thinking and problem-solving abilities”. Interviewee 7 believes that hiring people with the ability to learn a new technology is a more important skill to look for in a potential employee. Interviewee 12 said; “there are definitely people who are more progressive thinking, forward looking” and thereby highlights the need to hire individuals with progressive and forward-thinking competencies.

One participant highlights the importance of technical big data employees learning the business understanding as well by stating; “it is always that the importance is getting them to understand what we are trying to solve; they need to understand the business processes and things like that. It’s important they spend time in the business”. They also state that the opposite is also important by bringing in more and more big data competent resources into the business units to transfer learning of BDAC to the more business focused employees.

Interviewee 4 reveals that it is important that job descriptions be less confined when it comes to big data in order to account for necessary agility and adaptations that may be required to successfully implement a big data project. Interviewee 5 believes that BD implementation and application skills are more important to drive big data success than technical skills. Interviewee 6 makes reference to the competitive advantage that can be attained by employing and having access to the best big data skills.

Big Data Champions

Big Data Champions are important to drive enthusiasm and support for big data initiatives and projects. They are important to create awareness around big data and encourage business units to use big data as their main source of knowledge to solve problems or complete tasks. During the interviews everyone was in agreement about champions being required and participants from organisations with struggling big data initiatives generally did not have big data champions or they did not have the correct type and level of champion. It was revealed by participants from more successful big data organisations that they have high level and powerful members of the organisation as their big data champions. Interviewee 1 stated “there is definitely a champion… the project sponsor of the whole thing, the guy is quite powerful”. Interviewee 16 from an organisation just starting to implement big data said; “yes we do have a champion, in other words we have got one of the MD’s whose division we are trying it on, is set to be the first champion” and this is a good step to helping promote their big data drive.

Interviewee 4 revealed that the big data unit rather than a person is the champion. The big data unit is established enough not to require a champion that much anymore as they are well known throughout the business and respected. Interviewee 10 and 11 said that in their organisation there are big data champions, advocates or disciplines within each business unit that truly believe in and are passionate about big data. Interviewee 13 stated; “you definitely need a champion and you need a champion that understands business and understands the capability of the technology” and this raised an important point about the competency required from a big data champion being that of business acumen as well as big data understanding.

3.4.2. Openness and Awareness

The organisation and the culture of the organisation needs to be very open-minded to be successful at implementing big data initiatives. It is important that there is buy-in and a good sentiment towards big data across the whole organisation. For this to occur there needs to be a level of awareness of big data; what it is, what it can do for each business unit and how to go about utilising big data to improve organisational performance. Interviewee 7 stated; “I have actually got an engineer I hired. Her job is to go and visit people and find out what their problems are, and work out if there is data that can be utilised to fix this problem” and interviewee 4 mentioned the importance of people within units in the organisation to bridge the gap between the unit and the big data hub and help with understanding throughout the organisation. Interviewee 5 agreed with this sentiment by saying that the link between big data analytics and the rest of the organisation is important.

The focus should also be on collaboration, in order to leverage data across business units and be able to gather the best knowledge and datasets by combining data from each business unit into one database. Each business unit can then leverage off of the whole organisation’s efforts to collect data and not just their limited ability. Collaboration should remove siloes and allow the business to work as a whole with big data as the underlying driver linking all business units together. Interviewee 7 stated; ‘The thing with data though is you want to be able to leverage each other’s data and you want to be able to collaborate. So, part of the strategy now is to get collaboration particularly around data. So, what we did is we created a group wide strategy so anyone can come with a problem that they believe data will solve. They need to business case it and then volunteers from multiple business units would get together and we will create a sort of a virtual team that will solve that problem… and the data then gets hosted centrally”. Interviewee 9 revealed the importance of the central unit encouraging collaboration across the business units and stated; “we share data a lot, across the company. I think that is important, there is no ‘you can’t have this because it is IP or whatever’ – we really do share data”.

An organisation needs to create an environment where it is not only the data scientists using big data but the whole organisation. Interviewee 1 said that part of their success is that anyone can approach the big data unit with business problems to use big data to help solve; “everybody within the business who has a problem that they need to resolve or a problem, big data is there for them… there is no cut off point here… as long as you have a clear defined problem and data available, then we are able to use big data and data science to resolve whatever challenges which you might have and there is no cut off there… any executive that thinks big data or machine learning will be able to resolve his particular problem we are to here to help”.

A more mature organisation has realised that it is not only about business units approaching the big data team to solve problems but about fostering a culture where business units can use the big data platform themselves in order to solve their own problems. This creates a deeper understanding of utilising big data throughout the organisation and ensures the limited big data team resources are not a bottle-neck to achieving success with big data across the entire business. Interviewee 7 agreed with this statement and mentioned how this importantly allows organisations to increase the speed at which big data can be implemented and said; “we don’t want to have a dependency on an IT”. Interviewee 2 stated; “by next year Feb we will probably be at our full mature stage, which means that big clients can have access to our application, put their own data sets in around scoring models off our database they can do everything themselves. Right now, we do everything for the client; it becomes a bit tedious and becomes a bit time consuming”.

As not all employees will have the necessary skills to do data science, automation and ease of use of business platforms is important to enable self-service. Interviewee 7 stated; “part of the journey is to make it user-friendly. The other big part of the journey though is…the bigger issue here is automation and self-service”. One participant mentioned the importance of providing basic big data education to the business units in order to make this possible because they need the awareness to start thinking about big data in the right way and start asking the right questions for the big data platform to be utilised effectively and successfully throughout the business.

3.4.3. Competency

Data Insights

A big aim and emphasis is to gain as much insight as possible from data and therefore the emphasis needs to be on insights attained rather than data itself. This will allow big data to become a core competency and facilitate competitive advantage if done well. To gain the correct insights it is important that the business side of the problem is understood and the technical employees have been involved from the start in order to facilitate this occurring. Interviewee 7 stated; “our strong aim is to gather as much data as humanly possible and leverage insights off it, because data without insights is worthless”.

Data visualisation

The organisation needs to be aware of how the results of big data problems should be visualised to be understandable and applicable to business and not just the technical specialists. Interviewee 1 mentioned the importance of understanding how the results of certain big data problem types should be represented and using the way they should be measured to guide visualisation. Interviewee 11 mentions how their organisation ensures big data results are made very graphic and visual so that it is easy to understand; “you find that it shows pictures and all those things just for anyone to see, because once it has come through visual graphs and all that it becomes easy then”.

Implementation

Big data competency is not only about deploying a technology, but in its implementation and application. This will enable organisations to derive the most value out of their investment in the technology. Interviewee 7 said; “you can have the best tech in the world, if you are doing arbitrary stuff with it, it is not going to help; whereas you can have medium tech but you utilise the thing till breaking point and you get a lot of business value”. Interviewee 5 stated the importance of implementation to derive value from big data by saying that big data has its most success if there is follow through. Working on implementation of solutions, as much as solutions themselves is important; “the idea is not that we kind of partner with them and then run away, the idea is that the delivery is just the first step of the project; after that a consultative process starts and we keep working with them to make sure that they can make use of what we have delivered them”.

Implementation needs momentum and drive to continue moving forward and being able to act and implement quickly in important for success. Interviewee 9 stated; “you know this company go, go, go, goes – so we don’t look back too often but if we implement and it breaks then we fix and move on… we just drive it forward and improve it”. Organisations must be able to quickly make structural changes required to optimise and streamline roles for the big data solution implementation. It is also important to review and get feedback on implementations to understand each time what went wrong and what could be done better. Interviewee 5 concurs with this statement and said that identifying and working on the obstacles to implementation is extremely important. Many organisations are able to innovate with ideas of what big data can do but implementation is behind and it is important to foster an implementation culture in order for implementation competency to catch up with the ability to generate ideas and potential solutions.

3.4.4. Data Competencies

Developing the competencies in order to collect as much of the right data for the best knowledge is important to big data success. A better knowledgebase from which insights are attainable is a competitive advantage to companies. Interviewee 1 said that their organisation is aggressive when it comes to data collection and currently collects more data than it can use but understands the importance of having information available. Interviewee 5 states the importance of the quality and condition the data is in. Less mature organisation or participants from organisations where big data as not been that successful often mentioned that they did not capture data even though they could have and this was knowledge that was lost. Interviewee 12 said; “it is simply because we do not collect big data, as maybe we should have been doing because of the new sources of big data or data that has become available”.

Interviewee 2 mentions how data has become a valuable commodity and interviewee 1 mentions that they are storing more data than they use as they understand they may be able to sell it one day. Interviewee 2 mentions that there can be competitive advantage in data, the data incorporated, how it is incorporated and what is derived from the data. Interviewee 9 believes that there are two sources of competitive advantage from data; capturing unique and exclusive data and using more common data in unique ways.

Span

Interviewee 2 mentions the importance of knowing what data to source to where to find the data. Data can be everywhere and just collecting everything with no plan can hinder an organisation more than help to achieve better performance. A plan for collecting the right data and knowing where this data can be sourced is step 1 to ensuring a more organised and direct ability to utilise data. Data can then be collected across all business units in an organisation and stored in a central depository for data but interviewee 7 mentions that there is a greater span of available data by increasing collaboration across a group of companies that the organisation belongs too and not just the organisation itself. Interviewee 2 mentions how this is taken a step further and there is a lot of buying and selling of data between companies to allow each company to build better and more updated profiles and they also mention how a lot of data sharing takes place between companies; “data in exchange for data”.

Sophistication

Data analytics has come a long way and increased in sophistication throughout the years. There are more sources of data available and the ability to combine sources to have even more knowledge in one place about any particular subject has increased exponentially. Interviewee 2 mentions how data analytics started with basic data capturing and basic profiles’ “captured through credit bureaus” but many research companies, data providers and data builders have entered the field and they generate a lot more information on business and consumers. It is important for companies to understand this progression and take advantage by being aware and utilising the new sources of data.

Sophistication can also be in the form of an organisation being smart in where it looks for data, what and how it measures and how it implements and this sentiment is mentioned by interviewee 9. Interviewee 7 goes on to say how sophistication in data competency moves into doing tasks such as using meta data and other methods to improve data and test data quality and integrity; “part of our structure now is it is not enough to just get data in, you have got to have meta data, you have got to have quality, lineage, know where it comes from, you need to know if the data shifts or if it gets corrupted or anything like that. So part of our infrastructure investment is it is not just about getting the data into our central platform and making it available, it has got to get in there and we have got to know that this data is good data; it has been validated, it has been checked and so basically it is the purity of our data.”.

Uniqueness

A source of competitive advantage is gathering unique data to your other competitors because this is exclusive knowledge from which to gather insights for better decision making than the competitors. Interviewee 9 states; “we are getting information that other companies are not getting because of just our unique products”. Technology is therefore a means to acquiring unique data and interviewee 9 points out that they can measure things other companies are not able too and this generates unique data for their organisation. Interviewee 5 also touches on the importance of being able to have access to exclusive data; “there is no other company in SA that has such an extensive read on that. So, that’s really unique”.

Innovativeness

In order to gather unique data, you need to be innovative in the thinking around how you are going to source data, this is demonstrated well by the organisation of interviewee 9. This and then innovativeness with the way in which more common data is used are the two ways innovativeness can provide competitive advantage for an organisation. Interviewee 2 stated that data sets and data profiles may involve a lot of innovation to build, with a lot of indirect ways of gathering the needed information. They also mention how being able to build very innovative data sets and data bases from scratch, opens the door to new business and marketing opportunities; “You’ve got to think outside the box, if someone wants information somewhere being a data expert you never say I can’t get you that because you can always get it”.

Software

The interviews revealed an interesting debate between using vendor-bought software versus open source software. For example, interviewee 8 and 11 stated that the organisation mostly uses vendor software, rather than open source software with interviewee 8 adding that the vendor software will be made more user friendly, more interactive, and faster in terms of getting results. But interviewee 7 and 9 agree that open source software was the key to allowing big data to be utilised by many more organisations and gives organisations the freedom to quickly adapt to new methods and technologies required. Interviewee 10 agrees with open source and states; “embrace open source because one issue that like if you get stuck with vendor lock in then if you bring new people in then they are not going to know how to use that product and you are also reliant on that company that is providing the product to give you new features, and then there is like a whole onboarding process. So I think they need to do a lot more around teaching people to use Python and R and using some of the machine learning modules that are out there”. Interviewee 14 mentions how their organisation use a combination of vendor and open source software. Interviewee 12 gives their summary where they state; “typically I say if you are a SAS user you fall into one category, if you are an open source user you fall into a different category. And the people who use open source are typically more entrepreneurial and innovative in their approach; SAS and those kinds of environments do not allow for much.”. SAS is an example of a vendor software.

Based on the interviews, more mature organisations are moving to open source software due to the flexibility it provides as well as the skills development being easier than needed third party system training. This removes reliance on a third party and allows the organisation to use the software in a way suitable to their organisation and their unique requirements rather than a more standard approach as is found in the vendor software. Open source is also more cost effective as it does not require a large upfront fee to purchase or large licence fees per user to utilise. Building data competency in open source software that can meet the requirements of the big data project can lead to more successful BDAC.

3.4.5. Development

It is important to develop big data systems and technology as well as big data skills.

Technology

Interviewee 2 said that they believe in the importance of optimising, standardising and automating the big data platform in order to allow for quicker results and the ability to complete more big data projects at one time. Automation also facilitates removing reliance on the big data specialists and allowing business units to use the systems themselves effectively.

One participant mentioned that the development of front-end platforms helps to introduce user-friendliness into the system in order to allow non-technical business unit employees to access and utilise big data. Interviewee 7 specifically highlighted the development of visualisation for simplicity and user-friendliness. Another participant said it was important to develop in order to improve the response times of the big data systems to reduce the time taken to solve big data queries and problems. Interviewee 9 said development was important to keep up with the latest technology.

Exploration

Interviewee 1 stated; “before you can do anything with your data the first thing you have to do is to do exploratory data analysis otherwise you are not going to get any value implementing any big data solutions… you have to understand in order to implement, so one of the first things you need to do is to explore the data”. This highlights the importance of exploration to gain new understanding and develop big data capability. Interviewee 9 mentioned that their organisation allows a lot of freedom at the idea generation phase to facilitate exploration and develop skills to be innovative and generate new ideas.

Skills

Training is important to develop skills in utilising big data, from systems through to implementation and from big data specialists to employees in all business units. Interviewee 7 stated the importance of training to help business units build solutions; “what we have to do with them though is teach them, the solution has to fit the platform”. Interviewee 9 mentions how they develop skills through conferences, hackathons and combined sessions to bring people together to find solutions to big data problems. This is important to further skills development as people learn from the challenge and from each other. Interviewee 15 states; ‘I think what is really nice is we have got a lot of user group training sessions and we actually host some of them as well and that is basically based around specific platforms that we use”.

3.4.6. Existing Experience

Existing experience in analytics can help an organisation develop their BDAC capabilities by forming a starting point from which to understand the power of data as well as benchmark against to ensure big data delivers more value than traditional analytics. It also allows organisations to take smaller steps towards using big data for problem solving by starting small, proving value and then scaling-up the big data operation. Example of doing this successfully came from interviewee 1 who stated; “customer value management has been here for quite a while, but as big data grows further and further the intention is for big data to replace that customer value management” and interviewee 5 who said that analytics preceded big data in their organisation. Interviewee 9 said; “the big data being this sort of huge thing, that’s quite tricky, but there are a lot of interim steps that you can take” and they believe that traditional analytics; “the excels and that” are important as you need to start by understanding what you are doing and build from there.

Interviewee 3 mentioned starting small and gave the example; “I think in their call centre they could really use it”. They eluded to the fact that it was important to gain some experience first by getting it right in some area or location and snowballing from there. Interviewee 5 stated; “their exposure to it as well… when we work with an organisation with stakeholders in the business that have worked with this before and are familiar and comfortable with it, they will champion it, because they understand how good it is”. This highlights the fact that existing experience in analytics can provide the necessary proof to get buy-in from employees that big data analytics will provide value and is worth learning and understanding. Interviewee 5 mentioned that it was less daunting when you already had basic analytics knowledge.

3.4.7. Forward Thinking Focus

It is important to be forward thinking when it comes to developing and implementing big data. There is a common sentiment amoungst the interviewees that big data will continue to grow in focus and support in the future. As a result, organisations will become more dependent on big data as the basis for decision making in the future. Interviewee 1 said; “big data going forward is still going to be a measurable driver and most of the decisions will be based on it” and also stated that big data will further shift decision making to be data, insight and fact based. Interviewee 1 expects big data and the insights it provides to further influence organisational and business unit strategies, further change how business is done and how the business thinks, and will further change how and what the business implements. One participant stated that they believe there is still untapped potential and massive opportunities for big data to add value in the organisation and that big data has the potential to offer plenty more solutions and improvements. Interviewee 2 mentioned that they believe big data has the potential to open up new markets for organisations in the future. This was repeated by interviewee 9 who said that expanding reach and scope was eminent in the future as well as companies entering new areas, applications and markets. Interviewee 7 stated that organisations that are not data or data analytics competent are likely to be disrupted and made redundant in the future.

Big data is predicted by the participants to form a greater part of revenue generation as it is better utilised and implemented as well as more innovative solutions generated as skills and competencies develop. Interviewee 1 stated; ‘we definitely have to generate revenue going forward and it is quite central from a digital migration platform point of view”. Participants from more mature organisations have started to become better at big data implementation but organisations still growing their BDAC have said they expect better implementation competency in the future. Interviewee 1 mentioned that their organisation is looking at driving sentiment analysis in the future while interviewee 7 is more focused on the convergence of industries they believe will occur as more organisations become big data driven. Interviewee 12 stated; “traditionally data was collected for auditing and accounting purposes, and there is a key change that needs to be made so that data is a key asset or core competency” and this highlights the shift from data for compliance and management (auditing and accounting) to data as an asset or core competency.

3.4.8. Cost and Benefit

Organisations in early maturity with regards to deploying BDAC or organisations that have not been successful in implementing big data solutions, often would refer to the cost of big data technology and skills as a prohibitive factor. This is mainly because they have not managed to derive the benefit and value from big data, which the participants from the more mature organisations mention far outweighs the cost and creates the scenario where a big data business case can easily prove its value and make sure any upfront cost for hardware or skill is not a barrier to implementation. Interviewee 1 stated; “costs are going to be insignificant and the return on investment on this thing is going to be massive” and also said that although big data has an upfront deployment cost, this can easily and soon be negated by the value and benefit big data unlocks and provides. Interviewee 7 agreed and said; “if I do a cost benefit analysis, the amount we have spent for what we have made is nothing”. Interviewee 14 stated that truly innovative organisations are less concerned about cost, because they have benefit in mind.

Open source software is also seeing the decline in upfront costs to big data projects while returns on investments are increase as organisations gain experience and improve in utilising BDAC. Interviewee 1 echoed this sentiment, as did interviewee 7 who stated; “the costs are high but the return on investment seems to be even higher”. Interviewee 15 demonstrated the negative effects of not demonstrating the benefit of big data initially by saying; “you find that if companies have had a previous run in where they were sold something, ‘oh this is going to be brilliant’, and then things didn’t work out or they didn’t quite get what they were expecting then you know the sentiment is not so great”. These organisations must now first be sold and made to understand how big data works and how big data will deliver value before they will begin to utilise it properly.

3.4.9. Security and Trust

Data security is crucial for the accuracy of the data as well as the trust it will build with clients or within business units. Interviewee’s 2 and 7 stated how imperative data security was to their organisations with interviewee 7 saying; “the security side of it is massive. So, protecting the data, keeping the data safe, making sure our data doesn’t get breached”. Interviewee 7 went further to say that this will be part of maintaining and building customer trust along the way. This was echoed by interviewee 9 who said it was important to build trust within the community because the believe in the security of their data at that organisation. Interviewee 15 stated that building trust relationships are important because it opens the door wider for further and larger big data projects.

3.5. Results for Research Question 3

Research question 3: What are the biggest challenges/risks to converting big data insights into organisational performance?

The last section of the interview asked participants for inhibitors of big data performance and established the main challenges and risks to a company’s big data performance. The question also attempted to confirm the challenges found within the current literature, and see if there were different challenges that emerged within a South African context across the five industries sampled.

During the interviews, participants mentioned what they believed to be challenges inhibiting big data success. These factors were grouped into constructs and then further grouped into fourteen main themes that emerged from the sixteen interviews conducted. Table 7 below shows the frequency count of how many participants mentioned each theme during their interview. The higher frequency means that more participants experienced this challenge or believe this is a challenge to big data success. This section will now outline each of the themes and their constructs in the order of their frequency.

Table 7. Frequency of Big Data Challenges

3.5.1. Skills and Resources

Skills required for big data is viewed as a challenge by all but one participant. The availability of skills to hire, the development of skills as well as the cost of skills and diversity of skills required to make a success of big data are factors making skills and resources a big data challenge.

Human Resource Scarcity

Interviewee 1 said that skills for big data are broad and scarce by stating; “skill is an important factor, there are a lot of skills required in the big data environment, we think the data scientists, the geniuses, the production engineers, all those things are quite a skill on their own and to find people like that in South Africa is almost impossible”. Interviewee’s 5, 7, 9, 14, 15 and 16 agree and say the skills shortage is a challenge and Interviewee 3 says that big data trained professionals are scarce. Interviewee 13 believes lack of skills hold back big data implementations.

Training and Experience

Interviewee 1 states the importance of investing in training but that most training is online and as people then have to be self-disciplined and passionate about upskilling, not everyone improves at the rate they could if they are not self-motivated to complete the self-training. Training can also be by third parties but there is a cost involved and when budgets are tight, this may be a deterring factor for driving and supporting BDAC. Interviewee 2 concurs with the above sentiment in that there is a lot of self-learning and learning through experience involved with big data by stating “training on data is not anywhere near where we need to be… It is owner managed”. This participant also mentions how big data timelines may be missed if there is missing experience and maturity in the big data employees.

Interviewee’s 7 and 9 mention how attaining big data implementation skill is important and this takes time and experience and can therefore be a challenge. They also agree with the above maturity sentiment and states that a challenge is novice users lacking expertise. This participant also said that practical experience and self-learning is common but added that education may take too long because by the time the person is done studying, the big data environment has moved on from what was taught during the degree.

Retention

Interviewee 1 makes mention of retention being a challenge. This is because skills are scare and therefore there is a demand which reduces loyalty and increases turnover. Interviewee 7 mentions the difficulty in securing and protecting scarce big data skill; “one of our main challenges, is to try and get the resourcing and then trying to keep the resourcing”. And also how traditional HR practices are not appropriate for this environment due to the fast changes in the big data market and the inability to change the job requirements and salary offerings quick enough; “traditional HR practices are way behind, you know the whole got a salary survey, we peg this role at this, and you know our answer is six months ago that was the right salary for that role but now it’s not”.

3.5.2. Proof of Relevance and Value

A challenge many organisations are having is that big data is still hype and seen as a buzzword. This occurs due to the challenge of proving the relevance and value of big data through successes when the organisation is still immature in its big data capability. Interviewee 1 states; “I think it is a buzz word but a lot of people still don’t understand it”.

Value can only be demonstrated when there is a clear vision for big data implementation and a challenge is that many organisations still do not know how to use and exploit big data for their needs. Interviewee 1 said; “they just know that there is big data but they don’t know how to use it or how it feeds into the network”. Interviewee 2 stated that an application needs to first be found for data before the value will be discovered and this can be challenging. Interviewee 9 said that there can be an uncertainty about the benefits and value of big data where people do not know how to make big data work. Interviewee 16 stated; “I think people generally complain about it being a fad, or for a lot of people they don’t really see how it helps it is very difficult to explain it to a number of people to say this is what it can do for you”.

Interviewee 7 stated that a lot of effort is required to ensure no bad big data related experiences occur in the business or this will deter future desire to utilise big data. The same was said by interviewee 9 who mentioned the challenge of ensuring reliability of implementation and ensuring the big data team delivers on its promises when it comes to benefits. Interviewee 9 indicated demonstrating value can be challenging because the big data contribution is not always directly measurable.

Some participants mentioned cost as a challenge to big data but as was established in section 5.4.8 above, cost is only seen as high or a problem when organisations have not yet matured to the point where they are deriving value and benefit that outweighs the cost. Therefore, this challenge mentioned is more a consequence of businesses finding it challenging to prove big data value.

3.5.3. Maturity and Competency Development

A challenge with big data is that an immense amount of competency building needs to take place in order to develop and mature with regards to big data implementation success. This is mentioned by both interviewee 1 and 2 where they both admit their organisations still have a lot of room for big data growth. Immaturity often brings immature processes which can make implementing on insights more difficult. Interviewee 1 states that they are immature with regards to their big data processes and this immaturity is challenging for the business because it takes too long to implement a big data model and provide answers to business problems; “we tend to take a fairly long time before we can put a model into production and that is a big issue”. Interviewee agreed that time to implement is an important factor of competition and therefore maturity to ensure quick turn around times are important but can be challenging.

One participant mentioned that lack of competency leads to the big data unit being unable to do as much as it wants to, or at a level that is needed. Interviewee 12 mentioned that there is a lot of competency that still needs to be developed; “there are some very intelligent, very sharp people out there, but they may not necessarily have the experience and it will take a while to upskill people into the data and methods the business needs”. Interviewee 7 mentions that competency of data scientists is challenging because otherwise they do not always understand the system and can cause errors; “one of the nice analogies I like from data is the whole water system. So, you have a reservoir, a water purification plant and then it gets consumed all over the place. If the water purification plant messes up, you can poison the whole city. That is the kind of problem we are trying to deal with”. Interviewee 4 stated that lack of competency in the country is a challenge and still needs to be developed.

3.5.4. Politics and Sentiment

Politics can occur in an organisation and be a challenge to big data success. Some people may have a negative sentiment towards big data because they do not want to change their way of doing business or risk losing the power they have if big data plays a larger role. Interviewee 9 describes this as complacency and bureaucracy challenges; “You know they have been running a successful company for so long, ‘really, am I going to get benefit from this?’ The market works. That is the sort of complacency. You can’t blame them because it is established companies and then they have got too much bureaucracy”. Interviewee 15 describes this as a mindset being a predominant obstacle to big data. More politics challenges are seen around sharing of data between business units or departments within organisations. Interviewee 7 stated; “so it is politics in terms of people wanting to own stuff, you know it’s mine, it’s not yours”. Interviewee 2 mentioned the challenge of fear and mistrust around information sharing because of risk aversion and benefits being overshadowed by dangers. Interviewee 6 said it is a challenge when key stakeholders withhold data; “people sitting on data and not wanting to give it to you”.

Another sentiment that challenges big data is the privacy and security of information. Interviewee 9 said that people do not want their information known and are afraid of organisations knowing them to the extent big data can provide knowledge; “there is a fear of big data”. Interviewee 7 echoes the challenge and says that you need to produce actionable insights from big data using many sources for the best knowledge but while respecting privacy. This participant provided a good example of privacy protection; “In the US they sent a girl a pregnancy something or other and the father freaked out because his daughter was 15 but she had just bought a pregnancy test” and called this “creepy data”, which is what you need to stay away from and there can be a fine line.

3.5.5. Sophistication and Complexity

Big data systems can be complex in nature and so can big data outputs. This can be a challenge to the business when they do not know how to properly utilise the system or interpret the output. Interviewee 7 stated; “it is guys who don’t know how to use the tools properly… but we are having to put a lot of checks and balances in to protect the platform from the users of the platform… That is probably our biggest problem at the moment”. Interviewee 3 said that poor representation and interpretation of results is a challenge because results are difficult to consume and not user friendly. Interviewee 5 stated; “Usability is one, you know it needs to be easy and accessible for people to use it”. Interviewee 12 mentioned that the results can be challenging to grasp; “it’s maybe the results, that are sometimes different to what people have seen before. So it is non-traditional, it doesn’t fall into the normal scope of things, so it is just a bit more challenging, the result – and sometimes it challenges conventional thinking”.

3.5.6. Tasks, Routines and Processes

Big data tasks that need to be performed in the background in order to ensure accuracy of results are time consuming and this is challenging when the business wants results quickly. Interviewee 7 said this challenge is about getting the data ready, accessible and available quickly while ensuring data quality; “I would say currently data onboarding is probably our most time. So, we spend most of our time on data onboarding, data preparation…”. This participant also mentioned the time to test or check the model’s users wish to run for practicality and errors as well as stated; “between getting our data clean and protecting our data I would say that probably consumes about 50 or 60% of our time”. Another challenge this participant mentioned was the power and lead time or latency of the system delaying answers. Interviewee 1 also talked about the time required for data preparation tasks being a challenge.

Interviewee 10 stated that meeting business response time is a challenge because end users expect quick responses and answers, and don't want to wait long for BD solutions to business problems and queries. A lack of clearly defined and optimised big data processes is a challenge and this is mentioned by interviewee 1 who says this opens the door for issues to arise and they challenge they are now having is to optimise their big data processes.

3.5.7. Accuracy of Big Data Outputs

Big data may be relatively accurate but due to the dynamic nature of big data it is not possible to be 100% accurate or flawless (Interviewee 2). A good example is forecast or prediction, which is only as good as its input data and may not be an exact replica of the actual future. A lack of understanding can then cause business to distrust big data in the future. A challenge is to try and get big data inputs as well as insights as accurate as possible to gain business trust and produce the outcomes in organisational performance desired. Interviewee 2 gave an example; “if you tell someone that this is Stephen and he’s looking for a fibre line (for example) and they phone Stephen on that basis and Stephen says he’s not interested, you then have a problem”.

Interviewee 7 mentioned an important point for accuracy, which is not that all mistakes can be avoided but that the challenge is recognising mistakes in time to avoid implementation on a mistake taking place; “Sometimes the results are very subtly skewed and it takes you six months to work out that it is skewed and you have done a whole lot of bad business, or you have made a lot of bad decisions. So, that is the massive risk, the masses of big data, and getting it wrong”. Interviewee 5 and 6 also mentioned data integrity as a big challenge of big data.

3.5.8. Legacy Systems

A challenge for big data can be converting or integrating into the legacy systems an organisation uses (interviewee 1). Interviewee 7 agreed and said this can be challenging, time consuming and require a lot of new deployment of tools and infrastructure if legacy systems are to be replaced. Interviewee 11 said that the transition from older, conventional or legacy systems may also be an issue and interviewee 12 agreed this is a challenge. Interviewee 9 stated; “we are already starting to build up legacy which does complicate things. So, we are continually looking at ways to reset our structures to get all the data and systems effective” and “their legacy systems, you have to do something, if you are looking at a big organisation, for them it is very tricky to get started. So if you have something where there never was anything it is easier, because you can start collecting the data in the way that you want it; as soon as you have something well then you are a bit stuffed, they come up with these IT design systems and it is three years to implement, and you think ‘what, like three years?’ Like an Ipad came in within a year everybody has an Ipad. You know, technology moves so fast, how can you justify system implementation taking three years, yet in the corporate world that is where you are sitting”.

3.5.9. Regulation

A few participants, including interviewee 12, seem to be aware that although the data market is not currently regulated in South Africa, it may become regulated in the future. As a consulting company this awareness is greater as it may affect the business and cause challenges. This was mainly discussed by interviewee 2 who mentioned the implication for then needing to register and the restrictions it may place on what data they can acquire. This participant believes this may hinder big data progression and Interviewee 1 agreed when they said that privacy and regulation may inhibit the freedom an organisation has to explore within big data and big data implementation. Interviewee 9 stated that they view regulation as a challenge to embrace and opportunity to find solutions so although they believe it is a challenge, they do not feel it will hinder their big data ability but rather provide them with an opportunity to discover a competitive advantage.

3.5.10. Management

Management of the big data process from beginning to end is important and if not done correctly can cause problems. Interviewee 2 mentioned how easily data scientists can “run-away” with big data and so it needs real management to rein it in and focus on what is necessary for business objectives. This participant mentioned how big data requires constraint and focus because it is too easy to collect too much data and spend too much time on models and datasets. This can decrease profitability of the big data project. Management of big data inputs and outputs is required as well as effort required and the level of output desired. Interviewee 7 mentioned the challenge of needing to manage expectations in terms of what big data can do and used an anecdote; “No, you can’t have that because Star Trek computers don’t exist yet”. Interviewee 9 mentioned the managing of structures, processes and speed of decision making.

3.5.11. Data

One of the data challenges was mentioned by interviewee 2 who stated that data is dynamic and changes continuously. It is difficult to keep updated. This participant used some examples to demonstrate this point; “People go from being fitness freaks to being super unhealthy to being fitness freaks. People go from wanting a BMW to buying a BMW to getting into debt selling it and buying a Polo and end up financing a car again”.

Initially, establishing a basic database can be a challenge and data required for big data can be expensive to procure through buying or researching. Interviewee 2 also mentioned data misconception as a challenge; “I think people have a misconception around their data. I think people believe that they have all got the best data. The only way you can have the best data is to buy the updates from someone who has done the updates over and over again”. Data is also a challenge in that to remain competitive in the future, it may not be sufficient to utilise only one data set. Interviewee 2 stated, “You can’t just use one set”. Interviewee 2 and 9 mentioned the challenge of ensuring the data is structured and ordered correctly for the best results. Interviewee 2 also mentioned the fact that there is still missing and incomplete data where not all data is readily available or directly capturable. Finding ways to find and capture this data can be a challenge.

3.5.12. Awareness and Involvement

There is a big data challenge that exists to create awareness and drive involvement from more employees in all business units or divisions in organisations. Interviewee 1 stated that big data implementation is predominantly driven by data scientists and business problem owners. Interviewee 7 said a challenge was the general understanding of big data throughout the organisation because it varies; “Across the organisation, depending on some divisions we deal with, there is massive buy-in and understanding; others are more focused on what they are doing”. This participant also said; “a lot of areas of the organisation are benefiting from it, they are just probably not aware of what is going on behind the scenes”, which shows that data is provided without the knowledge of where t originates and this creates an awareness and disconnect challenge. Interviewee 9 revealed that their organisation is similar in that some business units do understand and some do not understand big data and its application and value; “So to a certain extent yes, but also to a certain extent no. So, they don’t necessarily see the big data link”. This participant also said that those that do not work directly with the data or have exposure to it, for example their call centre, do not necessarily follow or understand big data.

Communication around big data is required to further awareness and interviewee 1 stated that the organisation is trying to get horizontal as well as vertical communication correct for big data.

3.5.13. Ethics

There is an ethics challenge when it comes to what data to collect and what data to buy and sell. Interviewee 2 revealed that views on what is ethical are not consistent and many organisations are afraid of getting into trouble if they are deemed unethical with data collection and usage; “some people disagree with some things and call it unethical. There is always ethics involved in data”.

Interviewee 8 said; “People keep on bombarding the same subject looking for the same information because the first company or person that did this information from the same subject, is selling it, or I am made to pay a price to get it, so let me go back to the same person and try to get it myself. And by the time you do that you are not getting a willing participant.”. The challenge is trying to all collect the same data and bombarding customers for data, who then get irritated and do not want to share their data. Clear ethics around sharing and selling of data could solve this issue.

3.5.14. Research and Development

Conducting research and development within the big data environment is a challenge. Big data personnel are generally too busy doing tasks to prepare for big data projects and completing the projects approved to solve a presented business problem, that there is no room left for exploration in order to become more innovative with big data analytics. Interviewee 1 states that exploration around big data is mostly task related and part of job descriptions. It is not necessarily exploring new big data avenues. Interviewee 7 has sad that a challenge is finding time for big data planning and new big data implementation over and above the running of the big data platform; “The other challenge is we still have to run the organisation and do all of the existing stuff and there is a lot of work that people are doing, so the challenge is to do all of this and do what we are currently doing and prepare for the future”.

3.6. Conclusion

The results of the interview questions were presented in this section per research question. Themes and constructs found from the coding process were presented and discussed under each research question. The situation was established in the first research question, followed by main success factors of big data in research question 2 and lastly challenges presented in research question 3. The next section will discuss these findings in the context of the literature presented in section 2 and a proposed framework for success factors of big data will be presented based on these findings.

4. Discussion of Results

The themes are compared and contrasted to those identified from literature thereby contributing to the body of knowledge of big data success factors and challenges with regards to achieving organisational performance.

4.1. Discussion of Results for Research Question 1

Research Question 1: What is the general understanding of big data and its current situation?

Research question 1 aimed to identify the perceived understanding and current view of big data. This included the understanding of what big data means, what it adds to the organisation and how companies go about utilising big data. It was also established what organisations view as the main uses and application of big data. It was important to understand this context in order to then know what success factors (research question 2) are required to achieve utilising big data successfully in this way and what challenges (research question 3) are inhibiting organisations from utilising big data successfully.

4.1.1. Big Data Understanding and Applicable View

Understanding

Participants views varied on what big data means, with some participants taking a more academic definition while others a more practical definition. This is consistent with literature where no consensus on a single definition for big data exists either (Sheng et al., 2017). Participants in leadership roles tended to look at big data in terms of what it meant for the business while more technical interviewees saw big data more in terms of the data and technology itself. Many participants, when asked about their understanding of big data would go straight into tasks and uses of big data rather than talk through a holistic understanding of the concept, showing that many people only understand big data from the perspective of where they have seen it applied.

Most participants referred to the great volume of data when describing their understanding of big data but literature tells us that the word “big”, does not imply that big data is about the size or amount of data only, but rather the complexity of the data and how the data is captured and managed (Caesarius and Hohenthal, 2018). This fact was not articulated by the majority of participants. A few big data consultants or high-level business executives expanded on their understanding of big data to show that it is equally about more accurate information and the ability to build insight from data knowledge for better decision-making. This is consistent with what George et al. (2014) describe in the literature that big data is its ability to produce knowledge and generate insights that were not possible with traditional methods. Gupta and George (2016) also mention that the ability of the data to provide smart insights is more accurate of what distinguishes big data from traditional data and these few participants therefore showed their better understanding of the big data term.

Only a couple of participants took a more academic view of the definition of big data, showing their knowledge that big data can be explained in terms of the 5 V’s. Whereas one participant was able to describe the variety component of big data over and above volume the other participant managed to list but not detail velocity and variability as well. This is a start to understanding what literature describes as the definition of big data in terms of a number of V’s. Literature describes big data in terms of three V’s of volume, variety and velocity (Braganza et al., 2017; Sheng et al., 2017;Erevelles et al., 2016; Gupta and George, 2016) or five V’s, which adds value and veracity (Sheng et al., 2017;Wamba et al., 2015). The Five V’s of big data definition is extended to include valence and variability and become the Seven V’s definition, which provides an all-encompassing view of big data (Erevelles et al., 2016; Günther et al., 2017; Braganza et al., 2017; Janssen et al., 2017).

In general, this research aligns with literature (Sheng et al., 2017) in that the understanding of big data is not well articulated or consistent, with the most focus still being on the volume of data. Since most participants focused on functional aspects of big data when describing their understanding of the concept and this research is about the functional application of big data initiatives, Comuzzi and Patel (2016) definition of big data is still the most relevant to the research findings and states what the few higher-level business participants and consultants began to describe as their understanding, “the ability to harness information in novel ways to produce useful insights of goods and services of significant value and to extract new insights or create new forms of value”.

Importance

The overwhelming majority stated the importance of incorporating big data successfully in order to be competitive. There was an emphasis that this is for all types of organisations and especially customer focused businesses. Everyone believed that the environment required the use of big data to not only succeed but to survive and that organisations not utilising big data now are in trouble in the future. There is also a drive for companies to make BDAC a core competency and this is believed to be the way forward to ensure competitive advantage. This finding is in line with the literature which explains that there is an exponential rise in realisation of the importance of big data due to value creation opportunities the improved data collection, storage, processing and transportation technologies have allowed and the increased digital dependency arising from technology developments (Akter et al., 2016; Kabir and Carayannis, 2013; Caesarius and Hohenthal, 2018).

One participant added a new aspect, not seen in the previous literature about the consolidation of industries occurring due to big data becoming a core competency and this changing competitors from other industry companies to top big data companies where disruption could occur. Companies with a BDAC core competency that collect and utilise big data well are the companies where competition is going to emanate from in the near future. Being mature in big data competency will be key to compete and survive in this new dynamic.

4.1.2. Data Capturing

In general, most organisations use their business transactions and internal systems to capture data. The predominant sentiment was that companies do not utilise all the data that they generate. This is inline with the literature that also says big data generates more data than organisations are able to use and is characterised by unprecedented large volumes, high velocity and extensive variety (McAfee, Brynjolfsson, Davenport, Patil, and Barton, 2012; Caesarius and Hohenthal, 2018; Günther et al., 2017; Erevelles et al., 2016).

Mostly organisations are capturing and using structured data for insights but there does exist a lot of untapped potential in unstructured data. This finding is again in line with the literature where big data is now a mainstream activity of an organisation since routine business today generates an immense amount of structured and unstructured information (Janssen, van der Voort, and Wahyudi, 2017;Erevelles, Fukawa, and Swayne, 2016; Kabir and Carayannis, 2013). Knowing how to utilise the unstructured data is more limited and more big data mature organisations are starting to drive towards utilising more unstructured data.

The literature mentions how organisations are also able to capture communication data due to the general adoption of mobile devices, tablets, online social media activity (e.g. Facebook, Twitter) and use of instruments, machines and transmissions (Kabir and Carayannis, 2013; EY, 2014; Wamba, Akter, Edwards, Chopin, and Gnanzou, 2015; Liu, Wang, and Lin, 2017; Braganza, Brooks, Nepelski, Ali, and Moro, 2017). This was not a finding within this research as most organisations had not been able to utilise this ability but were looking to make use of communication data for sentiment analysis in the near future.

4.1.3. Politics

Organisations that are immature in big data implementation feel that more politics may arise as big data becomes more prevalent but more mature big data organisations report the opposite in that politics has either been resolved or the strong leadership drive leaves no room for politics to interfere. Politics around legacy systems exist due to people not wanting to change their current ways of working or risk losing power that they may currently possess. Politics can also exist around which new system to adopt and current or fostered relationships with technology providers.

4.1.4. Process

Most organisations initiate big data projects by an identified business problem but a more mature organisation will all look at business needs, objectives or targets of business units to trigger a big data project. After a project is identified a business case is required to determine if the project has value and the projects are then prioritised for analysis and implementation. More mature organisations are beginning to find methods to reduce the burden and bottlenecks at the data scientists and data teams by encouraging and enabling methods for business units to own their own big data projects and complete the analysis through big data platforms themselves.

Another initiator of a big data project is innovation, which this study found to be rare amoungst organisations. This is also not found in literature as literature focuses more on big data providing insights that can be used to innovate (Kabir and Carayannis, 2013; LaValle et al., 2011; Sheng et al., 2017) and not on innovation in the form of exploring, capturing and utilising the data differently or in innovative ways to find better procedures or new uses for big data that have not been seen before.

4.1.5. Value

Value is seen in terms of the business value that data can bring to a company. Most companies have struggled to derive value from big data while a few have managed to delivered plenty of value within their organisation and continue to do so. The majority of the companies are in line with literature which states that organisations find it difficult to sustainably use BDAC to achieve value and improve organisational performance (Comuzzi and Patel, 2016; Elgendy and Elragal, 2016; LaValle et al., 2011) but the few companies that are exceptions have managed to achieve sustainable value and improved performance from their BDAC and these companies continue to grow and improve their BDAC because more value generated allows them more freedom, trust and resource allocation.

There is a common sentiment that big data can do more and provide more value, even in the successful organisations. This research also highlighted the value of the data itself and that data is truly sought after. Building data sets and doing data analytics has a positive impact on the economy in two ways; firstly, through selling data and secondly, by providing businesses with valuable information that increases their efficiency, thereby positively contributing to the economy. This is a new viewpoint on value that has not been seen in the literature and needs to be understood due to the increasing awareness of the ability to buy and sell data to increase the knowledge base from just what is captured from internal systems.

4.1.6. Training

Training for BDAC is mostly self-study and learning by experience. Some organisations enhance training through online courses, third party training, programs and seminars. Companies that are serious about developing their BDAC skills have a training budget and resources that are allocated to big data skills enhancement and development. More mature big data organisations take training to the next level by having combined sessions where everybody comes together to find big data solutions and can then learn skills from each other. This is a particularly useful method to utilise for more complex problems that require more complex solutions and techniques. In general, little is done proactively to facilitate training of big data analytics skills and this could be a hindrance to the progression of BDAC in organisations. Wang, Kung, and Byrd (2018) recommend training personnel in big data analytics as a means to success but even the literature is scarce on talking about training when it comes to big data analytics and no details about what type of training can be found. This is an area that therefore needs to be addressed and better understood so organisations can be advised on what types of training will best bring about skills development. A well performing organisation could be a great example to explore their methods further as they have been successful in developing skills well enough to sustainably derive value from big data initiatives.

4.1.7. Organisation

Most organisations have not restructured to accommodate the introduction of BDAC. There was no consensus on whether or not a restructure is likely in the future as BDAC and big data projects increase. It is found that organisations that had managed to grow in big data maturity had gone through some organisational restructuring to allow big data to be the centre for decision making. This may provide some evidence that restructuring to best suit big data as a core competency is important to develop and grow in big data maturity. This included combining divisions or replacing divisions in order to allow a centralised big data team to form as well as introducing more big data professionals to each of the business units. It is important for big data teams to have both an IT and business aspect and for these two divisions to work together closely and begin to teach and understand each other better for efficient big data analysis. There is no current literature on restructuring to suit big data capabilities and processes.

4.1.8. Application and Uses

The main applications of big data to surface from this research are that of marketing opportunities, customer satisfaction, operational efficiencies, innovation, predictions and forecasts, and increasing revenue. Marketing refers to customer insights, reducing uncertainty, improving accuracy, targeting customers, understanding customers better, trend analysis, understand markets and directed or intentional marketing. Customer satisfaction consists of offering the customer exactly what they need and sentiment analysis. Operational efficiencies include; lean processes, improve response times, compliance, reporting, improve systems, automate transactions, fraud detection, evaluate KPI’s and HR hiring. Innovation is comprised of products and services development and improvement, and managing risk. Predictions and forecasts are used for customer behaviour and increasing revenue is due to unique offerings that can become available.

Most of these applications featured strongly in the data collection process and this research has demonstrated that there is a strong understanding of what big data can be used for in an organisation. How to achieve success in each of these applications is less known and the subject of research question two.

There is a vast number of papers in recent literature on big data and its applications, demonstrating the significance of big data (Chen, Chiang, and Storey, 2012; Chen and Zhang, 2014; Chen et al., 2015; De Mauro, Greco, and Grimaldi, 2015; Hashem et al., 2015; Wamba et al., 2015; Akter et al., 2016; Elgendy and Elragal, 2016; Erevelles et al., 2016; Ram, Zhang, and Koronios, 2016; Braganza et al., 2017; Gil, Song, Aldana, and Trujillo, 2017; Lee, 2017; Sheng et al., 2017; Vidgen et al., 2017; Wamba et al., 2017; Saggi and Jain, 2018). Many of these applications have been mentioned within this literature but there are some uses mentioned that have not featured strongly within literature and can be attributed to the fact that new uses are found all the time as technology improves and business units themselves utilise big data to solve their specific problems. Directed or intentional marketing is an example of one of these uses as it is a fairly new concept that would not be possible without a large database of knowledge on customers. Fraud detection and HR hiring are also seldom mentioned in literature and these are operational efficiencies which come from specific departments within a company.

4.1.9. Advanced Big Data

The research revealed that the sentiment around what advanced big data is, is still immature in thinking where the biggest focus was on the advancement of the technology and achieving automation, AI and machine learning. The term advanced was definitely relative based on the level of maturity and success of big data within the organisation. Less mature organisations struggled to see the potential for advanced big data that more mature organisations could. The literature describes organisations maturity and what their capabilities are at that level of maturity in terms of three maturity levels and at the top maturity level is where they will conduct the advanced big data analytics (LaValle et al., 2011). The transformed level (top level) is where organisations now organise their other capabilities to align with the BDAC requirements and use BDAC to differentiate themselves in the market (LaValle et al., 2011).

A novel view found in this research is viewing advanced big data across the big data analytics process where big data competency has the ability to exploit data and derive exceptional value from it, big data implementation has the ability to apply and use advanced big data tools and techniques and big data reporting has the ability to get insights to the clients or departments in user-friendly ways for them to action. This viewpoint is important as it draws attention to the importance of the whole big data process from the time data is collected to the time solutions are implemented and their value and benefit reported for measurement and feedback. Not completing the whole process well would mean not achieving success in big data projects.

4.1.10. Conclusive findings

The huge importance of having BDAC within an organisation is unanimously understood. There is generally a very surface level and poor understanding of exactly what big data means but functional understanding of what big data can be used for is much more comprehensive and organisations know what tasks can be achieved by utilising big data. The lack of proper understanding of big data may hinder their ability to properly utilise big data for each of the identified applications of big data and prevent the organisations from improving their maturity. Organisations generally do not use all the data they capture and since organisations mostly only use their structured data, there is huge untapped potential for utilisation of the unstructured data available to organisations. Even though organisations understand what uses can bring value, actually deriving this value is difficult and many organisations are struggling to show benefit in their big data initiatives. Training is mostly informal and with no directed methods to properly train competency in big data, improving could be hindered. Restructuring the organisation to better suit the requirements of the big data process is important to increase big data maturity. Although a lot is known about the uses of big data, what is required to make each of the uses successful in improving organisational performance is less explored and unknown to many organisations. This will be covered in research question two.

4.2. Discussion of Results for Research Question 2

Research Question 2: What capabilities are required in order to be successful at translating big data insights into organisational performance?

Research question 1 established the current situation with regards to understanding and uses of big data for value driving. Research question 2 establishes the perceived capabilities and reasons as to what enables an organisation to successfully use their BDAC for better organisational performance.

4.2.1. Success Factors of Big Data

The applications of BDAC are well documented in the literature and in section 6.1 above. BDAC has potential to improve organisational performance and create competitive advantage but there has not been a lot of success in this plan (LaValle et al., 2011; Wang, Liu, Feng, and Wang, 2014). This research concurs with that literature sentiment and very few organisations were found to be successful at demonstrating value and benefit from big data initiatives and as a result were unable to translate big data projects into organisational performance. The organisations that have managed to sustain organisational performance success from big data are a great source of understanding as to what they are doing well in order to isolate success factors that organisations can implements in order to achieve big data success.

The main success factors listed in the literature for big data are; management, people, technology, culture, strategy, processes, basic resources and governance (McAfee et al., 2012; Davenport and Dyché, 2013; Wamba et al., 2017; Kiron et al., 2014; Gupta and George, 2016; Wang, Kung, and Byrd, 2018; LaValle et al., 2011; George et al., 2014; Sheng et al., 2017; Braganza et al., 2017).

Literature refers to the importance of culture (Kiron et al., 2014); Gupta and George, 2016; Wang, Kung, and Byrd, 2018), but culture is broad and it does not sufficiently expand on what is important about culture and what aspects of a culture will lead to success in BDAC. The analysis in this report talks to more specific components that can be a part of a company’s culture, such as the openness and awareness of the organisation. Openness does seem to be a first step toward starting with big data and not enough to increase maturity in big data alone. Even organisations just starting with big data mentioned how their companies and the employees were open to big data and aware of big data. The companies have all bought into the notion of big data and its potential value and there is positive sentiment about building BDAC to improve organisational performance. Participants from companies that mainly focused on openness to big data as what is happening and required for big data are the less mature organisations where big data has either not been successful or they company has just started to utilise big data. Openness within a company culture and in the minds of all the business units and employees therefore seems to be an order qualifier for developing BDAC and something that is predominantly existent in most of the organisations from which a participant was interviewed.

The next success factor, which organisations require in order to take organisations from beginner maturity or big data out of just the “hype” mentality is commitment. Commitment in terms of actions that show the organisation is serious about making big data work. These are things such as restructuring to form a centralised BDAC team as well as deploying big data skills throughout business units. The literature mentions the importance of skills in BDAC (McAfee et al., 2012; Davenport and Dyché, 2013; Kiron et al., 2014; Wamba et al., 2017; Gupta and George, 2016; Wang, Kung, and Byrd, 2018) but never in conjunction with the deployment of such skills in a manner that shows commitment and utilising this to drive behaviour within business units to better utilising big data for performance. Commitment is a new success factor that has been derived from this research and has not been tested as a success factor in the literature before.

Organisational success factor constructs determined in this research are strategy, leadership, commitment, monitoring, measurement and incentivising, human resources and big data champions. This category covers a few of the mentioned success factors from literature and therefore mostly aligns to what literature has found to be success factors. The new constructs not found in literature are commitment (as discussed above), monitoring, measurement and incentivising and big data champions. These constructs are important for success and this research has demonstrated that in order for organisations to mature and improve their big data competency, they are necessary and without them, organisations tend to be behind or fail with regards to big data implementation.

This research found that organisations with clear big data strategy and leadership support, are more successful when it comes to big data. Companies need a strategy and vision for big data and big data implementation. For big data to be successful it needs to form part of the overall, long-term strategy as well as the digital strategy. Organisations need to make decisions based on data whenever possible and ensure there is a clear goal and vision within the organisation to help define what big data value and value generation looks like. Part of a strategy process is benchmarking and companies need to do this against top data companies and internationally when it comes to BDAC. This is consistent with the literature on strategy being important for BDAC.

Leadership support is extremely important because if big data is seen as a vision of leadership, the rest of the organisation will make it a priority to implement. This is consistent with literature (McAfee et al., 2012; Davenport and Dyché, 2013; Wamba et al., 2017) but literature lists more managerial tasks rather than leadership skill and it is leadership skill that is more important and therefore this research is refining and improving on this construct from literature. It is important that leadership takes place on each big data project itself where someone takes ownership of the project and holds the accountability for its success. This is also an aspect of leadership not mentioned in literature. Big data champions also help leadership to drive big data and ensure an awareness of the big data existence and capability. The champions help to further implement big data and disseminate big data knowledge within business units.

Measurement and incentivisation of big data helps to keep track of big data and its progress. This is essential to properly manage big data and is not a construct that has been mentioned in the BDAC success factor literature. Skilled individuals with the correct aptitude will provide organisations with an immediate competency advantage. Introducing experienced big data people to the organisation increases the chances of finding value from big data and implementing big data successfully. This research on skills is in line with literature (Gupta and George, 2016; Wang, Kung, and Byrd, 2018) and concurs with literatures findings that this is a success factor of big data.

Companies that successfully implement big data are able to derive value from big data, so that the benefit of big data exceeds the cost. To do so, they build on previous analytics experience, proper project management experience, and may involve outside experts. Demonstrating value will create by-in and this is important for involving everyone in the organisation to achieve the openness and awareness needed to begin with BDAC. The more every business unit and person in the organisation is aware of big data, the greater its application, and the more worthwhile projects are suggested. It also increases buy-in, if people can practically see the benefit from big data. In this regard, sending big data champions throughout the organisation to identify common and everyday problems big data can help with, also has good success.

Development and competency building are important to create success with BDAC. There is a second level of success activation, when the organisation can identify ways to advance its big data competency. It unlocks more advanced levels of value generation. Cultivating data competencies for big data is important for accuracy of analysis and therefore outcomes. Implementation competency is crucial to big data success because insights mean nothing if the organisation cannot act upon them. Existing data analytics experience is important because it helps tremendously to be able to derive value from big data projects and to identify where and how big data can offer value. It forms a starting point from which to understand the power of data. Smaller steps are taken towards using big data for problem solving by starting small, proving value and then scaling-up the big data operation.

4.2.2. Big Data Success Factor Model Development

The literature also breaks down BDAC into three dimensions to provide competitive advantage; big data analytics management capability (BDAMAC), big data analytics infrastructure capability and big data analytics talent capability (Wamba et al., 2017; Akter et al., 2016; Davenport and Dyché, 2013; McAfee et al., 2012). Niland (2017) uses these three dimensions and fourteen constructs in a model that he tests quantitatively to determine the variance in organisational performance and he finds 64% of the variance is attributed to these constructs (see Figure 2 below). This means that the constructs that make up the success factors of big data have not yet been fully determined and the constructs found in this research can be used to find constructs to be tested quantitatively in future research (see # below). It is hopeful that these constructs will account for more of the variance in organisational performance and add to the literature models to build the comprehensive model with which to take the research into success factors of big data forward instead of current trends which is many different researchers testing different models and the research progress is not driving forward.

Figure 2. Model of BDAC versus Organisational Performance from Literature

Source: Niland, 2017

Figure 3. Constructs and Themes of Success Factors of Big Data Found in this Research

4.3. Discussion of Results for Research Question 3

Research Question 3: What are the biggest challenges/risks to converting big data insights into organisational performance?

The last section explores inhibitors of big data performance and establishes the main challenges and risks to a company’s big data performance. The research question also attempted to confirm the challenges found within the current literature, as identified in section 2, and see if there were different challenges that emerged within a South African context across the five industries sampled.

4.3.1. Big Data Challenges

Literature identified the following challenges; Leadership, talent management, technology, decision-making, company culture, strategic positioning, infrastructure, data access, skills, decision-making, data privacy and regulation, processes, lack of understanding of maturity and implementation complexity (McAfee et al., 2012; Comuzzi and Patel, 2016; Akter et al., 2016; Wamba et al., 2017; Frizzo-Barker, Chow-White, Mozafari, and Ha, 2016; LaValle et al., 2011; Gupta and George, 2016).

Challenges that were identified in this research which are new and add to the literature are; proof of relevance and value, awareness and involvement, politics and sentiment, regulation, accuracy of big data outputs, ethics, research and development, and legacy systems.

Skills and resources: Big data skills are scarce and a lot of training and expertise building must still occur. The availability of skills to hire, the development of skills as well as the cost of skills and diversity of skills required to make a success of big data are factors making skills and resources a big data challenge. Skills scarcity, training and experience as well as retention are problematic with big data human resources. This is consistent with the literature that lists skills as a big data challenge on many occasions.

Proof of relevance and value: A number of organisations struggle to get off the ground, because they are unable to find proper application and use for big data, or they struggle to successfully implement big data. They make significant big data investments, with little to eventually show for it. They hear of the benefits of big data, but are unable to realise them. This is taken to be a factor of improper big data project management, lack of big data competency and skills (experienced BD people), and lack of data experience. In this regard, external big data consultants can have good results, provided that big data and the outcomes are properly managed.

Maturity and competency development: In a number of organisations, also more successful ones, big data is simply still immature, with limited competency. Competency must still be developed. A challenge with big data is that an immense amount of competency building needs to take place in order to develop and mature with regards to big data implementation success. Immaturity often brings immature processes which can make implementing on insights more difficult. This concurs with the literature which states the difficulty of moving between levels of maturity (LaValle et al., 2011).

Politics and sentiment: Business units may hold back data. Fear may cause people to not want to participate when it comes to big data and its implementation. Some people may have a negative sentiment towards big data because they do not want to change their way of doing business or risk losing the power they have. More politics challenges are seen around sharing of data between business units or departments within organisations Another big data challenge is the privacy and security of information.

Sophistication and complexity: Big data is simply sophisticated and complex. It is by nature not easy to implement, and can get very complex. Literature also list complexity of implementation as a big data challenge.

Tasks, routines, processes: Big data involves a lot of tasks, routines and processes that must be optimised, and perhaps automated, to free up scarce resources. Big data tasks that need to be performed in the background in order to ensure accuracy of results are time consuming and this is challenging when the business wants results quickly. Literature also mentions bad processes as a challenge to big data.

Accuracy of big data results: Big data generally has a high success or accuracy rate (80%), but it is still not perfect. It may also happen from time to time that the results are skewed, and it takes a significant while to pick it up. A lack of understanding of accuracy limitations can then cause business to distrust big data in the future.

Legacy systems: A number of organisations have legacy systems, that make the transition to big data systems and platforms difficult.

Regulation: Big data faces or may face regulation, that can impact its application and use, and that can constrain it.

Management: Big data requires proper management. It involves the business context, as much as the technical aspect of big data. Big data requires constraint and focus because it is too easy to collect too much data and spend too much time on models and datasets. This can decrease profitability of the big data project.

Data: Issues around data include the availability of data in some cases. Data is dynamic, and changes all the time. An initial basic database must be developed first. It may be necessary to triangulate and integrate a number of databases.

Awareness and involvement: It frequently occurs that big data awareness and basic knowledge are lacking throughout organisations, with very little basic training on big data and big data fundamentals, principles, and utility for ordinary employees. It means the majority within organisations are left in the dark, and cannot actively contribute and make use of big data. Communication around big data is required to further awareness.

Ethics: Big data typically has ethical concerns associated with it. There is an ethics challenge when it comes to what data to collect and what data to buy and sell.

Research and development: There is too little time, and by extension resources, available to advance and further big data implementation and competency. This also restricts big data development and growth. Too much time is spent on merely running platforms and too little time is spent on advancing big data thinking, competency, and solutions.

5. Conclusion

5.1. Principle Findings

The huge importance of having BDAC within an organisation is unanimously understood. There is generally a very surface level and poor understanding of exactly what big data means but functional understanding of what big data can be used for is much more comprehensive and organisations know what tasks can be achieved by utilising big data. The lack of proper understanding of big data may hinder their ability to properly utilise big data for each of the identified applications of big data and prevent the organisations from improving their maturity. Organisations generally do not use all the data they capture and since organisations mostly only use their structured data, there is huge untapped potential for utilisation of the unstructured data available to organisations. Even though organisations understand what uses can bring value, actually deriving this value is difficult and many organisations are struggling to show benefit in their big data initiatives. Training is mostly informal and with no directed methods to properly train competency in big data, improving could be hindered. Restructuring the organisation to better suit the requirements of the big data process is important to increase big data maturity. Although a lot is known about the uses of big data, what is required to make each of the uses successful in improving organisational performance is less explored and unknown to many organisations.

Big Data still has tremendous potential that is currently untapped, and yet to be exploited. Even more successful big data organisations attest to this. There is a lot of development behind big data. A number of organisations have also identified more advanced ways in which big data can unlock value. There is still a lot of big data competency to be cultivated but this generally takes time and grows as maturity is developed. Big data also still requires a lot of optimisation and automation of routines, processes and tasks. Automation will facilitate openness and allow non-technical employees the opportunity to themselves use big data to solve heir business problems. Most organisations that get big data off the ground, report benefits that far exceed costs and this demonstrated value drives interest in big data initiatives and creates the buy-in for the organisation. When an organisation is immature, it is important to start small and complete small projects successfully to demonstrate the value and benefit if big data. This will create the support for more resources in order to grow into bigger projects and develop big data maturity.

Common use cases of big data include using big data to improve customer offerings and satisfaction as well as marketing efforts and operational efficiencies. Proper strategy, leadership, commitment, champions and project management (implementation), are some of the organisational factors that promote big data success and the ability to successfully utilise big data to achieve the uses mentioned above in a way that promotes improved organisational performance. It also helps if the organisation has prior data analytics experience, or if the organisation can bring in experienced big data people, that are properly supported and managed. Prior data analytics experience allows people to better understand the data environment as well as leaves a smaller gap to bridge to the big data platforms. Some organisations have achieved significant buy-in with regards to big data and these organisations are maturing quickly and reaping the benefits big data affords. These organisations can still be performing better but skill shortages and competency level are two of the main reasons preventing the higher level of performance possible in these organisations.

There is still very basic training on big data analytics, big data implementation and utility training that the ordinary employee lacks. Training needs to be improved and better managed because self-training ensures that only some employees with a direct interest will take the time to upskill. Skills are a major limitation when it comes to big data. Skills are scarce and therefore expensive and retention can be difficult when there are many opportunities arriving for employees. Organisations follow a number of strategies to attempt to counter the skills problem, including training people themselves, using open source software that has self—study online training and can be learnt on the job and it is important to remove the bottleneck on the scarce skill employees by increasing the ability of the business unit employees to utilise the big data platforms themselves and solve their own business problems using big data. This will only be possible if there is data and system enablement where the platforms are made to be user-friendly for non-technical employees and business units.

Organisations typically fail when it comes to big data implementation, because they are unable to identify how big data can deliver value to the organisation or because they have not developed the implementation skills. Many organisations are learning how to get insights from big data but converting these insights into organisational performance through good big data implementation projects is one of the major competencies lacking within organisations. Focus on improving implementation is important and the success factors identified under research question two can help organisations to better develop and lead implementations. Other factors that impact the relative success/ failure of big data include: time, complexity and sophistication, legacy systems, and proper big data management.

The research achieved the aim of investigating the success factors of big data in order to understand how and when BDAC yields benefits and improves organisational performance and answered the call for exploration into the success factors of big data, which was part of the purpose.

5.2. Implications for Management

The research findings provide an outline of success factors and challenges for management to utilise when designing and planning around incorporation of big data into their organisation in order to make it a core competency. Management can now better understand how to structure themselves, lead the projects, design processes, capture data and implement with the success factors in mind. This will provide management with a clearer idea of what the roadmap, planning and change management should be in order to mature a a big data organisation. Management can now be more confident in their approaches to big data, knowing what is successful and what challenges they can expect and plan for. The research looked at key management capabilities which will then directly help management to better lead for big data success. BDAC is a growing field of interest and is used constantly within the business environment to attain and use knowledge for organisational performance and competitive advantage. Since organisations are unsure how to capture business value, research into how to generate value from BDAC is highly desirable and the findings can be directly applied by management. The success factors developed in this research were more specific and more practical than most of the current literature and this will help management with implementation.

5.3. Limitations of the Research

Based on the scope of organisations covered during the semi-structured interviews, the results may not be generalisable to other industries or geographies. The scope of constructs defining the semi-structured questionnaire only covered the literature reviewed and this may have missed the effects of other mediators or moderators of the relationship between BDAC and organisational performance. Integrating more variables may be an avenue for future research. Respondent bias could have occurred due to their personal opinions and interpretations, for example, their understanding of what big data actually means as well as their main role in the company not having a holistic view of the company’s BDAC. The research did not actually measure the improved organisational performance and therefore cannot prove any causal relationship between BDAC success factor constructs and improved organisational performance. The environment of data and technology is constantly changing and developing and therefore this research may only be valid for a specific time period after the research was conducted.

5.4. Suggestions for Future Research

Case studies, experiments and longitudinal studies can be conducted to measure the actual performance improvement of the success factors to test whether each success factor has a significant effect on organisational improvement as well as the extent to which each success factor contributes to the improved performance. A quantitative study on a larger number of participants, companies and industries to test if the success factors are impacting organisational performance and whether this list of success factors is more comprehensive than the ones currently found in the literature. This will help drive the academic literature forward in a direction rather than continuing to test newly proposed models and stagnating the literature at this point. The two best companies with data can be further explored through a case study on each organisation to see in more detail what they are doing that has put them so far ahead of other companies in South Africa. This will also bring out more detail around the success factors and challenges and can be the leading South African benchmark for other companies to learn from and adopt success factors that they are not currently doing to enhance their BDAC performance. Each of the constructs discussed under each success factor can be further explored in more detail individually and combined with other fields of study, such as behavioural psychology, leadership, organisational development and transformation, human behaviour etc. This could give further detail as to what companies need to do more of specifically to develop each competency to achieve the success factors and increase their big data maturity and success.

Appendices

Interview Schedule

Culture

1. How does your organisation see and understand big data (BD)? What is BD to your organisation?

2. How well do employees within the organisation understand BD?

3. How does the organisation make business and strategic decisions, and to what extent are decisions based on data?

4. How much feedback on BD and BD practices does the organisation collect?

5. Who consumes BD – how is BD consumed in the organisation - and how do BD projects initiate, kick-off, or commence?

6. To what extent has BD caused organisational redesign in your organisation, or to what extent is it necessary or becoming necessary?

Value

7. How is BD adding value within the organisation now and what about in the future?

8. How and to what extent is BD linked to business problems or business targets in the organisation? Are industry or business experts – people that truly know the business – adequately brought together with BD and BD scientists?

9. How innovative and agile are your organisation and its culture, and how does this affect BD and BD success?

10.To what extent is BD contributing to your organisation innovating – being innovative?

11.How much of BD done by your organisation is exploratory in nature? Is there a bit of freedom to play around?

12.What are the main uses of BD within the organisation?

13.To what extent do you find that conventional data analytics forms an entry point for BD to get a foot in the door and to build momentum?

14.Do you think your organisation needs BD to successfully compete? Is BD a core competency of the organisation, or can BD become a core competency of the organisation?

Core competency

15.What skills do you believe your organisation needs to develop in order to have BD as a competitive advantage and core competency?

16.How strong is your organisation on BD implementation – following through on BD projects?

17.How well are you able to articulate and visualise BD data and output?

18.How extensive is your data gathering? Do you capture unique sources of data?

19.What would you define as advanced BD in the context of your organisation?

Technology

20.How developed and sophisticated is your BD technology?

21.How open and user-friendly is your BD platform, and to what extent is it disseminated through the organisation?

22.Do you foresee new BD tools in the future to improve BD?

Challenges

23.What are the typical factors hindering BD and capturing value through BD?

24.Do you think BD is under-utilised and under-performing, or not? Can BD still do or contribute more?

25.Did BD deliver with the projects it was incorporated in?

26.What do people typically complain about when it comes to BD?

27.Are you able to source the skills and competencies you need to make a success out of BD?

28.Do you find the cost of BD to be prohibitive?

References

1. Agee, J., 2009. Developing qualitative research questions: a reflective process. International Journal of Qualitative Studies in Education, 22(4), pp.431–447.
2. Akter, S., Wamba, S. F., Gunasekaran, A., Dubey, R. and Childe, S. J., 2016. How to improve firm performance using big data analytics capability and business strategy alignment? International Journal of Production Economics, 182, pp.113–131.
3. Bloomberg, L. D. and Volpe, M., 2012. Completing your qualitative dissertation: A road map from beginning to end. London, UK: Sage Publications.
4. Braganza, A., Brooks, L., Nepelski, D., Ali, M. and Moro, R., 2017. Resource management in big data initiatives: Processes and dynamic capabilities. Journal of Business Research, 70, pp.328–337.
5. Braun, V. and Clarke, V., 2006. Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), pp.77–101.
6. Caesarius, L. M. and Hohenthal, J., 2018. Searching for big data: How incumbents explore a possible adoption of big data technologies. Scandinavian Journal of Management, 34(2), pp.129–140.
7. Carson, D., Gilmore, A., Perry, C. and Gronhaug, K., 2001. Qualitative marketing research. London, UK: Sage Publications.
8. Comuzzi, M. and Patel, A., 2016. How organisations leverage big data: A maturity model. Industrial Management and Data Systems, 116(8), pp.1468–1492.
9. Davenport, T. H. and Dyché, J., 2013. Big data in big companies. International Institute for Analytics, 3 [online] Available at: https://docs.media.bitpipe.com/io_10x/io_102267/item_725049/Big-Data-in-Big-Companies.pdf [Accessed on 11 May 2019].
10. Erevelles, S., Fukawa, N. and Swayne, L., 2016. Big Data consumer analytics and the transformation of marketing. Journal of Business Research, 69(2), pp.897–904.
11. EY., 2014. Big Data - Changing the way businesses compete and operate. [online] Available at: http://www.ey.com/Publication/vwLUAssets/EY_-_Big_data:_changing_the_way_businesses_operate/%24FILE/EY-Insights-on-GRC-Big-data.pdf [Accessed on 11 May 2019].
12. Flick, U., 2014. An introduction to qualitative research. London, UK: Sage Publications.
13. George, G., Haas, M. R. and Pentland, A., 2014. Big data and management. Academy of Management Journal, 57(2), pp.321–326.
14. Gil, D., Song, I.-Y., Aldana, J. F. and Trujillo, J., 2017. Big Data. New approaches of modelling and management. London, UK:Elsevier.
15. Guba, E. G. and Lincoln, Y. S., 1994. Competing paradigms in qualitative research. In N. K. Denzin and Y. S. Lincoln (Eds.), Handbook of qualitative research, pp. 105–117. London, UK: Sage Publications.
16. Günther, W. A., Mehrizi, M. H. R., Huysman, M. and Feldberg, F., 2017. Debating big data: A literature review on realizing value from big data. The Journal of Strategic Information Systems, 26, pp.191–209.
17. Gupta, M. and George, J. F., 2016. Toward the development of a big data analytics capability. Information and Management, 53(8), pp.1049–1064.
18. Hashem, I. A. T., Yaqoob, I., Anuar, N. B., Mokhtar, S., Gani, A. and Khan, S. U., 2015. The rise of “big data” on cloud computing: Review and open research issues. Information Systems, 47, pp.98–115.
19. Janssen, M., van der Voort, H. and Wahyudi, A., 2017. Factors influencing big data decision-making quality. Journal of Business Research, 70, pp.338–345.
20. Kabir, N. and Carayannis, E., 2013. Big data, tacit knowledge and organizational competitiveness. Journal of Intelligence Studies in Business, 3, pp.54–62.
21. Kelliher, F., 2011. Interpretivism and the pursuit of research legitimisation: an integrated approach to single case design. Leading Issues in Business Research Methods, 1(2), pp.123–131.
22. Kiron, D., Prentice, P. K. and Ferguson, R. B., 2014. The analytics mandate. MIT Sloan Management Review, 55(4), pp.1-25.
23. LaValle, S., Lesser, E., Shockley, R., Hopkins, M. S. and Kruschwitz, N., 2011. Big data, analytics and the path from insights to value. MIT Sloan Management Review, 52(2), p.21.
24. Lee, I., 2017. Big data: Dimensions, evolution, impacts, and challenges. Business Horizons, 60(3), pp.293–303.
25. Liu, C.-H., Wang, J. S. and Lin, C.-W., 2017. The concepts of big data applied in personal knowledge management. Journal of Knowledge Management, 21(1), pp.213–230.
26. McAfee, A., Brynjolfsson, E., Davenport, T. H., Patil, D. J. and Barton, D., 2012. Big data: the management revolution. Harvard Business Review, 90(10), pp.60–68.
27. McCracken, G., 1988. The long interview. London, UK: Sage.
28. Niland, M. J., 2017. Towards the Influence of the Organisation on Big data Analytics. Gordon Institute of Business Science.
29. Noblet, J.-P., Simon, E. and Parent, R., 2011. Absorptive capacity: a proposed operationalization. Knowledge Management Research and Practice, 9(4), pp.367–377.
30. Patton, M. Q., 2002. Qualitative research and evaluation methods. Thousand Oaks, CA: Sage Publications.
31. Ram, J., Zhang, C. and Koronios, A., 2016. The implications of big data analytics on business intelligence: A qualitative study in China. Procedia Computer Science, 87, pp.221–226.
32. Saggi, M. K. and Jain, S., 2018. A survey towards an integration of big data analytics to big insights for value-creation. Information Processing and Management, 54 (5), pp.758-790.
33. Saunders, M., Lewis, P. and Thornhill, A., 2009. Research methods for business students (5th ed.) London, UK: Pearson Education.
34. Saunders, M. N. K. and Lewis, P., 2012. Doing research in business and management: An essential guide to planning your project. London, UK: Pearson Education.
35. Sheng, J., Amankwah-Amoah, J. and Wang, X., 2017. A multidisciplinary perspective of big data in management research. International Journal of Production Economics, 191, pp.97–112.
36. Vidgen, R., 2014. Creating business value from Big Data and business analytics: organizational, managerial and human resource implications. Citeseer.
37. Vidgen, R., Shaw, S. and Grant, D. B., 2017. Management challenges in creating value from business analytics. European Journal of Operational Research, 261(2), pp.626–639.
38. Wamba, S. F., Akter, S., Edwards, A., Chopin, G. and Gnanzou, D., 2015. How ‘big data’can make big impact: Findings from a systematic review and a longitudinal case study. International Journal of Production Economics, 165, pp.234–246.
39. Wamba, S. F., Gunasekaran, A., Akter, S., Ren, S. J., Dubey, R. and Childe, S. J., 2017. Big data analytics and firm performance: Effects of dynamic capabilities. Journal of Business Research, 70, pp.356–365.
40. Wang, Y. and Byrd, T. A., 2017. Business analytics-enabled decision-making effectiveness through knowledge absorptive capacity in health care. Journal of Knowledge Management, 21(3), pp.517–539.
41. Wang, Y., Kung, L. and Byrd, T. A., 2018. Big data analytics: Understanding its capabilities and potential benefits for healthcare organizations. Technological Forecasting and Social Change, 126, pp.3–13.
42. Wang, Y., Kung, L., Wang, W. Y. C. and Cegielski, C. G., 2018. An integrated big data analytics-enabled transformation model: Application to health care. Information and Management, 55(1), pp.64–79.
43. Zikmund, W. G., 2000. Business research methods (6th ed.) Orlando, USA: The Dryden Press, Harcourt College Publishers.
44. Zikmund, W. G., 2003. Sample designs and sampling procedures. Business Research Methods, 7, pp.368–400.
45. Zikmund, W. G., Babin, B. J., Carr, J. C. and Griffin, M., 2013. Business research methods (7th ed). Mason, OH: Thomson/South-Western.

Article Rights and License

Corresponding Author