19 Data-Driven Agility
The Agile philosophy was originally introduced in 2001 in the software development industry (Beck et al., 2001). Since then, Agile principles, practices, and methods have continued to evolve as business needs for greater agility and responsiveness to changing customer demands have grown exponentially across industries. Firms are poised today to find new ways to develop self-learning capabilities and inform decision-making processes based on more comprehensive and accurate insights from customers. Not surprisingly, the application of Agile principles to the organizational context has developed strongly in recent years, in line with the unstoppable transition towards the smart firm. The combination of data processes with Agile methods is a great opportunity to make value creation in the smart firm more efficient and profitable and have a greater impact on performance.
Big Data is dramatically impacting business firms’ use of information while increasing the challenges facing owners and managers. Recent advances in data science and data analytics techniques are further driving progress towards smarter organizations. As the very essence of the tourism business changes, so do the information and knowledge needs of tourism firms. This, in turn, is pushing changes in the way organizations apply Agile principles and practices (Larson & Chang, 2016). The challenges that smart firms must address are similar in nature to the founding principles of Agile. As such, Agile provides a framework to support smart business transformation by enabling a more dynamic, less formal, and more customer-centric way of working.
The Agile framework is well coordinated and aligned with the various stages of the data life cycle, which means it can be a suitable approach for working with Big Data as well as developing data analytics projects. In those cases, Agile practices provide the opportunity to spend little or no time establishing requirements up front and documenting them in detail before work begins. Instead, they emphasize the need to develop small projects in very short iterations to deliver rapid incremental improvements. This is consistent with the way prescriptive and predictive data analytics projects work, as they seek iterative discovery and continuous validation of behavioral patterns from data (Ambler & Lines, 2016). Noting these similarities between data-centric projects and Agile methods, the smart firm’s use of Agile requires a number of adjustments to highlight the primacy of information and knowledge management over software development.
19.1 Agile in Big Data
The rise of Big Data in tourism and the new challenges it poses for the management of tourism firms explains why many organizations are inclined to implement Agile methods. The large influx of data generated by Big Data and the Internet of Things (IoT) requires new practices that allow the firm to manage data on distributed platforms (e.g., data warehouses, cloud, social media), keep applications up to date, and incorporate continuous testing solutions. This is generally an iterative process that, when in place, may allow the firm to proactively respond to changes in the business and customer environment.
Big Data and agility go hand in hand from an operational and strategic point of view. Big Data facilitates the firm’s transition to agility by providing the insights and resources needed to make fast, timely decisions to deliver better customer value, while agility provides Big Data with the management practices and processes needed to extract value from the data more easily and consistently. In this way, owners and managers can leverage Big Data to identify common customer behavior patterns, personalize their product and service offerings, and set prices based on customer preferences and market trends. Ultimately, together Big Data and agility drive customer-centric management, while optimizing the use of organizational resources (Rialti et al., 2018).
The combination of organizational agility and Big Data further encourages integrated but decentralized decision-making, but only after the organization has become a self-learning organization and continuous interactions have been established between employees, customers, and the firm’s stakeholders. For example, creating cross-functional teams that work in rapid cycles allows the firm to make incremental decisions, which is a big step forward towards organizational agility (Unhelkar, 2017). Self-learning organizations strengthen, update, and optimize Agile values and practices.
Notwithstanding its benefits for improving business performance and creating value, why haven’t firms rushed then to implement Agile frameworks? No doubt business owners and managers have their own reasons. There are several obstacles that hinder Agile development in almost all functional areas of the business, which can be grouped into two categories: 1) lack of leadership; and 2) inadequate organizational culture (Joiner, 2019). Professionals working on the implementation of Agile practices also know very well that the lack of knowledge and experience in relation to Agile principles and methods explains why many organizations do not trust the Agile way of working and do not apply Agile practices to their data and transformation projects.
19.2 Agile Analytics
Big Data analytics initiatives are very different from any software development project on which Agile principles were originally built. For example, Agile project management approaches often fail to take into account many of the challenges of working with large amounts of data from which to extract actionable insights that inform decision-making. As a result, the methodologies used in software features development cannot be applied “as is” in Big Data analytics projects. The other side of the coin is that many teams working on data science and analytics application projects fail right when the projects are deployed to production. Very often, teams don’t work collaboratively on short, continuous development and testing cycles, and when they deliver development products, they must wait to test them and get user feedback, thus making it difficult to manage resources and meet established time targets (Ullah, 2019).
Agile analytics is the application (or adaptation) of Agile practices and methods to data analytics projects with the aim of providing faster time to value, greater flexibility, and responsiveness. In practice, Agile analytics is a set of practices and methods that guide the organization in the construction of data warehouses and data analytics applications that respond to the unique and changing needs of the firm. Moreover, Agile analytics fosters collaborative relationships between data analysts, IT developers, and business users to ensure effective technical delivery by the execution team (Collier, 2012; Nagle et al., 2013).
Agile analytics focuses on developing iterative data discovery cycles that aim to extract value for the business faster by analyzing large amounts of data from multiple perspectives and dimensions, which is a more appropriate approach to the real circumstances in which tourism firms operate. The data to be analyzed can be structured and unstructured, as priority is given to the extraction of value and not to the process itself. Once the results are obtained, Agile analytics focuses on revealing new meanings and interpretations from data and finding new answers by avoiding bias. This implies important changes in the organizational and behavioral approach of the firm, in addition to the development of a creative mentality (Nagle et al., 2013). The Agile smart firm does not work according to a detailed work plan and a strict long-term schedule with detailed documentation, nor does it invest millions in data analytics projects. Instead, the firm establishes use cases that can deliver realistic short-term results for the business and can be specified as requirements for development. The tasks leading to the development of use cases are then assigned to cross-functional teams specialized in data analytics, IT, and other related functions. If some tasks require the intervention of experts from other teams, they can temporarily join the team. In this way, Agile analytics offers the ability to accommodate changes as they occur and rapidly scale projects as new requirements arise or impacts become apparent.
19.3 Agile Practices for Data-Driven Projects
Firms willing to move towards data-driven management need to improve their agility, which means having a good understanding of what an Agile framework is about and choosing which (Agile) practices and methodologies are best suited to implement in their data projects. Indeed, one major stumbling block for the adoption of an Agile framework in data-driven projects is the sometimes ill-informed choice of suitable tools and techniques that are valuable to teams. So, what Agile practices can be applied to data-driven projects to effectively improve the organization’s business performance?
The fact is that this question is not easy to answer. There are many and varied Agile practices that can be applied in a wide variety of situations and contexts by teams seeking to extract value from data. Therefore, before deciding to apply Agile practices, it is always a good idea to first understand how Agile and data projects can fit together (Larson & Chang, 2016; Ullah, 2019). A thorough discussion on the Agile principles that are best suited to data projects and how they differ from traditional methods can be a good starting point for owners and managers. Some of the most important ones are presented below.
19.3.1 Incremental and iterative development
The Agile project development philosophy is embodied in iterative, short, incremental development cycles that are executed in an evolutionary style. Agile development cycles replace the traditional waterfall style by prioritizing the development of those requirements that the user values and continuously adapt development tasks as the customer or user provides feedback to the team. In each iteration, the team produces incremental improvements to the final product that are quickly made available to the customer/user for feedback. These iterations generally last from 2 to 4 weeks, although the final time depends on the nature of the tasks involved. At the end of each iteration an evaluation meeting is held to assess the new functionalities delivered by the team according to the requirements and the deviations from the work agreed upon in the kickoff meeting. Each new feature is only accepted after the customer or user has given their final approval.
19.3.2 Collaboration and self-management
Big Data and analytics projects are usually very complex. They require people with proficiency in multiple areas, from software and IT development to data science, and customer and industry expertise. Without the collaboration of people with different backgrounds it is almost impossible for data projects to be successful. Agile data projects enable mechanisms that promote collaboration among team members, as well as between team members and the customer/end user. Agile teams, unlike traditional project management teams, are independent and self-organizing to manage their tasks and make their own decisions. They decide the sequence in which the development tasks are carried out in each iteration and are responsible for the deliveries to the customer/user according to the committed requirements. Furthermore, Agile team members are discouraged from performing routine or repetitive tasks manually and, if there are any, attempts are made to replace them with automated tasks so as not to waste valuable time that should be spent developing quality features.
19.3.3 Interactions between individuals
Teams that work with formal and highly structured processes but have little experience are not as effective and fast as teams of experienced people who work together but without the burden of such formal processes. In the world of Big Data and analytics, it is essential to work quickly to create value, moving from data to knowledge and from knowledge to the interpretation of results in an agile way. The heaviest analytics processes (e.g., processes with transactional data) that used to take days to complete are now performed in just a few minutes using new Big Data technologies. Additionally, new professional profiles have emerged, such as the Chief Data Officer and data scientists, who are increasingly in charge of data processes. They put Agile principles into practice by applying machine learning algorithms for iterative pattern discovery and seek ongoing interaction with stakeholders for interpretation and guidance.
19.3.4 Early and continuous testing
For Agile teams to continuously deliver incremental data products to customers/users, they need to set up testing mechanisms in advance and test very frequently. Having automated test systems (e.g., analytics-driven testing) makes it easy for teams to test early in the development cycle and ensures that the data products created are reliable and accurate to customer specifications. On other occasions, Agile teams may use test-driven development whereby developers write a test after they have developed enough code to fail. After the first tests are completed, the code is updated, and new tests are rerun as needed until approved. If the code fails, the developers update the working code and test it again. Once the tests pass, the next step is to start a new development cycle to implement new features.
19.3.5 Speed of data analysis
As Big Data and data analytics projects are running at an ever-increasing speed, producing comprehensive documentation in data projects does not seem the best practice. A basic principle of Agile teams is that documentation is not essential, but communication is. As long as Agile teams meet the requirements of product development and these have been properly tested, not much literature is needed to explain things. Moreover, complete documentation not only does not help ensure the success of the project but may even contribute to its failure.
Documentation in Agile data projects is a secondary priority, justified only when the benefits to be gained from it outweigh the costs of preparing and keeping it up to date. Instead, the priority is to streamline data analytics processes as much as possible, ignoring documentation when it does not contribute to faster analysis. When for any reason firms decide to prepare documentation (i.e., when requested by stakeholders, to formalize contracts, to support communication with external stakeholders, for audit purposes), the content is only that which is strictly necessary, not vague or abstract, but concise and not detailed. Very often, Agile teams use the Agile Modeling method to identify what kind of documentation needs to be prepared in data projects, for example, contract templates, executive overviews, system documentation, user documentation, etc.
19.3.6 Response to change
In data project management, the traditional approach generally involves signing a contract between customer and provider with terms that tend to be as complete and detailed as possible. The scope of the contract includes the rights and obligations of the parties, the objective being that it is always strictly fulfilled, deviating as little as possible. Any change in the conditions established in the contract is considered negative and the parties must try to do everything possible to avoid it since it affects the time, resources, and cost of the project.
In Big Data and analytics projects, change is the rule, not the exception, as input data is constantly changing and is not structured or clean enough to be processed. Therefore, being prepared to respond to the changing conditions is a must in data projects. Unlike the traditional approach, the Agile approach attempts to eliminate the bureaucracy and resources required to draft a comprehensive contract between the parties. Instead, the parties must be prepared for changes and respond accordingly.
19.3.7 Collaboration and communication
Big Data and analytics projects consist of iterative discovery processes often carried out by data scientists who process the data and deliver results that must be verified and validated by stakeholders. This generally requires the use of multiple data sources and presentations in different formats using various tools to display graphs and figures. Timely and continuous collaboration between technicians and users/customers is thus essential to achieve the objectives pursued by data analytics projects.
However, while contracts may address expectations of collaboration between stakeholders at the beginning of a data project, it is difficult to gather all the necessary requirements, as they are often unknown, and it is difficult to anticipate all needs in advance. The way to solve this is to increase communication between stakeholders to continually approximate expectations and achieve a satisfactory delivery. The need to communicate effectively becomes a critical factor in developing an Agile data project. Agile communication encompasses all that occurs between team members, with customers and end users, as well as with operations staff and the firm’s management.
Agile team members can choose from several modes of communication when working together, but the most effective communication is person-toperson and face-to-face, especially if shared modeling means are used (e.g., whiteboard, flip chart, paper, cards). It is crucial that communication between members of Agile teams is continuous and kept as simple as possible, and that members trust each other. If there is no mutual trust between team members, the information received and the people who provide it will not be trusted, and therefore the fundamental objective of communication will be lost.
19.3.8 Continuous Delivery
Continuous Delivery is a set of activities and workflow automation process used to develop a new functionality or feature, from ideation to delivery of value to the customer or end user. When firms use a Continuous Delivery approach, development-to-production time can be dramatically reduced, and data Agile teams can focus more on delivering quality data products. The process of Continuous Delivery consists of four stages: Continuous Exploration (CE); Continuous Integration (CI); Continuous Deployment (CD); and Release on Demand (this one intended for commercial software development firms).
CE lays the groundwork for what needs to be developed. Design thinking is generally used by teams to understand a market problem (or a customer need) and find a solution. CE takes the development requirements of the program’s backlog and implements them. After implementing a new functionality or feature, the work is tested from start to finish before is validated in a staging environment. CD moves developments from the staging environment to a production environment, where they are verified and monitored to ensure that they work correctly. After testing in the production environment, the development team determines the right time to release them to customers/users. It must be noted that although these stages are described sequentially, a real-life project is not strictly linear. Rather, it is a learning cycle where team members work on all aspects in parallel, build a solution, test it, and iteratively learn from the feedback received.
19.3.9 Continuous review and improvement
Continuous improvement is a core Agile practice that makes teams more effective when they spend time regularly reviewing what works and what doesn’t in a data project. Once the Agile team receives customer or user feedback, it takes immediate action to implement that feedback. The team uses the feedback received to learn and improve. Improvements can be introduced throughout the development process (i.e., changes in the duration of the iteration cycle), or can be made to the delivered product (i.e., changing data privacy requirements, data sources). Sprint retrospectives are a way to ensure the team takes time to review what they did well and where they need to improve. This encourages incremental improvement over time, thus becoming part of the DNA of the Agile team.
19.4 Agile Data-Driven Framework
As the number and scale of Agile data projects grows, there is an increasing need for owners and managers to understand the framework within which they will develop Agile projects. Agile frameworks seek to go beyond the usual linear approach based on a rigid, pre-established plan and timeline for data product development (e.g., waterfall development). Instead, they apply a less formal, adaptive approach focused on delivering a minimum viable product as quickly as possible that the organization can improve over successive iterative development cycles. This approach based on continuous iterations will keep teams working on small incremental improvements to the data product at regular time intervals, staying focused on responding to customer and user needs that arise throughout the project. Therefore, rather than waiting to have all the requirements developed and implemented before delivering the data product to the customer/user, Agile organizations seek to engage stakeholders early on so they can review the results and modify the requirements to meet their business goals.
Agile data-driven frameworks are customer-centric. Their purpose is to provide business owners and managers with the capabilities to identify common patterns of customer behavior and the insights needed to customize the organization’s product and service offerings and pricing according to customer preferences. Besides a strong customer orientation, Agile frameworks pursue operational optimization, better risk management, and superior workforce utilization through Agile practices that can improve the performance of multidisciplinary teams working with data (Rialti et al., 2018). Figure 19.1 illustrates the workflow of an Agile data-driven framework that tourism firms can work on. This is not a rigid framework that organizations implement blindly, but rather a high-level approach that tourism firms can adapt to their own context and particular needs.
The workflow begins by stating the business objectives that the development of each data product pursues. It involves collecting functional and nonfunctional requirements from users and stakeholders and then translating them into work items that are noted down in the Product Backlog. Note that not all the requirements need to be collected at this stage. The aim of the Product Backlog is to manage all project work and it is filled in by the Product Owner (PO), who is also responsible for discussing each work item with the team to ensure they have the necessary capabilities to take on all required development tasks. Items in the Product Backlog that are selected for a development cycle are moved to the Iteration Backlog. This means that each work item goes through a data product development cycle comprising of acquisition, discovery, prediction, development, and presentation tasks. Once the work item meets all the conditions and acceptance criteria established in the Definition of Done (DoD) and is accepted by the PO, the product is considered ready for delivery to the user. At the end of each iteration, the work items are first moved to a stage environment to be presented to stakeholders and receive their feedback at evaluation meetings. Once the stakeholders are satisfied with the product, the operations team releases the product into the production environment. From this moment on the support team will provide reports on possible improvements and/or errors detected in the product to the development team so that they can make improvements in the next iterations.
Throughout the Agile work process, the status of the different project tasks is visible to everyone who is working on the product so that they are aware of the ins and outs of the tasks performed. As for the product users and stakeholders, they participate during the product development cycle and are aware of the progress of the work, offering feedback at the end of each iteration so that the development team delivers a high-quality and useful product. When the product does not deliver the result expected by the stakeholders, the product and/or the tasks are adapted as soon as possible to minimize deviations. Finally, it should not be forgotten that those firms that are willing to lead Agile datadriven projects must provide training, workshops, and meetings that make their teams improve their expertise in data-driven product development and accelerate the adoption of Agile practices in their daily work.
19.5 Challenges of Data-Driven Agility
When it comes to Agile data-driven project challenges, people skills and abilities rank high at the top of the list. The demand for data-savvy professionals has exploded in recent years in virtually every industry and especially in the data-intensive tourism sector. Today, data scientists are considered one of the most promising careers of the 21st century by the world’s leading staffing companies, and universities are rushing to create training programs to educate qualified data specialists. Investing wisely in skills and technologies is one of the most pressing challenges for all organizations that want to adopt Agile practices successfully in their data projects.
The need for increasingly skilled people in data science and related technologies encompasses not only technical personnel, but also business leaders who need to be more knowledgeable in Big Data and analytics. This challenge is closely related to a major barrier noted by data project practitioners, which is a lack of sufficient commitment from leaders to undertake Agile projects. According to surveys, the lack of corporate sponsorship in the development of Agile analytics projects almost doubles in importance (35%) the construction of business cases (18%). This suggests that while the justification of data projects from a business viewpoint is already a major challenge, the actual implementation of a data project is even more so (Nagle et al., 2013).
There are challenges to data-driven agility at every stage of the data value life cycle. One of the most critical is the confidence that organizations have in the accuracy of their data. There is still a long way to go in this regard, as there are still many organizations that do not have much confidence in the accuracy of their data, nor do they have established processes that ensure compliance with data definitions and data standards between operating systems and technologies. Moreover, very few firms have their data in a single data repository or have implemented a master data management (MDM) system (or “master data warehouse”) to create and maintain accurate and consistent master data sets, which is the way data-centric organizations have a single source of truth to improve business processes. This highlights the need for organizations to integrate the data they have and build a single truth of data across all business functions; even more so given the storm of new data they find daily.
Significant challenges also arise in the firm’s ability to perform data analytics and report the results. Most firms are unable to find patterns and dependencies in multivariate analysis or perform predictive analysis on product or process performance. This amounts to saying that most firms rarely or never perform what can be considered standard data analysis. The same applies to the average time it takes an organization to develop a new or ad-hoc report/ dashboard. Some surveys report that it would take most firms several days and that only a few could do it in less than 1 hour, which reflects the severe restrictions incurred by firms that work with data. Surveys also report that firms have significant difficulties when it comes to integrating new data sources, and that only a few can mine a new data set in less than an hour. All the above highlights the low maturity of data agility in organizations and the size of the challenge ahead until firms can unlock the full potential that data offers.
Data governance is one of the three main differences that distinguish firms that are capable of capturing value from data and those that are not. However, the number of firms that do not have data governance plans in place is still very high. Some surveys report that nearly 30% of firms’ total time is spent on non-value-added tasks due to poor quality and availability (Petzold et al., 2020). These are reasons to believe that most firms do not have a data strategy in place and, therefore, are not treating their data as a corporate asset. This situation is especially alarming considering that data quality is one of the most common reasons for the failure of data initiatives. Organizations do not seem to be investing enough in the capabilities needed to achieve data quality maturity, and their behavior to maintain data quality are somewhat poor. Furthermore, firms should calculate the cost of working with bad or inaccurate data, which they rarely do. Consequently, organizations must rethink the way they manage their data quality policies and understand the implications of data governance policies.
Another key challenge of Agile data-driven projects is technology. Although less relevant than the above challenges, data technologies are generally not seen by owners and managers as a significant limiting factor for Agile data products. Only a relatively small number of firms consider the flexibility and speed of current database software (26% and 10%, respectively), and the rigidity of data warehouses and OLAP data models (12%) as a major barrier (Nagle et al., 2013). Agile data projects are greatly enhanced when the organization already has a pre-existing data warehouse that has been fed with data, otherwise building a data warehouse becomes a real challenge for the average organization. Agile is a value-based approach where customer/user knowledge and intelligence needs drive the development of data warehouse components and not the other way around. This prevents the firm’s data warehouse from being “overbuilt” beyond its intended purposes (Collier, 2012). Owners and managers must note that although the data warehouse can help simplify data product development efforts, its implementation is not a prerequisite to perform data analytics and generate business intelligence for the organization.
Another technology-related challenge is the pervasive reliance organizations continue to have on spreadsheets for data management. The number of firms that still use spreadsheets as the only (or practically the only) tool for information processing and data analysis is still very high, even though spreadsheets are often riddled with errors and there are newer, less errorprone technologies available that are affordable. There really is a significant gap between the claims about big technology trends made by the industry’s leading solution providers and the reality of how comfortable organizations are with them. This situation is further aggravated if we consider tourism SMEs with limited resources and lack of technical skills to tackle technology-based data projects. The responsibility to reverse this situation lies with each firm, but it is also necessary for technology providers to be aware of the challenges faced by tourism SMEs. Perhaps the technology industry should focus more on data tools and applications that are easy to use and for which users do not need highly specialized technical skills and knowledge. This could help firms reduce their dependence on spreadsheets and raise their level of technological maturity to meet the challenges of smartization. Recently, new start-ups (e.g., DataHero, DataCamp) have emerged aware of the opportunity to provide data management knowledge and tools in low-cost, on-demand models. Ultimately, it is important for organizations to implement a data governance program that enables them to meet data standards and maintain data quality, thus reducing reliance on spreadsheets as the primary tool for extracting value from data. Implementing an MDM solution that makes the organization share a common understanding and language of business data objects (e.g., customers, products, services) would be a big step forward in advancing Agile data project management.
19.6 Discussion Questions
Are tourism firms ready to implement an Agile data-driven framework? What factors are for or against?
What kind of Agile practices do you think would be easier (or more difficult) to implement in tourism firms? How could the difficulties be overcome?
Can a tourism firm start implementing an Agile framework without having previous experience working with data? Does a firm need to reach a certain level of maturity in data management before starting to think about Agile practices?
What benefits can a tourism firm expect from implementing an Agile data-driven framework? What threats can be expected?
What Agile practices do you think best respond to the needs created by the smart transformation process of tourism firms?
To what extent can Agile practices improve the work of people in tourism firms?
Do you have any experience working with Agile practices, or know someone who has? Write down those aspects that can help to improve the performance of the firm.