Guide to Big Data

Big data refers to large, complex datasets that can’t be managed with traditional data processing methods. If you consider something that continuously collects every online interaction, including purchases, social media likes, and website visits, you’ll begin to see just how extensive big data is.

It's hard to believe, but up until 2000, only 25% of the world's stored information was digital. In 2026, around 221 zettabytes (that’s 221 trillion gigabytes) of data is expected to be generated, and that number keeps growing every year.

The massive volume of information is where the concept of big data comes into play. Why does it matter? Your own business generates data every day, and so do your customers. Capturing and analysing that information helps you not only improve your business but also develop a better customer experience.

Of course, getting on top of big data is more than a moment’s work, which is why we developed this article. It explores the different types of big data, how it can be used, and some of the challenges it creates. There’s a lot to cover, so let’s dive in.

• Big data defined
• Importance of big data
• Types of big data
• Big data and data strategy
• Applications and use cases of big data
• Big data challenges
• Big data best practices
• Big data strategy and solutions
• Connecting with Salesforce for big data solutions
• Frequently asked questions

At its core, big data is information in such large quantities that traditional tools, like spreadsheets, simply can’t manage it. Rather than thinking of big data as just a large-scale database, it’s better to imagine a system that’s continuously receiving and processing information in many different formats (text, images, video, sensor readings, and more).

The difference between big data vs. small data comes down to scale and manageability. Small data is limited enough in its scope that it can be managed and analysed at a local level, while big data often requires purpose-built platforms, and increasingly the use of AI, to receive and make sense of it all.

That challenge has grown exponentially over the past decade. Data and analytics leaders say their organisations' overall data volumes are increasing by an estimated 30% annually. As data volumes continue to grow, so do the opportunities to use big data to drive even more business value, while also providing the foundation for powerful AI solutions.

Learning the basics of big data is critical for everyone who needs to understand how data and AI can be used to solve business challenges. Whether you’re a data analyst or a business leader, this article will help you get started.

A helpful way to think about big data is to consider its five characteristics: volume, variety, velocity, veracity, and value (known as the “five Vs”).

Volume refers to the sheer amount of data being generated and collected. Online retailers, for example, generate vast amounts of data, ranging from transaction history and website interactions to inventory management and customer service systems.

Velocity refers to the speed at which data is being generated and processed. In the world of finance, stock exchanges process high-velocity data streams to make split-second trading decisions based on real-time market data. Banks and credit card companies analyse transaction data in real-time to identify and prevent fraudulent activities.

Variety refers to the different types and formats of data. For healthcare providers, this can include everything from patient demographics and medical history to lab results, imaging data, and clinical notes.

Veracity refers to the accuracy, trustworthiness, and reliability of your data. Even large volumes of fast-moving, varied data are only useful if you can trust them. Data and analytics leaders estimate that 26% of their organisations’ data is untrustworthy, which is why data veracity is now considered just as critical to a big data strategy as volume, velocity, or variety.

In sectors like healthcare and financial services, where decisions carry real consequences, poor data veracity isn't just inconvenient; it can be costly or even dangerous.

Value refers to your organisation's ability to turn big data into meaningful business outcomes. Collecting and storing data doesn’t do much if you can’t understand and act on it. Seventy-six per cent of business leaders say they feel increasing pressure to drive business value with data, which is why having a big data strategy is a key part of the concept.

According to Salesforce's latest State of Data and Analytics Report, 63% of business leaders now describe their organisation as highly data-driven, up ten points from the previous edition. But to translate data into actionable insights, you need data analytics . Data analytics refers to the tools and processes that allow you to extract valuable insights from all of this data and make better decisions.

For example, businesses can analyse historical and real-time big datasets to identify trends, patterns, and correlations, and make more confident decisions based on the data. A data-driven approach replaces intuition and guesswork, and also supports more accurate and effective business strategies.

Seventy per cent of business leaders believe their most valuable insights are sitting in the 19% of company data that remains siloed or inaccessible. One way to solve such challenges is by implementing platforms that unify the data from your internal systems, as well as external data lakes and warehouses.

By unifying all your customer data, you can develop a 360-degree view of your customer and use AI to find actionable insights that help you make better decisions, and deliver business results.

AI, including generative artificial intelligence (GAI), relies on big data to train and improve its algorithms, and the results are only as good as the data behind them. Salesforce's State Eighty-nine per cent of data leaders believe a strong data foundation is critical to successful AI, which is why data quality and diversity matter just as much as data volume when it comes to building effective AI solutions.

When your data is unified on one platform, you can begin to reap the benefits:

• Analysts can view unified profiles and gain insights about your customers that can be used for personalised marketing, product innovation, better service delivery, and more.
• Developers can create applications, such as health-scoring apps, using unified data. Apps can provide critical information to health-care providers who receive real-time alerts when a patient’s score dips below a certain number.
• Your sales reps need to understand their prospects at a granular level. Are they opening emails? What are they looking at on the website? Based on all prospect data, how likely are they to buy? Unified data can help answer all of these questions.
• E-commerce managers can use data to personalise their website based on a customer’s browsing behaviour and current product stock levels, ensuring a relevant and targeted customer experience.

Once you’re able to see the big picture, it’s easier to identify gaps in the market, discover new opportunities, and develop products and services that customers want and need.

The “mosaic” of big data consists of many approaches. Knowing the types of big data listed below is the first step to understanding how to effectively manage, analyse, and extract valuable insights.

This type of big data is formatted in fields that can be easily sorted and analysed. Think spreadsheets and lists arranged in columns and rows. Examples include:

• customer information
• point-of-sale data
• inventory records
• financial data, such as transaction records.

Unstructured data includes categories of information that can’t be organised into rows and columns. Think social media posts, emails, customer reviews, and multimedia content like photos and videos. Unstructured data is nearly 80% of all data, but it hasn't always been easy to analyse. Thanks to AI and machine learning, it’s possible to gain valuable insights from this vast category of data.

This type of data has some organisation but doesn't fit neatly into traditional databases. Semi-structured data often contains tags and labels (metadata) that is used to group data and describe how it is stored. Examples include XML files, JSON data, and log files.

Metadata is information that describes other data. This “data about data” provides details about the characteristics, attributes, and properties of a dataset. The metadata of a video file, for example, may include information about its duration, resolution, and encoding format.

Time-series data is used to analyse trends, patterns, and changes over a specific period. Examples include stock market prices, weather data, sensor readings from IoT devices, and website traffic data.

Geospatial data is related to specific locations. It includes coordinates, addresses, maps, and satellite imagery. Geospatial data is used in applications like GPS navigation, urban planning, and environmental monitoring.

Streaming data comes from sources like social media feeds, sensor networks, and online transactions. Analysis of streaming data provides organisations with information needed to make immediate decisions and respond to events as they happen.

In AI-driven projects, all of these data types can be used together to help your organisation gain a well-rounded understanding of your customer. Unstructured data, in particular, holds enormous potential in the age of AI.

By applying advanced analytics and machine learning, organisations are able to extract insights from mounds of messy information and deliver targeted customer experiences, product innovations, and more.

Now that you understand the many different types of data, you can begin to see that insights from all of these data types are necessary for gaining deep understanding of your customer, driving innovation, and making better decisions.

Your data is only valuable when it’s properly managed, protected, and available for analysis. Let’s look at a few key concepts about the data life cycle and learn how it all comes together.

How do you get from raw data to actionable insights? Your approach is guided by your data strategy. This strategy is an essential framework that guides your organisation's approach to collecting, managing, and using its data.

Your strategy should include defining goals, identifying data sources, and establishing processes for data acquisition, storage, and analysis. A solid data strategy aligns data initiatives with business objectives, so the entire organisation is on the same page about how data should be used to support the business.

Data governance is an important part of your data strategy. Your data governance framework includes standards for data quality, integrity, and compliance for your business. A good governance framework specifies the policies, procedures, and controls for data management, access, and usage that apply to each team in the organisation.

Effective data governance helps ensure that data is accurate, consistent, and trustworthy, allowing organisations to make informed decisions and comply with regulatory requirements. Yet, fewer than half of technical leaders (43%) currently have a formal data governance framework in place, a significant gap given how central governance has become to AI readiness.

Security and big data are two sides of the same coin, and the stakes are rising as AI adoption accelerates. In response to our State of Data and Analytics survey, 88% of data and analytics leaders agreed that AI demands entirely new governance and security approaches.

To address this, organisations need comprehensive data security measures to protect data from unauthorised access, breaches, and cyberthreats. This includes using a wide range of security solutions, from encryption and event monitoring to multifactor authentication and employee security training, to ensure the highest standards of confidentiality, integrity, and availability.

Strong data security measures build trust with both customers and stakeholders, and help safeguard the organisation's reputation.

Data privacy is critical in the era of big data. Safeguarding your customers' privacy is directly tied to your business success. Organisations should ensure that they always maintain compliance with regulations regarding customer data handling. This includes obtaining consent for data collection, establishing privacy controls, protecting sensitive information, and adopting security policies to safeguard sensitive data.

What can you do with big data and big data analytics? Once an organisation has analysed the information, the opportunities are as endless as the data.

Big data is informing the business strategies of some of the world’s largest companies — and delivering substantial value to customers and the bottom line. Here are just a few examples of how data, AI, and customer relationship management (CRM) are changing the game for these big brands:

Big data is powering AI algorithms, and data is required to ensure that AI-powered solutions are trained with the right sources to learn and adapt. Fashion house Gucci put AI at the centre of Gucci 9, its global client service network, with Salesforce’s Salesforce Platform. Now, AI generates bite-size conversation replies in a "Guccified" brand voice to help advisors assist customers in every interaction.

Multinational food company General Mills wanted a better way to use its data to reach consumers on a more personal level while maintaining privacy. With its wealth of purchase and behavioural data, the company zeros in on data, such as recipe views and receipt scans, to recommend personalised content to consumers, based on their diet or geography.

This helped them triple consumer engagement, increase known site users 170% year over year, and save millions of dollars in paid media.

Specialty retailer Williams-Sonoma Inc. uses data from past purchases to customise the emails and ads it sends to customers, which has tripled engagement. Using Data 360 an advanced data platform, the company creates a 360-degree view of each customer and segments the data into niche audiences. “Personalisation is table stakes,” said Abby Teisch, SVP of Marketing. “We not only want to show you the product, we want to tell you how to use it, how to cook with it, and what it’ll look like in your home.”

“Personalisation is table stakes. We not only want to show you the product, we want to tell you how to use it, how to cook with it, and what it’ll look like in your home.” — Abby Teisch, SVP of Marketing, Williams-Sonoma Inc.

By connecting all its guest data, including booking history and resort interactions, Turtle Bay Resort in Hawaii is using big data and Data 360 an advanced data platform to deliver a customised experience to its guests. 

Guest data is used to create highly targeted guest segments — and the resulting email campaigns have led to a 20% increase in booking conversions and a 15% rise in repeat bookings for adventure experiences.

Humana makes the path to wellness easier with tailored messaging, service, and care coordination with a 360-degree view of their customers. This has resulted in faster care delivery, significant cost savings, and ultimately improving patient well-being. In fact, the company now delivers care three to four times faster and saves $6 million in information security costs in the process.

Considering big data’s depth and breadth and rapid expansion, it doesn’t come without challenges. Let’s look at three of the most common issues in more detail.

A big data challenge facing organisations is data literacy, ensuring their employees have the right skills and training to effectively understand and use the data. In fact, improving data literacy is a top priority, according to a recent survey of analytics and IT leaders. Investing in data literacy programs and training is a good first step.

Big data is critical to improving customer experience, but it’s also a big challenge. Getting to valuable, actionable insights requires advanced analytics tactics like machine learning and natural language processing. To succeed, businesses need to invest in the right tools and expertise to personalise experiences and drive customer satisfaction.

Considering the complexities of handling big data, your organisation needs to establish best practices. Data best practices are important to ensure data accuracy, integrity, security, and compliance. With a common understanding of what’s expected, your teams can be confident that they are using trusted data, and that their data-based decisions are sound.

According to our latest State of Data and Analytics report and the Data Strategy Guide, some best practices for big data include:

1. Clear objectives: Establish clear business objectives and align them with data initiatives to ensure that data efforts are focused and aligned with organisational goals.
2. Data quality and governance: Establish robust data quality processes and governance frameworks to ensure data accuracy, consistency, and compliance with regulations.
3. Data security and privacy: Implement strong security measures to protect data from breaches and ensure compliance with privacy regulations, building trust with customers and stakeholders.
4. Data unification: Data unification is the process of bringing together all of your data into a unified view. Using a data unification platform like Data 360, you can store, manage, and analyse data from every data source and gain deep insight into every aspect of your business.
5. Advanced analytics: Leverage advanced analytics techniques like machine learning and predictive modeling to extract valuable insights from big data and drive informed decision-making.
6. Scalable infrastructure: Invest in scalable and flexible infrastructure that can handle the volume, velocity, and variety of your data — ensuring that your systems can store, process, and analyse all your data with efficiency.
7. Data storage: Data lakes, data warehouses and data lakehouses play a central role in achieving scalable infrastructure, and organisations that unify these with a platform like Data 360 are far better placed to realise the full value of their data.
8. Data-driven culture: Promote a culture that prioritises data-driven decision-making, so every employee understands the importance of data and is empowered to use it effectively.
9. Collaboration and cross-functional teams: Encourage collaboration across departments and teams to share insights, expertise, and data, so everyone can derive value from your data assets.

Simply stated, big data strategy and solutions are the tools and platforms that help organisations like yours obtain insights from large volumes of data.

Data unification and activation solutions bring together your customer data from multiple sources and provide a unified view of your customer. Data 360/an advanced data platform is designed to connect and unify data from any source — and activate it within the CRM and workflows you use every day to power AI solutions and drive effective decision making and customer success.

Data security platforms are a basic requirement for protecting sensitive data from unauthorised access, breaches, and cyber-threats. These platforms protect against unauthorised access, attacks, and theft to ensure the integrity and compliance of your customer and business data.

Business intelligence (BI) platforms allow organisations to transform raw data into actionable insights. These platforms offer an array of advanced analytics capabilities, interactive dashboards, and reporting tools that are foundational for data-driven decision-making.

Data visualisation platforms bring analytics to life by presenting data in a visually appealing and easily understandable format. These platforms use charts, graphs, and interactive visualisations to communicate complex data insights effectively.

CRM (customer relationship management) platforms that integrate data and AI, such as Salesforce with Agentforce, combine customer data with AI capabilities to take customer relationship management to the next level. With AI-enabled CRM, it's easy to analyse customer behaviour, predict trends, provide personalised recommendations, and deliver exceptional customer experiences.

All of these solutions help organisations to manage and use big data, which in turn drives informed decision-making, personalised experiences, and business success.