Updated on 9 November 2026 by Nicolas Finet, CEO of Sortlist. This guide has been refreshed for 2026: it keeps the classic big data framework (volume, variety, velocity) and the main management and analytics challenges, and adds the cost of poor data quality, UK GDPR and ICO expectations, and how AI raises the stakes.
Big data analytics means extracting insight from datasets too large, varied or fast-moving for traditional tools. Used well, it improves decisions, customer experience and efficiency. Used badly, it becomes expensive storage with unreliable conclusions and legal risk. This guide covers the main challenges and practical solutions.
Key takeaways
- Gartner research from 2020 estimated that poor data quality costs organisations at least $12.9 million a year on average, and more recent commentary suggests this is likely an underestimate as AI spending grows.
- The UK GDPR's data minimisation principle (Article 5(1)(c)) requires you to process the minimum personal data you need for your purpose, which limits "collect everything" approaches.
- Start with a business question, then build the data, tools and governance needed to answer it.
The three Vs of big data
| V | Meaning | Example | Challenge |
|---|---|---|---|
| Volume | The amount of data | Years of transactions, clickstreams, sensor readings | Storage and processing cost |
| Variety | The different forms of data | Tables, text, images, video, logs, social posts | Integrating different formats |
| Velocity | The speed data arrives and must be handled | Real-time payments, streaming events, IoT devices | Real-time processing and decisions |
Many practitioners add Veracity (how trustworthy the data is) and Value (whether it helps decisions). Veracity is often the biggest practical problem.
Challenges in data management
Data storage and infrastructure
- Challenge: growing data volumes make on-premises storage costly and inflexible.
- Solutions: cloud data warehouses and lakes, tiered storage (hot, warm, cold), compression and retention policies that delete what you no longer need. Cost control matters as much as capacity.
Data integration and quality
- Challenge: data sits in silos (CRM, website, finance, operations), with inconsistent definitions, duplicates and gaps.
- Solutions: a shared data dictionary, master data management, automated validation, data lineage and clear ownership.
“Poor data quality costs organizations at least $12.9 million a year on average.”
Gartner, Data quality (research from 2020)
This widely cited figure is a decade-long industry estimate, not a precise measure for any business. What matters locally is the time analysts spend cleaning data and the decisions made on faulty numbers.
Scalability and performance
- Challenge: queries and pipelines that worked on small data slow down or fail at scale.
- Solutions: distributed processing, partitioning, caching, columnar storage, incremental loads and monitoring of pipeline costs and performance.
Analytics challenges
Processing and analysing large datasets
Choose tools to match the job: SQL in a cloud warehouse for most business analysis, distributed frameworks for very large or complex processing, and streaming platforms for real-time. Do not use a heavy architecture for a problem that fits in a spreadsheet or a database.
Extracting meaningful insights from noisy data
- Start with a clear question and hypothesis.
- Check for bias, sampling problems and spurious correlations.
- Use statistical methods and test results, including on holdout data.
- Combine quantitative findings with qualitative context.
- Visualise clearly. See data storytelling.
Predictive models and AI
Machine learning turns data into predictions: demand, churn, fraud, recommendations. But models inherit the flaws in their data. Monitor for drift, bias and accuracy, and keep humans accountable for decisions. See predictive analytics in marketing.
Skills and culture
Many organisations lack data engineers, analysts and people who can interpret results. Build skills, hire carefully, and make sure that business teams know how to ask good questions and question the answers.
Data security and privacy in the UK
- Lawful basis and transparency: you need a lawful basis to process personal data, and must tell people how you use it.
- Data minimisation: the UK GDPR requires that you identify the minimum amount of personal data you need for your purpose and process only that. This limits indiscriminate collection "in case it is useful".
- Purpose limitation and retention: use data only for the stated purpose and delete it when no longer needed.
- Security: encryption, access controls, logging, backups and supplier checks.
- AI and automated decisions: the ICO publishes guidance on AI and data protection, including fairness and Article 22 rules on solely automated decisions with significant effects.
- International transfers: check safeguards when data leaves the UK.
- Anonymisation: properly anonymised data is outside the UK GDPR, but true anonymisation is hard, so test the risk of re-identification.
Consult your data protection officer or a specialist adviser for complex projects. This is general information, not legal advice. For tracking and consent in marketing, see software development in digital marketing and web analytics tools.
A practical approach
- Define the business question and the decision it supports.
- Audit your data: what exists, where, quality and legal basis.
- Fix the foundations: definitions, integration and quality checks.
- Choose proportionate tools and watch costs.
- Build, test and validate models and dashboards with users.
- Govern: owners, access rules, documentation and review.
- Measure value: time saved, revenue gained, risk reduced.
Common mistakes
- Collecting data without a purpose.
- Ignoring data quality until models fail.
- Choosing complex technology before defining the question.
- Treating privacy as an afterthought.
- Building dashboards that nobody uses.
- Overtrusting AI outputs.
Conclusion
Big data analytics is only as good as the questions, the data quality and the governance behind it. Start with a decision that matters, build proportionate foundations and respect privacy rules. The organisations that benefit are those that treat data as a managed asset, not a by-product.
Frequently asked questions about big data analytics
What are the three Vs of big data?
Volume, variety and velocity: how much data, how many types and how fast it arrives.
What are the main challenges of big data?
Data quality and integration, storage cost, scalability, skills, extracting reliable insights and meeting security and privacy requirements.
How much does poor data quality cost?
Gartner research from 2020 estimated at least $12.9 million a year on average per organisation, and later commentary suggests the true figure may be higher.
How does UK GDPR affect big data projects?
It requires a lawful basis, transparency, data minimisation, purpose limitation and security, which limits collecting everything.
Do I need big data tools?
Not always. Many questions can be answered with a standard database or warehouse. Use more complex tools only when volume or speed requires them.
How can AI help?
It can automate analysis and prediction, but it depends on good data and needs monitoring for bias and errors.
Need analytics support? Post your project on Sortlist and compare UK data and analytics agencies for free.
Sources: Gartner, data quality, IBM, cost of poor data quality, ICO, guidance on AI and data protection, ICO, data minimisation in AI. This is general information, not legal advice.
