The layers of a sound data architecture have not changed. What sits inside them, and who uses the output, has. Here is what modern enterprise data infrastructure looks like as organisations plan for 2027.

Ask a room of data leaders to sketch their current architecture and you will rarely see the same picture twice. Most enterprise data estates were never designed in one sitting. They grew: a warehouse for finance reporting, a data lake for the data science team, a streaming platform for a fraud use case, and a governance layer that was always due to arrive next quarter. Few would describe the result as a modern data architecture.

For years that was tolerable. AI has made it expensive. Gartner expects the share of AI spending that goes to AI data readiness to increase seven times between 2025 and 2029. Organisations are finding that the model is rarely the hardest part of an AI initiative. The data underneath it is.

As 2027 budgets are set, it is a good moment to look at what a modern data architecture needs to include, and what can stay exactly as it is.

If you are new to the topic, start with our knowledge base article What is Big Data Architecture? It describes five interconnected layers that most enterprise data architectures share: ingestion, storage, processing, serving, and visualisation and consumption. Long before that, the NIST Big Data Reference Architecture gave the industry a vendor-neutral model of the same flow, from data provider to data consumer, with management, security and privacy wrapped around every component.

Neither model is out of date. Data still has to be collected, stored, processed and served before anyone can use it, and governance still has to reach every step. What has changed is the technology filling each layer, the kind of data moving through it and, most significantly, who or what sits at the end of the pipeline.

One principle from our knowledge base is worth repeating before going further: no single architectural pattern is universally correct. The right design depends on data volumes, latency needs, team capability and regulation, not on industry trends.

Layers as described in the DASCIN knowledge base, with data flowing from ingestion through to consumption.

The common thread is that architecture now has to serve machines as well as people. A dashboard user who sees a strange number can ask a colleague about it. An automated process usually cannot.

Much of the architecture debate is framed as a contest between approaches. DASCIN’s Enterprise Big Data Engineer curriculum takes a different view. It covers six architectures and treats them as approaches that organisations frequently combine, not as rivals: the relational data warehouse, the data lake, the modern data warehouse, the data lakehouse, data fabric and data mesh. Each one answers a different question.

The lakehouse answers a storage question. Lakes and warehouses were never really rivals. As our article on data lakes vs data warehousing explains, most organisations run both, with data moving from the lake into the warehouse once it has been cleansed. The lakehouse brings the two together. Open table formats add transactional integrity, schema evolution and the ability to query earlier versions of a table, which is why the lakehouse has become a common starting point for organisations modernising legacy warehouses.

Data fabric answers a connection question. It is best understood as a conceptual model rather than a single tool: a metadata-driven layer that links existing warehouses, lakes and applications so data can be discovered and governed where it already sits. For a closer look at how its layers fit together, from data sources through metadata and catalogue to consumers, see What is Data Fabric?

Data mesh answers an ownership question. Domain teams own their data as products, supported by self-serve infrastructure and governed through federated standards.

An organisation can run a lakehouse, connect it to older systems using fabric principles and assign ownership along mesh lines, all at the same time. The outcome to avoid is a data lake with no governance. That is how a lake becomes a data swamp.

Both the NIST model and our own description of the consumption layer allow for consumers that are systems rather than people. A few years ago that mostly meant another application. Going into 2027, it increasingly means AI agents that query data, combine sources and act on what they find. Gartner predicts that by 2027, half of business decisions will be augmented or automated by AI agents for decision intelligence.

That raises the bar for everything upstream. Our own knowledge base already counts AI systems among the consumers a data fabric must serve, and those consumers need data that is described, not just stored: consistent business definitions, documented lineage, visible quality signals and access rules a machine can read.

This is where semantics earns its place. According to Gartner, organisations that prioritise semantics in AI-ready data by 2027 will increase generative AI model accuracy by up to 80% and reduce costs by up to 60%. Semantics is not a new discipline, either. The taxonomies and classification structures that information architects have refined for years, covered in our white paper An Introduction to Information Architecture for Data Professionals, are what give a machine a consistent understanding of what the business means by “customer”, “order” or “revenue”.

Unstructured data is the second shift. Contracts, emails, images and call transcripts were once stored and largely left alone by analytics teams. Gartner expects spending on multistructured data management to account for 40% of all data management spending by 2027. It also offers a useful caution: through 2028, organisations that build their own unstructured metadata solutions are expected to spend more than 300% more than those that use existing document and records management solutions, skills and practices. Not every problem needs a new platform.

The business objective still comes first. Our Enterprise Big Data Analyst curriculum starts every analysis with a defined business objective, before any data is collected. Architecture follows the same logic: design for the decisions the organisation needs to make, not for the tools on the market.

Pipelines still need discipline. The EBDFA Data Pipeline Architecture sets out four steps:

The tools behind each step have changed considerably. The sequence has not. Our guide to data pipeline orchestration, management and monitoring covers the fourth step in more detail.

Governance belongs in the design. NIST placed security and privacy around every component of its reference architecture, and What is Big Data Architecture? makes the same case: governance should be built in from the start rather than bolted on afterwards. With regulation of personal data and AI tightening across many jurisdictions, the cost of retrofitting it keeps rising.

Modern data architecture is less about a single platform decision than about keeping a clear structure while its contents change. The organisations best placed to benefit from AI in 2027 will be those whose data is described, governed and reachable, whatever technology it happens to run on.

Building that structure depends on people who understand how the layers fit together. DASCIN’s Enterprise Big Data Engineer certification covers data pipelines, the six architectures discussed here, and security and privacy in data engineering. The Enterprise Big Data Analyst certification focuses on turning well-architected data into decisions. If either matches where you or your team are heading, the credential pages set out the details.