O que é um sistema de arquivos distribuídos?

In general, distributed files systems are IT solutions that allow multiple users to access and share data in what appears to be a single seamless storage pool. The back-end enabling systems can follow one of a few architectural patterns, in popular order they are client-server, which tends to be the most common, cluster-based architectures are most useful in large data centers, and decentralized files systems also exist.

These architectures comprise multiple back-end systems connected together via a network, with middleware orchestrating file storage, and employing many techniques to ensure the “distributed” system’s performance meets the needs of users. In this way, the distributed system has a capacity of service, and the load on that service is the total demand by all the active users. When load approaches or exceeds that capacity, system performance will degrade, and will show signs of lag, or service outages.

The chief benefit rests on the fact that sharing data is fundamental to distributed systems, and therefore forms the basis for many distributed applications. Specifically, distributed files systems are proven ways to securely and reliably accommodate the data sharing between multiple processes over long periods. This makes them ideal as a foundational layer for distributed systems and applications.

Distributed systems form the modern concept of “the cloud” and support the idea that the cloud is essentially limitless in storage capacity. These systems can expand behind the scenes and match any growth in demand. They can manage massive volumes of information, safeguard its integrity, and ensure its availability to users 99.9995% of the time. And in that small sliver of downtime, there are contingencies upon contingencies in place. For cloud data centers, this is their business, and so are able to benefit from economies of scale more readily than enterprises or smaller businesses that deploy their own distributed systems.

Enterprises and small businesses may deploy their own distributed file systems to facilitate business operations, regionally, even globally. For instance, distributed systems may support private clouds, parallel computing, even real-time control systems. Municipalities deploy real-time traffic control and monitoring systems to better manage commuter times, all made possible by DFS supported applications. Sophisticated parallel computing models are deployed across many participating computing systems in collaborations that help compute large data sets, as in astronomical calculations where one computer simply won’t do the work.

Technically, distributed file systems need to achieve several goals to produce the effect of active file sharing for multiple users. The basic distributed model connects multiple local file systems together, by mounting them, and then abstracts a layer of storage management from the user interface layer, hiding how those files are optimally stored on underlying infrastructure, so that engaging clients view the multiple system as one. From the user’s perspective, they will often expect simultaneous user data access, while accessing the freshest version. These demands raise many considerations for system designers.

To answer many of these considerations, system designers use different DFS mechanisms, including:

• Mounting — Connecting the file namespaces of multiple systems is a key feature of distributed file systems. Mounting mechanism allows different file namespaces to be connected into a single hierarchical namespace.
• Client Caching — Because files are stored across a network, file access time inherently includes network transmission times, so network performance directly impacts the whole system performance. Poor network performance reduces client experience. To mitigate poor access time due to lag, caching features are employed to store copy files or parts of files locally at the client, for example, accessing a folder may cache the metadata of all files inside, once a file is requested, then the content data may be transferred.
• Bulk Data Transfers — Closely linked to system performance is the transfer of data in bulk. Because network performance is key to system performance, techniques like burst transfers are utilized to optimize network transfers.
• Encryption — Distributed systems are network supported, so encryption of data in transit is used to enforce security.
• Stateful and Stateless File Servers — Sometimes users will interact with a file, and expect to edit it, while other users are doing the same thing. This complicates distributed file system management. This can be done using stateful servers, which maintains session information. However, in general, stateless servers are used, and stateful ones avoided, because stateless are more fault tolerant—client crashes do not impact the stateless server like it would a stateful server actively managing sessions.

The main benefit of distributed file systems is to connect together multiple systems through a network and expand storage capabilities while maximizing user access. Subsequent benefits include:

• Unified view and control over file storage assets
• File access availability to users anytime, anywhere
• File access controls prioritizing access to data
• Storage system interoperability
• Storage Scaling capabilities
• Load-balancing management
• Data access transparency and location transparency

Three distributed file system architectures are in use today. Client/server file systems are most common, and readily available. Decentralized file systems are often found in peer-to-peer community-based networks. And cluster-based file systems are useful in large datacenters.

• Client/Server Architecture — The most common DFS today is the client/server, where files are managed by a file server, and files are stored on one of several servers within the distributed system. This model relies on remote access, where a client sends requests to work on a file that is physically located on a server, in contrast to downloading a file, and uploading an updated version if there are any edits.
• Decentralized Architecture — In a decentralized architecture, files are dispersed throughout the participating network, and transferred peer-to-peer (P2P). This style is popular with torrent networks.
• Cluster-based Architecture — Typical solutions for client/server architectures become burdensome when scaling to very large applications. Cluster-based architecture is deployed in large data centers, and uses more enhanced methods of data management, like file-striping, chunking, and parallelism, to make data access, transfer, and processing more efficient. The leading solutions in this space are Google and Hadoop