Skip to main content

File Explorer

Overview​

File Explorer is a centralized tool designed to simplify the exploration of multiple storage systems, regardless of their type—whether it's S3, Google Cloud Storage (GCS), Hadoop Distributed File System (HDFS), or Azure Blob Storage (WASBS).

Problem:​

In the process of developing data pipelines, it's often necessary to access storage systems and inspect their contents. For instance, accessing Google Cloud Storage typically requires logging into the web console, managing user permissions, and configuring access. This challenge becomes significantly more complex when dealing with multiple storage types across different clusters.

Solution:​

Ilum File Explorer abstracts these complexities and provides a unified interface to explore storage systems from a single location. By using this tool, data engineers can streamline their workflow and focus on pipeline development, without needing to manage individual access permissions or storage configurations across various platforms.

Key Features:​

  • Unified Access: View the file structure across all supported storage systems and clusters from one central interface.
  • Simplified Management: Eliminate the need for separate access and credentials for each storage system.
  • Focus on Development: Allows data engineers to concentrate on building and maintaining data pipelines rather than managing storage complexities.

How does it work?​

  1. Information about clusters and storage systems is maintained in a central database.
  2. Cloud Storage (S3, GCS, WASBS): File structure data is retrieved using HTTP requests.
  3. HDFS: File structure is accessed via RPC-library .

How does it look?​

Ilum

Ilum