Quobyte, a leader in high-performance storage solutions, has introduced its new distributed File Query Engine. This latest innovation is designed to enhance the management of large-scale storage for enterprises and research environments by offering high-speed querying of file system metadata.
The File Query Engine provides a variety of functionalities, including the capability to query user-defined metadata essential for AI and machine learning (AI/ML) training. This allows users to directly label files with data rather than relying on managing numerous small "metadata files." Administrators can also utilise the engine to efficiently identify and manage space-consuming cold files or locate files belonging to specific users. It replaces the slower traditional method of performing file system tree walks with a more rapid and efficient solution suitable for handling large volumes of data.
A significant feature of the File Query Engine is its integration with Quobyte's distributed and replicated key-value store, which is responsible for storing metadata. Unlike other products, Quobyte's engine bypasses the need for an additional database layer, thereby accelerating query performance and conserving resources. Queries are executed in parallel across all metadata servers, enabling quick scans across an entire cluster or select volumes. The engine operates on up-to-date data to ensure accuracy and efficiency.
"The File Query Engine is a game-changer for our customers," commented Bjorn Kolbeck, CEO of Quobyte. "It streamlines the process of querying file system metadata, offering fast and efficient results even for large datasets and AI and machine-learning workloads."
The File Query Engine is featured in Quobyte's latest release, version 3.22, and is available automatically without requiring any additional configuration. Users can execute file metadata queries using the command-line tool "qmgmt," which supports output in both CSV and JSON formats. Additionally, queries can be initiated through the Quobyte API for added flexibility and ease of use.
