megfile

name: megfile description: Use megfile’s APIs, SmartPath, configuration, and CLI to perform filesystem tasks easy across local FS, S3/OSS-style object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.

Overview

Guide to using megfile’s unified APIs, SmartPath, configuration, and CLI for file operations across various backends, like local FS, S3/OSS-compatible object storage, SFTP, WebDAV, HTTP, HDFS, and stdio.

Quick Start

Install base: pip install megfile; add extras per backend (megfile[cli], megfile[hdfs], megfile[webdav]).
Configure credentials/endpoints (env vars > config files). S3 examples: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_ENDPOINT_URL / OSS_ENDPOINT / AWS_ENDPOINT_URL_S3, AWS_S3_ADDRESSING_STYLE.
Path format: protocol://bucket/key or alias (e.g., tos://); bare POSIX paths are treated as file://.
Import functional APIs (from megfile import smart_open, smart_sync, ...) or SmartPath (from megfile.smart_path import SmartPath).

Supported Protocols & Extras

Local FS (file:// or bare paths) — base install.
S3/OSS-compatible (s3://, plus aliases) — base install.
SFTP (sftp://) — install megfile[cli] or megfile with SFTP deps.
HTTP/HTTPS (http://, https://) — base install.
Stdio (stdio://) — base install.
HDFS (hdfs://) — install megfile[hdfs].
WebDAV (webdav://) — install megfile[webdav].
Full Protocol path format reference: references/path_format.md.

Core Tasks

File IO

smart_open(path, mode='r', encoding=None, **options): open binary/text handles.
Convenience loaders/savers: smart_load_content / smart_save_content (bytes), smart_load_text / smart_save_text (str), smart_save_as(file_obj, path), smart_load_from(path) returns BinaryIO.
smart_combine_open(glob, mode='rb', open_func=smart_open): sequentially reads multiple files as one stream.

Existence & Metadata

smart_exists, smart_isfile, smart_isdir, smart_islink, smart_isabs.
smart_access(path, mode=Access.READ/WRITE).
smart_stat / smart_lstat (via SmartPath), smart_getsize, smart_getmtime, smart_getmd5(recalculate=False, followlinks=False).

Listing & Globbing

Directory traversal: smart_listdir, smart_scandir, smart_walk, smart_scan, smart_scan_stat (returns FileEntry/StatResult).
Pattern matching: smart_glob, smart_iglob, smart_glob_stat for stat-rich globbing.

Data Transfer & Lifecycle

Copy/sync: smart_copy, smart_sync, smart_sync_with_progress (progress-friendly wrapper), smart_concat to merge multiple sources.
Moves/deletes: smart_move, smart_rename, smart_remove, smart_unlink, smart_touch, smart_makedirs.
Links: smart_symlink, smart_readlink.

Path Utilities

smart_path_join, smart_abspath, smart_realpath, smart_relpath, smart_isabs.
SmartPath mirrors pathlib semantics but routes to the right backend: path = SmartPath("s3://bucket/key"); path.exists(); path.open(mode="rb").

Caching

smart_cache(path, cacher=SmartCacher, **options): cache remote resources locally for tools that only support local files.

Configuration

Use CLI helpers to persist credentials/endpoints; environment variables take precedence.
Profiles enable multiple endpoints (e.g., s3+prod://...). See references/configuration/ for protocol-specific flags and env vars.
Full config reference: references/configuration/.

CLI Essentials

Install CLI extras: pip install 'megfile[cli]'.
Common commands (ls/cp/sync/stat/md5sum/mkdir/rm/touch) mirror POSIX semantics across backends.
Completion scripts: megfile completion zsh.
Full command list and flags: references/cli.md.

Usage Notes

Prefer smart_* for protocol-agnostic code paths; avoid branching per backend.
Ensure required extras are installed for target protocols before invoking APIs.
For high-volume sync/copy, supply map_func (e.g., ThreadPoolExecutor.map) and callback to report progress.
Use aliases via megfile config alias <alias> <protocol> to shorten paths (e.g., tos://).

References

API surface: references/megfile.smart.md and references/megfile.smart_path.md.
Configuration flags, env vars, and profiles: references/configuration/
CLI commands and flags: references/cli.md
Full Protocol path format reference: references/path_format.md.
Glob patterns reference: references/advanced/glob.md.