Pincer Engine Documentation

Welcome to the Pincer Engine docs. Start from the top and work your way down — everything is numbered in reading order.

Built for Grabbit

I originally developed the Pincer engine to power my native macOS Download Manager, Grabbit. If you are looking for a beautiful, polished GUI that leverages the full power of this engine, check it out at grabbit.fyi!

Getting Started

1. How It Works — A jargon-free look at what Pincer does under the hood.
2. User Guide — Start downloading files in minutes.
3. CLI Reference — Every flag and option, with examples.

JSON-RPC API

How to control Pincer programmatically over WebSocket.

1. Connecting — WebSocket URL, port, and token authentication.
2. Managing Downloads — Adding, pausing, removing, and queuing tasks.
3. Checking Status — Querying progress, file lists, and global stats.
4. Configuration — Engine defaults, task options, and system keys.
5. Advanced Features — URL resolution, format conversion, Metalink, FTP/SFTP, and more.
6. Events — Real-time WebSocket notifications.
7. System Controls — Version info, shutdown, session saves, and batch calls.

Guides

Deep-dive walkthroughs for specific use cases.

1. Session & Resume — How auto-save and download resumption work.
2. Swift Integration — Embedding Pincer in a macOS Swift/SwiftUI app.
3. API Comparison — Pincer’s methods mapped against standard download engines.

Contributing

Everything you need to contribute code to Pincer.

1. Getting Started — Setup, code style, and PR workflow.
2. Architecture — System design, modules, and concurrency model.
3. Testing — Test suite architecture and how to run tests.
4. Build & Release — Compiling from source and cutting releases.
5. Roadmap — Dependency migration strategy and optimization paths.
6. Feature Status — What’s done, what’s next.

License

Pincer Engine is free and open-source software licensed under the GNU General Public License v3.0 (GPLv3).

Downloads & Releases

Get the latest pre-compiled binaries for Pincer Engine, inspect versions, or report bugs.

Latest Release

Quick Installation (macOS)

1. Install via Homebrew

You can install Pincer using our custom Homebrew tap:

brew trust im-adnan/pincer/pincer
brew tap im-adnan/pincer
brew install pincer

2. Verify Installation

Check that Pincer is installed and view its version:

pincer --version

Start the daemon or interactive CLI:

# Run in daemon mode with RPC on port 6800
pincer --daemon --port 6800

# Or download a file directly
pincer "https://example.com/file.zip"

All Releases & Versions

To view past release notes, changelogs, and historical binaries:

• Browse all releases: GitHub Releases Archive
• Review changes in source: CHANGELOG.md

Found a Bug or Have a Feature Request?

We welcome community feedback, bug reports, and suggestions!

How Pincer Works (For Everyone)

Welcome! If you’ve ever wondered how Pincer downloads files so quickly, or what terms like “BitTorrent” and “Concurrent Downloads” mean, this guide is for you. We’ve stripped away the technical jargon to explain how the engine works under the hood.

The Problem: Slow Downloads

Imagine you need to move 1,000 bricks from a factory to your house.

If you use a single truck (a standard web browser download), the truck has to drive to the factory, load a brick, drive back to your house, unload it, and repeat this process 1,000 times. If there is a traffic jam (a slow network connection) on the route, the truck gets stuck, and your bricks arrive very slowly.

The Solution: Concurrent Downloads

Pincer solves this problem by using multiple trucks simultaneously.

When you ask Pincer to download a file, it does the following:

1. It asks the factory, “How many bricks are there in total?” (Checking the file size).
2. It divides the total number of bricks into equal chunks (e.g., 8 chunks of 125 bricks).
3. It sends 8 trucks at the same time (Concurrent Workers). Each truck takes a different route to the factory, picks up its assigned chunk, and drives it straight to your house.

If one truck gets stuck in a traffic jam, the other 7 trucks keep moving. This ensures that you get your bricks (your file) as quickly as your driveway (your internet connection) can possibly handle.

BitTorrent: The Community Factory

Sometimes, a single factory might not have enough trucks to give you your bricks quickly, or the factory might be closed.

BitTorrent is a different way of getting bricks. Instead of going to one central factory, you ask your neighborhood: “Does anyone have these bricks?”

• Your neighbor Bob might have bricks 1 to 50.
• Your neighbor Alice might have bricks 51 to 100.

Pincer connects to Bob, Alice, and anyone else who has the bricks you need. It sends trucks to all of their houses at the same time. The more people who have the file (called Seeders), the faster you can get it. Once you have some of the bricks, Pincer will politely let other neighbors copy those bricks from you (this is called Uploading or Seeding).

Zero-Allocation Writing (The Magic House)

Normally, when a truck arrives at your house, it drops the bricks on the lawn, and later someone has to move them inside and build the wall (copying data from computer memory to the hard drive).

Pincer uses a technique called Zero-Allocation Writing. We build a “magic house” where the wall is already built, but all the bricks are hollow (pre-allocating the file on your hard drive). When a truck arrives, it places its bricks exactly where they belong in the wall immediately. There’s no moving things around on the lawn, which saves a massive amount of time and computer power.

Media Conversion

Sometimes, you download a video or an image, but it’s not in the format you want (like downloading an .iso when you wanted an .mp4). Pincer can automatically use tools built into your computer to translate the file for you as soon as the download finishes. It’s like having an automatic translator waiting at your front door!

Next Steps: Ready to start downloading? Head over to the User Guide to learn how to use the Pincer application!

Pincer User Guide

Welcome to the Pincer Engine! This guide will walk you through how to start downloading files quickly and easily using the Command Line Interface (CLI).

If you are a developer looking to integrate Pincer into your own app or use the advanced API, check out the API docs. For the full CLI flag reference, see CLI Reference.

1. Downloading a Standard File

To download a file from the internet, you just need its URL (the web address, like https://example.com/movie.mp4).

Open your terminal and type:

./target/release/pincer "https://example.com/movie.mp4" --log

Pincer will automatically split the file and start downloading it as fast as possible. By passing the --log or -l flag, you’ll see a live progress bar on your screen showing the speed, time remaining, and the percentage completed.

Choosing Where to Save

By default, Pincer saves the file in the folder where you ran the command. If you want to save it somewhere else (like your Downloads folder), use the --dir option:

./target/release/pincer "https://example.com/movie.mp4" --dir ~/Downloads --log

2. Speeding It Up (Or Slowing It Down)

If you are on a slow connection or a shared network, you might not want Pincer to use all of the available internet speed.

You can limit the speed using the --max-download-limit option (e.g., 5M for 5 Megabytes per second):

./target/release/pincer "https://example.com/movie.mp4" --max-download-limit 5M --log

If you want to speed things up by using even more “trucks” (concurrent connections), you can change the --split option. The default is 4, but you can increase it up to 16:

./target/release/pincer "https://example.com/movie.mp4" --split 16 --log

(Note: Setting this too high might cause the server to temporarily block you, so 4 to 8 is usually a safe sweet spot!)

3. Downloading BitTorrent (Magnet Links)

Pincer has a powerful, built-in BitTorrent engine. To download a torrent, you don’t need any special settings. Just pass the Magnet link (which usually starts with magnet:?xt=...) just like a normal URL:

./target/release/pincer "magnet:?xt=urn:btih:dd8255ecdc7ca55fb0bbf81323d87062db1f6d1c&dn=Ubuntu" --dir ~/Downloads --log

Pincer will automatically connect to the BitTorrent network, find neighbors who have the file, and download it for you.

4. Pausing and Resuming

If your internet drops or you need to turn off your computer, don’t worry!

Pincer creates a special .download file while it is working. If you cancel the download (by pressing Ctrl+C in your terminal) or if it gets interrupted, simply run the exact same command again later. Pincer will see the .download file, figure out exactly where it left off, and resume without losing any progress!

5. Converting Files Automatically

If you are downloading a video or an image and want it in a different format, you can tell Pincer to convert it the moment the download finishes using the --format (or -f) option:

./target/release/pincer "https://example.com/image.png" --format jpg --log

Pincer will download the PNG image, and then automatically convert it into a JPG for you!

Need More Help?

If you ever forget a command, you can ask Pincer for help by typing:

./target/release/pincer --help

This will print out a list of all the cool things Pincer can do!

CLI Reference

Complete reference for using Pincer as a standalone command-line download tool.

See also: User Guide for quick-start examples · Session & Resume for persistence details

Basic Usage

For quick direct downloads without using the RPC server:

./target/release/pincer "https://example.com/file.zip"

This will download the file to the current directory using multiple threads by default.

Argument Reference

Planned CLI Flags

The CLI parser (using lexopt) will be expanded to support additional flags:

• -v or --version
• -V or --check-integrity
• -j or --max-concurrent-downloads
• -c or --continue (Resumption)

Examples

High-Speed 16-Thread Download:

./pincer "https://example.com/movie.mp4" -s 16 -d ~/Downloads -o holiday_video.mp4

How Threading Works

• Rust-Native Performance: Built on tokio for non-blocking I/O and zero-allocation disk writes.
• Dynamic Thread Scaling: Supports up to 99 threads per task. The engine automatically splits the file into equal byte-ranges and assigns a dedicated worker to each.
• Range Support Detection: If a server does not support Accept-Ranges, Pincer gracefully falls back to a single thread to ensure data integrity.

Download threads are dynamically controlled via the JSON-RPC interface when adding or modifying a task using options like split and min-split-size. See Configuration for all available options.

Verifying Multi-Threading (Worker Logs)

To verify that Pincer is correctly utilizing multi-threading, observe the internal worker logs during a CLI download.

Example Log Output (-s 16):

🚀 Pincer CLI Mode
URL: https://images.pexels.com/.../photo.jpeg
Threads: 16
-------------------------------------------
[INFO] [Worker 0] Assigned range: 0 - 47938 (47939 bytes)
[INFO] [Worker 1] Assigned range: 47939 - 95877 (47939 bytes)
...
[INFO] [Worker 0] Connection established. Segment range: 0 - 47938
[INFO] [Worker 2] Connection established. Segment range: 95878 - 143816
...
[========================================] 100.00%
✅ Download complete!

Connecting to Pincer

How to connect to Pincer’s JSON-RPC interface and authenticate your requests.

Next: Managing Downloads

Connection Details

• Default Port: 6842 (configurable via -p / --port CLI option)
• WebSocket URL: ws://127.0.0.1:<PORT>/jsonrpc
• Protocol: JSON-RPC 2.0

Pincer features an integrated RPC layer powered by Axum, which handles incoming WebSocket connections and processes JSON-RPC commands. All methods use the pin.* namespace.

Authentication

If an RPC secret is configured, pass it as the first element of the params array in the format "token:YOUR_SECRET".

Example (pin.addUri):

{
  "jsonrpc": "2.0",
  "method": "pin.addUri",
  "id": "1",
  "params": [
    "token:mysecret",
    ["https://example.com/file.zip"],
    {"dir": "/downloads", "split": 8}
  ]
}

What Makes Pincer Different

• Native Memory Safety: Written entirely in Rust, preventing the segfaults and memory leaks possible in other C-based engines.
• Async I/O Worker Pool: Uses a highly concurrent tokio pattern for zero-allocation disk writes, offering lower CPU overhead on high-speed connections.
• First-Class WebSocket Layer: Powered by axum for modern, efficient communication.

Managing Downloads

Methods for adding, removing, pausing, and controlling download tasks.

Previous: Connecting · Next: Checking Status

Methods

Task Options

When passing an options object to pin.addUri or pin.changeOption, these are the most common keys:

• dir: Target directory to store the file.
• out: The file name of the downloaded file.
• split: (Integer) Number of connections to use (Default: 5).
• max-connection-per-server: (Integer) Max connections to a single server (Default: 1).
• min-split-size: (String) Minimum size to split a chunk (e.g., 1M).
• max-download-limit: (String) Speed limit for the download (e.g., 50K, 0 for unlimited).
• header: (Array of Strings) Custom HTTP Headers.
• user-agent: (String) Custom User-Agent string.

For the complete list of all configuration keys and defaults, see Configuration.

Checking Status

Methods for querying download progress, task metadata, and global engine statistics.

Previous: Managing Downloads · Next: Configuration

Methods

Configuration

Methods for reading and modifying engine configuration, plus the complete reference of all configuration keys and their defaults.

Previous: Checking Status · Next: Advanced Features

Configuration Methods

Default Engine Configuration (pincer.conf style)

RPC

• enable-rpc=true
• rpc-allow-origin-all=true
• rpc-listen-all=true

File System

• auto-save-interval=10
• disk-cache=64M
• file-allocation=none
• no-file-allocation-limit=64M
• save-session-interval=10

Task Parameters

• check-certificate=false
• max-file-not-found=10
• max-tries=0
• retry-wait=10
• connect-timeout=10
• timeout=10
• min-split-size=1M
• http-accept-gzip=true
• remote-time=true
• summary-interval=0
• content-disposition-default-utf8=true

BitTorrent Parameters

• bt-detach-seed-only=true
• bt-enable-lpd=true
• bt-hash-check-seed=true
• bt-max-peers=128
• bt-prioritize-piece=head
• bt-remove-unselected-file=true
• bt-seed-unverified=false
• bt-tracker-connect-timeout=10
• bt-tracker-timeout=10
• enable-dht=true
• enable-dht6=true
• enable-peer-exchange=true
• dht-entry-point=dht.transmissionbt.com:6881
• dht-entry-point6=dht.transmissionbt.com:6881
• peer-agent=Transmission/3.00
• peer-id-prefix=-TR3000-

Note: Changes to rpc-listen-port, rpc-secret, listen-port, and dht-listen-port require an engine restart to take effect.

User Preferences (UI / Application Level)

These keys are tracked by the frontend and used for application-level state:

auto-check-update, auto-hide-window, auto-sync-tracker, cookie, enable-upnp, engine-bin-path, engine-max-connection-per-server, favorite-directories, hide-app-menu, history-directories, keep-seeding, keep-window-state, last-check-update-time, last-sync-tracker-time, locale, log-level, new-task-show-downloading, no-confirm-before-delete-task, open-at-login, protocols, proxy, resume-all-when-app-launched, run-mode, show-progress-bar, task-notification, theme, tracker-source, tray-speedometer.

System Keys (Global/Session Level)

These keys map directly to global or task option updates (pin.changeGlobalOption or pin.changeOption):

all-proxy-passwd, all-proxy-user, all-proxy, allow-overwrite, allow-piece-length-change, always-resume, async-dns, auto-file-renaming, bt-enable-hook-after-hash-check, bt-enable-lpd, bt-exclude-tracker, bt-external-ip, bt-force-encryption, bt-hash-check-seed, bt-load-saved-metadata, bt-max-peers, bt-metadata-only, bt-min-crypto-level, bt-prioritize-piece, bt-remove-unselected-file, bt-request-peer-speed-limit, bt-require-crypto, bt-save-metadata, bt-seed-unverified, bt-stop-timeout, bt-tracker-connect-timeout, bt-tracker-interval, bt-tracker-timeout, bt-tracker, check-integrity, checksum, conditional-get, connect-timeout, content-disposition-default-utf8, continue, dht-file-path, dht-file-path6, dht-listen-port, dir, dry-run, enable-http-keep-alive, enable-http-pipelining, enable-mmap, enable-peer-exchange, file-allocation, follow-metalink, follow-torrent, force-save, force-sequential, ftp-passwd, ftp-pasv, ftp-proxy-passwd, ftp-proxy-user, ftp-proxy, ftp-reuse-connection, ftp-type, ftp-user, gid, hash-check-only, header, http-accept-gzip, http-auth-challenge, http-no-cache, http-passwd, http-proxy-passwd, http-proxy-user, http-proxy, http-user, https-proxy-passwd, https-proxy-user, https-proxy, index-out, listen-port, lowest-speed-limit, max-concurrent-downloads, max-connection-per-server, max-download-limit, max-file-not-found, max-mmap-limit, max-overall-download-limit, max-overall-upload-limit, max-resume-failure-tries, max-tries, max-upload-limit, metalink-base-uri, metalink-enable-unique-protocol, metalink-language, metalink-location, metalink-os, metalink-preferred-protocol, metalink-version, min-split-size, no-file-allocation-limit, no-netrc, no-proxy, no-want-digest-header, out, parameterized-uri, pause-metadata, pause, piece-length, proxy-method, realtime-chunk-checksum, referer, remote-time, remove-control-file, retry-wait, reuse-uri, rpc-listen-port, rpc-save-upload-metadata, rpc-secret, seed-ratio, seed-time, select-file, split, ssh-host-key-md, stream-piece-selector, timeout, uri-selector, use-head, user-agent.

Advanced Features

Detailed documentation of Pincer’s advanced capabilities beyond basic download management.

Previous: Configuration · Next: Events

URL Resolution (pin.resolveUrl)

Pincer features a powerful URL resolution engine that handles more than just simple redirects.

• Universal Redirects: Follows HTTP redirects to find the final CDN link.
• Metadata Extraction: Extracts filenames from Content-Disposition headers.
• Platform Scraping: Automatically identifies and extracts high-quality video links from platforms like Pexels by parsing NEXT_DATA or meta tags.
• Resumability Check: Verifies if the server supports range requests before starting.

Real-Time File Format Conversion

Pincer features an integrated format conversion engine that triggers automatically upon download completion if a different file format is requested.

• Image Conversion: Utilizes macOS sips for standard image transcoding between formats like png, jpg/jpeg, webp, heic/heif.
  • PDF Support: Utilizes sips with an automatic built-in fallback to macOS’s native cupsfilter utility for extremely reliable PDF generation.
• Audio/Video Conversion: Uses ffmpeg (if globally installed) or falls back to macOS’s native afconvert utility for audio (mp3, wav, m4a, aac).
• Converting Status: During the transcoding phase, the task’s state changes to "converting" before final completion.

Multi-Protocol Support (FTP & SFTP)

Beyond standard HTTP/HTTPS, Pincer Engine fully supports legacy and secure file transfer protocols.

• FTP (ftp://): Supports anonymous login natively. Connections seamlessly plug into Pincer’s DownloadWorker pool for multi-threaded downloads.
• SFTP (sftp://): Supports completely secure file transfers over SSH. Handles passwordless authentication automatically by leveraging the user’s local SSH agent.

Metalink & Multi-Source Failover

Pincer can parse .meta4 (Metalink) XML files using the pin.addMetalink JSON-RPC method, granting highly resilient downloads.

• XML Parsing: Uses safe and fast XML parsing via quick-xml. Simply provide the base64-encoded XML document to the RPC method.
• Multi-Source Failover: Metalink files often provide multiple fallback URIs for a single file. If Pincer encounters a connection error (e.g. timeout, connection refused) from the priority 1 server while downloading a specific chunk, the worker will automatically retry that chunk using the priority 2 fallback server without aborting the task.
• Checksum Validation: If the Metalink XML provides a <hash type="sha-256"> node, Pincer engine automatically computes the SHA-256 hash of the final file on a separate non-blocking thread post-download. If corruption is detected, the task is safely set to an error state.

Global Speed Modes

The speed-mode option in pin.changeGlobalOption allows for high-level bandwidth control:

• max_bandwidth (or max): No limit applied.
• half_bandwidth (or half): Limits speed to 50% of the maximum speed seen during the current session.
• min_bandwidth (or min): Limits speed to a sub-kb/s level (approx. 768 bytes/s) without pausing, keeping connections alive.

Trash Integration

The pin.removeAndFile method utilizes the system trash (e.g., macOS Trash) rather than performing a permanent deletion. This provides a safety net for users who may want to recover a deleted download.

Native Safe Restarts (Non-Resumable Tasks)

The engine natively protects against corrupted files when dealing with non-resumable connections. If you invoke pin.unpause on a non-resumable task that previously failed or was interrupted, the backend engine automatically resets the task’s offset to zero and permanently deletes (rm -rf) the partially downloaded corrupt file from the disk before restarting the worker. This handles the complex teardown process instantaneously without requiring multi-step RPC interactions.

Chronological Queue Order (FIFO Scheduling)

Pincer implements strict First-In-First-Out (FIFO) queueing. Every task is tagged with a precise created_at Unix millisecond timestamp upon addition. When running multiple downloads under a concurrency limit, the engine strictly schedules and spawns waiting tasks chronologically.

Events

Pincer streams real-time status notifications to all connected WebSocket clients. You do not need to poll for updates.

Previous: Advanced Features · Next: System Controls

Event Reference

Notification Format

All events follow the standard JSON-RPC 2.0 notification format (no id field):

{
  "jsonrpc": "2.0",
  "method": "pin.onDownloadComplete",
  "params": [{"gid": "2089b05ecca3d829"}]
}

The params array contains a single object with the gid (task ID) that triggered the event.

System Controls

Methods for managing engine lifecycle, session control, and batch operations.

Previous: Events

Methods

Session & Resume

How Pincer automatically saves progress and resumes interrupted downloads.

Overview

Pincer includes a built-in persistence layer. All tasks, global options, and progress are saved to a session file automatically.

How It Works

• Session File: pincer.session (JSON format).
• Auto-Save: Triggered on every significant state change (adding a task, pausing, or changing options).
• Auto-Load: On startup, Pincer detects the session file and restores all tasks.
• Resumption: Interrupted downloads will automatically resume from the last successfully written byte using HTTP Range requests.

Manual Session Control

You can also manage sessions explicitly via the JSON-RPC API:

For the full system methods reference, see System Controls.

Swift Integration Guide

Step-by-step guide to integrating Pincer Engine into a macOS Swift/SwiftUI application.

Overview

Pincer Engine is designed to run silently as a backend daemon for frontend graphical user interfaces, such as macOS applications built with Swift/SwiftUI.

To integrate Pincer Engine’s capabilities (like FTP/SFTP support, multi-source Metalink fallback, auto-format conversion, etc.) into your UI application, follow this standardized 4-step architecture:

Step 1: Spawn the Engine as a Background Daemon

Bundle the compiled pincer binary inside your macOS App bundle. When your app launches, use Process (NSTask) to spawn the engine in daemon mode (-D) on a specific port.

let task = Process()
task.executableURL = Bundle.main.url(forResource: "pincer", withExtension: nil)
// Enable RPC daemon on port 6800, using the background detached flag
task.arguments = ["--enable-rpc=true", "--rpc-listen-port=6800", "-D"]
try? task.run()

Step 2: Connect to the WebSocket RPC Server

Once spawned, Pincer runs a lightweight WebSocket server locally. Establish a persistent WebSocket connection from your Swift app to send commands and receive real-time updates.

let url = URL(string: "ws://localhost:6800/jsonrpc")!
let session = URLSession(configuration: .default)
let webSocketTask = session.webSocketTask(with: url)
webSocketTask.resume()

For connection details (ports, auth tokens), see Connecting.

Step 3: Trigger Features via JSON-RPC

Instead of executing complex CLI commands, your UI will trigger Pincer’s features by sending standardized JSON-RPC payloads over the WebSocket.

All of Pincer’s features are invoked similarly. For example, to trigger the Metalink multi-source download feature:

// Example: Triggering a Metalink Task
let payload: [String: Any] = [
    "jsonrpc": "2.0",
    "id": UUID().uuidString,
    "method": "pin.addMetalink",
    "params": [
        "PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0i...", // Base64 encoded payload
        ["dir": "/Users/Shared/Downloads", "split": "16"] // Engine options
    ]
]
let jsonData = try! JSONSerialization.data(withJSONObject: payload)
webSocketTask.send(.data(jsonData)) { error in ... }

(Whether you are using pin.addUri for FTP links, or pin.changeGlobalOption to throttle speeds, the JSON-RPC interface remains identical).

For all available methods, see Managing Downloads.

Step 4: Map Introspection Models and Listen to Events

Pincer streams real-time status events back to your WebSocket. You do not need to poll manually. Create Swift Codable structs that map to Pincer’s introspection models (e.g., PincerFile, PincerUri, TaskStatus) to cleanly decode these incoming JSON payloads.

// Ensure you catch Pincer's standardized events to update your UI:
// - pin.onDownloadStart
// - pin.onDownloadComplete
// - pin.onDownloadError (e.g. emitted if a checksum validation fails!)

webSocketTask.receive { result in
    switch result {
    case .success(.string(let text)):
        // Decode the JSON-RPC response and update your UI progress bars
        let response = try? JSONDecoder().decode(PincerRPCResponse.self, from: text.data(using: .utf8)!)
    case .failure(let error):
        print("WebSocket Error: \(error)")
    }
}

For the full list of events, see Events.

Summary

By keeping the heavy lifting inside the Pincer Rust engine, your Swift frontend can remain incredibly lightweight—simply sending JSON-RPC commands and painting the UI based on the incoming WebSocket event stream.

API Comparison

How Pincer’s API methods map to standard download management patterns, demonstrating full coverage.

Task Addition & Management

Status & Monitoring

Configuration, History & System

Events

Getting Started as a Contributor

Thank you for considering contributing to Pincer! Your help makes this high-performance download engine better for everyone.

Prerequisites

You’ll need the Rust toolchain. See Build & Release for full setup instructions.

Code Style & SRP Rules

Before submitting a pull request, please adhere to the following rules:

1. Adhere to SRP (~100 lines per file): Keep modules focused on a single responsibility. See Architecture for how the codebase is organized.
2. Follow Rust naming conventions: Use snake_case for variables/functions, CamelCase for types/structs. Run cargo clippy to catch common mistakes.
3. Write descriptive commit messages: Start with a short summary in the imperative mood (e.g., “Add proxy support”).
4. Update Documentation: When modifying user-facing behavior or adding new features, update the relevant documentation in docs/.
5. Zero regressions: Ensure all tests pass. See Testing for how to run the test suite.

Pull Request Workflow

1. Fork the repository and create your feature branch.
2. Ensure your code builds cleanly and passes all lints:

   cargo fmt --all -- --check
   cargo clippy --all-targets -- -D warnings
   

3. Write test cases where appropriate.
4. Submit a Pull Request (PR) with a clear description of changes.

Happy coding!

Architecture & Design

This document provides a comprehensive technical overview of the Pincer architecture, internal subsystems, concurrency models, and domain modules.

Architectural Philosophy

Pincer is architected strictly around the Single Responsibility Principle (SRP). Every source file is isolated to a focused domain responsibility (~100 lines per module) to ensure maintainability, auditability, and safety.

pincer-engine/
├── src/
│   ├── cli/            # Command-line interface, argument parsing, ANSI UI, daemonizer
│   ├── common/         # Common utilities, error types, URI expansion, sanitization
│   ├── converter/      # Media transcoding subsystem (sips, cupsfilter, ffmpeg, afconvert)
│   ├── engine/         # Core HTTP/FTP chunking, worker streams, disk allocation, throttling
│   ├── manager/        # Task registry, scheduling, persistence, lifecycle, query facade
│   ├── metalink/       # Metalink XML 3.0 / 4.0 file and hash parser
│   ├── models/         # Strongly-typed serde data transfer and domain models
│   ├── protocol/       # Protocol adapters (HTTP/S, FTP, SFTP)
│   ├── resolver/       # URL, Magnet, Torrent, and media scraping resolvers
│   ├── rpc/            # Axum WebSocket server, JSON-RPC 2.0 router, token auth, method handlers
│   └── torrent/        # BitTorrent integration engine (librqbit session, stats, file selector)

Core Subsystems

Protocol Adapter Layer (src/protocol/)

Provides uniform stream abstractions via the ProtocolAdapter trait:

• HttpAdapter: Asynchronous byte-range streaming and metadata discovery via reqwest.
• FtpAdapter: FTP/FTPS segmented streaming using suppaftp.
• SftpAdapter: SSH/SFTP streaming and metadata discovery using russh and russh-sftp.

Core Execution Engine (src/engine/)

• DiskAllocator: Zero-fragmentation space pre-allocation (set_len).
• DownloadBundle: macOS .download staging bundle structure, Info.plist UTI assignment, and com.apple.quarantine handling.
• RangeChunker: Parallel byte-range partitioner and resume offset arithmetic.
• DownloadWorker: Non-blocking zero-allocation disk writes using POSIX write_at on an Arc<File> across concurrent threads without lock contention.
• RateThrottler: Proportional bandwidth allocation using the ThreadGuard RAII active thread counter.

BitTorrent Integration Engine (src/torrent/)

• TorrentSessionManager: Configures librqbit::Session with port range 6881..6891 and automatic ephemeral port fallback (0..1).
• TorrentTaskSpawner: Inspects Magnet and Torrent byte sources and prepares isolated directory trees.
• TorrentStatsTracker: Computes instantaneous download/upload speeds, seeders, and peer snapshots.
• TorrentFileSelector: Cleans up unselected files and prunes empty directory branches.

Universal Media & Metadata Resolver (src/resolver/)

• TorrentResolver: List-only metadata extraction for Magnet links and Base64-encoded .torrent files.
• DirectHttpResolver: Content-Type, Content-Length, Content-Disposition, and Accept-Ranges resolution.
• HtmlScraper & ScriptExtractor: OpenGraph/Twitter card extraction, Next.js embedded JSON scraping, and direct video stream detection.

Media Transcoder (src/converter/)

• ImageSips: macOS sips image conversion (jpeg, png, webp, heic).
• CupsPdf: macOS cupsfilter raster-to-PDF conversion.
• Ffmpeg: Universal media transcoding.
• Afconvert: macOS afconvert audio encoding.
• Includes rollback restoration that re-establishes original files if conversion fails.

Central Manager (src/manager/)

• TaskSpawner & TaskRunner: Coordinates task lifecycle, execution, SHA256 integrity verification, and quarantine removal.
• TaskLifecycleManager: Safe pause/unpause state machine transitions.
• TaskRemovalManager: Atomic task deletion and non-blocking background Trash operations (trash::delete).
• SessionPersistence: Session serialization to ~/.pincer/pincer.session and .download/state.json.

JSON-RPC 2.0 Layer (src/rpc/)

• Axum WebSocket server at ws://127.0.0.1:6842/jsonrpc.
• Native pin.* and system.* namespace dispatch.
• Per-connection authentication with token:<secret> support.
• Real-time broadcast notification channel (pin.onDownloadStart, pin.onDownloadProgress, pin.onDownloadComplete, etc.).

Concurrency & Disk I/O Model

                    ┌─────────────────────────┐
                    │    DownloadManager      │
                    └────────────┬────────────┘
                                 │
           ┌─────────────────────┼─────────────────────┐
           ▼                     ▼                     ▼
   ┌───────────────┐     ┌───────────────┐     ┌───────────────┐
   │ Worker 1      │     │ Worker 2      │     │ Worker N      │
   │ (Range 0..A)  │     │ (Range A..B)  │     │ (Range B..End)│
   └───────┬───────┘     └───────┬───────┘     └───────┬───────┘
           │                     │                     │
           └─────────────────────┼─────────────────────┘
                                 ▼
                     ┌───────────────────────┐
                     │ Shared Arc<File>      │
                     │ (write_at POSIX I/O)  │
                     └───────────────────────┘
                                 │
                                 ▼
                     ┌───────────────────────┐
                     │ Staging .download     │
                     │ Bundle (with UTI)     │
                     └───────────────────────┘

Testing Pincer

This document outlines the testing architecture and procedures for the Pincer RPC server.

Why Python for Testing?

Using Python (specifically unittest.IsolatedAsyncioTestCase and websockets) allows us to perform black-box integration testing. By interacting with Pincer over WebSockets exactly as a real client would, we ensure that the compiled Rust binary and its RPC interface function correctly in real-world scenarios. It also allows us to quickly validate JSON-RPC structures without the boilerplate of a compiled test harness.

Test Suite Architecture

The test suite is located in the tests/ directory and is split into two parts:

1. tests/test_rpc.py: Tests the WebSocket JSON-RPC server daemon.
2. tests/test_cli.py: Tests the direct Command Line Interface (CLI) downloads.

It utilizes Python’s built-in unittest module to minimize external dependencies.

What is Tested?

The test suite covers the active features exposed by the modular src/rpc/ backend, grouped into logical blocks:

1. System & Info (test_01_get_version, test_06_misc):
   • pin.getVersion, pin.resolveUrl, pin.resolveTorrent, pin.saveSession
2. Global Options (test_02_global_options):
   • pin.getGlobalOption, pin.changeGlobalOption
3. Task Lifecycle (test_03_lifecycle, test_08_pause_resume_integrity):
   • pin.addUri, pin.addTorrent, pin.addMetalink, pin.tellStatus, pin.pause, pin.unpause, pin.changeOption, pin.getOption, pin.remove, pin.removeAndFile, pin.forceRemove
4. Bulk Operations & Stats (test_04_bulk_and_stats):
   • pin.pauseAll, pin.unpauseAll, pin.tellActive, pin.tellWaiting, pin.tellStopped, pin.getGlobalStat
5. Result Management (test_05_cleanup):
   • pin.purgeDownloadResult, pin.removeDownloadResult
6. Security & Path Traversal (test_07_path_traversal):
   • Path escaping mitigation and filename sanitization verification.

CLI Tests (tests/test_cli.py)

This suite tests the direct binary execution (pincer [URL] [OPTIONS]) without starting the daemon. It covers:

1. Help & Version: Output of --help and --version.
2. Direct Downloads: Downloading a file directly via CLI arguments (--out, --split, --dir, etc.) and verifying the file is written to disk successfully.

How to Run the Tests

The easiest and recommended way to run the entire test suite is using the automated test runner script.

Note

If your goal is to validate the codebase before cutting a new release, please refer to Build & Release. The automated release script delegates its checks directly to the unified test runner discussed below.

The Automated Test Runner (scripts/run_tests.py)

Pincer includes an all-in-one test runner script at scripts/run_tests.py. This script manages the entire lifecycle of CI verification, compilation, execution, and cleanup.

What the Runner Does:

1. Conflicting Process Check: Automatically scans for and terminates any pre-existing running Pincer processes to prevent port bind conflicts on 6842.
2. CI Verification Checks:
   • Runs cargo fmt --all -- --check (attempts to auto-format using cargo fmt if check fails).
   • Runs cargo check --all-targets to verify code compiles.
   • Runs cargo clippy --all-targets -- -D warnings to verify zero lint warnings.
3. Build Stage: Builds the debug binary (cargo build).
4. Session Cleanup: Removes any stale/leftover session file from ~/.pincer/pincer.session.
5. CLI Tests: Runs tests/test_cli.py to verify direct download functionality.
6. Background Server: Spawns Pincer in the background, waiting for it to spin up and bind the socket.
7. RPC Tests: Runs tests/test_rpc.py to test websocket JSON-RPC methods end-to-end.
8. Server Shutdown: Properly terminates the background server.
9. Cleanup Prompt: Asks if you want to clean up temporary test files in tests/temporary.

How to Run:

# Ensure dependency is installed
pip install websockets

# Run the test suite
python3 scripts/run_tests.py

Running Tests Manually

If you prefer to run CLI or RPC tests independently:

RPC Tests (tests/test_rpc.py)

The Pincer server must be running in a separate process.

1. Start the Rust backend:

   cargo run
   

   The server should log that it is listening on ws://0.0.0.0:6842/jsonrpc.

2. In a separate terminal:

   python3 -m unittest tests/test_rpc.py
   

CLI Tests (tests/test_cli.py)

These do not require the server to be running:

python3 -m unittest tests/test_cli.py

Build & Release

Everything you need to know about building Pincer from source, setting up the development environment, and managing new releases.

Prerequisites

Pincer is written entirely in Rust. You will need the Rust toolchain installed:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After installation, reload your environment:

source $HOME/.cargo/env

How to Build

1. Clone the repository:

   git clone https://github.com/<GITHUB_OWNER>/Pincer-Engine.git
   cd Pincer-Engine
   

   If you use environment variables in your scripts or CI, set GITHUB_OWNER to the GitHub account and GITHUB_REPO to Pincer-Engine.

   If you use local automation, make sure GITHUB_OWNER and GITHUB_REPO are defined before running scripts that reference repository URLs.

2. Run in development/RPC server mode (defaults to port 6842 over WebSocket):

   cargo run
   

3. Run the standalone CLI to test direct downloads:

   TEST_URL="https://images.unsplash.com/photo-1446941303752-a64bb1048d54?ixlib=rb-4.1.0&q=85&fm=jpg&crop=entropy&cs=srgb&dl=nasa-U2uKrI4lci8-unsplash.jpg"
   cargo run -- "$TEST_URL" -s 8 -d ./
   

4. Compile a highly optimized standalone binary for production:

   cargo build --release
   

   After a release build, the executable is located at target/release/pincer.

How to Release

We use a unified release script to ensure that all checks pass before a release version is tagged.

The scripts/release.sh script automates:

1. CI validation: Executes the unified test suite (python3 tests/run_tests.py --ci), which performs formatting checks, linting, Rust unit tests, and the full suite of Python integration tests.
2. Release build: Ensures the application compiles successfully under --release.
3. Version bump: Updates the version automatically inside Cargo.toml.
4. Git Tagging: Commits the version bump and tags the commit with the new version (e.g. v1.5.0).

Running the Release Script

You must provide the new semantic version as an argument:

./scripts/release.sh <new_version>

Example:

./scripts/release.sh 1.5.0

Once the script completes successfully, you will be prompted to push the new tag and commit:

git push origin <branch-name> --tags

Pushing the v* tag triggers the .github/workflows/release.yml GitHub Action, which builds and publishes the pre-compiled .zip artifacts to the GitHub Releases page automatically.

Documentation Website (GitHub Pages)

The documentation is hosted on GitHub Pages and powered by mdBook. All source markdown files in docs/ are rendered into an interactive documentation site with dynamic release downloads and bug tracking.

Local Documentation Preview

To preview the documentation locally with live-reload:

# Install mdbook via Cargo
cargo install mdbook

# Serve and open the documentation at http://localhost:3000
mdbook serve --open

Automatic Deployment

Whenever updates to docs/, theme/, or book.toml are pushed to the main branch, the .github/workflows/deploy-docs.yml workflow automatically builds the documentation site and deploys it to GitHub Pages.

Roadmap

Dependency migration strategy and technical optimization paths for Pincer Engine.

For what features are done and what’s next, see Feature Status.

Current Architecture Overview

Pincer is built on the tokio async runtime and consists of three primary layers:

1. RPC Layer (Axum): Handles incoming WebSocket connections and processes JSON-RPC commands.
2. Management Layer: Tracks download states, provides thread-safe access to task metadata, and manages global throughput stats.
3. Worker Pool: Each task spawns multiple workers that independently fetch segments and perform non-blocking concurrent writes to the disk using zero-allocation writes.

For a deep-dive into each module, see Architecture.

Networking & Transfer (The Engine)

The core of your tool. Higher levels offer safety and edge-case handling; lower levels offer raw speed and absolute control.

Server Interface (API/Remote Control)

Serialization (Data Handling)

Command Line Interface (CLI)

Utilities & Logic

Technical Strategy for Faster Download Speeds

As you move from High-Level to Low-Level, your ability to optimize speed increases, but your “Edge-Case” safety decreases.

To Maintain Max Speed with Min Dependencies:

1. Direct I/O (The “Pincer” Secret): Regardless of the library, use file.set_len() to pre-allocate space on disk. This prevents filesystem fragmentation during multi-connection downloads.

2. Zero-Copy Buffering: Instead of using bytes crate (Level 4), use std::io::BufWriter with a manually tuned buffer size (usually 64KB or 128KB) to match the CPU cache line.

3. The Resumption Logic: To handle edge cases (flaky servers) without reqwest:

   • Level 4: Let the library handle retries.
   • Level 2: Implement a “Retry Loop” around your TcpStream. Check the Content-Length vs the local file size. Send Range: bytes=N- where $N$ is your current progress.

Summary Checklist for Migration

• Priority 1: Speed? Stick with tokio for the executor, but move from reqwest to hyper. This keeps the async efficiency but removes high-level bloat.
• Priority 2: Zero Dependencies? Move everything to the Level 1/2 column. You will gain a <1MB binary, but you must manually code the “Handshake” and “Retry” logic for every protocol.
• Priority 3: Edge Case Handling? Stay at Level 4. The open-source community has already fixed the bugs you haven’t encountered yet (e.g., specific header formats for old Nginx servers).

Feature Status

Current implementation status of Pincer Engine features. This document tracks what has been achieved, what remains, and the prioritized roadmap for upcoming work.

For dependency migration strategy, see Roadmap.

Achieved Features

• Command-line interface (lexopt)
• Download files through HTTP(S)
• FTP / SFTP Protocol Support: Seamless integration for anonymous and authenticated secure file transfers.
• Metalink Support: Full support for addMetalink & XML parsing, enabling robust multi-source failover and SHA-256 checksum verification.
• Daemon Mode: Running pincer-engine as a detached background service without a terminal window (--daemon).
• Concurrent Segmented downloading (up to 99 threads)
• Sequential Resume utilizing HTTP Range headers
• JSON-RPC (over WebSocket) interface for real-time status updates
• Session Persistence & Resumption (pincer.session auto-saving)
• Configuration File & Dynamic Options (pincer.conf support, changeOption)
• Download / Upload Speed Throttling (max-download-limit, speed-mode)
• Basic Advanced HTTP (Proxy and Auth properties integrated into settings)

Features Left to Implement

• Full RPC Standard Methods: system.multicall, system.listMethods, system.listNotifications, getSessionInfo.
• Detailed Task Introspection RPCs: getFiles, getUris, getPeers, getServers.
• Advanced Task Modification: changeUri, changePosition, forcePause, forcePauseAll.
• Batch Downloading (Parameterized URIs, Reading URIs from a text file)
• Netrc Support
• BitTorrent Support

Prioritized Next Steps

1. Missing RPC Standard & Introspection Methods (getFiles, getUris, system.multicall): Critical for GUI/Web UI frontends to display file contents.
2. Advanced Task Modification (changeUri, changePosition): Highly requested for dynamic download environments.
3. Batch Downloading & Parameterized URIs: Useful for downloading sequences (e.g., image_{1..100}.jpg).