Extract Structured HTML Data
using Unix-style Pipes

Pipsel is a Node.js and TypeScript compiler toolchain for a custom HTML parsing DSL. Write declarative, hierarchical extraction selectors, parse, format, and lint them programmatically or in the CLI.

🛡️ 100% Code Coverage 🎭 Playwright & Puppeteer Ready v0.0.7 Release 📜 MIT License

$ npm install pipsel

⛓️ Unix Pipe Transformers

Chain element content converters like text, html, and attr into value manipulators like trim, regex, split, and float using standard | pipe indicators.

🔍 Native Scoping / Nesting

Supports list blocks (with brackets []) and nested child objects. Scopes evaluation relative to matched CSS selectors for robust structure scraping.

🛠️ Robust Linting & AST

Catches duplicate keys, wrong arguments, unknown pipes, empty selectors, and invalid pipe order. Exposes a clean compiler API and CLI interface.

🤖 Browser Integration

Integrates natively with Playwright & Puppeteer. Exposes the fluent pipsel(page) / pipsel(locator) API with visual diagnostic debugging.

Live Scraper Playground

Write rules in the custom PSL editor on the left and see linting, AST structure, and browser-executed evaluation on the right!

PSL Extraction Rules (.psl)

HTML Source Document

Documentation Reference

1. Installation

Install Pipsel using your favorite package manager. It targets Node.js 20+ environments and exports standard TypeScript compiler declarations.

npm install pipsel

2. DSL Syntax Guide

Pipsel's domain syntax is JSON-like but structured specifically for DOM crawling. Definitions map properties to target selector rules.

Syntax Type	Example Snippet	Description
Simple Field	`title: "h1" \| text`	Extracts value from selector and applies converters.
Optional Field	`price?: ".price" \| text`	If selector is missing, returns `null` rather than raising warnings.
List Block	`items[]: ".item" { name: "h3" \| text }`	Iterates over selector array. Scopes children context to each item.
Primitive List	`tags[]: ".tag" \| text \| trim`	Extracts all matching elements into a flat primitive array (strings, numbers, etc.).
Meta Variables	`source: @url`	Injects run parameters (`@url`, `@timestamp`). No selector needed.
Smart Match	`title: @match("title")`	Resolves semantic selector matches based on normalized class names, test IDs, and synonyms.

3. Built-in Pipe Functions

The parser validates pipe names and parameter counts. The pipes execute in order from left to right.

Pipe Name	Arguments	Description
`text`	None	Extract text content from the selected element.
`html`	None	Extract outer/inner HTML content from the selected element.
`attr(name)`	`(name: string)`	Extract specified element attribute.
`trim`	None	Trim spacing around strings.
`lowercase / lower`	None	Converts string to lowercase.
`uppercase / upper`	None	Converts string to uppercase.
`titlecase / title`	None	Converts string to Title Case.
`slugify`	None	Converts string into a URL-safe slug.
`clean`	None	Collapses multiple spaces/newlines into a single space.
`prefix(value)`	`(value: string)`	Prepends the prefix string value.
`suffix(value)`	`(value: string)`	Appends the suffix string value.
`substring / slice`	`(start: number, end?: number)`	Extracts characters between start and end indices.
`trim_start / trim_end`	None	Trims whitespace from start or end of string.
`replace(from, to)`	`(from: string, to: string)`	Replace all occurrences of character/string sequences.
`regex(pattern)`	`(pattern: string)`	Match values using RegEx. Returns capture group or full match.
`split(separator)`	`(separator: string)`	Split string values into array.
`int`	None	Parse clean values into integer values.
`float`	None	Parse clean values into floating-point numbers.
`abs`	None	Returns the absolute value of a number.
`round(decimals)`	`(decimals?: number)`	Rounds a number to the specified decimal places (default 0).
`ceil / floor`	None	Rounds a number up (ceil) or down (floor).
`add(value) / subtract(value)`	`(value: number)`	Add or subtract a value.
`multiply(value) / divide(value)`	`(value: number)`	Multiply or divide by a value.
`min / max`	None	Returns the minimum or maximum of a numeric array.
`sum / avg / average`	None	Calculates sum or average of a numeric array.
`bool / boolean`	None	Converts values to booleans. Absent elements resolve to false.
`fallback(value)`	`(value: any)`	Default output if element selection evaluates to null or empty.
`filter(pattern)`	`(pattern: string)`	Filters array elements or checks strings against regex.
`unique(key)`	`(key?: string)`	De-duplicates arrays. Can filter object arrays using a target key.
`json_parse / json`	None	Parses serialized JSON string content into a structured JavaScript object.
`required(message)`	`(message?: string)`	Throws a validation error if the field/list matches 0 elements or evaluates nullish.
`url_hostname / url_param`	`(param?: string)`	Parses active URL structures to extract hostname, protocol, origin, or search query parameters.
`> / < / >= / <= / == / !=`	`(value: number \| string)`	Evaluates numerical or string comparison checks directly inside pipe sequences.

4. Programmatic API

Expose Pipsel in your backend services or custom tooling scripts. ESM modules are standard.

import { parse, format, lint } from "pipsel";

const psl = 'title: "h1" | text | trim';

// Format and clean rules code
const formatted = format(psl);

// Lint compiler configurations
const issues = lint(psl);
if (issues.length === 0) {
  // Produce compiler AST trees
  const ast = parse(psl);
  console.log(ast);
}

5. CLI Interface

Pipsel packages a CLI utility to check files locally. Run commands directly in your shell terminal.

# Format a rules file in-place
pipsel fmt rules.psl

# Lint diagnostic checks (exits with non-zero on error)
pipsel lint rules.psl

6. License

Pipsel is open-source software licensed under the MIT License.

Extract Structured HTML Data using Unix-style Pipes