Pipsel is a Node.js and TypeScript compiler toolchain for a custom HTML parsing DSL. Write declarative, hierarchical extraction selectors, parse, format, and lint them programmatically or in the CLI.
Chain element content converters like text, html, and attr into value manipulators like trim, regex, split, and float using standard | pipe indicators.
Supports list blocks (with brackets []) and nested child objects. Scopes evaluation relative to matched CSS selectors for robust structure scraping.
Catches duplicate keys, wrong arguments, unknown pipes, empty selectors, and invalid pipe order. Exposes a clean compiler API and CLI interface.
Write rules in the custom PSL editor on the left and see linting, AST structure, and browser-executed evaluation on the right!
Install Pipsel using your favorite package manager. It targets Node.js 20+ environments and exports standard TypeScript compiler declarations.
npm install pipselPipsel's domain syntax is JSON-like but structured specifically for DOM crawling. Definitions map properties to target selector rules.
| Syntax Type | Example Snippet | Description |
|---|---|---|
| Simple Field | title: "h1" | text |
Extracts value from selector and applies converters. |
| Optional Field | price?: ".price" | text |
If selector is missing, returns null rather than raising warnings. |
| List Block | items[]: ".item" { name: "h3" | text } |
Iterates over selector array. Scopes children context to each item. |
| Meta Variables | source: @url |
Injects run parameters (@url, @timestamp). No selector needed. |
The parser validates pipe names and parameter counts. The pipes execute in order from left to right.
| Pipe Name | Arguments | Description |
|---|---|---|
text |
None | Extract text content from the selected element. |
html |
None | Extract outer/inner HTML content from the selected element. |
attr(name) |
(name: string) |
Extract specified element attribute. |
trim |
None | Trim spacing around strings. |
replace(from, to) |
(from: string, to: string) |
Replace all occurrences of character/string sequences. |
regex(pattern) |
(pattern: string) |
Match values using RegEx. Returns capture group or full match. |
split(separator) |
(separator: string) |
Split string values into array. |
int |
None | Parse clean values into integer values. |
float |
None | Parse clean values into floating-point numbers. |
fallback(value) |
(value: any) |
Default output if element selection evaluates to null or empty. |
filter(pattern) |
(pattern: string) |
Filters array elements or checks strings against regex. |
Expose Pipsel in your backend services or custom tooling scripts. ESM modules are standard.
import { parse, format, lint } from "pipsel";
const psl = 'title: "h1" | text | trim';
// Format and clean rules code
const formatted = format(psl);
// Lint compiler configurations
const issues = lint(psl);
if (issues.length === 0) {
// Produce compiler AST trees
const ast = parse(psl);
console.log(ast);
}Pipsel packages a CLI utility to check files locally. Run commands directly in your shell terminal.
# Format a rules file in-place
pipsel fmt rules.psl
# Lint diagnostic checks (exits with non-zero on error)
pipsel lint rules.psl