初始化项目: Web Capture API v1.0.0

功能:
- 网页截图(全页或视口)
- HTML代码提取
- 自动滚动加载动态内容
- 自定义视口大小
- Web界面和RESTful API
- 支持两种后端: agent-browser 和 Playwright
- Docker部署支持

包含:
- Flask REST API
- 前端Web界面
- 测试脚本
- 安装脚本
- Dockerfile和docker-compose
- systemd服务配置
This commit is contained in:
2026-07-04 23:13:33 +08:00
commit ed42f65c56
11 changed files with 1554 additions and 0 deletions

36
.gitignore vendored Normal file
View File

@@ -0,0 +1,36 @@
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
venv/
env/
ENV/
.venv
# IDE
.vscode/
.idea/
*.swp
*.swo
# 测试
.pytest_cache/
.coverage
htmlcov/
# 临时文件
*.log
*.tmp
.DS_Store
Thumbs.db
# 截图输出(可选)
captures/
*.png
*.html
# 环境变量
.env
.env.local

58
Dockerfile Normal file
View File

@@ -0,0 +1,58 @@
FROM python:3.12-slim
LABEL maintainer="hz4th_coder@tphai.com"
LABEL description="Web Capture API - 网页截图与代码提取服务"
# 设置环境变量
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1
# 安装系统依赖
RUN apt-get update && apt-get install -y --no-install-recommends \
wget \
gnupg \
curl \
ca-certificates \
fonts-liberation \
libasound2 \
libatk-bridge2.0-0 \
libatk1.0-0 \
libatspi2.0-0 \
libcups2 \
libdbus-1-3 \
libdrm2 \
libgbm1 \
libgtk-3-0 \
libnspr4 \
libnss3 \
libwayland-client0 \
libxcomposite1 \
libxdamage1 \
libxfixes3 \
libxkbcommon0 \
libxrandr2 \
xdg-utils \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# 安装Python依赖
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# 安装Playwright浏览器
RUN playwright install chromium --with-deps
# 复制应用代码
COPY . .
# 创建临时目录
RUN mkdir -p /tmp/web_captures
EXPOSE 5000
# 健康检查
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
CMD curl -f http://localhost:5000/health || exit 1
CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "--timeout", "120", "app:app"]

277
README.md Normal file
View File

@@ -0,0 +1,277 @@
# Web Capture API
网页截图与代码提取服务
## 功能特性
- 📸 **网页截图** - 全页或视口截图
- 📄 **HTML提取** - 提取完整网页源码
- 🔄 **动态内容** - 支持滚动加载动态内容
- 🎯 **自定义视口** - 可调整浏览器窗口大小
-**RESTful API** - 简单易用的HTTP接口
- 🌐 **Web界面** - 直观的前端页面
## 系统要求
- Python 3.10+ 或 Python 3.12
- Node.js 18+ (用于 agent-browser)
- 现代浏览器Chromium/Chrome
## 快速开始
### 方案一:使用 agent-browser推荐
agent-browser 是基于 Rust 的高性能浏览器自动化工具。
```bash
# 1. 安装 Python 依赖
pip install --user flask flask-cors gunicorn
# 2. 安装 agent-browser
npm install -g agent-browser
# 3. 安装浏览器依赖(需要 sudo
sudo apt-get install -y \
libxcb-shm0 libx11-xcb1 libx11-6 libxcb1 libxext6 \
libxrandr2 libxcomposite1 libxcursor1 libxdamage1 \
libxfixes3 libxi6 libgtk-3-0 libpangocairo-1.0-0 \
libpango-1.0-0 libatk1.0-0 libcairo-gobject2 libcairo2 \
libgdk-pixbuf-2.0-0 libxrender1 libasound2 libfreetype6 \
libfontconfig1 libdbus-1-3 libnss3 libnspr4 libatk-bridge2.0-0 \
libdrm2 libxkbcommon0 libatspi2.0-0 libcups2 libxshmfence1 \
libgbm1 fonts-noto-color-emoji fonts-noto-cjk fonts-freefont-ttf
# 4. 安装浏览器
agent-browser install
# 5. 启动服务
python3.12 app.py
```
### 方案二:使用 Playwright
Playwright 是微软开发的浏览器自动化库。
```bash
# 1. 安装依赖
pip install --user playwright flask flask-cors gunicorn
# 2. 安装浏览器
python3.12 -m playwright install chromium
# 3. 启动服务
python3.12 app.py
```
## API 接口
### POST /api/capture
**请求参数:**
```json
{
"url": "https://example.com", // 必填:目标网址
"action": "screenshot", // 必填screenshot 或 html
"scroll_times": 3, // 可选滚动次数默认0
"scroll_delay": 1000, // 可选:滚动间隔(ms)默认1000
"full_page": true, // 可选全页截图默认false
"viewport": { // 可选:视口大小
"width": 1920,
"height": 1080
},
"wait_time": 2000, // 可选:页面加载等待(ms)默认2000
"backend": "auto" // 可选auto/agent-browser/playwright
}
```
**响应:**
- `screenshot` 模式:返回 PNG 图片
- `html` 模式:返回 JSON `{"success": true, "html": "..."}`
### GET /health
健康检查接口
### GET /api
API 信息接口
## 使用示例
### 命令行调用
```bash
# 基础截图
curl -X POST http://localhost:5000/api/capture \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "action": "screenshot"}' \
--output screenshot.png
# 提取HTML
curl -X POST http://localhost:5000/api/capture \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com", "action": "html"}'
# 滚动加载后全页截图
curl -X POST http://localhost:5000/api/capture \
-H "Content-Type: application/json" \
-d '{
"url": "https://news.ycombinator.com",
"action": "screenshot",
"scroll_times": 5,
"scroll_delay": 1500,
"full_page": true
}' \
--output full_page.png
# 自定义视口
curl -X POST http://localhost:5000/api/capture \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"action": "screenshot",
"viewport": {"width": 375, "height": 812}
}' \
--output mobile.png
```
### Python 调用
```python
import requests
# 截图
response = requests.post(
'http://localhost:5000/api/capture',
json={
'url': 'https://example.com',
'action': 'screenshot'
}
)
with open('screenshot.png', 'wb') as f:
f.write(response.content)
# 提取HTML
response = requests.post(
'http://localhost:5000/api/capture',
json={
'url': 'https://example.com',
'action': 'html'
}
)
html = response.json()['html']
print(html)
```
### JavaScript 调用
```javascript
// 截图
const response = await fetch('http://localhost:5000/api/capture', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
url: 'https://example.com',
action: 'screenshot',
scroll_times: 3,
full_page: true
})
});
const blob = await response.blob();
// 提取HTML
const response = await fetch('http://localhost:5000/api/capture', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({
url: 'https://example.com',
action: 'html'
})
});
const data = await response.json();
console.log(data.html);
```
## 生产部署
### 使用 Gunicorn
```bash
gunicorn -w 4 -b 0.0.0.0:5000 app:app
```
### 使用 Systemd 服务
```bash
# 复制服务文件
sudo cp web-capture-api.service /etc/systemd/system/
# 启动服务
sudo systemctl start web-capture-api
sudo systemctl enable web-capture-api
```
### 使用 Docker
```dockerfile
FROM python:3.12-slim
# 安装依赖
RUN apt-get update && apt-get install -y \
wget gnupg curl \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
RUN playwright install chromium --with-deps
COPY . .
EXPOSE 5000
CMD ["gunicorn", "-w", "4", "-b", "0.0.0.0:5000", "app:app"]
```
```bash
docker build -t web-capture-api .
docker run -p 5000:5000 web-capture-api
```
## 性能优化
- 使用 `--headless` 模式减少资源消耗
- 合理设置 `wait_time` 避免不必要的等待
- 滚动次数适度,避免过长处理时间
- 生产环境使用 Gunicorn 多进程
## 故障排除
### 浏览器启动失败
检查是否安装了所有依赖:
```bash
# agent-browser
agent-browser install --with-deps
# playwright
python3.12 -m playwright install-deps chromium
```
### 权限错误
确保用户有权限访问浏览器缓存目录:
```bash
mkdir -p ~/.cache/ms-playwright
chmod -R 755 ~/.cache/ms-playwright
```
### 内存不足
减少并发进程数或增加系统内存。
## 许可证
MIT License

401
app.py Normal file
View File

@@ -0,0 +1,401 @@
#!/home/hz1/miniconda3/envs/openclaw/bin/python3.12
"""
Web Capture API - 网页截图与代码提取服务
使用 agent-browser (Rust + Playwright) 或 Playwright
"""
import os
import json
import asyncio
import subprocess
import tempfile
import uuid
import time
from pathlib import Path
from flask import Flask, request, jsonify, send_file, render_template
from flask_cors import CORS
from datetime import datetime
# 检查 Playwright 是否可用
try:
from playwright.async_api import async_playwright
PLAYWRIGHT_AVAILABLE = True
except ImportError:
PLAYWRIGHT_AVAILABLE = False
# 检查 agent-browser 是否可用
def check_agent_browser():
try:
result = subprocess.run(
["which", "agent-browser"],
capture_output=True,
text=True
)
return result.returncode == 0
except:
return False
AGENT_BROWSER_AVAILABLE = check_agent_browser()
app = Flask(__name__)
CORS(app)
# 配置
CAPTURE_DIR = Path(tempfile.gettempdir()) / "web_captures"
CAPTURE_DIR.mkdir(exist_ok=True)
class AgentBrowserSession:
"""agent-browser 会话管理器"""
def __init__(self, session_id=None):
self.session_id = session_id or str(uuid.uuid4())[:8]
self.base_cmd = ["agent-browser", "--session", self.session_id]
def run(self, cmd, timeout=60000):
"""执行 agent-browser 命令"""
full_cmd = self.base_cmd + cmd
try:
result = subprocess.run(
full_cmd,
capture_output=True,
text=True,
timeout=timeout / 1000
)
return result.returncode == 0, result.stdout, result.stderr
except subprocess.TimeoutExpired:
return False, "", "Command timeout"
except Exception as e:
return False, "", str(e)
def open(self, url):
"""打开网页"""
return self.run(["open", url], timeout=30000)
def set_viewport(self, width, height):
"""设置视口大小"""
return self.run(["set", "viewport", str(width), str(height)])
def screenshot(self, output_path, full_page=False):
"""截图"""
cmd = ["screenshot", str(output_path)]
if full_page:
cmd.append("--full")
return self.run(cmd, timeout=30000)
def get_html(self):
"""获取HTML"""
success, output, error = self.run(
["eval", "document.documentElement.outerHTML"],
timeout=10000
)
return success, output, error
def scroll_down(self, pixels=800):
"""向下滚动"""
return self.run(["scroll", "down", str(pixels)])
def wait(self, ms):
"""等待"""
return self.run(["wait", str(ms)])
def close(self):
"""关闭浏览器"""
try:
self.run(["close"])
except:
pass
def capture_with_agent_browser(
url: str,
action: str = "screenshot",
scroll_times: int = 0,
scroll_delay: int = 1000,
full_page: bool = False,
viewport: dict = None,
wait_time: int = 2000
):
"""
使用 agent-browser 捕获网页
"""
session = AgentBrowserSession()
temp_file = None
try:
# 设置视口
if viewport:
session.set_viewport(viewport.get("width", 1920), viewport.get("height", 1080))
# 打开网页
success, stdout, stderr = session.open(url)
if not success:
return {"success": False, "error": f"Failed to open URL: {stderr}"}
# 等待页面加载
session.wait(wait_time)
# 滚动加载动态内容
if scroll_times > 0:
for i in range(scroll_times):
session.scroll_down(800)
session.wait(scroll_delay)
# 执行操作
if action == "screenshot":
temp_file = CAPTURE_DIR / f"{session.session_id}.png"
success, _, error = session.screenshot(temp_file, full_page=full_page)
if success and temp_file.exists():
return {"success": True, "file_path": str(temp_file)}
else:
return {"success": False, "error": f"Screenshot failed: {error}"}
elif action == "html":
success, html, error = session.get_html()
if success:
return {"success": True, "html": html}
else:
return {"success": False, "error": f"Get HTML failed: {error}"}
else:
return {"success": False, "error": f"Unknown action: {action}"}
except Exception as e:
return {"success": False, "error": str(e)}
finally:
session.close()
async def capture_with_playwright(
url: str,
action: str = "screenshot",
scroll_times: int = 0,
scroll_delay: int = 1000,
full_page: bool = False,
viewport: dict = None,
wait_time: int = 2000
):
"""
使用 Playwright 捕获网页
"""
if not PLAYWRIGHT_AVAILABLE:
return {"success": False, "error": "Playwright not installed"}
session_id = str(uuid.uuid4())[:8]
try:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
viewport_settings = viewport or {"width": 1920, "height": 1080}
context = await browser.new_context(
viewport={
"width": viewport_settings.get("width", 1920),
"height": viewport_settings.get("height", 1080)
},
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
)
page = await context.new_page()
try:
await page.goto(url, wait_until="networkidle", timeout=30000)
except:
await page.goto(url, wait_until="domcontentloaded", timeout=30000)
if wait_time > 0:
await page.wait_for_timeout(wait_time)
if scroll_times > 0:
for i in range(scroll_times):
await page.evaluate("window.scrollBy(0, 800)")
await page.wait_for_timeout(scroll_delay)
await page.wait_for_timeout(500)
result = {}
if action == "screenshot":
temp_file = CAPTURE_DIR / f"{session_id}.png"
await page.screenshot(path=str(temp_file), full_page=full_page)
result = {"success": True, "file_path": str(temp_file)}
elif action == "html":
html = await page.content()
result = {"success": True, "html": html}
else:
result = {"success": False, "error": f"Unknown action: {action}"}
await browser.close()
return result
except Exception as e:
return {"success": False, "error": str(e)}
def capture_webpage(
url: str,
action: str = "screenshot",
scroll_times: int = 0,
scroll_delay: int = 1000,
full_page: bool = False,
viewport: dict = None,
wait_time: int = 2000,
backend: str = "auto"
):
"""
捕获网页
Args:
backend: "auto", "agent-browser", "playwright"
"""
# 选择后端
if backend == "auto":
if AGENT_BROWSER_AVAILABLE:
backend = "agent-browser"
elif PLAYWRIGHT_AVAILABLE:
backend = "playwright"
else:
return {"success": False, "error": "No browser backend available. Install agent-browser or playwright."}
# 使用对应后端
if backend == "agent-browser":
if not AGENT_BROWSER_AVAILABLE:
return {"success": False, "error": "agent-browser not available"}
return capture_with_agent_browser(
url, action, scroll_times, scroll_delay, full_page, viewport, wait_time
)
elif backend == "playwright":
if not PLAYWRIGHT_AVAILABLE:
return {"success": False, "error": "Playwright not available"}
try:
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
result = loop.run_until_complete(
capture_with_playwright(
url, action, scroll_times, scroll_delay, full_page, viewport, wait_time
)
)
loop.close()
return result
except Exception as e:
return {"success": False, "error": str(e)}
else:
return {"success": False, "error": f"Unknown backend: {backend}"}
@app.route('/')
def index():
"""主页"""
return render_template('index.html')
@app.route('/api')
def api_info():
"""API信息"""
return jsonify({
"service": "Web Capture API",
"version": "2.0.0",
"backends": {
"agent-browser": "available" if AGENT_BROWSER_AVAILABLE else "unavailable",
"playwright": "available" if PLAYWRIGHT_AVAILABLE else "unavailable"
},
"endpoints": {
"/api/capture": "POST - Capture webpage screenshot or HTML",
"/health": "GET - Health check"
}
})
@app.route('/health')
def health():
"""健康检查"""
status = "healthy" if (AGENT_BROWSER_AVAILABLE or PLAYWRIGHT_AVAILABLE) else "degraded"
return jsonify({
"status": status,
"agent-browser": "installed" if AGENT_BROWSER_AVAILABLE else "not found",
"playwright": "installed" if PLAYWRIGHT_AVAILABLE else "not found",
"timestamp": datetime.now().isoformat()
})
@app.route('/api/capture', methods=['POST'])
def capture():
"""
捕获网页接口
请求体:
{
"url": "https://example.com",
"action": "screenshot" | "html",
"scroll_times": 0,
"scroll_delay": 1000,
"full_page": false,
"viewport": {"width": 1920, "height": 1080},
"wait_time": 2000,
"backend": "auto" | "agent-browser" | "playwright"
}
"""
data = request.get_json()
if not data:
return jsonify({"success": False, "error": "No JSON data provided"}), 400
url = data.get("url")
if not url:
return jsonify({"success": False, "error": "URL is required"}), 400
# 添加协议前缀
if not url.startswith(("http://", "https://")):
url = "https://" + url
action = data.get("action", "screenshot")
scroll_times = int(data.get("scroll_times", 0))
scroll_delay = int(data.get("scroll_delay", 1000))
full_page = bool(data.get("full_page", False))
viewport = data.get("viewport")
wait_time = int(data.get("wait_time", 2000))
backend = data.get("backend", "auto")
result = capture_webpage(
url=url,
action=action,
scroll_times=scroll_times,
scroll_delay=scroll_delay,
full_page=full_page,
viewport=viewport,
wait_time=wait_time,
backend=backend
)
if not result["success"]:
return jsonify(result), 400
if action == "screenshot":
file_path = result["file_path"]
return send_file(file_path, mimetype='image/png')
elif action == "html":
return jsonify(result)
if __name__ == '__main__':
print("🌐 Web Capture API Starting...")
print(f"📁 Capture directory: {CAPTURE_DIR}")
print(f"🔧 Backends:")
print(f" - agent-browser: {'' if AGENT_BROWSER_AVAILABLE else ''}")
print(f" - playwright: {'' if PLAYWRIGHT_AVAILABLE else ''}")
if not AGENT_BROWSER_AVAILABLE and not PLAYWRIGHT_AVAILABLE:
print("\n⚠️ 没有可用的浏览器后端,请安装其中一个:")
print(" agent-browser: npm install -g agent-browser && agent-browser install")
print(" playwright: pip install playwright && playwright install chromium")
print("\n🚀 Server running on http://0.0.0.0:5000")
app.run(host='0.0.0.0', port=5000, debug=True)

26
deploy.sh Executable file
View File

@@ -0,0 +1,26 @@
#!/bin/bash
# 部署脚本
set -e
echo "🚀 开始部署 Web Capture API..."
# 进入项目目录
cd /home/openclaw/.openclaw/workspace-hz4th_coder/works/web-capture-api
# 安装 Python 依赖
echo "📥 安装 Python 依赖..."
pip3.12 install --user playwright flask flask-cors gunicorn
# 安装 Playwright 浏览器
echo "🌐 安装 Playwright 浏览器..."
python3.12 -m playwright install chromium
echo ""
echo "✅ 部署完成!"
echo ""
echo "运行方式:"
echo " 开发模式: python3.12 app.py"
echo " 生产模式: gunicorn -w 4 -b 0.0.0.0:5000 app:app"
echo ""
echo "访问地址: http://localhost:5000"

20
docker-compose.yml Normal file
View File

@@ -0,0 +1,20 @@
version: '3.8'
services:
web-capture-api:
build: .
image: web-capture-api:latest
container_name: web-capture-api
ports:
- "5000:5000"
environment:
- TZ=Asia/Shanghai
volumes:
- ./captures:/tmp/web_captures
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
interval: 30s
timeout: 10s
retries: 3
start_period: 10s

116
install.sh Executable file
View File

@@ -0,0 +1,116 @@
#!/bin/bash
# 一键安装脚本
set -e
echo "🚀 Web Capture API 安装脚本"
echo "================================"
echo ""
# 检测系统
if [[ "$OSTYPE" == "linux-gnu"* ]]; then
OS="linux"
elif [[ "$OSTYPE" == "darwin"* ]]; then
OS="macos"
else
echo "❌ 不支持的操作系统: $OSTYPE"
exit 1
fi
echo "📋 系统检测: $OS"
# 检查 Python
echo ""
echo "📦 检查 Python..."
if command -v python3.12 &> /dev/null; then
PYTHON=python3.12
elif command -v python3 &> /dev/null; then
PYTHON=python3
else
echo "❌ 未找到 Python 3请先安装 Python 3.10+"
exit 1
fi
echo "✅ 找到 Python: $($PYTHON --version)"
# 检查 pip
if ! $PYTHON -m pip --version &> /dev/null; then
echo "❌ 未找到 pip请先安装 pip"
exit 1
fi
# 检查 Node.js (用于 agent-browser)
echo ""
echo "📦 检查 Node.js..."
if command -v node &> /dev/null; then
NODE_VERSION=$(node --version)
echo "✅ 找到 Node.js: $NODE_VERSION"
HAS_NODE=true
else
echo "⚠️ 未找到 Node.js将跳过 agent-browser 安装"
HAS_NODE=false
fi
# 安装 Python 依赖
echo ""
echo "📥 安装 Python 依赖..."
$PYTHON -m pip install --user playwright flask flask-cors gunicorn
# 询问使用哪个后端
echo ""
echo "🔧 选择浏览器后端:"
echo " 1) Playwright (推荐,自动安装浏览器)"
echo " 2) agent-browser (高性能,需要手动安装浏览器依赖)"
read -p "请选择 [1/2]: " choice
case $choice in
1)
echo ""
echo "🌐 安装 Playwright 浏览器..."
$PYTHON -m playwright install chromium
if [ "$OS" == "linux" ]; then
echo ""
echo "⚠️ 如果浏览器启动失败,可能需要安装系统依赖:"
echo " $PYTHON -m playwright install-deps chromium"
fi
;;
2)
if [ "$HAS_NODE" = false ]; then
echo "❌ 安装 agent-browser 需要 Node.js"
exit 1
fi
echo ""
echo "🌐 安装 agent-browser..."
npm install -g agent-browser
if [ "$OS" == "linux" ]; then
echo ""
echo "⚠️ agent-browser 需要浏览器依赖,请运行:"
echo " agent-browser install --with-deps"
echo ""
echo "如果需要 sudo 权限,请运行:"
echo " sudo agent-browser install --with-deps"
else
agent-browser install
fi
;;
*)
echo "❌ 无效选择"
exit 1
;;
esac
# 完成
echo ""
echo "✅ 安装完成!"
echo ""
echo "🚀 启动服务:"
echo " cd $(pwd)"
echo " $PYTHON app.py"
echo ""
echo "📖 访问地址:"
echo " http://localhost:5000"
echo ""
echo "📚 API 文档:"
echo " http://localhost:5000/api"

4
requirements.txt Normal file
View File

@@ -0,0 +1,4 @@
Flask==3.0.0
flask-cors==6.0.5
gunicorn==26.0.0
playwright==1.48.0

480
templates/index.html Normal file
View File

@@ -0,0 +1,480 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Web Capture - 网页截图与代码提取</title>
<style>
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, Oxygen, Ubuntu, Cantarell, sans-serif;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
min-height: 100vh;
padding: 20px;
}
.container {
max-width: 1200px;
margin: 0 auto;
}
.header {
text-align: center;
color: white;
margin-bottom: 30px;
}
.header h1 {
font-size: 2.5em;
margin-bottom: 10px;
}
.header p {
opacity: 0.9;
}
.card {
background: white;
border-radius: 16px;
padding: 30px;
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
margin-bottom: 20px;
}
.form-group {
margin-bottom: 20px;
}
label {
display: block;
font-weight: 600;
margin-bottom: 8px;
color: #333;
}
input[type="text"],
input[type="number"],
select {
width: 100%;
padding: 12px;
border: 2px solid #e0e0e0;
border-radius: 8px;
font-size: 16px;
transition: border-color 0.3s;
}
input:focus,
select:focus {
outline: none;
border-color: #667eea;
}
.radio-group {
display: flex;
gap: 20px;
}
.radio-label {
display: flex;
align-items: center;
gap: 8px;
cursor: pointer;
}
.radio-label input[type="radio"] {
width: 20px;
height: 20px;
}
.checkbox-group {
display: flex;
gap: 20px;
flex-wrap: wrap;
}
.checkbox-label {
display: flex;
align-items: center;
gap: 8px;
cursor: pointer;
}
.checkbox-label input[type="checkbox"] {
width: 20px;
height: 20px;
}
.row {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(150px, 1fr));
gap: 15px;
}
.btn {
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
border: none;
padding: 15px 40px;
font-size: 18px;
font-weight: 600;
border-radius: 8px;
cursor: pointer;
transition: transform 0.2s, box-shadow 0.2s;
width: 100%;
margin-top: 10px;
}
.btn:hover {
transform: translateY(-2px);
box-shadow: 0 10px 30px rgba(102, 126, 234, 0.4);
}
.btn:active {
transform: translateY(0);
}
.btn:disabled {
opacity: 0.6;
cursor: not-allowed;
transform: none;
}
.loading {
display: none;
text-align: center;
padding: 20px;
}
.loading.active {
display: block;
}
.spinner {
border: 4px solid #f3f3f3;
border-top: 4px solid #667eea;
border-radius: 50%;
width: 40px;
height: 40px;
animation: spin 1s linear infinite;
margin: 0 auto 15px;
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
.result {
display: none;
}
.result.active {
display: block;
}
.result-image {
width: 100%;
border-radius: 8px;
box-shadow: 0 4px 15px rgba(0, 0, 0, 0.1);
}
.result-code {
background: #1e1e1e;
color: #d4d4d4;
padding: 20px;
border-radius: 8px;
overflow-x: auto;
max-height: 600px;
font-family: 'Courier New', monospace;
font-size: 14px;
line-height: 1.6;
}
.result-code pre {
margin: 0;
white-space: pre-wrap;
word-wrap: break-word;
}
.actions {
margin-top: 20px;
display: flex;
gap: 10px;
}
.btn-secondary {
background: #6c757d;
padding: 10px 20px;
font-size: 14px;
}
.error {
background: #fee;
border: 1px solid #fcc;
color: #c00;
padding: 15px;
border-radius: 8px;
margin-top: 20px;
display: none;
}
.error.active {
display: block;
}
.api-docs {
margin-top: 30px;
padding: 20px;
background: #f8f9fa;
border-radius: 8px;
}
.api-docs h3 {
margin-bottom: 15px;
color: #333;
}
.api-docs code {
background: #e9ecef;
padding: 2px 6px;
border-radius: 4px;
font-size: 14px;
}
.api-docs pre {
background: #1e1e1e;
color: #d4d4d4;
padding: 15px;
border-radius: 8px;
overflow-x: auto;
margin-top: 10px;
}
</style>
</head>
<body>
<div class="container">
<div class="header">
<h1>🌐 Web Capture</h1>
<p>网页截图与代码提取服务</p>
</div>
<div class="card">
<form id="captureForm">
<div class="form-group">
<label for="url">目标网址</label>
<input type="text" id="url" placeholder="https://example.com" required>
</div>
<div class="form-group">
<label>操作类型</label>
<div class="radio-group">
<label class="radio-label">
<input type="radio" name="action" value="screenshot" checked>
📸 截图
</label>
<label class="radio-label">
<input type="radio" name="action" value="html">
📄 提取HTML
</label>
</div>
</div>
<div class="form-group">
<label>高级选项</label>
<div class="checkbox-group">
<label class="checkbox-label">
<input type="checkbox" id="fullPage">
全页截图
</label>
</div>
</div>
<div class="form-group">
<label>滚动设置(用于加载动态内容)</label>
<div class="row">
<div>
<label for="scrollTimes">滚动次数</label>
<input type="number" id="scrollTimes" value="0" min="0" max="20">
</div>
<div>
<label for="scrollDelay">滚动间隔(ms)</label>
<input type="number" id="scrollDelay" value="1000" min="500" max="5000" step="100">
</div>
<div>
<label for="waitTime">页面加载等待(ms)</label>
<input type="number" id="waitTime" value="2000" min="0" max="10000" step="500">
</div>
</div>
</div>
<div class="form-group">
<label>视口大小</label>
<div class="row">
<div>
<label for="viewportWidth">宽度</label>
<input type="number" id="viewportWidth" value="1920" min="320" max="3840">
</div>
<div>
<label for="viewportHeight">高度</label>
<input type="number" id="viewportHeight" value="1080" min="240" max="2160">
</div>
</div>
</div>
<button type="submit" class="btn" id="submitBtn">
🚀 开始捕获
</button>
</form>
<div class="loading" id="loading">
<div class="spinner"></div>
<p>正在处理,请稍候...</p>
</div>
<div class="error" id="error"></div>
<div class="result" id="result">
<h3 style="margin-bottom: 15px;">捕获结果</h3>
<div id="resultContent"></div>
<div class="actions">
<button class="btn btn-secondary" onclick="downloadResult()">💾 下载</button>
<button class="btn btn-secondary" onclick="copyResult()">📋 复制</button>
<button class="btn btn-secondary" onclick="resetForm()">🔄 重新捕获</button>
</div>
</div>
</div>
<div class="card api-docs">
<h3>📚 API 文档</h3>
<p>POST /api/capture</p>
<pre><code>{
"url": "https://example.com",
"action": "screenshot", // 或 "html"
"scroll_times": 3,
"scroll_delay": 1000,
"full_page": true,
"viewport": {"width": 1920, "height": 1080},
"wait_time": 2000
}</code></pre>
</div>
</div>
<script>
const form = document.getElementById('captureForm');
const loading = document.getElementById('loading');
const result = document.getElementById('result');
const error = document.getElementById('error');
const submitBtn = document.getElementById('submitBtn');
let currentData = null;
let currentBlob = null;
form.addEventListener('submit', async (e) => {
e.preventDefault();
const url = document.getElementById('url').value;
const action = document.querySelector('input[name="action"]:checked').value;
const fullPage = document.getElementById('fullPage').checked;
const scrollTimes = parseInt(document.getElementById('scrollTimes').value);
const scrollDelay = parseInt(document.getElementById('scrollDelay').value);
const waitTime = parseInt(document.getElementById('waitTime').value);
const viewportWidth = parseInt(document.getElementById('viewportWidth').value);
const viewportHeight = parseInt(document.getElementById('viewportHeight').value);
currentData = {
url,
action,
scroll_times: scrollTimes,
scroll_delay: scrollDelay,
full_page: fullPage,
wait_time: waitTime,
viewport: {
width: viewportWidth,
height: viewportHeight
}
};
// 显示加载
loading.classList.add('active');
result.classList.remove('active');
error.classList.remove('active');
submitBtn.disabled = true;
try {
const response = await fetch('/api/capture', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify(currentData)
});
if (!response.ok) {
const errData = await response.json();
throw new Error(errData.error || '请求失败');
}
if (action === 'screenshot') {
currentBlob = await response.blob();
const imageUrl = URL.createObjectURL(currentBlob);
document.getElementById('resultContent').innerHTML =
`<img src="${imageUrl}" class="result-image" alt="截图结果">`;
} else {
const data = await response.json();
currentBlob = new Blob([data.html], { type: 'text/html' });
document.getElementById('resultContent').innerHTML =
`<div class="result-code"><pre>${escapeHtml(data.html)}</pre></div>`;
}
result.classList.add('active');
} catch (err) {
error.textContent = '❌ ' + err.message;
error.classList.add('active');
} finally {
loading.classList.remove('active');
submitBtn.disabled = false;
}
});
function escapeHtml(text) {
const div = document.createElement('div');
div.textContent = text;
return div.innerHTML;
}
function downloadResult() {
if (!currentBlob) return;
const url = URL.createObjectURL(currentBlob);
const a = document.createElement('a');
a.href = url;
a.download = currentData.action === 'screenshot' ? 'screenshot.png' : 'page.html';
a.click();
URL.revokeObjectURL(url);
}
function copyResult() {
if (!currentBlob) return;
const reader = new FileReader();
reader.onload = () => {
navigator.clipboard.writeText(reader.result).then(() => {
alert('已复制到剪贴板');
});
};
reader.readAsText(currentBlob);
}
function resetForm() {
result.classList.remove('active');
error.classList.remove('active');
currentData = null;
currentBlob = null;
}
</script>
</body>
</html>

122
test_api.py Executable file
View File

@@ -0,0 +1,122 @@
#!/usr/bin/env python3
"""
测试脚本 - 测试 Web Capture API
"""
import requests
import json
import sys
API_URL = "http://localhost:5000"
def test_health():
"""测试健康检查"""
print("\n🔍 测试健康检查...")
resp = requests.get(f"{API_URL}/health")
print(f"状态: {resp.status_code}")
print(f"响应: {json.dumps(resp.json(), indent=2, ensure_ascii=False)}")
return resp.status_code == 200
def test_screenshot():
"""测试截图功能"""
print("\n📸 测试截图功能...")
data = {
"url": "https://example.com",
"action": "screenshot",
"wait_time": 2000
}
resp = requests.post(
f"{API_URL}/api/capture",
json=data
)
if resp.status_code == 200:
with open("test_screenshot.png", "wb") as f:
f.write(resp.content)
print("✅ 截图成功,已保存为 test_screenshot.png")
return True
else:
print(f"❌ 截图失败: {resp.text}")
return False
def test_html():
"""测试HTML提取"""
print("\n📄 测试HTML提取...")
data = {
"url": "https://example.com",
"action": "html"
}
resp = requests.post(
f"{API_URL}/api/capture",
json=data
)
if resp.status_code == 200:
result = resp.json()
print(f"✅ HTML提取成功长度: {len(result.get('html', ''))}")
print(f"前200字符: {result.get('html', '')[:200]}...")
return True
else:
print(f"❌ HTML提取失败: {resp.text}")
return False
def test_scroll():
"""测试滚动加载"""
print("\n🔄 测试滚动加载...")
data = {
"url": "https://news.ycombinator.com",
"action": "screenshot",
"scroll_times": 3,
"scroll_delay": 1000,
"full_page": True
}
resp = requests.post(
f"{API_URL}/api/capture",
json=data
)
if resp.status_code == 200:
with open("test_scroll.png", "wb") as f:
f.write(resp.content)
print("✅ 滚动截图成功,已保存为 test_scroll.png")
return True
else:
print(f"❌ 滚动截图失败: {resp.text}")
return False
if __name__ == "__main__":
print("🚀 Web Capture API 测试")
print("=" * 50)
tests = [
("健康检查", test_health),
("截图功能", test_screenshot),
("HTML提取", test_html),
("滚动加载", test_scroll)
]
results = []
for name, test_func in tests:
try:
result = test_func()
results.append((name, result))
except Exception as e:
print(f"{name} 异常: {e}")
results.append((name, False))
print("\n" + "=" * 50)
print("测试结果汇总:")
for name, result in results:
status = "✅ 通过" if result else "❌ 失败"
print(f" {name}: {status}")

14
web-capture-api.service Normal file
View File

@@ -0,0 +1,14 @@
[Unit]
Description=Web Capture API - 网页截图与代码提取服务
After=network.target
[Service]
Type=simple
User=openclaw
WorkingDirectory=/home/openclaw/.openclaw/workspace-hz4th_coder/works/web-capture-api
ExecStart=/home/openclaw/.openclaw/workspace-hz4th_coder/works/web-capture-api/venv/bin/gunicorn -w 4 -b 0.0.0.0:5000 app:app
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target