Compare commits

...

47 Commits

Author SHA1 Message Date
dmy
2475619228 feat: 添加爬虫状态更新中的显示支持
在爬虫状态列中增加"正在更新"状态显示,并在爬虫过程中更新状态
2026-01-22 14:15:07 +08:00
dmy
eaed16a12e feat: 在AI推荐面板中显示推荐生成时间 2026-01-22 12:51:34 +08:00
dmy
4bace565e4 feat: 为项目标题添加可点击链接功能 2026-01-20 23:47:51 +08:00
dmy
bdc62a2975 feat: 优化移动端顶部导航栏样式和布局 2026-01-19 17:05:04 +08:00
dmy
e3880408b7 style: 优化移动端表格样式和响应式布局 2026-01-19 17:01:16 +08:00
dmy
1453e03c76 style: 调整表格列宽 2026-01-19 14:10:37 +08:00
dmy
882c8b5b9f fix: 移动端隐藏标题和logo 2026-01-19 13:50:53 +08:00
dmy
95dfcd0278 feat: 添加移动端响应式布局和样式优化 2026-01-19 13:47:41 +08:00
dmy
fffc17b9ad feat: 提示语中增加不修改工程名称的要求 2026-01-19 12:54:57 +08:00
dmy
91e44018f0 chore: 在.gitignore中添加.idea目录 2026-01-18 20:48:43 +08:00
dmy
cf5a0b179e feat: 添加单个数据源手动更新功能 2026-01-18 20:48:15 +08:00
dmy
9fc455cca4 feat: 添加API连接状态显示页脚
在应用底部添加固定页脚显示当前连接的API地址
2026-01-18 19:08:04 +08:00
dmy
8f6e5c8423 feat: 添加投标项目查看器的Ionic移动端应用
新增基于Ionic + Vue 3 + TypeScript + Tailwind CSS的移动端应用,适配Android平台。主要功能包括AI推荐项目展示、爬虫信息统计、置顶项目管理和下拉刷新功能。同时更新前后端API端口配置为3300。
2026-01-18 18:53:35 +08:00
dmy
f08c513bbe feat: 添加环境变量配置并优化部署路径
添加开发和生产环境变量文件
更新前端构建命令以使用不同模式
优化部署脚本中的前端部署路径
在认证守卫中添加日志记录
2026-01-18 15:44:37 +08:00
dmy
b6a6398864 feat: 添加用户认证系统
引入基于 Basic Auth 的用户认证系统,包括用户管理、登录界面和 API 鉴权
- 新增用户实体和管理功能
- 实现前端登录界面和凭证管理
- 重构 API 鉴权为 Basic Auth 模式
- 添加用户管理脚本工具
2026-01-18 12:47:16 +08:00
dmy
a55dfd78d2 feat: 更新部署配置和API类型定义
将部署配置更改为生产环境服务器设置
将AxiosRequestConfig替换为InternalAxiosRequestConfig
2026-01-18 11:17:12 +08:00
dmy
810a420a46 feat(date): enhance date formatting and timezone handling
- Update formatDateTime function to handle timezone-aware date strings directly.
- Introduce utcToBeijingISOString function for consistent Beijing time formatting in ISO string.
- Modify BidsService to utilize the new utcToBeijingISOString for date conversions.
- Add unit tests for AuthGuard to validate API key authentication and access control.
- Create end-to-end tests for API key handling in various scenarios.
- Update .gitignore to exclude 'qingyun' directory.
- Add environment configuration for production settings in .env.production.
- Include Tailwind CSS setup in the uni-app version for styling.
2026-01-16 23:31:58 +08:00
dmy
300e930c64 feat(auth): implement API key authentication and enhance request handling
- Add AuthGuard to validate API key for public access.
- Create AuthModule to provide the AuthGuard globally.
- Update API request interceptor to automatically include API key for non-localhost requests.
- Modify .env and .env.example to include API_KEY configuration.
- Enhance API request handling with improved error logging and client IP detection.
2026-01-16 11:26:02 +08:00
dmy
9257c78e72 feat(timezone): implement UTC and Beijing time conversion utilities and update bid handling
- Add functions to convert between UTC and Beijing time in timezone utility.
- Update BidsService to return latest update and publish dates in Beijing time.
- Modify BidCrawlerService to store publish dates in UTC format.
- Change database timezone configuration to UTC for consistency.
2026-01-16 00:00:00 +08:00
dmy
e8beeec2b9 chore(env): update log level to debug and modify .gitignore
- Change log level from 'info' to 'debug' for enhanced logging during development.
- Add '.cursor' to .gitignore to prevent tracking of cursor files.
2026-01-15 23:48:07 +08:00
dmy
9dc01eeb46 fix(api): increase request timeout and format latest update date
- Update API request timeout from 60 seconds to 120 seconds for improved performance.
- Format latest update date to append 'Z' if it exists, ensuring consistency in date handling.
2026-01-15 23:31:05 +08:00
dmy
811ad927f3 feat(logging): add logging middleware for HTTP requests and update API timeout
- Introduce LoggingMiddleware to log incoming requests and their response times
- Update AppModule to apply the new logging middleware globally
- Increase API request timeout from 10 seconds to 60 seconds
- Modify PromptString to include additional project types of interest
2026-01-15 21:52:40 +08:00
dmy
3033eb622f feat(deploy): implement new deployment script using ssh2-sftp-client
- Replace PowerShell deploy script with TypeScript-based deploy script
- Add SSH2 SFTP client for secure file uploads to remote server
- Include error handling and logging for deployment process
- Update package.json to reflect changes in deploy command and add new dependencies
2026-01-15 21:02:36 +08:00
dmy
5edebd9d55 refactor: improve date handling and timezone consistency
- Add timezone support to database module (+08:00)
- Extract date formatting utilities to shared modules
- Standardize timezone handling across frontend and backend
- Improve date formatting consistency in UI components
- Refactor crawler page.goto options for better readability
2026-01-15 16:17:41 +08:00
dmy
eba5c7e5c5 feat(crawler): add human-like behavior simulation to prevent detection
refactor(manifest): update appid and format manifest file
2026-01-15 15:34:00 +08:00
dmy
36cbb6fda1 feat(api): add unified request utility with type definitions
Implement a centralized API request utility with proper TypeScript interfaces
for bid items, AI recommendations, and crawl statistics. The utility handles
base URL configuration, request/response typing, and error handling.
2026-01-15 14:28:04 +08:00
dmy
20c7c0da0c feat: 添加代理隧道连接失败的重试机制
refactor(crawler): 在各爬虫服务中实现代理错误重试逻辑
feat(uni-app): 新增投标项目查看器的uni-app版本
2026-01-15 14:03:46 +08:00
dmy
37200aa115 fix(electron): 更新开发模式支持和控制台编码设置
- 在主进程中添加对Windows控制台UTF-8编码的支持
- 优化开发模式检测逻辑,确保在开发环境下正确打开开发者工具
- 改进后端进程的启动逻辑,确保在启动失败时提供详细错误信息
2026-01-15 10:14:54 +08:00
dmy
af78fd0682 chore: 更新数据库连接配置 2026-01-15 02:20:04 +08:00
dmy
20619bb87b fix: 移除Windows控制台配置并修改后端进程stdio设置 2026-01-15 02:19:38 +08:00
dmy
70f0498c44 build(electron): 在Windows配置中禁用控制台窗口
修改electron-builder.json配置,将win.console设置为false以隐藏应用启动时的控制台窗口
2026-01-15 01:32:16 +08:00
dmy
0f510554ed fix(electron): 修复生产环境后端路径问题并优化构建配置
调整 electron-builder 配置文件以正确处理打包路径
修复生产环境下后端服务路径问题
添加对开发工具中 Autofill 错误信息的过滤
更新构建脚本确保前端先构建
2026-01-15 01:30:47 +08:00
dmy
5a7cbc6daa feat: 禁用应用菜单以提供更简洁的界面
添加 Menu.setApplicationMenu(null) 来隐藏默认的 Electron 应用菜单,提升用户体验
2026-01-15 00:45:30 +08:00
dmy
e804e3998f refactor: 移除表格列的fixed属性并更新electron启动脚本
移除CrawlInfo组件中操作列的fixed属性以改善布局
将electron:start重命名为electron:dev并更新构建命令
2026-01-15 00:41:56 +08:00
dmy
eca3f4f9fd feat(electron): 添加Electron桌面应用支持
- 新增Electron主进程、预加载脚本和构建配置
- 修改前端配置以支持Electron打包
- 更新项目文档和依赖
- 重构API调用使用统一axios实例
2026-01-15 00:35:19 +08:00
dmy
f736f30248 refactor(crawler): 优化爬虫类型定义和接口导出
将 Crawler 接口替换为 AnyCrawler 类型,统一爬虫类型定义
导出 CrawlInfoAddStats 接口以便其他模块使用
简化爬虫调用方式,使用类型断言替代接口方法调用
2026-01-14 22:51:36 +08:00
dmy
82f5a81887 chore: 更新.gitignore并添加新文件
在.gitignore中添加对*.png、*.log、*-lock.json、*.woff2文件的忽略规则,并新增OFL.txt文件。同时,添加vue.svg图标文件以支持前端展示。更新多个TypeScript文件以优化代码格式和增强可读性。
2026-01-14 22:26:32 +08:00
dmy
10565af001 fix: 处理错误信息字段的空值和空字符串
更新多个文件以确保错误信息字段在处理时将空字符串和未定义值转换为null,增强数据一致性和展示逻辑。包括前端组件和后端服务的相应调整。
2026-01-14 21:33:35 +08:00
dmy
740c11527f feat: 更新数据库配置并添加同步脚本
将数据库配置从本地MariaDB切换为腾讯云MySQL
添加数据库同步脚本命令到package.json
禁用数据库自动同步功能
2026-01-14 20:58:07 +08:00
dmy
3f6d10061d feat: 添加单源爬取功能并优化数据库同步
新增单源爬取功能,支持在界面上单独更新每个数据源
添加数据库同步脚本,支持主从数据库结构同步和数据同步
优化华能集团爬虫的页面导航和稳定性
新增系统托盘功能,支持最小化到托盘
2026-01-14 16:25:01 +08:00
dmy
bcd7af4e69 feat: 添加爬虫状态监控功能
新增爬虫统计信息展示组件,包括后端数据查询接口和前端展示界面。同时简化日期显示格式并添加刷新提示功能。
2026-01-14 09:26:04 +08:00
dmy
571eea0f66 build: 更新.gitignore并添加TailwindCSS和OpenAI依赖 2026-01-14 01:10:48 +08:00
dmy
6a9c52fe10 feat: 添加AI推荐功能及自动刷新机制
新增AI推荐模块,包括后端数据获取接口和前端展示组件
实现自动刷新功能,每5分钟自动更新当前标签页数据
添加手动刷新按钮,优化用户交互体验
2026-01-14 01:06:15 +08:00
dmy
8e4429558c feat: 添加投标项目查看器组件及后端支持 2026-01-14 00:56:30 +08:00
dmy
2fcfb452ec docs: 更新README以反映投标监控系统项目详情 2026-01-13 21:32:59 +08:00
dmy
e410053ddd refactor: 从Dashboard-AI组件中移除未使用的ArrowDown图标 2026-01-13 21:26:52 +08:00
dmy
f32c04b8df refactor: 移除未使用的 ArrowDown 图标导入 2026-01-13 21:26:25 +08:00
189 changed files with 14152 additions and 1358 deletions

52
.env
View File

@@ -1,17 +1,53 @@
DATABASE_TYPE=mariadb
DATABASE_HOST=127.0.0.1
DATABASE_PORT=23306
DATABASE_USERNAME=root
DATABASE_PASSWORD=410491
# DATABASE_TYPE=mariadb
# DATABASE_HOST=127.0.0.1
# DATABASE_PORT=23306
# DATABASE_USERNAME=root
# DATABASE_PASSWORD=410491
# DATABASE_NAME=bidding
# DATABASE_SYNCHRONIZE=true
# DATABASE_TYPE=mysql
# DATABASE_HOST=bj-cynosdbmysql-grp-r3a4c658.sql.tencentcdb.com
# DATABASE_PORT=21741
# DATABASE_USERNAME=root
# DATABASE_PASSWORD=}?cRa1f[,}`J
# DATABASE_NAME=bidding
# DATABASE_SYNCHRONIZE=false
DATABASE_TYPE=mysql
DATABASE_HOST=mysql-35aea0ff-ijustforregister-858d.h.aivencloud.com
DATABASE_PORT=14129
DATABASE_USERNAME=avnadmin
DATABASE_PASSWORD=AVNS_PJLxfsWSKa4_FAq_PBt
DATABASE_NAME=bidding
DATABASE_SYNCHRONIZE=true
DATABASE_SYNCHRONIZE=false
# Slave 数据库配置(用于数据同步)
SLAVE_DATABASE_TYPE=mysql
SLAVE_DATABASE_HOST=bj-cynosdbmysql-grp-r3a4c658.sql.tencentcdb.com
SLAVE_DATABASE_PORT=21741
SLAVE_DATABASE_USERNAME=root
SLAVE_DATABASE_PASSWORD=}?cRa1f[,}`J
SLAVE_DATABASE_NAME=bidding
# 代理配置(可选)
PROXY_HOST=127.0.0.1
PROXY_PORT=3211
# 日志级别可选error, warn, info, debug, verbose
LOG_LEVEL=info
LOG_LEVEL=debug
# OpenAI API Key (用于 AI 推荐)
ARK_API_KEY=a63d58b6-cf56-434b-8a42-5c781ba0822a
ARK_API_KEY=a63d58b6-cf56-434b-8a42-5c781ba0822a
SSH_PASSPHRASE=x
API_KEY=22c64b60-6e60-433c-991d-f6d658024b9e
# 是否启用 Basic Auth 认证true/false
ENABLE_BASIC_AUTH=true
PORT=3300
HOST=0.0.0.0

View File

@@ -6,6 +6,14 @@ DATABASE_PASSWORD=root
DATABASE_NAME=bidding
DATABASE_SYNCHRONIZE=true
# Slave 数据库配置(用于数据同步)
SLAVE_DATABASE_TYPE=mariadb
SLAVE_DATABASE_HOST=localhost
SLAVE_DATABASE_PORT=3306
SLAVE_DATABASE_USERNAME=root
SLAVE_DATABASE_PASSWORD=root
SLAVE_DATABASE_NAME=bidding_slave
# 代理配置(可选)
PROXY_HOST=127.0.0.1
PROXY_PORT=6000
@@ -16,4 +24,9 @@ PROXY_PORT=6000
LOG_LEVEL=info
# OpenAI API Key (用于 AI 推荐)
ARK_API_KEY=your_openai_api_key_here
ARK_API_KEY=your_openai_api_key_here
API_KEY=your_secure_api_key_here
# 是否启用 Basic Auth 认证true/false
ENABLE_BASIC_AUTH=true

19
.gitignore vendored
View File

@@ -4,4 +4,21 @@ dist
public
*.xls*
pw-browsers
logs
logs
build
*.exe
*.png
*.log
*-lock.json
*.woff2
widget/looker/frontend/src/assets/fonts/OFL.txt
dist-electron
unpackage
.cursor
qingyun
plan
.trae
plans
android
docs
.idea

451
README.md
View File

@@ -1,106 +1,129 @@
<p align="center">
<a href="http://nestjs.com/" target="blank"><img src="https://nestjs.com/img/logo-small.svg" width="120" alt="Nest Logo" /></a>
</p>
# 投标信息智能监控系统
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
[circleci-url]: https://circleci.com/gh/nestjs/nest
一个基于 TypeScript 的 Web 应用,用于自动爬取商务投标平台的最新信息,将符合条件的投标项目突出显示,为用户提供精准的投标信息监控服务。
<p align="center">A progressive <a href="http://nodejs.org" target="_blank">Node.js</a> framework for building efficient and scalable server-side applications.</p>
<p align="center">
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/v/@nestjs/core.svg" alt="NPM Version" /></a>
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/l/@nestjs/core.svg" alt="Package License" /></a>
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/dm/@nestjs/common.svg" alt="NPM Downloads" /></a>
<a href="https://circleci.com/gh/nestjs/nest" target="_blank"><img src="https://img.shields.io/circleci/build/github/nestjs/nest/master" alt="CircleCI" /></a>
<a href="https://discord.gg/G7Qnnhy" target="_blank"><img src="https://img.shields.io/badge/discord-online-brightgreen.svg" alt="Discord"/></a>
<a href="https://opencollective.com/nest#backer" target="_blank"><img src="https://opencollective.com/nest/backers/badge.svg" alt="Backers on Open Collective" /></a>
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://opencollective.com/nest/sponsors/badge.svg" alt="Sponsors on Open Collective" /></a>
<a href="https://paypal.me/kamilmysliwiec" target="_blank"><img src="https://img.shields.io/badge/Donate-PayPal-ff3f59.svg" alt="Donate us"/></a>
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://img.shields.io/badge/Support%20us-Open%20Collective-41B883.svg" alt="Support us"></a>
<a href="https://twitter.com/nestframework" target="_blank"><img src="https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow" alt="Follow us on Twitter"></a>
</p>
<!--[![Backers on Open Collective](https://opencollective.com/nest/backers/badge.svg)](https://opencollective.com/nest#backer)
[![Sponsors on Open Collective](https://opencollective.com/nest/sponsors/badge.svg)](https://opencollective.com/nest#sponsor)-->
## 技术栈
## Description
### 后端
- **框架**: NestJS
- **语言**: TypeScript
- **数据库**: PostgreSQL (TypeORM)
- **爬虫**: axios
- **任务调度**: @nestjs/schedule
- **AI 服务**: OpenAI API
[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository.
### 前端
- **框架**: Vue.js 3
- **构建工具**: Vite
- **UI**: Element Plus
- **图标**: @element-plus/icons-vue
- **HTTP 客户端**: Axios
- **Tailwind CSS**: 用于样式辅助
## Project setup
## 项目结构
```bash
$ npm install
```
src/
├── ai/ # AI 模块
│ ├── entities/
│ │ └── ai-recommendation.entity.ts
│ ├── Prompt.ts
│ ├── ai.controller.ts
│ ├── ai.module.ts
│ └── ai.service.ts
├── bids/ # 投标业务模块
│ ├── controllers/
│ │ └── bid.controller.ts
│ ├── entities/
│ │ └── bid-item.entity.ts
│ ├── services/
│ │ └── bid.service.ts
│ └── bids.module.ts
├── crawler/ # 爬虫模块
│ ├── entities/
│ │ └── crawl-info-add.entity.ts
│ ├── services/
│ │ ├── bid-crawler.service.ts
│ │ ├── cdt_target.ts
│ │ ├── chng_target.ts
│ │ ├── ceic_target.ts
│ │ ├── cgnpc_target.ts
│ │ ├── chdtp_target.ts
│ │ ├── cnncecp_target.ts
│ │ ├── cnooc_target.ts
│ │ ├── eps_target.ts
│ │ ├── espic_target.ts
│ │ ├── powerbeijing_target.ts
│ │ ├── sdicc_target.ts
│ │ └── szecp_target.ts
│ ├── crawler.controller.ts
│ └── crawler.module.ts
├── database/ # 数据库模块
│ └── database.module.ts
├── keywords/ # 关键词管理模块
│ ├── keyword.entity.ts
│ ├── keywords.controller.ts
│ ├── keywords.module.ts
│ └── keywords.service.ts
├── schedule/ # 定时任务
│ ├── tasks/
│ │ └── bid-crawl.task.ts
│ └── schedule.module.ts
├── scripts/ # 脚本工具
│ ├── ai-recommendations.ts
│ ├── crawl.ts
│ ├── deploy.ps1
│ ├── remove-duplicates.ts
│ ├── sync.ts
│ └── update-source.ts
├── common/ # 公共模块
│ └── logger/
│ ├── logger.module.ts
│ ├── logger.service.ts
│ └── winston.config.ts
├── app.controller.ts
├── app.module.ts
├── app.service.ts
└── main.ts
frontend/
├── src/
│ ├── assets/
│ │ └── vue.svg
│ ├── components/
│ │ ├── Dashboard.vue
│ │ ├── Dashboard-AI.vue
│ │ ├── PinnedProject.vue
│ │ ├── Bids.vue
│ │ ├── Keywords.vue
│ │ └── CrawlInfo.vue
│ ├── utils/
│ │ └── api.ts
│ ├── App.vue
│ ├── main.ts
│ └── style.css
├── .env
├── .env.example
├── .gitignore
├── README.md
├── index.html
├── package.json
├── postcss.config.js
├── tsconfig.app.json
├── tsconfig.json
├── tsconfig.node.json
└── vite.config.ts
```
## Compile and run the project
## 快速开始
```bash
# development
$ npm run start
### 1. 环境准备
# watch mode
$ npm run start:dev
确保已安装 Node.js (18+) 和 PostgreSQL。
# production mode
$ npm run start:prod
```
### 2. 数据库配置
## Run tests
```bash
# unit tests
$ npm run test
# e2e tests
$ npm run test:e2e
# test coverage
$ npm run test:cov
```
## Deployment
When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information.
If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:
```bash
$ npm install -g @nestjs/mau
$ mau deploy
```
With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure.
## Resources
Check out a few resources that may come in handy when working with NestJS:
- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework.
- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks.
- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).
## Support
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).
## Stay in touch
- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec)
- Website - [https://nestjs.com](https://nestjs.com/)
- Twitter - [@nestframework](https://twitter.com/nestframework)
## License
Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE).
## How to Run
### 1. Database Setup
Update the `.env` file with your PostgreSQL credentials:
复制 `.env.example``.env` 并配置数据库连接:
```env
DATABASE_TYPE=postgres
@@ -110,73 +133,195 @@ DATABASE_USERNAME=your_username
DATABASE_PASSWORD=your_password
DATABASE_NAME=bidding
DATABASE_SYNCHRONIZE=true
```
### 2. Install Dependencies
```bash
npm install
cd frontend && npm install
```
### 3. Build and Start
```bash
# From the root directory
cd frontend && npm run build
cd ..
npm run build
npm run start
```
## Features
### Frontend Features
- **Dashboard**: View high priority bids and today's bids
- **Date Filtering**:
- Click "3天" or "7天" buttons to filter bids from the last 3 or 7 days
- The filter only limits the start date, showing all data from the selected start date onwards (including data newer than the end date)
- **Keyword Filtering**: Filter bids by keywords (saved in localStorage)
- **All Bids**: View all bids with pagination and source filtering
- **Keyword Management**: Add and delete keywords with weight-based priority
### Backend Features
- **Multi-Source Crawling**: Crawls bidding information from multiple sources:
- ChdtpCrawler
- ChngCrawler
- SzecpCrawler
- CdtCrawler
- EpsCrawler
- CnncecpCrawler
- CgnpcCrawler
- CeicCrawler
- EspicCrawler
- PowerbeijingCrawler
- **Automatic Retry**: If a crawler returns 0 items, it will be retried after all crawlers complete
- **Proxy Support**: Configurable proxy settings via environment variables
- **Scheduled Tasks**: Automatic crawling at scheduled intervals
### Environment Variables
```env
# Database
DATABASE_TYPE=postgres
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_USERNAME=your_username
DATABASE_PASSWORD=your_password
DATABASE_NAME=bidding
DATABASE_SYNCHRONIZE=true
# Proxy (optional)
# 代理配置(可选)
PROXY_HOST=your_proxy_host
PROXY_PORT=your_proxy_port
PROXY_USERNAME=your_proxy_username
PROXY_PASSWORD=your_proxy_password
# AI 配置
OPENAI_API_KEY=your_openai_api_key
```
## Initial Setup
### 3. 安装依赖
The system will automatically initialize with the preset keywords: "山东", "海", "建设", "工程", "采购". You can manage these and view crawled bidding information at http://localhost:3000.
```bash
# 安装后端依赖
npm install
# 安装前端依赖
cd frontend && npm install
```
### 4. 运行项目
```bash
# 开发模式 - 后端
npm run start:dev
# 开发模式 - 前端
cd frontend && npm run dev
```
### 5. 构建生产版本
```bash
# 构建前端
cd frontend && npm run build
# 构建后端
cd .. && npm run build
# 运行生产版本
npm run start:prod
```
## 核心功能
### 智能爬虫模块
- **多源爬取**: 支持 12 个主流招标网站
- 中国华电集团有限公司电子商务平台 (CHDTP)
- 中国华能集团有限公司电子商务平台 (CHNG)
- 深圳交易集团有限公司 (SZECP)
- 中国大唐集团电子商务平台 (CDT)
- 中国电力招标网 (EPS)
- 国家能源投资集团有限责任公司 (CNNCECP)
- 中国石油天然气集团有限公司 (CGNPC)
- 中国能源建设集团有限公司 (CEIC)
- 中国电力建设集团有限公司 (ESPIC)
- 北京电力交易中心 (POWERBEIJING)
- 山东能源集团有限公司 (SDICC)
- 中国海洋石油集团有限公司 (CNOOC)
- **智能防封策略**:
- 随机请求间隔 (1-3 秒)
- 固定 User-Agent
- 异常检测与自动重试机制
- 代理支持
- **定时任务**:
- 爬虫任务:已暂停(默认每天午夜执行)
- 数据清理:每天午夜自动执行
### 数据处理与存储
- **数据模型**:
- 投标项目标题
- 详细页面 URL
- 发布时间
- 来源网站
- 置顶标记
- 创建时间
- 更新时间
- **增量存储**:
- 通过 URL 哈希值判断是否为新数据
- 仅存储当天和最近 7 天的历史数据
- 每日自动清理 30 天前的数据
### 关键词智能监控
- **预设关键词**: "山东", "海", "建设", "工程", "采购"
- **自定义关键词**: 通过 Web 界面添加/删除关键词
- **权重设置**: 可设置关键词权重 (1-5 级)
- **匹配逻辑**:
- 标题完全匹配和部分匹配
- 多关键词叠加权重
- 支持正则表达式高级匹配
### AI 智能推荐
- **智能分析**: 使用 AI 分析投标信息的相关性
- **推荐评分**: 基于关键词匹配和内容分析生成推荐评分
- **智能摘要**: 自动生成投标信息摘要
### Web 展示界面
- **仪表盘**:
- 高优先级投标信息(匹配自定义关键词)
- 今日新增投标列表(按时间倒序)
- AI 推荐投标信息
- 置顶项目
- **交互功能**:
- 关键词管理面板
- 按日期/来源/关键词筛选
- 信息标记已读/未读状态
- 项目置顶功能
- 爬取信息查看
- **响应式设计**: 适配桌面和移动设备
## API 接口
### 投标信息
- `GET /api/bids` - 获取投标列表(支持分页、筛选)
- `GET /api/bids/recent` - 获取最近投标
- `GET /api/bids/pinned` - 获取置顶投标
- `GET /api/bids/sources` - 获取来源列表
- `GET /api/bids/by-date-range` - 按日期范围获取投标
- `GET /api/bids/crawl-info-stats` - 获取爬取信息统计
- `PATCH /api/bids/:title/pin` - 更新置顶状态
### 关键词管理
- `GET /api/keywords` - 获取所有关键词
- `POST /api/keywords` - 添加新关键词
- `DELETE /api/keywords/:id` - 删除关键词
### AI 服务
- `POST /api/ai/recommendations` - 获取 AI 推荐
- `POST /api/ai/save-recommendations` - 保存 AI 推荐
- `GET /api/ai/latest-recommendations` - 获取最新 AI 推荐
### 爬虫管理
- `GET /api/crawler/status` - 获取爬虫状态
- `POST /api/crawler/run` - 运行爬虫
- `POST /api/crawler/crawl/:sourceName` - 爬取单个来源
## 前端路由
- `/` - 仪表盘(默认页面)
- `/bids` - 全部投标信息
- `/keywords` - 关键词管理
- `/ai` - AI 推荐页面
- `/crawl-info` - 爬取信息
## 测试
```bash
# 单元测试
npm run test
# E2E 测试
npm run test:e2e
# 测试覆盖率
npm run test:cov
```
## 部署
项目包含 PowerShell 部署脚本 `src/scripts/deploy.ps1`,用于自动化部署流程。
## 环境变量
| 变量名 | 说明 | 默认值 |
|--------|------|--------|
| DATABASE_TYPE | 数据库类型 | postgres |
| DATABASE_HOST | 数据库主机 | localhost |
| DATABASE_PORT | 数据库端口 | 5432 |
| DATABASE_USERNAME | 数据库用户名 | - |
| DATABASE_PASSWORD | 数据库密码 | - |
| DATABASE_NAME | 数据库名称 | bidding |
| DATABASE_SYNCHRONIZE | 自动同步数据库 | true |
| PROXY_HOST | 代理主机 | - |
| PROXY_PORT | 代理端口 | - |
| PROXY_USERNAME | 代理用户名 | - |
| PROXY_PASSWORD | 代理密码 | - |
| OPENAI_API_KEY | OpenAI API 密钥 | - |
## 许可证
MIT

42
app/electron-builder.json Normal file
View File

@@ -0,0 +1,42 @@
{
"productName": "投标应用",
"appId": "com.bidding.app",
"directories": {
"output": "dist-electron",
"app": "."
},
"files": [
"app/**/*",
"dist/**/*",
"frontend/dist/**/*",
".env",
"package.json"
],
"asarUnpack": [
"dist/**/*",
"node_modules/**/*"
],
"win": {
"target": "nsis",
"icon": "frontend/public/favicon.ico",
"requestedExecutionLevel": "asInvoker"
},
"nsis": {
"oneClick": false,
"allowToChangeInstallationDirectory": true,
"createDesktopShortcut": true,
"createStartMenuShortcut": true,
"shortcutName": "投标应用"
},
"extraResources": [
{
"from": ".env",
"to": ".env",
"filter": ["**/*"]
}
],
"publish": {
"provider": "generic",
"url": "http://localhost:3000/"
}
}

267
app/main.js Normal file
View File

@@ -0,0 +1,267 @@
const { app, BrowserWindow, ipcMain, Menu } = require('electron');
const path = require('path');
const { spawn } = require('child_process');
const dotenv = require('dotenv');
const fs = require('fs');
// 设置控制台输出编码为 UTF-8Windows 兼容)
if (process.platform === 'win32') {
// 设置 stdout 和 stderr 的编码
if (process.stdout.setDefaultEncoding) {
process.stdout.setDefaultEncoding('utf8');
}
if (process.stderr.setDefaultEncoding) {
process.stderr.setDefaultEncoding('utf8');
}
// 尝试设置控制台代码页为 UTF-8
try {
require('child_process').exec('chcp 65001', { encoding: 'utf8' });
} catch (e) {
// 忽略错误
}
}
// 加载环境变量
const envPath = path.join(__dirname, '..', '.env');
if (fs.existsSync(envPath)) {
dotenv.config({ path: envPath });
}
let mainWindow;
let backendProcess;
// 判断是否为开发模式
// 主要依赖 app.isPackaged如果为 false 则是开发环境
// 或者检查路径是否包含 app.asar打包后的应用
const isDevelopment = !app.isPackaged || process.env.NODE_ENV === 'development';
console.log('运行模式检测:');
console.log(' - app.isPackaged:', app.isPackaged);
console.log(' - process.resourcesPath:', process.resourcesPath);
console.log(' - process.env.NODE_ENV:', process.env.NODE_ENV);
console.log(' - isDevelopment:', isDevelopment);
/**
* 创建Electron主窗口
*/
function createWindow() {
mainWindow = new BrowserWindow({
width: 1200,
height: 800,
webPreferences: {
preload: path.join(__dirname, 'preload.js'),
nodeIntegration: false,
contextIsolation: true,
},
autoHideMenuBar: false,
title: '投标应用',
});
// 加载前端页面
const indexPath = path.join(__dirname, '..', 'frontend', 'dist', 'index.html');
mainWindow.loadFile(indexPath);
// 开发环境下打开开发者工具
if (isDevelopment || process.env.NODE_ENV === 'development') {
console.log('开发模式:打开开发者工具');
mainWindow.webContents.openDevTools();
// 过滤掉 DevTools 的 Autofill 相关错误
mainWindow.webContents.on('console-message', (event, level, message, line, sourceId) => {
if (message.includes('Autofill.enable') || message.includes('Autofill.setAddresses')) {
event.preventDefault();
}
});
}
mainWindow.on('closed', () => {
mainWindow = null;
});
}
/**
* 等待后端服务启动
*/
function waitForBackend(port = 3000, maxRetries = 30, interval = 1000) {
return new Promise((resolve, reject) => {
let retries = 0;
const checkBackend = () => {
const net = require('net');
const client = new net.Socket();
client.once('connect', () => {
client.destroy();
console.log('后端服务已启动');
resolve();
});
client.once('error', () => {
client.destroy();
retry();
});
client.once('timeout', () => {
client.destroy();
retry();
});
client.connect(port, 'localhost');
client.setTimeout(1000);
function retry() {
retries++;
if (retries >= maxRetries) {
reject(new Error('后端服务启动超时'));
} else {
console.log(`等待后端服务启动... (${retries}/${maxRetries})`);
setTimeout(checkBackend, interval);
}
}
};
checkBackend();
});
}
/**
* 启动后端服务
*/
async function startBackend() {
let backendPath;
if (app.isPackaged) {
// 生产环境:使用 app.asar.unpacked 中的文件
backendPath = path.join(process.resourcesPath, 'app.asar.unpacked', 'dist', 'main.js');
} else {
// 开发环境:使用项目根目录下的 dist 文件夹
backendPath = path.join(__dirname, '..', 'dist', 'main.js');
}
// 检查后端构建文件是否存在
if (!fs.existsSync(backendPath)) {
console.error('后端服务构建文件不存在,路径:', backendPath);
console.error('请先执行 npm run build');
return;
}
console.log('启动后端服务:', backendPath);
// 启动后端服务
backendProcess = spawn('node', [backendPath], {
env: {
...process.env,
NODE_ENV: isDevelopment ? 'development' : (process.env.NODE_ENV || 'production'),
// 设置编码环境变量Windows
...(process.platform === 'win32' && {
PYTHONIOENCODING: 'utf-8',
LANG: 'en_US.UTF-8'
}),
},
stdio: 'pipe',
// Windows 上设置 shell 选项以确保编码正确
...(process.platform === 'win32' && { shell: false }),
});
// 捕获并显示后端进程的输出
backendProcess.stdout.on('data', (data) => {
// 确保正确解码 UTF-8 编码的数据
const output = Buffer.isBuffer(data) ? data.toString('utf8') : data.toString();
console.log(`[后端输出] ${output}`);
});
backendProcess.stderr.on('data', (data) => {
// 确保正确解码 UTF-8 编码的数据
const output = Buffer.isBuffer(data) ? data.toString('utf8') : data.toString();
console.error(`[后端错误] ${output}`);
});
backendProcess.on('error', (error) => {
console.error('后端服务启动失败:', error);
});
let backendExited = false;
backendProcess.on('exit', (code, signal) => {
backendExited = true;
if (code !== 0 && code !== null) {
console.error(`后端服务异常退出,退出码: ${code}, 信号: ${signal}`);
} else {
console.log(`后端服务退出,退出码: ${code}`);
}
backendProcess = null;
});
// 等待后端服务启动完成
try {
await waitForBackend();
// 检查后端是否在等待期间就退出了
if (backendExited) {
throw new Error('后端服务在启动过程中退出');
}
} catch (error) {
console.error('等待后端服务启动失败:', error.message);
if (backendProcess) {
console.error('正在停止后端进程...');
backendProcess.kill();
backendProcess = null;
}
throw error; // 重新抛出错误,让调用者知道启动失败
}
}
/**
* 停止后端服务
*/
function stopBackend() {
if (backendProcess) {
console.log('正在停止后端服务...');
backendProcess.kill();
backendProcess = null;
}
}
// 应用就绪时启动后端服务,然后创建窗口
app.on('ready', async () => {
try {
await startBackend();
createWindow();
Menu.setApplicationMenu(null);
} catch (error) {
console.error('应用启动失败:', error);
// 显示错误对话框
const { dialog } = require('electron');
dialog.showErrorBox(
'启动失败',
`后端服务启动失败: ${error.message}\n\n请检查控制台输出以获取更多信息。`
);
}
});
// 所有窗口关闭时退出应用
app.on('window-all-closed', () => {
stopBackend();
if (process.platform !== 'darwin') {
app.quit();
}
});
// MacOS上点击dock图标时重新创建窗口
app.on('activate', async () => {
if (BrowserWindow.getAllWindows().length === 0) {
if (!backendProcess) {
await startBackend();
}
createWindow();
}
});
// 应用退出前停止后端服务
app.on('before-quit', () => {
stopBackend();
});
// 处理来自渲染进程的IPC消息
ipcMain.handle('get-env', (event, key) => {
return process.env[key];
});

13
app/package.json Normal file
View File

@@ -0,0 +1,13 @@
{
"name": "bidding-app",
"version": "0.0.1",
"description": "投标应用Electron版本",
"main": "main.js",
"scripts": {
"start": "electron .",
"build": "electron-builder"
},
"keywords": ["electron", "bidding", "app"],
"author": "",
"license": "UNLICENSED"
}

14
app/preload.js Normal file
View File

@@ -0,0 +1,14 @@
const { contextBridge, ipcRenderer } = require('electron');
/**
* 预加载脚本,用于在渲染进程和主进程之间通信
* 提供安全的API给渲染进程访问主进程功能
*/
contextBridge.exposeInMainWorld('electronAPI', {
/**
* 获取环境变量值
* @param {string} key - 环境变量名称
* @returns {Promise<string>} - 环境变量值
*/
getEnv: (key) => ipcRenderer.invoke('get-env', key),
});

View File

@@ -29,7 +29,7 @@ export default tseslint.config(
'@typescript-eslint/no-explicit-any': 'off',
'@typescript-eslint/no-floating-promises': 'warn',
'@typescript-eslint/no-unsafe-argument': 'warn',
"prettier/prettier": ["error", { endOfLine: "auto" }],
"prettier/prettier": "off",
},
},
);

View File

@@ -1,2 +1,5 @@
# ARK API Key (用于 AI 推荐)
VITE_ARK_API_KEY=a63d58b6-cf56-434b-8a42-5c781ba0822a
# 后端 API 地址
VITE_API_BASE_URL=http://localhost:3300/

5
frontend/.env.production Normal file
View File

@@ -0,0 +1,5 @@
# ARK API Key (用于 AI 推荐)
VITE_ARK_API_KEY=a63d58b6-cf56-434b-8a42-5c781ba0822a
# 后端 API 地址
VITE_API_BASE_URL=http://139.180.190.142:3300/

View File

@@ -1,5 +0,0 @@
# Vue 3 + TypeScript + Vite
This template should help get you started developing with Vue 3 and TypeScript in Vite. The template uses Vue 3 `<script setup>` SFCs, check out the [script setup docs](https://v3.vuejs.org/api/sfc-script-setup.html#sfc-script-setup) to learn more.
Learn more about the recommended Project Setup and IDE Support in the [Vue Docs TypeScript Guide](https://vuejs.org/guide/typescript/overview.html#project-setup).

View File

@@ -3,7 +3,7 @@
<head>
<meta charset="UTF-8" />
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no" />
<title>投标</title>
</head>
<body>

View File

@@ -11,17 +11,35 @@
"@element-plus/icons-vue": "^2.3.2",
"axios": "^1.13.2",
"element-plus": "^2.13.1",
"openai": "^6.16.0",
"vue": "^3.5.24"
},
"devDependencies": {
"@tailwindcss/postcss": "^4.1.18",
"@types/node": "^24.10.1",
"@vitejs/plugin-vue": "^6.0.1",
"@vue/tsconfig": "^0.8.1",
"autoprefixer": "^10.4.23",
"postcss": "^8.5.6",
"tailwindcss": "^4.1.18",
"typescript": "~5.9.3",
"vite": "^7.2.4",
"vue-tsc": "^3.1.4"
}
},
"node_modules/@alloc/quick-lru": {
"version": "5.2.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/@alloc/quick-lru/-/quick-lru-5.2.0.tgz",
"integrity": "sha512-UrcABB+4bUrFABwbluTIBErXwvbsU/V7TZWfmbgJfbkwiBuziS9gxdODUyuiecfdGQ85jglMW6juS3+z5TsKLw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/@babel/helper-string-parser": {
"version": "7.27.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/@babel/helper-string-parser/-/helper-string-parser-7.27.1.tgz",
@@ -553,12 +571,55 @@
"integrity": "sha512-aGTxbpbg8/b5JfU1HXSrbH3wXZuLPJcNEcZQFMxLs3oSzgtVu6nFPkbbGGUvBcUjKV2YyB9Wxxabo+HEH9tcRQ==",
"license": "MIT"
},
"node_modules/@jridgewell/gen-mapping": {
"version": "0.3.13",
"resolved": "https://mirrors.cloud.tencent.com/npm/@jridgewell/gen-mapping/-/gen-mapping-0.3.13.tgz",
"integrity": "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/sourcemap-codec": "^1.5.0",
"@jridgewell/trace-mapping": "^0.3.24"
}
},
"node_modules/@jridgewell/remapping": {
"version": "2.3.5",
"resolved": "https://mirrors.cloud.tencent.com/npm/@jridgewell/remapping/-/remapping-2.3.5.tgz",
"integrity": "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/gen-mapping": "^0.3.5",
"@jridgewell/trace-mapping": "^0.3.24"
}
},
"node_modules/@jridgewell/resolve-uri": {
"version": "3.1.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/@jridgewell/resolve-uri/-/resolve-uri-3.1.2.tgz",
"integrity": "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6.0.0"
}
},
"node_modules/@jridgewell/sourcemap-codec": {
"version": "1.5.5",
"resolved": "https://mirrors.cloud.tencent.com/npm/@jridgewell/sourcemap-codec/-/sourcemap-codec-1.5.5.tgz",
"integrity": "sha512-cYQ9310grqxueWbl+WuIUIaiUaDcj7WOq5fVhEljNVgRfOUhY9fy2zTvfoqWsnebh8Sl70VScFbICvJnLKB0Og==",
"license": "MIT"
},
"node_modules/@jridgewell/trace-mapping": {
"version": "0.3.31",
"resolved": "https://mirrors.cloud.tencent.com/npm/@jridgewell/trace-mapping/-/trace-mapping-0.3.31.tgz",
"integrity": "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/resolve-uri": "^3.1.0",
"@jridgewell/sourcemap-codec": "^1.4.14"
}
},
"node_modules/@popperjs/core": {
"name": "@sxzz/popperjs-es",
"version": "2.11.7",
@@ -927,6 +988,277 @@
"win32"
]
},
"node_modules/@tailwindcss/node": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/node/-/node-4.1.18.tgz",
"integrity": "sha512-DoR7U1P7iYhw16qJ49fgXUlry1t4CpXeErJHnQ44JgTSKMaZUdf17cfn5mHchfJ4KRBZRFA/Coo+MUF5+gOaCQ==",
"dev": true,
"license": "MIT",
"dependencies": {
"@jridgewell/remapping": "^2.3.4",
"enhanced-resolve": "^5.18.3",
"jiti": "^2.6.1",
"lightningcss": "1.30.2",
"magic-string": "^0.30.21",
"source-map-js": "^1.2.1",
"tailwindcss": "4.1.18"
}
},
"node_modules/@tailwindcss/oxide": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide/-/oxide-4.1.18.tgz",
"integrity": "sha512-EgCR5tTS5bUSKQgzeMClT6iCY3ToqE1y+ZB0AKldj809QXk1Y+3jB0upOYZrn9aGIzPtUsP7sX4QQ4XtjBB95A==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">= 10"
},
"optionalDependencies": {
"@tailwindcss/oxide-android-arm64": "4.1.18",
"@tailwindcss/oxide-darwin-arm64": "4.1.18",
"@tailwindcss/oxide-darwin-x64": "4.1.18",
"@tailwindcss/oxide-freebsd-x64": "4.1.18",
"@tailwindcss/oxide-linux-arm-gnueabihf": "4.1.18",
"@tailwindcss/oxide-linux-arm64-gnu": "4.1.18",
"@tailwindcss/oxide-linux-arm64-musl": "4.1.18",
"@tailwindcss/oxide-linux-x64-gnu": "4.1.18",
"@tailwindcss/oxide-linux-x64-musl": "4.1.18",
"@tailwindcss/oxide-wasm32-wasi": "4.1.18",
"@tailwindcss/oxide-win32-arm64-msvc": "4.1.18",
"@tailwindcss/oxide-win32-x64-msvc": "4.1.18"
}
},
"node_modules/@tailwindcss/oxide-android-arm64": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-android-arm64/-/oxide-android-arm64-4.1.18.tgz",
"integrity": "sha512-dJHz7+Ugr9U/diKJA0W6N/6/cjI+ZTAoxPf9Iz9BFRF2GzEX8IvXxFIi/dZBloVJX/MZGvRuFA9rqwdiIEZQ0Q==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"android"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-darwin-arm64": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-darwin-arm64/-/oxide-darwin-arm64-4.1.18.tgz",
"integrity": "sha512-Gc2q4Qhs660bhjyBSKgq6BYvwDz4G+BuyJ5H1xfhmDR3D8HnHCmT/BSkvSL0vQLy/nkMLY20PQ2OoYMO15Jd0A==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-darwin-x64": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-darwin-x64/-/oxide-darwin-x64-4.1.18.tgz",
"integrity": "sha512-FL5oxr2xQsFrc3X9o1fjHKBYBMD1QZNyc1Xzw/h5Qu4XnEBi3dZn96HcHm41c/euGV+GRiXFfh2hUCyKi/e+yw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-freebsd-x64": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-freebsd-x64/-/oxide-freebsd-x64-4.1.18.tgz",
"integrity": "sha512-Fj+RHgu5bDodmV1dM9yAxlfJwkkWvLiRjbhuO2LEtwtlYlBgiAT4x/j5wQr1tC3SANAgD+0YcmWVrj8R9trVMA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"freebsd"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-linux-arm-gnueabihf": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-linux-arm-gnueabihf/-/oxide-linux-arm-gnueabihf-4.1.18.tgz",
"integrity": "sha512-Fp+Wzk/Ws4dZn+LV2Nqx3IilnhH51YZoRaYHQsVq3RQvEl+71VGKFpkfHrLM/Li+kt5c0DJe/bHXK1eHgDmdiA==",
"cpu": [
"arm"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-linux-arm64-gnu": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-linux-arm64-gnu/-/oxide-linux-arm64-gnu-4.1.18.tgz",
"integrity": "sha512-S0n3jboLysNbh55Vrt7pk9wgpyTTPD0fdQeh7wQfMqLPM/Hrxi+dVsLsPrycQjGKEQk85Kgbx+6+QnYNiHalnw==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-linux-arm64-musl": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-linux-arm64-musl/-/oxide-linux-arm64-musl-4.1.18.tgz",
"integrity": "sha512-1px92582HkPQlaaCkdRcio71p8bc8i/ap5807tPRDK/uw953cauQBT8c5tVGkOwrHMfc2Yh6UuxaH4vtTjGvHg==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-linux-x64-gnu": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-linux-x64-gnu/-/oxide-linux-x64-gnu-4.1.18.tgz",
"integrity": "sha512-v3gyT0ivkfBLoZGF9LyHmts0Isc8jHZyVcbzio6Wpzifg/+5ZJpDiRiUhDLkcr7f/r38SWNe7ucxmGW3j3Kb/g==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-linux-x64-musl": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-linux-x64-musl/-/oxide-linux-x64-musl-4.1.18.tgz",
"integrity": "sha512-bhJ2y2OQNlcRwwgOAGMY0xTFStt4/wyU6pvI6LSuZpRgKQwxTec0/3Scu91O8ir7qCR3AuepQKLU/kX99FouqQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-wasm32-wasi": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-wasm32-wasi/-/oxide-wasm32-wasi-4.1.18.tgz",
"integrity": "sha512-LffYTvPjODiP6PT16oNeUQJzNVyJl1cjIebq/rWWBF+3eDst5JGEFSc5cWxyRCJ0Mxl+KyIkqRxk1XPEs9x8TA==",
"bundleDependencies": [
"@napi-rs/wasm-runtime",
"@emnapi/core",
"@emnapi/runtime",
"@tybys/wasm-util",
"@emnapi/wasi-threads",
"tslib"
],
"cpu": [
"wasm32"
],
"dev": true,
"license": "MIT",
"optional": true,
"dependencies": {
"@emnapi/core": "^1.7.1",
"@emnapi/runtime": "^1.7.1",
"@emnapi/wasi-threads": "^1.1.0",
"@napi-rs/wasm-runtime": "^1.1.0",
"@tybys/wasm-util": "^0.10.1",
"tslib": "^2.4.0"
},
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/@tailwindcss/oxide-win32-arm64-msvc": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-win32-arm64-msvc/-/oxide-win32-arm64-msvc-4.1.18.tgz",
"integrity": "sha512-HjSA7mr9HmC8fu6bdsZvZ+dhjyGCLdotjVOgLA2vEqxEBZaQo9YTX4kwgEvPCpRh8o4uWc4J/wEoFzhEmjvPbA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/oxide-win32-x64-msvc": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/oxide-win32-x64-msvc/-/oxide-win32-x64-msvc-4.1.18.tgz",
"integrity": "sha512-bJWbyYpUlqamC8dpR7pfjA0I7vdF6t5VpUGMWRkXVE3AXgIZjYUYAK7II1GNaxR8J1SSrSrppRar8G++JekE3Q==",
"cpu": [
"x64"
],
"dev": true,
"license": "MIT",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 10"
}
},
"node_modules/@tailwindcss/postcss": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/@tailwindcss/postcss/-/postcss-4.1.18.tgz",
"integrity": "sha512-Ce0GFnzAOuPyfV5SxjXGn0CubwGcuDB0zcdaPuCSzAa/2vII24JTkH+I6jcbXLb1ctjZMZZI6OjDaLPJQL1S0g==",
"dev": true,
"license": "MIT",
"dependencies": {
"@alloc/quick-lru": "^5.2.0",
"@tailwindcss/node": "4.1.18",
"@tailwindcss/oxide": "4.1.18",
"postcss": "^8.4.41",
"tailwindcss": "4.1.18"
}
},
"node_modules/@types/estree": {
"version": "1.0.8",
"resolved": "https://mirrors.cloud.tencent.com/npm/@types/estree/-/estree-1.0.8.tgz",
@@ -1255,6 +1587,43 @@
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==",
"license": "MIT"
},
"node_modules/autoprefixer": {
"version": "10.4.23",
"resolved": "https://mirrors.cloud.tencent.com/npm/autoprefixer/-/autoprefixer-10.4.23.tgz",
"integrity": "sha512-YYTXSFulfwytnjAPlw8QHncHJmlvFKtczb8InXaAx9Q0LbfDnfEYDE55omerIJKihhmU61Ft+cAOSzQVaBUmeA==",
"dev": true,
"funding": [
{
"type": "opencollective",
"url": "https://opencollective.com/postcss/"
},
{
"type": "tidelift",
"url": "https://tidelift.com/funding/github/npm/autoprefixer"
},
{
"type": "github",
"url": "https://github.com/sponsors/ai"
}
],
"license": "MIT",
"dependencies": {
"browserslist": "^4.28.1",
"caniuse-lite": "^1.0.30001760",
"fraction.js": "^5.3.4",
"picocolors": "^1.1.1",
"postcss-value-parser": "^4.2.0"
},
"bin": {
"autoprefixer": "bin/autoprefixer"
},
"engines": {
"node": "^10 || ^12 || >=14"
},
"peerDependencies": {
"postcss": "^8.1.0"
}
},
"node_modules/axios": {
"version": "1.13.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/axios/-/axios-1.13.2.tgz",
@@ -1266,6 +1635,51 @@
"proxy-from-env": "^1.1.0"
}
},
"node_modules/baseline-browser-mapping": {
"version": "2.9.14",
"resolved": "https://mirrors.cloud.tencent.com/npm/baseline-browser-mapping/-/baseline-browser-mapping-2.9.14.tgz",
"integrity": "sha512-B0xUquLkiGLgHhpPBqvl7GWegWBUNuujQ6kXd/r1U38ElPT6Ok8KZ8e+FpUGEc2ZoRQUzq/aUnaKFc/svWUGSg==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"baseline-browser-mapping": "dist/cli.js"
}
},
"node_modules/browserslist": {
"version": "4.28.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/browserslist/-/browserslist-4.28.1.tgz",
"integrity": "sha512-ZC5Bd0LgJXgwGqUknZY/vkUQ04r8NXnJZ3yYi4vDmSiZmC/pdSN0NbNRPxZpbtO4uAfDUAFffO8IZoM3Gj8IkA==",
"dev": true,
"funding": [
{
"type": "opencollective",
"url": "https://opencollective.com/browserslist"
},
{
"type": "tidelift",
"url": "https://tidelift.com/funding/github/npm/browserslist"
},
{
"type": "github",
"url": "https://github.com/sponsors/ai"
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"baseline-browser-mapping": "^2.9.0",
"caniuse-lite": "^1.0.30001759",
"electron-to-chromium": "^1.5.263",
"node-releases": "^2.0.27",
"update-browserslist-db": "^1.2.0"
},
"bin": {
"browserslist": "cli.js"
},
"engines": {
"node": "^6 || ^7 || ^8 || ^9 || ^10 || ^11 || ^12 || >=13.7"
}
},
"node_modules/call-bind-apply-helpers": {
"version": "1.0.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz",
@@ -1279,6 +1693,27 @@
"node": ">= 0.4"
}
},
"node_modules/caniuse-lite": {
"version": "1.0.30001764",
"resolved": "https://mirrors.cloud.tencent.com/npm/caniuse-lite/-/caniuse-lite-1.0.30001764.tgz",
"integrity": "sha512-9JGuzl2M+vPL+pz70gtMF9sHdMFbY9FJaQBi186cHKH3pSzDvzoUJUPV6fqiKIMyXbud9ZLg4F3Yza1vJ1+93g==",
"dev": true,
"funding": [
{
"type": "opencollective",
"url": "https://opencollective.com/browserslist"
},
{
"type": "tidelift",
"url": "https://tidelift.com/funding/github/npm/caniuse-lite"
},
{
"type": "github",
"url": "https://github.com/sponsors/ai"
}
],
"license": "CC-BY-4.0"
},
"node_modules/combined-stream": {
"version": "1.0.8",
"resolved": "https://mirrors.cloud.tencent.com/npm/combined-stream/-/combined-stream-1.0.8.tgz",
@@ -1312,6 +1747,16 @@
"node": ">=0.4.0"
}
},
"node_modules/detect-libc": {
"version": "2.1.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/detect-libc/-/detect-libc-2.1.2.tgz",
"integrity": "sha512-Btj2BOOO83o3WyH59e8MgXsxEQVcarkUOpEYrubB0urwnN10yQ364rsiByU11nZlqWYZm05i/of7io4mzihBtQ==",
"dev": true,
"license": "Apache-2.0",
"engines": {
"node": ">=8"
}
},
"node_modules/dunder-proto": {
"version": "1.0.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/dunder-proto/-/dunder-proto-1.0.1.tgz",
@@ -1326,6 +1771,13 @@
"node": ">= 0.4"
}
},
"node_modules/electron-to-chromium": {
"version": "1.5.267",
"resolved": "https://mirrors.cloud.tencent.com/npm/electron-to-chromium/-/electron-to-chromium-1.5.267.tgz",
"integrity": "sha512-0Drusm6MVRXSOJpGbaSVgcQsuB4hEkMpHXaVstcPmhu5LIedxs1xNK/nIxmQIU/RPC0+1/o0AVZfBTkTNJOdUw==",
"dev": true,
"license": "ISC"
},
"node_modules/element-plus": {
"version": "2.13.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/element-plus/-/element-plus-2.13.1.tgz",
@@ -1351,6 +1803,20 @@
"vue": "^3.3.0"
}
},
"node_modules/enhanced-resolve": {
"version": "5.18.4",
"resolved": "https://mirrors.cloud.tencent.com/npm/enhanced-resolve/-/enhanced-resolve-5.18.4.tgz",
"integrity": "sha512-LgQMM4WXU3QI+SYgEc2liRgznaD5ojbmY3sb8LxyguVkIg5FxdpTkvk72te2R38/TGKxH634oLxXRGY6d7AP+Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"graceful-fs": "^4.2.4",
"tapable": "^2.2.0"
},
"engines": {
"node": ">=10.13.0"
}
},
"node_modules/entities": {
"version": "7.0.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/entities/-/entities-7.0.0.tgz",
@@ -1450,6 +1916,16 @@
"@esbuild/win32-x64": "0.27.2"
}
},
"node_modules/escalade": {
"version": "3.2.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/escalade/-/escalade-3.2.0.tgz",
"integrity": "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/estree-walker": {
"version": "2.0.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/estree-walker/-/estree-walker-2.0.2.tgz",
@@ -1510,6 +1986,20 @@
"node": ">= 6"
}
},
"node_modules/fraction.js": {
"version": "5.3.4",
"resolved": "https://mirrors.cloud.tencent.com/npm/fraction.js/-/fraction.js-5.3.4.tgz",
"integrity": "sha512-1X1NTtiJphryn/uLQz3whtY6jK3fTqoE3ohKs0tT+Ujr1W59oopxmoEh7Lu5p6vBaPbgoM0bzveAW4Qi5RyWDQ==",
"dev": true,
"license": "MIT",
"engines": {
"node": "*"
},
"funding": {
"type": "github",
"url": "https://github.com/sponsors/rawify"
}
},
"node_modules/fsevents": {
"version": "2.3.3",
"resolved": "https://mirrors.cloud.tencent.com/npm/fsevents/-/fsevents-2.3.3.tgz",
@@ -1583,6 +2073,13 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/graceful-fs": {
"version": "4.2.11",
"resolved": "https://mirrors.cloud.tencent.com/npm/graceful-fs/-/graceful-fs-4.2.11.tgz",
"integrity": "sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==",
"dev": true,
"license": "ISC"
},
"node_modules/has-symbols": {
"version": "1.1.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/has-symbols/-/has-symbols-1.1.0.tgz",
@@ -1622,6 +2119,279 @@
"node": ">= 0.4"
}
},
"node_modules/jiti": {
"version": "2.6.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/jiti/-/jiti-2.6.1.tgz",
"integrity": "sha512-ekilCSN1jwRvIbgeg/57YFh8qQDNbwDb9xT/qu2DAHbFFZUicIl4ygVaAvzveMhMVr3LnpSKTNnwt8PoOfmKhQ==",
"dev": true,
"license": "MIT",
"peer": true,
"bin": {
"jiti": "lib/jiti-cli.mjs"
}
},
"node_modules/lightningcss": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss/-/lightningcss-1.30.2.tgz",
"integrity": "sha512-utfs7Pr5uJyyvDETitgsaqSyjCb2qNRAtuqUeWIAKztsOYdcACf2KtARYXg2pSvhkt+9NfoaNY7fxjl6nuMjIQ==",
"dev": true,
"license": "MPL-2.0",
"peer": true,
"dependencies": {
"detect-libc": "^2.0.3"
},
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
},
"optionalDependencies": {
"lightningcss-android-arm64": "1.30.2",
"lightningcss-darwin-arm64": "1.30.2",
"lightningcss-darwin-x64": "1.30.2",
"lightningcss-freebsd-x64": "1.30.2",
"lightningcss-linux-arm-gnueabihf": "1.30.2",
"lightningcss-linux-arm64-gnu": "1.30.2",
"lightningcss-linux-arm64-musl": "1.30.2",
"lightningcss-linux-x64-gnu": "1.30.2",
"lightningcss-linux-x64-musl": "1.30.2",
"lightningcss-win32-arm64-msvc": "1.30.2",
"lightningcss-win32-x64-msvc": "1.30.2"
}
},
"node_modules/lightningcss-android-arm64": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-android-arm64/-/lightningcss-android-arm64-1.30.2.tgz",
"integrity": "sha512-BH9sEdOCahSgmkVhBLeU7Hc9DWeZ1Eb6wNS6Da8igvUwAe0sqROHddIlvU06q3WyXVEOYDZ6ykBZQnjTbmo4+A==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"android"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-darwin-arm64": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-darwin-arm64/-/lightningcss-darwin-arm64-1.30.2.tgz",
"integrity": "sha512-ylTcDJBN3Hp21TdhRT5zBOIi73P6/W0qwvlFEk22fkdXchtNTOU4Qc37SkzV+EKYxLouZ6M4LG9NfZ1qkhhBWA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-darwin-x64": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-darwin-x64/-/lightningcss-darwin-x64-1.30.2.tgz",
"integrity": "sha512-oBZgKchomuDYxr7ilwLcyms6BCyLn0z8J0+ZZmfpjwg9fRVZIR5/GMXd7r9RH94iDhld3UmSjBM6nXWM2TfZTQ==",
"cpu": [
"x64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"darwin"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-freebsd-x64": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-freebsd-x64/-/lightningcss-freebsd-x64-1.30.2.tgz",
"integrity": "sha512-c2bH6xTrf4BDpK8MoGG4Bd6zAMZDAXS569UxCAGcA7IKbHNMlhGQ89eRmvpIUGfKWNVdbhSbkQaWhEoMGmGslA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"freebsd"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-linux-arm-gnueabihf": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-linux-arm-gnueabihf/-/lightningcss-linux-arm-gnueabihf-1.30.2.tgz",
"integrity": "sha512-eVdpxh4wYcm0PofJIZVuYuLiqBIakQ9uFZmipf6LF/HRj5Bgm0eb3qL/mr1smyXIS1twwOxNWndd8z0E374hiA==",
"cpu": [
"arm"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-linux-arm64-gnu": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-linux-arm64-gnu/-/lightningcss-linux-arm64-gnu-1.30.2.tgz",
"integrity": "sha512-UK65WJAbwIJbiBFXpxrbTNArtfuznvxAJw4Q2ZGlU8kPeDIWEX1dg3rn2veBVUylA2Ezg89ktszWbaQnxD/e3A==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-linux-arm64-musl": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-linux-arm64-musl/-/lightningcss-linux-arm64-musl-1.30.2.tgz",
"integrity": "sha512-5Vh9dGeblpTxWHpOx8iauV02popZDsCYMPIgiuw97OJ5uaDsL86cnqSFs5LZkG3ghHoX5isLgWzMs+eD1YzrnA==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-linux-x64-gnu": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-linux-x64-gnu/-/lightningcss-linux-x64-gnu-1.30.2.tgz",
"integrity": "sha512-Cfd46gdmj1vQ+lR6VRTTadNHu6ALuw2pKR9lYq4FnhvgBc4zWY1EtZcAc6EffShbb1MFrIPfLDXD6Xprbnni4w==",
"cpu": [
"x64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-linux-x64-musl": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-linux-x64-musl/-/lightningcss-linux-x64-musl-1.30.2.tgz",
"integrity": "sha512-XJaLUUFXb6/QG2lGIW6aIk6jKdtjtcffUT0NKvIqhSBY3hh9Ch+1LCeH80dR9q9LBjG3ewbDjnumefsLsP6aiA==",
"cpu": [
"x64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"linux"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-win32-arm64-msvc": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-win32-arm64-msvc/-/lightningcss-win32-arm64-msvc-1.30.2.tgz",
"integrity": "sha512-FZn+vaj7zLv//D/192WFFVA0RgHawIcHqLX9xuWiQt7P0PtdFEVaxgF9rjM/IRYHQXNnk61/H/gb2Ei+kUQ4xQ==",
"cpu": [
"arm64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lightningcss-win32-x64-msvc": {
"version": "1.30.2",
"resolved": "https://mirrors.cloud.tencent.com/npm/lightningcss-win32-x64-msvc/-/lightningcss-win32-x64-msvc-1.30.2.tgz",
"integrity": "sha512-5g1yc73p+iAkid5phb4oVFMB45417DkRevRbt/El/gKXJk4jid+vPFF/AXbxn05Aky8PapwzZrdJShv5C0avjw==",
"cpu": [
"x64"
],
"dev": true,
"license": "MPL-2.0",
"optional": true,
"os": [
"win32"
],
"engines": {
"node": ">= 12.0.0"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/parcel"
}
},
"node_modules/lodash": {
"version": "4.17.21",
"resolved": "https://mirrors.cloud.tencent.com/npm/lodash/-/lodash-4.17.21.tgz",
@@ -1717,12 +2487,40 @@
"node": "^10 || ^12 || ^13.7 || ^14 || >=15.0.1"
}
},
"node_modules/node-releases": {
"version": "2.0.27",
"resolved": "https://mirrors.cloud.tencent.com/npm/node-releases/-/node-releases-2.0.27.tgz",
"integrity": "sha512-nmh3lCkYZ3grZvqcCH+fjmQ7X+H0OeZgP40OierEaAptX4XofMh5kwNbWh7lBduUzCcV/8kZ+NDLCwm2iorIlA==",
"dev": true,
"license": "MIT"
},
"node_modules/normalize-wheel-es": {
"version": "1.2.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/normalize-wheel-es/-/normalize-wheel-es-1.2.0.tgz",
"integrity": "sha512-Wj7+EJQ8mSuXr2iWfnujrimU35R2W4FAErEyTmJoJ7ucwTn2hOUSsRehMb5RSYkxXGTM7Y9QpvPmp++w5ftoJw==",
"license": "BSD-3-Clause"
},
"node_modules/openai": {
"version": "6.16.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/openai/-/openai-6.16.0.tgz",
"integrity": "sha512-fZ1uBqjFUjXzbGc35fFtYKEOxd20kd9fDpFeqWtsOZWiubY8CZ1NAlXHW3iathaFvqmNtCWMIsosCuyeI7Joxg==",
"license": "Apache-2.0",
"bin": {
"openai": "bin/cli"
},
"peerDependencies": {
"ws": "^8.18.0",
"zod": "^3.25 || ^4.0"
},
"peerDependenciesMeta": {
"ws": {
"optional": true
},
"zod": {
"optional": true
}
}
},
"node_modules/path-browserify": {
"version": "1.0.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/path-browserify/-/path-browserify-1.0.1.tgz",
@@ -1769,6 +2567,7 @@
}
],
"license": "MIT",
"peer": true,
"dependencies": {
"nanoid": "^3.3.11",
"picocolors": "^1.1.1",
@@ -1778,6 +2577,13 @@
"node": "^10 || ^12 || >=14"
}
},
"node_modules/postcss-value-parser": {
"version": "4.2.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/postcss-value-parser/-/postcss-value-parser-4.2.0.tgz",
"integrity": "sha512-1NNCs6uurfkVbeXG4S8JFT9t19m45ICnif8zWLd5oPSZ50QnwMfK+H3jv408d4jw/7Bttv5axS5IiHoLaVNHeQ==",
"dev": true,
"license": "MIT"
},
"node_modules/proxy-from-env": {
"version": "1.1.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
@@ -1838,6 +2644,27 @@
"node": ">=0.10.0"
}
},
"node_modules/tailwindcss": {
"version": "4.1.18",
"resolved": "https://mirrors.cloud.tencent.com/npm/tailwindcss/-/tailwindcss-4.1.18.tgz",
"integrity": "sha512-4+Z+0yiYyEtUVCScyfHCxOYP06L5Ne+JiHhY2IjR2KWMIWhJOYZKLSGZaP5HkZ8+bY0cxfzwDE5uOmzFXyIwxw==",
"dev": true,
"license": "MIT"
},
"node_modules/tapable": {
"version": "2.3.0",
"resolved": "https://mirrors.cloud.tencent.com/npm/tapable/-/tapable-2.3.0.tgz",
"integrity": "sha512-g9ljZiwki/LfxmQADO3dEY1CbpmXT5Hm2fJ+QaGKwSXUylMybePR7/67YW7jOrrvjEgL1Fmz5kzyAjWVWLlucg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=6"
},
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/webpack"
}
},
"node_modules/tinyglobby": {
"version": "0.2.15",
"resolved": "https://mirrors.cloud.tencent.com/npm/tinyglobby/-/tinyglobby-0.2.15.tgz",
@@ -1877,6 +2704,37 @@
"dev": true,
"license": "MIT"
},
"node_modules/update-browserslist-db": {
"version": "1.2.3",
"resolved": "https://mirrors.cloud.tencent.com/npm/update-browserslist-db/-/update-browserslist-db-1.2.3.tgz",
"integrity": "sha512-Js0m9cx+qOgDxo0eMiFGEueWztz+d4+M3rGlmKPT+T4IS/jP4ylw3Nwpu6cpTTP8R1MAC1kF4VbdLt3ARf209w==",
"dev": true,
"funding": [
{
"type": "opencollective",
"url": "https://opencollective.com/browserslist"
},
{
"type": "tidelift",
"url": "https://tidelift.com/funding/github/npm/browserslist"
},
{
"type": "github",
"url": "https://github.com/sponsors/ai"
}
],
"license": "MIT",
"dependencies": {
"escalade": "^3.2.0",
"picocolors": "^1.1.1"
},
"bin": {
"update-browserslist-db": "cli.js"
},
"peerDependencies": {
"browserslist": ">= 4.21.0"
}
},
"node_modules/vite": {
"version": "7.3.1",
"resolved": "https://mirrors.cloud.tencent.com/npm/vite/-/vite-7.3.1.tgz",

View File

@@ -5,8 +5,8 @@
"type": "module",
"scripts": {
"dev": "vite",
"build": "vue-tsc -b && vite build",
"build:watch": "concurrently \"vue-tsc -b --watch\" \"vite build --watch\"",
"build": "vue-tsc -b && vite build --mode production",
"build:watch": "concurrently \"vue-tsc -b --watch\" \"vite build --watch --mode development\"",
"preview": "vite preview"
},
"dependencies": {

View File

@@ -1,10 +1,33 @@
<template>
<el-container class="layout-container" style="height: 100vh">
<el-aside width="200px" style="background-color: #545c64">
<div class="logo">投标信息一览</div>
<el-menu active-text-color="#ffd04b" background-color="#545c64" class="el-menu-vertical-demo" default-active="1"
text-color="#fff" @select="handleSelect">
<div class="layout-container" :class="{ 'is-mobile': isMobile }">
<!-- 移动端顶部导航栏 -->
<el-header v-if="isMobile" class="mobile-header">
<div class="mobile-header-content">
<el-button type="primary" link @click="toggleSidebar" class="header-btn">
<el-icon :size="24"><Fold /></el-icon>
</el-button>
<span v-if="isMobile" class="mobile-title">投标信息一览</span>
<el-button type="primary" link @click="handleLogout" v-if="currentUser" class="header-btn">
<el-icon :size="24"><SwitchButton /></el-icon>
</el-button>
</div>
</el-header>
<!-- 移动端侧边栏遮罩层 -->
<div v-if="isMobile && sidebarVisible" class="sidebar-overlay" @click="toggleSidebar"></div>
<!-- 侧边栏 - 桌面端固定显示移动端可滑动 -->
<el-aside :class="{ 'mobile-sidebar': isMobile, 'sidebar-visible': sidebarVisible }" width="200px">
<div v-if="!isMobile" class="logo">投标信息一览</div>
<el-menu
active-text-color="#ffd04b"
background-color="#545c64"
class="el-menu-vertical-demo"
:default-active="activeIndex"
text-color="#fff"
@select="handleSelect"
:collapse="isMobile"
>
<el-menu-item index="1">
<el-icon>
<MagicStick />
@@ -39,34 +62,63 @@
</el-aside>
<el-container>
<el-header style="text-align: right; font-size: 12px">
<span>Admin</span>
<!-- 桌面端顶部导航栏 -->
<el-header v-if="!isMobile" class="desktop-header">
<div class="header-content">
<span v-if="currentUser" class="username">{{ currentUser }}</span>
<el-button v-if="currentUser" type="primary" link @click="handleLogout">退出登录</el-button>
</div>
</el-header>
<el-main>
<DashboardAI v-if="activeIndex === '1'" :bids="bids" />
<Dashboard v-if="activeIndex === '2'" :today-bids="todayBids"
:keywords="keywords" :loading="loading" :is-crawling="isCrawling" @refresh="fetchData"
@update-bids="updateBidsByDateRange" />
<Dashboard
v-if="activeIndex === '2'"
:today-bids="todayBids"
:keywords="keywords"
:loading="loading"
:is-crawling="isCrawling"
@refresh="fetchData"
@update-bids="updateBidsByDateRange"
/>
<Bids v-if="activeIndex === '3'" :bids="bids" :source-options="sourceOptions" :loading="loading" :total="total"
@fetch="handleFetchBids" />
<Bids
v-if="activeIndex === '3'"
:bids="bids"
:source-options="sourceOptions"
:loading="loading"
:total="total"
@fetch="handleFetchBids"
/>
<Keywords v-if="activeIndex === '4'" :keywords="keywords" :loading="loading" @refresh="fetchData" />
<CrawlInfo v-if="activeIndex === '5'" />
</el-main>
</el-container>
</el-container>
</div>
<!-- 登录对话框 -->
<el-dialog v-model="loginDialogVisible" title="用户登录" width="90%" :style="{ maxWidth: '400px' }" :close-on-click-modal="false" :show-close="false">
<el-form :model="loginForm" label-width="80px">
<el-form-item label="用户名">
<el-input v-model="loginForm.username" placeholder="请输入用户名" @keyup.enter="handleLogin" />
</el-form-item>
<el-form-item label="密码">
<el-input v-model="loginForm.password" type="password" placeholder="请输入密码" show-password @keyup.enter="handleLogin" />
</el-form-item>
</el-form>
<template #footer>
<el-button type="primary" @click="handleLogin" :loading="loginLoading">登录</el-button>
</template>
</el-dialog>
</template>
<script setup lang="ts">
import { ref, onMounted } from 'vue'
import axios from 'axios'
import { ref, onMounted, onUnmounted } from 'vue'
import api, { setAuthCredentials, clearAuthCredentials, isAuthenticated } from './utils/api'
import { ElMessage } from 'element-plus'
import { DataBoard, Document, Setting, MagicStick, Connection } from '@element-plus/icons-vue'
import { DataBoard, Document, Setting, MagicStick, Connection, Fold, SwitchButton } from '@element-plus/icons-vue'
import Dashboard from './components/Dashboard.vue'
import DashboardAI from './components/Dashboard-AI.vue'
import Bids from './components/Bids.vue'
@@ -82,14 +134,43 @@ const isCrawling = ref(false)
const total = ref(0)
const sourceOptions = ref<string[]>([])
// 移动端状态
const isMobile = ref(false)
const sidebarVisible = ref(false)
// 登录相关状态
const loginDialogVisible = ref(false)
const loginLoading = ref(false)
const loginForm = ref({
username: '',
password: ''
})
const currentUser = ref<string | null>(null)
// 检测屏幕宽度
const checkScreenSize = () => {
isMobile.value = window.innerWidth < 768
if (!isMobile.value) {
sidebarVisible.value = true
}
}
const toggleSidebar = () => {
sidebarVisible.value = !sidebarVisible.value
}
const handleSelect = (key: string) => {
activeIndex.value = key
// 移动端选择后关闭侧边栏
if (isMobile.value) {
sidebarVisible.value = false
}
}
const handleFetchBids = async (page: number, limit: number, source?: string) => {
loading.value = true
try {
const res = await axios.get('/api/bids', {
const res = await api.get('/api/bids', {
params: {
page,
limit,
@@ -109,16 +190,16 @@ const fetchData = async () => {
loading.value = true
try {
const [bidsRes, recentRes, kwRes, sourcesRes, statusRes] = await Promise.all([
axios.get('/api/bids', {
api.get('/api/bids', {
params: {
page: 1,
limit: 10
}
}),
axios.get('/api/bids/recent'),
axios.get('/api/keywords'),
axios.get('/api/bids/sources'),
axios.get('/api/crawler/status')
api.get('/api/bids/recent'),
api.get('/api/keywords'),
api.get('/api/bids/sources'),
api.get('/api/crawler/status')
])
bids.value = bidsRes.data.items
total.value = bidsRes.data.total
@@ -145,7 +226,7 @@ const updateBidsByDateRange = async (startDate: string, endDate?: string, keywor
params.keywords = keywords.join(',')
}
const response = await axios.get('/api/bids/by-date-range', { params })
const response = await api.get('/api/bids/by-date-range', { params })
todayBids.value = response.data
ElMessage.success(`更新成功,共 ${response.data.length} 条数据`)
} catch (error) {
@@ -156,12 +237,104 @@ const updateBidsByDateRange = async (startDate: string, endDate?: string, keywor
}
}
// 处理登录
const handleLogin = async () => {
if (!loginForm.value.username || !loginForm.value.password) {
ElMessage.warning('请输入用户名和密码')
return
}
loginLoading.value = true
try {
// 保存凭证到 localStorage
setAuthCredentials(loginForm.value.username, loginForm.value.password)
// 测试凭证是否有效
await api.get('/api/bids', { params: { page: 1, limit: 1 } })
// 登录成功
currentUser.value = loginForm.value.username
loginDialogVisible.value = false
ElMessage.success('登录成功')
// 清空表单
loginForm.value.username = ''
loginForm.value.password = ''
// 加载数据
fetchData()
} catch (error: any) {
console.error('登录失败:', error)
// 清除无效凭证
clearAuthCredentials()
if (error.response?.status === 401) {
ElMessage.error('用户名或密码错误')
} else {
ElMessage.error('登录失败,请稍后重试')
}
} finally {
loginLoading.value = false
}
}
// 处理登出
const handleLogout = () => {
clearAuthCredentials()
currentUser.value = null
ElMessage.success('已退出登录')
}
// 处理认证要求事件
const handleAuthRequired = () => {
loginDialogVisible.value = true
}
onMounted(() => {
fetchData()
// 检查是否已登录
if (isAuthenticated()) {
// 从凭证中提取用户名
const credentials = localStorage.getItem('authCredentials')
if (credentials) {
try {
const decoded = atob(credentials)
const [username] = decoded.split(':')
currentUser.value = username || null
fetchData()
} catch (e) {
console.error('解析凭证失败:', e)
clearAuthCredentials()
loginDialogVisible.value = true
}
}
} else {
loginDialogVisible.value = true
}
// 监听认证要求事件
window.addEventListener('auth-required', handleAuthRequired)
// 监听屏幕大小变化
checkScreenSize()
window.addEventListener('resize', checkScreenSize)
})
onUnmounted(() => {
window.removeEventListener('auth-required', handleAuthRequired)
window.removeEventListener('resize', checkScreenSize)
})
</script>
<style scoped>
.layout-container {
display: flex;
height: 100vh;
width: 100%;
}
.layout-container .el-container {
height: 100%;
}
.layout-container .el-header {
background-color: #fff;
color: var(--el-text-color-primary);
@@ -171,6 +344,13 @@ onMounted(() => {
.layout-container .el-aside {
color: var(--el-text-color-primary);
transition: transform 0.3s ease;
display: flex;
flex-direction: column;
}
.layout-container .el-menu {
flex: 1;
}
.logo {
@@ -182,4 +362,109 @@ onMounted(() => {
font-size: 18px;
background-color: #434a50;
}
/* 桌面端样式 */
.desktop-header .header-content {
display: flex;
justify-content: flex-end;
align-items: center;
height: 100%;
padding-right: 20px;
}
.username {
margin-right: 15px;
font-size: 14px;
color: #333;
}
/* 移动端样式 */
.is-mobile .el-container {
flex: 1;
display: flex;
flex-direction: column;
}
.is-mobile .el-main {
padding-top: 60px;
}
.mobile-header {
position: fixed;
top: 0;
left: 0;
right: 0;
background-color: #434a50;
color: white;
padding: 0;
height: 50px !important;
line-height: 50px;
z-index: 1001;
}
.mobile-header-content {
display: flex;
justify-content: space-between;
align-items: center;
height: 100%;
padding: 0 10px;
}
.header-btn {
min-width: 44px;
min-height: 44px;
}
.mobile-title {
font-size: 16px;
font-weight: bold;
}
.mobile-sidebar {
position: fixed;
left: 0;
top: 0;
height: 100vh;
z-index: 1000;
transform: translateX(-100%);
transition: transform 0.3s ease;
}
.mobile-sidebar.sidebar-visible {
transform: translateX(0);
}
.sidebar-overlay {
position: fixed;
top: 0;
left: 0;
right: 0;
bottom: 0;
background-color: rgba(0, 0, 0, 0.5);
z-index: 999;
}
.is-mobile .el-main {
flex: 1;
padding: 10px;
overflow: auto;
}
/* Element Plus 菜单在移动端的样式调整 */
.is-mobile .el-menu {
border-right: none;
}
.is-mobile .el-menu-item {
padding: 0 20px !important;
min-height: 48px;
}
.is-mobile .el-button {
min-height: 44px;
}
.is-mobile .el-icon {
padding: 4px;
}
</style>

View File

@@ -0,0 +1 @@
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" aria-hidden="true" role="img" class="iconify iconify--logos" width="37.07" height="36" preserveAspectRatio="xMidYMid meet" viewBox="0 0 256 198"><path fill="#41B883" d="M204.8 0H256L128 220.8L0 0h97.92L128 51.2L157.44 0h47.36Z"></path><path fill="#41B883" d="m0 0l128 220.8L256 0h-51.2L128 132.48L50.56 0H0Z"></path><path fill="#35495E" d="M50.56 0L128 133.12L204.8 0h-47.36L128 51.2L97.92 0H50.56Z"></path></svg>

After

Width:  |  Height:  |  Size: 496 B

View File

@@ -1,8 +1,8 @@
<template>
<div>
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px;">
<h2 style="margin: 0;">All Bids</h2>
<el-select v-model="selectedSource" placeholder="Filter by Source" clearable style="width: 200px" @change="handleSourceChange">
<div class="bids-container">
<div class="bids-header">
<h2 class="bids-title">All Bids</h2>
<el-select v-model="selectedSource" placeholder="按来源筛选" clearable class="source-select" @change="handleSourceChange">
<el-option
v-for="source in sourceOptions"
:key="source"
@@ -11,8 +11,8 @@
/>
</el-select>
</div>
<el-table :data="bids" v-loading="loading" style="width: 100%">
<el-table-column label="Pin" width="60" align="center">
<el-table :data="bids" v-loading="loading" style="width: 100%" class="bids-table">
<el-table-column label="Pin" width="45" align="center">
<template #default="scope">
<el-icon
:style="{
@@ -26,13 +26,22 @@
</el-icon>
</template>
</el-table-column>
<el-table-column prop="title" label="Title">
<el-table-column label="Title" min-width="150">
<template #default="scope">
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
<a
v-if="scope.row.url"
:href="scope.row.url"
target="_blank"
class="project-link"
@click.stop
>
{{ scope.row.title }}
</a>
<span v-else>{{ scope.row.title }}</span>
</template>
</el-table-column>
<el-table-column prop="source" label="Source" width="200" />
<el-table-column prop="publishDate" label="Date" width="150">
<el-table-column prop="source" label="Source" min-width="100" />
<el-table-column prop="publishDate" label="Date" width="95">
<template #default="scope">{{ formatDate(scope.row.publishDate) }}</template>
</el-table-column>
</el-table>
@@ -44,16 +53,17 @@
layout="total, sizes, prev, pager, next, jumper"
@current-change="handlePageChange"
@size-change="handleSizeChange"
style="margin-top: 20px; justify-content: flex-end;"
class="pagination"
/>
</div>
</template>
<script setup lang="ts">
import { ref } from 'vue'
import axios from 'axios'
import api from '../utils/api'
import { ElMessage } from 'element-plus'
import { Paperclip } from '@element-plus/icons-vue'
import { formatDate } from '../utils/date.util'
interface Props {
bids: any[]
@@ -72,11 +82,6 @@ const selectedSource = ref('')
const currentPage = ref(1)
const pageSize = ref(10)
const formatDate = (dateString: string) => {
if (!dateString) return '-'
return new Date(dateString).toLocaleDateString()
}
const handleSourceChange = () => {
currentPage.value = 1
emit('fetch', currentPage.value, pageSize.value, selectedSource.value || undefined)
@@ -97,7 +102,7 @@ const handleSizeChange = (size: number) => {
const togglePin = async (item: any) => {
try {
const newPinStatus = !item.pin
await axios.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: newPinStatus })
await api.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: newPinStatus })
item.pin = newPinStatus
ElMessage.success(newPinStatus ? '已置顶' : '已取消置顶')
} catch (error) {
@@ -107,6 +112,52 @@ const togglePin = async (item: any) => {
</script>
<style scoped>
.bids-container {
padding: 0;
}
.bids-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 16px;
flex-wrap: wrap;
gap: 10px;
}
.bids-title {
margin: 0;
font-size: 1.25rem;
}
.source-select {
width: 100%;
max-width: 200px;
}
.bids-table {
width: 100%;
}
.project-link {
color: var(--el-text-color-primary);
text-decoration: none;
cursor: pointer;
transition: color 0.2s;
}
.project-link:hover {
color: var(--el-color-primary);
}
.pagination {
margin-top: 20px;
display: flex;
justify-content: flex-end;
flex-wrap: wrap;
gap: 10px;
}
a {
text-decoration: none;
color: inherit;
@@ -115,4 +166,32 @@ a {
a:hover {
color: #409eff;
}
/* 移动端响应式样式 */
@media (max-width: 768px) {
.bids-header {
flex-direction: column;
align-items: flex-start;
}
.bids-title {
font-size: 1.1rem;
}
.source-select {
max-width: 100%;
}
.bids-table {
overflow-x: auto;
}
.bids-table :deep(.el-table__cell) {
padding: 6px;
}
.pagination {
justify-content: center;
}
}
</style>

View File

@@ -1,9 +1,9 @@
<template>
<div class="crawl-info">
<el-card>
<el-card class="crawl-card">
<template #header>
<div class="card-header">
<span>爬虫统计信息</span>
<span class="card-title">爬虫统计信息</span>
<el-button type="primary" size="small" @click="fetchCrawlStats" :loading="loading">
<el-icon><Refresh /></el-icon>
刷新
@@ -11,36 +11,54 @@
</div>
</template>
<el-table :data="crawlStats" stripe style="width: 100%" v-loading="loading">
<el-table-column prop="source" label="爬虫来源" width="200" />
<el-table-column prop="count" label="本次获取数量" width="120" sortable />
<el-table-column label="最近更新时间" width="180">
<el-table :data="crawlStats" stripe style="width: 100%" v-loading="loading" class="crawl-table" :cell-class-name="tableCellClassName">
<el-table-column prop="source" label="爬虫来源" min-width="120" />
<el-table-column prop="count" label="本次获取数量" width="110" sortable />
<el-table-column label="最近更新时间" min-width="140">
<template #default="{ row }">
{{ formatDate(row.latestUpdate) }}
{{ formatDateTime(row.latestUpdate) }}
</template>
</el-table-column>
<el-table-column label="最新工程时间" width="180">
<el-table-column label="最新工程时间" min-width="140">
<template #default="{ row }">
{{ formatDate(row.latestPublishDate) }}
{{ formatDateTime(row.latestPublishDate) }}
</template>
</el-table-column>
<el-table-column label="状态" width="100">
<el-table-column label="状态" width="80">
<template #default="{ row }">
<el-tag :type="row.error ? 'danger' : (row.count > 0 ? 'success' : 'info')">
{{ row.error ? '出错' : (row.count > 0 ? '正常' : '无数据') }}
<el-tag
v-if="row.count === -1"
type="warning"
size="small"
>
正在更新...
</el-tag>
<el-tag
v-else
:type="row.error && row.error.trim() ? 'danger' : (row.count > 0 ? 'success' : 'info')"
size="small"
>
{{ row.error && row.error.trim() ? '出错' : (row.count > 0 ? '正常' : '无数据') }}
</el-tag>
</template>
</el-table-column>
<el-table-column label="错误信息" min-width="200">
<el-table-column label="操作" width="60">
<template #default="{ row }">
<span v-if="row.error" style="color: #f56c6c">{{ row.error }}</span>
<span v-else>-</span>
<el-button
type="primary"
size="small"
@click="crawlSingleSource(row.source)"
:loading="crawlingSources.has(row.source)"
:disabled="crawlingSources.has(row.source)"
>
<el-icon><Refresh /></el-icon>
</el-button>
</template>
</el-table-column>
</el-table>
<div class="summary" v-if="crawlStats.length > 0">
<el-descriptions :column="3" border>
<el-descriptions :column="2" border class="descriptions">
<el-descriptions-item label="爬虫来源总数">
{{ crawlStats.length }}
</el-descriptions-item>
@@ -60,10 +78,11 @@
</template>
<script setup lang="ts">
import { ref, computed, onMounted } from 'vue'
import axios from 'axios'
import { ref, computed, onMounted, onBeforeUnmount } from 'vue'
import api from '../utils/api'
import { ElMessage } from 'element-plus'
import { Refresh } from '@element-plus/icons-vue'
import { formatDateTime } from '../utils/date.util'
interface CrawlStat {
source: string
@@ -75,6 +94,8 @@ interface CrawlStat {
const crawlStats = ref<CrawlStat[]>([])
const loading = ref(false)
const crawlingSources = ref<Set<string>>(new Set())
let refreshTimer: number | null = null
const totalCount = computed(() => {
return crawlStats.value.reduce((sum, item) => sum + item.count, 0)
@@ -85,25 +106,13 @@ const activeSources = computed(() => {
})
const errorSources = computed(() => {
return crawlStats.value.filter(item => item.error).length
return crawlStats.value.filter(item => item.error && item.error.trim()).length
})
const formatDate = (dateStr: string | null) => {
if (!dateStr) return '-'
const date = new Date(dateStr)
return date.toLocaleString('zh-CN', {
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit'
})
}
const fetchCrawlStats = async () => {
loading.value = true
try {
const res = await axios.get('/api/bids/crawl-info-stats')
const res = await api.get('/api/bids/crawl-info-stats')
crawlStats.value = res.data
} catch (error) {
console.error('Failed to fetch crawl stats:', error)
@@ -113,14 +122,70 @@ const fetchCrawlStats = async () => {
}
}
// 表格单元格类名,用于响应式处理
const tableCellClassName = ({ columnIndex }: { columnIndex: number }) => {
// 移动端隐藏错误信息列
if (columnIndex === 5) {
return 'hidden-on-mobile'
}
return ''
}
const crawlSingleSource = async (sourceName: string) => {
crawlingSources.value.add(sourceName)
try {
ElMessage.info(`正在更新 ${sourceName}...`)
const res = await api.post(`/api/crawler/crawl/${encodeURIComponent(sourceName)}`)
if (res.data.success) {
ElMessage.success(`${sourceName} 更新成功,获取 ${res.data.count} 条数据`)
} else {
ElMessage.error(`${sourceName} 更新失败: ${res.data.error || '未知错误'}`)
}
await fetchCrawlStats()
} catch (error) {
console.error('Failed to crawl single source:', error)
ElMessage.error(`${sourceName} 更新失败`)
} finally {
crawlingSources.value.delete(sourceName)
}
}
const startAutoRefresh = () => {
if (refreshTimer !== null) {
clearInterval(refreshTimer)
}
// 取消自动刷新
// refreshTimer = window.setInterval(() => {
// fetchCrawlStats()
// }, REFRESH_INTERVAL)
}
const stopAutoRefresh = () => {
if (refreshTimer !== null) {
clearInterval(refreshTimer)
refreshTimer = null
}
}
onMounted(() => {
fetchCrawlStats()
startAutoRefresh()
})
onBeforeUnmount(() => {
stopAutoRefresh()
})
</script>
<style scoped>
.crawl-info {
padding: 20px;
padding: 0;
}
.crawl-card {
width: 100%;
}
.card-header {
@@ -129,7 +194,67 @@ onMounted(() => {
align-items: center;
}
.card-title {
font-size: 1.1rem;
font-weight: bold;
}
.crawl-table {
width: 100%;
}
.summary {
margin-top: 20px;
}
.descriptions {
width: 100%;
}
a {
text-decoration: none;
color: inherit;
}
a:hover {
color: #409eff;
}
/* 移动端响应式样式 */
@media (max-width: 768px) {
.card-header {
flex-direction: column;
align-items: flex-start;
gap: 10px;
}
.card-title {
font-size: 1rem;
}
.crawl-table {
overflow-x: auto;
}
.crawl-table :deep(.el-table__cell) {
padding: 6px;
}
.el-table {
font-size: 12px;
}
.el-descriptions {
font-size: 12px;
}
.summary {
margin-top: 16px;
}
/* 移动端隐藏错误信息列 */
.crawl-table :deep(.hidden-on-mobile) {
display: none;
}
}
</style>

View File

@@ -1,22 +1,29 @@
<template>
<div>
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px;">
<h2 style="margin: 0;">Dashboard AI</h2>
<div class="dashboard-ai-container">
<div class="dashboard-header">
<div class="dashboard-title-wrapper">
<h2 class="dashboard-title">Dashboard AI</h2>
</div>
<el-button type="primary" :loading="loading" @click="fetchAIRecommendations">
<el-icon style="margin-right: 5px"><MagicStick /></el-icon>
获取 AI 推荐
</el-button>
</div>
<PinnedProject ref="pinnedProjectRef" @pin-changed="handlePinChanged" />
<el-row :gutter="20">
<el-row :gutter="20" class="ai-section">
<el-col :span="24">
<el-card class="box-card" shadow="hover">
<!-- <template #header> -->
<div class="card-header">
<span>AI 推荐项目</span>
<el-tag type="success">{{ aiRecommendations.length }} 个推荐</el-tag>
</div>
<!-- </template> -->
<div class="card-header">
<span>AI 推荐项目</span>
<span v-if="lastRecommendationTime" class="last-recommendation-time">
生成时间: {{ lastRecommendationTime }}
</span>
<span v-else class="last-recommendation-time text-muted">
暂无推荐时间
</span>
<el-tag type="success">{{ aiRecommendations.length }} 个推荐</el-tag>
</div>
<div v-if="loading" style="text-align: center; padding: 40px;">
<el-icon class="is-loading" :size="30"><Loading /></el-icon>
<p style="margin-top: 10px; color: #909399;">AI 正在分析中...</p>
@@ -27,8 +34,8 @@
<p style="font-size: 12px; margin-top: 5px;">点击上方按钮获取 AI 推荐</p>
</div>
<div v-else>
<el-table :data="aiRecommendations" style="width: 100%" size="small">
<el-table-column label="Pin" width="60" align="center">
<el-table :data="aiRecommendations" style="width: 100%" size="small" class="ai-table">
<el-table-column label="Pin" width="45" align="center">
<template #default="scope">
<el-icon
:style="{
@@ -42,20 +49,29 @@
</el-icon>
</template>
</el-table-column>
<el-table-column prop="title" label="项目名称">
<el-table-column label="项目名称" min-width="150">
<template #default="scope">
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
<a
v-if="scope.row.url"
:href="scope.row.url"
target="_blank"
class="project-link"
@click.stop
>
{{ scope.row.title }}
</a>
<span v-else>{{ scope.row.title }}</span>
</template>
</el-table-column>
<el-table-column prop="source" label="来源" width="200" />
<el-table-column prop="publishDate" label="发布日期" width="180">
<el-table-column prop="source" label="来源" min-width="80" />
<el-table-column prop="publishDate" label="发布日期" width="95">
<template #default="scope">
{{ formatDate(scope.row.publishDate) }}
</template>
</el-table-column>
<el-table-column prop="confidence" label="推荐度" width="120">
<el-table-column prop="confidence" label="推荐度" width="70">
<template #default="scope">
<el-tag :type="getConfidenceType(scope.row.confidence)">
<el-tag :type="getConfidenceType(scope.row.confidence)" size="small">
{{ scope.row.confidence }}%
</el-tag>
</template>
@@ -65,26 +81,28 @@
</el-card>
</el-col>
</el-row>
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 10px;">
<h3 style="margin: 0;">选择日期范围</h3>
<div style="display: flex; gap: 10px;">
<div class="filter-section">
<h3 class="filter-title">选择日期范围</h3>
<div class="filter-controls">
<el-date-picker
v-model="dateRange"
type="daterange"
range-separator="To"
start-placeholder="Start Date"
end-placeholder="End Date"
range-separator=""
start-placeholder="开始日期"
end-placeholder="结束日期"
format="YYYY-MM-DD"
value-format="YYYY-MM-DD"
clearable
style="width: 240px;"
class="date-picker"
/>
<el-button type="primary" @click="setLast3Days">3天</el-button>
<el-button type="primary" @click="setLast7Days">7</el-button>
<el-button type="success" :loading="bidsLoading" @click="fetchBidsByDateRange">
<el-icon style="margin-right: 5px"><List /></el-icon>
列出时间范围内所有工程
</el-button>
<div class="button-group">
<el-button type="primary" @click="setLast3Days">3</el-button>
<el-button type="primary" @click="setLast7Days">7天</el-button>
<el-button type="success" :loading="bidsLoading" @click="fetchBidsByDateRange">
<el-icon style="margin-right: 5px"><List /></el-icon>
列出时间范围内所有工程
</el-button>
</div>
</div>
</div>
<el-row v-if="showAllBids" :gutter="20" style="margin-top: 20px;">
@@ -105,14 +123,23 @@
<p style="margin-top: 10px;">该时间范围内暂无工程</p>
</div>
<div v-else>
<el-table :data="bidsByDateRange" style="width: 100%" size="small">
<el-table-column prop="title" label="项目名称">
<el-table :data="bidsByDateRange" style="width: 100%" size="small" class="bids-table">
<el-table-column label="项目名称" min-width="150">
<template #default="scope">
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
<a
v-if="scope.row.url"
:href="scope.row.url"
target="_blank"
class="project-link"
@click.stop
>
{{ scope.row.title }}
</a>
<span v-else>{{ scope.row.title }}</span>
</template>
</el-table-column>
<el-table-column prop="source" label="来源" width="200" />
<el-table-column prop="publishDate" label="发布日期" width="180">
<el-table-column prop="source" label="来源" width="220" />
<el-table-column prop="publishDate" label="发布日期" width="95">
<template #default="scope">
{{ formatDate(scope.row.publishDate) }}
</template>
@@ -127,9 +154,9 @@
<script setup lang="ts">
import { ref, watch } from 'vue'
import axios from 'axios'
import api from '../utils/api'
import { ElMessage } from 'element-plus'
import { MagicStick, Loading, InfoFilled, List, ArrowDown, Paperclip } from '@element-plus/icons-vue'
import { MagicStick, Loading, InfoFilled, List, Paperclip } from '@element-plus/icons-vue'
import PinnedProject from './PinnedProject.vue'
@@ -154,6 +181,7 @@ const dateRange = ref<[string, string] | null>(null)
const showAllBids = ref(false)
const bidsLoading = ref(false)
const bidsByDateRange = ref<any[]>([])
const lastRecommendationTime = ref<string | null>(null)
// 从 localStorage 加载保存的日期范围
const loadSavedDateRange = () => {
@@ -175,13 +203,16 @@ watch(dateRange, (newDateRange) => {
// 从数据库加载最新的 AI 推荐
const loadLatestRecommendations = async () => {
try {
const response = await axios.get('/api/ai/latest-recommendations')
const recommendations = response.data
const response = await api.get('/api/ai/latest-recommendations')
const { recommendations, generatedAt } = response.data
// 更新生成时间
lastRecommendationTime.value = generatedAt
// 获取所有置顶的项目
const pinnedResponse = await axios.get('/api/bids/pinned')
const pinnedResponse = await api.get('/api/bids/pinned')
const pinnedTitles = new Set(pinnedResponse.data.map((b: any) => b.title))
// 更新每个推荐项目的 pin 状态
aiRecommendations.value = recommendations.map((rec: any) => ({
...rec,
@@ -240,7 +271,7 @@ const fetchAIRecommendations = async () => {
}))
// 调用后端 API
const response = await axios.post('/api/ai/recommendations', {
const response = await api.post('/api/ai/recommendations', {
bids: bidsData
})
@@ -267,10 +298,13 @@ const fetchAIRecommendations = async () => {
aiRecommendations.value = recommendations
// 保存推荐结果到数据库
await axios.post('/api/ai/save-recommendations', {
await api.post('/api/ai/save-recommendations', {
recommendations
})
// 更新时间戳
lastRecommendationTime.value = new Date().toLocaleString('zh-CN', { hour12: false })
ElMessage.success('AI 推荐获取成功')
} catch (error: any) {
ElMessage.error('获取 AI 推荐失败')
@@ -301,7 +335,7 @@ const fetchBidsByDateRange = async () => {
params.endDate = endDate
}
const response = await axios.get('/api/bids/by-date-range', { params })
const response = await api.get('/api/bids/by-date-range', { params })
bidsByDateRange.value = response.data
ElMessage.success(`获取成功,共 ${response.data.length} 个工程`)
} catch (error: any) {
@@ -344,7 +378,7 @@ const handlePinChanged = async (title: string) => {
const togglePin = async (item: AIRecommendation) => {
try {
const newPinStatus = !item.pin
await axios.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: newPinStatus })
await api.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: newPinStatus })
item.pin = newPinStatus
ElMessage.success(newPinStatus ? '已置顶' : '已取消置顶')
// 刷新 PinnedProject 组件的数据
@@ -358,18 +392,98 @@ const togglePin = async (item: AIRecommendation) => {
</script>
<style scoped>
.dashboard-ai-container {
padding: 0;
}
.dashboard-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 16px;
flex-wrap: wrap;
gap: 10px;
}
.dashboard-title-wrapper {
display: flex;
flex-direction: column;
gap: 4px;
}
.dashboard-title {
margin: 0;
font-size: 1.25rem;
}
.last-recommendation-time {
font-size: 0.75rem;
color: var(--el-text-color-secondary);
}
.last-recommendation-time.text-muted {
color: var(--el-text-color-placeholder);
}
.ai-section {
margin-top: 16px;
}
.card-header {
display: flex;
justify-content: space-between;
align-items: center;
}
.box-card{
.box-card {
margin-top: 10px;
margin-bottom: 10px;
}
.filter-section {
margin-bottom: 16px;
margin-top: 20px;
}
.filter-title {
margin: 0 0 12px 0;
font-size: 1rem;
}
.filter-controls {
display: flex;
flex-wrap: wrap;
gap: 10px;
align-items: center;
}
.date-picker {
width: 100%;
max-width: 280px;
}
.button-group {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.ai-table,
.bids-table {
width: 100%;
}
.project-link {
color: var(--el-text-color-primary);
text-decoration: none;
cursor: pointer;
transition: color 0.2s;
}
.project-link:hover {
color: var(--el-color-primary);
}
a {
text-decoration: none;
color: inherit;
@@ -378,4 +492,50 @@ a {
a:hover {
color: #409eff;
}
/* 移动端响应式样式 */
@media (max-width: 768px) {
.dashboard-header {
flex-direction: column;
align-items: flex-start;
}
.dashboard-title-wrapper {
flex-direction: column;
gap: 4px;
}
.dashboard-title {
font-size: 1.1rem;
}
.filter-controls {
flex-direction: column;
align-items: stretch;
}
.date-picker {
max-width: 100%;
}
.button-group {
justify-content: center;
}
.ai-table,
.bids-table {
overflow-x: auto;
}
.ai-table :deep(.el-table__cell),
.bids-table :deep(.el-table__cell) {
padding: 6px;
}
.card-header {
flex-direction: column;
align-items: flex-start;
gap: 8px;
}
}
</style>

View File

@@ -1,7 +1,7 @@
<template>
<div>
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px;">
<h2 style="margin: 0;">Dashboard</h2>
<div class="dashboard-container">
<div class="dashboard-header">
<h2 class="dashboard-title">Dashboard</h2>
<el-button type="primary" :loading="crawling" :disabled="isCrawling" @click="handleCrawl">
<el-icon style="margin-right: 5px"><Refresh /></el-icon>
立刻抓取
@@ -9,34 +9,36 @@
</div>
<PinnedProject ref="pinnedProjectRef" @pin-changed="handlePinChanged" />
<el-divider />
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 10px;">
<h3 style="margin: 0;">Today's Bids</h3>
<div style="display: flex; gap: 10px;">
<div class="filter-section">
<h3 class="filter-title">Today's Bids</h3>
<div class="filter-controls">
<el-date-picker
v-model="dateRange"
type="daterange"
range-separator="To"
start-placeholder="Start Date"
end-placeholder="End Date"
range-separator=""
start-placeholder="开始日期"
end-placeholder="结束日期"
format="YYYY-MM-DD"
value-format="YYYY-MM-DD"
clearable
style="width: 240px;"
class="date-picker"
/>
<el-button type="primary" @click="setLast3Days">3天</el-button>
<el-button type="primary" @click="setLast7Days">7天</el-button>
<el-button type="success" :loading="updating" @click="updateBidsByDateRange">
<el-icon style="margin-right: 5px"><Refresh /></el-icon>
更新
</el-button>
<div class="button-group">
<el-button type="primary" @click="setLast3Days">3天</el-button>
<el-button type="primary" @click="setLast7Days">7天</el-button>
<el-button type="success" :loading="updating" @click="updateBidsByDateRange">
<el-icon style="margin-right: 5px"><Refresh /></el-icon>
更新
</el-button>
</div>
<el-select
v-model="selectedKeywords"
multiple
collapse-tags
collapse-tags-tooltip
placeholder="Filter by Keywords"
placeholder="按关键字筛选"
clearable
style="width: 300px;"
class="keyword-select"
>
<el-option
v-for="keyword in keywords"
@@ -47,8 +49,8 @@
</el-select>
</div>
</div>
<el-table :data="filteredTodayBids" v-loading="loading" style="width: 100%">
<el-table-column label="Pin" width="60" align="center">
<el-table :data="filteredTodayBids" v-loading="loading" style="width: 100%" class="bids-table">
<el-table-column label="Pin" width="50" align="center">
<template #default="scope">
<el-icon
:style="{
@@ -62,13 +64,22 @@
</el-icon>
</template>
</el-table-column>
<el-table-column prop="title" label="Title">
<el-table-column label="Title" min-width="150">
<template #default="scope">
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
<a
v-if="scope.row.url"
:href="scope.row.url"
target="_blank"
class="project-link"
@click.stop
>
{{ scope.row.title }}
</a>
<span v-else>{{ scope.row.title }}</span>
</template>
</el-table-column>
<el-table-column prop="source" label="Source" width="220" />
<el-table-column prop="publishDate" label="Date" width="150">
<el-table-column prop="source" label="Source" min-width="90" />
<el-table-column prop="publishDate" label="Date" width="100">
<template #default="scope">{{ formatDate(scope.row.publishDate) }}</template>
</el-table-column>
</el-table>
@@ -77,10 +88,11 @@
<script setup lang="ts">
import { ref, computed, watch } from 'vue'
import axios from 'axios'
import api from '../utils/api'
import { ElMessage } from 'element-plus'
import { Refresh, ArrowDown, Paperclip } from '@element-plus/icons-vue'
import { Refresh, Paperclip } from '@element-plus/icons-vue'
import PinnedProject from './PinnedProject.vue'
import { formatDate } from '../utils/date.util'
interface Props {
todayBids: any[]
@@ -121,7 +133,7 @@ watch(dateRange, (newDateRange) => {
localStorage.setItem('dashboard_dateRange', JSON.stringify(newDateRange))
}, { deep: true })
// 从 localStorage 加载保存的关键字
// 从 localStorage 保存的关键字
const loadSavedKeywords = () => {
const saved = localStorage.getItem('selectedKeywords')
if (saved) {
@@ -160,11 +172,6 @@ watch(dateRange, () => {
}
})
const formatDate = (dateString: string) => {
if (!dateString) return '-'
return new Date(dateString).toLocaleDateString()
}
// 过滤 Today's Bids只显示包含所选关键字的项目并且在日期范围内
const filteredTodayBids = computed(() => {
let result = props.todayBids
@@ -275,7 +282,7 @@ const handleCrawl = async () => {
}
crawling.value = true
try {
await axios.post('/api/crawler/run')
await api.post('/api/crawler/run')
ElMessage.success('Crawl completed successfully')
emit('refresh') // Refresh data after crawl
} catch (error) {
@@ -301,7 +308,7 @@ const handlePinChanged = async (title: string) => {
const togglePin = async (item: any) => {
try {
const newPinStatus = !item.pin
await axios.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: newPinStatus })
await api.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: newPinStatus })
item.pin = newPinStatus
ElMessage.success(newPinStatus ? '已置顶' : '已取消置顶')
// 刷新 PinnedProject 组件的数据
@@ -324,6 +331,71 @@ if (!dateRange.value) {
</script>
<style scoped>
.dashboard-container {
padding: 0;
}
.dashboard-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 16px;
flex-wrap: wrap;
gap: 10px;
}
.dashboard-title {
margin: 0;
font-size: 1.25rem;
}
.filter-section {
margin-bottom: 16px;
}
.filter-title {
margin: 0 0 12px 0;
font-size: 1rem;
}
.filter-controls {
display: flex;
flex-wrap: wrap;
gap: 10px;
align-items: center;
}
.date-picker {
width: 100%;
max-width: 280px;
}
.button-group {
display: flex;
gap: 8px;
flex-wrap: wrap;
}
.keyword-select {
width: 100%;
max-width: 200px;
}
.bids-table {
width: 100%;
}
.project-link {
color: var(--el-text-color-primary);
text-decoration: none;
cursor: pointer;
transition: color 0.2s;
}
.project-link:hover {
color: var(--el-color-primary);
}
.card-header {
display: flex;
justify-content: space-between;
@@ -338,4 +410,41 @@ a {
a:hover {
color: #409eff;
}
/* 移动端响应式样式 */
@media (max-width: 768px) {
.dashboard-header {
flex-direction: column;
align-items: flex-start;
}
.dashboard-title {
font-size: 1.1rem;
}
.filter-controls {
flex-direction: column;
align-items: stretch;
}
.date-picker {
max-width: 100%;
}
.button-group {
justify-content: center;
}
.keyword-select {
max-width: 100%;
}
.bids-table {
overflow-x: auto;
}
.bids-table :deep(.el-table__cell) {
padding: 6px;
}
}
</style>

View File

@@ -1,37 +1,37 @@
<template>
<div>
<div class="card-header" style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px;">
<h2>Keyword Management</h2>
<el-button type="primary" @click="dialogVisible = true">Add Keyword</el-button>
<div class="keywords-container">
<div class="keywords-header">
<h2 class="keywords-title">Keyword Management</h2>
<el-button type="primary" @click="dialogVisible = true">添加关键字</el-button>
</div>
<div v-loading="loading" style="min-height: 200px;">
<div v-loading="loading" style="min-height: 200px;" class="keywords-list">
<el-tag
v-for="keyword in keywords"
:key="keyword.id"
closable
:type="getTagType(keyword.weight)"
@close="handleDeleteKeyword(keyword.id)"
style="margin: 5px;"
class="keyword-tag"
>
{{ keyword.word }}
</el-tag>
<el-empty v-if="keywords.length === 0" description="No keywords" />
<el-empty v-if="keywords.length === 0" description="暂无关键字" />
</div>
<el-dialog v-model="dialogVisible" title="Add Keyword" width="30%">
<el-form :model="form" label-width="120px">
<el-form-item label="Keyword">
<el-input v-model="form.word" />
<el-dialog v-model="dialogVisible" title="添加关键字" width="90%" :style="{ maxWidth: '400px' }">
<el-form :model="form" label-width="80px">
<el-form-item label="关键字">
<el-input v-model="form.word" placeholder="请输入关键字" />
</el-form-item>
<el-form-item label="Weight">
<el-form-item label="权重">
<el-input-number v-model="form.weight" :min="1" :max="5" />
</el-form-item>
</el-form>
<template #footer>
<span class="dialog-footer">
<el-button @click="dialogVisible = false">Cancel</el-button>
<el-button type="primary" @click="handleAddKeyword">Confirm</el-button>
<el-button @click="dialogVisible = false">取消</el-button>
<el-button type="primary" @click="handleAddKeyword">确认</el-button>
</span>
</template>
</el-dialog>
@@ -40,7 +40,7 @@
<script setup lang="ts">
import { ref, reactive } from 'vue'
import axios from 'axios'
import api from '../utils/api'
import { ElMessage } from 'element-plus'
interface Props {
@@ -72,36 +72,86 @@ const getTagType = (weight: number) => {
const handleAddKeyword = async () => {
if (!form.word) {
ElMessage.warning('Please enter a keyword')
ElMessage.warning('请输入关键字')
return
}
try {
await axios.post('/api/keywords', form)
ElMessage.success('Keyword added')
await api.post('/api/keywords', form)
ElMessage.success('关键字添加成功')
dialogVisible.value = false
form.word = ''
form.weight = 1
emit('refresh')
} catch (error) {
ElMessage.error('Failed to add keyword')
ElMessage.error('添加关键字失败')
}
}
const handleDeleteKeyword = async (id: string) => {
try {
await axios.delete(`/api/keywords/${id}`)
ElMessage.success('Keyword deleted')
await api.delete(`/api/keywords/${id}`)
ElMessage.success('关键字删除成功')
emit('refresh')
} catch (error) {
ElMessage.error('Failed to delete keyword')
ElMessage.error('删除关键字失败')
}
}
</script>
<style scoped>
.card-header {
.keywords-container {
padding: 0;
}
.keywords-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 16px;
flex-wrap: wrap;
gap: 10px;
}
.keywords-title {
margin: 0;
font-size: 1.25rem;
}
.keywords-list {
min-height: 200px;
}
.keyword-tag {
margin: 5px;
}
a {
text-decoration: none;
color: inherit;
}
a:hover {
color: #409eff;
}
/* 移动端响应式样式 */
@media (max-width: 768px) {
.keywords-header {
flex-direction: column;
align-items: flex-start;
}
.keywords-title {
font-size: 1.1rem;
}
.keyword-tag {
margin: 4px;
font-size: 12px;
}
.keywords-list {
min-height: 150px;
}
}
</style>

View File

@@ -2,7 +2,7 @@
<el-card class="box-card" shadow="hover">
<template #header>
<div class="card-header">
<span>Pinned</span>
<span class="card-title">Pinned</span>
<el-tag type="danger">{{ pinnedBids.length }} 个置顶</el-tag>
</div>
</template>
@@ -15,8 +15,8 @@
<p style="margin-top: 10px;">暂无置顶项目</p>
</div>
<div v-else>
<el-table :data="pinnedBids" style="width: 100%" size="small">
<el-table-column label="Pin" width="60" align="center">
<el-table :data="pinnedBids" style="width: 100%" size="small" class="pinned-table">
<el-table-column label="Pin" width="45" align="center">
<template #default="scope">
<el-icon
:style="{
@@ -30,15 +30,24 @@
</el-icon>
</template>
</el-table-column>
<el-table-column prop="title" label="项目名称">
<el-table-column label="项目名称" min-width="150">
<template #default="scope">
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
<a
v-if="scope.row.url"
:href="scope.row.url"
target="_blank"
class="project-link"
@click.stop
>
{{ scope.row.title }}
</a>
<span v-else>{{ scope.row.title }}</span>
</template>
</el-table-column>
<el-table-column prop="source" label="来源" width="200" />
<el-table-column prop="publishDate" label="发布日期" width="180">
<el-table-column prop="source" label="来源" min-width="100" />
<el-table-column prop="publishDate" label="发布日期" width="95">
<template #default="scope">
{{ formatDate(scope.row.publishDate) }}
{{ formatSimpleDate(scope.row.publishDate) }}
</template>
</el-table-column>
</el-table>
@@ -48,9 +57,10 @@
<script setup lang="ts">
import { ref, onMounted } from 'vue'
import axios from 'axios'
import api from '../utils/api'
import { ElMessage } from 'element-plus'
import { Loading, InfoFilled, Paperclip } from '@element-plus/icons-vue'
import { formatSimpleDate } from '../utils/date.util'
const emit = defineEmits<{
pinChanged: [title: string]
@@ -63,7 +73,7 @@ const pinnedLoading = ref(false)
const loadPinnedBids = async () => {
pinnedLoading.value = true
try {
const response = await axios.get('/api/bids/pinned')
const response = await api.get('/api/bids/pinned')
pinnedBids.value = response.data
} catch (error) {
console.error('Failed to load pinned bids:', error)
@@ -75,7 +85,7 @@ const loadPinnedBids = async () => {
// 切换置顶列表的 Pin 状态
const togglePin = async (item: any) => {
try {
await axios.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: false })
await api.patch(`/api/bids/${encodeURIComponent(item.title)}/pin`, { pin: false })
const index = pinnedBids.value.findIndex(b => b.title === item.title)
if (index !== -1) {
pinnedBids.value.splice(index, 1)
@@ -88,16 +98,6 @@ const togglePin = async (item: any) => {
}
}
// 格式化日期,只显示年月日
const formatDate = (dateStr: string) => {
if (!dateStr) return ''
const date = new Date(dateStr)
const year = date.getFullYear()
const month = String(date.getMonth() + 1).padStart(2, '0')
const day = String(date.getDate()).padStart(2, '0')
return `${year}-${month}-${day}`
}
// 初始化时加载置顶项目
onMounted(() => {
loadPinnedBids()
@@ -116,11 +116,31 @@ defineExpose({
align-items: center;
}
.card-title {
font-size: 1rem;
font-weight: bold;
}
.box-card {
margin-top: 10px;
margin-bottom: 10px;
}
.pinned-table {
width: 100%;
}
.project-link {
color: var(--el-text-color-primary);
text-decoration: none;
cursor: pointer;
transition: color 0.2s;
}
.project-link:hover {
color: var(--el-color-primary);
}
a {
text-decoration: none;
color: inherit;
@@ -129,4 +149,30 @@ a {
a:hover {
color: #409eff;
}
/* 移动端响应式样式 */
@media (max-width: 768px) {
.card-header {
flex-direction: column;
align-items: flex-start;
gap: 8px;
}
.card-title {
font-size: 0.9rem;
}
.pinned-table {
overflow-x: auto;
}
.pinned-table :deep(.el-table__cell) {
padding: 6px;
}
.box-card {
margin-top: 8px;
margin-bottom: 8px;
}
}
</style>

View File

@@ -1,5 +1,6 @@
@import "tailwindcss";
/* 基础样式 */
:root {
font-family: system-ui, Avenir, Helvetica, Arial, sans-serif;
line-height: 1.5;
@@ -15,6 +16,19 @@
-moz-osx-font-smoothing: grayscale;
}
body {
margin: 0;
min-width: 320px;
min-height: 100vh;
}
#app {
width: 100%;
max-width: 1280px;
margin: 0 auto;
padding: 0;
}
a {
font-weight: 500;
color: #646cff;
@@ -24,19 +38,6 @@ a:hover {
color: #535bf2;
}
body {
margin: 0;
display: flex;
place-items: center;
min-width: 320px;
min-height: 100vh;
}
h1 {
font-size: 3.2em;
line-height: 1.1;
}
button {
border-radius: 8px;
border: 1px solid transparent;
@@ -56,17 +57,6 @@ button:focus-visible {
outline: 4px auto -webkit-focus-ring-color;
}
.card {
padding: 2em;
}
#app {
max-width: 1280px;
margin: 0 auto;
padding: 2rem;
text-align: center;
}
@media (prefers-color-scheme: light) {
:root {
color: #213547;

80
frontend/src/utils/api.ts Normal file
View File

@@ -0,0 +1,80 @@
import axios, { type InternalAxiosRequestConfig, type AxiosError } from 'axios';
/**
* 认证相关工具函数
*/
/**
* 设置 Basic Auth 凭证到 localStorage
*/
export const setAuthCredentials = (username: string, password: string) => {
const credentials = btoa(`${username}:${password}`);
localStorage.setItem('authCredentials', credentials);
};
/**
* 从 localStorage 获取 Basic Auth 凭证
*/
export const getAuthCredentials = (): string | null => {
return localStorage.getItem('authCredentials');
};
/**
* 清除认证凭证
*/
export const clearAuthCredentials = () => {
localStorage.removeItem('authCredentials');
};
/**
* 检查是否已登录
*/
export const isAuthenticated = (): boolean => {
return !!localStorage.getItem('authCredentials');
};
/**
* API配置
* 配置axios实例设置baseURL和请求拦截器
*/
const api = axios.create({
baseURL:
(import.meta.env.VITE_API_BASE_URL as string) || 'http://localhost:3000', // 设置后端服务地址
timeout: 120000, // 请求超时时间120秒
});
// 请求拦截器
api.interceptors.request.use(
(config: InternalAxiosRequestConfig) => {
// 添加 Basic Auth 头
const credentials = getAuthCredentials();
if (credentials && config.headers) {
config.headers['Authorization'] = `Basic ${credentials}`;
}
return config;
},
(error: AxiosError) => {
return Promise.reject(error);
},
);
// 响应拦截器
api.interceptors.response.use(
(response) => {
return response;
},
(error: AxiosError) => {
// 处理 401 未授权错误
if (error.response?.status === 401) {
// 清除无效的凭证
clearAuthCredentials();
// 触发自定义事件,通知应用需要重新登录
window.dispatchEvent(new CustomEvent('auth-required'));
}
console.error('API请求错误:', error);
return Promise.reject(error);
},
);
export default api;

View File

@@ -0,0 +1,69 @@
/**
* 日期格式化工具函数
* 统一处理东八区(Asia/Shanghai)时间显示
*/
/**
* 格式化日期为 YYYY-MM-DD 格式
* @param dateStr 日期字符串或Date对象
* @returns 格式化后的日期字符串
*/
export function formatDate(dateStr: string | null | undefined): string {
if (!dateStr) return '-'
const date = new Date(dateStr)
if (isNaN(date.getTime())) return '-'
return date.toLocaleDateString('zh-CN', {
year: 'numeric',
month: '2-digit',
day: '2-digit',
timeZone: 'Asia/Shanghai'
}).replace(/\//g, '-')
}
/**
* 格式化日期时间为 YYYY-MM-DD HH:mm 格式
* @param dateStr 日期字符串或Date对象
* @returns 格式化后的日期时间字符串
*/
export function formatDateTime(dateStr: string | null | undefined): string {
if (!dateStr) return '-'
// 如果时间字符串已经包含时区信息(如 +08:00说明已经是正确的北京时间
// 直接从字符串中提取日期和时间部分,避免时区转换问题
const timezoneMatch = dateStr.match(/^(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}):\d{2}(?:\.\d{3})?[+-]\d{2}:\d{2}$/)
if (timezoneMatch) {
// 时间字符串已包含时区,直接提取日期和时间
return `${timezoneMatch[1]} ${timezoneMatch[2]}`
}
// 没有时区信息或格式不匹配,使用 Date 对象解析并转换
const date = new Date(dateStr)
if (isNaN(date.getTime())) return '-'
return date.toLocaleString('zh-CN', {
year: 'numeric',
month: '2-digit',
day: '2-digit',
hour: '2-digit',
minute: '2-digit',
timeZone: 'Asia/Shanghai'
}).replace(/\//g, '-')
}
/**
* 格式化日期为简洁的 YYYY-MM-DD 格式(用于置顶项目等)
* @param dateStr 日期字符串或Date对象
* @returns 格式化后的日期字符串
*/
export function formatSimpleDate(dateStr: string | null | undefined): string {
if (!dateStr) return ''
const date = new Date(dateStr)
if (isNaN(date.getTime())) return ''
const year = date.getFullYear()
const month = String(date.getMonth() + 1).padStart(2, '0')
const day = String(date.getDate()).padStart(2, '0')
return `${year}-${month}-${day}`
}

View File

@@ -3,6 +3,7 @@ import vue from '@vitejs/plugin-vue'
// https://vite.dev/config/
export default defineConfig({
base: './',
plugins: [vue()],
server: {
proxy: {

View File

@@ -0,0 +1 @@
VITE_API_BASE_URL=http://localhost:3300

View File

@@ -0,0 +1 @@
VITE_API_BASE_URL=http://139.180.190.142:3300/

129
ionic-app/README.md Normal file
View File

@@ -0,0 +1,129 @@
# 投标项目查看器 - Ionic 版本
基于 Ionic + Vue 3 + TypeScript + Tailwind CSS 的移动端应用,适配 Android 平台。
## 功能特性
- **AI 推荐** - 显示 AI 推荐的投标项目,带推荐度标签
- **爬虫信息** - 显示各爬虫来源的统计信息和状态
- **置顶项目** - 显示用户置顶的投标项目
- **下拉刷新** - 支持下拉刷新数据
## 技术栈
- **Ionic 8** - 移动端 UI 框架
- **Vue 3** - 前端框架
- **TypeScript** - 类型安全
- **Tailwind CSS** - 样式框架
- **Capacitor** - 原生功能集成
- **Axios** - HTTP 客户端
## 安装依赖
```bash
npm install
```
## 开发
启动开发服务器:
```bash
npm run dev
```
应用将在 http://localhost:8100 运行。
## 构建
构建生产版本:
```bash
npm run build
```
## Android 构建
### 添加 Android 平台
```bash
npm install -g @capacitor/cli
npx cap init
npx cap add android
```
### 同步资源
```bash
npm run build
npx cap sync
```
### 在 Android Studio 中打开
```bash
npx cap open android
```
## 环境变量
创建 `.env.development` 文件:
```env
VITE_API_BASE_URL=http://localhost:3001
```
创建 `.env.production` 文件:
```env
VITE_API_BASE_URL=https://your-production-api.com
```
## 项目结构
```
ionic-app/
├── src/
│ ├── components/ # 组件
│ │ ├── AiRecommendations.vue
│ │ ├── CrawlInfo.vue
│ │ └── PinnedBids.vue
│ ├── pages/ # 页面
│ │ └── HomePage.vue
│ ├── router/ # 路由
│ │ └── index.ts
│ ├── theme/ # 主题
│ │ └── variables.css
│ ├── types/ # 类型定义
│ │ └── index.ts
│ ├── utils/ # 工具函数
│ │ └── api.ts
│ ├── App.vue # 根组件
│ ├── main.ts # 入口文件
│ └── vite-env.d.ts # Vite 类型声明
├── public/ # 静态资源
├── .env.development # 开发环境变量
├── .env.production # 生产环境变量
├── capacitor.config.ts # Capacitor 配置
├── ionic.config.json # Ionic 配置
├── package.json
├── tailwind.config.js # Tailwind 配置
├── tsconfig.json # TypeScript 配置
└── vite.config.ts # Vite 配置
```
## API 接口
| 接口 | 方法 | 说明 |
|------|------|------|
| `/api/ai/latest-recommendations` | GET | 获取最新 AI 推荐 |
| `/api/bids/crawl-info-stats` | GET | 获取爬虫统计信息 |
| `/api/bids/pinned` | GET | 获取置顶项目 |
| `/api/bids/{title}/pin` | PATCH | 切换置顶状态 |
## 注意事项
1. 确保后端 API 服务正在运行
2. 开发环境默认 API 地址为 `http://localhost:3001`
3. 首次运行需要安装所有依赖
4. Android 构建需要安装 Android Studio 和 Android SDK

101
ionic-app/android/.gitignore vendored Normal file
View File

@@ -0,0 +1,101 @@
# Using Android gitignore template: https://github.com/github/gitignore/blob/HEAD/Android.gitignore
# Built application files
*.apk
*.aar
*.ap_
*.aab
# Files for the ART/Dalvik VM
*.dex
# Java class files
*.class
# Generated files
bin/
gen/
out/
# Uncomment the following line in case you need and you don't have the release build type files in your app
# release/
# Gradle files
.gradle/
build/
# Local configuration file (sdk path, etc)
local.properties
# Proguard folder generated by Eclipse
proguard/
# Log Files
*.log
# Android Studio Navigation editor temp files
.navigation/
# Android Studio captures folder
captures/
# IntelliJ
*.iml
.idea/workspace.xml
.idea/tasks.xml
.idea/gradle.xml
.idea/assetWizardSettings.xml
.idea/dictionaries
.idea/libraries
# Android Studio 3 in .gitignore file.
.idea/caches
.idea/modules.xml
# Comment next line if keeping position of elements in Navigation Editor is relevant for you
.idea/navEditor.xml
# Keystore files
# Uncomment the following lines if you do not want to check your keystore files in.
#*.jks
#*.keystore
# External native build folder generated in Android Studio 2.2 and later
.externalNativeBuild
.cxx/
# Google Services (e.g. APIs or Firebase)
# google-services.json
# Freeline
freeline.py
freeline/
freeline_project_description.json
# fastlane
fastlane/report.xml
fastlane/Preview.html
fastlane/screenshots
fastlane/test_output
fastlane/readme.md
# Version control
vcs.xml
# lint
lint/intermediates/
lint/generated/
lint/outputs/
lint/tmp/
# lint/reports/
# Android Profiling
*.hprof
# Cordova plugins for Capacitor
capacitor-cordova-android-plugins
# Copied web assets
app/src/main/assets/public
# Generated Config files
app/src/main/assets/capacitor.config.json
app/src/main/assets/capacitor.plugins.json
app/src/main/res/xml/config.xml

2
ionic-app/android/app/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
/build/*
!/build/.npmkeep

View File

@@ -0,0 +1,54 @@
apply plugin: 'com.android.application'
android {
namespace "com.bidding.app"
compileSdk rootProject.ext.compileSdkVersion
defaultConfig {
applicationId "com.bidding.app"
minSdkVersion rootProject.ext.minSdkVersion
targetSdkVersion rootProject.ext.targetSdkVersion
versionCode 1
versionName "1.0"
testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
aaptOptions {
// Files and dirs to omit from the packaged assets dir, modified to accommodate modern web apps.
// Default: https://android.googlesource.com/platform/frameworks/base/+/282e181b58cf72b6ca770dc7ca5f91f135444502/tools/aapt/AaptAssets.cpp#61
ignoreAssetsPattern '!.svn:!.git:!.ds_store:!*.scc:.*:!CVS:!thumbs.db:!picasa.ini:!*~'
}
}
buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
}
}
}
repositories {
flatDir{
dirs '../capacitor-cordova-android-plugins/src/main/libs', 'libs'
}
}
dependencies {
implementation fileTree(include: ['*.jar'], dir: 'libs')
implementation "androidx.appcompat:appcompat:$androidxAppCompatVersion"
implementation "androidx.coordinatorlayout:coordinatorlayout:$androidxCoordinatorLayoutVersion"
implementation "androidx.core:core-splashscreen:$coreSplashScreenVersion"
implementation project(':capacitor-android')
testImplementation "junit:junit:$junitVersion"
androidTestImplementation "androidx.test.ext:junit:$androidxJunitVersion"
androidTestImplementation "androidx.test.espresso:espresso-core:$androidxEspressoCoreVersion"
implementation project(':capacitor-cordova-android-plugins')
}
apply from: 'capacitor.build.gradle'
try {
def servicesJSON = file('google-services.json')
if (servicesJSON.text) {
apply plugin: 'com.google.gms.google-services'
}
} catch(Exception e) {
logger.info("google-services.json not found, google-services plugin not applied. Push Notifications won't work")
}

View File

@@ -0,0 +1,19 @@
// DO NOT EDIT THIS FILE! IT IS GENERATED EACH TIME "capacitor update" IS RUN
android {
compileOptions {
sourceCompatibility JavaVersion.VERSION_17
targetCompatibility JavaVersion.VERSION_17
}
}
apply from: "../capacitor-cordova-android-plugins/cordova.variables.gradle"
dependencies {
implementation project(':capacitor-app')
}
if (hasProperty('postBuildExtras')) {
postBuildExtras()
}

View File

@@ -0,0 +1,29 @@
// Top-level build file where you can add configuration options common to all sub-projects/modules.
buildscript {
repositories {
google()
mavenCentral()
}
dependencies {
classpath 'com.android.tools.build:gradle:8.2.1'
classpath 'com.google.gms:google-services:4.4.0'
// NOTE: Do not place your application dependencies here; they belong
// in the individual module build.gradle files
}
}
apply from: "variables.gradle"
allprojects {
repositories {
google()
mavenCentral()
}
}
task clean(type: Delete) {
delete rootProject.buildDir
}

View File

@@ -0,0 +1,6 @@
// DO NOT EDIT THIS FILE! IT IS GENERATED EACH TIME "capacitor update" IS RUN
include ':capacitor-android'
project(':capacitor-android').projectDir = new File('../node_modules/@capacitor/android/capacitor')
include ':capacitor-app'
project(':capacitor-app').projectDir = new File('../node_modules/@capacitor/app/android')

View File

@@ -0,0 +1,22 @@
# Project-wide Gradle settings.
# IDE (e.g. Android Studio) users:
# Gradle settings configured through the IDE *will override*
# any settings specified in this file.
# For more details on how to configure your build environment visit
# http://www.gradle.org/docs/current/userguide/build_environment.html
# Specifies the JVM arguments used for the daemon process.
# The setting is particularly useful for tweaking memory settings.
org.gradle.jvmargs=-Xmx1536m
# When configured, Gradle will run in incubating parallel mode.
# This option should only be used with decoupled projects. More details, visit
# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
# org.gradle.parallel=true
# AndroidX package structure to make it clearer which packages are bundled with the
# Android operating system, and which are packaged with your app's APK
# https://developer.android.com/topic/libraries/support-library/androidx-rn
android.useAndroidX=true

248
ionic-app/android/gradlew vendored Normal file
View File

@@ -0,0 +1,248 @@
#!/bin/sh
#
# Copyright © 2015-2021 the original authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
##############################################################################
#
# Gradle start up script for POSIX generated by Gradle.
#
# Important for running:
#
# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
# noncompliant, but you have some other compliant shell such as ksh or
# bash, then to run this script, type that shell name before the whole
# command line, like:
#
# ksh Gradle
#
# Busybox and similar reduced shells will NOT work, because this script
# requires all of these POSIX shell features:
# * functions;
# * expansions «$var», «${var}», «${var:-default}», «${var+SET}»,
# «${var#prefix}», «${var%suffix}», and «$( cmd )»;
# * compound commands having a testable exit status, especially «case»;
# * various built-in commands including «command», «set», and «ulimit».
#
# Important for patching:
#
# (2) This script targets any POSIX shell, so it avoids extensions provided
# by Bash, Ksh, etc; in particular arrays are avoided.
#
# The "traditional" practice of packing multiple parameters into a
# space-separated string is a well documented source of bugs and security
# problems, so this is (mostly) avoided, by progressively accumulating
# options in "$@", and eventually passing that to Java.
#
# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
# see the in-line comments for details.
#
# There are tweaks for specific operating systems such as AIX, CygWin,
# Darwin, MinGW, and NonStop.
#
# (3) This script is generated from the Groovy template
# https://github.com/gradle/gradle/blob/HEAD/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
# within the Gradle project.
#
# You can find Gradle at https://github.com/gradle/gradle/.
#
##############################################################################
# Attempt to set APP_HOME
# Resolve links: $0 may be a link
app_path=$0
# Need this for daisy-chained symlinks.
while
APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
[ -h "$app_path" ]
do
ls=$( ls -ld "$app_path" )
link=${ls#*' -> '}
case $link in #(
/*) app_path=$link ;; #(
*) app_path=$APP_HOME$link ;;
esac
done
# This is normally unused
# shellcheck disable=SC2034
APP_BASE_NAME=${0##*/}
APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
# Use the maximum available, or set MAX_FD != -1 to use that value.
MAX_FD=maximum
warn () {
echo "$*"
} >&2
die () {
echo
echo "$*"
echo
exit 1
} >&2
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
case "$( uname )" in #(
CYGWIN* ) cygwin=true ;; #(
Darwin* ) darwin=true ;; #(
MSYS* | MINGW* ) msys=true ;; #(
NONSTOP* ) nonstop=true ;;
esac
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
# Determine the Java command to use to start the JVM.
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
JAVACMD=$JAVA_HOME/jre/sh/java
else
JAVACMD=$JAVA_HOME/bin/java
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
JAVACMD=java
if ! command -v java >/dev/null 2>&1
then
die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
fi
# Increase the maximum file descriptors if we can.
if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
case $MAX_FD in #(
max*)
# In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC3045
MAX_FD=$( ulimit -H -n ) ||
warn "Could not query maximum file descriptor limit"
esac
case $MAX_FD in #(
'' | soft) :;; #(
*)
# In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked.
# shellcheck disable=SC3045
ulimit -n "$MAX_FD" ||
warn "Could not set maximum file descriptor limit to $MAX_FD"
esac
fi
# Collect all arguments for the java command, stacking in reverse order:
# * args from the command line
# * the main class name
# * -classpath
# * -D...appname settings
# * --module-path (only if needed)
# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
# For Cygwin or MSYS, switch paths to Windows format before running java
if "$cygwin" || "$msys" ; then
APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
JAVACMD=$( cygpath --unix "$JAVACMD" )
# Now convert the arguments - kludge to limit ourselves to /bin/sh
for arg do
if
case $arg in #(
-*) false ;; # don't mess with options #(
/?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
[ -e "$t" ] ;; #(
*) false ;;
esac
then
arg=$( cygpath --path --ignore --mixed "$arg" )
fi
# Roll the args list around exactly as many times as the number of
# args, so each arg winds up back in the position where it started, but
# possibly modified.
#
# NB: a `for` loop captures its iteration list before it begins, so
# changing the positional parameters here affects neither the number of
# iterations, nor the values presented in `arg`.
shift # remove old arg
set -- "$@" "$arg" # push replacement arg
done
fi
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Collect all arguments for the java command;
# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
# shell script including quotes and variable substitutions, so put them in
# double quotes to make sure that they get re-expanded; and
# * put everything else in single quotes, so that it's not re-expanded.
set -- \
"-Dorg.gradle.appname=$APP_BASE_NAME" \
-classpath "$CLASSPATH" \
org.gradle.wrapper.GradleWrapperMain \
"$@"
# Stop when "xargs" is not available.
if ! command -v xargs >/dev/null 2>&1
then
die "xargs is not available"
fi
# Use "xargs" to parse quoted args.
#
# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
#
# In Bash we could simply go:
#
# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
# set -- "${ARGS[@]}" "$@"
#
# but POSIX shell has neither arrays nor command substitution, so instead we
# post-process each arg (as a line of input to sed) to backslash-escape any
# character that might be a shell metacharacter, then use eval to reverse
# that process (while maintaining the separation between arguments), and wrap
# the whole thing up as a single "set" statement.
#
# This will of course break if any of these variables contains a newline or
# an unmatched quote.
#
eval "set -- $(
printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
xargs -n1 |
sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
tr '\n' ' '
)" '"$@"'
exec "$JAVACMD" "$@"

92
ionic-app/android/gradlew.bat vendored Normal file
View File

@@ -0,0 +1,92 @@
@rem
@rem Copyright 2015 the original author or authors.
@rem
@rem Licensed under the Apache License, Version 2.0 (the "License");
@rem you may not use this file except in compliance with the License.
@rem You may obtain a copy of the License at
@rem
@rem https://www.apache.org/licenses/LICENSE-2.0
@rem
@rem Unless required by applicable law or agreed to in writing, software
@rem distributed under the License is distributed on an "AS IS" BASIS,
@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
@rem See the License for the specific language governing permissions and
@rem limitations under the License.
@rem
@if "%DEBUG%"=="" @echo off
@rem ##########################################################################
@rem
@rem Gradle startup script for Windows
@rem
@rem ##########################################################################
@rem Set local scope for the variables with windows NT shell
if "%OS%"=="Windows_NT" setlocal
set DIRNAME=%~dp0
if "%DIRNAME%"=="" set DIRNAME=.
@rem This is normally unused
set APP_BASE_NAME=%~n0
set APP_HOME=%DIRNAME%
@rem Resolve any "." and ".." in APP_HOME to make it shorter.
for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"
@rem Find java.exe
if defined JAVA_HOME goto findJavaFromJavaHome
set JAVA_EXE=java.exe
%JAVA_EXE% -version >NUL 2>&1
if %ERRORLEVEL% equ 0 goto execute
echo.
echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
echo.
echo Please set the JAVA_HOME variable in your environment to match the
echo location of your Java installation.
goto fail
:findJavaFromJavaHome
set JAVA_HOME=%JAVA_HOME:"=%
set JAVA_EXE=%JAVA_HOME%/bin/java.exe
if exist "%JAVA_EXE%" goto execute
echo.
echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
echo.
echo Please set the JAVA_HOME variable in your environment to match the
echo location of your Java installation.
goto fail
:execute
@rem Setup the command line
set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
@rem Execute Gradle
"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %*
:end
@rem End local scope for the variables with windows NT shell
if %ERRORLEVEL% equ 0 goto mainEnd
:fail
rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
rem the _cmd.exe /c_ return code!
set EXIT_CODE=%ERRORLEVEL%
if %EXIT_CODE% equ 0 set EXIT_CODE=1
if not ""=="%GRADLE_EXIT_CONSOLE%" exit %EXIT_CODE%
exit /b %EXIT_CODE%
:mainEnd
if "%OS%"=="Windows_NT" endlocal
:omega

View File

@@ -0,0 +1,5 @@
include ':app'
include ':capacitor-cordova-android-plugins'
project(':capacitor-cordova-android-plugins').projectDir = new File('./capacitor-cordova-android-plugins/')
apply from: 'capacitor.settings.gradle'

View File

@@ -0,0 +1,16 @@
ext {
minSdkVersion = 22
compileSdkVersion = 34
targetSdkVersion = 34
androidxActivityVersion = '1.8.0'
androidxAppCompatVersion = '1.6.1'
androidxCoordinatorLayoutVersion = '1.2.0'
androidxCoreVersion = '1.12.0'
androidxFragmentVersion = '1.6.2'
coreSplashScreenVersion = '1.0.1'
androidxWebkitVersion = '1.9.0'
junitVersion = '4.13.2'
androidxJunitVersion = '1.1.5'
androidxEspressoCoreVersion = '3.5.1'
cordovaAndroidVersion = '10.1.1'
}

View File

@@ -0,0 +1,25 @@
{
"appId": "com.bidding.app",
"appName": "BiddingApp",
"webDir": "dist",
"server": {
"androidScheme": "https"
},
"plugins": {
"SplashScreen": {
"launchShowDuration": 0,
"launchAutoHide": true,
"backgroundColor": "#3880ff",
"androidSplashResourceName": "splash",
"androidScaleType": "CENTER_CROP",
"showSpinner": true,
"androidSpinnerStyle": "large",
"iosSpinnerStyle": "small",
"spinnerColor": "#3880ff",
"splashFullScreen": true,
"splashImmersive": true,
"layoutName": "launch_screen",
"useDialog": true
}
}
}

29
ionic-app/index.html Normal file
View File

@@ -0,0 +1,29 @@
<!DOCTYPE html>
<html lang="zh-CN">
<head>
<meta charset="utf-8" />
<title>投标项目查看器</title>
<base href="/" />
<meta name="color-scheme" content="light dark" />
<meta
name="viewport"
content="viewport-fit=cover, width=device-width, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no"
/>
<meta name="format-detection" content="telephone=no" />
<meta name="msapplication-tap-highlight" content="no" />
<link rel="icon" type="image/png" href="/favicon.ico" />
<!-- add to homescreen for ios -->
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
</head>
<body>
<div id="app"></div>
<div id="api-footer" style="position: fixed; bottom: 0; left: 0; right: 0; z-index: 99999; background: var(--ion-toolbar-background, #f8f9fa); padding: 10px; text-align: center; font-size: 12px; color: var(--ion-text-color, #333); border-top: 1px solid #ddd;"></div>
<script type="module" src="/src/main.ts"></script>
</body>
</html>

View File

@@ -0,0 +1,7 @@
{
"name": "bidding-looker-ionic",
"integrations": {
"capacitor": {}
},
"type": "vue"
}

36
ionic-app/package.json Normal file
View File

@@ -0,0 +1,36 @@
{
"name": "bidding-looker-ionic",
"version": "1.0.0",
"description": "投标项目查看器 - Ionic 版本",
"type": "module",
"scripts": {
"dev": "vite",
"build": "vite build",
"preview": "vite preview",
"ionic:build": "npm run build",
"ionic:serve": "vite --port 8100 --host",
"cap:sync": "capacitor sync",
"cap:open:android": "capacitor open android"
},
"dependencies": {
"@capacitor/android": "^6.1.2",
"@capacitor/app": "^6.0.1",
"@capacitor/core": "^6.0.0",
"@ionic/vue": "^8.3.2",
"@ionic/vue-router": "^8.3.2",
"axios": "^1.7.9",
"ionicons": "^7.4.0",
"vue": "^3.5.13",
"vue-router": "^4.5.0"
},
"devDependencies": {
"@capacitor/cli": "^6.0.0",
"@vitejs/plugin-vue": "^5.2.1",
"autoprefixer": "^10.4.20",
"postcss": "^8.4.49",
"tailwindcss": "^3.4.17",
"typescript": "^5.7.2",
"vite": "^6.0.3",
"vue-tsc": "^2.1.10"
}
}

View File

@@ -0,0 +1,6 @@
export default {
plugins: {
tailwindcss: {},
autoprefixer: {},
},
}

31
ionic-app/src/App.vue Normal file
View File

@@ -0,0 +1,31 @@
<script setup lang="ts">
import { IonApp, IonRouterOutlet } from '@ionic/vue'
import { onMounted } from 'vue'
const apiHost = import.meta.env.VITE_API_BASE_URL || 'http://localhost:3001'
onMounted(() => {
const footer = document.getElementById('api-footer')
if (footer) {
footer.textContent = `连接: ${apiHost}`
}
})
</script>
<template>
<IonApp>
<IonRouterOutlet />
</IonApp>
</template>
<style>
* {
box-sizing: border-box;
}
body {
margin: 0;
padding: 0;
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, 'Helvetica Neue', Arial, sans-serif;
}
</style>

View File

@@ -0,0 +1,219 @@
<script setup lang="ts">
import { ref, onMounted } from 'vue'
import { IonCard, IonCardHeader, IonCardSubtitle, IonCardTitle, IonCardContent, IonChip, IonLabel, IonSpinner, IonRefresher, IonRefresherContent, RefresherEventDetail } from '@ionic/vue'
import { getAiRecommendations, togglePin } from '@/utils/api'
import type { AiRecommendation } from '@/types'
const recommendations = ref<AiRecommendation[]>([])
const loading = ref<boolean>(true)
const error = ref<string>('')
const loadRecommendations = async () => {
try {
loading.value = true
error.value = ''
const items = await getAiRecommendations()
recommendations.value = items || []
} catch (err: any) {
error.value = `加载失败: ${err.message || err}`
console.error('加载 AI 推荐失败:', err)
} finally {
loading.value = false
}
}
const getConfidenceColor = (confidence: number): string => {
if (confidence >= 80) return '#27ae60'
if (confidence >= 60) return '#f39c12'
return '#e74c3c'
}
const getConfidenceLabel = (confidence: number): string => {
if (confidence >= 80) return '高'
if (confidence >= 60) return '中'
return '低'
}
const handleRefresh = async (event: CustomEvent<RefresherEventDetail>) => {
await loadRecommendations()
const target = event.target as any
if (target && target.complete) {
target.complete()
}
}
const handleTogglePin = async (item: AiRecommendation) => {
try {
const newPinStatus = !item.pin
await togglePin(item.title, newPinStatus)
item.pin = newPinStatus
} catch (err) {
console.error('切换置顶状态失败:', err)
}
}
const formatDate = (dateStr?: string): string => {
if (!dateStr) return ''
const date = new Date(dateStr)
const year = date.getFullYear()
const month = String(date.getMonth() + 1).padStart(2, '0')
const day = String(date.getDate()).padStart(2, '0')
return `${year}-${month}-${day}`
}
onMounted(() => {
loadRecommendations()
})
defineExpose({
loadRecommendations
})
</script>
<template>
<IonRefresher slot="fixed" @ionRefresh="handleRefresh">
<IonRefresherContent></IonRefresherContent>
</IonRefresher>
<div v-if="loading" class="loading-container">
<IonSpinner name="crescent" />
<p class="loading-text">加载中...</p>
</div>
<div v-else-if="error" class="error-container">
<p class="error-text">{{ error }}</p>
</div>
<div v-else-if="recommendations.length === 0" class="empty-container">
<p class="empty-text">暂无推荐项目</p>
</div>
<div v-else class="recommendation-list">
<IonCard v-for="item in recommendations" :key="item.id" class="recommendation-card">
<IonCardHeader>
<div class="card-header">
<IonChip :style="{ backgroundColor: getConfidenceColor(item.confidence) }" class="confidence-chip">
<IonLabel class="confidence-label">
{{ getConfidenceLabel(item.confidence) }} ({{ item.confidence }}%)
</IonLabel>
</IonChip>
<IonCardSubtitle class="date-text">{{ formatDate(item.publishDate) }}</IonCardSubtitle>
</div>
<IonCardTitle class="title-text">{{ item.title }}</IonCardTitle>
</IonCardHeader>
<IonCardContent>
<div class="card-footer">
<span class="source-text">{{ item.source }}</span>
<button
@click="handleTogglePin(item)"
:class="['pin-button', { pinned: item.pin }]"
>
{{ item.pin ? '已置顶' : '置顶' }}
</button>
</div>
</IonCardContent>
</IonCard>
</div>
</template>
<style scoped>
.loading-container,
.error-container,
.empty-container {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 40px 20px;
text-align: center;
}
.loading-text,
.error-text,
.empty-text {
margin-top: 16px;
color: #666;
font-size: 14px;
}
.error-text {
color: #e74c3c;
}
.recommendation-list {
padding: 16px;
display: flex;
flex-direction: column;
gap: 16px;
}
.recommendation-card {
margin: 0;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1);
}
.card-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 8px;
}
.confidence-chip {
color: #fff;
padding: 4px 12px;
border-radius: 16px;
font-size: 12px;
font-weight: 500;
}
.confidence-label {
color: #fff;
font-size: 12px;
}
.date-text {
font-size: 12px;
color: #999;
}
.title-text {
font-size: 16px;
font-weight: 600;
color: #333;
line-height: 1.4;
}
.card-footer {
display: flex;
justify-content: space-between;
align-items: center;
margin-top: 8px;
}
.source-text {
font-size: 12px;
color: #666;
}
.pin-button {
padding: 6px 16px;
border: 1px solid #ddd;
border-radius: 16px;
background: #fff;
color: #666;
font-size: 12px;
cursor: pointer;
transition: all 0.2s;
}
.pin-button.pinned {
background: #e74c3c;
color: #fff;
border-color: #e74c3c;
}
.pin-button:active {
transform: scale(0.95);
}
</style>

View File

@@ -0,0 +1,345 @@
<script setup lang="ts">
import { ref, onMounted, computed } from 'vue'
import { IonCard, IonCardHeader, IonCardTitle, IonCardContent, IonBadge, IonSpinner, IonRefresher, IonRefresherContent, IonButton, RefresherEventDetail } from '@ionic/vue'
import { getCrawlInfoStats, crawlSingleSource } from '@/utils/api'
import type { CrawlInfoStat } from '@/types'
const crawlStats = ref<CrawlInfoStat[]>([])
const loading = ref<boolean>(true)
const error = ref<string>('')
const updatingSources = ref<Set<string>>(new Set())
const loadCrawlStats = async () => {
try {
loading.value = true
error.value = ''
const stats = await getCrawlInfoStats()
crawlStats.value = stats || []
} catch (err: any) {
error.value = `加载失败: ${err.message || err}`
console.error('加载爬虫统计信息失败:', err)
} finally {
loading.value = false
}
}
const handleUpdateSource = async (sourceName: string) => {
if (updatingSources.value.has(sourceName)) {
return
}
try {
updatingSources.value.add(sourceName)
await crawlSingleSource(sourceName)
// 更新成功后重新加载统计数据
await loadCrawlStats()
} catch (err: any) {
console.error(`更新 ${sourceName} 失败:`, err)
alert(`更新失败: ${err.message || err}`)
} finally {
updatingSources.value.delete(sourceName)
}
}
const getStatusText = (stat: CrawlInfoStat): string => {
if (stat.error) {
return '出错'
}
if (stat.count > 0) {
return '正常'
}
return '无数据'
}
const getStatusColor = (stat: CrawlInfoStat): string => {
if (stat.error) {
return '#f8d7da'
}
if (stat.count > 0) {
return '#d4edda'
}
return '#e2e3e5'
}
const getStatusTextColor = (stat: CrawlInfoStat): string => {
if (stat.error) {
return '#721c24'
}
if (stat.count > 0) {
return '#155724'
}
return '#383d41'
}
const handleRefresh = async (event: CustomEvent<RefresherEventDetail>) => {
await loadCrawlStats()
const target = event.target as any
if (target && target.complete) {
target.complete()
}
}
const formatDateTime = (dateStr: string | null): string => {
if (!dateStr) return '-'
const date = new Date(dateStr)
const year = date.getFullYear()
const month = String(date.getMonth() + 1).padStart(2, '0')
const day = String(date.getDate()).padStart(2, '0')
const hours = String(date.getHours()).padStart(2, '0')
const minutes = String(date.getMinutes()).padStart(2, '0')
return `${year}-${month}-${day} ${hours}:${minutes}`
}
const totalCount = computed(() => {
return crawlStats.value.reduce((sum: number, item: CrawlInfoStat) => sum + item.count, 0)
})
const activeSources = computed(() => {
return crawlStats.value.filter((item: CrawlInfoStat) => item.count > 0).length
})
const errorSources = computed(() => {
return crawlStats.value.filter((item: CrawlInfoStat) => item.error && item.error.trim()).length
})
onMounted(() => {
loadCrawlStats()
})
defineExpose({
loadCrawlStats
})
</script>
<template>
<IonRefresher slot="fixed" @ionRefresh="handleRefresh">
<IonRefresherContent></IonRefresherContent>
</IonRefresher>
<div v-if="loading" class="loading-container">
<IonSpinner name="crescent" />
<p class="loading-text">加载中...</p>
</div>
<div v-else-if="error" class="error-container">
<p class="error-text">{{ error }}</p>
</div>
<div v-else-if="crawlStats.length === 0" class="empty-container">
<p class="empty-text">暂无爬虫统计信息</p>
</div>
<div v-else>
<IonCard class="summary-card">
<IonCardHeader>
<IonCardTitle>统计摘要</IonCardTitle>
</IonCardHeader>
<IonCardContent>
<div class="summary-grid">
<div class="summary-item">
<span class="summary-label">爬虫来源总数</span>
<span class="summary-value">{{ crawlStats.length }}</span>
</div>
<div class="summary-item">
<span class="summary-label">本次获取总数</span>
<span class="summary-value">{{ totalCount }}</span>
</div>
<div class="summary-item">
<span class="summary-label">有数据来源</span>
<span class="summary-value">{{ activeSources }}</span>
</div>
<div class="summary-item">
<span class="summary-label">出错来源</span>
<span class="summary-value error">{{ errorSources }}</span>
</div>
</div>
</IonCardContent>
</IonCard>
<div class="crawl-list">
<IonCard v-for="stat in crawlStats" :key="stat.source" class="crawl-card">
<IonCardContent class="crawl-card-content">
<div class="crawl-header">
<span class="source-name">{{ stat.source }}</span>
<div class="header-actions">
<IonBadge
:style="{
backgroundColor: getStatusColor(stat),
color: getStatusTextColor(stat)
}"
class="status-badge"
>
{{ getStatusText(stat) }}
</IonBadge>
<IonButton
size="small"
fill="outline"
:disabled="updatingSources.has(stat.source)"
@click="handleUpdateSource(stat.source)"
class="update-button"
>
<span v-if="updatingSources.has(stat.source)">更新中...</span>
<span v-else>更新</span>
</IonButton>
</div>
</div>
<div class="crawl-details">
<div class="detail-item">
<span class="detail-label">本次获取数量:</span>
<span class="detail-value">{{ stat.count }}</span>
</div>
<div class="detail-item">
<span class="detail-label">最近更新时间:</span>
<span class="detail-value">{{ formatDateTime(stat.latestUpdate) }}</span>
</div>
<div class="detail-item">
<span class="detail-label">最新工程时间:</span>
<span class="detail-value">{{ formatDateTime(stat.latestPublishDate) }}</span>
</div>
<div v-if="stat.error && stat.error.trim()" class="detail-item error">
<span class="detail-label">错误信息:</span>
<span class="detail-value">{{ stat.error }}</span>
</div>
</div>
</IonCardContent>
</IonCard>
</div>
</div>
</template>
<style scoped>
.loading-container,
.error-container,
.empty-container {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 40px 20px;
text-align: center;
}
.loading-text,
.error-text,
.empty-text {
margin-top: 16px;
color: #666;
font-size: 14px;
}
.error-text {
color: #e74c3c;
}
.summary-card {
margin: 16px;
}
.summary-grid {
display: grid;
grid-template-columns: repeat(2, 1fr);
gap: 12px;
}
.summary-item {
display: flex;
flex-direction: column;
align-items: center;
padding: 12px;
background: #f8f9fa;
border-radius: 8px;
}
.summary-label {
font-size: 12px;
color: #666;
margin-bottom: 4px;
}
.summary-value {
font-size: 20px;
font-weight: 600;
color: #333;
}
.summary-value.error {
color: #e74c3c;
}
.crawl-list {
padding: 0 16px 16px;
display: flex;
flex-direction: column;
gap: 12px;
}
.crawl-card {
margin: 0;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1);
}
.crawl-card-content {
padding: 16px;
}
.crawl-header {
display: flex;
justify-content: space-between;
align-items: center;
margin-bottom: 12px;
}
.source-name {
font-size: 14px;
font-weight: 600;
color: #333;
}
.header-actions {
display: flex;
align-items: center;
gap: 8px;
}
.status-badge {
padding: 4px 12px;
border-radius: 12px;
font-size: 12px;
font-weight: 500;
}
.update-button {
--padding-start: 8px;
--padding-end: 8px;
--padding-top: 4px;
--padding-bottom: 4px;
font-size: 12px;
height: 28px;
}
.crawl-details {
display: flex;
flex-direction: column;
gap: 8px;
}
.detail-item {
display: flex;
justify-content: space-between;
align-items: center;
font-size: 12px;
}
.detail-item.error {
color: #e74c3c;
}
.detail-label {
color: #666;
}
.detail-value {
color: #333;
font-weight: 500;
}
</style>

View File

@@ -0,0 +1,199 @@
<script setup lang="ts">
import { ref, onMounted } from 'vue'
import { IonCard, IonCardHeader, IonCardTitle, IonCardContent, IonSpinner, IonRefresher, IonRefresherContent, RefresherEventDetail } from '@ionic/vue'
import { getPinnedBids, togglePin } from '@/utils/api'
import type { BidItem } from '@/types'
const pinnedBids = ref<BidItem[]>([])
const loading = ref<boolean>(true)
const error = ref<string>('')
const loadPinnedBids = async () => {
try {
loading.value = true
error.value = ''
const items = await getPinnedBids()
pinnedBids.value = items || []
} catch (err: any) {
error.value = `加载失败: ${err.message || err}`
console.error('加载置顶项目失败:', err)
} finally {
loading.value = false
}
}
const handleRefresh = async (event: CustomEvent<RefresherEventDetail>) => {
await loadPinnedBids()
const target = event.target as any
if (target && target.complete) {
target.complete()
}
}
const handleTogglePin = async (item: BidItem) => {
try {
await togglePin(item.title, false)
pinnedBids.value = pinnedBids.value.filter((bid: BidItem) => bid.title !== item.title)
} catch (err) {
console.error('取消置顶失败:', err)
}
}
const formatDate = (dateStr: string): string => {
if (!dateStr) return ''
const date = new Date(dateStr)
const year = date.getFullYear()
const month = String(date.getMonth() + 1).padStart(2, '0')
const day = String(date.getDate()).padStart(2, '0')
return `${year}-${month}-${day}`
}
onMounted(() => {
loadPinnedBids()
})
defineExpose({
loadPinnedBids
})
</script>
<template>
<IonRefresher slot="fixed" @ionRefresh="handleRefresh">
<IonRefresherContent></IonRefresherContent>
</IonRefresher>
<div v-if="loading" class="loading-container">
<IonSpinner name="crescent" />
<p class="loading-text">加载中...</p>
</div>
<div v-else-if="error" class="error-container">
<p class="error-text">{{ error }}</p>
</div>
<div v-else-if="pinnedBids.length === 0" class="empty-container">
<p class="empty-text">暂无置顶项目</p>
</div>
<div v-else class="pinned-list">
<IonCard v-for="item in pinnedBids" :key="item.id" class="pinned-card">
<IonCardHeader>
<IonCardTitle class="title-text">{{ item.title }}</IonCardTitle>
</IonCardHeader>
<IonCardContent>
<div class="card-details">
<div class="detail-item">
<span class="detail-label">来源:</span>
<span class="detail-value">{{ item.source }}</span>
</div>
<div class="detail-item">
<span class="detail-label">发布日期:</span>
<span class="detail-value">{{ formatDate(item.publishDate) }}</span>
</div>
</div>
<div class="card-footer">
<a :href="item.url" target="_blank" class="view-link">查看详情</a>
<button @click="handleTogglePin(item)" class="unpin-button">取消置顶</button>
</div>
</IonCardContent>
</IonCard>
</div>
</template>
<style scoped>
.loading-container,
.error-container,
.empty-container {
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
padding: 40px 20px;
text-align: center;
}
.loading-text,
.error-text,
.empty-text {
margin-top: 16px;
color: #666;
font-size: 14px;
}
.error-text {
color: #e74c3c;
}
.pinned-list {
padding: 16px;
display: flex;
flex-direction: column;
gap: 16px;
}
.pinned-card {
margin: 0;
box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1);
}
.title-text {
font-size: 16px;
font-weight: 600;
color: #333;
line-height: 1.4;
}
.card-details {
display: flex;
flex-direction: column;
gap: 8px;
margin-bottom: 16px;
}
.detail-item {
display: flex;
justify-content: space-between;
font-size: 14px;
}
.detail-label {
color: #666;
}
.detail-value {
color: #333;
font-weight: 500;
}
.card-footer {
display: flex;
justify-content: space-between;
align-items: center;
padding-top: 12px;
border-top: 1px solid #eee;
}
.view-link {
color: #3498db;
text-decoration: none;
font-size: 14px;
font-weight: 500;
}
.unpin-button {
padding: 6px 16px;
border: 1px solid #e74c3c;
border-radius: 16px;
background: #fff;
color: #e74c3c;
font-size: 12px;
cursor: pointer;
transition: all 0.2s;
}
.unpin-button:active {
transform: scale(0.95);
background: #e74c3c;
color: #fff;
}
</style>

28
ionic-app/src/main.ts Normal file
View File

@@ -0,0 +1,28 @@
import { createApp } from 'vue'
import { IonicVue } from '@ionic/vue'
/* Core CSS required for Ionic components to work properly */
import '@ionic/vue/css/core.css'
/* Basic CSS for apps built with Ionic */
import '@ionic/vue/css/normalize.css'
import '@ionic/vue/css/structure.css'
import '@ionic/vue/css/typography.css'
import '@ionic/vue/css/padding.css'
import '@ionic/vue/css/float-elements.css'
import '@ionic/vue/css/text-alignment.css'
import '@ionic/vue/css/text-transformation.css'
import '@ionic/vue/css/flex-utils.css'
import '@ionic/vue/css/display.css'
/* Theme variables */
import './theme/variables.css'
import App from './App.vue'
import router from './router'
const app = createApp(App)
app.use(IonicVue)
app.use(router)
app.mount('#app')

View File

@@ -0,0 +1,81 @@
<script setup lang="ts">
import { ref } from 'vue'
import { IonPage, IonHeader, IonToolbar, IonTitle, IonContent, IonSegment, IonSegmentButton, IonIcon, IonLabel, IonButtons, IonButton } from '@ionic/vue'
import { sparkles, statsChart, pin, logOutOutline } from 'ionicons/icons'
import AiRecommendations from '@/components/AiRecommendations.vue'
import CrawlInfo from '@/components/CrawlInfo.vue'
import PinnedBids from '@/components/PinnedBids.vue'
import { useAuthStore } from '@/stores/auth'
const activeTab = ref('pinned')
const authStore = useAuthStore()
const aiRecommendationsRef = ref()
const crawlInfoRef = ref()
const pinnedBidsRef = ref()
const handleTabChange = (tab: string) => {
activeTab.value = tab
// 刷新对应标签页的数据
if (tab === 'recommendations' && aiRecommendationsRef.value) {
aiRecommendationsRef.value.loadRecommendations()
} else if (tab === 'crawl' && crawlInfoRef.value) {
crawlInfoRef.value.loadCrawlStats()
} else if (tab === 'pinned' && pinnedBidsRef.value) {
pinnedBidsRef.value.loadPinnedBids()
}
}
const handleLogout = () => {
authStore.logout()
// 路由守卫会自动跳转到登录页
}
</script>
<template>
<IonPage>
<IonHeader>
<IonToolbar>
<IonTitle>投标项目查看器</IonTitle>
<IonButtons slot="end">
<IonButton @click="handleLogout">
<IonIcon :icon="logOutOutline" />
</IonButton>
</IonButtons>
</IonToolbar>
</IonHeader>
<IonContent>
<IonSegment :value="activeTab" @ionChange="(e: any) => handleTabChange(e.detail.value)" class="segment-container">
<IonSegmentButton value="pinned">
<IonIcon :icon="pin" />
<IonLabel>置顶项目</IonLabel>
</IonSegmentButton>
<IonSegmentButton value="recommendations">
<IonIcon :icon="sparkles" />
<IonLabel>AI 推荐</IonLabel>
</IonSegmentButton>
<IonSegmentButton value="crawl">
<IonIcon :icon="statsChart" />
<IonLabel>爬虫信息</IonLabel>
</IonSegmentButton>
</IonSegment>
<div class="tab-content">
<AiRecommendations v-if="activeTab === 'recommendations'" ref="aiRecommendationsRef" />
<CrawlInfo v-if="activeTab === 'crawl'" ref="crawlInfoRef" />
<PinnedBids v-if="activeTab === 'pinned'" ref="pinnedBidsRef" />
</div>
</IonContent>
</IonPage>
</template>
<style scoped>
.segment-container {
margin: 16px;
}
.tab-content {
padding: 0 16px 16px;
}
</style>

View File

@@ -0,0 +1,121 @@
<script setup lang="ts">
import { ref } from 'vue'
import { IonPage, IonHeader, IonToolbar, IonTitle, IonContent, IonItem, IonLabel, IonInput, IonButton, IonCard, IonCardHeader, IonCardTitle, IonCardContent, IonSpinner } from '@ionic/vue'
import { useIonRouter } from '@ionic/vue'
import { useAuthStore } from '@/stores/auth'
const router = useIonRouter()
const authStore = useAuthStore()
const username = ref('')
const password = ref('')
const loading = ref(false)
const error = ref('')
const handleLogin = async () => {
if (!username.value || !password.value) {
error.value = '请输入用户名和密码'
return
}
loading.value = true
error.value = ''
try {
await authStore.login(username.value, password.value)
router.push('/home')
} catch (err: any) {
error.value = err.message || '登录失败,请检查用户名和密码'
} finally {
loading.value = false
}
}
</script>
<template>
<IonPage>
<IonHeader>
<IonToolbar>
<IonTitle>登录</IonTitle>
</IonToolbar>
</IonHeader>
<IonContent class="login-content">
<IonCard class="login-card">
<IonCardHeader>
<IonCardTitle>投标项目查看器</IonCardTitle>
</IonCardHeader>
<IonCardContent>
<div v-if="error" class="error-message">
{{ error }}
</div>
<IonItem class="input-item">
<IonLabel position="floating">用户名</IonLabel>
<IonInput
v-model="username"
type="text"
placeholder="请输入用户名"
:disabled="loading"
/>
</IonItem>
<IonItem class="input-item">
<IonLabel position="floating">密码</IonLabel>
<IonInput
v-model="password"
type="password"
placeholder="请输入密码"
:disabled="loading"
/>
</IonItem>
<IonButton
expand="block"
@click="handleLogin"
:disabled="loading"
class="login-button"
>
<IonSpinner v-if="loading" name="crescent" slot="start" />
{{ loading ? '登录中...' : '登录' }}
</IonButton>
</IonCardContent>
</IonCard>
</IonContent>
</IonPage>
</template>
<style scoped>
.login-content {
--background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
display: flex;
align-items: center;
justify-content: center;
}
.login-card {
margin: 20px;
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.1);
}
.error-message {
background: #f8d7da;
color: #721c24;
padding: 12px;
border-radius: 8px;
margin-bottom: 16px;
font-size: 14px;
}
.input-item {
margin-bottom: 16px;
}
.login-button {
margin-top: 24px;
--background: #667eea;
--color: #fff;
height: 48px;
font-size: 16px;
font-weight: 600;
}
</style>

View File

@@ -0,0 +1,42 @@
import { createRouter, createWebHistory } from '@ionic/vue-router'
import HomePage from '@/pages/HomePage.vue'
import LoginPage from '@/pages/LoginPage.vue'
import { useAuthStore } from '@/stores/auth'
const routes = [
{
path: '/',
redirect: '/login'
},
{
path: '/login',
component: LoginPage
},
{
path: '/home',
component: HomePage,
meta: { requiresAuth: true }
}
]
const router = createRouter({
history: createWebHistory(),
routes
})
// 路由守卫
router.beforeEach((to, from, next) => {
const authStore = useAuthStore()
if (to.meta.requiresAuth && !authStore.state.value.isAuthenticated) {
// 需要认证但未登录,跳转到登录页
next('/login')
} else if (to.path === '/login' && authStore.state.value.isAuthenticated) {
// 已登录用户访问登录页,跳转到首页
next('/home')
} else {
next()
}
})
export default router

View File

@@ -0,0 +1,79 @@
import { ref } from 'vue'
import api from '@/utils/api'
interface AuthState {
isAuthenticated: boolean
username: string | null
token: string | null
}
const state = ref<AuthState>({
isAuthenticated: false,
username: null,
token: null
})
export const useAuthStore = () => {
const login = async (username: string, password: string) => {
// 创建 Basic Auth token
const credentials = btoa(`${username}:${password}`)
const token = `Basic ${credentials}`
// 保存 token 到 localStorage
localStorage.setItem('auth_token', token)
localStorage.setItem('auth_username', username)
// 更新 API 实例的默认 headers
api.defaults.headers.common['Authorization'] = token
// 更新状态
state.value = {
isAuthenticated: true,
username,
token
}
}
const logout = () => {
// 清除 localStorage
localStorage.removeItem('auth_token')
localStorage.removeItem('auth_username')
// 清除 API headers
delete api.defaults.headers.common['Authorization']
// 更新状态
state.value = {
isAuthenticated: false,
username: null,
token: null
}
}
const checkAuth = () => {
const token = localStorage.getItem('auth_token')
const username = localStorage.getItem('auth_username')
if (token && username) {
api.defaults.headers.common['Authorization'] = token
state.value = {
isAuthenticated: true,
username,
token
}
return true
}
return false
}
// 初始化时检查认证状态
checkAuth()
return {
state,
login,
logout,
checkAuth
}
}

View File

@@ -0,0 +1,229 @@
/* Ionic Variables and Theming. For more info, please see:
http://ionicframework.com/docs/theming/theming-base-variables */
/** Ionic CSS Variables **/
:root {
/** primary **/
--ion-color-primary: #3880ff;
--ion-color-primary-rgb: 56, 128, 255;
--ion-color-primary-contrast: #ffffff;
--ion-color-primary-contrast-rgb: 255, 255, 255;
--ion-color-primary-shade: #3171e0;
--ion-color-primary-tint: #4c8dff;
/** secondary **/
--ion-color-secondary: #3dc2ff;
--ion-color-secondary-rgb: 61, 194, 255;
--ion-color-secondary-contrast: #ffffff;
--ion-color-secondary-contrast-rgb: 255, 255, 255;
--ion-color-secondary-shade: #36abe0;
--ion-color-secondary-tint: #50c8ff;
/** tertiary **/
--ion-color-tertiary: #5260ff;
--ion-color-tertiary-rgb: 82, 96, 255;
--ion-color-tertiary-contrast: #ffffff;
--ion-color-tertiary-contrast-rgb: 255, 255, 255;
--ion-color-tertiary-shade: #4854e0;
--ion-color-tertiary-tint: #6370ff;
/** success **/
--ion-color-success: #2dd36f;
--ion-color-success-rgb: 45, 211, 111;
--ion-color-success-contrast: #ffffff;
--ion-color-success-contrast-rgb: 255, 255, 255;
--ion-color-success-shade: #28ba62;
--ion-color-success-tint: #42d77d;
/** warning **/
--ion-color-warning: #ffc409;
--ion-color-warning-rgb: 255, 196, 9;
--ion-color-warning-contrast: #000000;
--ion-color-warning-contrast-rgb: 0, 0, 0;
--ion-color-warning-shade: #e0ac08;
--ion-color-warning-tint: #ffca22;
/** danger **/
--ion-color-danger: #eb445a;
--ion-color-danger-rgb: 235, 68, 90;
--ion-color-danger-contrast: #ffffff;
--ion-color-danger-contrast-rgb: 255, 255, 255;
--ion-color-danger-shade: #cf3c4f;
--ion-color-danger-tint: #ed576b;
/** dark **/
--ion-color-dark: #222428;
--ion-color-dark-rgb: 34, 36, 40;
--ion-color-dark-contrast: #ffffff;
--ion-color-dark-contrast-rgb: 255, 255, 255;
--ion-color-dark-shade: #1e2023;
--ion-color-dark-tint: #383a3e;
/** medium **/
--ion-color-medium: #92949c;
--ion-color-medium-rgb: 146, 148, 156;
--ion-color-medium-contrast: #ffffff;
--ion-color-medium-contrast-rgb: 255, 255, 255;
--ion-color-medium-shade: #808289;
--ion-color-medium-tint: #9d9fa6;
/** light **/
--ion-color-light: #f4f5f8;
--ion-color-light-rgb: 244, 245, 248;
--ion-color-light-contrast: #000000;
--ion-color-light-contrast-rgb: 0, 0, 0;
--ion-color-light-shade: #d7d8da;
--ion-color-light-tint: #f5f6f9;
}
@media (prefers-color-scheme: dark) {
/*
* Dark Colors
* -------------------------------------------
*/
body.dark {
--ion-color-primary: #428cff;
--ion-color-primary-rgb: 66, 140, 255;
--ion-color-primary-contrast: #ffffff;
--ion-color-primary-contrast-rgb: 255, 255, 255;
--ion-color-primary-shade: #3a7be0;
--ion-color-primary-tint: #5598ff;
--ion-color-secondary: #50c8ff;
--ion-color-secondary-rgb: 80, 200, 255;
--ion-color-secondary-contrast: #ffffff;
--ion-color-secondary-contrast-rgb: 255, 255, 255;
--ion-color-secondary-shade: #46b0e0;
--ion-color-secondary-tint: #62ceff;
--ion-color-tertiary: #6a64ff;
--ion-color-tertiary-rgb: 106, 100, 255;
--ion-color-tertiary-contrast: #ffffff;
--ion-color-tertiary-contrast-rgb: 255, 255, 255;
--ion-color-tertiary-shade: #5d58e0;
--ion-color-tertiary-tint: #7974ff;
--ion-color-success: #2fdf75;
--ion-color-success-rgb: 47, 223, 117;
--ion-color-success-contrast: #000000;
--ion-color-success-contrast-rgb: 0, 0, 0;
--ion-color-success-shade: #29c467;
--ion-color-success-tint: #44e283;
--ion-color-warning: #ffd534;
--ion-color-warning-rgb: 255, 213, 52;
--ion-color-warning-contrast: #000000;
--ion-color-warning-contrast-rgb: 0, 0, 0;
--ion-color-warning-shade: #e0bb2e;
--ion-color-warning-tint: #ffd948;
--ion-color-danger: #ff4961;
--ion-color-danger-rgb: 255, 73, 97;
--ion-color-danger-contrast: #ffffff;
--ion-color-danger-contrast-rgb: 255, 255, 255;
--ion-color-danger-shade: #e04055;
--ion-color-danger-tint: #ff5b71;
--ion-color-dark: #f4f5f8;
--ion-color-dark-rgb: 244, 245, 248;
--ion-color-dark-contrast: #000000;
--ion-color-dark-contrast-rgb: 0, 0, 0;
--ion-color-dark-shade: #d7d8da;
--ion-color-dark-tint: #f5f6f9;
--ion-color-medium: #989aa2;
--ion-color-medium-rgb: 152, 154, 162;
--ion-color-medium-contrast: #000000;
--ion-color-medium-contrast-rgb: 0, 0, 0;
--ion-color-medium-shade: #86888f;
--ion-color-medium-tint: #a2a4ab;
--ion-color-light: #222428;
--ion-color-light-rgb: 34, 36, 40;
--ion-color-light-contrast: #ffffff;
--ion-color-light-contrast-rgb: 255, 255, 255;
--ion-color-light-shade: #1e2023;
--ion-color-light-tint: #383a3e;
}
/*
* iOS Dark Theme
* -------------------------------------------
*/
.ios body.dark {
--ion-background-color: #03060b;
--ion-background-color-rgb: 3, 6, 11;
--ion-text-color: #ffffff;
--ion-text-color-rgb: 255, 255, 255;
--ion-color-step-50: #0d0d0d;
--ion-color-step-100: #1a1a1a;
--ion-color-step-150: #262626;
--ion-color-step-200: #333333;
--ion-color-step-250: #404040;
--ion-color-step-300: #4d4d4d;
--ion-color-step-350: #595959;
--ion-color-step-400: #666666;
--ion-color-step-450: #737373;
--ion-color-step-500: #808080;
--ion-color-step-550: #8c8c8c;
--ion-color-step-600: #999999;
--ion-color-step-650: #a6a6a6;
--ion-color-step-700: #b3b3b3;
--ion-color-step-750: #bfbfbf;
--ion-color-step-800: #cccccc;
--ion-color-step-850: #d9d9d9;
--ion-color-step-900: #e6e6e6;
--ion-color-step-950: #f2f2f2;
--ion-item-background: #000000;
--ion-card-background: #1c1c1d;
}
/*
* Material Design Dark Theme
* -------------------------------------------
*/
.md body.dark {
--ion-background-color: #121212;
--ion-background-color-rgb: 18, 18, 18;
--ion-text-color: #ffffff;
--ion-text-color-rgb: 255, 255, 255;
--ion-border-color: #222222;
--ion-color-step-50: #1e1e1e;
--ion-color-step-100: #2a2a2a;
--ion-color-step-150: #363636;
--ion-color-step-200: #414141;
--ion-color-step-250: #4d4d4d;
--ion-color-step-300: #595959;
--ion-color-step-350: #656565;
--ion-color-step-400: #717171;
--ion-color-step-450: #7d7d7d;
--ion-color-step-500: #898989;
--ion-color-step-550: #949494;
--ion-color-step-600: #a0a0a0;
--ion-color-step-650: #acacac;
--ion-color-step-700: #b8b8b8;
--ion-color-step-750: #c4c4c4;
--ion-color-step-800: #d0d0d0;
--ion-color-step-850: #dbdbdb;
--ion-color-step-900: #e7e7e7;
--ion-color-step-950: #f3f3f3;
--ion-item-background: #1e1e1e;
--ion-toolbar-background: #1f1f1f;
--ion-tab-bar-background: #1f1f1f;
--ion-card-background: #1e1e1e;
}
}

View File

@@ -0,0 +1,40 @@
// 投标项目类型
export interface BidItem {
id: string
title: string
url: string
publishDate: string
source: string
pin: boolean
createdAt?: string
updatedAt?: string
}
// AI 推荐类型
export interface AiRecommendation {
id: string
title: string
url: string
source: string
confidence: number
publishDate?: string
pin?: boolean
}
// 爬虫统计信息类型
export interface CrawlInfoStat {
source: string
count: number
latestUpdate: string | null
latestPublishDate: string | null
error: string | null
}
// API 响应类型
export interface ApiResponse<T> {
data: T
message?: string
}
// 日期范围类型
export type DateRange = [string, string] | null

View File

@@ -0,0 +1,97 @@
import axios from 'axios'
import type { BidItem, AiRecommendation, CrawlInfoStat, DateRange } from '@/types'
// 从环境变量读取 API 基础地址
const BASE_URL = import.meta.env.VITE_API_BASE_URL || 'http://localhost:3001'
// 创建 axios 实例
const api = axios.create({
baseURL: BASE_URL,
timeout: 30000,
headers: {
'Content-Type': 'application/json'
}
})
// 请求拦截器
api.interceptors.request.use(
(config) => {
return config
},
(error) => {
return Promise.reject(error)
}
)
// 响应拦截器
api.interceptors.response.use(
(response) => {
return response
},
(error) => {
console.error('API 请求错误:', error)
return Promise.reject(error)
}
)
/**
* 获取置顶投标项目
*/
export function getPinnedBids(): Promise<BidItem[]> {
return api.get('/api/bids/pinned').then(res => res.data)
}
/**
* 获取最新 AI 推荐
*/
export function getAiRecommendations(): Promise<AiRecommendation[]> {
return api.get('/api/ai/latest-recommendations').then(res => res.data)
}
/**
* 获取爬虫统计信息
*/
export function getCrawlInfoStats(): Promise<CrawlInfoStat[]> {
return api.get('/api/bids/crawl-info-stats').then(res => res.data)
}
/**
* 爬取单个数据源
*/
export function crawlSingleSource(sourceName: string): Promise<any> {
return api.post(`/api/crawler/crawl/${encodeURIComponent(sourceName)}`).then(res => res.data)
}
/**
* 切换置顶状态
*/
export function togglePin(title: string, pin: boolean): Promise<void> {
return api.patch(`/api/bids/${encodeURIComponent(title)}/pin`, { pin }).then(res => res.data)
}
/**
* 按日期范围获取工程
*/
export function getBidsByDateRange(startDate: string, endDate?: string): Promise<BidItem[]> {
const params: any = { startDate }
if (endDate) {
params.endDate = endDate
}
return api.get('/api/bids/by-date-range', { params }).then(res => res.data)
}
/**
* 获取 AI 推荐(发送 bids 数据)
*/
export function fetchAiRecommendations(bids: { title: string }[]): Promise<AiRecommendation[]> {
return api.post('/api/ai/recommendations', { bids }).then(res => res.data)
}
/**
* 保存 AI 推荐结果
*/
export function saveAiRecommendations(recommendations: AiRecommendation[]): Promise<void> {
return api.post('/api/ai/save-recommendations', { recommendations }).then(res => res.data)
}
export default api

9
ionic-app/src/vite-env.d.ts vendored Normal file
View File

@@ -0,0 +1,9 @@
/// <reference types="vite/client" />
interface ImportMetaEnv {
readonly VITE_API_BASE_URL: string
}
interface ImportMeta {
readonly env: ImportMetaEnv
}

View File

@@ -0,0 +1,11 @@
/** @type {import('tailwindcss').Config} */
export default {
content: [
"./index.html",
"./src/**/*.{vue,js,ts,jsx,tsx}",
],
theme: {
extend: {},
},
plugins: [],
}

25
ionic-app/tsconfig.json Normal file
View File

@@ -0,0 +1,25 @@
{
"compilerOptions": {
"target": "ES2020",
"useDefineForClassFields": true,
"module": "ESNext",
"lib": ["ES2020", "DOM", "DOM.Iterable"],
"skipLibCheck": true,
"moduleResolution": "bundler",
"allowImportingTsExtensions": true,
"resolveJsonModule": true,
"isolatedModules": true,
"noEmit": true,
"jsx": "preserve",
"strict": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"noFallthroughCasesInSwitch": true,
"baseUrl": ".",
"paths": {
"@/*": ["src/*"]
}
},
"include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"],
"references": [{ "path": "./tsconfig.node.json" }]
}

View File

@@ -0,0 +1,11 @@
{
"compilerOptions": {
"composite": true,
"skipLibCheck": true,
"module": "ESNext",
"moduleResolution": "bundler",
"allowSyntheticDefaultImports": true,
"strict": true
},
"include": ["vite.config.ts"]
}

19
ionic-app/vite.config.ts Normal file
View File

@@ -0,0 +1,19 @@
import { defineConfig } from 'vite'
import vue from '@vitejs/plugin-vue'
import { resolve } from 'path'
import { fileURLToPath } from 'url'
const __dirname = fileURLToPath(new URL('.', import.meta.url))
export default defineConfig({
plugins: [vue()],
resolve: {
alias: {
'@': resolve(__dirname, 'src')
}
},
server: {
port: 8100,
host: true
}
})

View File

@@ -5,13 +5,14 @@
"author": "",
"private": true,
"license": "UNLICENSED",
"main": "app/main.js",
"scripts": {
"build": "nest build",
"format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
"start": "nest start",
"start:dev": "nest start --watch",
"start:debug": "nest start --debug --watch",
"start:prod": "node dist/main",
"start:prod": "node dist/main.js",
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
"test": "jest",
"test:watch": "jest --watch",
@@ -21,7 +22,13 @@
"crawl": "ts-node -r tsconfig-paths/register src/scripts/crawl.ts",
"update-source": "ts-node -r tsconfig-paths/register src/scripts/update-source.ts",
"ai-recommendations": "ts-node -r tsconfig-paths/register src/scripts/ai-recommendations.ts",
"deploy": "powershell -ExecutionPolicy Bypass -File src/scripts/deploy.ps1"
"sync": "ts-node -r tsconfig-paths/register src/scripts/sync.ts",
"deploy": "ts-node src/scripts/deploy.ts",
"user:create": "ts-node -r tsconfig-paths/register src/scripts/create-user.ts",
"user:list": "ts-node -r tsconfig-paths/register src/scripts/list-users.ts",
"user:delete": "ts-node -r tsconfig-paths/register src/scripts/delete-user.ts",
"electron:dev": "chcp 65001 >nul 2>&1 & npm run -prefix frontend build && npm run build && set NODE_ENV=development && electron ./app",
"electron:build": "npm run -prefix frontend build && npm run build && electron-builder --config ./app/electron-builder.json"
},
"dependencies": {
"@nestjs/common": "^11.0.1",
@@ -32,6 +39,7 @@
"@nestjs/serve-static": "^5.0.4",
"@nestjs/typeorm": "^11.0.0",
"axios": "^1.13.2",
"bcrypt": "^6.0.0",
"class-transformer": "^0.5.1",
"class-validator": "^0.14.3",
"dotenv": "^16.4.7",
@@ -42,6 +50,7 @@
"puppeteer-extra-plugin-stealth": "^2.11.2",
"reflect-metadata": "^0.2.2",
"rxjs": "^7.8.1",
"ssh2": "^1.17.0",
"typeorm": "^0.3.28",
"winston": "^3.19.0",
"winston-daily-rotate-file": "^5.0.0"
@@ -52,11 +61,20 @@
"@nestjs/cli": "^11.0.14",
"@nestjs/schematics": "^11.0.0",
"@nestjs/testing": "^11.0.1",
"@types/bcrypt": "^6.0.0",
"@types/cacheable-request": "^6.0.3",
"@types/express": "^5.0.0",
"@types/fs-extra": "^11.0.4",
"@types/http-cache-semantics": "^4.0.4",
"@types/jest": "^30.0.0",
"@types/node": "^22.10.7",
"@types/responselike": "^1.0.3",
"@types/ssh2": "^1.15.5",
"@types/ssh2-sftp-client": "^9.0.6",
"@types/supertest": "^6.0.2",
"concurrently": "^9.2.1",
"electron": "^39.2.7",
"electron-builder": "^26.4.0",
"eslint": "^9.18.0",
"eslint-config-prettier": "^10.0.1",
"eslint-plugin-prettier": "^5.2.2",
@@ -64,6 +82,7 @@
"jest": "^30.0.0",
"prettier": "^3.4.2",
"source-map-support": "^0.5.21",
"ssh2-sftp-client": "^12.0.1",
"supertest": "^7.0.0",
"ts-jest": "^29.2.5",
"ts-loader": "^9.5.2",

View File

@@ -1 +1 @@
export const PromptString: string = `先给我说你统计了多少个项目。我只对辽宁、山东、江苏、浙江、福建、广东、广西、海南、河北这些地方的海上风电、海上光伏、漂浮式光伏、滩涂光伏、滩涂风电、渔光互补项目感兴趣。从我提供的这些工程里面找到我感兴趣的工程无论如何至少推荐10个工程。如果没有推荐的也要给出思考过程。`;
export const PromptString: string = `先给我说你统计了多少个项目。我只对辽宁、山东、江苏、浙江、福建、广东、广西、海南、河北这些地方的海上风电、海上光伏、漂浮式光伏、滩涂光伏、滩涂风电、渔光互补、风光互补项目感兴趣。从我提供的这些工程里面找到我感兴趣的工程无论如何至少推荐10个工程。如果没有推荐的也要给出思考过程。不要修改或简化返回的工程名称。`;

View File

@@ -17,6 +17,7 @@ export interface AIRecommendation {
source: string;
confidence: number;
publishDate?: Date;
generatedAt?: string;
}
@Injectable()
@@ -31,14 +32,13 @@ export class AiService {
@InjectRepository(BidItem)
private readonly bidItemRepository: Repository<BidItem>,
) {
const apiKey = this.configService.get<string>('ARK_API_KEY');
// this.openai = new OpenAI({
// apiKey: apiKey || '',
// apiKey: this.configService.get<string>('ARK_API_KEY') || '',
// baseURL: 'https://ark.cn-beijing.volces.com/api/v3',
// timeout: 120000, // 120秒超时
// });
this.openai = new OpenAI({
apiKey: 'sk-5sSOxrJl31MGz76bE14d2fDbA55b44869fCcA0C813Fc893a' ,
apiKey: 'sk-5sSOxrJl31MGz76bE14d2fDbA55b44869fCcA0C813Fc893a',
baseURL: 'https://aihubmix.com/v1',
timeout: 120000, // 120秒超时
});
@@ -49,7 +49,9 @@ export class AiService {
this.logger.log(`发送给 AI 的数据数量: ${bids.length}`);
try {
const prompt =PromptString+ `请根据以下投标项目标题列表,筛选出我关心的项目。请以 JSON 格式返回,格式如下:
const prompt =
PromptString +
`请根据以下投标项目标题列表,筛选出我关心的项目。请以 JSON 格式返回,格式如下:
[
{
"title": "项目标题",
@@ -58,7 +60,11 @@ export class AiService {
]
投标项目标题列表:
${JSON.stringify(bids.map(b => b.title), null, 2)}`;
${JSON.stringify(
bids.map((b) => b.title),
null,
2,
)}`;
// this.logger.log('发给AI的内容',prompt);
const completion = await this.openai.chat.completions.create({
model: 'mimo-v2-flash-free',
@@ -97,7 +103,9 @@ ${JSON.stringify(bids.map(b => b.title), null, 2)}`;
}
}
async saveRecommendations(recommendations: AIRecommendation[]): Promise<void> {
async saveRecommendations(
recommendations: AIRecommendation[],
): Promise<void> {
this.logger.log('开始保存 AI 推荐结果');
try {
@@ -105,7 +113,7 @@ ${JSON.stringify(bids.map(b => b.title), null, 2)}`;
await this.aiRecommendationRepository.clear();
// 保存新的推荐结果(只保存 title 和 confidence
const entities = recommendations.map(rec => {
const entities = recommendations.map((rec) => {
const entity = new AiRecommendationEntity();
entity.title = rec.title;
entity.confidence = rec.confidence;
@@ -120,19 +128,31 @@ ${JSON.stringify(bids.map(b => b.title), null, 2)}`;
}
}
async getLatestRecommendations(): Promise<AIRecommendation[]> {
async getLatestRecommendations(): Promise<{ recommendations: AIRecommendation[]; generatedAt: string | null }> {
this.logger.log('获取最新的 AI 推荐结果');
try {
// 查询最大的 createdAt 作为生成时间
const maxCreatedAtResult = await this.aiRecommendationRepository
.createQueryBuilder('rec')
.select('MAX(rec.createdAt)', 'maxCreatedAt')
.getRawOne();
const generatedAt = maxCreatedAtResult?.maxCreatedAt
? new Date(maxCreatedAtResult.maxCreatedAt).toLocaleString('zh-CN', { hour12: false })
: null;
this.logger.log(`AI 推荐生成时间: ${generatedAt}`);
const entities = await this.aiRecommendationRepository.find({
order: { confidence: 'DESC' }
order: { confidence: 'DESC' },
});
// 从 bid-items 表获取 url、source 和 publishDate
const result: AIRecommendation[] = [];
for (const entity of entities) {
const bidItem = await this.bidItemRepository.findOne({
where: { title: entity.title }
where: { title: entity.title },
});
result.push({
@@ -140,7 +160,7 @@ ${JSON.stringify(bids.map(b => b.title), null, 2)}`;
url: bidItem?.url || '',
source: bidItem?.source || '',
confidence: entity.confidence,
publishDate: bidItem?.publishDate
publishDate: bidItem?.publishDate,
});
}
@@ -148,10 +168,12 @@ ${JSON.stringify(bids.map(b => b.title), null, 2)}`;
result.sort((a, b) => {
if (!a.publishDate) return 1;
if (!b.publishDate) return -1;
return new Date(b.publishDate).getTime() - new Date(a.publishDate).getTime();
return (
new Date(b.publishDate).getTime() - new Date(a.publishDate).getTime()
);
});
return result;
return { recommendations: result, generatedAt };
} catch (error) {
this.logger.error('获取最新 AI 推荐失败:', error);
throw error;

View File

@@ -1,4 +1,9 @@
import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn } from 'typeorm';
import {
Entity,
PrimaryGeneratedColumn,
Column,
CreateDateColumn,
} from 'typeorm';
@Entity('ai_recommendations')
export class AiRecommendation {

View File

@@ -1,4 +1,4 @@
import { Module } from '@nestjs/common';
import { MiddlewareConsumer, Module, NestModule } from '@nestjs/common';
import { ConfigModule } from '@nestjs/config';
import { ScheduleModule } from '@nestjs/schedule';
import { ServeStaticModule } from '@nestjs/serve-static';
@@ -9,7 +9,10 @@ import { KeywordsModule } from './keywords/keywords.module';
import { CrawlerModule } from './crawler/crawler.module';
import { TasksModule } from './schedule/schedule.module';
import { LoggerModule } from './common/logger/logger.module';
import { LoggingMiddleware } from './common/logger/logging.middleware';
import { AiModule } from './ai/ai.module';
import { AuthModule } from './common/auth/auth.module';
import { UsersModule } from './users/users.module';
@Module({
imports: [
@@ -20,6 +23,8 @@ import { AiModule } from './ai/ai.module';
exclude: ['/api'],
}),
LoggerModule,
AuthModule,
UsersModule,
DatabaseModule,
BidsModule,
KeywordsModule,
@@ -28,4 +33,8 @@ import { AiModule } from './ai/ai.module';
AiModule,
],
})
export class AppModule {}
export class AppModule implements NestModule {
configure(consumer: MiddlewareConsumer) {
consumer.apply(LoggingMiddleware).forRoutes('*');
}
}

View File

@@ -1,12 +1,19 @@
import { Controller, Get, Query, Patch, Param, Body } from '@nestjs/common';
import { BidsService } from '../services/bid.service';
interface FindAllQuery {
page?: number;
limit?: number;
source?: string;
keyword?: string;
}
@Controller('api/bids')
export class BidsController {
constructor(private readonly bidsService: BidsService) {}
@Get()
findAll(@Query() query: any) {
findAll(@Query() query: FindAllQuery) {
return this.bidsService.findAll(query);
}
@@ -26,9 +33,17 @@ export class BidsController {
}
@Get('by-date-range')
getByDateRange(@Query('startDate') startDate: string, @Query('endDate') endDate?: string, @Query('keywords') keywords?: string) {
getByDateRange(
@Query('startDate') startDate: string,
@Query('endDate') endDate?: string,
@Query('keywords') keywords?: string,
) {
const keywordsArray = keywords ? keywords.split(',') : undefined;
return this.bidsService.getBidsByDateRange(startDate, endDate, keywordsArray);
return this.bidsService.getBidsByDateRange(
startDate,
endDate,
keywordsArray,
);
}
@Get('crawl-info-stats')

View File

@@ -1,4 +1,10 @@
import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn, UpdateDateColumn } from 'typeorm';
import {
Entity,
PrimaryGeneratedColumn,
Column,
CreateDateColumn,
UpdateDateColumn,
} from 'typeorm';
@Entity('bid_items')
export class BidItem {

View File

@@ -1,8 +1,42 @@
import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository, LessThan, MoreThanOrEqual } from 'typeorm';
import { Repository, LessThan } from 'typeorm';
import { BidItem } from '../entities/bid-item.entity';
import { CrawlInfoAdd } from '../../crawler/entities/crawl-info-add.entity';
import {
getDaysAgo,
setStartOfDay,
setEndOfDay,
utcToBeijing,
utcToBeijingISOString,
} from '../../common/utils/timezone.util';
interface FindAllQuery {
page?: number;
limit?: number;
source?: string;
keyword?: string;
}
interface SourceResult {
source: string;
}
export interface CrawlInfoAddStats {
source: string;
count: number;
latestUpdate: Date | string | null;
latestPublishDate: Date | string | null;
error: string | null;
}
interface CrawlInfoAddRawResult {
source: string;
count: number;
latestPublishDate: Date | string | null;
error: string | null;
latestUpdate: Date | string;
}
@Injectable()
export class BidsService {
@@ -13,7 +47,7 @@ export class BidsService {
private crawlInfoRepository: Repository<CrawlInfoAdd>,
) {}
async findAll(query?: any) {
async findAll(query?: FindAllQuery) {
const { page = 1, limit = 10, source, keyword } = query || {};
const qb = this.bidRepository.createQueryBuilder('bid');
@@ -26,8 +60,8 @@ export class BidsService {
}
qb.orderBy('bid.publishDate', 'DESC')
.skip((page - 1) * limit)
.take(limit);
.skip((Number(page) - 1) * Number(limit))
.take(Number(limit));
const [items, total] = await qb.getManyAndCount();
return { items, total };
@@ -35,7 +69,9 @@ export class BidsService {
async createOrUpdate(data: Partial<BidItem>) {
// Use title or a hash of title to check for duplicates
let item = await this.bidRepository.findOne({ where: { title: data.title } });
const item = await this.bidRepository.findOne({
where: { title: data.title },
});
if (item) {
Object.assign(item, data);
return this.bidRepository.save(item);
@@ -44,28 +80,25 @@ export class BidsService {
}
async cleanOldData() {
const thirtyDaysAgo = new Date();
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
const thirtyDaysAgo = getDaysAgo(30);
return this.bidRepository.delete({
createdAt: LessThan(thirtyDaysAgo),
});
}
async getSources() {
async getSources(): Promise<string[]> {
const result = await this.bidRepository
.createQueryBuilder('bid')
.select('DISTINCT bid.source')
.select('DISTINCT bid.source', 'source')
.where('bid.source IS NOT NULL')
.orderBy('bid.source', 'ASC')
.getRawMany();
return result.map((item: any) => item.source);
.getRawMany<SourceResult>();
return result.map((item) => item.source);
}
async getRecentBids() {
const thirtyDaysAgo = new Date();
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
thirtyDaysAgo.setHours(0, 0, 0, 0);
const thirtyDaysAgo = setStartOfDay(getDaysAgo(30));
return this.bidRepository
.createQueryBuilder('bid')
.where('bid.publishDate >= :thirtyDaysAgo', { thirtyDaysAgo })
@@ -81,29 +114,36 @@ export class BidsService {
.getMany();
}
async getBidsByDateRange(startDate?: string, endDate?: string, keywords?: string[]) {
async getBidsByDateRange(
startDate?: string,
endDate?: string,
keywords?: string[],
) {
const qb = this.bidRepository.createQueryBuilder('bid');
if (startDate) {
const start = new Date(startDate);
start.setHours(0, 0, 0, 0);
const start = setStartOfDay(new Date(startDate));
qb.andWhere('bid.publishDate >= :startDate', { startDate: start });
}
if (endDate) {
const end = new Date(endDate);
end.setHours(23, 59, 59, 999);
const end = setEndOfDay(new Date(endDate));
qb.andWhere('bid.publishDate <= :endDate', { endDate: end });
}
if (keywords && keywords.length > 0) {
const keywordConditions = keywords.map((keyword, index) => {
return `bid.title LIKE :keyword${index}`;
}).join(' OR ');
qb.andWhere(`(${keywordConditions})`, keywords.reduce((params, keyword, index) => {
params[`keyword${index}`] = `%${keyword}%`;
return params;
}, {}));
const keywordConditions = keywords
.map((keyword, index) => {
return `bid.title LIKE :keyword${index}`;
})
.join(' OR ');
qb.andWhere(
`(${keywordConditions})`,
keywords.reduce((params, keyword, index) => {
params[`keyword${index}`] = `%${keyword}%`;
return params;
}, {}),
);
}
return qb.orderBy('bid.publishDate', 'DESC').getMany();
@@ -118,7 +158,7 @@ export class BidsService {
return this.bidRepository.save(item);
}
async getCrawlInfoAddStats() {
async getCrawlInfoAddStats(): Promise<CrawlInfoAddStats[]> {
// 获取每个来源的最新一次爬虫记录(按 createdAt 降序)
const query = `
SELECT
@@ -136,14 +176,30 @@ export class BidsService {
ORDER BY source ASC
`;
const results = await this.crawlInfoRepository.query(query);
const results =
await this.crawlInfoRepository.query<CrawlInfoAddRawResult[]>(query);
return results.map((item: any) => ({
source: item.source,
count: item.count,
latestUpdate: item.latestUpdate,
latestPublishDate: item.latestPublishDate,
error: item.error,
}));
return results.map((item) => {
// 将UTC时间转换为北京时间的ISO字符串格式
// 这样前端接收到的时间字符串已经是正确的北京时间,不需要再次转换
const latestUpdateBeijing = item.latestUpdate
? utcToBeijingISOString(new Date(item.latestUpdate))
: null;
const latestPublishDateBeijing = item.latestPublishDate
? utcToBeijingISOString(new Date(item.latestPublishDate))
: null;
return {
source: String(item.source),
count: Number(item.count),
latestUpdate: latestUpdateBeijing,
latestPublishDate: latestPublishDateBeijing,
// 确保 error 字段正确处理null 或空字符串都转换为 null非空字符串保留
error:
item.error && String(item.error).trim() !== ''
? String(item.error)
: null,
};
});
}
}

View File

@@ -0,0 +1,329 @@
import { Test, TestingModule } from '@nestjs/testing';
import { ConfigService } from '@nestjs/config';
import { ExecutionContext, UnauthorizedException } from '@nestjs/common';
import { Request } from 'express';
import { AuthGuard } from './auth.guard';
describe('AuthGuard', () => {
let guard: AuthGuard;
let configService: ConfigService;
let mockExecutionContext: ExecutionContext;
let mockRequest: Partial<Request>;
const createMockExecutionContext = (request: Partial<Request>): ExecutionContext => {
return {
switchToHttp: () => ({
getRequest: () => request as Request,
getResponse: () => ({}),
getNext: () => ({}),
}),
} as ExecutionContext;
};
beforeEach(async () => {
const module: TestingModule = await Test.createTestingModule({
providers: [
AuthGuard,
{
provide: ConfigService,
useValue: {
get: jest.fn(),
},
},
],
}).compile();
guard = module.get<AuthGuard>(AuthGuard);
configService = module.get<ConfigService>(ConfigService);
});
describe('本地IP访问', () => {
it('应该允许 127.0.0.1 访问', () => {
mockRequest = {
ip: '127.0.0.1',
socket: { remoteAddress: '127.0.0.1' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该允许 ::1 (IPv6本地地址) 访问', () => {
mockRequest = {
ip: '::1',
socket: { remoteAddress: '::1' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该允许 localhost 访问', () => {
mockRequest = {
ip: 'localhost',
socket: { remoteAddress: 'localhost' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该允许通过 X-Forwarded-For 传递的本地IP访问', () => {
mockRequest = {
ip: '192.168.1.1',
socket: { remoteAddress: '192.168.1.1' },
headers: {
'x-forwarded-for': '127.0.0.1',
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
});
describe('公网IP访问 - 已配置API_KEY', () => {
const validApiKey = 'test-api-key-12345';
beforeEach(() => {
jest.spyOn(configService, 'get').mockReturnValue(validApiKey);
});
it('应该允许提供正确API Key的公网访问', () => {
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {
'x-api-key': validApiKey,
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该拒绝未提供API Key的公网访问', () => {
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
'Invalid or missing API Key',
);
});
it('应该拒绝提供错误API Key的公网访问', () => {
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {
'x-api-key': 'wrong-api-key',
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
'Invalid or missing API Key',
);
});
it('应该拒绝提供空字符串API Key的公网访问', () => {
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {
'x-api-key': '',
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
it('应该正确处理 X-Forwarded-For 中的公网IP', () => {
mockRequest = {
ip: '127.0.0.1',
socket: { remoteAddress: '127.0.0.1' },
headers: {
'x-forwarded-for': '8.8.8.8',
'x-api-key': validApiKey,
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该正确处理多个IP的 X-Forwarded-For 头(取第一个)', () => {
mockRequest = {
ip: '127.0.0.1',
socket: { remoteAddress: '127.0.0.1' },
headers: {
'x-forwarded-for': '8.8.8.8, 192.168.1.1',
'x-api-key': validApiKey,
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
});
describe('公网IP访问 - 未配置API_KEY', () => {
beforeEach(() => {
jest.spyOn(configService, 'get').mockReturnValue(undefined);
});
it('应该允许所有公网访问(开发环境)', () => {
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该允许未提供API Key的公网访问', () => {
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
});
describe('内网IP访问', () => {
beforeEach(() => {
jest.spyOn(configService, 'get').mockReturnValue('test-api-key');
});
it('应该要求内网IP提供API Key', () => {
mockRequest = {
ip: '192.168.1.100',
socket: { remoteAddress: '192.168.1.100' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
it('应该允许提供正确API Key的内网访问', () => {
mockRequest = {
ip: '192.168.1.100',
socket: { remoteAddress: '192.168.1.100' },
headers: {
'x-api-key': 'test-api-key',
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
const result = guard.canActivate(mockExecutionContext);
expect(result).toBe(true);
});
it('应该要求 10.x.x.x 网段提供API Key', () => {
mockRequest = {
ip: '10.0.0.1',
socket: { remoteAddress: '10.0.0.1' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
it('应该要求 172.16-31.x.x 网段提供API Key', () => {
mockRequest = {
ip: '172.16.0.1',
socket: { remoteAddress: '172.16.0.1' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
});
describe('边界情况', () => {
beforeEach(() => {
jest.spyOn(configService, 'get').mockReturnValue('test-api-key');
});
it('应该处理 unknown IP 地址', () => {
mockRequest = {
ip: 'unknown',
socket: { remoteAddress: 'unknown' },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
it('应该处理缺少 IP 信息的请求', () => {
mockRequest = {
ip: undefined,
socket: { remoteAddress: undefined },
headers: {},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
it('应该处理 API Key 大小写敏感', () => {
jest.spyOn(configService, 'get').mockReturnValue('Test-API-Key');
mockRequest = {
ip: '8.8.8.8',
socket: { remoteAddress: '8.8.8.8' },
headers: {
'x-api-key': 'test-api-key', // 小写,应该被拒绝
},
};
mockExecutionContext = createMockExecutionContext(mockRequest);
expect(() => guard.canActivate(mockExecutionContext)).toThrow(
UnauthorizedException,
);
});
});
});

View File

@@ -0,0 +1,72 @@
import {
CanActivate,
ExecutionContext,
Injectable,
UnauthorizedException,
Logger,
} from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { Request } from 'express';
import { UsersService } from '../../users/users.service';
@Injectable()
export class AuthGuard implements CanActivate {
private readonly logger = new Logger(AuthGuard.name);
constructor(
private configService: ConfigService,
private usersService: UsersService,
) {}
async canActivate(context: ExecutionContext): Promise<boolean> {
const request = context.switchToHttp().getRequest<Request>();
// 检查是否启用 Basic Auth
const enableBasicAuth =
this.configService.get<string>('ENABLE_BASIC_AUTH') === 'true';
this.logger.log(`Basic Auth enabled: ${enableBasicAuth}`);
if (!enableBasicAuth) {
// 如果未启用 Basic Auth允许所有访问
return true;
}
// 解析 Authorization header
const authHeader = request.headers['authorization'] as string;
if (!authHeader || !authHeader.startsWith('Basic ')) {
this.logger.warn('Missing or invalid Authorization header');
throw new UnauthorizedException('Missing or invalid Authorization header');
}
// 解码 Basic Auth
const base64Credentials = authHeader.split(' ')[1];
const credentials = Buffer.from(base64Credentials, 'base64').toString(
'utf-8',
);
const [username, password] = credentials.split(':');
if (!username || !password) {
this.logger.warn('Invalid credentials format');
throw new UnauthorizedException('Invalid credentials format');
}
this.logger.log(`Attempting login for user: ${username}`);
// 验证用户
const user = await this.usersService.validateUser(username, password);
if (!user) {
this.logger.warn(`Login failed for user: ${username} - Invalid username or password`);
throw new UnauthorizedException('Invalid username or password');
}
this.logger.log(`User ${username} logged in successfully`);
// 将用户信息附加到请求对象
(request as any).user = user;
return true;
}
}

View File

@@ -0,0 +1,15 @@
import { Module } from '@nestjs/common';
import { APP_GUARD } from '@nestjs/core';
import { AuthGuard } from './auth.guard';
import { UsersModule } from '../../users/users.module';
@Module({
imports: [UsersModule],
providers: [
{
provide: APP_GUARD,
useClass: AuthGuard,
},
],
})
export class AuthModule {}

View File

@@ -1,6 +1,21 @@
import { Injectable, LoggerService, Scope } from '@nestjs/common';
import { winstonLogger } from './winston.config';
type LogMessage = string | Error | Record<string, unknown>;
function formatMessage(message: LogMessage): string {
if (typeof message === 'string') {
return message;
}
if (message instanceof Error) {
return message.message;
}
if (typeof message === 'object' && message !== null) {
return JSON.stringify(message);
}
return String(message);
}
@Injectable({ scope: Scope.TRANSIENT })
export class CustomLogger implements LoggerService {
private context?: string;
@@ -9,23 +24,34 @@ export class CustomLogger implements LoggerService {
this.context = context;
}
log(message: any, context?: string) {
winstonLogger.info(message, { context: context || this.context });
log(message: LogMessage, context?: string) {
winstonLogger.info(formatMessage(message), {
context: context || this.context,
});
}
error(message: any, trace?: string, context?: string) {
winstonLogger.error(message, { context: context || this.context, trace });
error(message: LogMessage, trace?: string, context?: string) {
winstonLogger.error(formatMessage(message), {
context: context || this.context,
trace,
});
}
warn(message: any, context?: string) {
winstonLogger.warn(message, { context: context || this.context });
warn(message: LogMessage, context?: string) {
winstonLogger.warn(formatMessage(message), {
context: context || this.context,
});
}
debug(message: any, context?: string) {
winstonLogger.debug(message, { context: context || this.context });
debug(message: LogMessage, context?: string) {
winstonLogger.debug(formatMessage(message), {
context: context || this.context,
});
}
verbose(message: any, context?: string) {
winstonLogger.verbose(message, { context: context || this.context });
verbose(message: LogMessage, context?: string) {
winstonLogger.verbose(formatMessage(message), {
context: context || this.context,
});
}
}

View File

@@ -0,0 +1,29 @@
import { Injectable, NestMiddleware } from '@nestjs/common';
import { Request, Response, NextFunction } from 'express';
import { CustomLogger } from './logger.service';
@Injectable()
export class LoggingMiddleware implements NestMiddleware {
constructor(private readonly logger: CustomLogger) {
this.logger.setContext('HTTP');
}
use(req: Request, res: Response, next: NextFunction) {
const { method, originalUrl, ip } = req;
const userAgent = req.get('user-agent') || '';
const startTime = Date.now();
// 收到请求时立即输出
this.logger.debug(`--> ${method} ${originalUrl} - ${ip} - ${userAgent}`);
res.on('finish', () => {
const { statusCode } = res;
const duration = Date.now() - startTime;
this.logger.debug(
`<-- ${method} ${originalUrl} ${statusCode} - ${duration}ms`,
);
});
next();
}
}

View File

@@ -16,13 +16,33 @@ const logFormat = winston.format.combine(
winston.format.errors({ stack: true }),
winston.format.splat(),
winston.format.printf(({ timestamp, level, message, context, stack }) => {
let log = `${timestamp} [${level}]`;
if (context) {
log += ` [${context}]`;
}
log += ` ${message}`;
const timestampStr =
typeof timestamp === 'string' ? timestamp : String(timestamp);
const levelStr = typeof level === 'string' ? level : String(level);
const messageStr = typeof message === 'string' ? message : String(message);
const contextStr = context
? typeof context === 'string'
? context
: JSON.stringify(context)
: '';
let stackStr = '';
if (stack) {
log += `\n${stack}`;
if (typeof stack === 'string') {
stackStr = stack;
} else if (typeof stack === 'object' && stack !== null) {
stackStr = JSON.stringify(stack);
} else {
stackStr = String(stack);
}
}
let log = `${timestampStr} [${levelStr}]`;
if (contextStr) {
log += ` [${contextStr}]`;
}
log += ` ${messageStr}`;
if (stackStr) {
log += `\n${stackStr}`;
}
return log;
}),
@@ -30,10 +50,7 @@ const logFormat = winston.format.combine(
// 控制台传输
const consoleTransport = new winston.transports.Console({
format: winston.format.combine(
winston.format.colorize(),
logFormat,
),
format: winston.format.combine(winston.format.colorize(), logFormat),
});
// 应用日志传输(按天轮转)
@@ -43,7 +60,6 @@ const appLogTransport = new DailyRotateFile({
datePattern: 'YYYY-MM-DD',
maxSize: '20m',
maxFiles: '30d',
format: logFormat,
});
// 错误日志传输(按天轮转)
@@ -51,10 +67,9 @@ const errorLogTransport = new DailyRotateFile({
dirname: logDir,
filename: 'error-%DATE%.log',
datePattern: 'YYYY-MM-DD',
level: 'error',
maxSize: '20m',
maxFiles: '30d',
format: logFormat,
level: 'error',
});
// 创建 winston logger 实例
@@ -63,8 +78,8 @@ export const winstonLogger = winston.createLogger({
format: logFormat,
transports: [
consoleTransport,
appLogTransport,
errorLogTransport,
appLogTransport as any,
errorLogTransport as any,
],
exitOnError: false,
});

View File

@@ -0,0 +1,143 @@
/**
* 时区工具函数
* 统一处理东八区(Asia/Shanghai)时间相关的操作
*/
const TIMEZONE_OFFSET = 8 * 60 * 60 * 1000;
/**
* 将北京时间(+8)转换为UTC
* 用于将爬取的北京时间字符串解析后的Date对象转为UTC存储
* @param date 北京时间的Date对象
* @returns UTC时间的Date对象
*/
export function beijingToUtc(date: Date): Date {
return new Date(date.getTime() - TIMEZONE_OFFSET);
}
/**
* 将UTC时间转换为北京时间(+8)
* 用于将数据库中的UTC时间转为北京时间显示
* @param date UTC时间的Date对象
* @returns 北京时间的Date对象
*/
export function utcToBeijing(date: Date): Date {
return new Date(date.getTime() + TIMEZONE_OFFSET);
}
/**
* 获取当前时间的东八区Date对象
* @returns Date 当前时间的东八区表示
*/
export function getCurrentDateInTimezone(): Date {
const now = new Date();
const utc = now.getTime() + now.getTimezoneOffset() * 60 * 1000;
return new Date(utc + TIMEZONE_OFFSET);
}
/**
* 将任意Date对象转换为东八区时间
* @param date 原始Date对象
* @returns Date 转换后的东八区时间
*/
export function convertToTimezone(date: Date): Date {
const utc = date.getTime() + date.getTimezoneOffset() * 60 * 1000;
return new Date(utc + TIMEZONE_OFFSET);
}
/**
* 格式化日期为 YYYY-MM-DD 格式
* @param date Date对象
* @returns 格式化后的日期字符串
*/
export function formatDate(date: Date): string {
const timezoneDate = convertToTimezone(date);
const year = timezoneDate.getFullYear();
const month = String(timezoneDate.getMonth() + 1).padStart(2, '0');
const day = String(timezoneDate.getDate()).padStart(2, '0');
return `${year}-${month}-${day}`;
}
/**
* 格式化日期时间为 YYYY-MM-DD HH:mm:ss 格式
* @param date Date对象
* @returns 格式化后的日期时间字符串
*/
export function formatDateTime(date: Date): string {
const timezoneDate = convertToTimezone(date);
const year = timezoneDate.getFullYear();
const month = String(timezoneDate.getMonth() + 1).padStart(2, '0');
const day = String(timezoneDate.getDate()).padStart(2, '0');
const hours = String(timezoneDate.getHours()).padStart(2, '0');
const minutes = String(timezoneDate.getMinutes()).padStart(2, '0');
const seconds = String(timezoneDate.getSeconds()).padStart(2, '0');
return `${year}-${month}-${day} ${hours}:${minutes}:${seconds}`;
}
/**
* 设置时间为当天的开始时间 (00:00:00.000)
* @param date Date对象
* @returns 设置后的Date对象
*/
export function setStartOfDay(date: Date): Date {
const timezoneDate = convertToTimezone(date);
timezoneDate.setHours(0, 0, 0, 0);
return timezoneDate;
}
/**
* 设置时间为当天的结束时间 (23:59:59.999)
* @param date Date对象
* @returns 设置后的Date对象
*/
export function setEndOfDay(date: Date): Date {
const timezoneDate = convertToTimezone(date);
timezoneDate.setHours(23, 59, 59, 999);
return timezoneDate;
}
/**
* 获取指定天数前的日期
* @param days 天数
* @returns 指定天数前的Date对象
*/
export function getDaysAgo(days: number): Date {
const date = getCurrentDateInTimezone();
date.setDate(date.getDate() - days);
return date;
}
/**
* 解析日期字符串为东八区Date对象
* @param dateStr 日期字符串 (支持 YYYY-MM-DD 或 YYYY-MM-DD HH:mm:ss 格式)
* @returns 解析后的Date对象
*/
export function parseDateString(dateStr: string): Date {
const date = new Date(dateStr);
return convertToTimezone(date);
}
/**
* 将UTC时间转换为北京时间的ISO字符串格式
* 用于API返回确保前端接收到的时间字符串已经是北京时间
* @param date UTC时间的Date对象
* @returns 北京时间的ISO字符串 (格式: YYYY-MM-DDTHH:mm:ss+08:00)
*/
export function utcToBeijingISOString(date: Date): string {
// 获取UTC时间戳毫秒
const utcTimestamp = date.getTime();
// 计算北京时间戳UTC + 8小时
const beijingTimestamp = utcTimestamp + TIMEZONE_OFFSET;
// 创建UTC Date对象来格式化避免本地时区影响
const beijingDate = new Date(beijingTimestamp);
// 使用UTC方法获取时间组件确保不受本地时区影响
const year = beijingDate.getUTCFullYear();
const month = String(beijingDate.getUTCMonth() + 1).padStart(2, '0');
const day = String(beijingDate.getUTCDate()).padStart(2, '0');
const hours = String(beijingDate.getUTCHours()).padStart(2, '0');
const minutes = String(beijingDate.getUTCMinutes()).padStart(2, '0');
const seconds = String(beijingDate.getUTCSeconds()).padStart(2, '0');
const milliseconds = String(beijingDate.getUTCMilliseconds()).padStart(3, '0');
return `${year}-${month}-${day}T${hours}:${minutes}:${seconds}.${milliseconds}+08:00`;
}

View File

@@ -1,15 +1,19 @@
import { Controller, Post, Get } from '@nestjs/common';
import { Controller, Post, Get, Param, Body } from '@nestjs/common';
import { BidCrawlerService } from './services/bid-crawler.service';
@Controller('api/crawler')
export class CrawlerController {
private isCrawling = false;
private crawlingSources = new Set<string>();
constructor(private readonly crawlerService: BidCrawlerService) {}
@Get('status')
getStatus() {
return { isCrawling: this.isCrawling };
return {
isCrawling: this.isCrawling,
crawlingSources: Array.from(this.crawlingSources),
};
}
@Post('run')
@@ -17,15 +21,15 @@ export class CrawlerController {
if (this.isCrawling) {
return { message: 'Crawl is already running' };
}
this.isCrawling = true;
// We don't await this because we want it to run in the background
// We don't await this because we want it to run in the background
// and return immediately, or we can await if we want to user to wait.
// Given the requirement "Immediate Crawl", usually implies triggering it.
// However, for a better UI experience, we might want to wait or just trigger.
// Let's await it so that user knows when it's done (or failed),
// assuming it doesn't take too long for the mock.
// Let's await it so that user knows when it's done (or failed),
// assuming it doesn't take too long for the mock.
// Real crawling might take long, so background is better.
// For this prototype, I'll await it to show completion.
try {
@@ -35,4 +39,30 @@ export class CrawlerController {
this.isCrawling = false;
}
}
@Post('crawl/:sourceName')
async crawlSingleSource(@Param('sourceName') sourceName: string) {
if (this.crawlingSources.has(sourceName)) {
return { message: `Source ${sourceName} is already being crawled` };
}
this.crawlingSources.add(sourceName);
try {
// 设置状态为正在更新count = -1
await this.crawlerService.updateCrawlStatus(sourceName, -1);
const result = await this.crawlerService.crawlSingleSource(sourceName);
// 更新状态为实际数量
await this.crawlerService.updateCrawlStatus(
sourceName,
result.success ? result.count : 0,
);
return result;
} finally {
this.crawlingSources.delete(sourceName);
}
}
}

View File

@@ -1,4 +1,9 @@
import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn } from 'typeorm';
import {
Entity,
PrimaryGeneratedColumn,
Column,
CreateDateColumn,
} from 'typeorm';
@Entity('crawl_info_add')
export class CrawlInfoAdd {
@@ -15,7 +20,7 @@ export class CrawlInfoAdd {
latestPublishDate: Date | null;
@Column({ type: 'text', nullable: true })
error: string;
error: string | null;
@CreateDateColumn()
createdAt: Date;

View File

@@ -4,6 +4,7 @@ import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import * as puppeteer from 'puppeteer';
import { BidsService } from '../../bids/services/bid.service';
import { beijingToUtc } from '../../common/utils/timezone.util';
import { CrawlInfoAdd } from '../entities/crawl-info-add.entity';
import { ChdtpCrawler } from './chdtp_target';
import { ChngCrawler } from './chng_target';
@@ -18,6 +19,26 @@ import { PowerbeijingCrawler } from './powerbeijing_target';
import { SdiccCrawler } from './sdicc_target';
import { CnoocCrawler } from './cnooc_target';
interface CrawlResult {
title: string;
publishDate: Date;
url: string;
}
type AnyCrawler =
| typeof ChdtpCrawler
| typeof ChngCrawler
| typeof SzecpCrawler
| typeof CdtCrawler
| typeof EpsCrawler
| typeof CnncecpCrawler
| typeof CgnpcCrawler
| typeof CeicCrawler
| typeof EspicCrawler
| typeof PowerbeijingCrawler
| typeof SdiccCrawler
| typeof CnoocCrawler;
@Injectable()
export class BidCrawlerService {
private readonly logger = new Logger(BidCrawlerService.name);
@@ -31,17 +52,15 @@ export class BidCrawlerService {
async crawlAll() {
this.logger.log('Starting crawl task with Puppeteer...');
// 设置最大执行时间为3小时
const maxExecutionTime = 3 * 60 * 60 * 1000; // 3小时毫秒
const startTime = Date.now();
// 统计结果
const crawlResults: Record<string, { success: number; error?: string }> = {};
const crawlResults: Record<string, { success: number; error?: string }> =
{};
// 记录数据为0的爬虫用于重试
const zeroDataCrawlers: any[] = [];
const zeroDataCrawlers: AnyCrawler[] = [];
// 从环境变量读取代理配置
const proxyHost = this.configService.get<string>('PROXY_HOST');
const proxyPort = this.configService.get<string>('PROXY_PORT');
@@ -60,9 +79,10 @@ export class BidCrawlerService {
];
if (proxyHost && proxyPort) {
const proxyUrl = proxyUsername && proxyPassword
? `http://${proxyUsername}:${proxyPassword}@${proxyHost}:${proxyPort}`
: `http://${proxyHost}:${proxyPort}`;
const proxyUrl =
proxyUsername && proxyPassword
? `http://${proxyUsername}:${proxyPassword}@${proxyHost}:${proxyPort}`
: `http://${proxyHost}:${proxyPort}`;
args.push(`--proxy-server=${proxyUrl}`);
this.logger.log(`Using proxy: ${proxyHost}:${proxyPort}`);
}
@@ -72,24 +92,43 @@ export class BidCrawlerService {
args,
});
const crawlers = [ChdtpCrawler, ChngCrawler, SzecpCrawler, CdtCrawler, EpsCrawler, CnncecpCrawler, CgnpcCrawler, CeicCrawler, EspicCrawler, PowerbeijingCrawler, SdiccCrawler, CnoocCrawler];
const crawlers = [
ChdtpCrawler,
ChngCrawler,
SzecpCrawler,
CdtCrawler,
EpsCrawler,
CnncecpCrawler,
CgnpcCrawler,
CeicCrawler,
EspicCrawler,
PowerbeijingCrawler,
SdiccCrawler,
CnoocCrawler,
];
try {
for (const crawler of crawlers) {
this.logger.log(`Crawling: ${crawler.name}`);
// 检查是否超时
const elapsedTime = Date.now() - startTime;
if (elapsedTime > maxExecutionTime) {
this.logger.warn(`⚠️ Crawl task exceeded maximum execution time of 3 hours. Stopping...`);
this.logger.warn(`⚠️ Total elapsed time: ${Math.floor(elapsedTime / 1000 / 60)} minutes`);
this.logger.warn(
`⚠️ Crawl task exceeded maximum execution time of 3 hours. Stopping...`,
);
this.logger.warn(
`⚠️ Total elapsed time: ${Math.floor(elapsedTime / 1000 / 60)} minutes`,
);
break;
}
try {
const results = await crawler.crawl(browser);
this.logger.log(`Extracted ${results.length} items from ${crawler.name}`);
const results = await (crawler as any).crawl(browser);
this.logger.log(
`Extracted ${results.length} items from ${crawler.name}`,
);
// 记录成功数量
crawlResults[crawler.name] = { success: results.length };
@@ -99,64 +138,81 @@ export class BidCrawlerService {
}
// 获取最新的发布日期
const latestPublishDate = results.length > 0
? results.reduce((latest, item) => {
const itemDate = new Date(item.publishDate);
return itemDate > latest ? itemDate : latest;
}, new Date(0))
: null;
for (const item of results) {
await this.bidsService.createOrUpdate({
title: item.title,
url: item.url,
publishDate: item.publishDate,
source: crawler.name,
});
}
// 保存爬虫统计信息到数据库
await this.saveCrawlInfo(crawler.name, results.length, latestPublishDate);
} catch (err) {
this.logger.error(`Error crawling ${crawler.name}: ${err.message}`);
// 记录错误信息
crawlResults[crawler.name] = { success: 0, error: err.message };
// 保存错误信息到数据库
await this.saveCrawlInfo(crawler.name, 0, null, err.message);
}
}
// 对数据为0的爬虫进行重试
if (zeroDataCrawlers.length > 0) {
this.logger.log(`Retrying ${zeroDataCrawlers.length} crawlers with zero data...`);
for (const crawler of zeroDataCrawlers) {
this.logger.log(`Retrying: ${crawler.name}`);
// 检查是否超时
const elapsedTime = Date.now() - startTime;
if (elapsedTime > maxExecutionTime) {
this.logger.warn(`⚠️ Crawl task exceeded maximum execution time of 3 hours. Stopping retry...`);
this.logger.warn(`⚠️ Total elapsed time: ${Math.floor(elapsedTime / 1000 / 60)} minutes`);
break;
}
try {
const results = await crawler.crawl(browser);
this.logger.log(`Retry extracted ${results.length} items from ${crawler.name}`);
// 更新统计结果
crawlResults[crawler.name] = { success: results.length };
// 获取最新的发布日期
const latestPublishDate = results.length > 0
const latestPublishDate =
results.length > 0
? results.reduce((latest, item) => {
const itemDate = new Date(item.publishDate);
return itemDate > latest ? itemDate : latest;
}, new Date(0))
: null;
for (const item of results) {
// 将北京时间转换为UTC存储
const publishDateUtc = beijingToUtc(new Date(item.publishDate));
await this.bidsService.createOrUpdate({
title: item.title,
url: item.url,
publishDate: publishDateUtc,
source: crawler.name,
});
}
// 保存爬虫统计信息到数据库将北京时间转为UTC
await this.saveCrawlInfo(
crawler.name,
results.length,
latestPublishDate ? beijingToUtc(latestPublishDate) : null,
);
} catch (err) {
const errorMessage = err instanceof Error ? err.message : String(err);
this.logger.error(`Error crawling ${crawler.name}: ${errorMessage}`);
// 记录错误信息
crawlResults[crawler.name] = { success: 0, error: errorMessage };
// 保存错误信息到数据库
await this.saveCrawlInfo(crawler.name, 0, null, errorMessage);
}
}
// 对数据为0的爬虫进行重试
if (zeroDataCrawlers.length > 0) {
this.logger.log(
`Retrying ${zeroDataCrawlers.length} crawlers with zero data...`,
);
for (const crawler of zeroDataCrawlers) {
this.logger.log(`Retrying: ${crawler.name}`);
// 检查是否超时
const elapsedTime = Date.now() - startTime;
if (elapsedTime > maxExecutionTime) {
this.logger.warn(
`⚠️ Crawl task exceeded maximum execution time of 3 hours. Stopping retry...`,
);
this.logger.warn(
`⚠️ Total elapsed time: ${Math.floor(elapsedTime / 1000 / 60)} minutes`,
);
break;
}
try {
const results = await (crawler as any).crawl(browser);
this.logger.log(
`Retry extracted ${results.length} items from ${crawler.name}`,
);
// 更新统计结果
crawlResults[crawler.name] = { success: results.length };
// 获取最新的发布日期
const latestPublishDate =
results.length > 0
? results.reduce((latest, item) => {
const itemDate = new Date(item.publishDate);
return itemDate > latest ? itemDate : latest;
}, new Date(0))
: null;
for (const item of results) {
await this.bidsService.createOrUpdate({
title: item.title,
@@ -166,56 +222,191 @@ export class BidCrawlerService {
});
}
// 更新爬虫统计信息到数据库
await this.saveCrawlInfo(crawler.name, results.length, latestPublishDate);
// 更新爬虫统计信息到数据库将北京时间转为UTC
await this.saveCrawlInfo(
crawler.name,
results.length,
latestPublishDate ? beijingToUtc(latestPublishDate) : null,
);
} catch (err) {
this.logger.error(`Error retrying ${crawler.name}: ${err.message}`);
const errorMessage =
err instanceof Error ? err.message : String(err);
this.logger.error(
`Error retrying ${crawler.name}: ${errorMessage}`,
);
// 记录错误信息
crawlResults[crawler.name] = { success: 0, error: err.message };
crawlResults[crawler.name] = { success: 0, error: errorMessage };
// 更新错误信息到数据库
await this.saveCrawlInfo(crawler.name, 0, null, err.message);
await this.saveCrawlInfo(crawler.name, 0, null, errorMessage);
}
}
}
} catch (error) {
this.logger.error(`Crawl task failed: ${error.message}`);
const errorMessage =
error instanceof Error ? error.message : String(error);
this.logger.error(`Crawl task failed: ${errorMessage}`);
} finally {
await browser.close();
const totalTime = Date.now() - startTime;
const minutes = Math.floor(totalTime / 1000 / 60);
this.logger.log(`Crawl task finished. Total time: ${minutes} minutes`);
if (totalTime > maxExecutionTime) {
this.logger.warn(`⚠️ Crawl task exceeded maximum execution time of 3 hours.`);
this.logger.warn(
`⚠️ Crawl task exceeded maximum execution time of 3 hours.`,
);
}
// 输出统计总结
this.logger.log('='.repeat(50));
this.logger.log('爬虫执行总结 / Crawl Summary');
this.logger.log('='.repeat(50));
let totalSuccess = 0;
let errorCount = 0;
for (const [source, result] of Object.entries(crawlResults)) {
if (result.error) {
this.logger.error(`${source}: 出错 - ${result.error}`);
errorCount++;
} else {
this.logger.log(`${source}: 成功获取 ${result.success} 条工程信息`);
this.logger.log(
`${source}: 成功获取 ${result.success} 条工程信息`,
);
totalSuccess += result.success;
}
}
this.logger.log('='.repeat(50));
this.logger.log(`总计: ${totalSuccess} 条工程信息, ${errorCount} 个来源出错`);
this.logger.log(`Total: ${totalSuccess} items, ${errorCount} sources failed`);
this.logger.log(
`总计: ${totalSuccess} 条工程信息, ${errorCount} 个来源出错`,
);
this.logger.log(
`Total: ${totalSuccess} items, ${errorCount} sources failed`,
);
this.logger.log('='.repeat(50));
}
}
async crawlSingleSource(sourceName: string) {
this.logger.log(`Starting single source crawl for: ${sourceName}`);
// 从环境变量读取代理配置
const proxyHost = this.configService.get<string>('PROXY_HOST');
const proxyPort = this.configService.get<string>('PROXY_PORT');
const proxyUsername = this.configService.get<string>('PROXY_USERNAME');
const proxyPassword = this.configService.get<string>('PROXY_PASSWORD');
// 构建代理参数
const args = [
'--no-sandbox',
'--disable-setuid-sandbox',
'--disable-blink-features=AutomationControlled',
'--disable-infobars',
'--window-position=0,0',
'--ignore-certifcate-errors',
'--ignore-certifcate-errors-spki-list',
];
if (proxyHost && proxyPort) {
const proxyUrl =
proxyUsername && proxyPassword
? `http://${proxyUsername}:${proxyPassword}@${proxyHost}:${proxyPort}`
: `http://${proxyHost}:${proxyPort}`;
args.push(`--proxy-server=${proxyUrl}`);
this.logger.log(`Using proxy: ${proxyHost}:${proxyPort}`);
}
const browser = await puppeteer.launch({
headless: false,
args,
});
const crawlers = [
ChdtpCrawler,
ChngCrawler,
SzecpCrawler,
CdtCrawler,
EpsCrawler,
CnncecpCrawler,
CgnpcCrawler,
CeicCrawler,
EspicCrawler,
PowerbeijingCrawler,
SdiccCrawler,
CnoocCrawler,
];
const targetCrawler = crawlers.find((c) => c.name === sourceName);
if (!targetCrawler) {
await browser.close();
throw new Error(`Crawler not found for source: ${sourceName}`);
}
try {
this.logger.log(`Crawling: ${targetCrawler.name}`);
const results = await (targetCrawler as any).crawl(browser);
this.logger.log(
`Extracted ${results.length} items from ${targetCrawler.name}`,
);
// 获取最新的发布日期
const latestPublishDate =
results.length > 0
? results.reduce((latest, item) => {
const itemDate = new Date(item.publishDate);
return itemDate > latest ? itemDate : latest;
}, new Date(0))
: null;
for (const item of results) {
// 将北京时间转换为UTC存储
const publishDateUtc = beijingToUtc(new Date(item.publishDate));
await this.bidsService.createOrUpdate({
title: item.title,
url: item.url,
publishDate: publishDateUtc,
source: targetCrawler.name,
});
}
// 保存爬虫统计信息到数据库将北京时间转为UTC
await this.saveCrawlInfo(
targetCrawler.name,
results.length,
latestPublishDate ? beijingToUtc(latestPublishDate) : null,
);
return {
success: true,
source: targetCrawler.name,
count: results.length,
latestPublishDate,
};
} catch (err) {
const errorMessage = err instanceof Error ? err.message : String(err);
this.logger.error(
`Error crawling ${targetCrawler.name}: ${errorMessage}`,
);
// 保存错误信息到数据库
await this.saveCrawlInfo(targetCrawler.name, 0, null, errorMessage);
return {
success: false,
source: targetCrawler.name,
count: 0,
error: errorMessage,
};
} finally {
await browser.close();
}
}
private async saveCrawlInfo(
source: string,
count: number,
@@ -227,12 +418,50 @@ export class BidCrawlerService {
source,
count,
latestPublishDate,
error,
// 确保 error 字段正确处理undefined 或空字符串都转换为 null
error: error && error.trim() !== '' ? error : null,
});
await this.crawlInfoRepository.save(crawlInfo);
this.logger.log(`Saved crawl info for ${source}: ${count} items`);
} catch (err) {
this.logger.error(`Failed to save crawl info for ${source}: ${err.message}`);
const errorMessage = err instanceof Error ? err.message : String(err);
this.logger.error(
`Failed to save crawl info for ${source}: ${errorMessage}`,
);
}
}
// 更新爬虫状态count = -1 表示正在更新
async updateCrawlStatus(source: string, count: number) {
try {
// 使用原生查询实现 upsert 逻辑
await this.crawlInfoRepository.manager.transaction(
async (manager) => {
// 检查记录是否存在
const existing = await manager.findOne(CrawlInfoAdd, {
where: { source },
});
if (existing) {
// 更新现有记录
await manager.update(CrawlInfoAdd, { source }, { count });
} else {
// 插入新记录
await manager.save(CrawlInfoAdd, {
source,
count,
latestPublishDate: null,
error: null,
});
}
},
);
this.logger.log(`Updated crawl status for ${source}: ${count}`);
} catch (err) {
const errorMessage = err instanceof Error ? err.message : String(err);
this.logger.error(
`Failed to update crawl status for ${source}: ${errorMessage}`,
);
}
}
}

View File

@@ -2,7 +2,7 @@ import { CdtCrawler } from './cdt_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('CdtCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('CdtCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${CdtCrawler.name}`);
console.log(`Target URL: ${CdtCrawler.url}`);
const results = await CdtCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('CdtCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -13,11 +13,11 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -31,19 +31,69 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export interface CdtResult {
@@ -52,12 +102,22 @@ export interface CdtResult {
url: string;
}
interface CdtCrawlerType {
name: string;
url: string;
baseUrl: string;
extract(html: string): CdtResult[];
}
export const CdtCrawler = {
name: '中国大唐集团电子商务平台',
url: 'https://tang.cdt-ec.com/home/index.html',
baseUrl: 'https://tang.cdt-ec.com',
async crawl(browser: puppeteer.Browser): Promise<CdtResult[]> {
async crawl(
this: CdtCrawlerType,
browser: puppeteer.Browser,
): Promise<CdtResult[]> {
const logger = new Logger('CdtCrawler');
const page = await browser.newPage();
@@ -67,7 +127,9 @@ export const CdtCrawler = {
await page.authenticate({ username, password });
}
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
);
const allResults: CdtResult[] = [];
let currentPage = 1;
@@ -75,7 +137,17 @@ export const CdtCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
@@ -86,19 +158,26 @@ export const CdtCrawler = {
// 点击"招标公告"标签
logger.log('Looking for "招标公告" tab...');
await page.waitForFunction(() => {
const tabs = Array.from(document.querySelectorAll('span.notice-tab'));
return tabs.some(tab => tab.textContent && tab.textContent.includes('招标公告'));
}, { timeout: 60000 });
await page.waitForFunction(
() => {
const tabs = Array.from(document.querySelectorAll('span.notice-tab'));
return tabs.some(
(tab) => tab.textContent && tab.textContent.includes('招标公告'),
);
},
{ timeout: 60000 },
);
await page.evaluate(() => {
const tabs = Array.from(document.querySelectorAll('span.notice-tab'));
const target = tabs.find(tab => tab.textContent && tab.textContent.includes('招标公告')) as HTMLElement;
const target = tabs.find(
(tab) => tab.textContent && tab.textContent.includes('招标公告'),
) as HTMLElement;
if (target) target.click();
});
logger.log('Clicked "招标公告" tab.');
await new Promise(r => setTimeout(r, 2000));
await new Promise((r) => setTimeout(r, 2000));
// 模拟人类行为
logger.log('Simulating human mouse movements...');
@@ -109,26 +188,43 @@ export const CdtCrawler = {
// 点击"招标公告"下的"更多+"链接
logger.log('Looking for "更多+" link under "招标公告"...');
await page.waitForFunction(() => {
const titles = Array.from(document.querySelectorAll('span.h-notice-title'));
return titles.some(title => title.textContent && title.textContent.includes('招标公告'));
}, { timeout: 30000 });
await page.waitForFunction(
() => {
const titles = Array.from(
document.querySelectorAll('span.h-notice-title'),
);
return titles.some(
(title) =>
title.textContent && title.textContent.includes('招标公告'),
);
},
{ timeout: 30000 },
);
await page.evaluate(() => {
const titles = Array.from(document.querySelectorAll('span.h-notice-title'));
const targetTitle = titles.find(title => title.textContent && title.textContent.includes('招标公告'));
const titles = Array.from(
document.querySelectorAll('span.h-notice-title'),
);
const targetTitle = titles.find(
(title) =>
title.textContent && title.textContent.includes('招标公告'),
);
if (targetTitle) {
const parent = targetTitle.parentElement;
if (parent) {
const moreLink = parent.querySelector('a.h-notice-more') as HTMLElement;
const moreLink = parent.querySelector(
'a.h-notice-more',
) as HTMLElement;
if (moreLink) moreLink.click();
}
}
});
logger.log('Clicked "更多+" link under "招标公告".');
await page.waitForNavigation({ waitUntil: 'networkidle2', timeout: 60000 }).catch(() => {});
await new Promise(r => setTimeout(r, 3000));
await page
.waitForNavigation({ waitUntil: 'networkidle2', timeout: 60000 })
.catch(() => {});
await new Promise((r) => setTimeout(r, 3000));
// 模拟人类行为
logger.log('Simulating human mouse movements...');
@@ -155,7 +251,9 @@ export const CdtCrawler = {
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
// 模拟人类行为 - 翻页前
logger.log('Simulating human mouse movements before pagination...');
@@ -172,7 +270,9 @@ export const CdtCrawler = {
}, nextButtonSelector);
if (!nextButtonExists) {
logger.log('Next page button not found or disabled. Reached end of list.');
logger.log(
'Next page button not found or disabled. Reached end of list.',
);
break;
}
@@ -186,18 +286,25 @@ export const CdtCrawler = {
}, nextButtonSelector);
// 等待 AJAX 请求完成(通过监听网络请求)
await page.waitForFunction(() => {
// 检查表格是否正在加载
const loading = document.querySelector('.layui-table-loading');
return !loading;
}, { timeout: 30000 }).catch(() => {});
await page
.waitForFunction(
() => {
// 检查表格是否正在加载
const loading = document.querySelector('.layui-table-loading');
return !loading;
},
{ timeout: 30000 },
)
.catch(() => {});
// 额外等待确保数据加载完成
await new Promise(r => setTimeout(r, 2000));
await new Promise((r) => setTimeout(r, 2000));
// 检查是否真的翻页了(通过检查当前页码)
const currentActivePage = await page.evaluate(() => {
const activeSpan = document.querySelector('.layui-laypage-curr em:last-child');
const activeSpan = document.querySelector(
'.layui-laypage-curr em:last-child',
);
return activeSpan ? parseInt(activeSpan.textContent || '1') : 1;
});
@@ -217,25 +324,29 @@ export const CdtCrawler = {
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): CdtResult[] {
extract(this: CdtCrawlerType, html: string): CdtResult[] {
const results: CdtResult[] = [];
/**
* Regex groups for tang.cdt-ec.com:
@@ -243,23 +354,24 @@ export const CdtCrawler = {
* 2: Title (项目名称)
* 3: Date (发布时间)
*/
const regex = /<tr[^>]*data-index="[^"]*"[^>]*>[\s\S]*?<a[^>]*class="layui-table-link"[^>]*href="([^"]*)"[^>]*>([^<]*)<\/a>[\s\S]*?<td[^>]*data-field="publish_time"[^>]*>[\s\S]*?<div[^>]*class="layui-table-cell[^"]*"[^>]*>([^<]*)<\/div>[\s\S]*?<\/td>[\s\S]*?<\/tr>/gs;
const regex =
/<tr[^>]*data-index="[^"]*"[^>]*>[\s\S]*?<a[^>]*class="layui-table-link"[^>]*href="([^"]*)"[^>]*>([^<]*)<\/a>[\s\S]*?<td[^>]*data-field="publish_time"[^>]*>[\s\S]*?<div[^>]*class="layui-table-cell[^"]*"[^>]*>([^<]*)<\/div>[\s\S]*?<\/td>[\s\S]*?<\/tr>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const url = match[1]?.trim();
const title = match[2]?.trim();
const dateStr = match[3]?.trim();
const url = match[1]?.trim() ?? '';
const title = match[2]?.trim() ?? '';
const dateStr = match[3]?.trim() ?? '';
if (title && url) {
const fullUrl = url.startsWith('http') ? url : this.baseUrl + url;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
},
};

View File

@@ -29,7 +29,7 @@ describe('CeicCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Run in non-headless mode
args: [
@@ -40,14 +40,14 @@ describe('CeicCrawler Real Site Test', () => {
'--disable-infobars',
...proxyArgs,
],
defaultViewport: null
defaultViewport: null,
});
});
afterAll(async () => {
if (browser) {
// Keep open for a few seconds after test to see result
await new Promise(r => setTimeout(r, 50000));
await new Promise((r) => setTimeout(r, 50000));
await browser.close();
}
});
@@ -56,29 +56,33 @@ describe('CeicCrawler Real Site Test', () => {
console.log(`
Starting crawl for: ${CeicCrawler.name}`);
console.log(`Target URL: ${CeicCrawler.url}`);
const results = await CeicCrawler.crawl(browser);
console.log(`
Successfully found ${results.length} items:
`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
expect(results).toBeDefined();
expect(Array.isArray(results)).toBeTruthy();
if (results.length === 0) {
console.warn('Warning: No items found. Observe browser window to see if content is loading or if there is a verification challenge.');
console.warn(
'Warning: No items found. Observe browser window to see if content is loading or if there is a verification challenge.',
);
} else {
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -12,13 +12,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -28,23 +28,79 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
interface CeicCrawlerType {
name: string;
url: string;
baseUrl: string;
}
export const CeicCrawler = {
@@ -52,7 +108,10 @@ export const CeicCrawler = {
url: 'https://ceic.dlnyzb.com/3001',
baseUrl: 'https://ceic.dlnyzb.com/',
async crawl(browser: puppeteer.Browser): Promise<ChdtpResult[]> {
async crawl(
this: CeicCrawlerType,
browser: puppeteer.Browser,
): Promise<ChdtpResult[]> {
const logger = new Logger('CeicCrawler');
const page = await browser.newPage();
@@ -65,10 +124,14 @@ export const CeicCrawler = {
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', { get: () => [1, 2, 3, 4, 5] });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: ChdtpResult[] = [];
@@ -77,12 +140,22 @@ export const CeicCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
@@ -90,16 +163,25 @@ export const CeicCrawler = {
logger.log(`Processing page ${currentPage}...`);
// Wait for content to load - MUI list items
await page.waitForFunction(() => {
return document.querySelectorAll('li.MuiListItem-root').length > 0;
}, { timeout: 60000 }).catch(() => logger.warn('Content not found. Site might be slow.'));
await page
.waitForFunction(
() => {
return (
document.querySelectorAll('li.MuiListItem-root').length > 0
);
},
{ timeout: 60000 },
)
.catch(() => logger.warn('Content not found. Site might be slow.'));
const pageResults = await page.evaluate(() => {
const results: { title: string; dateStr: string; url: string }[] = [];
// Extract from MUI list items
const listItems = Array.from(document.querySelectorAll('li.MuiListItem-root'));
listItems.forEach(item => {
const listItems = Array.from(
document.querySelectorAll('li.MuiListItem-root'),
);
listItems.forEach((item) => {
// Find the title link
const titleLink = item.querySelector('a.css-1vdw90h');
const title = titleLink?.textContent?.trim() || '';
@@ -125,15 +207,19 @@ export const CeicCrawler = {
});
if (pageResults.length === 0) {
logger.warn(`No results found on page ${currentPage}. Extraction failed.`);
logger.warn(
`No results found on page ${currentPage}. Extraction failed.`,
);
break;
}
allResults.push(...pageResults.map(r => ({
title: r.title,
publishDate: r.dateStr ? new Date(r.dateStr) : new Date(),
url: r.url.replace(/\/\//g, '/')
})));
allResults.push(
...pageResults.map((r) => ({
title: r.title,
publishDate: r.dateStr ? new Date(r.dateStr) : new Date(),
url: r.url.replace(/\/\//g, '/'),
})),
);
logger.log(`Extracted ${pageResults.length} items.`);
@@ -142,27 +228,30 @@ export const CeicCrawler = {
if (!nextButton) break;
await nextButton.click();
await new Promise(r => setTimeout(r, 3000));
await new Promise((r) => setTimeout(r, 3000));
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
currentPage++;
}
return allResults;
} catch (error) {
logger.error(`Crawl failed: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Crawl failed: ${errorMessage}`);
throw error;
} finally {
if (page) await page.close();
}
},
extract() { return []; }
extract() {
return [];
},
};

View File

@@ -2,7 +2,7 @@ import { CgnpcCrawler } from './cgnpc_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('CgnpcCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('CgnpcCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${CgnpcCrawler.name}`);
console.log(`Target URL: ${CgnpcCrawler.url}`);
const results = await CgnpcCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('CgnpcCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -11,13 +11,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -27,23 +27,73 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export interface CgnpcResult {
@@ -52,12 +102,22 @@ export interface CgnpcResult {
url: string;
}
interface CgnpcCrawlerType {
name: string;
url: string;
baseUrl: string;
extract(html: string): CgnpcResult[];
}
export const CgnpcCrawler = {
name: '中广核电子商务平台',
url: 'https://ecp.cgnpc.com.cn/zbgg.html',
baseUrl: 'https://ecp.cgnpc.com.cn/',
async crawl(browser: puppeteer.Browser): Promise<CgnpcResult[]> {
async crawl(
this: CgnpcCrawlerType,
browser: puppeteer.Browser,
): Promise<CgnpcResult[]> {
const logger = new Logger('CgnpcCrawler');
const page = await browser.newPage();
@@ -69,11 +129,15 @@ export const CgnpcCrawler = {
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => "zh-CN"});
Object.defineProperty(navigator, 'plugins', { get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: CgnpcResult[] = [];
@@ -82,12 +146,22 @@ export const CgnpcCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
@@ -103,12 +177,14 @@ export const CgnpcCrawler = {
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
// 模拟人类行为 - 翻页前
logger.log('Simulating human mouse movements before pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling before pagination...');
await simulateHumanScrolling(page);
@@ -127,9 +203,13 @@ export const CgnpcCrawler = {
try {
// 点击下一页按钮
await nextButton.click();
await new Promise(r => setTimeout(r, 3000)); // 等待页面加载
await new Promise((r) => setTimeout(r, 3000)); // 等待页面加载
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
@@ -138,26 +218,27 @@ export const CgnpcCrawler = {
// 模拟人类行为 - 翻页后
logger.log('Simulating human mouse movements after pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling after pagination...');
await simulateHumanScrolling(page);
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): CgnpcResult[] {
extract(this: CgnpcCrawlerType, html: string): CgnpcResult[] {
const results: CgnpcResult[] = [];
/**
* Regex groups for ecp.cgnpc.com.cn:
@@ -181,24 +262,25 @@ export const CgnpcCrawler = {
* </div>
* </div>
*/
const regex = /<div class="zbnr">[\s\S]*?<a[^>]*title="([^"]*)"[^>]*href="([^"]*)"[^>]*>[\s\S]*?<dt>[\s\S]*?<p><\/p>[\s\S]*?<h2>\s*(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2})\s*<\/h2>[\s\S]*?<\/div>/gs;
const regex =
/<div class="zbnr">[\s\S]*?<a[^>]*title="([^"]*)"[^>]*href="([^"]*)"[^>]*>[\s\S]*?<dt>[\s\S]*?<p><\/p>[\s\S]*?<h2>\s*(\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2})\s*<\/h2>[\s\S]*?<\/div>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const title = match[1]?.trim();
const url = match[2]?.trim();
const dateStr = match[3]?.trim();
const title = match[1]?.trim() ?? '';
const url = match[2]?.trim() ?? '';
const dateStr = match[3]?.trim() ?? '';
if (title && url) {
const fullUrl = url.startsWith('http') ? url : this.baseUrl + url;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
},
};

View File

@@ -29,7 +29,7 @@ describe('ChdtpCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: true, // Change to false to see the browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('ChdtpCrawler Real Site Test', () => {
it('should visit the website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${ChdtpCrawler.name}`);
console.log(`Target URL: ${ChdtpCrawler.url}`);
const results = await ChdtpCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('ChdtpCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if the website structure has changed or if the list is currently empty.');
console.warn(
'Warning: No items found. Check if the website structure has changed or if the list is currently empty.',
);
} else {
// Check data integrity of the first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of the first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});
});

View File

@@ -1,10 +1,107 @@
import * as puppeteer from 'puppeteer';
import { Logger } from '@nestjs/common';
async function simulateHumanMouseMovement(page: puppeteer.Page) {
const viewport = page.viewport();
if (!viewport) return;
const movements = 5 + Math.floor(Math.random() * 5);
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20),
});
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
async function simulateHumanScrolling(page: puppeteer.Page) {
const scrollCount = 3 + Math.floor(Math.random() * 5);
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400);
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth',
});
}, scrollDistance);
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise((r) => setTimeout(r, 1000));
}
export interface ChdtpResult {
title: string;
publishDate: Date;
url: string; // Necessary for system uniqueness
url: string;
}
interface ChdtpCrawlerType {
name: string;
url: string;
baseUrl: string;
extract(html: string): ChdtpResult[];
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export const ChdtpCrawler = {
@@ -12,17 +109,22 @@ export const ChdtpCrawler = {
url: 'https://www.chdtp.com/webs/queryWebZbgg.action?zbggType=1',
baseUrl: 'https://www.chdtp.com/webs/',
async crawl(browser: puppeteer.Browser): Promise<ChdtpResult[]> {
async crawl(
this: ChdtpCrawlerType,
browser: puppeteer.Browser,
): Promise<ChdtpResult[]> {
const logger = new Logger('ChdtpCrawler');
const page = await browser.newPage();
const username = process.env.PROXY_USERNAME;
const password = process.env.PROXY_PASSWORD;
if (username && password) {
await page.authenticate({ username, password });
}
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36',
);
const allResults: ChdtpResult[] = [];
let currentPage = 1;
@@ -30,19 +132,43 @@ export const ChdtpCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
while (currentPage <= maxPages) {
const content = await page.content();
const pageResults = this.extract(content);
if (pageResults.length === 0) {
logger.warn(`No results found on page ${currentPage}, stopping.`);
break;
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
logger.log('Simulating human mouse movements before pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling before pagination...');
await simulateHumanScrolling(page);
// Find the "Next Page" button
// Using partial match for src to be robust against path variations
@@ -54,39 +180,50 @@ export const ChdtpCrawler = {
break;
}
// Optional: Check if the button is disabled (though image inputs usually aren't "disabled" in the same way)
// For this specific site, we'll try to click.
logger.log(`Navigating to page ${currentPage + 1}...`);
try {
await Promise.all([
page.waitForNavigation({ waitUntil: 'networkidle2', timeout: 60000 }),
page.waitForNavigation({
waitUntil: 'networkidle2',
timeout: 60000,
}),
nextButton.click(),
]);
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
currentPage++;
logger.log('Simulating human mouse movements after pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling after pagination...');
await simulateHumanScrolling(page);
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults; // Return what we have so far
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): ChdtpResult[] {
extract(this: ChdtpCrawlerType, html: string): ChdtpResult[] {
const results: ChdtpResult[] = [];
/**
* Regex groups for chdtp.com:
@@ -96,23 +233,24 @@ export const ChdtpCrawler = {
* 4: Business Type
* 5: Date
*/
const regex = /<tr[^>]*>\s*<td class="td_1">.*?<span[^>]*>\s*(.*?)\s*<\/span>.*?<\/td>\s*<td class="td_2">\s*<a[^>]*href="javascript:toGetContent\('(.*?)'\)" title="(.*?)">.*?<\/a><\/td>\s*<td class="td_3">\s*<a[^>]*>\s*(.*?)\s*<\/a>\s*<\/td>\s*<td class="td_4"><span>\[(.*?)\]<\/span><\/td>/gs;
const regex =
/<tr[^>]*>\s*<td class="td_1">.*?<span[^>]*>\s*(.*?)\s*<\/span>.*?<\/td>\s*<td class="td_2">\s*<a[^>]*href="javascript:toGetContent\('(.*?)'\)" title="(.*?)">.*?<\/a><\/td>\s*<td class="td_3">\s*<a[^>]*>\s*(.*?)\s*<\/a>\s*<\/td>\s*<td class="td_4"><span>\[(.*?)\]<\/span><\/td>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const urlSuffix = match[2]?.trim();
const title = match[3]?.trim();
const dateStr = match[5]?.trim();
const urlSuffix = match[2]?.trim() ?? '';
const title = match[3]?.trim() ?? '';
const dateStr = match[5]?.trim() ?? '';
if (title && urlSuffix) {
const fullUrl = this.baseUrl + urlSuffix;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
};
},
};

View File

@@ -31,13 +31,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -47,23 +47,23 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
describe('ChngCrawler Real Site Test', () => {
@@ -74,7 +74,7 @@ describe('ChngCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Run in non-headless mode
args: [
@@ -82,7 +82,7 @@ describe('ChngCrawler Real Site Test', () => {
'--disable-setuid-sandbox',
'--disable-blink-features=AutomationControlled',
'--window-size=1920,1080',
"--disable-infobars",
'--disable-infobars',
...proxyArgs,
// "--headless=new",
// '--disable-dev-shm-usage',
@@ -94,15 +94,14 @@ describe('ChngCrawler Real Site Test', () => {
// '--disable-webgl',
// '--disable-javascript',
],
defaultViewport: null
defaultViewport: null,
});
});
afterAll(async () => {
if (browser) {
// Keep open for a few seconds after test to see result
await new Promise(r => setTimeout(r, 50000));
await new Promise((r) => setTimeout(r, 50000));
await browser.close();
}
});
@@ -111,43 +110,51 @@ describe('ChngCrawler Real Site Test', () => {
console.log(`
Starting crawl for: ${ChngCrawler.name}`);
console.log(`Target URL: ${ChngCrawler.url}`);
// 创建一个临时页面用于模拟人类行为
const tempPage = await browser.newPage();
await tempPage.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1 });
await tempPage.setViewport({
width: 1920,
height: 1080,
deviceScaleFactor: 1,
});
// 模拟人类鼠标移动
console.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(tempPage);
// 模拟人类滚动
console.log('Simulating human scrolling...');
await simulateHumanScrolling(tempPage);
await tempPage.close();
const results = await ChngCrawler.crawl(browser);
console.log(`
Successfully found ${results.length} items:
`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
expect(results).toBeDefined();
expect(Array.isArray(results)).toBeTruthy();
if (results.length === 0) {
console.warn('Warning: No items found. Observe the browser window to see if content is loading or if there is a verification challenge.');
console.warn(
'Warning: No items found. Observe the browser window to see if content is loading or if there is a verification challenge.',
);
} else {
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});
});

View File

@@ -4,55 +4,138 @@ import { ChdtpResult } from './chdtp_target';
// 模拟人类鼠标移动
async function simulateHumanMouseMovement(page: puppeteer.Page) {
const viewport = page.viewport();
if (!viewport) return;
try {
const viewport = page.viewport();
if (!viewport) return;
const movements = 5 + Math.floor(Math.random() * 5); // 5-10次随机移动
const movements = 5 + Math.floor(Math.random() * 5); // 5-10次随机移动
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
for (let i = 0; i < movements; i++) {
// 检查页面是否仍然有效
if (page.isClosed()) {
console.log('Page was closed during mouse movement simulation');
return;
}
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
console.log('Mouse movement simulation interrupted:', errorMessage);
}
}
// 模拟人类滚动
async function simulateHumanScrolling(page: puppeteer.Page) {
const scrollCount = 3 + Math.floor(Math.random() * 5); // 3-7次滚动
try {
const scrollCount = 3 + Math.floor(Math.random() * 5); // 3-7次滚动
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
for (let i = 0; i < scrollCount; i++) {
// 检查页面是否仍然有效
if (page.isClosed()) {
console.log('Page was closed during scrolling simulation');
return;
}
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
if (!page.isClosed()) {
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
}, scrollDistance);
await new Promise((r) => setTimeout(r, 1000));
}
} catch (error) {
const errorMessage = error instanceof Error ? error.message : String(error);
console.log('Scrolling simulation interrupted:', errorMessage);
}
}
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
throw lastError;
}
interface ChngCrawlerType {
name: string;
url: string;
baseUrl: string;
}
export const ChngCrawler = {
name: '华能集团电子商务平台',
url: 'https://ec.chng.com.cn/ecmall/index.html#/purchase/home?top=0',
baseUrl: 'https://ec.chng.com.cn/ecmall/index.html',
url: 'https://ec.chng.com.cn/channel/home/#/purchase?top=0',
baseUrl: 'https://ec.chng.com.cn/channel/home/#',
async crawl(browser: puppeteer.Browser): Promise<ChdtpResult[]> {
async crawl(
this: ChngCrawlerType,
browser: puppeteer.Browser,
): Promise<ChdtpResult[]> {
const logger = new Logger('ChngCrawler');
let page = await browser.newPage();
// await page.setViewport({ width: 1920, height: 1080, deviceScaleFactor: 1 });
@@ -62,42 +145,55 @@ export const ChngCrawler = {
if (username && password) {
await page.authenticate({ username, password });
}
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => "zh-CN"});
Object.defineProperty(navigator, 'plugins', { get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: ChdtpResult[] = [];
let currentPage = 1;
const maxPages = 5;
const maxPages = 5;
try {
logger.log('Navigating to Bing...');
await page.goto('https://cn.bing.com', { waitUntil: 'networkidle2' });
await delayRetry(
async () => {
await page.goto('https://cn.bing.com', { waitUntil: 'networkidle2' });
},
3,
5000,
logger,
);
logger.log('Searching for target site...');
const searchBoxSelector = 'input[name="q"]';
const searchBoxSelector = 'input[name="q"]';
await page.waitForSelector(searchBoxSelector);
await page.type(searchBoxSelector, 'https://ec.chng.com.cn/');
await page.keyboard.press('Enter');
await page.waitForNavigation({ waitUntil: 'networkidle2' });
logger.log('Clicking search result...');
// await page.screenshot({ path: 'bing.png' });
const firstResultSelector = '#b_results .b_algo h2 a';
await page.waitForSelector(firstResultSelector);
const newTargetPromise = browser.waitForTarget(target => target.opener() === page.target());
const newTargetPromise = browser.waitForTarget(
(target) => target.opener() === page.target(),
);
await page.click(firstResultSelector);
const newTarget = await newTargetPromise;
const newPage = await newTarget.page();
if (newPage) {
// await newPage.screenshot({ path: 'newPage.png' });
await page.close();
@@ -106,129 +202,177 @@ export const ChngCrawler = {
await page.authenticate({ username, password });
}
}
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
await page.waitForNavigation({ waitUntil: 'domcontentloaded' }).catch(() => {});
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
// PAUSE 15 SECONDS as requested
logger.log('Pausing 15 seconds before looking for "采购专栏"...');
await new Promise(r => setTimeout(r, 15000));
// await page.screenshot({ path: 'huaneng.png' });
logger.log('Looking for "采购专栏" link...');
await page.waitForFunction(() => {
const divs = Array.from(document.querySelectorAll('div.text'));
return divs.some(div => div.textContent && div.textContent.includes('采购专栏'));
}, { timeout: 60000 });
const purchaseTargetPromise = browser.waitForTarget(target => target.opener() === page.target(), { timeout: 15000 }).catch(() => null);
await page.evaluate(() => {
const divs = Array.from(document.querySelectorAll('div.text'));
const target = divs.find(div => div.textContent && div.textContent.includes('采购专栏')) as HTMLElement;
if (target) target.click();
});
const purchaseTarget = await purchaseTargetPromise;
if (purchaseTarget) {
const pPage = await purchaseTarget.page();
if (pPage) {
logger.log('Switched to Purchase Page tab.');
page = pPage;
if (username && password) {
await page.authenticate({ username, password });
}
await new Promise(r => setTimeout(r, 5000));
}
}
logger.log(`Active URL: ${page.url()}`);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
// 等待页面稳定,不强制等待导航
await new Promise((r) => setTimeout(r, 3000));
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
// PAUSE 15 SECONDS as requested
logger.log('Pausing 15 seconds before looking for "采购专栏"...');
await new Promise((r) => setTimeout(r, 15000));
// await page.screenshot({ path: 'huaneng.png' });
logger.log('Looking for "采购专栏" link...');
await page.waitForFunction(
() => {
const divs = Array.from(document.querySelectorAll('div.text'));
return divs.some(
(div) => div.textContent && div.textContent.includes('采购专栏'),
);
},
{ timeout: 60000 },
);
const purchaseTargetPromise = browser
.waitForTarget((target) => target.opener() === page.target(), {
timeout: 15000,
})
.catch(() => null);
await page.evaluate(() => {
const divs = Array.from(document.querySelectorAll('div.text'));
const target = divs.find(
(div) => div.textContent && div.textContent.includes('采购专栏'),
) as HTMLElement;
if (target) target.click();
});
const purchaseTarget = await purchaseTargetPromise;
if (purchaseTarget) {
const pPage = await purchaseTarget.page();
if (pPage) {
logger.log('Switched to Purchase Page tab.');
page = pPage;
if (username && password) {
await page.authenticate({ username, password });
}
await new Promise((r) => setTimeout(r, 5000));
}
}
logger.log(`Active URL: ${page.url()}`);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
while (currentPage <= maxPages) {
logger.log(`Processing page ${currentPage}...`);
// Wait for table rows to load
await page.waitForFunction(() => {
return document.querySelectorAll('tr.ant-table-row').length > 0;
}, { timeout: 60000 }).catch(() => logger.warn('Content not found. Site might be slow.'));
await page
.waitForFunction(
() => {
return document.querySelectorAll('tr.ant-table-row').length > 0;
},
{ timeout: 60000 },
)
.catch(() => logger.warn('Content not found. Site might be slow.'));
const pageResults = await page.evaluate((baseUrl) => {
// Extract from table rows
const items = Array.from(document.querySelectorAll('tr.ant-table-row'));
return items.map(item => {
const titleSpan = item.querySelector('span.list-text');
const dateCell = item.querySelector('td.ant-table-row-cell-break-word p');
if (titleSpan && dateCell) {
const title = titleSpan.textContent?.trim() || '';
const dateStr = dateCell.textContent?.trim() || '';
if (title.length < 5) return null; // Filter noise
// URL is not directly available in the table, need to construct from data-row-key
const rowKey = item.getAttribute('data-row-key');
const url = rowKey ? `${baseUrl}#/purchase/detail?id=${rowKey}` : '';
return {
title,
dateStr,
url
};
}
return null;
}).filter(i => i !== null);
const items = Array.from(
document.querySelectorAll('tr.ant-table-row'),
);
return items
.map((item) => {
const titleSpan = item.querySelector('span.list-text');
const dateCell = item.querySelector(
'td.ant-table-row-cell-break-word p',
);
if (titleSpan && dateCell) {
const title = titleSpan.textContent?.trim() || '';
const dateStr = dateCell.textContent?.trim() || '';
if (title.length < 5) return null; // Filter noise
// URL is not directly available in the table, need to construct from data-row-key
const rowKey = item.getAttribute('data-row-key');
const url = rowKey
? `${baseUrl}#/purchase/detail?id=${rowKey}`
: '';
return {
title,
dateStr,
url,
};
}
return null;
})
.filter((i) => i !== null);
}, this.baseUrl);
if (pageResults.length === 0) {
logger.warn(`No results found on page ${currentPage}. Extraction failed.`);
break;
logger.warn(
`No results found on page ${currentPage}. Extraction failed.`,
);
break;
}
allResults.push(...pageResults.map(r => ({
title: r!.title,
publishDate: new Date(r!.dateStr),
url: r!.url.replace(/\/\//g, '/')
})));
allResults.push(
...pageResults.map((r) => ({
title: r.title,
publishDate: new Date(r.dateStr),
url: r.url.replace(/\/\//g, '/'),
})),
);
logger.log(`Extracted ${pageResults.length} items.`);
// Pagination: look for the "right" icon SVG
const nextButton = await page.$('svg[data-icon="right"]');
if (!nextButton) break;
// 点击下一页前保存当前页面状态
const currentUrl = page.url();
await nextButton.click();
await new Promise(r => setTimeout(r, 5000));
// 等待页面导航完成
try {
await page.waitForFunction(
(oldUrl) => window.location.href !== oldUrl,
{ timeout: 10000 },
currentUrl,
);
} catch {
logger.warn('Navigation timeout, continuing anyway');
}
// 等待页面内容加载
await new Promise((r) => setTimeout(r, 15000));
currentPage++;
}
return allResults;
} catch (error) {
logger.error(`Crawl failed: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Crawl failed: ${errorMessage}`);
throw error;
} finally {
if (page) await page.close();
}
},
extract() { return []; }
};
extract() {
return [];
},
};

View File

@@ -2,7 +2,7 @@ import { CnncecpCrawler } from './cnncecp_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('CnncecpCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('CnncecpCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${CnncecpCrawler.name}`);
console.log(`Target URL: ${CnncecpCrawler.url}`);
const results = await CnncecpCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('CnncecpCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -11,13 +11,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -27,23 +27,73 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export interface CnncecpResult {
@@ -52,12 +102,22 @@ export interface CnncecpResult {
url: string;
}
interface CnncecpCrawlerType {
name: string;
url: string;
baseUrl: string;
extract(html: string): CnncecpResult[];
}
export const CnncecpCrawler = {
name: '中核集团电子采购平台',
url: 'https://www.cnncecp.com/xzbgg/index.jhtml',
baseUrl: 'https://www.cnncecp.com/',
async crawl(browser: puppeteer.Browser): Promise<CnncecpResult[]> {
async crawl(
this: CnncecpCrawlerType,
browser: puppeteer.Browser,
): Promise<CnncecpResult[]> {
const logger = new Logger('CnncecpCrawler');
const page = await browser.newPage();
@@ -69,11 +129,15 @@ export const CnncecpCrawler = {
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => "zh-CN"});
Object.defineProperty(navigator, 'plugins', { get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: CnncecpResult[] = [];
@@ -82,12 +146,22 @@ export const CnncecpCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
@@ -103,12 +177,14 @@ export const CnncecpCrawler = {
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
// 模拟人类行为 - 翻页前
logger.log('Simulating human mouse movements before pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling before pagination...');
await simulateHumanScrolling(page);
@@ -126,9 +202,13 @@ export const CnncecpCrawler = {
try {
// 点击下一页按钮
await nextButton.click();
await new Promise(r => setTimeout(r, 3000)); // 等待页面加载
await new Promise((r) => setTimeout(r, 3000)); // 等待页面加载
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
@@ -137,26 +217,27 @@ export const CnncecpCrawler = {
// 模拟人类行为 - 翻页后
logger.log('Simulating human mouse movements after pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling after pagination...');
await simulateHumanScrolling(page);
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): CnncecpResult[] {
extract(this: CnncecpCrawlerType, html: string): CnncecpResult[] {
const results: CnncecpResult[] = [];
/**
* Regex groups for cnncecp.com:
@@ -172,24 +253,25 @@ export const CnncecpCrawler = {
* <a href="https://www.cnncecp.com/xzbgg/1862778.jhtml">中核四0四有限公司2026-2028年度质量流量控制器等采购项目二次变更公告</a>
* </li>
*/
const regex = /<li>[\s\S]*?<span class="Right Gray">\s*(\d{4}-\d{2}-\d{2})\s*<\/span>[\s\S]*?<a[^>]*href="([^"]*)"[^>]*>([^<]*)<\/a>[\s\S]*?<\/li>/gs;
const regex =
/<li>[\s\S]*?<span class="Right Gray">\s*(\d{4}-\d{2}-\d{2})\s*<\/span>[\s\S]*?<a[^>]*href="([^"]*)"[^>]*>([^<]*)<\/a>[\s\S]*?<\/li>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const dateStr = match[1]?.trim();
const url = match[2]?.trim();
const title = match[3]?.trim();
const dateStr = match[1]?.trim() ?? '';
const url = match[2]?.trim() ?? '';
const title = match[3]?.trim() ?? '';
if (title && url) {
const fullUrl = url.startsWith('http') ? url : this.baseUrl + url;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
},
};

View File

@@ -2,7 +2,7 @@ import { CnoocCrawler } from './cnooc_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('CnoocCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('CnoocCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${CnoocCrawler.name}`);
console.log(`Target URL: ${CnoocCrawler.url}`);
const results = await CnoocCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('CnoocCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -11,13 +11,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -27,23 +27,73 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export interface CnoocResult {
@@ -52,12 +102,22 @@ export interface CnoocResult {
url: string;
}
interface CnoocCrawlerType {
name: string;
url: string;
baseUrl: string;
extract(html: string): CnoocResult[];
}
export const CnoocCrawler = {
name: '中海油招标平台',
url: 'https://buy.cnooc.com.cn/cbjyweb/001/001001/moreinfo.html',
baseUrl: 'https://buy.cnooc.com.cn/',
async crawl(browser: puppeteer.Browser): Promise<CnoocResult[]> {
async crawl(
this: CnoocCrawlerType,
browser: puppeteer.Browser,
): Promise<CnoocResult[]> {
const logger = new Logger('CnoocCrawler');
const page = await browser.newPage();
@@ -69,11 +129,15 @@ export const CnoocCrawler = {
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => "zh-CN"});
Object.defineProperty(navigator, 'plugins', { get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: CnoocResult[] = [];
@@ -82,12 +146,22 @@ export const CnoocCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
@@ -103,12 +177,14 @@ export const CnoocCrawler = {
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
// 模拟人类行为 - 翻页前
logger.log('Simulating human mouse movements before pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling before pagination...');
await simulateHumanScrolling(page);
@@ -127,9 +203,13 @@ export const CnoocCrawler = {
try {
// 点击下一页按钮
await nextButton.click();
await new Promise(r => setTimeout(r, 3000)); // 等待页面加载
await new Promise((r) => setTimeout(r, 3000)); // 等待页面加载
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
@@ -138,26 +218,27 @@ export const CnoocCrawler = {
// 模拟人类行为 - 翻页后
logger.log('Simulating human mouse movements after pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling after pagination...');
await simulateHumanScrolling(page);
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): CnoocResult[] {
extract(this: CnoocCrawlerType, html: string): CnoocResult[] {
const results: CnoocResult[] = [];
/**
* Regex groups for buy.cnooc.com.cn:
@@ -173,24 +254,25 @@ export const CnoocCrawler = {
* <span class="now-span" style="width:100px">2026-01-12</span>
* </li>
*/
const regex = /<li class="now-hd-items clearfix">[\s\S]*?<a[^>]*href="([^"]*)"[^>]*>[\s\S]*?<font[^>]*>([^<]*)<\/font>[\s\S]*?<span class="now-span"[^>]*>\s*(\d{4}-\d{2}-\d{2})\s*<\/span>[\s\S]*?<\/li>/gs;
const regex =
/<li class="now-hd-items clearfix">[\s\S]*?<a[^>]*href="([^"]*)"[^>]*>[\s\S]*?<font[^>]*>([^<]*)<\/font>[\s\S]*?<span class="now-span"[^>]*>\s*(\d{4}-\d{2}-\d{2})\s*<\/span>[\s\S]*?<\/li>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const url = match[1]?.trim();
const title = match[2]?.trim();
const dateStr = match[3]?.trim();
const url = match[1]?.trim() ?? '';
const title = match[2]?.trim() ?? '';
const dateStr = match[3]?.trim() ?? '';
if (title && url) {
const fullUrl = url.startsWith('http') ? url : this.baseUrl + url;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
},
};

View File

@@ -2,7 +2,7 @@ import { EpsCrawler } from './eps_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('EpsCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('EpsCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${EpsCrawler.name}`);
console.log(`Target URL: ${EpsCrawler.url}`);
const results = await EpsCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('EpsCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -11,13 +11,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -27,23 +27,73 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export interface EpsResult {
@@ -52,12 +102,22 @@ export interface EpsResult {
url: string;
}
interface EpsCrawlerType {
name: string;
url: string;
baseUrl: string;
extract(html: string): EpsResult[];
}
export const EpsCrawler = {
name: '中国三峡集团电子商务平台',
url: 'https://eps.ctg.com.cn/cms/channel/1ywgg1/index.htm',
baseUrl: 'https://eps.ctg.com.cn/',
async crawl(browser: puppeteer.Browser): Promise<EpsResult[]> {
async crawl(
this: EpsCrawlerType,
browser: puppeteer.Browser,
): Promise<EpsResult[]> {
const logger = new Logger('EpsCrawler');
const page = await browser.newPage();
@@ -69,11 +129,15 @@ export const EpsCrawler = {
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => "zh-CN"});
Object.defineProperty(navigator, 'plugins', { get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: EpsResult[] = [];
@@ -82,12 +146,22 @@ export const EpsCrawler = {
try {
logger.log(`Navigating to ${this.url}...`);
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(this.url, {
waitUntil: 'networkidle2',
timeout: 60000,
});
},
3,
5000,
logger,
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
@@ -103,12 +177,14 @@ export const EpsCrawler = {
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
// 模拟人类行为 - 翻页前
logger.log('Simulating human mouse movements before pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling before pagination...');
await simulateHumanScrolling(page);
@@ -127,9 +203,13 @@ export const EpsCrawler = {
try {
// 点击下一页按钮,等待页面更新
await nextButton.click();
await new Promise(r => setTimeout(r, 3000)); // 等待页面加载
await new Promise((r) => setTimeout(r, 3000)); // 等待页面加载
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
@@ -138,26 +218,27 @@ export const EpsCrawler = {
// 模拟人类行为 - 翻页后
logger.log('Simulating human mouse movements after pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling after pagination...');
await simulateHumanScrolling(page);
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): EpsResult[] {
extract(this: EpsCrawlerType, html: string): EpsResult[] {
const results: EpsResult[] = [];
/**
* Regex groups for eps.ctg.com.cn:
@@ -179,24 +260,25 @@ export const EpsCrawler = {
* </a>
* </li>
*/
const regex = /<li[^>]*name="li_name"[^>]*>[\s\S]*?<a[^>]*href="([^"]*)"[^>]*title="([^"]*)"[^>]*>[\s\S]*?<em>\s*(\d{4}-\d{2}-\d{2})\s*<\/em>[\s\S]*?<\/a>[\s\S]*?<\/li>/gs;
const regex =
/<li[^>]*name="li_name"[^>]*>[\s\S]*?<a[^>]*href="([^"]*)"[^>]*title="([^"]*)"[^>]*>[\s\S]*?<em>\s*(\d{4}-\d{2}-\d{2})\s*<\/em>[\s\S]*?<\/a>[\s\S]*?<\/li>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const url = match[1]?.trim();
const title = match[2]?.trim();
const dateStr = match[3]?.trim();
const url = match[1]?.trim() ?? '';
const title = match[2]?.trim() ?? '';
const dateStr = match[3]?.trim() ?? '';
if (title && url) {
const fullUrl = url.startsWith('http') ? url : this.baseUrl + url;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
},
};

View File

@@ -2,7 +2,7 @@ import { EspicCrawler } from './espic_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('EspicCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('EspicCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${EspicCrawler.name}`);
console.log(`Target URL: ${EspicCrawler.getUrl()}`);
const results = await EspicCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('EspicCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

View File

@@ -11,13 +11,13 @@ async function simulateHumanMouseMovement(page: puppeteer.Page) {
for (let i = 0; i < movements; i++) {
const x = Math.floor(Math.random() * viewport.width);
const y = Math.floor(Math.random() * viewport.height);
await page.mouse.move(x, y, {
steps: 10 + Math.floor(Math.random() * 20) // 10-30步使移动更平滑
steps: 10 + Math.floor(Math.random() * 20), // 10-30步使移动更平滑
});
// 随机停顿 100-500ms
await new Promise(r => setTimeout(r, 100 + Math.random() * 400));
await new Promise((r) => setTimeout(r, 100 + Math.random() * 400));
}
}
@@ -27,23 +27,73 @@ async function simulateHumanScrolling(page: puppeteer.Page) {
for (let i = 0; i < scrollCount; i++) {
const scrollDistance = 100 + Math.floor(Math.random() * 400); // 100-500px
await page.evaluate((distance) => {
window.scrollBy({
top: distance,
behavior: 'smooth'
behavior: 'smooth',
});
}, scrollDistance);
// 随机停顿 500-1500ms
await new Promise(r => setTimeout(r, 500 + Math.random() * 1000));
await new Promise((r) => setTimeout(r, 500 + Math.random() * 1000));
}
// 滚动回顶部
await page.evaluate(() => {
window.scrollTo({ top: 0, behavior: 'smooth' });
});
await new Promise(r => setTimeout(r, 1000));
await new Promise((r) => setTimeout(r, 1000));
}
// 检查错误是否为代理隧道连接失败
function isTunnelConnectionFailedError(error: unknown): boolean {
if (error instanceof Error) {
return (
error.message.includes('net::ERR_TUNNEL_CONNECTION_FAILED') ||
error.message.includes('ERR_TUNNEL_CONNECTION_FAILED')
);
}
return false;
}
// 延迟重试函数
async function delayRetry(
operation: () => Promise<void>,
maxRetries: number = 3,
delayMs: number = 5000,
logger?: Logger,
): Promise<void> {
let lastError: Error | unknown;
for (let attempt = 1; attempt <= maxRetries; attempt++) {
try {
await operation();
return;
} catch (error) {
lastError = error;
if (isTunnelConnectionFailedError(error)) {
if (attempt < maxRetries) {
const delay = delayMs * attempt; // 递增延迟
logger?.warn(
`代理隧道连接失败,第 ${attempt} 次尝试失败,${delay / 1000} 秒后重试...`,
);
await new Promise((resolve) => setTimeout(resolve, delay));
} else {
logger?.error(
`代理隧道连接失败,已达到最大重试次数 ${maxRetries}`,
);
throw error;
}
} else {
// 非代理错误,直接抛出
throw error;
}
}
}
throw lastError;
}
export interface EspicResult {
@@ -52,12 +102,19 @@ export interface EspicResult {
url: string;
}
interface EspicCrawlerType {
name: string;
baseUrl: string;
getUrl(page?: number): string;
extract(html: string): EspicResult[];
}
export const EspicCrawler = {
name: '电能e招采平台国电投',
baseUrl: 'https://ebid.espic.com.cn/',
// 生成动态 URL使用当前日期
getUrl(page: number = 1): string {
getUrl(this: EspicCrawlerType, page: number = 1): string {
const now = new Date();
const year = now.getFullYear();
const month = now.getMonth() + 1; // 月份从0开始
@@ -66,7 +123,10 @@ export const EspicCrawler = {
return `https://ebid.espic.com.cn/newgdtcms//category/iframe.html?dates=300&categoryId=2&tenderMethod=01&tabName=&page=${page}&time=${timeStr}`;
},
async crawl(browser: puppeteer.Browser): Promise<EspicResult[]> {
async crawl(
this: EspicCrawlerType,
browser: puppeteer.Browser,
): Promise<EspicResult[]> {
const logger = new Logger('EspicCrawler');
const page = await browser.newPage();
@@ -78,11 +138,15 @@ export const EspicCrawler = {
await page.evaluateOnNewDocument(() => {
Object.defineProperty(navigator, 'webdriver', { get: () => false });
Object.defineProperty(navigator, 'language', { get: () => "zh-CN"});
Object.defineProperty(navigator, 'plugins', { get: () => [1,2,3,4,5]});
Object.defineProperty(navigator, 'language', { get: () => 'zh-CN' });
Object.defineProperty(navigator, 'plugins', {
get: () => [1, 2, 3, 4, 5],
});
});
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36');
await page.setUserAgent(
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36',
);
await page.setViewport({ width: 1920, height: 1080 });
const allResults: EspicResult[] = [];
@@ -92,7 +156,14 @@ export const EspicCrawler = {
try {
const url = this.getUrl(currentPage);
logger.log(`Navigating to ${url}...`);
await page.goto(url, { waitUntil: 'networkidle2', timeout: 60000 });
await delayRetry(
async () => {
await page.goto(url, { waitUntil: 'networkidle2', timeout: 60000 });
},
3,
5000,
logger,
);
// 等待 WAF 验证通过
logger.log('Waiting for WAF verification...');
@@ -100,15 +171,18 @@ export const EspicCrawler = {
() => {
// 检查是否已经通过验证(页面不再是 WAF 页面)
const bodyText = document.body?.textContent || '';
return !bodyText.includes('人机识别检测') && !bodyText.includes('WEB 应用防火墙');
return (
!bodyText.includes('人机识别检测') &&
!bodyText.includes('WEB 应用防火墙')
);
},
{ timeout: 30000 }
{ timeout: 30000 },
);
// 模拟人类行为
logger.log('Simulating human mouse movements...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling...');
await simulateHumanScrolling(page);
@@ -124,12 +198,14 @@ export const EspicCrawler = {
}
allResults.push(...pageResults);
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
logger.log(
`Extracted ${pageResults.length} items from page ${currentPage}`,
);
// 模拟人类行为 - 翻页前
logger.log('Simulating human mouse movements before pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling before pagination...');
await simulateHumanScrolling(page);
@@ -141,7 +217,7 @@ export const EspicCrawler = {
'a[aria-label="Next"]',
'a.next',
'li.next a',
'a.layui-laypage-next:not(.layui-disabled)'
'a.layui-laypage-next:not(.layui-disabled)',
];
let nextButton: puppeteer.ElementHandle<Element> | null = null;
@@ -149,7 +225,7 @@ export const EspicCrawler = {
try {
nextButton = await page.$(selector);
if (nextButton) break;
} catch (e) {
} catch {
// 继续尝试下一个选择器
}
}
@@ -164,9 +240,13 @@ export const EspicCrawler = {
try {
// 点击下一页按钮
await nextButton.click();
await new Promise(r => setTimeout(r, 3000)); // 等待页面加载
await new Promise((r) => setTimeout(r, 3000)); // 等待页面加载
} catch (navError) {
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
const navErrorMessage =
navError instanceof Error ? navError.message : String(navError);
logger.error(
`Navigation to page ${currentPage + 1} failed: ${navErrorMessage}`,
);
break;
}
@@ -175,26 +255,27 @@ export const EspicCrawler = {
// 模拟人类行为 - 翻页后
logger.log('Simulating human mouse movements after pagination...');
await simulateHumanMouseMovement(page);
logger.log('Simulating human scrolling after pagination...');
await simulateHumanScrolling(page);
// Random delay between pages
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
await new Promise(resolve => setTimeout(resolve, delay));
await new Promise((resolve) => setTimeout(resolve, delay));
}
return allResults;
} catch (error) {
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
return allResults;
const errorMessage =
error instanceof Error ? error.message : String(error);
logger.error(`Failed to crawl ${this.name}: ${errorMessage}`);
throw error;
} finally {
await page.close();
}
},
extract(html: string): EspicResult[] {
extract(this: EspicCrawlerType, html: string): EspicResult[] {
const results: EspicResult[] = [];
/**
* Regex groups for ebid.espic.com.cn:
@@ -225,24 +306,25 @@ export const EspicCrawler = {
* </a>
* </li>
*/
const regex = /<li>[\s\S]*?<a[^>]*href="([^"]*)"[^>]*title="([^"]*)"[^>]*>[\s\S]*?<div class="newsDate">[\s\S]*?<div>\s*(\d{4}-\d{2}-\d{2})\s*<\/div>[\s\S]*?<\/div>[\s\S]*?<\/a>[\s\S]*?<\/li>/gs;
const regex =
/<li>[\s\S]*?<a[^>]*href="([^"]*)"[^>]*title="([^"]*)"[^>]*>[\s\S]*?<div class="newsDate">[\s\S]*?<div>\s*(\d{4}-\d{2}-\d{2})\s*<\/div>[\s\S]*?<\/div>[\s\S]*?<\/a>[\s\S]*?<\/li>/gs;
let match;
let match: RegExpExecArray | null;
while ((match = regex.exec(html)) !== null) {
const url = match[1]?.trim();
const title = match[2]?.trim();
const dateStr = match[3]?.trim();
const url = match[1]?.trim() ?? '';
const title = match[2]?.trim() ?? '';
const dateStr = match[3]?.trim() ?? '';
if (title && url) {
const fullUrl = url.startsWith('http') ? url : this.baseUrl + url;
results.push({
title,
publishDate: dateStr ? new Date(dateStr) : new Date(),
url: fullUrl.replace(/\/\//g, '/')
url: fullUrl.replace(/\/\//g, '/'),
});
}
}
return results;
}
},
};

View File

@@ -2,7 +2,7 @@ import { PowerbeijingCrawler } from './powerbeijing_target';
import * as puppeteer from 'puppeteer';
// Increase timeout to 60 seconds for network operations
jest.setTimeout(60000*5);
jest.setTimeout(60000 * 5);
// 获取代理配置
const getProxyArgs = (): string[] => {
@@ -29,7 +29,7 @@ describe('PowerbeijingCrawler Real Site Test', () => {
if (proxyArgs.length > 0) {
console.log('Using proxy:', proxyArgs.join(' '));
}
browser = await puppeteer.launch({
headless: false, // Change to false to see browser UI
args: ['--no-sandbox', '--disable-setuid-sandbox', ...proxyArgs],
@@ -45,13 +45,15 @@ describe('PowerbeijingCrawler Real Site Test', () => {
it('should visit website and list all found bid information', async () => {
console.log(`\nStarting crawl for: ${PowerbeijingCrawler.name}`);
console.log(`Target URL: ${PowerbeijingCrawler.url}`);
const results = await PowerbeijingCrawler.crawl(browser);
console.log(`\nSuccessfully found ${results.length} items:\n`);
console.log('----------------------------------------');
results.forEach((item, index) => {
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
console.log(
`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`,
);
console.log(` Link: ${item.url}`);
console.log('----------------------------------------');
});
@@ -61,13 +63,15 @@ describe('PowerbeijingCrawler Real Site Test', () => {
expect(Array.isArray(results)).toBeTruthy();
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
if (results.length === 0) {
console.warn('Warning: No items found. Check if website structure has changed or if list is currently empty.');
console.warn(
'Warning: No items found. Check if website structure has changed or if list is currently empty.',
);
} else {
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
// Check data integrity of first item
const firstItem = results[0];
expect(firstItem.title).toBeTruthy();
expect(firstItem.url).toMatch(/^https?:\/\//);
expect(firstItem.publishDate).toBeInstanceOf(Date);
}
});
});

Some files were not shown because too many files have changed in this diff Show More