From 2fcfb452ec610ac8183178c3da1acc55ca2a34ad Mon Sep 17 00:00:00 2001 From: dmy Date: Tue, 13 Jan 2026 21:32:59 +0800 Subject: [PATCH] =?UTF-8?q?docs:=20=E6=9B=B4=E6=96=B0README=E4=BB=A5?= =?UTF-8?q?=E5=8F=8D=E6=98=A0=E6=8A=95=E6=A0=87=E7=9B=91=E6=8E=A7=E7=B3=BB?= =?UTF-8?q?=E7=BB=9F=E9=A1=B9=E7=9B=AE=E8=AF=A6=E6=83=85?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- README.md | 402 +++++++++++++++++++++++++++++++++--------------------- 1 file changed, 249 insertions(+), 153 deletions(-) diff --git a/README.md b/README.md index 2b469f5..67734a1 100644 --- a/README.md +++ b/README.md @@ -1,106 +1,88 @@ -

- Nest Logo -

+# 投标信息智能监控系统 -[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456 -[circleci-url]: https://circleci.com/gh/nestjs/nest +一个基于 TypeScript 的 Web 应用，用于自动爬取商务投标平台的最新信息，将符合条件的投标项目突出显示，为用户提供精准的投标信息监控服务。 -

A progressive Node.js framework for building efficient and scalable server-side applications.

-

-NPM Version -Package License -NPM Downloads -CircleCI -Discord -Backers on Open Collective -Sponsors on Open Collective - Donate us - Support us - Follow us on Twitter -

- +## 技术栈 -## Description +### 后端 +- **框架**: NestJS +- **语言**: TypeScript +- **数据库**: PostgreSQL (TypeORM) +- **爬虫**: axios +- **任务调度**: @nestjs/schedule +- **AI 服务**: OpenAI API -[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository. +### 前端 +- **框架**: Vue.js 3 +- **构建工具**: Vite +- **UI**: Tailwind CSS +- **状态管理**: Pinia -## Project setup +## 项目结构 -```bash -$ npm install +``` +src/ +├── ai/ # AI 模块 +│ ├── ai.controller.ts +│ ├── ai.service.ts +│ ├── Prompt.ts +│ └── entities/ +├── bids/ # 投标业务模块 +│ ├── controllers/ +│ ├── entities/ +│ └── services/ +├── crawler/ # 爬虫模块 +│ ├── services/ +│ │ ├── bid-crawler.service.ts +│ │ ├── cdt_target.ts +│ │ ├── chng_target.ts +│ │ ├── ceic_target.ts +│ │ ├── cgnpc_target.ts +│ │ ├── chdtp_target.ts +│ │ ├── cnncecp_target.ts +│ │ ├── cnooc_target.ts +│ │ ├── eps_target.ts +│ │ ├── espic_target.ts +│ │ ├── powerbeijing_target.ts +│ │ ├── sdicc_target.ts +│ │ └── szecp_target.ts +│ └── entities/ +├── database/ # 数据库模块 +├── keywords/ # 关键词管理模块 +├── schedule/ # 定时任务 +│ └── tasks/ +│ └── bid-crawl.task.ts +├── scripts/ # 脚本工具 +│ ├── ai-recommendations.ts +│ ├── crawl.ts +│ ├── deploy.ps1 +│ ├── remove-duplicates.ts +│ └── update-source.ts +└── common/ # 公共模块 + └── logger/ + +frontend/ +└── src/ + ├── components/ + │ ├── Dashboard.vue + │ ├── Dashboard-AI.vue + │ ├── PinnedProject.vue + │ ├── Bids.vue + │ ├── Keywords.vue + │ └── CrawlInfo.vue + ├── App.vue + └── main.ts ``` -## Compile and run the project +## 快速开始 -```bash -# development -$ npm run start +### 1. 环境准备 -# watch mode -$ npm run start:dev +确保已安装 Node.js (18+) 和 PostgreSQL。 -# production mode -$ npm run start:prod -``` +### 2. 数据库配置 -## Run tests - -```bash -# unit tests -$ npm run test - -# e2e tests -$ npm run test:e2e - -# test coverage -$ npm run test:cov -``` - -## Deployment - -When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information. - -If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps: - -```bash -$ npm install -g @nestjs/mau -$ mau deploy -``` - -With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure. - -## Resources - -Check out a few resources that may come in handy when working with NestJS: - -- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework. -- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy). -- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/). -- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks. -- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com). -- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com). -- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs). -- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com). - -## Support - -Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support). - -## Stay in touch - -- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec) -- Website - [https://nestjs.com](https://nestjs.com/) -- Twitter - [@nestframework](https://twitter.com/nestframework) - -## License - -Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE). - -## How to Run - -### 1. Database Setup -Update the `.env` file with your PostgreSQL credentials: +复制 `.env.example` 为 `.env` 并配置数据库连接： ```env DATABASE_TYPE=postgres @@ -110,73 +92,187 @@ DATABASE_USERNAME=your_username DATABASE_PASSWORD=your_password DATABASE_NAME=bidding DATABASE_SYNCHRONIZE=true -``` -### 2. Install Dependencies - -```bash -npm install -cd frontend && npm install -``` - -### 3. Build and Start - -```bash -# From the root directory -cd frontend && npm run build -cd .. -npm run build -npm run start -``` - -## Features - -### Frontend Features - -- **Dashboard**: View high priority bids and today's bids -- **Date Filtering**: - - Click "3天" or "7天" buttons to filter bids from the last 3 or 7 days - - The filter only limits the start date, showing all data from the selected start date onwards (including data newer than the end date) -- **Keyword Filtering**: Filter bids by keywords (saved in localStorage) -- **All Bids**: View all bids with pagination and source filtering -- **Keyword Management**: Add and delete keywords with weight-based priority - -### Backend Features - -- **Multi-Source Crawling**: Crawls bidding information from multiple sources: - - ChdtpCrawler - - ChngCrawler - - SzecpCrawler - - CdtCrawler - - EpsCrawler - - CnncecpCrawler - - CgnpcCrawler - - CeicCrawler - - EspicCrawler - - PowerbeijingCrawler -- **Automatic Retry**: If a crawler returns 0 items, it will be retried after all crawlers complete -- **Proxy Support**: Configurable proxy settings via environment variables -- **Scheduled Tasks**: Automatic crawling at scheduled intervals - -### Environment Variables - -```env -# Database -DATABASE_TYPE=postgres -DATABASE_HOST=localhost -DATABASE_PORT=5432 -DATABASE_USERNAME=your_username -DATABASE_PASSWORD=your_password -DATABASE_NAME=bidding -DATABASE_SYNCHRONIZE=true - -# Proxy (optional) +# 代理配置（可选） PROXY_HOST=your_proxy_host PROXY_PORT=your_proxy_port PROXY_USERNAME=your_proxy_username PROXY_PASSWORD=your_proxy_password + +# AI 配置 +OPENAI_API_KEY=your_openai_api_key ``` -## Initial Setup +### 3. 安装依赖 -The system will automatically initialize with the preset keywords: "山东", "海", "建设", "工程", "采购". You can manage these and view crawled bidding information at http://localhost:3000. \ No newline at end of file +```bash +# 安装后端依赖 +npm install + +# 安装前端依赖 +cd frontend && npm install +``` + +### 4. 运行项目 + +```bash +# 开发模式 - 后端 +npm run start:dev + +# 开发模式 - 前端 +cd frontend && npm run dev +``` + +### 5. 构建生产版本 + +```bash +# 构建前端 +cd frontend && npm run build + +# 构建后端 +cd .. && npm run build + +# 运行生产版本 +npm run start:prod +``` + +## 核心功能 + +### 智能爬虫模块 + +- **多源爬取**: 支持 12 个主流招标网站 + - 中国大唐集团电子商务平台 (CDT) + - 中国华能集团有限公司电子商务平台 (CHNG) + - 中国南方电网电子商务平台 (CSG) + - 中国海洋石油集团有限公司 (CNOOC) + - 中国华电集团有限公司电子商务平台 (CHDTP) + - 国家能源投资集团有限责任公司 (CNNCECP) + - 中国核工业集团有限公司 (CNNC) + - 中国电力建设集团有限公司 (POWERCHINA) + - 中国能源建设集团有限公司 (CEIC) + - 中国石油天然气集团有限公司 (CNPC) + - 国家电网有限公司 (SGCC) + - 北京电力交易中心 (POWERBEIJING) + +- **智能防封策略**: + - 随机请求间隔 (3-8 秒) + - 轮换 User-Agent + - 异常检测与自动重试机制 + - 代理支持 + +- **定时任务**: 每 30 分钟自动执行爬取 + +### 数据处理与存储 + +- **数据模型**: + - 投标项目标题 + - 详细页面 URL + - 发布时间 + - 招标单位 + - 截止日期 + - 关键词匹配 + - 优先级评分 + +- **增量存储**: + - 通过 URL 哈希值判断是否为新数据 + - 仅存储当天和最近 7 天的历史数据 + - 每日自动清理 30 天前的数据 + +### 关键词智能监控 + +- **预设关键词**: "山东", "海", "建设", "工程", "采购" +- **自定义关键词**: 通过 Web 界面添加/删除关键词 +- **权重设置**: 可设置关键词权重 (1-5 级) +- **匹配逻辑**: + - 标题完全匹配和部分匹配 + - 多关键词叠加权重 + - 支持正则表达式高级匹配 + +### AI 智能推荐 + +- **智能分析**: 使用 AI 分析投标信息的相关性 +- **推荐评分**: 基于关键词匹配和内容分析生成推荐评分 +- **智能摘要**: 自动生成投标信息摘要 + +### Web 展示界面 + +- **仪表盘**: + - 高优先级投标信息（匹配自定义关键词） + - 今日新增投标列表（按时间倒序） + - AI 推荐投标信息 + - 置顶项目 + +- **交互功能**: + - 关键词管理面板 + - 按日期/来源/关键词筛选 + - 信息标记已读/未读状态 + - 项目置顶功能 + - 爬取信息查看 + +- **响应式设计**: 适配桌面和移动设备 + +## API 接口 + +### 投标信息 +- `GET /api/bids` - 获取投标列表（支持分页、筛选） +- `GET /api/bids/high-priority` - 获取高优先级投标 +- `GET /api/bids/today` - 获取今日投标 + +### 关键词管理 +- `GET /api/keywords` - 获取所有关键词 +- `POST /api/keywords` - 添加新关键词 +- `DELETE /api/keywords/:id` - 删除关键词 + +### AI 服务 +- `GET /api/ai/recommendations` - 获取 AI 推荐投标 +- `POST /api/ai/analyze` - 分析投标信息 + +### 爬虫管理 +- `GET /api/crawler/info` - 获取爬取信息 +- `POST /api/crawler/trigger` - 手动触发爬取 + +## 前端路由 + +- `/` - 仪表盘（默认页面） +- `/bids` - 全部投标信息 +- `/keywords` - 关键词管理 +- `/ai` - AI 推荐页面 +- `/crawl-info` - 爬取信息 + +## 测试 + +```bash +# 单元测试 +npm run test + +# E2E 测试 +npm run test:e2e + +# 测试覆盖率 +npm run test:cov +``` + +## 部署 + +项目包含 PowerShell 部署脚本 `src/scripts/deploy.ps1`，用于自动化部署流程。 + +## 环境变量 + +| 变量名 | 说明 | 默认值 | +|--------|------|--------| +| DATABASE_TYPE | 数据库类型 | postgres | +| DATABASE_HOST | 数据库主机 | localhost | +| DATABASE_PORT | 数据库端口 | 5432 | +| DATABASE_USERNAME | 数据库用户名 | - | +| DATABASE_PASSWORD | 数据库密码 | - | +| DATABASE_NAME | 数据库名称 | bidding | +| DATABASE_SYNCHRONIZE | 自动同步数据库 | true | +| PROXY_HOST | 代理主机 | - | +| PROXY_PORT | 代理端口 | - | +| PROXY_USERNAME | 代理用户名 | - | +| PROXY_PASSWORD | 代理密码 | - | +| OPENAI_API_KEY | OpenAI API 密钥 | - | + +## 许可证 + +MIT