docs: 更新README以反映投标监控系统项目详情
This commit is contained in:
dmy
402
README.md
402
README.md
@@ -1,106 +1,88 @@
|
|||||||
<p align="center">
|
# 投标信息智能监控系统
|
||||||
<a href="http://nestjs.com/" target="blank"><img src="https://nestjs.com/img/logo-small.svg" width="120" alt="Nest Logo" /></a>
|
|
||||||
</p>
|
|
||||||
|
|
||||||
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
|
一个基于 TypeScript 的 Web 应用，用于自动爬取商务投标平台的最新信息，将符合条件的投标项目突出显示，为用户提供精准的投标信息监控服务。
|
||||||
[circleci-url]: https://circleci.com/gh/nestjs/nest
|
|
||||||
|
|
||||||
<p align="center">A progressive <a href="http://nodejs.org" target="_blank">Node.js</a> framework for building efficient and scalable server-side applications.</p>
|
## 技术栈
|
||||||
<p align="center">
|
|
||||||
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/v/@nestjs/core.svg" alt="NPM Version" /></a>
|
|
||||||
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/l/@nestjs/core.svg" alt="Package License" /></a>
|
|
||||||
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/dm/@nestjs/common.svg" alt="NPM Downloads" /></a>
|
|
||||||
<a href="https://circleci.com/gh/nestjs/nest" target="_blank"><img src="https://img.shields.io/circleci/build/github/nestjs/nest/master" alt="CircleCI" /></a>
|
|
||||||
<a href="https://discord.gg/G7Qnnhy" target="_blank"><img src="https://img.shields.io/badge/discord-online-brightgreen.svg" alt="Discord"/></a>
|
|
||||||
<a href="https://opencollective.com/nest#backer" target="_blank"><img src="https://opencollective.com/nest/backers/badge.svg" alt="Backers on Open Collective" /></a>
|
|
||||||
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://opencollective.com/nest/sponsors/badge.svg" alt="Sponsors on Open Collective" /></a>
|
|
||||||
<a href="https://paypal.me/kamilmysliwiec" target="_blank"><img src="https://img.shields.io/badge/Donate-PayPal-ff3f59.svg" alt="Donate us"/></a>
|
|
||||||
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://img.shields.io/badge/Support%20us-Open%20Collective-41B883.svg" alt="Support us"></a>
|
|
||||||
<a href="https://twitter.com/nestframework" target="_blank"><img src="https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow" alt="Follow us on Twitter"></a>
|
|
||||||
</p>
|
|
||||||
<!--[](https://opencollective.com/nest#backer)
|
|
||||||
[](https://opencollective.com/nest#sponsor)-->
|
|
||||||
|
|
||||||
## Description
|
### 后端
|
||||||
|
- **框架**: NestJS
|
||||||
|
- **语言**: TypeScript
|
||||||
|
- **数据库**: PostgreSQL (TypeORM)
|
||||||
|
- **爬虫**: axios
|
||||||
|
- **任务调度**: @nestjs/schedule
|
||||||
|
- **AI 服务**: OpenAI API
|
||||||
|
|
||||||
[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository.
|
### 前端
|
||||||
|
- **框架**: Vue.js 3
|
||||||
|
- **构建工具**: Vite
|
||||||
|
- **UI**: Tailwind CSS
|
||||||
|
- **状态管理**: Pinia
|
||||||
|
|
||||||
## Project setup
|
## 项目结构
|
||||||
|
|
||||||
```bash
|
```
|
||||||
$ npm install
|
src/
|
||||||
|
├── ai/ # AI 模块
|
||||||
|
│ ├── ai.controller.ts
|
||||||
|
│ ├── ai.service.ts
|
||||||
|
│ ├── Prompt.ts
|
||||||
|
│ └── entities/
|
||||||
|
├── bids/ # 投标业务模块
|
||||||
|
│ ├── controllers/
|
||||||
|
│ ├── entities/
|
||||||
|
│ └── services/
|
||||||
|
├── crawler/ # 爬虫模块
|
||||||
|
│ ├── services/
|
||||||
|
│ │ ├── bid-crawler.service.ts
|
||||||
|
│ │ ├── cdt_target.ts
|
||||||
|
│ │ ├── chng_target.ts
|
||||||
|
│ │ ├── ceic_target.ts
|
||||||
|
│ │ ├── cgnpc_target.ts
|
||||||
|
│ │ ├── chdtp_target.ts
|
||||||
|
│ │ ├── cnncecp_target.ts
|
||||||
|
│ │ ├── cnooc_target.ts
|
||||||
|
│ │ ├── eps_target.ts
|
||||||
|
│ │ ├── espic_target.ts
|
||||||
|
│ │ ├── powerbeijing_target.ts
|
||||||
|
│ │ ├── sdicc_target.ts
|
||||||
|
│ │ └── szecp_target.ts
|
||||||
|
│ └── entities/
|
||||||
|
├── database/ # 数据库模块
|
||||||
|
├── keywords/ # 关键词管理模块
|
||||||
|
├── schedule/ # 定时任务
|
||||||
|
│ └── tasks/
|
||||||
|
│ └── bid-crawl.task.ts
|
||||||
|
├── scripts/ # 脚本工具
|
||||||
|
│ ├── ai-recommendations.ts
|
||||||
|
│ ├── crawl.ts
|
||||||
|
│ ├── deploy.ps1
|
||||||
|
│ ├── remove-duplicates.ts
|
||||||
|
│ └── update-source.ts
|
||||||
|
└── common/ # 公共模块
|
||||||
|
└── logger/
|
||||||
|
|
||||||
|
frontend/
|
||||||
|
└── src/
|
||||||
|
├── components/
|
||||||
|
│ ├── Dashboard.vue
|
||||||
|
│ ├── Dashboard-AI.vue
|
||||||
|
│ ├── PinnedProject.vue
|
||||||
|
│ ├── Bids.vue
|
||||||
|
│ ├── Keywords.vue
|
||||||
|
│ └── CrawlInfo.vue
|
||||||
|
├── App.vue
|
||||||
|
└── main.ts
|
||||||
```
|
```
|
||||||
|
|
||||||
## Compile and run the project
|
## 快速开始
|
||||||
|
|
||||||
```bash
|
### 1. 环境准备
|
||||||
# development
|
|
||||||
$ npm run start
|
|
||||||
|
|
||||||
# watch mode
|
确保已安装 Node.js (18+) 和 PostgreSQL。
|
||||||
$ npm run start:dev
|
|
||||||
|
|
||||||
# production mode
|
### 2. 数据库配置
|
||||||
$ npm run start:prod
|
|
||||||
```
|
|
||||||
|
|
||||||
## Run tests
|
复制 `.env.example` 为 `.env` 并配置数据库连接：
|
||||||
|
|
||||||
```bash
|
|
||||||
# unit tests
|
|
||||||
$ npm run test
|
|
||||||
|
|
||||||
# e2e tests
|
|
||||||
$ npm run test:e2e
|
|
||||||
|
|
||||||
# test coverage
|
|
||||||
$ npm run test:cov
|
|
||||||
```
|
|
||||||
|
|
||||||
## Deployment
|
|
||||||
|
|
||||||
When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information.
|
|
||||||
|
|
||||||
If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ npm install -g @nestjs/mau
|
|
||||||
$ mau deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure.
|
|
||||||
|
|
||||||
## Resources
|
|
||||||
|
|
||||||
Check out a few resources that may come in handy when working with NestJS:
|
|
||||||
|
|
||||||
- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework.
|
|
||||||
- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
|
|
||||||
- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
|
|
||||||
- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks.
|
|
||||||
- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
|
|
||||||
- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
|
|
||||||
- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
|
|
||||||
- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).
|
|
||||||
|
|
||||||
## Support
|
|
||||||
|
|
||||||
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).
|
|
||||||
|
|
||||||
## Stay in touch
|
|
||||||
|
|
||||||
- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec)
|
|
||||||
- Website - [https://nestjs.com](https://nestjs.com/)
|
|
||||||
- Twitter - [@nestframework](https://twitter.com/nestframework)
|
|
||||||
|
|
||||||
## License
|
|
||||||
|
|
||||||
Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE).
|
|
||||||
|
|
||||||
## How to Run
|
|
||||||
|
|
||||||
### 1. Database Setup
|
|
||||||
Update the `.env` file with your PostgreSQL credentials:
|
|
||||||
|
|
||||||
```env
|
```env
|
||||||
DATABASE_TYPE=postgres
|
DATABASE_TYPE=postgres
|
||||||
@@ -110,73 +92,187 @@ DATABASE_USERNAME=your_username
|
|||||||
DATABASE_PASSWORD=your_password
|
DATABASE_PASSWORD=your_password
|
||||||
DATABASE_NAME=bidding
|
DATABASE_NAME=bidding
|
||||||
DATABASE_SYNCHRONIZE=true
|
DATABASE_SYNCHRONIZE=true
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Install Dependencies
|
# 代理配置（可选）
|
||||||
|
|
||||||
```bash
|
|
||||||
npm install
|
|
||||||
cd frontend && npm install
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Build and Start
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# From the root directory
|
|
||||||
cd frontend && npm run build
|
|
||||||
cd ..
|
|
||||||
npm run build
|
|
||||||
npm run start
|
|
||||||
```
|
|
||||||
|
|
||||||
## Features
|
|
||||||
|
|
||||||
### Frontend Features
|
|
||||||
|
|
||||||
- **Dashboard**: View high priority bids and today's bids
|
|
||||||
- **Date Filtering**:
|
|
||||||
- Click "3天" or "7天" buttons to filter bids from the last 3 or 7 days
|
|
||||||
- The filter only limits the start date, showing all data from the selected start date onwards (including data newer than the end date)
|
|
||||||
- **Keyword Filtering**: Filter bids by keywords (saved in localStorage)
|
|
||||||
- **All Bids**: View all bids with pagination and source filtering
|
|
||||||
- **Keyword Management**: Add and delete keywords with weight-based priority
|
|
||||||
|
|
||||||
### Backend Features
|
|
||||||
|
|
||||||
- **Multi-Source Crawling**: Crawls bidding information from multiple sources:
|
|
||||||
- ChdtpCrawler
|
|
||||||
- ChngCrawler
|
|
||||||
- SzecpCrawler
|
|
||||||
- CdtCrawler
|
|
||||||
- EpsCrawler
|
|
||||||
- CnncecpCrawler
|
|
||||||
- CgnpcCrawler
|
|
||||||
- CeicCrawler
|
|
||||||
- EspicCrawler
|
|
||||||
- PowerbeijingCrawler
|
|
||||||
- **Automatic Retry**: If a crawler returns 0 items, it will be retried after all crawlers complete
|
|
||||||
- **Proxy Support**: Configurable proxy settings via environment variables
|
|
||||||
- **Scheduled Tasks**: Automatic crawling at scheduled intervals
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
|
|
||||||
```env
|
|
||||||
# Database
|
|
||||||
DATABASE_TYPE=postgres
|
|
||||||
DATABASE_HOST=localhost
|
|
||||||
DATABASE_PORT=5432
|
|
||||||
DATABASE_USERNAME=your_username
|
|
||||||
DATABASE_PASSWORD=your_password
|
|
||||||
DATABASE_NAME=bidding
|
|
||||||
DATABASE_SYNCHRONIZE=true
|
|
||||||
|
|
||||||
# Proxy (optional)
|
|
||||||
PROXY_HOST=your_proxy_host
|
PROXY_HOST=your_proxy_host
|
||||||
PROXY_PORT=your_proxy_port
|
PROXY_PORT=your_proxy_port
|
||||||
PROXY_USERNAME=your_proxy_username
|
PROXY_USERNAME=your_proxy_username
|
||||||
PROXY_PASSWORD=your_proxy_password
|
PROXY_PASSWORD=your_proxy_password
|
||||||
|
|
||||||
|
# AI 配置
|
||||||
|
OPENAI_API_KEY=your_openai_api_key
|
||||||
```
|
```
|
||||||
|
|
||||||
## Initial Setup
|
### 3. 安装依赖
|
||||||
|
|
||||||
The system will automatically initialize with the preset keywords: "山东", "海", "建设", "工程", "采购". You can manage these and view crawled bidding information at http://localhost:3000.
|
```bash
|
||||||
|
# 安装后端依赖
|
||||||
|
npm install
|
||||||
|
|
||||||
|
# 安装前端依赖
|
||||||
|
cd frontend && npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. 运行项目
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 开发模式 - 后端
|
||||||
|
npm run start:dev
|
||||||
|
|
||||||
|
# 开发模式 - 前端
|
||||||
|
cd frontend && npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. 构建生产版本
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 构建前端
|
||||||
|
cd frontend && npm run build
|
||||||
|
|
||||||
|
# 构建后端
|
||||||
|
cd .. && npm run build
|
||||||
|
|
||||||
|
# 运行生产版本
|
||||||
|
npm run start:prod
|
||||||
|
```
|
||||||
|
|
||||||
|
## 核心功能
|
||||||
|
|
||||||
|
### 智能爬虫模块
|
||||||
|
|
||||||
|
- **多源爬取**: 支持 12 个主流招标网站
|
||||||
|
- 中国大唐集团电子商务平台 (CDT)
|
||||||
|
- 中国华能集团有限公司电子商务平台 (CHNG)
|
||||||
|
- 中国南方电网电子商务平台 (CSG)
|
||||||
|
- 中国海洋石油集团有限公司 (CNOOC)
|
||||||
|
- 中国华电集团有限公司电子商务平台 (CHDTP)
|
||||||
|
- 国家能源投资集团有限责任公司 (CNNCECP)
|
||||||
|
- 中国核工业集团有限公司 (CNNC)
|
||||||
|
- 中国电力建设集团有限公司 (POWERCHINA)
|
||||||
|
- 中国能源建设集团有限公司 (CEIC)
|
||||||
|
- 中国石油天然气集团有限公司 (CNPC)
|
||||||
|
- 国家电网有限公司 (SGCC)
|
||||||
|
- 北京电力交易中心 (POWERBEIJING)
|
||||||
|
|
||||||
|
- **智能防封策略**:
|
||||||
|
- 随机请求间隔 (3-8 秒)
|
||||||
|
- 轮换 User-Agent
|
||||||
|
- 异常检测与自动重试机制
|
||||||
|
- 代理支持
|
||||||
|
|
||||||
|
- **定时任务**: 每 30 分钟自动执行爬取
|
||||||
|
|
||||||
|
### 数据处理与存储
|
||||||
|
|
||||||
|
- **数据模型**:
|
||||||
|
- 投标项目标题
|
||||||
|
- 详细页面 URL
|
||||||
|
- 发布时间
|
||||||
|
- 招标单位
|
||||||
|
- 截止日期
|
||||||
|
- 关键词匹配
|
||||||
|
- 优先级评分
|
||||||
|
|
||||||
|
- **增量存储**:
|
||||||
|
- 通过 URL 哈希值判断是否为新数据
|
||||||
|
- 仅存储当天和最近 7 天的历史数据
|
||||||
|
- 每日自动清理 30 天前的数据
|
||||||
|
|
||||||
|
### 关键词智能监控
|
||||||
|
|
||||||
|
- **预设关键词**: "山东", "海", "建设", "工程", "采购"
|
||||||
|
- **自定义关键词**: 通过 Web 界面添加/删除关键词
|
||||||
|
- **权重设置**: 可设置关键词权重 (1-5 级)
|
||||||
|
- **匹配逻辑**:
|
||||||
|
- 标题完全匹配和部分匹配
|
||||||
|
- 多关键词叠加权重
|
||||||
|
- 支持正则表达式高级匹配
|
||||||
|
|
||||||
|
### AI 智能推荐
|
||||||
|
|
||||||
|
- **智能分析**: 使用 AI 分析投标信息的相关性
|
||||||
|
- **推荐评分**: 基于关键词匹配和内容分析生成推荐评分
|
||||||
|
- **智能摘要**: 自动生成投标信息摘要
|
||||||
|
|
||||||
|
### Web 展示界面
|
||||||
|
|
||||||
|
- **仪表盘**:
|
||||||
|
- 高优先级投标信息（匹配自定义关键词）
|
||||||
|
- 今日新增投标列表（按时间倒序）
|
||||||
|
- AI 推荐投标信息
|
||||||
|
- 置顶项目
|
||||||
|
|
||||||
|
- **交互功能**:
|
||||||
|
- 关键词管理面板
|
||||||
|
- 按日期/来源/关键词筛选
|
||||||
|
- 信息标记已读/未读状态
|
||||||
|
- 项目置顶功能
|
||||||
|
- 爬取信息查看
|
||||||
|
|
||||||
|
- **响应式设计**: 适配桌面和移动设备
|
||||||
|
|
||||||
|
## API 接口
|
||||||
|
|
||||||
|
### 投标信息
|
||||||
|
- `GET /api/bids` - 获取投标列表（支持分页、筛选）
|
||||||
|
- `GET /api/bids/high-priority` - 获取高优先级投标
|
||||||
|
- `GET /api/bids/today` - 获取今日投标
|
||||||
|
|
||||||
|
### 关键词管理
|
||||||
|
- `GET /api/keywords` - 获取所有关键词
|
||||||
|
- `POST /api/keywords` - 添加新关键词
|
||||||
|
- `DELETE /api/keywords/:id` - 删除关键词
|
||||||
|
|
||||||
|
### AI 服务
|
||||||
|
- `GET /api/ai/recommendations` - 获取 AI 推荐投标
|
||||||
|
- `POST /api/ai/analyze` - 分析投标信息
|
||||||
|
|
||||||
|
### 爬虫管理
|
||||||
|
- `GET /api/crawler/info` - 获取爬取信息
|
||||||
|
- `POST /api/crawler/trigger` - 手动触发爬取
|
||||||
|
|
||||||
|
## 前端路由
|
||||||
|
|
||||||
|
- `/` - 仪表盘（默认页面）
|
||||||
|
- `/bids` - 全部投标信息
|
||||||
|
- `/keywords` - 关键词管理
|
||||||
|
- `/ai` - AI 推荐页面
|
||||||
|
- `/crawl-info` - 爬取信息
|
||||||
|
|
||||||
|
## 测试
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 单元测试
|
||||||
|
npm run test
|
||||||
|
|
||||||
|
# E2E 测试
|
||||||
|
npm run test:e2e
|
||||||
|
|
||||||
|
# 测试覆盖率
|
||||||
|
npm run test:cov
|
||||||
|
```
|
||||||
|
|
||||||
|
## 部署
|
||||||
|
|
||||||
|
项目包含 PowerShell 部署脚本 `src/scripts/deploy.ps1`，用于自动化部署流程。
|
||||||
|
|
||||||
|
## 环境变量
|
||||||
|
|
||||||
|
| 变量名 | 说明 | 默认值 |
|
||||||
|
|--------|------|--------|
|
||||||
|
| DATABASE_TYPE | 数据库类型 | postgres |
|
||||||
|
| DATABASE_HOST | 数据库主机 | localhost |
|
||||||
|
| DATABASE_PORT | 数据库端口 | 5432 |
|
||||||
|
| DATABASE_USERNAME | 数据库用户名 | - |
|
||||||
|
| DATABASE_PASSWORD | 数据库密码 | - |
|
||||||
|
| DATABASE_NAME | 数据库名称 | bidding |
|
||||||
|
| DATABASE_SYNCHRONIZE | 自动同步数据库 | true |
|
||||||
|
| PROXY_HOST | 代理主机 | - |
|
||||||
|
| PROXY_PORT | 代理端口 | - |
|
||||||
|
| PROXY_USERNAME | 代理用户名 | - |
|
||||||
|
| PROXY_PASSWORD | 代理密码 | - |
|
||||||
|
| OPENAI_API_KEY | OpenAI API 密钥 | - |
|
||||||
|
|
||||||
|
## 许可证
|
||||||
|
|
||||||
|
MIT
|
||||||
|
|||||||
Reference in New Issue
Block a user