第一次提交
This commit is contained in:
dmy
7
.env
Normal file
7
.env
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
DATABASE_TYPE=mariadb
|
||||||
|
DATABASE_HOST=127.0.0.1
|
||||||
|
DATABASE_PORT=23306
|
||||||
|
DATABASE_USERNAME=root
|
||||||
|
DATABASE_PASSWORD=410491
|
||||||
|
DATABASE_NAME=bidding
|
||||||
|
DATABASE_SYNCHRONIZE=true
|
||||||
1
.gitignore
vendored
Normal file
1
.gitignore
vendored
Normal file
@@ -0,0 +1 @@
|
|||||||
|
node_modules
|
||||||
4
.prettierrc
Normal file
4
.prettierrc
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"singleQuote": true,
|
||||||
|
"trailingComma": "all"
|
||||||
|
}
|
||||||
122
README.md
Normal file
122
README.md
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
<p align="center">
|
||||||
|
<a href="http://nestjs.com/" target="blank"><img src="https://nestjs.com/img/logo-small.svg" width="120" alt="Nest Logo" /></a>
|
||||||
|
</p>
|
||||||
|
|
||||||
|
[circleci-image]: https://img.shields.io/circleci/build/github/nestjs/nest/master?token=abc123def456
|
||||||
|
[circleci-url]: https://circleci.com/gh/nestjs/nest
|
||||||
|
|
||||||
|
<p align="center">A progressive <a href="http://nodejs.org" target="_blank">Node.js</a> framework for building efficient and scalable server-side applications.</p>
|
||||||
|
<p align="center">
|
||||||
|
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/v/@nestjs/core.svg" alt="NPM Version" /></a>
|
||||||
|
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/l/@nestjs/core.svg" alt="Package License" /></a>
|
||||||
|
<a href="https://www.npmjs.com/~nestjscore" target="_blank"><img src="https://img.shields.io/npm/dm/@nestjs/common.svg" alt="NPM Downloads" /></a>
|
||||||
|
<a href="https://circleci.com/gh/nestjs/nest" target="_blank"><img src="https://img.shields.io/circleci/build/github/nestjs/nest/master" alt="CircleCI" /></a>
|
||||||
|
<a href="https://discord.gg/G7Qnnhy" target="_blank"><img src="https://img.shields.io/badge/discord-online-brightgreen.svg" alt="Discord"/></a>
|
||||||
|
<a href="https://opencollective.com/nest#backer" target="_blank"><img src="https://opencollective.com/nest/backers/badge.svg" alt="Backers on Open Collective" /></a>
|
||||||
|
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://opencollective.com/nest/sponsors/badge.svg" alt="Sponsors on Open Collective" /></a>
|
||||||
|
<a href="https://paypal.me/kamilmysliwiec" target="_blank"><img src="https://img.shields.io/badge/Donate-PayPal-ff3f59.svg" alt="Donate us"/></a>
|
||||||
|
<a href="https://opencollective.com/nest#sponsor" target="_blank"><img src="https://img.shields.io/badge/Support%20us-Open%20Collective-41B883.svg" alt="Support us"></a>
|
||||||
|
<a href="https://twitter.com/nestframework" target="_blank"><img src="https://img.shields.io/twitter/follow/nestframework.svg?style=social&label=Follow" alt="Follow us on Twitter"></a>
|
||||||
|
</p>
|
||||||
|
<!--[](https://opencollective.com/nest#backer)
|
||||||
|
[](https://opencollective.com/nest#sponsor)-->
|
||||||
|
|
||||||
|
## Description
|
||||||
|
|
||||||
|
[Nest](https://github.com/nestjs/nest) framework TypeScript starter repository.
|
||||||
|
|
||||||
|
## Project setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ npm install
|
||||||
|
```
|
||||||
|
|
||||||
|
## Compile and run the project
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# development
|
||||||
|
$ npm run start
|
||||||
|
|
||||||
|
# watch mode
|
||||||
|
$ npm run start:dev
|
||||||
|
|
||||||
|
# production mode
|
||||||
|
$ npm run start:prod
|
||||||
|
```
|
||||||
|
|
||||||
|
## Run tests
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# unit tests
|
||||||
|
$ npm run test
|
||||||
|
|
||||||
|
# e2e tests
|
||||||
|
$ npm run test:e2e
|
||||||
|
|
||||||
|
# test coverage
|
||||||
|
$ npm run test:cov
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deployment
|
||||||
|
|
||||||
|
When you're ready to deploy your NestJS application to production, there are some key steps you can take to ensure it runs as efficiently as possible. Check out the [deployment documentation](https://docs.nestjs.com/deployment) for more information.
|
||||||
|
|
||||||
|
If you are looking for a cloud-based platform to deploy your NestJS application, check out [Mau](https://mau.nestjs.com), our official platform for deploying NestJS applications on AWS. Mau makes deployment straightforward and fast, requiring just a few simple steps:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ npm install -g @nestjs/mau
|
||||||
|
$ mau deploy
|
||||||
|
```
|
||||||
|
|
||||||
|
With Mau, you can deploy your application in just a few clicks, allowing you to focus on building features rather than managing infrastructure.
|
||||||
|
|
||||||
|
## Resources
|
||||||
|
|
||||||
|
Check out a few resources that may come in handy when working with NestJS:
|
||||||
|
|
||||||
|
- Visit the [NestJS Documentation](https://docs.nestjs.com) to learn more about the framework.
|
||||||
|
- For questions and support, please visit our [Discord channel](https://discord.gg/G7Qnnhy).
|
||||||
|
- To dive deeper and get more hands-on experience, check out our official video [courses](https://courses.nestjs.com/).
|
||||||
|
- Deploy your application to AWS with the help of [NestJS Mau](https://mau.nestjs.com) in just a few clicks.
|
||||||
|
- Visualize your application graph and interact with the NestJS application in real-time using [NestJS Devtools](https://devtools.nestjs.com).
|
||||||
|
- Need help with your project (part-time to full-time)? Check out our official [enterprise support](https://enterprise.nestjs.com).
|
||||||
|
- To stay in the loop and get updates, follow us on [X](https://x.com/nestframework) and [LinkedIn](https://linkedin.com/company/nestjs).
|
||||||
|
- Looking for a job, or have a job to offer? Check out our official [Jobs board](https://jobs.nestjs.com).
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
Nest is an MIT-licensed open source project. It can grow thanks to the sponsors and support by the amazing backers. If you'd like to join them, please [read more here](https://docs.nestjs.com/support).
|
||||||
|
|
||||||
|
## Stay in touch
|
||||||
|
|
||||||
|
- Author - [Kamil Myśliwiec](https://twitter.com/kammysliwiec)
|
||||||
|
- Website - [https://nestjs.com](https://nestjs.com/)
|
||||||
|
- Twitter - [@nestframework](https://twitter.com/nestframework)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Nest is [MIT licensed](https://github.com/nestjs/nest/blob/master/LICENSE).
|
||||||
|
|
||||||
|
How to Run:
|
||||||
|
1. Database Setup: Update the .env file with your PostgreSQL credentials.
|
||||||
|
|
||||||
|
1 DATABASE_TYPE=postgres
|
||||||
|
2 DATABASE_HOST=localhost
|
||||||
|
3 DATABASE_PORT=5432
|
||||||
|
4 DATABASE_USERNAME=your_username
|
||||||
|
5 DATABASE_PASSWORD=your_password
|
||||||
|
6 DATABASE_NAME=bidding
|
||||||
|
7 DATABASE_SYNCHRONIZE=true
|
||||||
|
2. Install Dependencies:
|
||||||
|
1 npm install
|
||||||
|
2 cd frontend && npm install
|
||||||
|
3. Build and Start:
|
||||||
|
|
||||||
|
1 # From the root directory
|
||||||
|
2 cd frontend && npm run build
|
||||||
|
3 cd ..
|
||||||
|
4 npm run build
|
||||||
|
5 npm run start
|
||||||
|
|
||||||
|
The system will automatically initialize with the preset keywords: "山东", "海", "建设", "工程", "采购". You can
|
||||||
|
manage these and view crawled bidding information at http://localhost:3000.
|
||||||
35
eslint.config.mjs
Normal file
35
eslint.config.mjs
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
// @ts-check
|
||||||
|
import eslint from '@eslint/js';
|
||||||
|
import eslintPluginPrettierRecommended from 'eslint-plugin-prettier/recommended';
|
||||||
|
import globals from 'globals';
|
||||||
|
import tseslint from 'typescript-eslint';
|
||||||
|
|
||||||
|
export default tseslint.config(
|
||||||
|
{
|
||||||
|
ignores: ['eslint.config.mjs'],
|
||||||
|
},
|
||||||
|
eslint.configs.recommended,
|
||||||
|
...tseslint.configs.recommendedTypeChecked,
|
||||||
|
eslintPluginPrettierRecommended,
|
||||||
|
{
|
||||||
|
languageOptions: {
|
||||||
|
globals: {
|
||||||
|
...globals.node,
|
||||||
|
...globals.jest,
|
||||||
|
},
|
||||||
|
sourceType: 'commonjs',
|
||||||
|
parserOptions: {
|
||||||
|
projectService: true,
|
||||||
|
tsconfigRootDir: import.meta.dirname,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
rules: {
|
||||||
|
'@typescript-eslint/no-explicit-any': 'off',
|
||||||
|
'@typescript-eslint/no-floating-promises': 'warn',
|
||||||
|
'@typescript-eslint/no-unsafe-argument': 'warn',
|
||||||
|
"prettier/prettier": ["error", { endOfLine: "auto" }],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
24
frontend/.gitignore
vendored
Normal file
24
frontend/.gitignore
vendored
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
# Logs
|
||||||
|
logs
|
||||||
|
*.log
|
||||||
|
npm-debug.log*
|
||||||
|
yarn-debug.log*
|
||||||
|
yarn-error.log*
|
||||||
|
pnpm-debug.log*
|
||||||
|
lerna-debug.log*
|
||||||
|
|
||||||
|
node_modules
|
||||||
|
dist
|
||||||
|
dist-ssr
|
||||||
|
*.local
|
||||||
|
|
||||||
|
# Editor directories and files
|
||||||
|
.vscode/*
|
||||||
|
!.vscode/extensions.json
|
||||||
|
.idea
|
||||||
|
.DS_Store
|
||||||
|
*.suo
|
||||||
|
*.ntvs*
|
||||||
|
*.njsproj
|
||||||
|
*.sln
|
||||||
|
*.sw?
|
||||||
5
frontend/README.md
Normal file
5
frontend/README.md
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
# Vue 3 + TypeScript + Vite
|
||||||
|
|
||||||
|
This template should help get you started developing with Vue 3 and TypeScript in Vite. The template uses Vue 3 `<script setup>` SFCs, check out the [script setup docs](https://v3.vuejs.org/api/sfc-script-setup.html#sfc-script-setup) to learn more.
|
||||||
|
|
||||||
|
Learn more about the recommended Project Setup and IDE Support in the [Vue Docs TypeScript Guide](https://vuejs.org/guide/typescript/overview.html#project-setup).
|
||||||
13
frontend/index.html
Normal file
13
frontend/index.html
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head>
|
||||||
|
<meta charset="UTF-8" />
|
||||||
|
<link rel="icon" type="image/svg+xml" href="/vite.svg" />
|
||||||
|
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||||
|
<title>frontend</title>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<div id="app"></div>
|
||||||
|
<script type="module" src="/src/main.ts"></script>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
2003
frontend/package-lock.json
generated
Normal file
2003
frontend/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
25
frontend/package.json
Normal file
25
frontend/package.json
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
{
|
||||||
|
"name": "frontend",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.0.0",
|
||||||
|
"type": "module",
|
||||||
|
"scripts": {
|
||||||
|
"dev": "vite",
|
||||||
|
"build": "vue-tsc -b && vite build",
|
||||||
|
"preview": "vite preview"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@element-plus/icons-vue": "^2.3.2",
|
||||||
|
"axios": "^1.13.2",
|
||||||
|
"element-plus": "^2.13.1",
|
||||||
|
"vue": "^3.5.24"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/node": "^24.10.1",
|
||||||
|
"@vitejs/plugin-vue": "^6.0.1",
|
||||||
|
"@vue/tsconfig": "^0.8.1",
|
||||||
|
"typescript": "~5.9.3",
|
||||||
|
"vite": "^7.2.4",
|
||||||
|
"vue-tsc": "^3.1.4"
|
||||||
|
}
|
||||||
|
}
|
||||||
248
frontend/src/App.vue
Normal file
248
frontend/src/App.vue
Normal file
@@ -0,0 +1,248 @@
|
|||||||
|
<template>
|
||||||
|
<el-container class="layout-container" style="height: 100vh">
|
||||||
|
<el-aside width="200px" style="background-color: #545c64">
|
||||||
|
<div class="logo">BID MONITOR</div>
|
||||||
|
<el-menu
|
||||||
|
active-text-color="#ffd04b"
|
||||||
|
background-color="#545c64"
|
||||||
|
class="el-menu-vertical-demo"
|
||||||
|
default-active="1"
|
||||||
|
text-color="#fff"
|
||||||
|
@select="handleSelect"
|
||||||
|
>
|
||||||
|
<el-menu-item index="1">
|
||||||
|
<el-icon><DataBoard /></el-icon>
|
||||||
|
<span>Dashboard</span>
|
||||||
|
</el-menu-item>
|
||||||
|
<el-menu-item index="2">
|
||||||
|
<el-icon><Document /></el-icon>
|
||||||
|
<span>Bids</span>
|
||||||
|
</el-menu-item>
|
||||||
|
<el-menu-item index="3">
|
||||||
|
<el-icon><Setting /></el-icon>
|
||||||
|
<span>Keywords</span>
|
||||||
|
</el-menu-item>
|
||||||
|
</el-menu>
|
||||||
|
</el-aside>
|
||||||
|
|
||||||
|
<el-container>
|
||||||
|
<el-header style="text-align: right; font-size: 12px">
|
||||||
|
<span>Admin</span>
|
||||||
|
</el-header>
|
||||||
|
|
||||||
|
<el-main>
|
||||||
|
<div v-if="activeIndex === '1'">
|
||||||
|
<div style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px;">
|
||||||
|
<h2 style="margin: 0;">Dashboard</h2>
|
||||||
|
<el-button type="primary" :loading="crawling" @click="handleCrawl">
|
||||||
|
<el-icon style="margin-right: 5px"><Refresh /></el-icon>
|
||||||
|
立刻抓取
|
||||||
|
</el-button>
|
||||||
|
</div>
|
||||||
|
<el-row :gutter="20">
|
||||||
|
<el-col :span="24">
|
||||||
|
<el-card class="box-card" shadow="hover">
|
||||||
|
<template #header>
|
||||||
|
<div class="card-header">
|
||||||
|
<span>High Priority Bids</span>
|
||||||
|
<el-tag type="danger">Top 10</el-tag>
|
||||||
|
</div>
|
||||||
|
</template>
|
||||||
|
<el-table :data="highPriorityBids" style="width: 100%" size="small">
|
||||||
|
<el-table-column prop="title" label="Title">
|
||||||
|
<template #default="scope">
|
||||||
|
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
|
||||||
|
</template>
|
||||||
|
</el-table-column>
|
||||||
|
<el-table-column prop="source" label="Source" width="120" />
|
||||||
|
<el-table-column prop="publishDate" label="Date" width="120">
|
||||||
|
<template #default="scope">{{ formatDate(scope.row.publishDate) }}</template>
|
||||||
|
</el-table-column>
|
||||||
|
</el-table>
|
||||||
|
</el-card>
|
||||||
|
</el-col>
|
||||||
|
</el-row>
|
||||||
|
<el-divider />
|
||||||
|
<h3>Today's Bids</h3>
|
||||||
|
<el-table :data="bids" v-loading="loading" style="width: 100%">
|
||||||
|
<el-table-column prop="title" label="Title">
|
||||||
|
<template #default="scope">
|
||||||
|
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
|
||||||
|
</template>
|
||||||
|
</el-table-column>
|
||||||
|
<el-table-column prop="source" label="Source" width="150" />
|
||||||
|
<el-table-column prop="publishDate" label="Date" width="150">
|
||||||
|
<template #default="scope">{{ formatDate(scope.row.publishDate) }}</template>
|
||||||
|
</el-table-column>
|
||||||
|
</el-table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div v-if="activeIndex === '2'">
|
||||||
|
<h2>All Bids</h2>
|
||||||
|
<el-table :data="bids" v-loading="loading" style="width: 100%">
|
||||||
|
<el-table-column prop="title" label="Title">
|
||||||
|
<template #default="scope">
|
||||||
|
<a :href="scope.row.url" target="_blank">{{ scope.row.title }}</a>
|
||||||
|
</template>
|
||||||
|
</el-table-column>
|
||||||
|
<el-table-column prop="source" label="Source" width="150" />
|
||||||
|
<el-table-column prop="publishDate" label="Date" width="150">
|
||||||
|
<template #default="scope">{{ formatDate(scope.row.publishDate) }}</template>
|
||||||
|
</el-table-column>
|
||||||
|
</el-table>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div v-if="activeIndex === '3'">
|
||||||
|
<div class="card-header" style="display: flex; justify-content: space-between; align-items: center; margin-bottom: 20px;">
|
||||||
|
<h2>Keyword Management</h2>
|
||||||
|
<el-button type="primary" @click="dialogVisible = true">Add Keyword</el-button>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<el-table :data="keywords" v-loading="loading" style="width: 100%">
|
||||||
|
<el-table-column prop="word" label="Keyword" />
|
||||||
|
<el-table-column prop="weight" label="Weight" />
|
||||||
|
<el-table-column label="Action">
|
||||||
|
<template #default="scope">
|
||||||
|
<el-button type="danger" size="small" @click="handleDeleteKeyword(scope.row.id)">Delete</el-button>
|
||||||
|
</template>
|
||||||
|
</el-table-column>
|
||||||
|
</el-table>
|
||||||
|
</div>
|
||||||
|
</el-main>
|
||||||
|
</el-container>
|
||||||
|
|
||||||
|
<el-dialog v-model="dialogVisible" title="Add Keyword" width="30%">
|
||||||
|
<el-form :model="form" label-width="120px">
|
||||||
|
<el-form-item label="Keyword">
|
||||||
|
<el-input v-model="form.word" />
|
||||||
|
</el-form-item>
|
||||||
|
<el-form-item label="Weight">
|
||||||
|
<el-input-number v-model="form.weight" :min="1" :max="5" />
|
||||||
|
</el-form-item>
|
||||||
|
</el-form>
|
||||||
|
<template #footer>
|
||||||
|
<span class="dialog-footer">
|
||||||
|
<el-button @click="dialogVisible = false">Cancel</el-button>
|
||||||
|
<el-button type="primary" @click="handleAddKeyword">Confirm</el-button>
|
||||||
|
</span>
|
||||||
|
</template>
|
||||||
|
</el-dialog>
|
||||||
|
</el-container>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<script setup lang="ts">
|
||||||
|
import { ref, onMounted, reactive } from 'vue'
|
||||||
|
import axios from 'axios'
|
||||||
|
import { ElMessage } from 'element-plus'
|
||||||
|
import { DataBoard, Document, Setting, Refresh } from '@element-plus/icons-vue'
|
||||||
|
|
||||||
|
const activeIndex = ref('1')
|
||||||
|
const bids = ref<any[]>([])
|
||||||
|
const highPriorityBids = ref<any[]>([])
|
||||||
|
const keywords = ref<any[]>([])
|
||||||
|
const loading = ref(false)
|
||||||
|
const crawling = ref(false)
|
||||||
|
const dialogVisible = ref(false)
|
||||||
|
|
||||||
|
const form = reactive({
|
||||||
|
word: '',
|
||||||
|
weight: 1
|
||||||
|
})
|
||||||
|
|
||||||
|
const handleSelect = (key: string) => {
|
||||||
|
activeIndex.value = key
|
||||||
|
}
|
||||||
|
|
||||||
|
const formatDate = (dateString: string) => {
|
||||||
|
if (!dateString) return '-'
|
||||||
|
return new Date(dateString).toLocaleDateString()
|
||||||
|
}
|
||||||
|
|
||||||
|
const fetchData = async () => {
|
||||||
|
loading.value = true
|
||||||
|
try {
|
||||||
|
const [bidsRes, highRes, kwRes] = await Promise.all([
|
||||||
|
axios.get('/api/bids'),
|
||||||
|
axios.get('/api/bids/high-priority'),
|
||||||
|
axios.get('/api/keywords')
|
||||||
|
])
|
||||||
|
bids.value = bidsRes.data.items
|
||||||
|
highPriorityBids.value = highRes.data
|
||||||
|
keywords.value = kwRes.data
|
||||||
|
} catch (error) {
|
||||||
|
ElMessage.error('Failed to fetch data')
|
||||||
|
} finally {
|
||||||
|
loading.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleCrawl = async () => {
|
||||||
|
crawling.value = true
|
||||||
|
try {
|
||||||
|
await axios.post('/api/crawler/run')
|
||||||
|
ElMessage.success('Crawl completed successfully')
|
||||||
|
fetchData() // Refresh data after crawl
|
||||||
|
} catch (error) {
|
||||||
|
ElMessage.error('Failed to run crawl task')
|
||||||
|
} finally {
|
||||||
|
crawling.value = false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleAddKeyword = async () => {
|
||||||
|
if (!form.word) {
|
||||||
|
ElMessage.warning('Please enter a keyword')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await axios.post('/api/keywords', form)
|
||||||
|
ElMessage.success('Keyword added')
|
||||||
|
dialogVisible.value = false
|
||||||
|
form.word = ''
|
||||||
|
form.weight = 1
|
||||||
|
fetchData()
|
||||||
|
} catch (error) {
|
||||||
|
ElMessage.error('Failed to add keyword')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const handleDeleteKeyword = async (id: string) => {
|
||||||
|
try {
|
||||||
|
await axios.delete(`/api/keywords/${id}`)
|
||||||
|
ElMessage.success('Keyword deleted')
|
||||||
|
fetchData()
|
||||||
|
} catch (error) {
|
||||||
|
ElMessage.error('Failed to delete keyword')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
onMounted(() => {
|
||||||
|
fetchData()
|
||||||
|
})
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.layout-container .el-header {
|
||||||
|
background-color: #fff;
|
||||||
|
color: var(--el-text-color-primary);
|
||||||
|
line-height: 60px;
|
||||||
|
border-bottom: 1px solid #eee;
|
||||||
|
}
|
||||||
|
.layout-container .el-aside {
|
||||||
|
color: var(--el-text-color-primary);
|
||||||
|
}
|
||||||
|
.logo {
|
||||||
|
height: 60px;
|
||||||
|
line-height: 60px;
|
||||||
|
text-align: center;
|
||||||
|
color: white;
|
||||||
|
font-weight: bold;
|
||||||
|
font-size: 18px;
|
||||||
|
background-color: #434a50;
|
||||||
|
}
|
||||||
|
.card-header {
|
||||||
|
display: flex;
|
||||||
|
justify-content: space-between;
|
||||||
|
align-items: center;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
41
frontend/src/components/HelloWorld.vue
Normal file
41
frontend/src/components/HelloWorld.vue
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
<script setup lang="ts">
|
||||||
|
import { ref } from 'vue'
|
||||||
|
|
||||||
|
defineProps<{ msg: string }>()
|
||||||
|
|
||||||
|
const count = ref(0)
|
||||||
|
</script>
|
||||||
|
|
||||||
|
<template>
|
||||||
|
<h1>{{ msg }}</h1>
|
||||||
|
|
||||||
|
<div class="card">
|
||||||
|
<button type="button" @click="count++">count is {{ count }}</button>
|
||||||
|
<p>
|
||||||
|
Edit
|
||||||
|
<code>components/HelloWorld.vue</code> to test HMR
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<p>
|
||||||
|
Check out
|
||||||
|
<a href="https://vuejs.org/guide/quick-start.html#local" target="_blank"
|
||||||
|
>create-vue</a
|
||||||
|
>, the official Vue + Vite starter
|
||||||
|
</p>
|
||||||
|
<p>
|
||||||
|
Learn more about IDE Support for Vue in the
|
||||||
|
<a
|
||||||
|
href="https://vuejs.org/guide/scaling-up/tooling.html#ide-support"
|
||||||
|
target="_blank"
|
||||||
|
>Vue Docs Scaling up Guide</a
|
||||||
|
>.
|
||||||
|
</p>
|
||||||
|
<p class="read-the-docs">Click on the Vite and Vue logos to learn more</p>
|
||||||
|
</template>
|
||||||
|
|
||||||
|
<style scoped>
|
||||||
|
.read-the-docs {
|
||||||
|
color: #888;
|
||||||
|
}
|
||||||
|
</style>
|
||||||
14
frontend/src/main.ts
Normal file
14
frontend/src/main.ts
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
import { createApp } from 'vue'
|
||||||
|
import ElementPlus from 'element-plus'
|
||||||
|
import 'element-plus/dist/index.css'
|
||||||
|
import * as ElementPlusIconsVue from '@element-plus/icons-vue'
|
||||||
|
import App from './App.vue'
|
||||||
|
|
||||||
|
const app = createApp(App)
|
||||||
|
|
||||||
|
app.use(ElementPlus)
|
||||||
|
for (const [key, component] of Object.entries(ElementPlusIconsVue)) {
|
||||||
|
app.component(key, component)
|
||||||
|
}
|
||||||
|
|
||||||
|
app.mount('#app')
|
||||||
79
frontend/src/style.css
Normal file
79
frontend/src/style.css
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
:root {
|
||||||
|
font-family: system-ui, Avenir, Helvetica, Arial, sans-serif;
|
||||||
|
line-height: 1.5;
|
||||||
|
font-weight: 400;
|
||||||
|
|
||||||
|
color-scheme: light dark;
|
||||||
|
color: rgba(255, 255, 255, 0.87);
|
||||||
|
background-color: #242424;
|
||||||
|
|
||||||
|
font-synthesis: none;
|
||||||
|
text-rendering: optimizeLegibility;
|
||||||
|
-webkit-font-smoothing: antialiased;
|
||||||
|
-moz-osx-font-smoothing: grayscale;
|
||||||
|
}
|
||||||
|
|
||||||
|
a {
|
||||||
|
font-weight: 500;
|
||||||
|
color: #646cff;
|
||||||
|
text-decoration: inherit;
|
||||||
|
}
|
||||||
|
a:hover {
|
||||||
|
color: #535bf2;
|
||||||
|
}
|
||||||
|
|
||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
display: flex;
|
||||||
|
place-items: center;
|
||||||
|
min-width: 320px;
|
||||||
|
min-height: 100vh;
|
||||||
|
}
|
||||||
|
|
||||||
|
h1 {
|
||||||
|
font-size: 3.2em;
|
||||||
|
line-height: 1.1;
|
||||||
|
}
|
||||||
|
|
||||||
|
button {
|
||||||
|
border-radius: 8px;
|
||||||
|
border: 1px solid transparent;
|
||||||
|
padding: 0.6em 1.2em;
|
||||||
|
font-size: 1em;
|
||||||
|
font-weight: 500;
|
||||||
|
font-family: inherit;
|
||||||
|
background-color: #1a1a1a;
|
||||||
|
cursor: pointer;
|
||||||
|
transition: border-color 0.25s;
|
||||||
|
}
|
||||||
|
button:hover {
|
||||||
|
border-color: #646cff;
|
||||||
|
}
|
||||||
|
button:focus,
|
||||||
|
button:focus-visible {
|
||||||
|
outline: 4px auto -webkit-focus-ring-color;
|
||||||
|
}
|
||||||
|
|
||||||
|
.card {
|
||||||
|
padding: 2em;
|
||||||
|
}
|
||||||
|
|
||||||
|
#app {
|
||||||
|
max-width: 1280px;
|
||||||
|
margin: 0 auto;
|
||||||
|
padding: 2rem;
|
||||||
|
text-align: center;
|
||||||
|
}
|
||||||
|
|
||||||
|
@media (prefers-color-scheme: light) {
|
||||||
|
:root {
|
||||||
|
color: #213547;
|
||||||
|
background-color: #ffffff;
|
||||||
|
}
|
||||||
|
a:hover {
|
||||||
|
color: #747bff;
|
||||||
|
}
|
||||||
|
button {
|
||||||
|
background-color: #f9f9f9;
|
||||||
|
}
|
||||||
|
}
|
||||||
16
frontend/tsconfig.app.json
Normal file
16
frontend/tsconfig.app.json
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
{
|
||||||
|
"extends": "@vue/tsconfig/tsconfig.dom.json",
|
||||||
|
"compilerOptions": {
|
||||||
|
"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.app.tsbuildinfo",
|
||||||
|
"types": ["vite/client"],
|
||||||
|
|
||||||
|
/* Linting */
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"erasableSyntaxOnly": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"noUncheckedSideEffectImports": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts", "src/**/*.tsx", "src/**/*.vue"]
|
||||||
|
}
|
||||||
7
frontend/tsconfig.json
Normal file
7
frontend/tsconfig.json
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"files": [],
|
||||||
|
"references": [
|
||||||
|
{ "path": "./tsconfig.app.json" },
|
||||||
|
{ "path": "./tsconfig.node.json" }
|
||||||
|
]
|
||||||
|
}
|
||||||
26
frontend/tsconfig.node.json
Normal file
26
frontend/tsconfig.node.json
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"tsBuildInfoFile": "./node_modules/.tmp/tsconfig.node.tsbuildinfo",
|
||||||
|
"target": "ES2023",
|
||||||
|
"lib": ["ES2023"],
|
||||||
|
"module": "ESNext",
|
||||||
|
"types": ["node"],
|
||||||
|
"skipLibCheck": true,
|
||||||
|
|
||||||
|
/* Bundler mode */
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"allowImportingTsExtensions": true,
|
||||||
|
"verbatimModuleSyntax": true,
|
||||||
|
"moduleDetection": "force",
|
||||||
|
"noEmit": true,
|
||||||
|
|
||||||
|
/* Linting */
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"erasableSyntaxOnly": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"noUncheckedSideEffectImports": true
|
||||||
|
},
|
||||||
|
"include": ["vite.config.ts"]
|
||||||
|
}
|
||||||
15
frontend/vite.config.ts
Normal file
15
frontend/vite.config.ts
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
import { defineConfig } from 'vite'
|
||||||
|
import vue from '@vitejs/plugin-vue'
|
||||||
|
|
||||||
|
// https://vite.dev/config/
|
||||||
|
export default defineConfig({
|
||||||
|
plugins: [vue()],
|
||||||
|
server: {
|
||||||
|
proxy: {
|
||||||
|
'/api': {
|
||||||
|
target: 'http://localhost:3000',
|
||||||
|
changeOrigin: true,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
8
nest-cli.json
Normal file
8
nest-cli.json
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json.schemastore.org/nest-cli",
|
||||||
|
"collection": "@nestjs/schematics",
|
||||||
|
"sourceRoot": "src",
|
||||||
|
"compilerOptions": {
|
||||||
|
"deleteOutDir": true
|
||||||
|
}
|
||||||
|
}
|
||||||
11423
package-lock.json
generated
Normal file
11423
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
81
package.json
Normal file
81
package.json
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
{
|
||||||
|
"name": "bidding",
|
||||||
|
"version": "0.0.1",
|
||||||
|
"description": "",
|
||||||
|
"author": "",
|
||||||
|
"private": true,
|
||||||
|
"license": "UNLICENSED",
|
||||||
|
"scripts": {
|
||||||
|
"build": "nest build",
|
||||||
|
"format": "prettier --write \"src/**/*.ts\" \"test/**/*.ts\"",
|
||||||
|
"start": "nest start",
|
||||||
|
"start:dev": "nest start --watch",
|
||||||
|
"start:debug": "nest start --debug --watch",
|
||||||
|
"start:prod": "node dist/main",
|
||||||
|
"lint": "eslint \"{src,apps,libs,test}/**/*.ts\" --fix",
|
||||||
|
"test": "jest",
|
||||||
|
"test:watch": "jest --watch",
|
||||||
|
"test:cov": "jest --coverage",
|
||||||
|
"test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
|
||||||
|
"test:e2e": "jest --config ./test/jest-e2e.json"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"@nestjs/common": "^11.0.1",
|
||||||
|
"@nestjs/config": "^4.0.2",
|
||||||
|
"@nestjs/core": "^11.0.1",
|
||||||
|
"@nestjs/platform-express": "^11.0.1",
|
||||||
|
"@nestjs/schedule": "^6.1.0",
|
||||||
|
"@nestjs/serve-static": "^5.0.4",
|
||||||
|
"@nestjs/typeorm": "^11.0.0",
|
||||||
|
"axios": "^1.13.2",
|
||||||
|
"class-transformer": "^0.5.1",
|
||||||
|
"class-validator": "^0.14.3",
|
||||||
|
"mysql2": "^3.16.0",
|
||||||
|
"puppeteer": "^24.34.0",
|
||||||
|
"reflect-metadata": "^0.2.2",
|
||||||
|
"rxjs": "^7.8.1",
|
||||||
|
"typeorm": "^0.3.28"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@eslint/eslintrc": "^3.2.0",
|
||||||
|
"@eslint/js": "^9.18.0",
|
||||||
|
"@nestjs/cli": "^11.0.0",
|
||||||
|
"@nestjs/schematics": "^11.0.0",
|
||||||
|
"@nestjs/testing": "^11.0.1",
|
||||||
|
"@types/express": "^5.0.0",
|
||||||
|
"@types/jest": "^30.0.0",
|
||||||
|
"@types/node": "^22.10.7",
|
||||||
|
"@types/supertest": "^6.0.2",
|
||||||
|
"eslint": "^9.18.0",
|
||||||
|
"eslint-config-prettier": "^10.0.1",
|
||||||
|
"eslint-plugin-prettier": "^5.2.2",
|
||||||
|
"globals": "^16.0.0",
|
||||||
|
"jest": "^30.0.0",
|
||||||
|
"prettier": "^3.4.2",
|
||||||
|
"source-map-support": "^0.5.21",
|
||||||
|
"supertest": "^7.0.0",
|
||||||
|
"ts-jest": "^29.2.5",
|
||||||
|
"ts-loader": "^9.5.2",
|
||||||
|
"ts-node": "^10.9.2",
|
||||||
|
"tsconfig-paths": "^4.2.0",
|
||||||
|
"typescript": "^5.7.3",
|
||||||
|
"typescript-eslint": "^8.20.0"
|
||||||
|
},
|
||||||
|
"jest": {
|
||||||
|
"moduleFileExtensions": [
|
||||||
|
"js",
|
||||||
|
"json",
|
||||||
|
"ts"
|
||||||
|
],
|
||||||
|
"rootDir": "src",
|
||||||
|
"testRegex": ".*\\.spec\\.ts$",
|
||||||
|
"transform": {
|
||||||
|
"^.+\\.(t|j)s$": "ts-jest"
|
||||||
|
},
|
||||||
|
"collectCoverageFrom": [
|
||||||
|
"**/*.(t|j)s"
|
||||||
|
],
|
||||||
|
"coverageDirectory": "../coverage",
|
||||||
|
"testEnvironment": "node"
|
||||||
|
}
|
||||||
|
}
|
||||||
22
src/app.controller.spec.ts
Normal file
22
src/app.controller.spec.ts
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
import { Test, TestingModule } from '@nestjs/testing';
|
||||||
|
import { AppController } from './app.controller';
|
||||||
|
import { AppService } from './app.service';
|
||||||
|
|
||||||
|
describe('AppController', () => {
|
||||||
|
let appController: AppController;
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
const app: TestingModule = await Test.createTestingModule({
|
||||||
|
controllers: [AppController],
|
||||||
|
providers: [AppService],
|
||||||
|
}).compile();
|
||||||
|
|
||||||
|
appController = app.get<AppController>(AppController);
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('root', () => {
|
||||||
|
it('should return "Hello World!"', () => {
|
||||||
|
expect(appController.getHello()).toBe('Hello World!');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
12
src/app.controller.ts
Normal file
12
src/app.controller.ts
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
import { Controller, Get } from '@nestjs/common';
|
||||||
|
import { AppService } from './app.service';
|
||||||
|
|
||||||
|
@Controller()
|
||||||
|
export class AppController {
|
||||||
|
constructor(private readonly appService: AppService) {}
|
||||||
|
|
||||||
|
@Get()
|
||||||
|
getHello(): string {
|
||||||
|
return this.appService.getHello();
|
||||||
|
}
|
||||||
|
}
|
||||||
27
src/app.module.ts
Normal file
27
src/app.module.ts
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { ConfigModule } from '@nestjs/config';
|
||||||
|
import { ScheduleModule } from '@nestjs/schedule';
|
||||||
|
import { ServeStaticModule } from '@nestjs/serve-static';
|
||||||
|
import { join } from 'path';
|
||||||
|
import { DatabaseModule } from './database/database.module';
|
||||||
|
import { BidsModule } from './bids/bids.module';
|
||||||
|
import { KeywordsModule } from './keywords/keywords.module';
|
||||||
|
import { CrawlerModule } from './crawler/crawler.module';
|
||||||
|
import { TasksModule } from './schedule/schedule.module';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [
|
||||||
|
ConfigModule.forRoot({ isGlobal: true }),
|
||||||
|
ScheduleModule.forRoot(),
|
||||||
|
ServeStaticModule.forRoot({
|
||||||
|
rootPath: join(__dirname, '..', 'frontend', 'dist'),
|
||||||
|
exclude: ['/api*'],
|
||||||
|
}),
|
||||||
|
DatabaseModule,
|
||||||
|
BidsModule,
|
||||||
|
KeywordsModule,
|
||||||
|
CrawlerModule,
|
||||||
|
TasksModule,
|
||||||
|
],
|
||||||
|
})
|
||||||
|
export class AppModule {}
|
||||||
8
src/app.service.ts
Normal file
8
src/app.service.ts
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
import { Injectable } from '@nestjs/common';
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class AppService {
|
||||||
|
getHello(): string {
|
||||||
|
return 'Hello World!';
|
||||||
|
}
|
||||||
|
}
|
||||||
13
src/bids/bids.module.ts
Normal file
13
src/bids/bids.module.ts
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||||
|
import { BidItem } from './entities/bid-item.entity';
|
||||||
|
import { BidsService } from './services/bid.service';
|
||||||
|
import { BidsController } from './controllers/bid.controller';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [TypeOrmModule.forFeature([BidItem])],
|
||||||
|
providers: [BidsService],
|
||||||
|
controllers: [BidsController],
|
||||||
|
exports: [BidsService],
|
||||||
|
})
|
||||||
|
export class BidsModule {}
|
||||||
17
src/bids/controllers/bid.controller.ts
Normal file
17
src/bids/controllers/bid.controller.ts
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
import { Controller, Get, Query } from '@nestjs/common';
|
||||||
|
import { BidsService } from '../services/bid.service';
|
||||||
|
|
||||||
|
@Controller('api/bids')
|
||||||
|
export class BidsController {
|
||||||
|
constructor(private readonly bidsService: BidsService) {}
|
||||||
|
|
||||||
|
@Get()
|
||||||
|
findAll(@Query() query: any) {
|
||||||
|
return this.bidsService.findAll(query);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Get('high-priority')
|
||||||
|
getHighPriority() {
|
||||||
|
return this.bidsService.getHighPriorityCorrected();
|
||||||
|
}
|
||||||
|
}
|
||||||
28
src/bids/entities/bid-item.entity.ts
Normal file
28
src/bids/entities/bid-item.entity.ts
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn, UpdateDateColumn } from 'typeorm';
|
||||||
|
|
||||||
|
@Entity('bid_items')
|
||||||
|
export class BidItem {
|
||||||
|
@PrimaryGeneratedColumn('uuid')
|
||||||
|
id: string;
|
||||||
|
|
||||||
|
@Column()
|
||||||
|
title: string;
|
||||||
|
|
||||||
|
@Column()
|
||||||
|
url: string;
|
||||||
|
|
||||||
|
@Column({ type: 'datetime' })
|
||||||
|
publishDate: Date;
|
||||||
|
|
||||||
|
@Column()
|
||||||
|
source: string;
|
||||||
|
|
||||||
|
@Column({ default: false })
|
||||||
|
isRead: boolean;
|
||||||
|
|
||||||
|
@CreateDateColumn()
|
||||||
|
createdAt: Date;
|
||||||
|
|
||||||
|
@UpdateDateColumn()
|
||||||
|
updatedAt: Date;
|
||||||
|
}
|
||||||
68
src/bids/services/bid.service.ts
Normal file
68
src/bids/services/bid.service.ts
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
import { Injectable } from '@nestjs/common';
|
||||||
|
import { InjectRepository } from '@nestjs/typeorm';
|
||||||
|
import { Repository, LessThan } from 'typeorm';
|
||||||
|
import { BidItem } from '../entities/bid-item.entity';
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class BidsService {
|
||||||
|
constructor(
|
||||||
|
@InjectRepository(BidItem)
|
||||||
|
private bidRepository: Repository<BidItem>,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async findAll(query?: any) {
|
||||||
|
const { page = 1, limit = 10, source, keyword } = query || {};
|
||||||
|
const qb = this.bidRepository.createQueryBuilder('bid');
|
||||||
|
|
||||||
|
if (source) {
|
||||||
|
qb.andWhere('bid.source = :source', { source });
|
||||||
|
}
|
||||||
|
|
||||||
|
if (keyword) {
|
||||||
|
qb.andWhere('bid.title LIKE :keyword', { keyword: `%${keyword}%` });
|
||||||
|
}
|
||||||
|
|
||||||
|
qb.orderBy('bid.publishDate', 'DESC')
|
||||||
|
.skip((page - 1) * limit)
|
||||||
|
.take(limit);
|
||||||
|
|
||||||
|
const [items, total] = await qb.getManyAndCount();
|
||||||
|
return { items, total };
|
||||||
|
}
|
||||||
|
|
||||||
|
getHighPriority() {
|
||||||
|
return this.bidRepository.find({
|
||||||
|
where: { priority: LessThan(0) }, // This is just a placeholder logic, priority should be > 0
|
||||||
|
order: { priority: 'DESC', publishDate: 'DESC' },
|
||||||
|
take: 10,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
// Update logic for priority
|
||||||
|
async getHighPriorityCorrected() {
|
||||||
|
return this.bidRepository.createQueryBuilder('bid')
|
||||||
|
.where('bid.priority > 0')
|
||||||
|
.orderBy('bid.priority', 'DESC')
|
||||||
|
.addOrderBy('bid.publishDate', 'DESC')
|
||||||
|
.limit(10)
|
||||||
|
.getMany();
|
||||||
|
}
|
||||||
|
|
||||||
|
async createOrUpdate(data: Partial<BidItem>) {
|
||||||
|
// Use URL or a hash of URL to check for duplicates
|
||||||
|
let item = await this.bidRepository.findOne({ where: { url: data.url } });
|
||||||
|
if (item) {
|
||||||
|
Object.assign(item, data);
|
||||||
|
return this.bidRepository.save(item);
|
||||||
|
}
|
||||||
|
return this.bidRepository.save(data);
|
||||||
|
}
|
||||||
|
|
||||||
|
async cleanOldData() {
|
||||||
|
const thirtyDaysAgo = new Date();
|
||||||
|
thirtyDaysAgo.setDate(thirtyDaysAgo.getDate() - 30);
|
||||||
|
return this.bidRepository.delete({
|
||||||
|
createdAt: LessThan(thirtyDaysAgo),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
21
src/crawler/crawler.controller.ts
Normal file
21
src/crawler/crawler.controller.ts
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
import { Controller, Post } from '@nestjs/common';
|
||||||
|
import { BidCrawlerService } from './services/bid-crawler.service';
|
||||||
|
|
||||||
|
@Controller('api/crawler')
|
||||||
|
export class CrawlerController {
|
||||||
|
constructor(private readonly crawlerService: BidCrawlerService) {}
|
||||||
|
|
||||||
|
@Post('run')
|
||||||
|
async runCrawl() {
|
||||||
|
// We don't await this because we want it to run in the background
|
||||||
|
// and return immediately, or we can await if we want the user to wait.
|
||||||
|
// Given the requirement "Immediate Crawl", usually implies triggering it.
|
||||||
|
// However, for a better UI experience, we might want to wait or just trigger.
|
||||||
|
// Let's await it so the user knows when it's done (or failed),
|
||||||
|
// assuming it doesn't take too long for the mock.
|
||||||
|
// Real crawling might take long, so background is better.
|
||||||
|
// For this prototype, I'll await it to show completion.
|
||||||
|
await this.crawlerService.crawlAll();
|
||||||
|
return { message: 'Crawl completed successfully' };
|
||||||
|
}
|
||||||
|
}
|
||||||
12
src/crawler/crawler.module.ts
Normal file
12
src/crawler/crawler.module.ts
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { BidCrawlerService } from './services/bid-crawler.service';
|
||||||
|
import { CrawlerController } from './crawler.controller';
|
||||||
|
import { BidsModule } from '../bids/bids.module';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [BidsModule],
|
||||||
|
controllers: [CrawlerController],
|
||||||
|
providers: [BidCrawlerService],
|
||||||
|
exports: [BidCrawlerService],
|
||||||
|
})
|
||||||
|
export class CrawlerModule {}
|
||||||
46
src/crawler/services/bid-crawler.service.ts
Normal file
46
src/crawler/services/bid-crawler.service.ts
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
import { Injectable, Logger } from '@nestjs/common';
|
||||||
|
import * as puppeteer from 'puppeteer';
|
||||||
|
import { BidsService } from '../../bids/services/bid.service';
|
||||||
|
import { ChdtpCrawler } from './chdtp_target';
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class BidCrawlerService {
|
||||||
|
private readonly logger = new Logger(BidCrawlerService.name);
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private bidsService: BidsService,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async crawlAll() {
|
||||||
|
this.logger.log('Starting crawl task with Puppeteer...');
|
||||||
|
|
||||||
|
const browser = await puppeteer.launch({
|
||||||
|
headless: true,
|
||||||
|
args: ['--no-sandbox', '--disable-setuid-sandbox'],
|
||||||
|
});
|
||||||
|
|
||||||
|
try {
|
||||||
|
// Currently only supports ChdtpCrawler, but can be extended to a list of crawlers
|
||||||
|
const crawler = ChdtpCrawler;
|
||||||
|
this.logger.log(`Crawling: ${crawler.name}`);
|
||||||
|
|
||||||
|
const results = await crawler.crawl(browser);
|
||||||
|
this.logger.log(`Extracted ${results.length} items from ${crawler.name}`);
|
||||||
|
|
||||||
|
for (const item of results) {
|
||||||
|
await this.bidsService.createOrUpdate({
|
||||||
|
title,
|
||||||
|
url: itemUrl,
|
||||||
|
publishDate,
|
||||||
|
source: type || 'Unknown',
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
} catch (error) {
|
||||||
|
this.logger.error(`Crawl task failed: ${error.message}`);
|
||||||
|
} finally {
|
||||||
|
await browser.close();
|
||||||
|
this.logger.log('Crawl task finished.');
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
51
src/crawler/services/chdtp_target.spec.ts
Normal file
51
src/crawler/services/chdtp_target.spec.ts
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
import { ChdtpCrawler } from './chdtp_target';
|
||||||
|
import * as puppeteer from 'puppeteer';
|
||||||
|
|
||||||
|
// Increase timeout to 60 seconds for network operations
|
||||||
|
jest.setTimeout(60000);
|
||||||
|
|
||||||
|
describe('ChdtpCrawler Real Site Test', () => {
|
||||||
|
let browser: puppeteer.Browser;
|
||||||
|
|
||||||
|
beforeAll(async () => {
|
||||||
|
browser = await puppeteer.launch({
|
||||||
|
headless: true, // Change to false to see the browser UI
|
||||||
|
args: ['--no-sandbox', '--disable-setuid-sandbox'],
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
afterAll(async () => {
|
||||||
|
if (browser) {
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should visit the website and list all found bid information', async () => {
|
||||||
|
console.log(`\nStarting crawl for: ${ChdtpCrawler.name}`);
|
||||||
|
console.log(`Target URL: ${ChdtpCrawler.url}`);
|
||||||
|
|
||||||
|
const results = await ChdtpCrawler.crawl(browser);
|
||||||
|
|
||||||
|
console.log(`\nSuccessfully found ${results.length} items:\n`);
|
||||||
|
console.log('----------------------------------------');
|
||||||
|
results.forEach((item, index) => {
|
||||||
|
console.log(`${index + 1}. [${item.publishDate.toLocaleDateString()}] ${item.title}`);
|
||||||
|
console.log(` Link: ${item.url}`);
|
||||||
|
console.log('----------------------------------------');
|
||||||
|
});
|
||||||
|
|
||||||
|
// Basic assertions to ensure the crawler is working
|
||||||
|
expect(results).toBeDefined();
|
||||||
|
expect(Array.isArray(results)).toBeTruthy();
|
||||||
|
// Warn but don't fail if site returns 0 items (could be empty or changed structure)
|
||||||
|
if (results.length === 0) {
|
||||||
|
console.warn('Warning: No items found. Check if the website structure has changed or if the list is currently empty.');
|
||||||
|
} else {
|
||||||
|
// Check data integrity of the first item
|
||||||
|
const firstItem = results[0];
|
||||||
|
expect(firstItem.title).toBeTruthy();
|
||||||
|
expect(firstItem.url).toMatch(/^https?:\/\//);
|
||||||
|
expect(firstItem.publishDate).toBeInstanceOf(Date);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
110
src/crawler/services/chdtp_target.ts
Normal file
110
src/crawler/services/chdtp_target.ts
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
import * as puppeteer from 'puppeteer';
|
||||||
|
import { Logger } from '@nestjs/common';
|
||||||
|
|
||||||
|
export interface ChdtpResult {
|
||||||
|
title: string;
|
||||||
|
publishDate: Date;
|
||||||
|
url: string; // Necessary for system uniqueness
|
||||||
|
}
|
||||||
|
|
||||||
|
export const ChdtpCrawler = {
|
||||||
|
name: '中国华能集团',
|
||||||
|
url: 'https://www.chdtp.com/webs/queryWebZbgg.action?zbggType=1',
|
||||||
|
baseUrl: 'https://www.chdtp.com/webs/',
|
||||||
|
|
||||||
|
async crawl(browser: puppeteer.Browser): Promise<ChdtpResult[]> {
|
||||||
|
const logger = new Logger('ChdtpCrawler');
|
||||||
|
const page = await browser.newPage();
|
||||||
|
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36');
|
||||||
|
|
||||||
|
const allResults: ChdtpResult[] = [];
|
||||||
|
let currentPage = 1;
|
||||||
|
const maxPages = 5; // Safety limit to prevent infinite loops during testing
|
||||||
|
|
||||||
|
try {
|
||||||
|
logger.log(`Navigating to ${this.url}...`);
|
||||||
|
await page.goto(this.url, { waitUntil: 'networkidle2', timeout: 60000 });
|
||||||
|
|
||||||
|
while (currentPage <= maxPages) {
|
||||||
|
const content = await page.content();
|
||||||
|
const pageResults = this.extract(content);
|
||||||
|
|
||||||
|
if (pageResults.length === 0) {
|
||||||
|
logger.warn(`No results found on page ${currentPage}, stopping.`);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
allResults.push(...pageResults);
|
||||||
|
logger.log(`Extracted ${pageResults.length} items from page ${currentPage}`);
|
||||||
|
|
||||||
|
// Find the "Next Page" button
|
||||||
|
// Using partial match for src to be robust against path variations
|
||||||
|
const nextButtonSelector = 'input[type="image"][src*="page-next.png"]';
|
||||||
|
const nextButton = await page.$(nextButtonSelector);
|
||||||
|
|
||||||
|
if (!nextButton) {
|
||||||
|
logger.log('Next page button not found. Reached end of list.');
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Optional: Check if the button is disabled (though image inputs usually aren't "disabled" in the same way)
|
||||||
|
// For this specific site, we'll try to click.
|
||||||
|
|
||||||
|
logger.log(`Navigating to page ${currentPage + 1}...`);
|
||||||
|
|
||||||
|
try {
|
||||||
|
await Promise.all([
|
||||||
|
page.waitForNavigation({ waitUntil: 'networkidle2', timeout: 60000 }),
|
||||||
|
nextButton.click(),
|
||||||
|
]);
|
||||||
|
} catch (navError) {
|
||||||
|
logger.error(`Navigation to page ${currentPage + 1} failed: ${navError.message}`);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
|
currentPage++;
|
||||||
|
|
||||||
|
// Random delay between pages
|
||||||
|
const delay = Math.floor(Math.random() * (3000 - 1000 + 1)) + 1000;
|
||||||
|
await new Promise(resolve => setTimeout(resolve, delay));
|
||||||
|
}
|
||||||
|
|
||||||
|
return allResults;
|
||||||
|
|
||||||
|
} catch (error) {
|
||||||
|
logger.error(`Failed to crawl ${this.name}: ${error.message}`);
|
||||||
|
return allResults; // Return what we have so far
|
||||||
|
} finally {
|
||||||
|
await page.close();
|
||||||
|
}
|
||||||
|
},
|
||||||
|
|
||||||
|
extract(html: string): ChdtpResult[] {
|
||||||
|
const results: ChdtpResult[] = [];
|
||||||
|
/**
|
||||||
|
* Regex groups for chdtp.com:
|
||||||
|
* 1: Status
|
||||||
|
* 2: URL suffix
|
||||||
|
* 3: Title
|
||||||
|
* 4: Business Type
|
||||||
|
* 5: Date
|
||||||
|
*/
|
||||||
|
const regex = /<tr[^>]*>\s*<td class="td_1">.*?<span[^>]*>\s*(.*?)\s*<\/span>.*?<\/td>\s*<td class="td_2">\s*<a[^>]*href="javascript:toGetContent\('(.*?)'\)" title="(.*?)">.*?<\/a><\/td>\s*<td class="td_3">\s*<a[^>]*>\s*(.*?)\s*<\/a>\s*<\/td>\s*<td class="td_4"><span>\[(.*?)\]<\/span><\/td>/gs;
|
||||||
|
|
||||||
|
let match;
|
||||||
|
while ((match = regex.exec(html)) !== null) {
|
||||||
|
const urlSuffix = match[2]?.trim();
|
||||||
|
const title = match[3]?.trim();
|
||||||
|
const dateStr = match[5]?.trim();
|
||||||
|
|
||||||
|
if (title && urlSuffix) {
|
||||||
|
results.push({
|
||||||
|
title,
|
||||||
|
publishDate: dateStr ? new Date(dateStr) : new Date(),
|
||||||
|
url: this.baseUrl + urlSuffix
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
};
|
||||||
25
src/database/database.module.ts
Normal file
25
src/database/database.module.ts
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||||
|
import { ConfigModule, ConfigService } from '@nestjs/config';
|
||||||
|
import { BidItem } from '../bids/entities/bid-item.entity';
|
||||||
|
import { Keyword } from '../keywords/keyword.entity';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [
|
||||||
|
TypeOrmModule.forRootAsync({
|
||||||
|
imports: [ConfigModule],
|
||||||
|
inject: [ConfigService],
|
||||||
|
useFactory: (configService: ConfigService) => ({
|
||||||
|
type: configService.get<any>('DATABASE_TYPE', 'mariadb'),
|
||||||
|
host: configService.get<string>('DATABASE_HOST', 'localhost'),
|
||||||
|
port: configService.get<number>('DATABASE_PORT', 3306),
|
||||||
|
username: configService.get<string>('DATABASE_USERNAME', 'root'),
|
||||||
|
password: configService.get<string>('DATABASE_PASSWORD', 'root'),
|
||||||
|
database: configService.get<string>('DATABASE_NAME', 'bidding'),
|
||||||
|
entities: [BidItem, Keyword],
|
||||||
|
synchronize: configService.get<boolean>('DATABASE_SYNCHRONIZE', true),
|
||||||
|
}),
|
||||||
|
}),
|
||||||
|
],
|
||||||
|
})
|
||||||
|
export class DatabaseModule {}
|
||||||
19
src/keywords/keyword.entity.ts
Normal file
19
src/keywords/keyword.entity.ts
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
import { Entity, PrimaryGeneratedColumn, Column, CreateDateColumn, UpdateDateColumn } from 'typeorm';
|
||||||
|
|
||||||
|
@Entity('keywords')
|
||||||
|
export class Keyword {
|
||||||
|
@PrimaryGeneratedColumn('uuid')
|
||||||
|
id: string;
|
||||||
|
|
||||||
|
@Column({ unique: true })
|
||||||
|
word: string;
|
||||||
|
|
||||||
|
@Column({ default: 1 })
|
||||||
|
weight: number; // 1-5级
|
||||||
|
|
||||||
|
@CreateDateColumn()
|
||||||
|
createdAt: Date;
|
||||||
|
|
||||||
|
@UpdateDateColumn()
|
||||||
|
updatedAt: Date;
|
||||||
|
}
|
||||||
22
src/keywords/keywords.controller.ts
Normal file
22
src/keywords/keywords.controller.ts
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
import { Controller, Get, Post, Body, Delete, Param } from '@nestjs/common';
|
||||||
|
import { KeywordsService } from './keywords.service';
|
||||||
|
|
||||||
|
@Controller('api/keywords')
|
||||||
|
export class KeywordsController {
|
||||||
|
constructor(private readonly keywordsService: KeywordsService) {}
|
||||||
|
|
||||||
|
@Get()
|
||||||
|
findAll() {
|
||||||
|
return this.keywordsService.findAll();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Post()
|
||||||
|
create(@Body('word') word: string, @Body('weight') weight: number) {
|
||||||
|
return this.keywordsService.create(word, weight);
|
||||||
|
}
|
||||||
|
|
||||||
|
@Delete(':id')
|
||||||
|
remove(@Param('id') id: string) {
|
||||||
|
return this.keywordsService.remove(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
13
src/keywords/keywords.module.ts
Normal file
13
src/keywords/keywords.module.ts
Normal file
@@ -0,0 +1,13 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { TypeOrmModule } from '@nestjs/typeorm';
|
||||||
|
import { Keyword } from './keyword.entity';
|
||||||
|
import { KeywordsService } from './keywords.service';
|
||||||
|
import { KeywordsController } from './keywords.controller';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [TypeOrmModule.forFeature([Keyword])],
|
||||||
|
providers: [KeywordsService],
|
||||||
|
controllers: [KeywordsController],
|
||||||
|
exports: [KeywordsService],
|
||||||
|
})
|
||||||
|
export class KeywordsModule {}
|
||||||
35
src/keywords/keywords.service.ts
Normal file
35
src/keywords/keywords.service.ts
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
import { Injectable, OnModuleInit } from '@nestjs/common';
|
||||||
|
import { InjectRepository } from '@nestjs/typeorm';
|
||||||
|
import { Repository } from 'typeorm';
|
||||||
|
import { Keyword } from './keyword.entity';
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class KeywordsService implements OnModuleInit {
|
||||||
|
constructor(
|
||||||
|
@InjectRepository(Keyword)
|
||||||
|
private keywordRepository: Repository<Keyword>,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
async onModuleInit() {
|
||||||
|
// 初始预设关键词
|
||||||
|
const defaultKeywords = ["山东", "海", "建设", "工程", "采购"];
|
||||||
|
for (const word of defaultKeywords) {
|
||||||
|
const exists = await this.keywordRepository.findOne({ where: { word } });
|
||||||
|
if (!exists) {
|
||||||
|
await this.keywordRepository.save({ word, weight: 1 });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
findAll() {
|
||||||
|
return this.keywordRepository.find();
|
||||||
|
}
|
||||||
|
|
||||||
|
create(word: string, weight: number = 1) {
|
||||||
|
return this.keywordRepository.save({ word, weight });
|
||||||
|
}
|
||||||
|
|
||||||
|
remove(id: string) {
|
||||||
|
return this.keywordRepository.delete(id);
|
||||||
|
}
|
||||||
|
}
|
||||||
8
src/main.ts
Normal file
8
src/main.ts
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
import { NestFactory } from '@nestjs/core';
|
||||||
|
import { AppModule } from './app.module';
|
||||||
|
|
||||||
|
async function bootstrap() {
|
||||||
|
const app = await NestFactory.create(AppModule);
|
||||||
|
await app.listen(process.env.PORT ?? 3000);
|
||||||
|
}
|
||||||
|
bootstrap();
|
||||||
10
src/schedule/schedule.module.ts
Normal file
10
src/schedule/schedule.module.ts
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
import { Module } from '@nestjs/common';
|
||||||
|
import { BidCrawlTask } from './tasks/bid-crawl.task';
|
||||||
|
import { CrawlerModule } from '../crawler/crawler.module';
|
||||||
|
import { BidsModule } from '../bids/bids.module';
|
||||||
|
|
||||||
|
@Module({
|
||||||
|
imports: [CrawlerModule, BidsModule],
|
||||||
|
providers: [BidCrawlTask],
|
||||||
|
})
|
||||||
|
export class TasksModule {}
|
||||||
26
src/schedule/tasks/bid-crawl.task.ts
Normal file
26
src/schedule/tasks/bid-crawl.task.ts
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
import { Injectable, Logger } from '@nestjs/common';
|
||||||
|
import { Cron, CronExpression } from '@nestjs/schedule';
|
||||||
|
import { BidCrawlerService } from '../../crawler/services/bid-crawler.service';
|
||||||
|
import { BidsService } from '../../bids/services/bid.service';
|
||||||
|
|
||||||
|
@Injectable()
|
||||||
|
export class BidCrawlTask {
|
||||||
|
private readonly logger = new Logger(BidCrawlTask.name);
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private crawlerService: BidCrawlerService,
|
||||||
|
private bidsService: BidsService,
|
||||||
|
) {}
|
||||||
|
|
||||||
|
@Cron(CronExpression.EVERY_30_MINUTES)
|
||||||
|
async handleCron() {
|
||||||
|
this.logger.debug('Scheduled crawl task started');
|
||||||
|
await this.crawlerService.crawlAll();
|
||||||
|
}
|
||||||
|
|
||||||
|
@Cron(CronExpression.EVERY_DAY_AT_MIDNIGHT)
|
||||||
|
async handleCleanup() {
|
||||||
|
this.logger.debug('Scheduled cleanup task started');
|
||||||
|
await this.bidsService.cleanOldData();
|
||||||
|
}
|
||||||
|
}
|
||||||
25
test/app.e2e-spec.ts
Normal file
25
test/app.e2e-spec.ts
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
import { Test, TestingModule } from '@nestjs/testing';
|
||||||
|
import { INestApplication } from '@nestjs/common';
|
||||||
|
import request from 'supertest';
|
||||||
|
import { App } from 'supertest/types';
|
||||||
|
import { AppModule } from './../src/app.module';
|
||||||
|
|
||||||
|
describe('AppController (e2e)', () => {
|
||||||
|
let app: INestApplication<App>;
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
const moduleFixture: TestingModule = await Test.createTestingModule({
|
||||||
|
imports: [AppModule],
|
||||||
|
}).compile();
|
||||||
|
|
||||||
|
app = moduleFixture.createNestApplication();
|
||||||
|
await app.init();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('/ (GET)', () => {
|
||||||
|
return request(app.getHttpServer())
|
||||||
|
.get('/')
|
||||||
|
.expect(200)
|
||||||
|
.expect('Hello World!');
|
||||||
|
});
|
||||||
|
});
|
||||||
4
tsconfig.build.json
Normal file
4
tsconfig.build.json
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"extends": "./tsconfig.json",
|
||||||
|
"exclude": ["node_modules", "test", "dist", "**/*spec.ts"]
|
||||||
|
}
|
||||||
26
tsconfig.json
Normal file
26
tsconfig.json
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"module": "nodenext",
|
||||||
|
"moduleResolution": "nodenext",
|
||||||
|
"resolvePackageJsonExports": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"isolatedModules": true,
|
||||||
|
"declaration": true,
|
||||||
|
"removeComments": true,
|
||||||
|
"emitDecoratorMetadata": true,
|
||||||
|
"experimentalDecorators": true,
|
||||||
|
"allowSyntheticDefaultImports": true,
|
||||||
|
"target": "ES2023",
|
||||||
|
"sourceMap": true,
|
||||||
|
"outDir": "./dist",
|
||||||
|
"baseUrl": "./",
|
||||||
|
"incremental": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"strictNullChecks": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"noImplicitAny": false,
|
||||||
|
"strictBindCallApply": false,
|
||||||
|
"noFallthroughCasesInSwitch": false
|
||||||
|
},
|
||||||
|
"include": ["src", "test"]
|
||||||
|
}
|
||||||
158
开发需求.md
Normal file
158
开发需求.md
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
|
||||||
|
|
||||||
|
# 投标信息智能监控系统 - 需求说明书
|
||||||
|
|
||||||
|
## 项目概述
|
||||||
|
开发一个基于TypeScript的Web应用，用于自动爬取商务投标平台的最新信息，将符合条件的投标项目突出显示，为用户提供精准的投标信息监控服务。
|
||||||
|
|
||||||
|
## 技术栈要求
|
||||||
|
- **主语言**: TypeScript (4.0+)
|
||||||
|
- **后端框架**: **NestJS** (推荐理由：完整的TypeScript支持、模块化架构、内置依赖注入、与数据库ORM良好集成)
|
||||||
|
- **前端框架**: React (集成在NestJS中，使用SSR或API模式)
|
||||||
|
- **数据库**: MySQL/PostgreSQL (通过TypeORM或Prisma操作)
|
||||||
|
- **爬虫库**: 使用axios
|
||||||
|
- **任务调度**: @nestjs/schedule
|
||||||
|
- **UI组件库**: Ant Design (提供企业级UI组件，支持TypeScript)
|
||||||
|
|
||||||
|
## 核心功能需求
|
||||||
|
|
||||||
|
### 1. 智能爬虫模块
|
||||||
|
- 定时访问指定商务投标平台(初始配置为：至少3个主流招标网站)
|
||||||
|
- 每30分钟执行一次爬取任务
|
||||||
|
- 识别并提取以下信息：
|
||||||
|
- 投标项目标题
|
||||||
|
- 详细页面URL
|
||||||
|
- 发布时间
|
||||||
|
- 招标单位
|
||||||
|
- 截止日期
|
||||||
|
- 实现智能防封策略：
|
||||||
|
- 随机请求间隔(3-8秒)
|
||||||
|
- 轮换User-Agent
|
||||||
|
- 异常检测与自动重试机制
|
||||||
|
|
||||||
|
### 2. 数据处理与存储
|
||||||
|
- 投标信息数据模型：
|
||||||
|
```typescript
|
||||||
|
interface BidItem {
|
||||||
|
id: string;
|
||||||
|
title: string;
|
||||||
|
url: string;
|
||||||
|
publishDate: Date;
|
||||||
|
source: string;
|
||||||
|
keywordsMatched: string[]; // 匹配的关键词
|
||||||
|
priority: number; // 优先级，基于关键词匹配
|
||||||
|
createdAt: Date;
|
||||||
|
updatedAt: Date;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- 增量存储逻辑：
|
||||||
|
- 通过URL哈希值判断是否为新数据
|
||||||
|
- 仅存储当天和最近7天的历史数据
|
||||||
|
- 每日自动清理30天前的数据
|
||||||
|
|
||||||
|
### 3. 关键词智能监控
|
||||||
|
- 预设关键词：["山东", "海", "建设", "工程", "采购"]
|
||||||
|
- 支持用户自定义关键词：
|
||||||
|
- 通过Web界面添加/删除关键词
|
||||||
|
- 可设置关键词权重(1-5级)
|
||||||
|
- 匹配逻辑：
|
||||||
|
- 标题完全匹配和部分匹配
|
||||||
|
- 多关键词叠加权重
|
||||||
|
- 支持正则表达式高级匹配
|
||||||
|
|
||||||
|
### 4. Web展示界面
|
||||||
|
- 仪表盘设计：
|
||||||
|
- 顶部：高优先级投标信息(匹配自定义关键词)
|
||||||
|
- 中部：今日新增投标列表(按时间倒序)
|
||||||
|
- 底部：历史投标信息
|
||||||
|
- 交互功能：
|
||||||
|
- 关键词管理面板
|
||||||
|
- 按日期/来源/关键词筛选
|
||||||
|
- 信息标记已读/未读状态
|
||||||
|
- 邮件订阅提醒(基础框架)
|
||||||
|
- 响应式设计：适配桌面和移动设备
|
||||||
|
|
||||||
|
## 系统架构
|
||||||
|
|
||||||
|
```
|
||||||
|
src/
|
||||||
|
├── common/
|
||||||
|
│ ├── decorators/ # 自定义装饰器
|
||||||
|
│ ├── filters/ # 异常过滤器
|
||||||
|
│ └── utils/ # 工具函数
|
||||||
|
├── crawler/ # 爬虫模块
|
||||||
|
│ ├── services/
|
||||||
|
│ │ ├── bid-crawler.service.ts # 爬虫核心服务
|
||||||
|
│ │ └── anti-ban.strategy.ts # 防封策略
|
||||||
|
│ └── crawler.module.ts
|
||||||
|
├── database/ # 数据库模块
|
||||||
|
│ ├── entities/ # 实体定义
|
||||||
|
│ ├── migrations/ # 数据库迁移
|
||||||
|
│ └── database.module.ts
|
||||||
|
├── bids/ # 投标业务模块
|
||||||
|
│ ├── entities/
|
||||||
|
│ ├── dto/ # 数据传输对象
|
||||||
|
│ ├── controllers/
|
||||||
|
│ ├── services/
|
||||||
|
│ │ ├── bid.service.ts
|
||||||
|
│ │ └── keyword-analyzer.service.ts # 关键词分析
|
||||||
|
│ └── bids.module.ts
|
||||||
|
├── keywords/ # 关键词管理模块
|
||||||
|
├── schedule/ # 定时任务
|
||||||
|
│ └── tasks/
|
||||||
|
│ └── bid-crawl.task.ts # 定时爬取任务
|
||||||
|
├── app.controller.ts
|
||||||
|
├── app.module.ts
|
||||||
|
└── main.ts # 应用入口
|
||||||
|
```
|
||||||
|
|
||||||
|
## API接口设计
|
||||||
|
|
||||||
|
### 后端API
|
||||||
|
- `GET /api/bids` - 获取投标列表(支持分页、筛选)
|
||||||
|
- `GET /api/bids/high-priority` - 获取高优先级投标
|
||||||
|
- `GET /api/keywords` - 获取所有关键词
|
||||||
|
- `POST /api/keywords` - 添加新关键词
|
||||||
|
- `DELETE /api/keywords/:id` - 删除关键词
|
||||||
|
|
||||||
|
### 前端路由
|
||||||
|
- `/` - 仪表盘(默认页面)
|
||||||
|
- `/bids` - 全部投标信息
|
||||||
|
- `/keywords` - 关键词管理
|
||||||
|
- `/settings` - 系统设置
|
||||||
|
|
||||||
|
## 非功能性需求
|
||||||
|
|
||||||
|
1. **性能要求**：
|
||||||
|
- 首屏加载时间 ≤ 2秒
|
||||||
|
- 爬虫任务执行时间 ≤ 5分钟/站点
|
||||||
|
- 支持至少1000条投标记录的快速检索
|
||||||
|
|
||||||
|
2. **安全要求**：
|
||||||
|
- 防XSS攻击(对爬取内容进行消毒)
|
||||||
|
- API访问权限控制
|
||||||
|
- 环境变量管理敏感信息
|
||||||
|
|
||||||
|
3. **可维护性**：
|
||||||
|
- 100% TypeScript类型覆盖
|
||||||
|
- 模块化设计，低耦合
|
||||||
|
- 完整的Jest单元测试(覆盖率≥80%)
|
||||||
|
|
||||||
|
4. **部署要求**：
|
||||||
|
- 环境配置分离(开发/测试/生产)
|
||||||
|
- 日志集中管理
|
||||||
|
|
||||||
|
## 配置项
|
||||||
|
- 爬取目标网站列表(可配置)
|
||||||
|
- 定时任务执行频率(默认30分钟)
|
||||||
|
- 数据保留策略(默认30天)
|
||||||
|
- 关键词匹配规则(默认包含匹配)
|
||||||
|
|
||||||
|
## 交付物
|
||||||
|
1. 完整的TypeScript源代码
|
||||||
|
|
||||||
|
|
||||||
|
## 备注
|
||||||
|
- 数据库连接使用连接池
|
||||||
|
|
||||||
|
请基于此需求说明书实现该系统，确保代码质量符合企业级应用标准。
|
||||||
Reference in New Issue
Block a user