feat: 全面升级系统日志和反爬虫功能
- 新增专业日志系统:集成 Winston 日志框架,支持按天轮转和分级存储 - 增强反爬虫能力:集成 puppeteer-extra-plugin-stealth 插件,提升隐蔽性 - 新增独立爬虫脚本:可通过 npm run crawl 命令单独执行爬虫任务 - 优化前端日期筛选:添加日期范围选择器,支持3天/7天快速筛选 - 改进爬虫统计功能:详细记录每个平台的成功/失败情况和执行时间 - 移除默认关键词初始化:避免重复创建预设关键词 - 扩展环境配置:新增 LOG_LEVEL 日志级别配置选项 - 增强.gitignore:添加日志目录、构建产物等忽略规则 - 升级执行时间限制:将最大执行时间从1小时延长至3小时 - 完善错误处理:更好的异常捕获和日志记录机制
This commit is contained in:
@@ -17,7 +17,8 @@
|
||||
"test:watch": "jest --watch",
|
||||
"test:cov": "jest --coverage",
|
||||
"test:debug": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register node_modules/.bin/jest --runInBand",
|
||||
"test:e2e": "jest --config ./test/jest-e2e.json"
|
||||
"test:e2e": "jest --config ./test/jest-e2e.json",
|
||||
"crawl": "ts-node -r tsconfig-paths/register src/scripts/crawl.ts"
|
||||
},
|
||||
"dependencies": {
|
||||
"@nestjs/common": "^11.0.1",
|
||||
@@ -32,9 +33,13 @@
|
||||
"class-validator": "^0.14.3",
|
||||
"mysql2": "^3.16.0",
|
||||
"puppeteer": "^24.34.0",
|
||||
"puppeteer-extra": "^3.3.6",
|
||||
"puppeteer-extra-plugin-stealth": "^2.11.2",
|
||||
"reflect-metadata": "^0.2.2",
|
||||
"rxjs": "^7.8.1",
|
||||
"typeorm": "^0.3.28"
|
||||
"typeorm": "^0.3.28",
|
||||
"winston": "^3.19.0",
|
||||
"winston-daily-rotate-file": "^5.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/eslintrc": "^3.2.0",
|
||||
|
||||
Reference in New Issue
Block a user