Design a modular, scalable Node.js server architecture for ingesting Excel/CSV data, processing it with transaction-specific logic, storing it in MongoDB, and forwarding it to external APIs while ensuring idempotency and tracking processing time.
Design a modular, scalable Node.js server architecture for ingesting Excel/CSV data, processing it with transaction-specific logic, storing it in MongoDB, and forwarding it to external APIs while ensuring idempotency and tracking processing time.
Act as a Node.js Architect and Backend Developer. Design and implement a generic, modular, and scalable data porting server. The server must read data from Excel or CSV files, process it, save it to MongoDB, and forward it to external APIs.
transactionType and transactionNumber.yyyy-mm-dd Hh:Mm:Ss.├── config
│ ├── default.json
│ └── production.json
├── logs
├── src
│ ├── api
│ │ └── middleware # Express middleware
│ ├── controllers
│ ├── models
│ ├── services
│ │ ├── APIService.js
│ │ ├── CSVService.js
│ │ ├── ExcelService.js
│ │ ├── Logger.js
│ │ ├── MongoDBService.js
│ │ └── TransactionService.js
│ └── utils
│ ├── dateUtils.js
│ └── validationUtils.js
├── test
│ ├── integration
│ └── unit
├── scripts # Operational scripts, i.e., database migration
├── docs # Documentation
├── .env
├── .gitignore
├── package.json
└── server.js
server.js must utilize node-locksmith for process locking, express for the server, mongoose for database connection, and dynamic route loading. It must include detailed JSDoc comments and handle graceful shutdowns.