For ANT Common Data Engineering, I developed a Flask-based server application that executes Python scripts within isolated Docker containers.
The server receives tasks via a RESTful API endpoint, validates the incoming data, and spins up Docker containers to execute scripts in a secure, containerized environment.
The architecture incorporates Gunicorn for production-grade WSGI serving with configurable workers and threads for scalability. I implemented a robust task management system using SQLite to track active tasks and prevent duplicate execution, along with comprehensive error handling and logging capabilities.
The server supports configurable Python versions (3.8-3.12) and includes debug modes for development. The entire application is containerized with Docker Compose for easy deployment. All incoming scripts are wrapped with necessary imports and API connections, ensuring consistent execution environments across all tasks.
Technologies: PYTHON, FLASK, DOCKER, DEVOPS, SQLITE, GUNICORN, DOCKER-COMPOSE, POETRY, VSCODE, GIT
