Software
Python research libraries, constrained optimization benchmarks, and network orchestration prototypes.
I build software for studying and solving sequential decision problems: reusable numerical tools, comparable experiments, and interfaces that check constraints before exporting a decision. The projects below link directly to public code, documentation, and related research.
Featured software
Constrained optimization
service-placement-benchmarks
Research benchmark
Compare edge-network service placement and routing strategies under compute, link-capacity, and latency constraints.
- Problem
- Edge networks must decide where to host services and how to route demand without violating resource or latency budgets.
- Approach
- A shared Python interface compares mathematical optimization, heuristics, evolutionary search, and reinforcement learning on the same network scenarios.
- Implemented
-
- Configurable synthetic tree and star networks; a common validator checks placement, routing, flow, capacity, resources, and latency.
- CSV exports record objectives, runtimes, feasibility, seeds, and errors; invalid candidates receive no objective score.
- Engineering
- Separate domain, solver, and reporting modules; optional solver dependencies; unit, integration, and regression tests; CI on Windows and Linux; CLI and Docker runner.
Scope. An evaluation toolkit. Connecting decisions to a live network requires a system-specific adapter; mathematical backends require Gurobi and a license.
Sequential decision-making
mdpforge
Python research library
Build finite Markov decision processes and compare solvers at a shared target accuracy with reproducible experiments.
- Problem
- Solver comparisons need consistent models, accuracy checks, and experiment records to make runtime measurements interpretable.
- Approach
- A modular catalogue of finite discounted MDPs and solvers, with registration of custom transition/reward matrices and solve functions.
- Implemented
-
- Value iteration, policy iteration, modified policy iteration, and aggregation methods; sparse transition matrices and optional solver backends.
- Independent Bellman-residual accuracy checks, repeated trials, scaling experiments, result summaries, and plots.
- Engineering
- CSV results with JSON experiment manifests record model fingerprints, seeds, solver settings, versions, and machine details. Tests cover numerical precision, validation, solver selection, and result handling.
Scope. Research tooling for finite discounted MDPs. Empirical runtimes depend on the model, accuracy target, backend, and hardware.
Software integration
service-placement-deployable
Research / engineering prototype
Train PPO policies for edge-network service placement, routing, and bandwidth allocation, then validate and export decisions through CLI and HTTP interfaces.
- Problem
- A placement strategy needs explicit input contracts, feasibility checks, and machine-readable decisions before it can be integrated into an orchestrator.
- Approach
- A saved PPO policy guides a sequence of placement, directed-routing, and integer-bandwidth edits under CPU, link-capacity, and load-dependent latency constraints.
- Implemented
-
- Greedy and routing-lookahead baselines, optional learned-policy inference, and independent feasibility validation.
- Versioned JSON input/output, a decision CLI, and a small HTTP adapter with input validation and algorithm diagnostics.
- Engineering
- Deterministic synthetic scenarios, unit and integration tests, CI with an RL smoke check, and Docker packaging. Automatic candidate selection records failures and can fall back to a feasible baseline.
Scope. A synthetic-data demonstrator. It does not apply network changes, bundle a trained policy, or establish learned-policy superiority. Live orchestration requires an adapter; the HTTP demo is not a production service.
Research code & environments
State abstraction
State-space disaggregation
Publication research code
Progressive refinement of aggregated state spaces for infinite-horizon dynamic programming, accompanying the ICAPS 2024 paper.
- Problem
- Large MDPs make direct dynamic programming expensive; useful state distinctions must be recovered from coarse abstractions.
- Approach
- Aggregate Bellman updates alternate with partition refinement based on value or policy information.
- Implemented
-
- Models include Rooms, Garnet, racetracks, tandem queues, and weakly coupled MDPs.
- An experiment runner compares solvers and exports runtime measurements to spreadsheets.
- Engineering
- Separate model, solver, and partition utilities with a documented experiment entry point. This historical experiment repository has no automated test suite or CI workflow in the audited public tree.
Scope. Publication-specific research code; the full experiment campaign is documented as taking several days.
RL environments
nonogram-gymnasium
Experimental environment
A compact Gymnasium environment for learning to solve Nonogram puzzles from row and column clues.
- Problem
- Structured puzzles provide small, inspectable decision problems for exploring observations, rewards, and agent behavior.
- Approach
- A hidden binary grid, clue-based observations, and cell/value actions exposed through Gymnasium reset and step interfaces.
- Implemented
-
- Seeded puzzle generation, configurable board size and episode length, rendering, and explicit termination/truncation behavior.
- Random-agent and terminal-play examples, a notebook demo, and a DQN training example using an action wrapper.
- Engineering
- Environment and reward tests; linting, formatting, and typing checks in CI across Python 3.10–3.12.
Scope. A toy research environment with a simple reward function and no correction action for previously played cells.