SATELLITE
OCO-2
TLE-based LEO orbital trajectory.
REINFORCEMENT LEARNING · RESEARCH
LEO Satellite RL
Reinforcement Learning for
Ground Station Optimization
PERIOD
2024.11.18 — 2025.01.15
LAB
Distributed Platforms & Systems Laboratory
AUTHORSHIP
First Author
Python · PyTorch · OpenAI Gym · Skyfield · SGP4 · NumPy · Matplotlib
01 — OVERVIEW
RESEARCH QUESTION
Which ground station should a moving satellite choose at each moment?
Low Earth Orbit satellites continuously move relative to ground stations, which means visibility and communication conditions change over time.
Concentrating transmissions on particular ground stations can create queue overload, information imbalance, transmission delays, and unnecessary satellite energy consumption.
I reformulated ground-station selection as a sequential decision-making problem and designed a DQN-based reinforcement learning environment that considers dynamic visibility, data freshness, queue states, and satellite energy.
02 — ORBITAL SIMULATION
Building a dynamic environment from a real satellite orbit.
The simulation modeled NASA's OCO-2 satellite trajectory from TLE orbital information and used coordinates for 10 NASA Near Space Network ground stations.
GROUND STATIONS
10
NASA Near Space Network locations.
TIME STEP
1 min
The environment advances once per minute.
EPISODE
1,440
One full day represented by 1,440 steps.
OCO-2 TLE
Orbital Information
Skyfield / SGP4
Orbit Calculation
Satellite Position
Time-varying
Ground Stations
NASA NSN × 10
Visibility
Dynamic Link State
RL Environment
1-minute Step
03 — ENVIRONMENT DESIGN
MARKOV DECISION PROCESS
State → Action → Reward
I designed the environment so the agent could observe both communication conditions and resource constraints, choose a transmission action, and receive a reward based on the quality of that decision.
STATE
Dynamic Environment
Visibility · AoI · Energy · Queues · Distance
AGENT
DQN
Estimate action values
ACTION
Ground Station
Select one station or no transmission
REWARD
Transmission Quality
Freshness · Queue Balance · Energy
04 — STATE
Representing the satellite communication environment.
The observation space combined time-varying orbital conditions with communication and resource states so the agent could respond to a changing environment.
01
Visibility
Whether each ground station is currently visible from the satellite.
02
Visibility Duration
How long the current communication opportunity remains available.
03
Distance
Current distance between the satellite and each ground station.
04
Satellite Energy
Remaining satellite energy available for transmission.
05
Age of Information
Freshness of satellite and ground-station data.
06
Queue State
Satellite and ground-station queue conditions representing pending data load.
05 — ACTION & REWARD
DECISION DESIGN
Teaching the agent what a good transmission decision means.
ACTION
Select Ground Station
At each minute, the agent selects one ground station for transmission or chooses not to transmit.
CONSTRAINT
Dynamic Visibility
The valid set of ground stations changes continuously as the satellite moves along its orbit.
REWARD COMPONENTS
01
Age of Information
Data Freshness
Encourages transmission decisions that improve the freshness of ground-station data.
02
Ground Station Load
Queue Balance
Rewards available queue capacity while penalizing large differences in queue usage among stations.
03
Satellite Resource
Energy
Accounts for energy consumed during transmission and the satellite's remaining energy.
INVALID ACTION
Selecting a ground station that was not currently visible was treated as a failed transmission and assigned a penalty.
06 — DQN
Learning a policy for long-term transmission decisions.
A Deep Q-Network was used to estimate the long-term value of transmission actions under changing satellite conditions. I iteratively adjusted reward weights and DQN training parameters while developing the policy.
01
DQN
Learns action values for sequential ground-station selection decisions.
02
n-step Learning
Uses multi-step returns to incorporate rewards beyond a single transition.
03
Prioritized Replay
Uses Prioritized Experience Replay to focus learning on more informative transitions.
TRAINING
10,000
Episodes used in the reported experimental setup.
STEPS / EPISODE
1,440
One simulated day at one-minute resolution.
07 — DYNAMIC CONSTRAINTS
CHALLENGE
The action space changed as the satellite moved.
A ground station that is valid at one moment may become unavailable minutes later. This makes satellite communication different from a reinforcement learning problem with a permanently fixed set of valid actions.
Orbit Changes
Satellite Movement
Visibility Changes
Valid Stations
Observation
Visibility Included
DQN Action
Station Selection
Invalid Choice
Transmission Penalty
CONSIDERED
Action Masking
Explored restricting the policy to ground stations that were valid under the current visibility state.
IMPLEMENTED
Penalty-based Constraint
Included visibility in the observation and penalized invalid selections so the agent could learn the consequences of unavailable actions.
08 — RESEARCH RESULT
01
Dynamic RL Environment
Built a TLE-based orbital simulation with time-varying ground-station visibility and resource constraints.
02
DQN Selection Policy
Proposed a DQN-based policy considering data freshness, queue load, satellite energy, and communication availability.
03
First-author Paper
Organized the research methodology, experimental results, and limitations and wrote the paper as first author.
PUBLICATION
Energy-Efficient Ground Station Optimization for LEO Satellites via Reinforcement Learning
KICS Winter Conference · 2025
The research demonstrated the feasibility of applying reinforcement learning to a dynamic satellite communication problem. The study did not claim a production-ready optimal policy; further work was required to model the more complex trade-offs of real operational environments.
09 — WHAT I LEARNED
In reinforcement learning, environment design is part of the model.
This research showed me that reinforcement learning performance depends not only on the neural network, but also on how the real-world problem is translated into states, actions, constraints, and rewards.
Building the orbital simulation and repeatedly refining the reward structure helped me understand how modeling assumptions directly shape the behavior learned by an agent.
RESEARCH LINKS
←Back to Projects
LEO Satellite RL · 2024 — 2025