DTN Phase
II & III
Learning Algorithms for
Robust Networking
Robust Internetworking
in Disruptive Environments
Mark-Oliver Stehr &
Carolyn Talcott
José Joaquin
Garcia-Lunes-Aceves &
Ignacio Solis
SRI
International
PARC
Palo Alto Research Center
DTN Phase II Kickoff Meeting
Washington, DC
August 9, 2006
© 2006 SRI International
Overview
• Motivation
• Overview of LEARN & RIDE Collaboration
– Objective and Vision
– Core Technologies and Technical Approach
• The SRI LEARN Project
– General Framework, Challenges & Technical Approach
– Detailed Objectives for Phases II and III
– Schedule, Milestones & Deliverables
• Conclusion
© 2006 SRI International
2
SATCOM on the Move:
Yet Another Motivation for DTN
dB Relative to LOS
From Lincoln Labs, Marc Zissman and Mark Smith
© 2006 SRI International
3
Objective and Vision
• Objective:
– reliable communication in highly disruptive environments
without end-to-end connectivity
• Key Problem:
– Current generation Internet protocols hardly utilize storage
which is abundant in today’s networks
• Guiding visions:
– content-based networking
– knowledge-based networking
Content &
Dissemination
Goals
© 2006 SRI International
Interest in
Content
4
Core Technologies
• New content-based routing algorithms for
storage-rich disrupted environments
• Distributed knowledge management and
distributed learning as a cross-layer
technology
• Novel approaches to limit information flow
• Content-based algorithms for self-forming
and hierarchical virtual topologies
© 2006 SRI International
Phase II
Phase III
5
Technical Approach
• Routing
• Opportunistic routing driven by virtual
potentials of interest and resistance
• Learning-based routing with
multi-level learning
• Efficiency Improvements
Phase II
• Topology Formation
• Opportunistic virtual topology formation
• Learning-based virtual topology formation
• Hierarchical and agent-organizational
techniques for scalability and robustness
© 2006 SRI International
Phase III
6
Learning Algorithms
for Robust Networking
"Learning is constructing or modifying
representations of what is being experienced.”
Ryszard Michalski
"Learning denotes changes in a system that ...
enable a system to do the same task
more efficiently the next time.”
"Learning is making useful changes in our minds.”
© 2006 SRI International
Herbert Simon
Marvin Minsky
7
General Framework
• Foundation:
– Markov Decision Processes
– Reinforcement Learning
© 2006 SRI International
8
General Framework
• Foundation:
– Markov Decision Processes
– Reinforcement Learning
• To be modified to accommodate:
– Distributed and cooperative nature of DTN routing problem
– Network disruptions and extreme delays
– Distributed/delayed reward/punishment without unique origin
– Global vs. local optimization objectives
– Exploitation and Exploration for adaptivity in DTN
– Rich state and action space requires abstractions/generalizations
– Partial observability and uncertainty
– Nonstationary nature of network
© 2006 SRI International
9
Technical Approach
• Key Problem: Reinforcement Learning requires
reasonably stable environment (model)
• Solution: Use intermediate layer to learn stable
abstractions of the environment
Learning via
Interaction
Learning via
Observation
Learning-Based Routing
Learning Network Patterns
Distributed Knowledge Management
© 2006 SRI International
10
SATCOM on the Move:
Connnectivity Patterns
dB Relative to LOS
From Lincoln Labs, Marc Zissman and Mark Smith
© 2006 SRI International
11
Phase II Objectives
• Simulation Prototypes and Evaluation:
– Distributed Knowledge Management Algorithm
– Distributed Learning Algorithm
– Learning-based Routing Algorithm
• Implementation of a Routing Module for the
MITRE DTN Plug-in Architecture
– Precise functionality will depend on capabilities of the
architecture and the routing module interface
– As a minimum requirement we assume that neighbor
discovery and persistent storage services will be
available
© 2006 SRI International
12
Phase III Objectives
• Simulation Prototypes and Evaluation:
– Efficiency Enhancements of Phase II Algorithms

Learning-based techniques to limit propagation of information
– Learning-based Topology Formation Algorithm

active management of the topology and storage to adapt to network
capabilities and characteristics, its dynamics and the application
demands => Strategic selection/placement of custodians
– Improving Topology Formation using
Hierarchical & Agent Organizational Techniques
• Extending our Phase II DTN Routing Module
– Integrated Learning-based Routing and Topology Formation
Module for the MITRE DTN Plug-in Architecture
© 2006 SRI International
13
Phase III Objectives
New in Phase III
Learning-Based Routing
Learning-Based Topology Formation
Learning Network Patterns
Learning-Based Knowledge Management
Enhanced in Phase III
© 2006 SRI International
14
Schedule, Milestones & Deliverables
8/06 5/07
2/08
11/08 8/09
Learning-Based Routing & Supporting Alg.
Simulation Prototype
Efficiency Improvements
Simulation Prototype
Topology-Formation & Organizational Alg.
Simulation Prototype
Documentation and Evaluation
Implementation and Testing
Routing Module
= Preliminary Version
= Final Version
© 2006 SRI International
Phase II Phase III
15
Conclusion: Strengths & Impact
• Paradigm shift towards higher level objectives, e.g. from
message exchange to content dissemination driven by
application goals
• New generation of protocols will enable use of network
storage, a valuable resource virtually unutilized by
current protocols
• Technology independence enables seamless
interoperation with existing and future protocols
• Wide-spread use facilitated by technology independence
further increases available resources
• Multiparty communication becomes an emerging concept
of content-based networking
© 2006 SRI International
16
Project Team
SRI International
Computer Science Laboratory
PARC
Palo Alto Research Center
Mark-Oliver Stehr
Carolyn Talcott
José Joaquin Garcia-Luna-Aceves
Ignacio Solis
Expertise
Design of Network Protocols
Reasoning and Learning
Formal Modeling and Analysis
Semantic Models and Languages
© 2006 SRI International
Wireless, Mobile Ad Hoc Networks
Routing and Topology Formation
Multipoint Communication
Content-Based Networking
17