Considerations of network/system for AI services

In the Internet of Things (IoT), the volume of generated data has exploded alongside the rapid proliferation of devices due to industrial digitization and the development and dissemination of new devices. Various methods have been tried to effectively handle the growing number of IoT devices and the data they produce. One of them is to provide IoT services close to IoT devices and users, shifting away from centralized cloud computing frameworks that transmit all data generated by IoT devices to a cloud server . IoT services have also been moving away from the traditional method of analyzing collected IoT data in the cloud and delivering the analyzed results back to IoT objects or devices. AIoT (Artificial Intelligence of Things), a combination of IoT technology and artificial intelligence (AI) technology, has been discussed at international standardization organizations such as ITU-T. AIoT, as discussed by the ITU-T CG-AIoT group, is defined as a technology that combines AI technology and IoT infrastructure to achieve more efficient IoT operations, improve human-machine interaction, and improve data management and analysis . The first work started by the IETF to apply IoT technology to the Internet was to research a lightweight protocol stack instead of the existing TCP/IP protocol stack so that various types of IoT devices, not traditional Internet terminals, could access the Internet . These technologies have been developed by the 6LoWPAN, 6lo, 6TiSCH, and CoRE working groups and the T2TRG research group, among others. Just as IoT technology was integrated into resource-constrained devices and connected to the Internet, AI technology has also changed from running only on very high-performance servers to running on small hardware, including microcontrollers, low-performance CPUs, and AI accelerators. This direction of technology development is called On-device AI, TinyML, or, more broadly, edge AI . More recently, generative AI models, including Small Language Models (SLMs), have also begun to run on end devices and edge devices equipped with neural processing units, while Large Language Models (LLMs) continue to be served mainly from cloud data centers. In this document, we consider how to configure the network and system for AI inference services. The technology of collecting sensing data from various sensors and delivering it to the cloud has already been studied by many standardization organizations including the IETF, and many standards have been developed. After an AI model has been created based on the collected data, how to deploy this AI model as a system has become a main research question. Until now, it has been common to collect data and perform inference on the servers used for training, but as AI services spread, using expensive servers for every AI service is not appropriate. In addition, since the servers that collect data and train models mainly exist in the cloud, connecting a large number of terminals to these cloud servers to request AI services raises many problems. Therefore, requesting an AI service from an edge device located nearby, rather than from an AI server located in a distant cloud, may support real-time services, reduce network traffic, and keep important data local . Even if an edge device is used to serve AI services, it is still important to connect to an AI server in the cloud for tasks that take a lot of time or require a lot of data. Therefore, offloading techniques that properly distribute the workload between the cloud server and the edge device are also being actively studied. Furthermore, beyond the deployment of basic AI inference on distributed devices, the concept of Agentic AI is rapidly emerging. Agentic AI refers to AI systems capable of autonomous goal setting, complex task decomposition, planning, tool invocation, execution, and continuous learning and reflection, rather than simple one-off inference responses. Deploying such Agentic AI systems in a distributed environment (cloud, edge, and local) is expected to be an important building block for autonomous and intelligent network management and service provision. The considerations for distributed AI deployment in this document also apply to the network infrastructure required by Agentic AI. This document does not specify agent architectures or agent communication protocols. In this document, for the network structures described in , the points to be considered when a client connects to a server or an edge device and requests an AI service are derived and described. The following considerations and options are derived: AI inference service execution entity Hardware specifications of the machine that performs AI inference services Selection of AI models to perform AI inference services Method of providing AI services from cloud servers or edge devices Communication method to transmit data to request AI inference services Characteristics of generative AI and LLM inference workloads Distribution and lifecycle management of AI models across the cloud continuum Autonomous, agent-driven selection of the execution point (offloading) The proposed considerations are applied to describe the use cases of self-driving vehicles, network digital twins, and Physical AI, all of which can benefit from providing AI services in a distributed manner.

This document uses the following terms: A service that executes a trained AI model on input data provided by a client and returns the inference result. A software module that requests an AI inference service and receives its result. A software module that executes an AI model to perform an AI inference service. The machine on which the AI client module runs (e.g., an end device, a vehicle, or a user terminal). A computing node located in the edge network, close to the local machine, that can host an AI server module. A computing node located in a cloud data center that can host an AI server module. The set of local machines, edge devices, and cloud servers across which the components of an AI service can be placed. Also called the cloud-edge-local continuum. Transferring all or part of an AI task from one execution point to another (e.g., from a local machine to an edge device or a cloud server). Large Language Model / Small Language Model. Generative models whose inference produces output tokens sequentially. SLMs are small enough to run on end devices or edge devices. A software entity that uses an AI model (typically an LLM) to perceive its environment, plan, invoke tools, and act toward a goal with some degree of autonomy. AI systems composed of one or more AI agents. The function of an AI agent that decomposes tasks and decides where (local, edge, or cloud) each task is executed. It corresponds to the Intelligence Core described in . An external capability (e.g., an API or a network management interface) that an AI agent can invoke to obtain information or perform an action. A system in which multiple AI agents interact, negotiate, and collaborate toward common or individual objectives. AI systems that perceive the physical environment through sensors, perform AI inference, and act on the physical world through actuators (e.g., robots, drones, autonomous vehicles, and industrial automation systems). An AI service whose result must be delivered before a deadline. An AI service whose result must be delivered within a specified time range (neither too early nor too late).

Given the extensive history of research on AI services, various architectures exist for providing them. However, due to the nature of AI technology, a system for providing AI services generally consists of the following steps .

+----------+ +----------+ +----------+ +----------+ +----------+ |Collect & | |Analysis &| | Train | | Deploy & | |Monitor & | | Store |->|Preprocess|->| AI model |->|Inference |->| Maintain | | data | | data | | | | AI model | | Accuracy | +----------+ +----------+ +----------+ +----------+ +----------+ |<-------->| |<---------------------->| |<-------->| |<-------->| Sensor, DB AI Server Target AI Srvr & machine Tgt. machine |<-------->| |<---------------------->| |<-------->| |<-------->| Internet Local Internet Local & Internet Data collection & Storage Data Analysis & Preprocessing AI Model Training AI Model Deploy & Inference Monitor & Maintain Accuracy In the data collection step, data required for training is prepared by collecting data from sensors and IoT devices or by using data stored in a database. Equipment involved in this step includes sensors, IoT devices, servers that store their data, and database servers. Since the operations performed at this step are conducted through the Internet, many IoT technologies developed by the IETF are suitable for this step. In the data analysis and preprocessing step, the features of the prepared data are analyzed and preprocessing for training is performed. Equipment involved in this step includes a high-performance server equipped with a GPU and a database server, and this step is mainly performed in a local network. In the AI model training step, a model is created by applying an algorithm suitable for the characteristics of the data and the problem to be solved. Equipment involved in this step includes high-performance servers equipped with GPUs or other AI accelerators, and this step is mainly performed in a local or data center network. In the AI model deployment and inference service provision step, the problem to be solved (e.g., classification, regression, or content generation) is solved using the trained AI model. Equipment involved in this step may include a target machine, a client, edge devices, and cloud servers that provide AI services. Since various equipment is involved in this step, it is conducted through the Internet. This document summarizes the factors to be considered at this step. In the accuracy monitoring step, if performance deteriorates due to new data, a new model is created through re-training, and the AI service quality is maintained by using the newly created model. This step repeats the model training and model deployment steps described above. Two aspects have become prominent in addition to these classical steps. First, many AI services are no longer built by training a model from scratch; instead, a pre-trained foundation model is adapted to the target task through fine-tuning (including parameter-efficient methods) or is complemented at inference time by retrieving external knowledge (Retrieval-Augmented Generation, RAG). Second, the distribution of models, adapters, and their updates to many inference points has become a networking task in its own right, as discussed in .

In general, after an AI model has been trained, it can be deployed on a local machine to provide AI inference services. Alternatively, AI models can be placed on cloud servers or edge devices, and AI service requests can be made remotely. In addition, for overall service performance, some AI service requests can be sent to the cloud server and others to edge devices through appropriate load balancing.

The following figure shows a case where an AI client module on a local machine requests an AI service from an AI server module on the same machine.

+---------------------------------------------------------------------+ | | | +-----------------+ Request AI +-----------------+ | | | Client module | Inference service | Server module | | | | for AI service |----------------------->| for AI service | | | | |<-----------------------| | | | +-----------------+ Reply AI +-----------------+ | | Inference result | +---------------------------------------------------------------------+ Local machine This method is often used when configuring a system focused on training AI models to improve their inference accuracy and performance, without particular consideration of AI service deployment and inference. In this case, since the AI client module that requests the AI inference service and the AI server module that performs it are on the same machine, the communication/network environment and the service provision method need little consideration. This method can also be used to implement a self-contained AI inference service that is not expected to change, such as on an embedded or customized machine. In this case, the hardware does not need to be capable of training the AI model, but it must be capable of running the AI inference service.

The following figure shows the case where the AI client module that requests the AI service and the AI server module that performs the AI service run on different machines.

The following figure shows the case where the AI client module and the AI server module are separated, and the AI server module is located on an edge device. As more and more data is generated at the edge, it is necessary to perform AI processing at the edge due to performance, privacy, and the cost of moving data to a central location. For example, some enterprises owning both local machines and edge networks may hesitate to upload confidential datasets to the central cloud due to privacy and security concerns. Instead, they retrieve AI models from the cloud and use them for AI inference locally at the edge device.

The following figure shows the case where AI server modules are distributed over the cloud and edge devices.

+--------------------------------------+ +------------------------+ | +---------------------------+ | | +-----------------+ | | | +-----------------+ | | | | Client module |<-+-+----+-----+---->| Server module | | | | | for AI service |<-+-+ | | | for AI service | | | | +-----------------+ | | | | +-----------------+ | | +------------------------+ | | +---------------------------+ | Local machine | | Edge device | | +--------------------------------------+ | Edge network | | +--------------------------------------+ | | +---------------------------+ | | | | +-----------------+ | | +----+-----+---->| Server module | | | | | | for AI service | | | | | +-----------------+ | | | +---------------------------+ | | Server machine | +--------------------------------------+ Cloud (Internet) AI server modules in the cloud and on edge devices differ in AI inference service performance. Therefore, the client may distribute its AI inference service requests between the cloud and the edge device appropriately to obtain the desired AI service. For example, a request for which a short inference time matters more than the highest inference accuracy can be sent to the edge device. The distribution of AI inference service requests between the cloud and edge devices is a critical decision. In the context of Agentic AI, the AI client module can be extended with an AI agent planning module. This module determines the execution point (local, edge, or cloud) based on a dynamic assessment of inference accuracy requirements, latency constraints (e.g., in-time/on-time delay sensitivity), and resource costs. Since the decision making of an LLM-based agent itself consumes time and resources, the timescale at which such decisions are made needs to be considered, as discussed in .

The configurations in Sections to place AI server modules at different levels of a vertical hierarchy of local machines, edge devices, and cloud servers. Because the capabilities of each machine are different, the overall performance of a vertical hierarchy depends on each machine. Generally, a cloud server has the most powerful computing capabilities, followed by an edge device. In contrast, in a horizontal configuration, multiple server machines at the same level (e.g., several cloud server machines or several edge devices) can serve the same AI inference service. In this configuration, the AI service may show different performance according to the load level of each server, the computing capability of each server machine, and the link state between the local machine and each server machine. Thus, to find the server machine that can provide the best AI service, a network element is required to monitor the network link state and the current computing capability of the server machines, along with a load balancer that can enforce a scheduling policy. When a client requests an AI inference service, the request is routed to a single server machine based on the load balancer's decision. Alternatively, the request can be processed by several server machines that each contribute to the final inference result. The Computing-Aware Traffic Steering (CATS) framework provides an architecture for this kind of joint compute- and network-aware selection of a service instance. The monitoring element and the load balancer described above map to the CATS functional components that collect computing metrics and network metrics and select a service contact instance. AI inference services can benefit from CATS, and AI-specific metrics (e.g., the model hosted by an instance, its expected accuracy, and its current inference latency or queue length) are candidate inputs to such selection. These functions can be placed at different locations depending on the deployment. First, they can be collocated with the edge device or with the first network element that the request from the local machine reaches (e.g., an access router or an edge gateway), which allows the selection to be made close to the client and is suitable when the candidate server machines are edge devices. Second, they can be placed at the network element that interconnects the edge network and the cloud (e.g., a gateway or an ingress router of a provider network), which is suitable when the candidates include both edge devices and cloud servers. Third, a logically centralized controller can collect the computing and network metrics and compute the selection policy, which is then enforced by the network elements on the path. Finally, the local machine itself can select the server machine based on information obtained from the network. In terms of the CATS framework , the collection of computing metrics is typically collocated with the server machines, the collection of network metrics is performed by network elements or a controller, and the selection of the service instance is performed either at the ingress network element closest to the local machine or by a centralized controller. When utilizing horizontal multiple servers, Multi-Agent Systems (MAS) based on Agentic AI can also be applied. Instead of relying solely on a centralized load balancer, multiple AI agents, each deployed on a separate server machine or edge device, can negotiate tasks and collaborate toward a common objective. This structure allows a complex AI task to be decomposed into sub-tasks, with each agent specializing in a part, leading to a collectively processed final result, improved resilience, and dynamic resource optimization. Such collaboration requires agents to discover each other, exchange tasks and results, and resolve conflicting decisions. The interoperability of agent communication is being addressed by agent communication protocols such as MCP and A2A and by the IETF AGENTPROTO working group . The following figure shows the case where the local machine requests an AI service from horizontal multiple cloud servers.

+--------------------------------------+ | +---------------------------+ | | | +-----------------+ | | +----+-----+---->| Server module | | | | | | | for AI service | | | | | | +-----------------+ | | | | +---------------------------+ | | | Server machine 1 | | +--------------------------------------+ | Cloud (Internet) | | +--------------------------------------+ +------------------------+ | | +---------------------------+ | | +-----------------+ | | | | +-----------------+ | | | | Client module |<-+-+----+-----+---->| Server module | | | | | for AI service |<-+-+ | | | for AI service | | | | +-----------------+ | | | | +-----------------+ | | +------------------------+ | | +---------------------------+ | Local machine | | Server machine 2 | | +--------------------------------------+ | Cloud (Internet) | | +--------------------------------------+ | | +---------------------------+ | | | | +-----------------+ | | +----+-----+---->| Server module | | | | | | for AI service | | | | | +-----------------+ | | | +---------------------------+ | | Server machine 3 | +--------------------------------------+ Cloud (Internet)

Collecting and preprocessing data and training an AI model require high-performance resources such as CPU, GPU, power, and storage. To mitigate this requirement, a network-side configuration can be utilized. Federated learning is a machine learning technique that trains an AI model across multiple decentralized servers, in contrast to traditional centralized machine learning techniques where all local datasets are uploaded to one server. Federated learning enables multiple network nodes to build a common machine learning model. These network nodes or servers can be located in the same data center network or in multiple data center networks. The security implications of federated learning, such as model poisoning, are discussed in . Transfer learning is a machine learning technique that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. In transfer learning, a network configuration can be utilized to transfer common information and knowledge between different network nodes within one data center network or across multiple data center networks.

As described in the previous section, the AI server module that performs AI inference services can run on a local machine, a cloud server, or an edge device. In theory, if the AI inference service is performed on a local machine, the AI service can be provided without communication delay or packet loss, but a certain level of hardware performance is required to perform AI inference. As AI services become widespread, the cost of the machines that perform AI services becomes important. Therefore, whether the AI inference service is performed on the cloud server or on an edge device with lower infrastructure cost can be a determining factor in the system configuration.

When an AI inference service request is made to a distant cloud server, transmission may take a long time, but the cloud server can process many AI inference requests in a short time and can host larger models, which typically provide higher inference accuracy. Conversely, when an AI service request is made to a nearby edge device, the transmission time is short, but the edge device cannot process as many requests at once and usually hosts smaller models with lower inference accuracy. Note that the inference accuracy is determined by the deployed model, not by the location itself. Therefore, by analyzing the characteristics and requirements of the AI service to be performed, it is necessary to determine whether to perform the AI inference service on a local machine, a cloud server, or an edge device. The hardware characteristics of the machines performing AI services vary. In general, cloud servers are regarded as having higher performance than edge devices. However, the performance of an AI inference service depends on how the hardware, such as CPU, RAM, GPU or other AI accelerators, and network interfaces, is configured for each cloud server and edge device. Without cost constraints, a system for AI services would use the machines with the best hardware performance, but in reality, cost must always be considered. Therefore, the performance of the local machine, cloud server, and edge device must be determined according to the characteristics and requirements of the AI service to be performed. Performance evaluation is possible through the performance metrics presented by ETSI . These metrics are divided into two groups. Functional metrics assess user performance and include classical indexes such as latency in task execution, device energy efficiency, bit rate, loss rate, jitter, and Quality of Service (QoS). Non-functional metrics focus on the deployment and management of the MEC (Multi-access Edge Computing, formerly Mobile Edge Computing) network. Non-functional metrics include service life-cycle (instantiation, service deployment, service provisioning, service update (e.g., service scalability and elasticity), and service disposal), service availability and fault tolerance (reliability), service processing/computational load, global mobile equipment host load, number of API requests (more generally, number of events) processed per second on the mobile equipment host, delay to process API requests (north and south), and number of failed API requests. The sum of service instantiation, service deployment, and service provisioning provides the service boot time. For generative AI services, additional metrics are commonly used, such as time to first token (TTFT), time per output token, token throughput (tokens per second), energy consumed per inference or per token, and cost per inference or per token. Energy consumption in particular is becoming an important criterion when choosing an execution point, together with accuracy, latency, and cost.

Although not directly related to communication or networking, the AI model used is the biggest factor influencing an AI inference service. For example, for image classification there are various AI models, such as ResNet, EfficientNet, VGG, and Inception, and for language tasks there are language models ranging from SLMs with a few billion parameters or fewer to LLMs with hundreds of billions of parameters. These models differ not only in inference accuracy but also in model size, memory footprint, and inference time. Models with the highest inference accuracy typically have very large sizes and long inference times. Therefore, when constructing an AI service system, choosing the model with the highest inference accuracy is not always appropriate, and the model must be selected according to the characteristics and requirements of the AI service. Model compression techniques, such as quantization, pruning, and knowledge distillation, can reduce model size and inference time at some cost in accuracy, and allow a family of model variants to be derived from one model. A common practice is to deploy a model with high inference accuracy on the cloud server and a compressed or smaller model that provides fast inference, with somewhat lower accuracy, on the edge device. How AI services are delivered on cloud servers or edge devices also needs to be considered, although this is partly an implementation issue. AI server modules can be built with general-purpose web frameworks or with serving runtimes specialized for AI inference. General-purpose web frameworks run on many types of machines, but since they are designed for general web services, they do not provide inference-specific optimizations. Specialized inference serving runtimes provide optimizations such as request batching, hardware acceleration, and, for LLMs, efficient management of the key-value (KV) cache, but they may require specific hardware features (e.g., particular CPU instruction sets or GPUs). Therefore, the AI server module method should be decided in consideration of the hardware characteristics of the AI system that can be built.

The communication method used to transfer data when requesting an AI inference service is also an important decision in constructing an AI system. The traditional REST method can be used with various machines and services, but its performance is generally inferior to gRPC. gRPC uses HTTP/2 and protocol buffers, providing low latency, efficient data serialization, and efficient transfer of large data compared to REST. QUIC or other transport protocols can also be used to request an AI inference service. For generative AI services, the results are typically streamed token by token over long-lived connections, and agent-based services may maintain long-lived sessions between agents and tools. The selection or specification of a particular transport protocol is beyond the scope of this document. Cloud-edge-endpoint collaboration-based AI service development is actively underway. In particular, for AI services that are sensitive to network delay, such as object recognition and autonomous vehicle services, (micro)services for inference are placed on edge devices close to the client to obtain fast inference results. As such, in the development of intelligent IoT services, various devices that provide computing services within the network, such as edge devices, are being added as network elements, and the number of IoT devices using them is rapidly increasing. Therefore, new functions for computing resource management and operation are required to provide computing services within the network. In addition, to operate distributed AI services on the network, network policies for collaboration are required between edge devices, or between edge devices and endpoint devices, that provide computing resources for AI services. From a network policy perspective, to efficiently support distributed AI services, the network needs to support the collaboration of AI services between edge devices, such as multi-edge network configurations for AI-service-aware traffic steering. This allows the client to receive distributed AI services efficiently in various network environments and allows network resources and edge computing resources to be adjusted dynamically. The CATS framework is applicable to such AI-service-aware traffic steering. For example, AI task messages must be exchangeable between edge devices, or between edge devices and endpoint devices, to provide collective AI inference. Another example is caching at edge devices between the client and the central cloud: a copy of an accessed file or model is kept at the edge device and served for subsequent requests from the same or other clients. For generative AI services, caching can also be applied to results of semantically similar requests and to reusable inference state such as the KV cache of a shared prompt prefix. There are also various delay-sensitive AI services based on edge devices in the network. They are divided into in-time delay-sensitive AI services with a deadline and on-time delay-sensitive AI services with a time range. In particular, on-time delay-sensitive AI services need to return prediction results within the time range. Therefore, distributed AI services must provide appropriate delay performance through the network, and the client should be able to use both in-time and on-time delay-sensitive services by interacting with the edge devices where distributed AI services are built. To support on-time delay-sensitive AI services, the underlying network may need to interact with deterministic networking capabilities, such as DetNet or IEEE 802.1 Time-Sensitive Networking (TSN) , to guarantee bounded latency.

Inference of generative AI models, in particular LLMs and SLMs, has characteristics that differ from classical inference (e.g., security risk identification, anomalous flow behavior detection, and application identification, which are widely used in networks as embedded or lightweight AI capabilities) and that affect where it should be executed: Two-phase execution: LLM inference consists of a prefill phase, which processes the whole input context and is compute-bound, and a decode phase, which generates output tokens one by one and is memory-bandwidth-bound. These phases may be placed on different machines, in which case the intermediate state (KV cache) must be transferred between them over the network. Asymmetric and variable traffic: the input context (e.g., documents, images, or retrieved knowledge) can be large in the uplink, while the output is streamed in small increments over a period that depends on the output length. The response time is therefore not fixed in advance. Collaborative edge-cloud inference: an SLM at the edge or on the local machine can generate candidate tokens that are verified by an LLM in the cloud (speculative decoding), or simple requests can be answered at the edge while complex ones are escalated to the cloud. In such schemes, each round trip between the edge and the cloud adds network latency, so the benefit depends on the RTT and on how often escalation or verification is needed. Split inference: a model can be partitioned so that the first layers run on the local machine or edge device and the remaining layers run in the cloud. The size of the intermediate data at the split point and the available bandwidth determine whether such a split is beneficial. Context and state: multi-turn and agent-based services maintain a growing context. Moving a session from one execution point to another (e.g., due to mobility) requires either transferring this context or recomputing it. These characteristics mean that the network metrics relevant for placing generative AI workloads include not only latency and loss but also sustained bandwidth for context and state transfer and the stability of long-lived streams.

When AI inference is distributed over many edge devices and local machines, the models themselves must be distributed and kept up to date. Model files, including those of compressed models, range from megabytes to many gigabytes, and adapters produced by fine-tuning add further artifacts. Distributing them to a large number of inference points consumes significant network resources and needs to be planned, for example by scheduling updates, using caching or multicast-like distribution at the edge, and transferring only the differences between model versions where possible. Consistency is also a concern. When a service uses models in the cloud and at the edge together, the versions of these models need to be compatible, and a mechanism to roll back to a previous version is needed when a new version degrades the service. To support such rollback, each deployed version of a model, including its adapters and configuration, needs to be stored as a checkpoint or snapshot, for example in a model registry, so that an inference point can revert to any previously saved version. Where these checkpoints are stored also matters: keeping recent versions cached at the edge allows a fast rollback without transferring the model again from the cloud, at the cost of additional storage at the edge. The integrity and authenticity of distributed models need to be verifiable by the inference points, as discussed in . Interoperability challenges in deploying the same AI solution across different environments are discussed in Section 7.2 of .

Traditional offloading mechanisms predominantly rely on static, rule-based thresholds (e.g., hardcoded values for network latency or available bandwidth) to determine whether an AI task should be executed locally, at the edge, or in the cloud. However, such approaches may fail to adapt to highly dynamic network conditions, volatile computing resource availability, and the nuanced accuracy requirements of complex AI applications. To address these limitations, distributed AI deployments can consider an Agentic AI-driven autonomous offloading framework. An Agentic AI-driven framework complements hardcoded logic with a closed-loop reasoning system consisting of an LLM-based Intelligence Core (the AI agent planning module), a Contextual Memory component, and a Tool Use mechanism for environmental sensing: Intelligence Core and Planning: The LLM-based core interprets high-level application goals or network intents, rather than simple binary commands. When a complex AI task is requested, the core performs task decomposition, breaking the objective into sequential or parallel sub-tasks. It then formulates an execution plan by evaluating trade-offs among model inference accuracy, latency bounds (in-time/on-time sensitivities), and energy and financial costs. Contextual Memory: The memory component maintains both short-term environmental context and long-term historical knowledge. Short-term memory tracks immediate telemetry fluctuations, while long-term memory retains historical performance data of specific edge and cloud nodes, previous offloading success rates, and semantic context (e.g., geographical routing patterns in vehicular networks). This allows the agent to anticipate network degradation and make proactive, rather than reactive, offloading decisions. Tool Use for Resource Sensing: To ground its reasoning in the actual state of the system, the AI agent uses a predefined set of tools, such as APIs and network management interfaces. Through tool invocation, the agent queries the network link state (such as RTT, packet loss rate, and jitter) and the processing capacity of available nodes (such as CPU/GPU utilization, memory usage, and queueing delay). Existing network telemetry and YANG-based management interfaces can be exposed to the agent as tools, for example through an agent-to-tool protocol such as MCP . By orchestrating planning, memory, and tool-based sensing, the AI agent continuously perceives the cloud continuum. For example, it can decide to withdraw a cloud offloading plan and fall back to a lightweight on-device model when a tool detects a sudden drop in wireless link quality, improving the resilience of the AI service. The following points need to be considered when applying this framework: Timescale separation: the reasoning of an LLM-based agent typically takes from hundreds of milliseconds to several seconds, which can exceed the deadline of the tasks being offloaded (e.g., about 33 ms per frame at 30 FPS in ). Therefore, the agent is better suited to a slow control loop that sets policies, plans, and placements, while per-request or per-frame offloading decisions are made by a fast control loop based on lightweight rules or learned models that follow the policies set by the agent. Predictability and validation: LLM outputs are not deterministic and may be incorrect. Decisions produced by the agent need to be validated against constraints (e.g., resource limits and policies) before they are enforced, and a safe default behavior (e.g., local execution with a fallback model) needs to be defined for when the agent fails or does not respond in time. Approaches such as the incremental application of decisions with rollback and the verification of decisions against safety bounds are discussed in Section 5.5 of . Human oversight: actions with a significant impact on the network or on other services need to be subject to approval, auditing, and the possibility of intervention by operators, as discussed in . Overhead: the agent itself consumes computing resources and generates telemetry and tool traffic. This overhead needs to be accounted for when evaluating the benefit of agent-driven offloading.

The document describes the challenges of coupling AI and network management (NM). It first characterizes difficult NM problems with six problem characteristics (C1 to C6), and then describes the high-level challenges in adopting AI in NM, which are detailed in Sections 5, 6, and 7 of . This section describes how the concepts discussed in the present document are linked to those characteristics and challenges. In particular, Section 5.4 of identifies distributed AI as a research challenge in two directions. The first is network management for efficient distributed AI, in which the communication requirements of a distributed AI application are mapped to network requirements such as latency, bandwidth, and confidentiality, leading to research on AI-aware network management and network-aware AI applications. The second is distributed AI for network management. The present document elaborates the first direction for AI inference services and AI agents deployed across the cloud continuum ( and ), while the network digital twin use case in illustrates the second. In addition, lightweight AI, as described in Section 5.3 of , in which AI tasks are offloaded to network devices while costly learning tasks remain on high-end servers, corresponds to the placement of AI server modules on local machines and edge devices described in . The structures presented in this document can also be combined with management systems that introduce AI functions into network management cycles and express management goals as network intents . An example of such a framework is AINEMA , and an example of such a management system is shown in .

C1 (Fundamental Algorithmic Complexity) refers to a solution space that explodes combinatorially with the scale and topological footprint of the network. The present document proposes to tackle part of such complexity by separating the AI work across multiple elements of the network and by decomposing complex AI tasks into sub-tasks handled by different servers or agents (). The separation is asymmetrical, so that some elements (cloud side) contribute more computation power to the distributed service. C2 (Uncertainty and Unpredictability) refers to highly dynamic and non-deterministic variables, such as shifting user traffic patterns, limited visibility into end-to-end network state, and multi-domain interdependencies. The present document addresses part of this characteristic by confining AI models and data within pre-defined client-edge-cloud structures, so that AI functions executed on local machines or edge devices can rely on locally observed state rather than on an incomplete global view. In addition, the uncertainty of the outputs of AI agents themselves is mitigated by validating their decisions against constraints and by defining safe fallback behaviors (). C3 (Real-Time Constraints and Determinism) refers to computing solutions and executing closed-loop control actions within strictly bounded or deterministic timeframes. The key aspect discussed in this document to address this characteristic is the integration of end-node, in-network, edge, and cloud computing services, which allows AI computations to be performed in the best place possible. The time needed to obtain responses for easy operations is minimized by executing them in end nodes (e.g., vehicles) and edge devices, while the time needed to compute more complex operations is minimized by offloading them to cloud computing services. For on-time delay-sensitive AI services, the network may interact with deterministic networking capabilities (). Furthermore, the separation between a slow, agent-driven control loop and a fast control loop described in allows Agentic AI to be used without placing LLM reasoning on the critical path of time-sensitive decisions. In Physical AI services (), this characteristic applies to the entire closed loop from sensing to actuation. C4 (Data Dependency and Heterogeneity) refers to processing and analyzing massive volumes of network telemetry that are heterogeneous, possibly incomplete, and generated at high velocity. This characteristic is specific to each information domain and problem. However, the difficulties in C4 can be relieved by data processing functions, such as data homogenization, filtering, and aggregation, instantiated in the cloud continuum and used transparently by AI services. Moreover, processing data on edge devices or network devices close to where it is generated reduces the volume of data that needs to be collected and stored centrally (). C5 (Integration with Existing Practice) refers to interoperating with pre-existing automated procedures, legacy deterministic control loops, heterogeneous operator workflows, and established human-in-the-loop structures. The flexibility of the structures presented in this document allows them to be connected to existing systems. In the agent-driven framework described in , the fast control loop can be realized by existing deterministic mechanisms while the AI agent adjusts their policies, and actions with a significant impact are subject to operator approval. The result is exemplified in the support for network digital twin and vehicle environments. C6 (Economics and Computational Footprint) refers to ensuring cost-effectiveness across limited physical and operational resources, including link bandwidth, node CPU, and human overhead, particularly when model training or inference is distributed at the network edge. Enabling AI services to be deployed in the cloud continuum supports the maximization of effectiveness by dynamically relocating services to the cheapest point that is able to meet the computation requirements. This relocation can be guided by intents and implemented by a management cycle that is independent of the AI services, as shown in . As a result, the cost of a distributed AI system is continuously checked and adjusted by requesting the underlying infrastructure to modify the point of instantiation of each part of the AI system. The energy and cost metrics in can be used as inputs to such decisions, and the overhead of model distribution () and of the AI agents themselves () needs to be accounted for.

Section 4 of describes three high-level challenges in adopting AI in NM, which are detailed in Sections 5, 6, and 7 of , respectively. The first high-level challenge is that AI advances have historically been driven by other communities, such as image/video and natural language processing, so network-tailored AI has been overlooked and AI techniques must be extensively adapted for network problems. Although this is a complex challenge, this document supports its resolution by enabling AI models and algorithms to evolve separately. Multiple providers can offer multiple solutions, and they can be stitched together by following the structures introduced above. The second high-level challenge concerns network data as input to AI: AI techniques require sufficiently representative data, and the analysis of network data can expose sensitive information and raise privacy concerns. Although the present document does not directly address data representation, the structures presented in this document help alleviate the burden by relocating some AI processes as close as possible to the end points (e.g., vehicles and NDT). The same structures support the implementation of measures to protect privacy. The AI services that are closer to the edges deal with sensitive data and ensure privacy protection by, for example, anonymizing it. Those services are instantiated within the data domain, so private data does not cross the boundaries of its data domain. The third high-level challenge concerns the seamless integration of AI into existing NM processes and its acceptability to administrators and operators. Giving administrators and operators control over AI operations increases such acceptance. The structures presented in this document support the involvement of administrators and operators in AI system processes through the provision of policies and network intents. In particular, aligning the distributed AI system described in this document with intent-based networking enables administrators and operators to use high-level specifications (namely network intents) to communicate the policies, requirements, and goals that must be enforced on the AI system constructed over the cloud continuum. For Agentic AI, the validation, human oversight, and fallback mechanisms described in further support acceptance by keeping operators in control of the actions taken by AI agents. These mechanisms correspond to the safeguards discussed in Section 5.5 of , such as the incremental application of AI decisions with rollback, verification against safety bounds, and the use of digital twins as a sandbox, and to the human-in-the-loop approaches discussed in Section 7.3 of .

Various sensors are used in self-driving vehicles, and the final judgment is made by combining their data. Among them, camera-based object detection complements equipment such as LiDAR and RADAR. Camera-based object detection performs various tasks: in addition to lane recognition for keeping and changing lanes, it supports safe driving and parking assistance by recognizing pedestrians, signs, and parked vehicles along the road. To perform such driving assistance and autonomous driving, object detection needs to be performed in real time. A commonly used minimum frame rate for real-time object detection in autonomous driving is 30 FPS (Frames Per Second) . No matter how high the accuracy is, a detector cannot be used for autonomous driving if it does not meet this reference value. Task offloading refers to transferring computing tasks to other processing devices or systems. Task offloading can quickly process tasks that exceed the performance limits of a resource-constrained device by delivering them from devices with limited computing power, storage, and energy to devices that are rich in computing resources. For a device with low hardware performance, such as an embedded board, performing all operations locally without task offloading may not achieve the frame rate required for object-detection-based autonomous driving. On the other hand, if task offloading is applied so that object detection is performed on a device with high hardware performance and the rest of the work is performed on the client, the required frame rate can be achieved, provided that the communication delay between the client and the offloading target is sufficiently small. For AI services such as object detection that are difficult to perform on resource-constrained devices, the task offloading structure can thus be effective. However, offloading between network nodes affects the end-to-end time, because larger data leads to greater communication latency. In addition, a vehicle moves between the coverage areas of different access points and edge devices, so the offloading target may change during a service session, and the context and state of the service may need to be transferred or recomputed. Therefore, in such a distributed network environment, AI services should be designed in consideration of data size, link quality, mobility, and the capabilities of the available nodes. shows an example of distributed AI deployment for a self-driving vehicle that does not have enough capability to perform object detection in real time and offloads some tasks to edge devices and cloud servers.

+--------------------------------------+ +------------------------+ | +---------------------------+ | | +-----------------+ | | | +-----------------+ | | | |Object detection |<-+-+----+-----+---->|Object detection | | | | | service |<-+-+ | | | service | | | | +-----------------+ | | | | +-----------------+ | | +------------------------+ | | +---------------------------+ | Vehicle | | Edge device | | +--------------------------------------+ | Edge network | | +--------------------------------------+ | | +---------------------------+ | | | | +-----------------+ | | +----+-----+---->|Object detection | | | | | | service | | | | | +-----------------+ | | | +---------------------------+ | | Server machine | +--------------------------------------+ Cloud (Internet)

Network digital twins also need distributed AI services. The purpose of a network digital twin is described in . In particular, the network digital twin provides network operators with technology that enables data-driven network management and allows real-time interaction between the physical network and the twin network. To achieve this, the network digital twin uses AI capabilities for various purposes such as scenario planning, impact analysis, and change management. Various AI functions will be applied for optimal network operation and management. However, an actual physical network consists of various network devices and has a complex structure with various types of data, such as topology data, configuration data, and state data. In addition, in a large-scale network environment, collecting and storing information from many network devices in a centralized manner, and creating and changing network management policies based on it, causes very large and expensive network overhead. Therefore, there is a need for a method to apply AI functions in a distributed form for network operation and management. In particular, since an actual physical network is built as a logical hierarchical structure, it is necessary to apply a distributed AI method that considers this logical hierarchical network structure. To optimally perform network operation and management through distributed AI methods, it is necessary to generate AI-based network operation and management topology models and policy models, and an operational method to distribute the generated AI-based network policies across different networks or administrative domains. In particular, to operate a network digital twin in a large-scale network environment, AI-based network policy models need to be generated in a distributed manner. A federated learning algorithm or a transfer learning algorithm that can learn large-scale networks in a distributed manner can be applied.

+-----------------------------------------------------+ | | | Distributed network learning model | | in large-scale network environment | | | | +-------------+------------+ | | | Master | | | | (AI based Policy model) | | | +-------------+------------+ | | | | | +-----------+-----+-----+-----------+ | | | | | | | | +----+----+ +----+----+ +----+----+ +----+----+ | | | Worker | | Worker | | Worker | | Worker | | | | (Agent) | | (Agent) | | (Agent) | | (Agent) | | | +----+----+ +----+----+ +----+----+ +----+----+ | | | | | | | | +----+----+ +----+----+ +----+----+ +----+----+ | | | Local | | Local | | Local | | Local | | | | Data | | Data | | Data | | Data | | | | Repo- | | Repo- | | Repo- | | Repo- | | | | sitory | | sitory | | sitory | | sitory | | | +---------+ +---------+ +---------+ +---------+ | | | +-----------------------------------------------------+ As shown in , to learn a large-scale network through a distributed learning method, a local data repository to store network data must be established in each region, for example, based on location or AS (Autonomous System). The distributed learning method learns through each worker (agent) based on the local network data stored in the local data repository, and generates a large-scale network policy model through the master. This method can reduce the network overhead of centralized data collection and storage, and reduce the time required to create AI models for network operation and management policies for large-scale networks. In addition, the network policy model generated by a worker can be used as a locally optimized network policy model that provides AI-based network operation and management policy services optimized for local network operations. Trained AI network policy models can be deployed on network devices that manage and operate the local network, to minimize network data movement. For example, in a large-scale network consisting of multiple ASes, AI network policy models can be deployed per AS to optimize network operation and management. shows an example of operating and managing a network by distributing AI network policy models by AS.

+---------------------------------------------------------+ | +-------------+------------+ | | | Master | | | | (AI-based Network Policy | | | | model management) | | | +-------------+------------+ | | | | | +-------------+-------------+ | | Worker | Worker | | | +----------+----------+ +----------+----------+ | | | Network device | | Network device | | | | (AI-based network | | (AI-based network | | | | Policy model) | | Policy model) | | | +----------+----------+ +----------+----------+ | | | | | | +----------+----------+ +----------+----------+ | | | AS_1 | | AS_2 | | | +---------------------+ +---------------------+ | +---------------------------------------------------------+ The workers in and can also be realized as AI agents that use the network digital twin as a tool, for example to validate a candidate policy in the twin network before it is applied to the physical network. An architecture that integrates Agentic AI with the network digital twin is described in . The considerations in apply to the placement of such agents and of the AI models they use across the network.

Physical AI refers to AI systems that perceive the physical environment through sensors, perform AI inference, and act on the physical world through actuators. Examples include service and industrial robots, drones, industrial automation systems, and autonomous vehicles; the self-driving vehicle use case in is one instance of Physical AI. Unlike conventional AI applications, Physical AI continuously collects multimodal sensor data from physical devices and generates control decisions that directly affect physical systems. Therefore, the performance and reliability of Physical AI services depend not only on the capabilities of AI models but also on the availability, allocation, and coordination of the underlying communication and computing resources. A Physical AI service may consist of multiple distributed components, including sensors, end devices, edge computing platforms, AI accelerators, cloud infrastructures, and actuators. These components cooperate to execute a closed loop of sensor data acquisition, data processing, AI inference, decision generation, and control command delivery, as shown in . To meet application-specific timing requirements, the supporting infrastructure needs to provide predictable communication and computing performance across the entire execution path of this loop, not only on its individual segments. In particular, end-to-end latency, latency variation (jitter), packet loss, and computing execution time need to be considered together when determining whether the required service performance can be achieved. The metrics described in and the in-time and on-time delay sensitivity described in therefore apply to the loop as a whole.

+----------------------+ +------------------------+ | Physical device | Sensor data | Edge device | | (robot, drone, etc.) | (video, LiDAR, | | | | measurements) | +----------------+ | | +--------------+ |--------------->| | AI inference | | | | Sensors | | | | and decision | | | +--------------+ | | +----------------+ | | +--------------+ | Control | | | | Actuators | | commands | | | +--------------+ |<---------------| | +----------------------+ +-----------+------------+ | Model updates, | Telemetry, policies | non-time-critical | tasks | +-----------+------------+ | Cloud server | | (training, planning, | | global coordination) | +------------------------+ The communication network supporting a Physical AI service carries heterogeneous traffic within the same service, including high-resolution video streams, LiDAR data, sensor measurements, control messages, telemetry, and AI inference requests. These traffic types have different requirements in terms of bandwidth, latency, reliability, and priority. Traffic classification and differentiated forwarding, as in the Differentiated Services architecture , QoS-aware resource allocation, and scheduling mechanisms can be used to satisfy these requirements. For latency-sensitive and safety-critical operations, such as the delivery of control commands, resource reservation, admission control, and traffic prioritization can be used to maintain predictable communication performance under varying network conditions, for example by using the deterministic networking capabilities described in . In addition to communication resources, Physical AI services require distributed computing resources, such as CPUs, GPUs, NPUs, memory, storage, and other AI accelerators deployed across end devices, edge platforms, and cloud infrastructures, that can execute data processing and AI inference within application-specific time constraints. Depending on the computational complexity of AI workloads and their latency requirements, AI functions can be executed locally or distributed across multiple computing environments, as described in . Since cloud-based execution may introduce additional communication delay and variability, latency-sensitive functions, such as perception and control in the closed loop, are preferably executed close to the physical devices, while functions with looser timing constraints, such as model training, task planning, and global coordination, can be executed in the cloud. The computing infrastructure therefore needs to support workload placement, execution scheduling, and dynamic resource allocation based on computing requirements, resource availability, and network conditions, for which the CATS framework is applicable (). When AI agents are used for task planning in Physical AI, the timescale separation described in applies, so that the reasoning of the agent is kept outside the closed control loop.

This document has no IANA actions.

Distributing AI services across local machines, edge devices, and cloud servers increases the attack surface compared to a centralized deployment. Even an AI service performed entirely on a local machine is not free from security issues: the model and its inputs can be manipulated, and the machine itself can be compromised. The following threats need to be considered: Exposure of AI service endpoints: when AI services are provided through cloud servers or edge devices, their endpoints are reachable from the network and can be targets of unauthorized access and denial-of-service attacks. Inference requests can be computationally expensive, so resource exhaustion attacks can be effective even at low request rates. Authentication, authorization, and rate limiting of AI service requests are needed. Confidentiality and integrity of offloaded data: data offloaded from local machines to edge devices or cloud servers (e.g., camera images from vehicles, or network telemetry) can be sensitive. It needs to be protected in transit (e.g., by TLS or QUIC ) and at the execution point. Attacks on AI models: adversarial inputs can cause incorrect inference results, and repeated queries can be used to extract a model or to infer information about its training data. Models deployed on edge devices, which may be physically accessible, are particularly exposed to theft and tampering. Integrity of distributed models: models and their updates distributed to many inference points () can be tampered with in transit or at rest. Inference points need to verify the integrity and origin of models (e.g., by digital signatures) before using them. Attacks on distributed learning: in federated learning (, ), a compromised participant can send poisoned updates that degrade or bias the global model. Robust aggregation and the authentication of participants are needed. Risks specific to Agentic AI: AI agents that invoke tools and act on the network introduce additional risks. Content obtained through tools (e.g., telemetry, logs, or documents) can contain instructions crafted to manipulate the agent (prompt injection), so such content needs to be treated as data, not as instructions. Agents need to be identified and authenticated, and granted only the minimum privileges necessary for their tasks. Actions with a significant impact on the network need to be validated, logged for auditing, and, where appropriate, approved by operators (). In multi-agent systems, messages from other agents need to be authenticated, and the impact of a compromised agent needs to be contained. Safety of Physical AI: in Physical AI services (), incorrect inference results, manipulated sensor inputs, or delayed or lost control commands can directly cause physical harm. In addition to protecting sensor data and control messages, physical devices need to transition to a safe state when control commands are not received in time or when communication with the edge device or cloud server is lost. AI inference requests often carry personal or confidential data, such as images of people and vehicles, location information, user prompts, or operator network data, and where AI inference is executed determines which parties can access this data. Executing inference on the local machine or on an edge device within the same administrative or data domain can prevent such data from leaving that domain, as discussed for the second high-level challenge in . When data needs to be offloaded to another domain, data minimization, such as sending extracted features instead of raw data or anonymizing data, needs to be considered. The context and memory maintained by AI agents, the caches maintained at edge devices, and the telemetry collected to support offloading decisions may also contain personal or confidential data, such as location and mobility patterns, and their retention and access need to be controlled. When providing AI services by utilizing machines in the network such as cloud servers and edge devices, it is necessary to analyze the characteristics of the modules to be used, identify their security vulnerabilities, and take countermeasures.

Experts from the IETF/IRTF NMRG provided valuable input during the development of this document.