Below is an example figure that succinctly describes the complete Codezero runtime with regard to software components and multiple cores:
Provided below are sections that divide this runtime into parts, to describe the microkernel architecture and its embedded hypervisor capabilities in detail.
Codezero is a new L4 microkernel that has been written from scratch, following the latest development and research principles on microkernel design. Codezero aims to carry the L4 microkernel architecture to the next level by actively evolving and through the support of its open community.
The Codezero and L4 line of microkernels are founded on a few fundamental design principles. The primary principle is that only the most fundamental and abstract software mechanisms are incorporated into the microkernel, ruling out any policy from the implementation. Codezero implements only the mechanisms to manage threads, address spaces, and the communication mechanisms between them.
In relation to its main founding principle, the microkernel becomes simple, abstract, and flexible. Due to its abstract nature, it may be used for multiple independent purposes, such as a Hardware Abstraction Layer, a Virtualization Platform, or as a basis for implementing new operating systems. By its simple and abstract design, L4 has a distinguished position among other real-time executives.
The microkernel is the only component that runs in privileged CPU mode. Therefore it is the central point of trust on the platform, responsible for the overall security and stable operation of the system. The microkernel is kept rigorously small, therefore making the system secure and stable.
Since the microkernel has system-wide control, the division of components and resource partitioning are also managed by the microkernel. In this respect Codezero implements the notion of Capabilities to protect and safely multiplex all resources to its run-time components.
Codezero sources are a mere ~10K lines of C code, and its API consists of twelve main system calls that can be grasped within a few days. The size and simplicity of the kernel also reflects positively on its performance.
General technical features of Codezero are listed below.
Codezero's system calls provide the following mechanisms on an embedded system:
Generally, Codezero has been designed from the start to incorporate the necessary infrastructure for real-time performance.
Codezero implements a 1-to-1 thread model where each thread in the system is known and controlled by the microkernel. As a result, every thread on the system is scheduled by the microkernel scheduler.
Codezero supports multicore. Particularly support for ARM Cortex-A9 has been added and tested on a quad core SMP platform. Codezero transparently schedules threads to multiple cores. There is no restriction on how the threads can be scheduled to cores, though such restrictions may be introduced if needed.
Codezero supports the conventional IPC methods provided by other L4 microkernels. There are three types of IPC defined, namely short, full, and extended IPC. The message transfer sizes increase in respective order. Extended IPC is particularly flexible such that it has no restrictions on the buffer address and allows page faults to happen.
Shared-memory communication is also provided for no-copy message transfers. Shared-memory access may be protected by high-performance userspace locks. Also, access to shared memory may be defined in ways that limit access to one or more parties during the lifetime of the shared area.
Please refer to the Communication section for further information on interprocess communication methods.
Codezero provides OS virtualization mechanisms throughout its design. It introduces the notion of Containers which are essentially virtual, isolated rooms of execution for running virtualized operating systems or baremetal applications. The isolation aspect of containers provide a plain partitioning mechanism.
System's overall security and resource management policy are further refined with the concept of capabilities. Codezero capabilities allow fine-grain allocation of resources and strictly define the operations that each software component is allowed to do. Please refer to Containers and capabilities sections for further details on these concepts.
Codezero provides the notion of containers for virtualization. Each container implements a conceptually isolated execution environment with its own set of resources. For example, each container has its own set of threads, address spaces, and memory resources. Initially, a container consists of a single privileged task called the pager. The pager produces further children tasks to produce a rich multitasking environment in its container's boundaries. Below is a figure illustrating two containers running on top of Codezero:
Each container provides enough mechanism for development of any software architecture:
The above figure has example containers for both of these scenarios. In Container 0 the pager has created multiple children tasks that run in their own virtual memory space. This is a typical setup for a virtualized operating system, where the guest kernel becomes the pager, and guest applications become the children tasks.
In Container 1 the pager itself has created many threads in its own address space. This typically corresponds to a baremetal, self-contained application.
It is worth noting that in the above scenarios, each pager has rights to its own container only. This allows for a simple and powerful way to manage isolation between containers.
Below is a code snippet that illustrates how each container is precisely represented inside the microkernel:
struct container { l4id_t cid; /* Unique container ID */ int npagers; /* # of pagers */ struct link list; /* List ref for containers */ struct address_space_list space_list; /* List of address spaces */ char name[CONFIG_CONTAINER_NAMESIZE]; /* Name of container */ struct ktcb_list ktcb_list; /* List of threads */ struct link pager_list; /* List of pagers */ struct id_pool *thread_id_pool; /* ID pools for thread/spaces */ struct id_pool *space_id_pool; struct mutex_queue_head mutex_queue_head; /* Userspace mutex list */ struct cap_list cap_list; /* Capabilities shared by whole container */ struct pager *pager; /* Boot-time array of pagers */ };
As illustrated by the above code excerpt, a container is a mere collection of resources. When each system call is issued, the microkernel refers to the relevant container and only manipulates the container in question.
Containers boldly partition the system into isolated environments. Capabilities build up on the notion of containers with fine-grain and flexible resource access management.
Codezero implements capability-based security for all kernel-managed resources. All system calls, expendable resources such as memory regions, memory pools, and interprocess communication mechanisms are protected by capabilities. Below is a figure that illustrates how each container has been controlled by capabilities:
As the above figure illustrates, there are two levels of capability possession in Codezero:
In the above scenario, the children tasks have access to a mutex pool capability, and they have been allowed to communicate via IPC for any thread inside their container.
Typically, the pager task possesses all privileged capabilities inside the container. For example, the pager is allowed to create and manipulate threads via its thread control and exchange registers capabilities. The pager may also consume kernel resources, such as thread and address space pools via these capabilities.
Finally, the pager has capabilities to a set of virtual and physical memory resources. By having possession to these resources, a pager may map memory to its own address space, as well as the address space of its children.
For every resource access, the microkernel is concerned whether the accessing party is in possession of a corresponding capability. This design makes the overall security architecture elegant and simple for multiple reasons:
The complete capability architecture in Codezero is implemented in less than 1K lines of C code. Capabilities are a key security aspect as they provide the fine-grain resource management privileges on top of the abstract notion of containers.
Please refer to the Capability section in the API reference manual chapter for further information on capabilities.
In Codezero, communication is provided in the form of IPC and shared memory. Most common method of IPC requires the notion of a User Thread Control Block as a message buffer, shortly referred as the UTCB. A UTCB is defined for every unique thread on the system. This block of memory can be used as thread-local storage as well as a buffer for interthread communication.
UTCB memory regions are optionally predefined in the virtual memory address space. Unlike other L4 microkernels, the task of allocating UTCBs to threads is left to the userspace pagers. This relieves the microkernel from extra policy.
On the ARM architecture, UTCBs are memory blocks that consist of 64 words (256 bytes).
IPC communication is done synchronously via a rendez-vous implementation in the kernel. Asynchronous message-passing methods are also possible. However, synchronous communication is encouraged for maintaining predictable software behavior. Particularly, synchronous IPC is considered to be a critical building block for deterministic concurrency on multithreaded systems.
Three types of IPC are provided on Codezero:
Short IPC
This is the most common method of IPC between userspace threads. On short IPC, two threads that communicate only transfer their primary message registers. The primary message registers consist of those registers which are capable of mapping onto real registers on the system.
For example on an ARM system, short IPC would cover only the primary message registers MR0-MR5, which would map onto R3-R8 on real hardware registers. It is strongly advised not to rely on the register mappings as this is a notion that is transparently managed by the microkernel.
Full IPC
In Full IPC, the complete UTCB buffer is transferred among threads. In the ARM case, this would correspond to 64 machine words (256 bytes).
Extended (long) IPC
Extended IPC is the most flexible method of IPC on Codezero. Using extended IPC, a thread may transfer a message of up to 2 Kilobytes, and the message buffer may be located at any address on the caller's address space. The buffer may freely page fault, as any page faults would be handled by the pager during IPC. The downside of using extended IPC is that since it may page fault, there is an aspect of non-determinism. Also, it may be slightly slower due to extra copying that it requires.
Communication of larger-sized buffers are encouraged to be implemented via shared-page mappings.
Below figure illustrates possible IPC paths that can take place in a Codezero system:
As seen in the figure, each child task in a container may communicate with their pager via IPC. Children tasks may also communicate between each other. Furthermore, if appropriate capabilities are defined, each child task on a container may communicate with tasks in other containers. The type and allowed domain of communication is dependent on whether the relevant capabilities are possessed by the communicator. For example, if a container-wide capability has been defined for container 0 that targets container 1, all tasks in container 0 may send messages to container 1. Similarly, communication may be restricted by making them possessed by each task, and/or targeting only one task.
Shared memory
Similar to IPC, the extent of shared-memory communication between tasks and containers are dependent on the possession of relevant physical and virtual memory capabilities. For example, two pagers that reside in different containers may have capabilities to the same physical memory region. As a result, they may map the same physical area allowing them to communicate.
Virtualization of operating system kernels are typically based on para-virtualization methodology on Codezero. In para-virtualization, an operating system kernel is abstracted away from the hardware details by manual inspection of source code. Para-virtualization is the most powerful method of virtualization since there is a dedicated engineering effort involved in the process. As such, the virtualized kernel runs with high performance, and yet with a maintainable software layer for future enhancements.
In the Codezero runtime environment, a para-virtualized OS kernel typically takes the place of the pager in a container. The kernel runs as a userspace component with capabilities to create applications. The kernel is limited in its access by the boundaries of the container that it is running inside. The figure below illustrates how a para-virtualized kernel takes place in the Codezero runtime environment:
In the example figure, the Linux kernel is running as a privileged pager inside its own container. In this setup, the guest kernel and applications typically run in user mode. Codezero runtime implements a method where the pager representing the guest kernel can switch between guest applications and their kernel space efficiently, named as the HyperSwitch methodology.
The Linux kernel and its applications have control over hardware resources that are allocated to them by their container definition. Codezero embedded hypervisor implements threading for the Linux kernel and has complete awareness of every Linux application running inside the container.
As previously mentioned, paravirtualization typically involves modification of guest kernel operations and inserting of replacement calls into the hypervisor. One of the other significant challenges in paravirtualization is the way guest kernel services are provided to applications in the virtualized environment. Often, new virtualization constructs and extra privileged threads are introduced inside the guest kernel space, in order to provide a mechanism to make applications dispatch and conclude system calls and exceptions. An example mechanism is shown below:
In this paravirtualization methodology, guest application system calls are converted to an ipc mechanism inside the hypervisor. The system call ipc requests are then redirected to dedicated service threads inside the guest kernel. Similarly real irqs are redirected to dedicated irq threads inside the guest kernel space. There are multiple problems with this approach. Introduction of new virtualization threads creates extra overhead on the system. Furthermore, the original method that the cpu handles these operations are modified in a fundamental way. This results in further requirements of changes in the system. As an example, a dedicated timer irq thread would now have to figure out which guest process was running before the interrupt, in order to do process accounting on that thread.
Codezero implements a different approach to the virtualization problem which have ultimately eliminated all of the artifacts introduced above. The new method named as HyperSwitch may be seen as below.
In HyperSwitch methodology, the existing method of cpu exception handling is preserved. In the same way that a Linux application thread switches between its user mode and kernel mode, Codezero enables a user application thread to switch to its kernel context in user mode and continue its execution there.
Harnessing the HyperSwitch methodology, Codezero preserves the existing infrastructure built inside the guest kernel for handling privileged services. As a result, guest modification requirements are significantly reduced (approximately 900 lines of arch-specific C code for the 2.6.34 Linux kernel). Furthermore, native kernel and application development practices such as GDB debugging and kernel patching may take place in identical fashion in the guest environment.
Codezero supports multicore SMP and optionally AMP designs. Particularly support for the ARMv7/Cortex-A9 cpus have been added and demonstrated on a quad-core SMP design.
Codezero transparently schedules threads onto cores and imposes no restriction on which thread is running on which core. For example, the formerly mentioned concept of containers are not obliged to run on any one core each. This allows for flexible and fully transparent utilization of cores.
It is possible, however, that certain applications are tied to certain cores, for performance and real-time purposes. For example, IRQ handling threads may be tied to one core this way. Also, the CPU timeslice of each thread may be adjusted if needed.
The Codezero runtime consists of two main components: the microkernel and containers.
The microkernel is the only privileged component in the Codezero runtime. It is considered as the single point of trust, namely the trusted computing base. It exposes a minimal system call API to userspace containers. It is also responsible for security checking of resource accesses issued by each container, through the use of its system call API.
Containers are independent userspace projects that may consist of a single baremetal application, or a full-blown virtualized operating system with many applications. There may be any number of containers in a Codezero build, and each container may be developed independent of others.
The build system consists of the SCons build tool and a set of Python scripts. SCons is a Python-based build tool that has similar functionality to Make. The main difference is that build rules are written using Python functions.
All container-build scripts are located under the scripts/ directory. The microkernel is built by the top-level SConstruct file and with help of many SConscript files spread across the kernel build directories. The build system may be invoked via the ./build.py script at the top-level directory.
The Codezero project is configured using the Python-based CML2 kernel configuration system. CML2 allows a menu-based configuration of build parameters before building. Typically, configuration data produced by CML2 are used for configuring the microkernel, as well as kernel-related parameters of each container included in the build. In essence, the CML2 configuration is used for a complete userspace and microkernel build.
The source code for CML2 itself lies under the tools/cml2-tools directory. Codezero project configuration data and scripts reside under scripts/config directory.
CML2 uses configuration files with a .cml extension for configuration. Various predefined configuration files reside under the scripts/config/examples/cml directory.
For reference to CML2 syntax and internals, please refer to the CML2 Language Reference Manual.
The loader directory contains sources for an ELF library and the boot loader. The boot loader is a simple ELF program that is responsible for scatter-loading the kernel and container images.
The loader is the final project built during a Codezero system build. After all containers and the kernel are built, the loader project bundles all resulting ELF files under a single final.elf image, which also contains the elf loader program.
The loader/libs directory contains two libraries called “C” and “elf”. “elf” is the library used for loading the ELF files during boot. The C library is a partial C library that is used by the bootloader during boot.
This directory contains various tools and scripts that are used as part of the project. The tools/tagsgen directory contains scripts to produce ctags and cscope index data for the microkernel and any other project included in the build. For instance, ./tools/tagsgen/tagsgen_kernel.py may be used to create a kernel index. Similarly, ./tools/tagsgen/tagsgen_linux.py may be used for creating an index for the Linux kernel. These scripts assume an existing architecture and platform configuration in order to filter out only the relevant architecture and platform directories for an index generation. tools/pyelf is a host-side Python library for inspecting ELF elements of a file inside the host. For instance, it is possible to extract fields from ELF headers or make calculations using this library. The tools/run-qemu-insight script is an autogenerated script that allows the final.elf image to be instantiated on QEMU and Insight/GDB after a build.
This directory includes various self-contained documents about peculiarities of the Codezero runtime. The docs/man directory includes reference manual pages for each Codezero microkernel system call. They may be invoked by: % man -M docs/man <man_page_name>
In the future, more ad-hoc documentation will be added here.
These directories contain the main sources for the Codezero microkernel.
conts directory contains all the userspace sources.
Contains example userspace servers and applications, e.g., Hello World and a template for user space development.
This directory is used for building Virtualized Linux on top of Codezero. It does not exist in original Codezero sources but should be created if Linux virtualization functionality is going to be used. The typical instantiation of this directory includes copying Linux environment files from a separate repository, which includes the Linux filesystem and boot parameters. Finally, the virtualized Linux kernel should be copied or cloned from a separate kernel repository into this directory. The contents of this directory must be set up in order to initiate a Linux build via the Codezero build system.
This directory is used for building virtualized u-boot instances on top of Codezero. It does not exist in original Codezero sources, but should be created if u-boot functionality is going to be used at runtime. The typical instantiation of this directory includes copying or git-cloning virtualized u-boot sources from a separate repository into this directory, before initiating a u-boot build via the Codezero build system.
This directory contains all userspace libraries that are available for any userspace application. These libraries consist of libl4, libc, libmem, and libdev.
libl4 is the most fundamental library provided for userspace. This library presents raw Codezero system calls in a more user-friendly format. It is appropriate to link with this library from every project, including virtualized operating system kernels. It is recommended that system calls to Codezero are issued via this library.
libc is the userspace C runtime library. It is particularly useful for referring to C library functionality that does not require an operating system. For example, string and basic IO functionality is included.
libdev is a helper library for clcd, timer, and UART devices for different platforms. Device services that reside under conts/baremetal make use of this library.
libmem contains three different memory allocators. A fixed-size memory cache implementation, a kmalloc implementation, and a page allocator implementation for page-size granularity memory allocation. Out of the three, memcache is the most frequently used because it provides a simple and deterministic method of memory allocation.
The POSIX sources consist of a partially complete POSIX implementation on top of Codezero. Advanced capabilities such as demand paging, memory, and process management are provided. As of this writing, this is an experimental container for investigating advanced pager behavior on top of the microkernel.
As described in the Getting Started guide, the standard debugging method on Codezero is based on Insight/GDB debugger connected to a QEMU emulator instance. This setup provides an experience that is very close to an In-Circuit-Emulator debugging of a real hardware platform.
Using Insight, any debug capability may be leveraged such as placing breakpoints, creating watchpoints, inspecting local variables, registers, function call stack, and memory.
Emulation environment can be easily set up on any Linux-hosted PC, and debug-fix-reboot cycle is several times faster than a real platform. Therefore, emulation environment is recommended for general software development. It is also recommended that the QEMU instance used for emulation is backed by a real hardware environment, and that software is checked for real hardware boot up on regular intervals.
For the purpose of debugging Codezero, the emulation tools have been optimized to provide enhanced debugging capabilities. QEMU has been modified to support the Versatile Express quad-core Cortex-A9 platform. Virtualized Linux kernel environment has been optimized so that Linux boots can be instantiated on the debugger within a few seconds.
Before starting with debugging and development of source code, it is recommended that below sections are read for an understanding of how individual components of a typical Codezero build are laid out in the final executable image, physical memory, and virtual memory spaces.
The typical Codezero build involves creation of multiple ELF images that are bundled together in a cascaded fashion. All images are standard ELF files that can be manipulated and inspected using standard GNU ELF tools such as readelf, objdump.
As formerly mentioned, the typical Codezero runtime consists of the microkernel and multiple containers. After all these components are built, the ELF loader build is initiated. ELF loader is built into the final.elf image, with every container image and the kernel image embedded inside. At runtime, the loader would traverse its own sections to find each ELF image and load them to their respective physical memory addresses.
Below is a summary of how images are cascaded inside final.elf:
final.elf: Section cont.0 (Container0.elf) Section cont.1 (Container1.elf) ... Section cont.X (ContainerX.elf) Section .kernel (Kernel.elf)
where each containerX.elf image is further cascaded as follows:
container0.elf: Section img.0 (image0.elf) Section img.1 (image_XYZ.elf) ... Section img.X (imageX.elf)
When building each container project, note that the builder picks up any file with a .elf extension from the project directories and places them in a .img.X section for that container. This allows support for containers with multiple ELF images.
During boot-time, the ELF loader unpacks each .cont.X section, and further unpacks and loads the ELF files contained in .img.X sections. Finally, it loads the kernel.elf from the .kernel section. It is essential that every image has been linked to be loaded at a nonclashing physical address with respect to other images. The build system automatically ensures this during the configuration phase.
During boot up, the ELF loader placed into the final.elf image starts executing. It loads each image according to its defined physical memory locations.
A container image may be loaded at any possible physical memory address during load time. The load information is is obtained from its LMA (logical memory address) value defined in its linker script. It is essential that this LMA matches with any one of the physical memory regions (PHYSMEM capabilities) defined for the corresponding container. Otherwise, the kernel would refuse to execute the image. It is also worth noting that load addresses of images must not overlap. Generally parameters such as load address and physical memory regions are defined during kernel container configuration. The configuration process automatically ensures that the setup is sane.
Even though the location of container images in physical memory is highly configurable, below is an example setup for physical memory layout of images:
Physical memory layout on Codezero/ARM Versatile PB926 ====================================================== 0x8000 0000 .--------------------------. End of physical memory | Rest of Physmem | |--------------------------| | Loader | <Loader LMA> |--------------------------| | Container 2, Image 0 | <Cont2/Img.0 LMA> |--------------------------| 16KB alignment | Container 1, Image 0 | <Cont1/Img.0 LMA> |--------------------------| 16KB alignment | Container 0, Image 1 | <Cont0/Img.1 LMA> |--------------------------| 16KB alignment | Container 0, Image 0 | <Cont0/Img.0 LMA> |--------------------------| | | |--------------------------| | Codezero | 0x0000 8000 |--------------------------| | Reserved | 0x0000 0000 '--------------------------' Start of physical memory
In the above example, there are three containers defined where container 0 has two images, container 1 and container 2 have one image each. It is expected that each LMA address is within the physical memory regions defined for its respective container. It is also worth noting that the loader image itself is placed at a higher physical memory address that gets calculated during build so that it is placed at an address that will not overlap with any one of the container images.
After all images are loaded to their physical memory regions, the ELF loader starts running the Codezero microkernel. Codezero enables virtual memory and starts running at the fixed virtual address of 0xF0000000.
After kernel initialization, knowing LMA (logical memory address), VMA (virtual memory address), and total image size for each pager, the microkernel maps the pager for each container to its runtime virtual address. Similar to a pager's LMA, the VMA value must lie inside one of the virtual memory capabilities defined for the pager. Otherwise, this would be a permission violation, and Codezero would refuse to run the pager for that container.
Unlike the physical LMA regions, it is possible that a virtual memory region may be allocated to more than one pager. Since each pager has a different set of page tables, it may be a valid setup that they run in the same virtual memory address. This is discouraged, however, as there may be implications with regard to caching, in case any two such pager wishes to communicate via shared memory or IPC.
Even though the virtual memory layout of a Codezero runtime is highly dependent on the configuration, below is a fictional example for the virtual memory layout of containers that were listed earlier:
Runtime virtual-memory layout of a Codezero/ARM system: ========================================================= 0xFFFF FFFF .---------------. End of virtual memory | Syscall page | 0xFFFF F000 |---------------| | Reserved | 0xFFFF 1000 |---------------| | Vector page | 0xFFFF 0000 |---------------| | Reserved | 0xFF00 1000 |---------------| | KIP | 0xFF00 0000 |---------------| | Kernel IO | 0xF900 0000 |---------------| UTCB area ends (optional) | | | ... | | --------- | | UTCB page | | --------- | | UTCB page | 0xF800 0000 |---------------| UTCB area starts (optional) | | | Codezero | | Microkernel | | | 0xF000 0000 |---------------| Container 2 Task/Pager area ends | | | Container 2 | | Pager/Tasks | | | CONT2_VIRTMEM0 |---------------| Container 2 Task/Pager area starts | | | Container 1 | | Pager | | | CONT1_VIRTMEM1 |---------------| Container 1 User task area ends | | | | | Container 1 | | Task | | Address Space | | | | | CONT1_VIRTMEM0 |---------------| Container 0 Pager area ends | | | Container 0 | | Pager | | | CONT0_VIRTMEM1 |---------------| Container 0 User task area ends | | | | | Container 0 | | Task | | Address Space | | | | | CONT0_VIRTMEM0 |---------------| Container 0 User task area starts | Empty | 0x0000 8000 |---------------| | Reserved | 0x0 '---------------' Start of virtual memory
As seen from the figure above, the containers have a highly configurable virtual runtime environment. Each container may be placed at a different portion of virtual memory. In case the container consists of a virtualized OS kernel and its applications, they may be optionally placed in different or same virtual regions. The configuration of virtual spaces are ultimately determined by the kernel configuration system before the build phase.
In the ARM architecture, there are a few fixed virtual address regions defined. The system call page is placed at the highest virtual memory page. The exception vectors are placed at the architecture-defined high vector address of 0xFFFF0000. The kernel accesses very few devices during its lifetime, such as the timer and UART devices on the platform. To access any such device region with flexibility, a kernel IO region has been defined at 0xF800000. Finally, the lowest few pages of virtual memory space have been reserved, in order to catch null pointer dereference errors and avoid confusion.
One notable feature of the Codezero environment is the notion of UTCBs. UTCBs denote the per-thread storage spaces that are used during an IPC. For performance and simplicity reasons, it is a strict requirement that any UTCB allocated to a thread does not overlap with another UTCB in the virtual address space. This ensures that the kernel does not need to remap UTCB regions or issue other complex cache maintenance operations during the course of the IPC.
For simplicity reasons, Codezero does not maintain UTCB address regions inside the microkernel. In the above example, an optional UTCB region has been allocated between 0xF8000000 - 0xF9000000 so that pagers may possess and allocate virtual areas from this region for their children threads or themselves. Note, that this is only a suggestive allocation, as any address in the virtual address space may be used for this purpose.
This section provides a brief introduction to debugging userspace applications and the Codezero microkernel in general.
After successfully building all containers, the microkernel and the loader final.elf image is ready for running under the build/ directory. In an emulation environment, this can be invoked by:
% ./tools/run-qemu-insight
In a real hardware environment, final.elf image may be loaded to RAM via a remote debugger that has capabilities to load ELF files, such as RVD.
Any one of container images or the microkernel may be debugged while running the system.
In order to load symbols of the desired debug executable, the relevant symbols must be loaded. To debug the microkernel as an example, the following GDB command must be executed in the GDB console:
% sym /opt/codezero/build/kernel.elf
In order to debug a baremetal application, its symbols must be loaded:
% sym /opt/codezero/build/cont0/baremetal_app/app.elf
In order to debug the virtualized linux kernel, linux symbols must be loaded:
% sym /opt/codezero/build/cont0/linux0/kernel-2.6.34/vmlinux
The above commands may be placed into the .gdbinit file in the current user home directory. This would allow the user to start debugging at a particular state of execution in automated fashion. Using such a debug script may significantly speed up startup times of a debug session. As an example, the following .gdbinit file is used for debugging the Linux kernel:
target remote localhost:1234 load final.elf sym final.elf break break_virtual continue sym kernel.elf stepi set $var = 0xa0008000 break *$var continue sym cont0/linux/kernel-2.6.34/vmlinux stepi
where the initial head.S entry point of the Linux kernel is located at virtual address 0xa0008000 in this occasion.
A notable point to keep in mind here is that the debugger catches breakpoints on an executable by matching the symbol addresses loaded to the real object code executing at the particular moment. In that respect, there may be ambiguities as to which executable is being debugged if two executables are running at the same virtual address range.
As an example, if two applications are running at virtual address 0×1000 0000 and a breakpoint is set at an address in one of them, the debugger may naturally fall into the fallacy of breaking on the other task, if it hits the same address. On disjoint virtual memory addresses (e.g., the microkernel virtual address range) no problem occurs.
A frequently needed operation is to set a breakpoint at an address where there is no symbol defined. GDB does not allow setting a breakpoint at a raw address this way. As a workaround, the following GDB convention helps:
% set $var = [hex address to break] % break *$var