Core (Part2): Generating Core + Debugging Core

Implementation Goal: tinydbg core [corefile]

In this section, we introduce debugging based on core files using tinydbg core [corefile]. Typically, a core file is a memory snapshot generated by the operating system when a program terminates abnormally or crashes. It contains information about the program's state at the time of crash, which the debugger can use to reconstruct the execution context and help developers locate issues.

One of the most common operations when using core files for problem diagnosis is executing the bt command to locate the program's stack at the time of crash. This is particularly useful for identifying SEGMENTATION FAULT issues. Most mainstream programming languages now provide stack trace capabilities when exceptions or serious errors occur, making it easier for developers to examine the problem stack.

For example:

  1. Go language supports both panic recovery and stack trace printing through debug.Stack(); setting GOTRACKBACK=crash environment variable can generate core files on crashes;
  2. Java language can print current thread stack information through Thread.dumpStack() or Throwable.printStackTrace(); JVM generates hs_err_pid*.log files to record crash information;
  3. C++ can obtain stack information through functions like backtrace() and backtrace_symbols(); core dumps can be enabled by setting ulimit -c unlimited, generating core files on program crashes;

Core files are essentially snapshots of a process at a particular moment, and they don't necessarily have to be generated only during crashes. For example, gcore <pid> can generate a core file without terminating the process. Of course, this is typically done when trying to diagnose process issues, and for online services, traffic should be diverted before doing this since the process is paused during core file generation.

Basic Knowledge

What Information Does Core Contain

Part 1 provided a detailed introduction to core files, but let's briefly review here. A core file is a memory snapshot of a process that contains the program's memory content and register state at the time of crash. It mainly consists of the following parts:

  1. ELF Header Information: Identifies this as a core file, containing basic information like file type and machine architecture
  2. Program Header Table: Describes the location and attributes of various segments in the core file
  3. Memory Mapping Segments:
    • Contains program code segments, data segments, heap, stack, and other memory regions
    • Each segment has corresponding virtual address and access permission information
  4. Register State:
    • General register values for all threads
    • Floating-point register state
    • Special register state
  5. Other Information:
    • Process information like process ID, user ID
    • Signal information that caused the crash
    • Command line arguments and environment variables
    • Open file descriptor information

The debugger can read this information from the core file to reconstruct the program's execution context at the time of crash, helping developers perform post-mortem debugging analysis and problem review.

How Core Files Are Generated

Core File Generation in Linux

Generated by Linux Kernel

When a program receives certain specific signals (such as SIGSEGV, SIGABRT, etc.), if the system has core dump functionality enabled, the kernel will help generate a core file. The specific process is as follows:

  1. Common signals that trigger core dumps:

    • SIGSEGV: Segmentation fault, illegal memory access
    • SIGABRT: Called abort() function
    • SIGFPE: Floating point exception
    • SIGILL: Illegal instruction
    • SIGBUS: Bus error
    • SIGQUIT: User sent quit signal

  2. System Configuration:

    # Check if core dump is enabled
ulimit -c

# Set core file size limit (unlimited means no limit)
ulimit -c unlimited

# Configure core file path format
echo "/tmp/core-%e-%p-%t" > /proc/sys/kernel/core_pattern

  3. Kernel Processing Flow:

    1. When the process receives the above signals, the kernel intervenes
    2. Checks if system core dump configuration allows core file generation
    3. Kernel pauses all threads of the process
    4. Collects process memory mappings, register states, and other information
    5. Writes information to core file
    6. Terminates the process

  4. Core File Naming Rules (/proc/sys/kernel/core_pattern):

    • %p - Process ID
    • %u - User ID
    • %g - Group ID
    • %s - Signal number that caused core dump
    • %t - Core dump time (UNIX timestamp)
    • %h - Hostname
    • %e - Executable filename

So generating core files doesn't require debugger participation - this is an important feature provided by the Linux kernel. The debugger's role is to analyze this core file afterward, reconstructing the crash scene for debugging.

Generated by Custom Tools

Besides the above methods of sending signals to processes and utilizing the kernel's capabilities to automatically generate core files, our custom debugging tools can also implement core file dumping capabilities.

For example, gcore in the gdb software package can also generate core files for processes without actually terminating them. Although most of the time core files are generated for online services when they encounter serious errors, we can actually generate core files without killing the process, and the implementation is not complicated.

For example, if we want to generate a core file for a certain process, we can do this:

  • Use the ptrace system call to attach to the target process;
  • Read /proc/<pid>/maps to understand memory layout;
  • Use process_vm_readv() or ptrace(PTRACE_PEEKDATA, ...) to read memory regions;
  • Use ptrace(PTRACE_GETREGS, ...) to capture register state;
  • Get open files, thread information, etc.;
  • Get startup environment variables, startup parameters, build parameters, etc.;
  • ...
  • Organize the above information of interest into the core file format and write it.

OK, let's see how tinydbg generates core files and loads core files.

Code Implementation

Core file generation is actually implemented through the debug session command tinydbg> dump <corefile>, while loading core files and starting debugging is implemented through tinydbg core <executable> <corefile>. According to our directory arrangement, in this section we should first introduce the core command, then the debug session commands, and finally the dump command. However, the production and consumption of core file data are closely related, and having production and consumption separated by many chapters would make it difficult for readers to understand and learn.

Therefore, we'll first introduce how the dump command implements core file generation, and then discuss core file consumption.

tinydbg Generating Core Files

$ (tinydbg) help dump
Creates a core dump from the current process state

        dump <output file>

The core dump is always written in ELF, even on systems (windows, macOS) where this is not customary. For environments other than linux/amd64 threads and registers are dumped in a format that only Delve can read back.

Core code path for generating core files:

debug_other.go:debugCmd.cmdFn(...)
    \--> dump(s *Session, ctx callContext, args string)
            \--> dumpState, err := t.client.CoreDumpStart(args)
                    \--> c.call("DumpStart", DumpStartIn{Destination: dest}, out)
            \--> forloop
                    \--> print dumping progress
                    \--> if !dumpState.Dumping { break }
                    \--> else { 
                            dumpState = t.client.CoreDumpWait(1000)}
                                \--> c.call("DumpWait", DumpWaitIn{Wait: msec}, out)
                         }

For the debugger backend, the code path is:

tinydbg/service/rpc2.(*RPCServer).DumpStart(arg DumpStartIn, out *DumpStartOut)
    \--> s.debugger.DumpStart(arg.Destination)
            \--> (d *Debugger) DumpStart(dest string) error {
                    \--> (t *Target) Dump(out elfwriter.WriteCloserSeeker, flags DumpFlags, state *DumpState) 
                            \--> 1. dump os/machine/abi... info as file header
                            \--> 2. t.dumpMemory(state, w, mme): write mapped memory data
                                    \--> update DumpState.MemoryDone, DumpState.MemoryTotal
                            \--> 3. prepare notes of dlv header, process, threads and other info
                                    \--> prepare note of dlv header: ... 
                                    \--> prepare note of process: t.proc.DumpProcessNotes(notes, state.threadDone)
                                    \--> for each thread:
                                            \--> t.dumpThreadNotes(notes, state, th)
                                            \--> update DumpState.ThreadsDone, DumpState.ThreadsTotal
                            \--> 4. w.WriteNotes(notes): dump dlv header, process info, threads info, and others as 
                                    a new PT_NOTE type entry of ProgHeader table 
    \--> out.State = *api.ConvertDumpState(s.debugger.DumpWait(0))
    \--> return DumpState to rpc2.Client

Looking at the specific source code implementation, it's clear that the process dumping might take some time and won't complete immediately. So after the client requests DumpStart, the server will first return a DumpState, which is the current state and may not be completely finished. If not finished, the client will request dumpState := t.client.CoreDumpWait(...) again every 1 second to get the dumping progress.

After reading the Dump implementation below, you'll understand how the dumping progress is calculated - there are just two metrics: whether thread information has been completely dumped, and whether memory information has been completely dumped. These two parts might take longer depending on the process workload.

// DumpStart starts a core dump to arg.Destination.
func (s *RPCServer) DumpStart(arg DumpStartIn, out *DumpStartOut) error {
    err := s.debugger.DumpStart(arg.Destination)
    if err != nil {
        return err
    }
    out.State = *api.ConvertDumpState(s.debugger.DumpWait(0))
    return nil
}

// ConvertDumpState converts proc.DumpState to api.DumpState.
func ConvertDumpState(dumpState *proc.DumpState) *DumpState {
    ...
    return &DumpState{
        Dumping:      dumpState.Dumping,
        AllDone:      dumpState.AllDone,
        ThreadsDone:  dumpState.ThreadsDone,
        ThreadsTotal: dumpState.ThreadsTotal,
        MemDone:      dumpState.MemDone,
        MemTotal:     dumpState.MemTotal,
    }
}

// DumpStart starts a core dump to dest.
func (d *Debugger) DumpStart(dest string) error {
    ...
    fh, err := os.Create(dest)
    ...
    d.dumpState.Dumping = true
    d.dumpState.AllDone = false
    d.dumpState.Canceled = false
    d.dumpState.DoneChan = make(chan struct{})
    d.dumpState.ThreadsDone = 0
    d.dumpState.ThreadsTotal = 0
    d.dumpState.MemDone = 0
    d.dumpState.MemTotal = 0
    d.dumpState.Err = nil

    go d.target.Selected.Dump(fh, 0, &d.dumpState)
    return nil
}

Here, selected actually refers to a Target in TargetGroup, and Target refers to the process dimension. For single-process programs, there's only one Target in TargetGroup. For multi-process programs, if tinydbg> target follow-exec [-on [regex]] [-off] is enabled in debug mode, when a child process is created and its command matches the regular expression, the newly created process will be automatically managed. In this case, TargetGroup will have more than one Target. Of course, the Target layer control backend implementation must support controlling parent-child processes. The native backend supports this, and for the gdb debugger, set follow-fork-mode child is also supported.

For multi-process debugging scenarios where you want to simultaneously pause and resume execution of parent and child processes, TargetGroup manages this uniformly, making it convenient to perform corresponding pause and resume operations.

ps: Regarding the extensible and replaceable backend issue: In our demo tinydbg, we only kept Delve's own native debugger implementation. We removed the implementation logic that supported gdb, lldb, mozilla rr, and other debugger backends. Note that the term backend here doesn't refer to the debugger server in a front-end/back-end separated architecture, but rather the part of the debugger server that controls the Target layer. When mixing Chinese and English, please pay attention to the specific meaning of terminology.

OK, let's continue looking at how Target.Dump(...) is implemented:

// Dump writes a core dump to out. State is updated as the core dump is written.
func (t *Target) Dump(out elfwriter.WriteCloserSeeker, flags DumpFlags, state *DumpState) {
    defer func() {
        state.Dumping = false
        close(state.DoneChan)
        ...
    }()

    bi := t.BinInfo()

    // 1. write the ELF corefile header
    var fhdr elf.FileHeader
    fhdr.Class = elf.ELFCLASS64
    fhdr.Data = elf.ELFDATA2LSB
    fhdr.Version = elf.EV_CURRENT
    fhdr.OSABI = elf.ELFOSABI_LINUX
    fhdr.Type = elf.ET_CORE
    fhdr.Machine = elf.EM_X86_64
    fhdr.Entry = 0
    w := elfwriter.New(out, &fhdr) 
    ...

    // prepare notes of dlv header, process, threads and others
    notes := []elfwriter.Note{}
    // - note of dlv header
    entryPoint, _ := t.EntryPoint()
    notes = append(notes, elfwriter.Note{
        Type: elfwriter.DelveHeaderNoteType,
        Name: "Delve Header",
        Data: []byte(fmt.Sprintf("%s/%s\n%s\n%s%d\n%s%#x\n", bi.GOOS, bi.Arch.Name, version.DelveVersion.String(), elfwriter.DelveHeaderTargetPidPrefix, t.pid, elfwriter.DelveHeaderEntryPointPrefix, entryPoint)),
    })

    // - notes of threads
    state.setThreadsTotal(len(threads))

    // note of process
    var threadsDone bool
    if flags&DumpPlatformIndependent == 0 {
        threadsDone, notes, _ = t.proc.DumpProcessNotes(notes, state.threadDone)
    }
    // notes of threads
    threads := t.ThreadList()
    if !threadsDone {
        for _, th := range threads {
            notes = t.dumpThreadNotes(notes, state, th)
            state.threadDone()
        }
    }

    // 2. write mapped memory data into corefile
    memmap, _ := t.proc.MemoryMap()
    memmapFilter := make([]MemoryMapEntry, 0, len(memmap))
    memtot := uint64(0)
    for i := range memmap {
        if mme := &memmap[i]; t.shouldDumpMemory(mme) {
            memmapFilter = append(memmapFilter, *mme)
            memtot += mme.Size
        }
    }
    state.setMemTotal(memtot)
    for i := range memmapFilter {
        mme := &memmapFilter[i]
        t.dumpMemory(state, w, mme)
    }

    // 3. write these notes into corefile as a new entry of 
    // ProgHeader table, with type `PT_NOTE`.
    notesProg := w.WriteNotes(notes)
    w.Progs = append(w.Progs, notesProg)
    w.WriteProgramHeaders()
    if w.Err != nil {
        state.setErr(fmt.Errorf("error writing to output file: %v", w.Err))
    }
    state.Mutex.Lock()
    state.AllDone = true
    state.Mutex.Unlock()
}

tinydbg Loading Core Files

Core code path for loading core files:

main.go:main.main
    \--> cmds.New(false).Execute()
            \--> coreCommand.Run()
                    \--> coreCmd(...)
                            \--> execute(0, []string{args[0]}, conf, args[1], debugger.ExecutingOther, args, buildFlags)
                                    \--> server := rpccommon.NewServer(...)
                                    \--> server.Run()
                                            \--> debugger, _ := debugger.New(...)
                                                if attach startup: debugger.Attach(...)
                                                elif core startup: core.OpenCore(...)
                                                else others debugger.Launch(...)

For tinydbg core, it uses the core.OpenCore(...) method.

// OpenCore will open the core file and return a *proc.TargetGroup.
// If the DWARF information cannot be found in the binary, Delve will look
// for external debug files in the directories passed in.
//
// note: we remove the support of reading separate dwarfdata.
func OpenCore(corePath, exePath string) (*proc.TargetGroup, error) {
    p, currentThread, err := readLinuxOrPlatformIndependentCore(corePath, exePath)
    if err != nil {
        return nil, err
    }

    if currentThread == nil {
        return nil, ErrNoThreads
    }

    grp, addTarget := proc.NewGroup(p, proc.NewTargetGroupConfig{
        DisableAsyncPreempt: false,
        CanDump:             false,
    })
    _, err = addTarget(p, p.pid, currentThread, exePath, proc.StopAttached, "")
    return grp, err
}

The core logic for reading the core file and reconstructing the problem scene is here:

// readLinuxOrPlatformIndependentCore reads a core file from corePath
// corresponding to the executable at exePath. For details on the Linux ELF
// core format, see:
// https://www.gabriel.urdhr.fr/2015/05/29/core-file/,
// https://uhlo.blogspot.com/2012/05/brief-look-into-core-dumps.html,
// elf_core_dump in https://elixir.bootlin.com/linux/v4.20.17/source/fs/binfmt_elf.c,
// and, if absolutely desperate, readelf.c from the binutils source.
func readLinuxOrPlatformIndependentCore(corePath, exePath string) (*process, proc.Thread, error) {

    // read notes
    coreFile, _ := elf.Open(corePath)
    machineType := coreFile.Machine
    notes, platformIndependentDelveCore, err := readNotes(coreFile, machineType)
    ...

    // read executable
    exe, _ := os.Open(exePath)
    exeELF, _ := elf.NewFile(exe)
    ...

    // 1. build memory
    memory := buildMemory(coreFile, exeELF, exe, notes)

    // 2. build process
    bi := proc.NewBinaryInfo("linux", "amd64")
    entryPoint := findEntryPoint(notes, bi.Arch.PtrSize()) // saved in dlv header in PT_NOTE segment

    p := &process{
        mem:         memory,
        Threads:     map[int]*thread{},
        entryPoint:  entryPoint,
        bi:          bi,
        breakpoints: proc.NewBreakpointMap(),
    }

    if platformIndependentDelveCore {
        currentThread, err := threadsFromDelveNotes(p, notes)
        return p, currentThread, err
    }

    currentThread := linuxThreadsFromNotes(p, notes, machineType)
    return p, currentThread, nil
}

The two most crucial steps here are establishing the memory scene and process state scene.

We haven't introduced note types in detail before:

// Note is a note from the PT_NOTE prog.
// Relevant types:
// - NT_FILE: File mapping information, e.g. program text mappings. Desc is a LinuxNTFile.
// - NT_PRPSINFO: Information about a process, including PID and signal. Desc is a LinuxPrPsInfo.
// - NT_PRSTATUS: Information about a thread, including base registers, state, etc. Desc is a LinuxPrStatus.
// - NT_FPREGSET (Not implemented): x87 floating point registers.
// - NT_X86_XSTATE: Other registers, including AVX and such.
type note struct {
    Type elf.NType
    Name string
    Desc interface{} // Decoded Desc from the
}

OK, let's look at buildMemory. This function mainly processes PT_NOTE and PT_LOAD types in two steps: 1) For program headers of type PT_NOTE, notes with type note.Type=_NT_FILE represent non-anonymous VMA region mappings of some files; When Linux generates core files, it includes these; tinydbg dumps all memory regions as PT_LOAD. 2) For program headers of type PT_LOAD, it mainly reads some data from the executable program;

func buildMemory(core, exeELF *elf.File, exe io.ReaderAt, notes []*note) proc.MemoryReader {
    memory := &SplicedMemory{}

    // tinydbg doesn't generate note.Type=NT_FILE notes information,
    //
    // - For Go programs, if the core file is generated by the kernel, it will include this, see linux `fill_files_notes`
    // - For tinydbg> debug my.core, this information is not generated
    //
    // Here we assume all file mappings come from exe, which is obviously incorrect, as shared libraries and other external files are not included
    // - 1) For read-only files, they usually aren't stored in the core file (to save space), so we need to read from external files
    //      The support here is insufficient!!!
    //      Because the readNote function only reads VMA.start/end/offsetByPage, but doesn't read the actual mapped filenames!
    //
    // - 2) For read-write files, the kernel usually dumps this data during core dump, so we should prioritize core file data,
    //      to avoid blindly reading external file data and causing overwrites
    //
    // For now, assume all file mappings are to the exe.
    for _, note := range notes {

        if note.Type == _NT_FILE {
            fileNote := note.Desc.(*linuxNTFile)
            for _, entry := range fileNote.entries {
                r := &offsetReaderAt{
                    // why? Because it assumes Go is mostly statically compiled, doesn't use shared libraries, and doesn't involve mmap files,
                    // so this is the basic case when the kernel generates coredump. This implementation can be optimized
                    reader: exe, 
                    offset: entry.Start - (entry.FileOfs * fileNote.PageSize),
                }
                memory.Add(r, entry.Start, entry.End-entry.Start)
            }
        }
    }

    // Load memory segments from exe and then from the core file,
    // allowing the corefile to overwrite previously loaded segments
    for _, elfFile := range []*elf.File{exeELF, core} {
        if elfFile == nil {
            continue
        }
        for _, prog := range elfFile.Progs {
            if prog.Type == elf.PT_LOAD {
                if prog.Filesz == 0 {
                    continue
                }
                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }
                memory.Add(r, prog.Vaddr, prog.Filesz)
            }
        }
    }
    return memory
}

Note that for notes of type NT_FILE, these are generated when the kernel creates core files. In tinydbg's dump-generated core files, everything is dumped as PT_LOAD type, dumping all mapped memory as PT_LOAD segments for simplicity. When the kernel creates them, it dumps non-anonymous mapped VMA file information as PT_NOTE, with note.Type=NT_FILE. Although the above code assumes all mapped files come from the executable is not entirely correct, even so, it won't affect debugging accuracy, because these notes only record the mapping relationship between VMAs and files, not the actual data - the data still needs to be looked at in the PT_LOAD type sections. Actually, the already read file contents are already in the process address space, and when the kernel generates core files, it records the location of mapped data in the core file, so we can know the contents of already mapped files... So although offsetReaderAt{reader: exe, ...} looks incorrect, if this data has already been dumped through PT_LOAD segments, there's no problem, as the data can be read.

However, some articles mention that for read-only PT_LOAD segments where FileSZ==0 && MemSZ != 0, and they are Non-Anonymous VMA regions, to get the data we need to read from external storage based on the filenames in the PT_NOTE table's mapped files. But since readNote processing explicitly ignores these filenames, I believe tinydbg debugging might encounter issues in certain scenarios. However, this isn't a problem we want to solve comprehensively in this section, just something to understand.

// readNote reads a single note from r, decoding the descriptor if possible.
func readNote(r io.ReadSeeker, machineType elf.Machine) (*note, error) {
    // Notes are laid out as described in the SysV ABI:
    // https://www.sco.com/developers/gabi/latest/ch5.pheader.html#note_section
    note := &note{}
    hdr := &elfNotesHdr{}

    err := binary.Read(r, binary.LittleEndian, hdr)
    note.Type = elf.NType(hdr.Type)

    name := make([]byte, hdr.Namesz)
    note.Name = string(name)
    desc := make([]byte, hdr.Descsz)

    descReader := bytes.NewReader(desc)
    switch note.Type {
    case elf.NT_PRSTATUS:
        note.Desc = &linuxPrStatusAMD64{}
    case elf.NT_PRPSINFO:
        note.Desc = &linuxPrPsInfo{}
        binary.Read(descReader, binary.LittleEndian, note.Desc)
    case _NT_FILE:
        // No good documentation reference, but the structure is
        // simply a header, including entry count, followed by that
        // many entries, and then the file name of each entry,
        // null-delimited. Not reading the names here.
        data := &linuxNTFile{}
        binary.Read(descReader, binary.LittleEndian, &data.linuxNTFileHdr)
        for i := 0; i < int(data.Count); i++ {
            entry := &linuxNTFileEntry{}
            binary.Read(descReader, binary.LittleEndian, entry)
            data.entries = append(data.entries, entry)
        }
        note.Desc = data
    case _NT_X86_XSTATE:
        if machineType == _EM_X86_64 {
            var fpregs amd64util.AMD64Xstate
            amd64util.AMD64XstateRead(desc, true, &fpregs, 0)
            note.Desc = &fpregs
        }
    case _NT_AUXV, elfwriter.DelveHeaderNoteType, elfwriter.DelveThreadNodeType:
        note.Desc = desc
    }
    skipPadding(r, 4)
    return note, nil
}

Also, referring to the implementation of fill_files_note(struct memelfnote *note) in the kernel source code, this function shows the data format of NT_FILE notes. We can see that long start, long end, long file_ofs are positions in the VMA, not in the mapped files. So as mentioned earlier, as long as the mapped files' contents, besides the mapping relationships in PT_NOTE, are dumped to the core file's PT_LOAD segments, when we buildMemory from the core file and establish SplicedMemory, which contains all VMA regions of the process at coredump time, we can actually read from this SplicedMemory when reading memory later, and we can read it, without needing to read external files. But the premise is that it was dumped (FileSZ != 0).

Actually, although the VMA corresponding to a mapped file might be read-only during process execution, it might not be on the filesystem, and could still be modified, so reading from external files during debugging would be problematic. So I think, for debugging convenience, we should still dump this data into the core file, even though the core file will be larger. But we probably don't care that much about disk usage.

ps: The complete address space of the process, all these VMAs, won't be dumped to the core file. But some VMAs don't have physical memory mappings established, and when recording these in the core file, only necessary information is recorded, with no actual data and no zero values written, but there are indeed some holes left in the file. In this case, ls -h will show a larger file size, but du -hs will show a smaller size. When I was doing game server development, I observed that the battle server process core file size showed as high as 80GB with ls, but actually only about 800MB+ with du.

/*
 * Format of NT_FILE note:
 *
 * long count     -- how many files are mapped
 * long page_size -- units for file_ofs
 * array of [COUNT] elements of
 *   long start
 *   long end
 *   long file_ofs
 * followed by COUNT filenames in ASCII: "FILE1" NUL "FILE2" NUL...
 */
static int fill_files_note(struct memelfnote *note)
{
    struct vm_area_struct *vma;
    unsigned count, size, names_ofs, remaining, n;
    user_long_t *data;
    user_long_t *start_end_ofs;
    char *name_base, *name_curpos;

    /* *Estimated* file count and total data size needed */
    count = current->mm->map_count;
    size = count * 64;

    names_ofs = (2 + 3 * count) * sizeof(data[0]);
 alloc:
    size = round_up(size, PAGE_SIZE);
    data = kvmalloc(size, GFP_KERNEL);

    start_end_ofs = data + 2;
    name_base = name_curpos = ((char *)data) + names_ofs;
    remaining = size - names_ofs;
    count = 0;
    for (vma = current->mm->mmap; vma != NULL; vma = vma->vm_next) {
        struct file *file;
        const char *filename;

        file = vma->vm_file;
        filename = file_path(file, name_curpos, remaining);

        /* file_path() fills at the end, move name down */
        /* n = strlen(filename) + 1: */
        n = (name_curpos + remaining) - filename;
        remaining = filename - name_curpos;
        memmove(name_curpos, filename, n);
        name_curpos += n;

        *start_end_ofs++ = vma->vm_start;
        *start_end_ofs++ = vma->vm_end;
        *start_end_ofs++ = vma->vm_pgoff;
        count++;
    }

    /* Now we know exact count of files, can store it */
    data[0] = count;
    data[1] = PAGE_SIZE;
    ...

    size = name_curpos - (char *)data;
    fill_note(note, "CORE", NT_FILE, size, data);
    return 0;
}

Subsequent Memory Reading Operations

Note that when tracking process memory mappings during core file buildMemory:

type SplicedMemory struct {
    readers []readerEntry
}

func buildMemory(core, exeELF *elf.File, exe io.ReaderAt, notes []*note) proc.MemoryReader {
    memory := &SplicedMemory{}

    // For now, assume all file mappings are to the exe.
    for _, note := range notes {
        if note.Type == _NT_FILE {
            fileNote := note.Desc.(*linuxNTFile)
            for _, entry := range fileNote.entries {
                r := &offsetReaderAt{
                    reader: exe,
                    offset: entry.Start - (entry.FileOfs * fileNote.PageSize),
                }
                memory.Add(r, entry.Start, entry.End-entry.Start)
            }
        }
    }

    // Load memory segments from exe and then from the core file,
    // allowing the corefile to overwrite previously loaded segments
    for _, elfFile := range []*elf.File{exeELF, core} {
        if elfFile == nil {
            continue
        }
        for _, prog := range elfFile.Progs {
            if prog.Type == elf.PT_LOAD {
                if prog.Filesz == 0 {
                    continue
                }
                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }
                memory.Add(r, prog.Vaddr, prog.Filesz)
            }
        }
    }
    return memory
}

We focus on the readers construction in the lower part:

        for _, prog := range elfFile.Progs {
            if prog.Type == elf.PT_LOAD {
                if prog.Filesz == 0 {
                    continue
                }
                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }
                memory.Add(r, prog.Vaddr, prog.Filesz)
            }
        }

We only process parts that have mappings and FileSZ != 0. If FileSZ == 0, we simply skip processing (recall that we didn't record filenames in readNote either, so we can't read them, and even if we could read them, since these files themselves might have changed, it wouldn't be useful to us). Then we put these memory regions with data into our SplicedMemory, with each VMA corresponding to a reader like this:

                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }

Later, when we need to read memory, instead of reading through ptrace(PTRACE_PEEKTEXT/PEEKDATA, ...) like when debugging a process, we read directly from the readers in SplicedMemory:

  1. First determine which VMAs' corresponding readers the data is in based on the starting address and data size to be read;
  2. Then read from these readers;
  3. The starting address for reading from each reader is already recorded, and the starting address is actually the VirtSize of each PT_LOAD type in the core file. ps: In part 1 we mentioned that in executable programs, VirtSize represents the loading address of PT_LOAD type in the process address space, but in core files, it represents the offset in the core file.

Subsequent Register Reading Operations

This is naturally even simpler. This information is recorded in the PT_NOTE corresponding segment, and when we read it, we've already parsed it and placed it in appropriate data structures, so it's not a problem.

Subsequent Initialization and Debugging

Afterward, the debugger continues to initialize the debug session and network communication parts, and can then examine the problem scene based on the core file and try to locate issues.

Execution Testing

Even after opening a core file, it's just reading a snapshot. Although it reconstructs the problem scene, it doesn't reconstruct the process, so debug commands involving execution in the debug session cannot be executed. Core file debugging typically uses bt to observe the stack, frame to select stack frames, and locals/args to view function parameters and local variable information.

Test examples omitted.

Summary

This article has introduced how tinydbg generates and loads core files, including:

  1. The basic concepts and structure of core files
  2. How core files are generated by the Linux kernel and custom tools
  3. The implementation details of tinydbg's core file generation and loading
  4. How memory and register information is handled during core file operations
  5. The limitations and considerations in core file debugging

Through this implementation, tinydbg provides a powerful tool for post-mortem debugging, allowing developers to analyze program crashes and issues after they occur.

The only shortcoming is that for some non-anonymous mapped file VMAs with FileSZ==0, this data might not be written out by the kernel, and these mapped files might be modified afterward. Even if we read them back, they won't match the problem scene at that time. This is a real issue.

tinydbg doesn't handle reading these mapped files, but rather selectively ignores them. Because even if it supported reading them, it couldn't properly handle these real existing problems. tinydbg has done quite well as is, see discussion here: https://github.com/go-delve/delve/discussions/4031.

Subsequent Memory Reading Operations

Note that when tracking process memory mappings during core file buildMemory:

type SplicedMemory struct {
    readers []readerEntry
}

func buildMemory(core, exeELF *elf.File, exe io.ReaderAt, notes []*note) proc.MemoryReader {
    memory := &SplicedMemory{}

    // For now, assume all file mappings are to the exe.
    for _, note := range notes {
        if note.Type == _NT_FILE {
            fileNote := note.Desc.(*linuxNTFile)
            for _, entry := range fileNote.entries {
                r := &offsetReaderAt{
                    reader: exe,
                    offset: entry.Start - (entry.FileOfs * fileNote.PageSize),
                }
                memory.Add(r, entry.Start, entry.End-entry.Start)
            }
        }
    }

    // Load memory segments from exe and then from the core file,
    // allowing the corefile to overwrite previously loaded segments
    for _, elfFile := range []*elf.File{exeELF, core} {
        if elfFile == nil {
            continue
        }
        for _, prog := range elfFile.Progs {
            if prog.Type == elf.PT_LOAD {
                if prog.Filesz == 0 {
                    continue
                }
                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }
                memory.Add(r, prog.Vaddr, prog.Filesz)
            }
        }
    }
    return memory
}

We focus on the readers construction in the lower part:

        for _, prog := range elfFile.Progs {
            if prog.Type == elf.PT_LOAD {
                if prog.Filesz == 0 {
                    continue
                }
                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }
                memory.Add(r, prog.Vaddr, prog.Filesz)
            }
        }

We only process parts that have mappings and FileSZ != 0. If FileSZ == 0, we simply skip processing (recall that we didn't record filenames in readNote either, so we can't read them, and even if we could read them, since these files themselves might have changed, it wouldn't be useful to us). Then we put these memory regions with data into our SplicedMemory, with each VMA corresponding to a reader like this:

                r := &offsetReaderAt{
                    reader: prog.ReaderAt,
                    offset: prog.Vaddr,
                }

Later, when we need to read memory, instead of reading through ptrace(PTRACE_PEEKTEXT/PEEKDATA, ...) like when debugging a process, we read directly from the readers in SplicedMemory:

  1. First determine which VMAs' corresponding readers the data is in based on the starting address and data size to be read;
  2. Then read from these readers;
  3. The starting address for reading from each reader is already recorded, and the starting address is actually the VirtSize of each PT_LOAD type in the core file. ps: In part 1 we mentioned that in executable programs, VirtSize represents the loading address of PT_LOAD type in the process address space, but in core files, it represents the offset in the core file.

Subsequent Register Reading Operations

This is naturally even simpler. This information is recorded in the PT_NOTE corresponding segment, and when we read it, we've already parsed it and placed it in appropriate data structures, so it's not a problem.

Subsequent Initialization and Debugging

Afterward, the debugger continues to initialize the debug session and network communication parts, and can then examine the problem scene based on the core file and try to locate issues.

Execution Testing

Even after opening a core file, it's just reading a snapshot. Although it reconstructs the problem scene, it doesn't reconstruct the process, so debug commands involving execution in the debug session cannot be executed. Core file debugging typically uses bt to observe the stack, frame to select stack frames, and locals/args to view function parameters and local variable information.

Test examples omitted.

Summary

This article has introduced how tinydbg generates and loads core files, including:

  1. The basic concepts and structure of core files
  2. How core files are generated by the Linux kernel and custom tools
  3. The implementation details of tinydbg's core file generation and loading
  4. How memory and register information is handled during core file operations
  5. The limitations and considerations in core file debugging

Through this implementation, tinydbg provides a powerful tool for post-mortem debugging, allowing developers to analyze program crashes and issues after they occur.

results matching ""

    No results matching ""