scripts/gdb/symbols: make BPF debug info available to GDB
Subsystems (2)
cover scripts/gdb/symbols: make BPF debug info available to GDB
Hi,
This series greatly simplifies debugging BPF progs when using QEMU gdbstub by providing symbol names, sizes, and line numbers to GDB.
Patch 1 adds radix tree iteration, which is necessary for parsing prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.
Best regards,
Ilya
Ilya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-command scripts/gdb/symbols: make BPF debug info available to GDB
scripts/gdb/linux/bpf.py | 253 ++++++++++++++++++++++++++++++ scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++- 4 files changed, 462 insertions(+), 10 deletions(-) create mode 100644 scripts/gdb/linux/bpf.py
Signature · 1 line
2.50.0
PATCH 1/2 scripts/gdb/radix-tree: add lx-radix-tree-command
Add a function and a command to iterate over radix tree contents. Duplicate the C implementation in Python, but drop support for tagging.
--- scripts/gdb/linux/radixtree.py | 139 +++++++++++++++++++++++++++++++-- 1 file changed, 132 insertions(+), 7 deletions(-)
scripts/gdb/linux/radixtree.py +132 -7
@@ -30,13 +30,16 @@ def entry_to_node(node): def node_maxindex(node): return (constants.LX_RADIX_TREE_MAP_SIZE << node['shift']) - 1 -def lookup(root, index): +def resolve_root(root): + if root.type == radix_tree_root_type.get_type(): + return root if root.type == radix_tree_root_type.get_type().pointer(): - node = root.dereference() - elif root.type != radix_tree_root_type.get_type(): - raise gdb.GdbError("must be {} not {}" - .format(radix_tree_root_type.get_type(), root.type)) + return root.dereference() + raise gdb.GdbError("must be {} not {}" + .format(radix_tree_root_type.get_type(), root.type)) +def lookup(root, index): + root = resolve_root(root) node = root['xa_head'] if node == 0: return None
@@ -71,14 +74,120 @@ def lookup(root, index): return node -class LxRadixTree(gdb.Function): +def descend(parent, index): + offset = (index >> int(parent["shift"])) & constants.LX_RADIX_TREE_MAP_MASK + return offset, parent["slots"][offset] + +def load_root(root): + node = root["xa_head"] + nodep = node + + if is_internal_node(node): + node = entry_to_node(node) + maxindex = node_maxindex(node) + return int(node["shift"]) + constants.LX_RADIX_TREE_MAP_SHIFT, \ + nodep, maxindex + + return 0, nodep, 0 + +class RadixTreeIter: + def __init__(self, start): + self.index = 0 + self.next_index = start + self.node = None + +def xa_mk_internal(v): + return (v << 2) | 2 + +LX_XA_RETRY_ENTRY = xa_mk_internal(256) +LX_RADIX_TREE_RETRY = LX_XA_RETRY_ENTRY + +def next_chunk(root, iter): + mask = (1 << (utils.get_ulong_type().sizeof * 8)) - 1 + + index = iter.next_index + if index == 0 and iter.index != 0: + return None + + restart = True + while restart: + restart = False + + _, child, maxindex = load_root(root) + if index > maxindex: + return None + if not child: + return None + + if not is_internal_node(child): + iter.index = index + iter.next_index = (maxindex + 1) & mask + iter.node = None + return root["xa_head"].address + + while True: + node = entry_to_node(child) + offset, child = descend(node, index) + + if not child: + while True: + offset += 1 + if offset >= constants.LX_RADIX_TREE_MAP_SIZE: + break + slot = node["slots"][offset] + if slot: + break + index &= ~node_maxindex(node) + index = (index + (offset << int(node["shift"]))) & mask + if index == 0: + return None + if offset == constants.LX_RADIX_TREE_MAP_SIZE: + restart = True + break + child = node["slots"][offset] + + if not child: + restart = True + break + if child == LX_XA_RETRY_ENTRY: + break + if not node["shift"] or not is_internal_node(child): + break + + iter.index = (index & ~node_maxindex(node)) | offset + iter.next_index = ((index | node_maxindex(node)) + 1) & mask + iter.node = node + + return node["slots"][offset].address + +def next_slot(slot, iter): + mask = (1 << (utils.get_ulong_type().sizeof * 8)) - 1 + for _ in range(iter.next_index - iter.index - 1): + slot += 1 + iter.index = (iter.index + 1) & mask + if slot.dereference(): + return slot + return None + +def for_each_slot(root, start=0): + iter = RadixTreeIter(start) + slot = None + while True: + if not slot: + slot = next_chunk(root, iter) + if not slot: + break + yield iter.index, slot + slot = next_slot(slot, iter) + +class LxRadixTreeLookup(gdb.Function): """ Lookup and return a node from a RadixTree. $lx_radix_tree_lookup(root_node [, index]): Return the node at the given index. If index is omitted, the root node is dereference and returned.""" def __init__(self): - super(LxRadixTree, self).__init__("lx_radix_tree_lookup") + super(LxRadixTreeLookup, self).__init__("lx_radix_tree_lookup") def invoke(self, root, index=0): result = lookup(root, index)
@@ -87,4 +196,20 @@ If index is omitted, the root node is dereference and returned.""" return result +class LxRadixTree(gdb.Command): + """Show all values stored in a RadixTree.""" + + def __init__(self): + super(LxRadixTree, self).__init__("lx-radix-tree", gdb.COMMAND_DATA, + gdb.COMPLETE_NONE) + + def invoke(self, argument, from_tty): + args = gdb.string_to_argv(argument) + if len(args) != 1: + raise gdb.GdbError("Usage: lx-radix-tree ROOT") + root = gdb.parse_and_eval(args[0]) + for index, slot in for_each_slot(root): + gdb.write("[{}] = {}\n".format(index, slot.dereference())) + LxRadixTree() +LxRadixTreeLookup()
Signature · 1 line
2.50.0
PATCH 2/2 scripts/gdb/symbols: make BPF debug info available to GDB
In reply to Ilya LeoshkevichOne can debug BPF programs with QEMU gdbstub by setting a breakpoint on bpf_prog_kallsyms_add(), waiting for a hit with a matching aux.name, and then setting a breakpoint on bpf_func. This is tedious, error-prone, and also lacks line numbers.
Automate this in a way similar to the existing support for modules in lx-symbols.
Enumerate and monitor changes to both BPF kallsyms and JITed progs. For each ksym, generate and compile a synthetic .s file containing the name, code, and size. In addition, if this ksym is also a prog, and not a trampoline, add line number information.
Ensure that this is a no-op if the kernel is built without BPF support or if "as" is missing. In theory the "as" dependency may be dropped by generating the synthetic .o file manually, but this is too much complexity for too little benefit.
Now one can debug BPF progs out of the box like this:
(gdb) lx-symbols
(gdb) b bpf_prog_4e612a6a881a086b_arena_list_add
Breakpoint 2 (bpf_prog_4e612a6a881a086b_arena_list_add) pending.
# ./test_progs -t arena_list
Thread 4 hit Breakpoint 2, bpf_prog_4e612a6a881a086b_arena_list_add ()
at linux/tools/testing/selftests/bpf/progs/arena_list.c:51
51 list_head = &global_head;
(gdb) n
bpf_prog_4e612a6a881a086b_arena_list_add () at linux/tools/testing/selftests/bpf/progs/arena_list.c:53
53 for (i = zero; i < cnt && can_loop; i++) {This also works for subprogs.
--- scripts/gdb/linux/bpf.py | 253 ++++++++++++++++++++++++++++++ scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/symbols.py | 77 ++++++++- 3 files changed, 330 insertions(+), 3 deletions(-) create mode 100644 scripts/gdb/linux/bpf.py
scripts/gdb/linux/bpf.py new file +253 -0
@@ -0,0 +1,253 @@ +# SPDX-License-Identifier: GPL-2.0 + +import json +import subprocess +import tempfile + +import gdb + +from linux import constants, lists, radixtree, utils + + +if constants.LX_CONFIG_BPF and constants.LX_CONFIG_BPF_JIT: + bpf_ksym_type = utils.CachedType("struct bpf_ksym") +if constants.LX_CONFIG_BPF_SYSCALL: + bpf_prog_type = utils.CachedType("struct bpf_prog") + + +def get_ksym_name(ksym): + name = ksym["name"].bytes + end = name.find(b"\x00") + if end != -1: + name = name[:end] + return name.decode() + + +def list_ksyms(): + if not (constants.LX_CONFIG_BPF and constants.LX_CONFIG_BPF_JIT): + return [] + bpf_kallsyms = gdb.parse_and_eval("&bpf_kallsyms") + bpf_ksym_ptr_type = bpf_ksym_type.get_type().pointer() + return list(lists.list_for_each_entry(bpf_kallsyms, + bpf_ksym_ptr_type, + "lnode")) + + +class KsymAddBreakpoint(gdb.Breakpoint): + def __init__(self, monitor): + super(KsymAddBreakpoint, self).__init__("bpf_ksym_add", internal=True) + self.silent = True + self.monitor = monitor + + def stop(self): + self.monitor.add(gdb.parse_and_eval("ksym")) + return False + + +class KsymRemoveBreakpoint(gdb.Breakpoint): + def __init__(self, monitor): + super(KsymRemoveBreakpoint, self).__init__("bpf_ksym_del", + internal=True) + self.silent = True + self.monitor = monitor + + def stop(self): + self.monitor.remove(gdb.parse_and_eval("ksym")) + return False + + +class KsymMonitor: + def __init__(self, add, remove): + self.add = add + self.remove = remove + + self.add_bp = KsymAddBreakpoint(self) + self.remove_bp = KsymRemoveBreakpoint(self) + + self.notify_initial() + + def notify_initial(self): + for ksym in list_ksyms(): + self.add(ksym) + + def close(self): + self.add_bp.delete() + self.remove_bp.delete() + + +def list_progs(): + if not constants.LX_CONFIG_BPF_SYSCALL: + return [] + idr_rt = gdb.parse_and_eval("&prog_idr.idr_rt") + bpf_prog_ptr_type = bpf_prog_type.get_type().pointer() + progs = [] + for _, slot in radixtree.for_each_slot(idr_rt): + prog = slot.dereference().cast(bpf_prog_ptr_type) + progs.append(prog) + # Subprogs are not registered in prog_idr, fetch them manually. + # func[0] is the current prog. + aux = prog["aux"] + func = aux["func"] + real_func_cnt = int(aux["real_func_cnt"]) + for i in range(1, real_func_cnt): + progs.append(func[i]) + return progs + + +class ProgAddBreakpoint(gdb.Breakpoint): + def __init__(self, monitor): + super(ProgAddBreakpoint, self).__init__("bpf_prog_kallsyms_add", + internal=True) + self.silent = True + self.monitor = monitor + + def stop(self): + self.monitor.add(gdb.parse_and_eval("fp")) + return False + + +class ProgRemoveBreakpoint(gdb.Breakpoint): + def __init__(self, monitor): + super(ProgRemoveBreakpoint, self).__init__("bpf_prog_free_id", + internal=True) + self.silent = True + self.monitor = monitor + + def stop(self): + self.monitor.remove(gdb.parse_and_eval("prog")) + return False + + +class ProgMonitor: + def __init__(self, add, remove): + self.add = add + self.remove = remove + + self.add_bp = ProgAddBreakpoint(self) + self.remove_bp = ProgRemoveBreakpoint(self) + + self.notify_initial() + + def notify_initial(self): + for prog in list_progs(): + self.add(prog) + + def delete(self): + self.add_bp.delete() + self.remove_bp.delete() + + +def btf_str_by_offset(btf, offset): + while offset < btf["start_str_off"]: + btf = btf["base_btf"] + + offset -= btf["start_str_off"] + if offset < btf["hdr"]["str_len"]: + return (btf["strings"] + offset).string() + + return None + + +def bpf_line_info_line_num(line_col): + return line_col >> 10 + + +def bpf_line_info_line_col(line_col): + return line_col & 0x3ff + + +class LInfoIter: + def __init__(self, prog): + # See bpf_prog_get_file_line() for details. + self.pos = 0 + self.nr_linfo = 0 + + if prog is None: + return + + self.bpf_func = int(prog["bpf_func"]) + aux = prog["aux"] + self.btf = aux["btf"] + linfo_idx = aux["linfo_idx"] + self.nr_linfo = int(aux["nr_linfo"]) - linfo_idx + if self.nr_linfo == 0: + return + + linfo_ptr = aux["linfo"] + tpe = linfo_ptr.type.target().array(self.nr_linfo).pointer() + self.linfo = (linfo_ptr + linfo_idx).cast(tpe).dereference() + jited_linfo_ptr = aux["jited_linfo"] + tpe = jited_linfo_ptr.type.target().array(self.nr_linfo).pointer() + self.jited_linfo = (jited_linfo_ptr + linfo_idx).cast(tpe).dereference() + + self.filenos = {} + + def get_code_off(self): + if self.pos >= self.nr_linfo: + return -1 + return self.jited_linfo[self.pos] - self.bpf_func + + def advance(self): + self.pos += 1 + + def get_fileno(self): + file_name_off = int(self.linfo[self.pos]["file_name_off"]) + fileno = self.filenos.get(file_name_off) + if fileno is not None: + return fileno, None + file_name = btf_str_by_offset(self.btf, file_name_off) + fileno = len(self.filenos) + 1 + self.filenos[file_name_off] = fileno + return fileno, file_name + + def get_line_col(self): + line_col = int(self.linfo[self.pos]["line_col"]) + return bpf_line_info_line_num(line_col), \ + bpf_line_info_line_col(line_col) + + +def generate_debug_obj(ksym, prog): + name = get_ksym_name(ksym) + # Avoid read_memory(); it throws bogus gdb.MemoryError in some contexts. + start = ksym["start"] + code = start.cast(gdb.lookup_type("unsigned char") + .array(int(ksym["end"]) - int(start)) + .pointer()).dereference().bytes + linfo_iter = LInfoIter(prog) + + result = tempfile.NamedTemporaryFile(suffix=".o", mode="wb") + try: + with tempfile.NamedTemporaryFile(suffix=".s", mode="w") as src: + # ".loc" does not apply to ".byte"s, only to ".insn"s, but since + # this needs to work for all architectures, the latter are not an + # option. Ask the assembler to apply ".loc"s to labels as well, + # and generate dummy labels after each ".loc". + src.write(".loc_mark_labels 1\n") + + src.write(".globl {}\n".format(name)) + src.write(".type {},@function\n".format(name)) + src.write("{}:\n".format(name)) + for code_off, code_byte in enumerate(code): + if linfo_iter.get_code_off() == code_off: + fileno, file_name = linfo_iter.get_fileno() + if file_name is not None: + src.write(".file {} {}\n".format( + fileno, json.dumps(file_name))) + line, col = linfo_iter.get_line_col() + src.write(".loc {} {} {}\n".format(fileno, line, col)) + src.write("0:\n") + linfo_iter.advance() + src.write(".byte {}\n".format(code_byte)) + src.write(".size {},{}\n".format(name, len(code))) + src.flush() + + try: + subprocess.check_call(["as", "-c", src.name, "-o", result.name]) + except FileNotFoundError: + # "as" is not installed. + result.close() + return None + return result + except: + result.close() + raise
scripts/gdb/linux/constants.py.in +3 -0
@@ -163,3 +163,6 @@ LX_CONFIG(CONFIG_PAGE_OWNER) LX_CONFIG(CONFIG_SLUB_DEBUG) LX_CONFIG(CONFIG_SLAB_FREELIST_HARDENED) LX_CONFIG(CONFIG_MMU) +LX_CONFIG(CONFIG_BPF) +LX_CONFIG(CONFIG_BPF_JIT) +LX_CONFIG(CONFIG_BPF_SYSCALL)
scripts/gdb/linux/symbols.py +74 -3
118 lines not on this page · raw message
Signature · 1 line
2.50.0
Quoted 16 lines · Ilya Leoshkevich
Hi,
This series greatly simplifies debugging BPF progs when using QEMU gdbstub by providing symbol names, sizes, and line numbers to GDB.
Patch 1 adds radix tree iteration, which is necessary for parsing
prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.Best regards,
IlyaIlya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-commandscripts/gdb/symbols: make BPF debug info available to GDB
Quoted 7 lines
scripts/gdb/linux/bpf.py | 253
++++++++++++++++++++++++++++++
scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++-4 files changed, 462 insertions(+), 10 deletions(-)create mode 100644
scripts/gdb/linux/bpf.py
Gentle ping. Any opinions on whether this is valuable? Personally I've been using this for quite some time, and having source level debugging for BPF progs (even if variables can't be inspected) feels really nice.
Quoted 26 lines · Ilya Leoshkevich
Quoted 16 lines · Ilya Leoshkevich
Hi,
This series greatly simplifies debugging BPF progs when using QEMU gdbstub by providing symbol names, sizes, and line numbers to GDB.
Patch 1 adds radix tree iteration, which is necessary for parsing
prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.Best regards,
IlyaIlya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-commandscripts/gdb/symbols: make BPF debug info available to GDBQuoted 7 lines
scripts/gdb/linux/bpf.py | 253 ++++++++++++++++++++++++++++++ scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++- 4 files changed, 462 insertions(+), 10 deletions(-) create mode 100644 scripts/gdb/linux/bpf.py
Quoted 3 lines
Gentle ping. Any opinions on whether this is valuable? Personally I've been using this for quite some time, and having source level debugging for BPF progs (even if variables can't be inspected) feels really nice.
Looks very useful to me.
Not sure which git tree it should be routed to.
Quoted 37 lines · Alexei Starovoitov
Quoted 28 lines · Ilya Leoshkevich
Quoted 18 lines · Ilya Leoshkevich
Hi,
This series greatly simplifies debugging BPF progs when using QEMU
gdbstub by providing symbol names, sizes, and line numbers to GDB.Patch 1 adds radix tree iteration, which is necessary for parsing
prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.Best regards,
IlyaIlya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-commandscripts/gdb/symbols: make BPF debug info available to GDBQuoted 7 lines
scripts/gdb/linux/bpf.py | 253
++++++++++++++++++++++++++++++
scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++-4 files changed, 462 insertions(+), 10 deletions(-)create mode 100644
scripts/gdb/linux/bpf.pyQuoted 6 lines
Gentle ping. Any opinions on whether this is valuable? Personally I've
been using this for quite some time, and having source level debugging
for BPF progs (even if variables can't be inspected) feels really nice.
Quoted 2 lines
Looks very useful to me.
Not sure which git tree it should be routed to.
Thanks, glad to hear that!
I think it makes sense to route it via the Andrew Morton's tree, like all the other GDB patches. IIUC the proposal to ask subsystems to maintain GDB scripts relevant to them [1] didn't go anywhere.
Jan Kiszka
maintainerQuoted 16 lines · Ilya Leoshkevich
Hi,
This series greatly simplifies debugging BPF progs when using QEMU gdbstub by providing symbol names, sizes, and line numbers to GDB.
Patch 1 adds radix tree iteration, which is necessary for parsing
prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.Best regards,
IlyaIlya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-commandscripts/gdb/symbols: make BPF debug info available to GDB
Quoted 7 lines
scripts/gdb/linux/bpf.py | 253 ++++++++++++++++++++++++++++++ scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++- 4 files changed, 462 insertions(+), 10 deletions(-) create mode 100644 scripts/gdb/linux/bpf.py
This wasn't picked up yet, right? Sorry for the late reply, my part of the "maintenance" here is best effort based.
Looks good to me regarding integration. I haven't tried it out, I'm just wondering if it has notable performance impact on starting gdb or interacting or when that could be the case. BPF programs are not uncommon in common setups today. But if you don't want to debug them, does this add unneeded overhead?
Otherwise, I think it could move forward if it still applies (which it likely does).
Jan
Signature · 2 lines
Siemens AG, Foundational Technologies Linux Expert Center
Quoted 37 lines · Jan Kiszka
Quoted 16 lines · Ilya Leoshkevich
Hi,
This series greatly simplifies debugging BPF progs when using QEMU gdbstub by providing symbol names, sizes, and line numbers to GDB.
Patch 1 adds radix tree iteration, which is necessary for parsing
prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.Best regards,
IlyaIlya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-commandscripts/gdb/symbols: make BPF debug info available to GDBQuoted 8 lines
scripts/gdb/linux/bpf.py | 253
++++++++++++++++++++++++++++++
scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++-4 files changed, 462 insertions(+), 10 deletions(-)create mode 100644
scripts/gdb/linux/bpf.pyThis wasn't picked up yet, right? Sorry for the late reply, my part of
the "maintenance" here is best effort based.Looks good to me regarding integration. I haven't tried it out, I'm just
wondering if it has notable performance impact on starting gdb or interacting or when that could be the case. BPF programs are not uncommon in common setups today. But if you don't want to debug them, does this add unneeded overhead?
Quoted 5 lines
Otherwise, I think it could move forward if it still applies (which it
likely does).Jan
Thanks for taking a look!
I have to admit the performance implications are noticeable due to having to spawn an external process for each BPF prog.
What do you think about hiding this behind lx-symbols --bpf flag?
Jan Kiszka
maintainerQuoted 42 lines · Ilya Leoshkevich
Quoted 41 lines · Jan Kiszka
Quoted 16 lines · Ilya Leoshkevich
Hi,
This series greatly simplifies debugging BPF progs when using QEMU gdbstub by providing symbol names, sizes, and line numbers to GDB.
Patch 1 adds radix tree iteration, which is necessary for parsing
prog_idr. Patch 2 is the actual implementation; its description contains some details on how to use this.Best regards,
IlyaIlya Leoshkevich (2):
scripts/gdb/radix-tree: add lx-radix-tree-commandscripts/gdb/symbols: make BPF debug info available to GDBQuoted 8 lines
scripts/gdb/linux/bpf.py | 253
++++++++++++++++++++++++++++++
scripts/gdb/linux/constants.py.in | 3 + scripts/gdb/linux/radixtree.py | 139 +++++++++++++++- scripts/gdb/linux/symbols.py | 77 ++++++++-4 files changed, 462 insertions(+), 10 deletions(-)create mode 100644
scripts/gdb/linux/bpf.pyThis wasn't picked up yet, right? Sorry for the late reply, my part of
the "maintenance" here is best effort based.Looks good to me regarding integration. I haven't tried it out, I'm just
wondering if it has notable performance impact on starting gdb or interacting or when that could be the case. BPF programs are not uncommon in common setups today. But if you don't want to debug them, does this add unneeded overhead?Otherwise, I think it could move forward if it still applies (which it
likely does).
Quoted 8 lines
Quoted 1 line
Jan
Thanks for taking a look!
I have to admit the performance implications are noticeable due to having to spawn an external process for each BPF prog.
What do you think about hiding this behind
lx-symbols --bpfflag?
Sounds like a reasonable path.
Just one detail: I think gdb only uses a single dash for flags, thus:
lx-symbols -bpf.
Jan
Signature · 2 lines
Siemens AG, Foundational Technologies Linux Expert Center