Skip to contentSkip to conversation

kernel 'simulator' and wave-form analysis tool?

DISCUSSION Timothy Miller 7 participants 9 messages Mar 5, 2004 – Mar 12, 2004 1 message not in the archive

lore
Share & tools
lore
#0 lore raw

I wouldn't be surprised if someone's already done this, but...

I'm a chip designer, and when we design a chip, before we put it in silicon, we use simulator tools that emulate the logic we designed. One of the most important parts of the simulator is the wave-form analyzer.
We run the simulator for some period of time, and then we can look at
the history of every signal in the design.

Well, I've been looking at Bochs, and it has this 'instrumentation' facility which you can use to track everything that goes on in its simulation of an x86 processor. If I were to put a hook in to track all memory writes, then I could record all memory activity (I could hook much more!). When a crash occurs, someone could use the analogue to the wave-form tool to trace execution back to the event that caused the problem (because, for instance, heap corruption causes crashes much later than the bug).

Would it be a productive use of my time to work on this?

A message that is not in the archive (sent off-list or not yet received): [email protected]

#2 lore raw
Quoted 1 line · Timothy Miller

Would it be a productive use of my time to work on this?

Not sure what you mean with waveform tool in this context, but simulator logs can be indeed very useful to track down kernel bugs. I used this excessively in the early days of the x86-64 port first with the SimNow and later with the commercial Virtutech Simics simulator. They both can generate a log file with all instructions and all memory accesses. We wrote some post processing tools that added the kernel symbols to that. While processing these files (which can quickly become several GB) was quite a stress test for less and the VM it was often very useful. I can also only recommend XFS for such uses, it seems to be by far the best file system for such workloads.

For example to find out who previously changed something in memory you just need to search backwards for the same physical address. The real work usually was just to get a small enough "window" into the problem to keep the log log files manageable. With logging the simulator runs extremly slow and fills up disks quite quickly. I don't think it's useful for complicated problems like races that take longer to trigger (for longer running tests full logging is not practical), only for something small and tricky that is relatively easy to repeat.

Writing a post processing tool that adds the symbols is not that complicated, you can probably do it easily with any simulator log file. While it would be possible to write a frontend to look for memory addresses and automate other tasks just grep and less already work reasonably well. With the amount of data involved you would probably want to do most analysis as "batch" jobs.

-Andi

#3 lore raw
In reply to Timothy Miller
Quoted 17 lines · Timothy Miller

I wouldn't be surprised if someone's already done this, but...

I'm a chip designer, and when we design a chip, before we put it in silicon, we use simulator tools that emulate the logic we designed. One of the most important parts of the simulator is the wave-form analyzer.
We run the simulator for some period of time, and then we can look at
the history of every signal in the design.

Well, I've been looking at Bochs, and it has this 'instrumentation' facility which you can use to track everything that goes on in its simulation of an x86 processor. If I were to put a hook in to track all memory writes, then I could record all memory activity (I could hook much more!). When a crash occurs, someone could use the analogue to the wave-form tool to trace execution back to the event that caused the problem (because, for instance, heap corruption causes crashes much later than the bug).

Quoted 2 lines

Would it be a productive use of my time to work on this?

If you are making hardware that goes between the CPU and the rest of the world, then you can keep track of anything that's going on with some hardware-software combination, external to the chip you are analyzing. These things exist and they are called emulators, even though most don't emulate anything, they use the real chip, but provide the physical and logical connections to the user. However, in the case of an already-made machine, you are limited in what you can do on the machine with software. For instance, to trap every memory access, you would need a trap-handler and set all the memory to trap on an access. This would a bit hard to do within the kernel because all the code on that page would trap as instructions were fetched. So, some mere "hook" won't do it, you need a kernel that executes a kernel and I think one for Linux already exists. So, before you get too involved, you might want to check that out.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
Note 96.31% of all statistics are fiction.

#4 lore raw
Quoted 18 lines · Richard B. Johnson

If you are making hardware that goes between the CPU and the rest of the world, then you can keep track of anything that's going on with some hardware-software combination, external to the chip you are analyzing. These things exist and they are called emulators, even though most don't emulate anything, they use the real chip, but provide the physical and logical connections to the user. However, in the case of an already-made machine, you are limited in what you can do on the machine with software. For instance, to trap every memory access, you would need a trap-handler and set all the memory to trap on an access. This would a bit hard to do within the kernel because all the code on that page would trap as instructions were fetched. So, some mere "hook" won't do it, you need a kernel that executes a kernel and I think one for Linux already exists. So, before you get too involved, you might want to check that out.

I must have been unclear. I was not suggesting adding hardware. I was suggesting that we could run Linux under Bochs, which is a software x86 emulator. Being what it is, hooks can be added to track "cpu activity" is it occurs within the emulator. This is all a simulation. The key idea I was suggesting was to log processor activity (of the emulator) and develop a viewer program which would help people visualize the activity.

Bochs already has hooks. I could write a logger and a viewer.

#5 lore raw
Quoted 6 lines

I must have been unclear. I was not suggesting adding hardware. I was suggesting that we could run Linux under Bochs, which is a software x86 emulator. Being what it is, hooks can be added to track "cpu activity" is it occurs within the emulator. This is all a simulation. The key idea I was suggesting was to log processor activity (of the emulator) and develop a viewer program which would help people visualize the activity.

Doesn't Valgrind already do most of what you want?

John.

#6 lore raw
Quoted 4 lines · John Bradford
Quoted 1 line

I must have been unclear. I was not suggesting adding hardware. I was suggesting that we could run Linux under Bochs, which is a software x86 emulator. Being what it is, hooks can be added to track "cpu activity" is it occurs within the emulator. This is all a simulation. The key idea I was suggesting was to log processor activity (of the emulator) and develop a viewer program which would help people visualize the activity.

Doesn't Valgrind already do most of what you want?

Can you valgrind a UML process?

Tim, what will this give you that a stack trace won't?

#7 lore raw
Quoted 1 line · Mike Fedyk

Can you valgrind a UML process?

Sorta. I think that valgrind should be able to handle UML. It just needs to be told how the kernel memory allocators work before it will provide any useful debugging information about the kernel.

Jeff

#8 lore raw
In reply to Mike Fedyk
Quoted 6 lines · Mike Fedyk
Quoted 4 lines · John Bradford
Quoted 1 line

I must have been unclear. I was not suggesting adding hardware. I was suggesting that we could run Linux under Bochs, which is a software x86 emulator. Being what it is, hooks can be added to track "cpu activity" is it occurs within the emulator. This is all a simulation. The key idea I was suggesting was to log processor activity (of the emulator) and develop a viewer program which would help people visualize the activity.

Quoted 8 lines
Quoted 1 line

Doesn't Valgrind already do most of what you want?

Can you valgrind a UML process?

Tim, what will this give you that a stack trace won't?

If your stack gets hosed by a bug, a simulator with a complete history of memory writes will help you discover the problem.

I know nothing about Valgrind.

#9 lore raw
Quoted 5 lines · Timothy Miller
Quoted 2 lines · Mike Fedyk

John Bradford wrote:

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines

I must have been unclear. I was not suggesting adding hardware. I was suggesting that we could run Linux under Bochs, which is a software x86 emulator. Being what it is, hooks can be added to track "cpu activity" is it occurs within the emulator. This is all a simulation. The key idea I was suggesting was to log processor activity (of the emulator) and develop a viewer program which would help people visualize the activity.

sounds good and useful to me, have a look at other simulators/emulators too, for example I use QEMU[1] to test linux kernels, because it is much simpler and faster than Bochs (YMMV)

best,
Herbert

Quoted 4 lines

If your stack gets hosed by a bug, a simulator with a complete history of memory writes will help you discover the problem.

I know nothing about Valgrind.

[1] http://fabrice.bellard.free.fr/qemu/