Skip to contentSkip to conversation

journal log messages are missing with latest version

DISCUSSION Yong Li 3 participants 8 messages Dec 21, 2018 – Jan 4, 2019

lore
Share & tools
lore

Yong Li

author
#0 lore raw

Hi Patrick,

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the "journalctl -f".

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c
88d6a3b9a5c4d49

add sdjournal interface to inject tests

Thanks,

Yong

#1 lore raw
Quoted 7 lines · Yong Li

Hi Patrick,

Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

Quoted 7 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 5 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 6 lines

Thanks,

Yong

#2 lore raw
Quoted 8 lines · Patrick Venture
Quoted 7 lines · Yong Li

Hi Patrick,

Quoted 7 lines
Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

This showed up on all of our systems too, anything logged using phosphor-logging is missing from the journal. I even ended up using a git bisect to also narrow it down to:

phosphor-logging: srcrev bump 30047bf964..31552c05e2

Patrick Venture (5):
add sdjournal interface to inject tests
test: use only one main library
bugfix: add fixes to mock class
bugfix: rename method to match implementation
test: add test to force building of sdjournal mock

Quoted 31 lines
Quoted 8 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 5 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 6 lines

Thanks,

Yong

#3 lore raw
Quoted 26 lines · Matt Spinler
Quoted 9 lines · Patrick Venture
Quoted 7 lines · Yong Li

Hi Patrick,

Quoted 7 lines
Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

This showed up on all of our systems too, anything logged using phosphor-logging
is missing from the journal. I even ended up using a git bisect to also narrow
it down to:

Quoted 9 lines

phosphor-logging: srcrev bump 30047bf964..31552c05e2

Patrick Venture (5):
add sdjournal interface to inject tests
test: use only one main library
bugfix: add fixes to mock class
bugfix: rename method to match implementation
test: add test to force building of sdjournal mock

William, you've been digging into sdbus a lot recently, can you take a deep look at the patchset above and see what I may have messed up? The phosphor-logging unit-tests were broken until I got it "working." Which lead me to believe that it's not correct -- and the daemons that have unit-tests that hit the log message are seeing the calls.

Quoted 35 lines
Quoted 34 lines
Quoted 10 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 6 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 6 lines

Thanks,

Yong

#4 lore raw
Quoted 39 lines · Patrick Venture
Quoted 26 lines · Matt Spinler
Quoted 9 lines · Patrick Venture
Quoted 7 lines · Yong Li

Hi Patrick,

Quoted 7 lines
Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

This showed up on all of our systems too, anything logged using phosphor-logging
is missing from the journal. I even ended up using a git bisect to also narrow
it down to:

Quoted 9 lines

phosphor-logging: srcrev bump 30047bf964..31552c05e2

Patrick Venture (5):
add sdjournal interface to inject tests
test: use only one main library
bugfix: add fixes to mock class
bugfix: rename method to match implementation
test: add test to force building of sdjournal mock

Quoted 5 lines

William, you've been digging into sdbus a lot recently, can you take a deep look at the patchset above and see what I may have messed up? The phosphor-logging unit-tests were broken until I got it "working." Which lead me to believe that it's not correct -- and the daemons that have unit-tests that hit the log message are seeing the calls.

From reading more into the var_args pass-through, that may be the issue. One cannot pass the var args down directly like this. I propose to initially snip out this extra call and then revisit so that it'll resume working.

The problem is that sd_journal_send doesn't expect a va_list argument, but rather strictly var args, which aren't the same.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 11 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 6 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 6 lines

Thanks,

Yong

#5 lore raw
Quoted 47 lines · Patrick Venture
Quoted 39 lines · Patrick Venture
Quoted 26 lines · Matt Spinler
Quoted 9 lines · Patrick Venture
Quoted 7 lines · Yong Li

Hi Patrick,

Quoted 7 lines
Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

This showed up on all of our systems too, anything logged using phosphor-logging
is missing from the journal. I even ended up using a git bisect to also narrow
it down to:

Quoted 9 lines

phosphor-logging: srcrev bump 30047bf964..31552c05e2

Patrick Venture (5):
add sdjournal interface to inject tests
test: use only one main library
bugfix: add fixes to mock class
bugfix: rename method to match implementation
test: add test to force building of sdjournal mock

Quoted 5 lines

William, you've been digging into sdbus a lot recently, can you take a deep look at the patchset above and see what I may have messed up? The phosphor-logging unit-tests were broken until I got it "working." Which lead me to believe that it's not correct -- and the daemons that have unit-tests that hit the log message are seeing the calls.

Quoted 4 lines

From reading more into the var_args pass-through, that may be the issue. One cannot pass the var args down directly like this. I propose to initially snip out this extra call and then revisit so that it'll resume working.

https://gerrit.openbmc-project.xyz/17054

Quoted 40 lines

The problem is that sd_journal_send doesn't expect a va_list argument, but rather strictly var args, which aren't the same.

Quoted 36 lines
Quoted 36 lines
Quoted 35 lines
Quoted 11 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 6 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 6 lines

Thanks,

Yong

author
#6 lore raw

Thanks Patrick for your mails.

Will test https://gerrit.openbmc-project.xyz/17054, and update the results.

Quoted 77 lines · Patrick Venture
Quoted 39 lines · Patrick Venture
Quoted 33 lines · Patrick Venture
Quoted 22 lines · Matt Spinler
Quoted 8 lines · Patrick Venture
Quoted 6 lines · Yong Li

Hi Patrick,

Quoted 6 lines
Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

This showed up on all of our systems too, anything logged using phosphor-logging
is missing from the journal. I even ended up using a git bisect to also narrow
it down to:

Quoted 9 lines

phosphor-logging: srcrev bump 30047bf964..31552c05e2

Patrick Venture (5):
add sdjournal interface to inject tests
test: use only one main library
bugfix: add fixes to mock class
bugfix: rename method to match implementation
test: add test to force building of sdjournal mock

Quoted 5 lines

William, you've been digging into sdbus a lot recently, can you take a deep look at the patchset above and see what I may have messed up? The phosphor-logging unit-tests were broken until I got it "working." Which lead me to believe that it's not correct -- and the daemons that have unit-tests that hit the log message are seeing the calls.

Quoted 4 lines

From reading more into the var_args pass-through, that may be the
issue. One cannot pass the var args down directly like this. I propose to initially snip out this extra call and then revisit so that it'll resume working.

https://gerrit.openbmc-project.xyz/17054

Quoted 32 lines

The problem is that sd_journal_send doesn't expect a va_list argument, but rather strictly var args, which aren't the same.

Quoted 29 lines
Quoted 29 lines
Quoted 29 lines
Quoted 7 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 6 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 5 lines

Thanks,

Yong

#7 lore raw
Quoted 53 lines · Yong B

Thanks Patrick for your mails.

Will test https://gerrit.openbmc-project.xyz/17054, and update the results.

Quoted 47 lines · Patrick Venture
Quoted 39 lines · Patrick Venture
Quoted 33 lines · Patrick Venture
Quoted 22 lines · Matt Spinler
Quoted 8 lines · Patrick Venture
Quoted 6 lines · Yong Li

Hi Patrick,

Quoted 6 lines
Quoted 5 lines

I encountered a problem after upgrade to the latest opnebmc version:

These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

This showed up on all of our systems too, anything logged using phosphor-logging
is missing from the journal. I even ended up using a git bisect to also narrow
it down to:

Quoted 9 lines

phosphor-logging: srcrev bump 30047bf964..31552c05e2

Patrick Venture (5):
add sdjournal interface to inject tests
test: use only one main library
bugfix: add fixes to mock class
bugfix: rename method to match implementation
test: add test to force building of sdjournal mock

Quoted 5 lines

William, you've been digging into sdbus a lot recently, can you take a deep look at the patchset above and see what I may have messed up? The phosphor-logging unit-tests were broken until I got it "working." Which lead me to believe that it's not correct -- and the daemons that have unit-tests that hit the log message are seeing the calls.

Quoted 4 lines

From reading more into the var_args pass-through, that may be the
issue. One cannot pass the var args down directly like this. I propose to initially snip out this extra call and then revisit so that it'll resume working.

https://gerrit.openbmc-project.xyz/17054

Quoted 2 lines

The problem is that sd_journal_send doesn't expect a va_list argument, but rather strictly var args, which aren't the same.

I generally try to avoid copying implementations, but an approach to this that would work and allow testing would be to use sd_journal_sendv() and have the interface call walk the parameters and build a list of iovec structures and then pass that. This is effectively what is done by sd_journal_send, which just processes the input and sends to sd_journal_sendv. We could also pass to sd_journal_print which takes a va_list, however, the behavior isn't identical for structured log entries.

So my question is, if we lift the code from systemd and make it work, that'll presumably add another license piece to phosphor-logging --- and I'm not really sure how that works.

Quoted 31 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 30 lines
Quoted 8 lines

My tests shows that the blow change is related, after I revert it, I can get these log messages.

Quoted 6 lines

Did you meet such issue, or any suggestions/comments to narrow down this issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49

add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ? (so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging objects.

Quoted 5 lines

Thanks,

Yong