improve logging: add dates to verbose stderr, match file log level to verbosity (#1772)

## Summary

- **Add ISO date to verbose stderr format**: when running with `-v`, the
stderr timestamp changes from `HH:mm:ss.SSS` to `YYYY-MM-DD
HH:mm:ss.SSS`, matching the file log format. This makes it possible to
correlate entries across days when stderr is captured by launchd,
systemd, Docker, or file redirection.
- **File log respects verbosity**: `exo.log` now uses DEBUG level when
`-v` is passed, so the persistent log has the same detail as stderr.
Previously it was always INFO regardless of verbosity.
- **Startup banner with PID**: adds a visual separator and process ID to
the startup message, making it easy to identify session boundaries in
long-running logs (e.g. `grep "Starting EXO"`).

The non-verbose (default) stderr format is unchanged — end-user terminal
experience is not affected.

## Motivation

When exo runs as a service (launchd, systemd, Docker), stderr is
typically captured to a file. Without calendar dates in the timestamp,
it is impossible to tell which day a log entry belongs to. This caused
misidentification of log entries during a multi-day RDMA debugging
session on a 4-node cluster.

The file log (`exo.log`) already had dates and rotation, but only
captured INFO level — missing the DEBUG output needed for postmortem
analysis.

## Test plan

- [x] `basedpyright` — 0 errors, 0 warnings, 0 notes
- [x] `ruff check` — all checks passed
- [x] `pytest` — 249 passed, 1 skipped

Co-authored-by: user <[email protected]>
This commit is contained in:
vskiwi
2026-03-23 16:06:26 +00:00
committed by GitHub
co-authored by user
parent 07598a3af1
commit e9fdd8d4af
2 changed files with 5 additions and 3 deletions
+3 -1
View File
@@ -264,7 +264,9 @@ def main():
mp.set_start_method("spawn", force=True)
# TODO: Refactor the current verbosity system
logger_setup(EXO_LOG, args.verbosity)
logger.info("Starting EXO")
logger.info(f"{'=' * 40}")
logger.info(f"Starting EXO | pid={os.getpid()}")
logger.info(f"{'=' * 40}")
logger.info(f"EXO_LIBP2P_NAMESPACE: {os.getenv('EXO_LIBP2P_NAMESPACE')}")
if args.offline:
+2 -2
View File
@@ -66,7 +66,7 @@ def logger_setup(log_file: Path | None, verbosity: int = 0):
else:
logger.add(
sys.__stderr__, # type: ignore
format="[ {time:HH:mm:ss.SSS} | <level>{level: <8}</level> | {name}:{function}:{line} ] <level>{message}</level>",
format="[ {time:YYYY-MM-DD HH:mm:ss.SSS} | <level>{level: <8}</level> | {name}:{function}:{line} ] <level>{message}</level>",
level="DEBUG",
colorize=True,
enqueue=True,
@@ -76,7 +76,7 @@ def logger_setup(log_file: Path | None, verbosity: int = 0):
logger.add(
log_file,
format="[ {time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | {name}:{function}:{line} ] {message}",
level="INFO",
level="DEBUG" if verbosity > 0 else "INFO",
colorize=False,
enqueue=True,
rotation=lambda _, __: next(rotate_once),