logger_guide.md (6249B)
1 # Logging 2 3 The function `custom_logger` is a wrapper over the `Logging.jl` and `LoggingExtras.jl` libraries. 4 I made them such that I could fine tune the type of log I use repeatedly across projects. 5 6 The things I find most useful: 7 8 1. four different log files for each different level of logging from *error* to *debug* 9 2. six output formats: `:pretty` for the REPL, `:oneline`, `:json`, `:logfmt`, `:syslog`, and `:log4j_standard` for files 10 3. filtering out messages of verbose packages (`TranscodingStreams`, etc...) which sometimes slows down julia because of excessive logging 11 4. exact-level filtering so each file gets only its own level (or cascading for the old behavior) 12 5. thread-safe writes via per-stream locking 13 14 There are still a few things that might be useful down the line: 15 (1) a catch-all log file where filters do not apply; (2) filtering out specific functions of packages; 16 17 Overall this is working fine for me. 18 19 ## Basic usage 20 21 Say at the beginning of a script you would have something like: 22 ```julia 23 using BazerUtils 24 custom_logger("/tmp/log_test"; 25 filtered_modules_all=[:StatsModels, :TranscodingStreams, :Parquet2], 26 create_log_files=true, 27 overwrite=true, 28 log_format=:oneline); 29 30 ┌ Info: Creating 4 log files: 31 │ ⮑ /tmp/log_test_error.log 32 │ /tmp/log_test_warn.log 33 │ /tmp/log_test_info.log 34 └ /tmp/log_test_debug.log 35 ``` 36 37 The REPL will see all messages above debug level: 38 ```julia 39 > @error "This is an error level message" 40 ┌ [08:28:08 2025-02-12] Error | @ Main[REPL[17]:1] 41 └ This is an error level message 42 43 > @warn "This is an warn level message" 44 ┌ [08:28:08 2025-02-12] Warn | @ Main[REPL[18]:1] 45 └ This is an warn level message 46 47 > @info "This is an info level message" 48 ┌ [08:28:08 2025-02-12] Info | @ Main[REPL[19]:1] 49 └ This is an info level message 50 51 > @debug "This is an debug level message" 52 53 ``` 54 Then each of the respective log-levels will be redirected to the individual files. With the `:oneline` format they will look like: 55 ``` 56 [/home/user] 2025-02-12 08:28:08 ERROR Main[./REPL[17]:1] This is an error level message 57 [/home/user] 2025-02-12 08:28:08 WARN Main[./REPL[18]:1] This is an warn level message 58 [/home/user] 2025-02-12 08:28:08 INFO Main[./REPL[19]:1] This is an info level message 59 [/home/user] 2025-02-12 08:28:08 DEBUG Main[./REPL[20]:1] This is an debug level message 60 ``` 61 62 63 ## Options 64 65 ### Log Formats 66 67 The `log_format` kwarg controls how file logs are formatted. Default is `:oneline`. 68 The `log_format_stdout` kwarg controls REPL output. Default is `:pretty`. 69 70 All formats are available for both file and stdout. 71 72 | Format | Symbol | Best for | 73 |--------|--------|----------| 74 | Pretty | `:pretty` | Human reading in the REPL. Box-drawing characters + ANSI colors. | 75 | Oneline | `:oneline` | File logs. Single line with timestamp, level, module, file:line, message. | 76 | JSON | `:json` | Structured log aggregation (ELK, Datadog, Loki). One JSON object per line, zero external dependencies. | 77 | logfmt | `:logfmt` | Grep-friendly structured logs. `key=value` pairs, popular with Splunk/Heroku. | 78 | Syslog | `:syslog` | RFC 5424 syslog collectors. | 79 | Log4j Standard | `:log4j_standard` | Java tooling interop. Actual Apache Log4j PatternLayout with thread ID and milliseconds. | 80 81 Example: 82 ```julia 83 # JSON logs for a data pipeline 84 custom_logger("/tmp/pipeline"; 85 log_format=:json, 86 create_log_files=true, 87 overwrite=true) 88 89 # logfmt for grep-friendly output 90 custom_logger("/tmp/pipeline"; 91 log_format=:logfmt, 92 overwrite=true) 93 ``` 94 95 > **Deprecation note:** `:log4j` still works as an alias for `:oneline` but emits a deprecation warning. Use `:oneline` for the single-line format or `:log4j_standard` for the actual Apache Log4j format. 96 97 98 ### Level Filtering: `cascading_loglevels` 99 100 By default (`cascading_loglevels=false`), each file gets only messages at its exact level: 101 - `app_error.log` — only errors 102 - `app_warn.log` — only warnings 103 - `app_info.log` — only info 104 - `app_debug.log` — only debug 105 106 With `cascading_loglevels=true`, each file gets its level **and everything above**: 107 - `app_error.log` — only errors 108 - `app_warn.log` — warnings + errors 109 - `app_info.log` — info + warnings + errors 110 - `app_debug.log` — everything 111 112 ```julia 113 # Old cascading behavior 114 custom_logger("/tmp/log_test"; 115 create_log_files=true, 116 cascading_loglevels=true, 117 overwrite=true) 118 ``` 119 120 121 ### Files 122 123 The default is to write all levels to a single file. 124 Set `create_log_files=true` to create one file per level: 125 126 ```julia 127 # Single file (default) 128 custom_logger("/tmp/log_test"; overwrite=true) 129 130 # Separate files per level 131 custom_logger("/tmp/log_test"; 132 create_log_files=true, overwrite=true) 133 ``` 134 135 You can also select only specific levels: 136 ```julia 137 custom_logger("/tmp/log_test"; 138 create_log_files=true, 139 file_loggers=[:warn, :debug], # only warn and debug files 140 overwrite=true) 141 ``` 142 143 Use `overwrite=false` (the default) to append to existing log files across script runs. 144 145 146 ### Filtering 147 148 - `filtered_modules_specific::Vector{Symbol}`: filter modules from stdout and info-level file logs only. 149 Some packages write too much — filter them from info but still see them in debug. 150 - `filtered_modules_all::Vector{Symbol}`: filter modules from ALL logs. 151 Use for extremely verbose packages like `TranscodingStreams` that can slow down I/O. 152 153 ```julia 154 custom_logger("/tmp/log_test"; 155 filtered_modules_all=[:TranscodingStreams], 156 filtered_modules_specific=[:HTTP], 157 overwrite=true) 158 ``` 159 160 161 ### Thread Safety 162 163 All file writes are wrapped in per-stream `ReentrantLock`s. Multiple threads can log concurrently without interleaving output. In single-file mode, all levels share one lock. In multi-file mode, each file has its own lock so writes to different files don't block each other. 164 165 166 ## Other 167 168 For single-line formats (`:oneline`, `:logfmt`, `:syslog`, `:log4j_standard`), multi-line messages are collapsed to a single line: `\n` is replaced by ` | `. The `:json` format escapes newlines as `\n` in the JSON string. 169 170 There is also a path shortener (`shorten_path`) that reduces file paths. Options: `:relative_path` (default), `:truncate_middle`, `:truncate_to_last`, `:truncate_from_right`, `:truncate_to_unique`, `:no`.