Logging in Factomd

Secured
#1
Factomd is able to produce a vast quantity of logs to aid in debugging, information, and filling your harddrive. Getting to them is a different matter, as there are multiple different loggers.

standard out and standard error log
You can already pipe the stdout/stderr of factomd to a file via OS calls but if you want output on both the console AND a file, you can use these. Use command line --stdoutlog=filepath to specify an output file, and same for --stderrlog=filepath.

Additional Logging
Factomd uses a package called logrus to log stuff to standard out. You'll often find a declaration at the top of the file that looks like this:
Code:
var discoLogger = packageLogger.WithField("subpack", "discovery")
Inc. already has a very descriptive explanation of how they work: https://github.com/FactomProject/factomd/blob/master/LogInstructions.md

The key takeaway is that you can start factomd with the parameter --loglevel=lvl, where lvl can be debug, info, warning, error, or panic. Debug is exceptionally spammy and you won't be able to use the data in real time, so it's recommended to pair this with stdoutlog.

(Note: it seems that Inc. uses Logstash to parse these. See the command line options "logstash" and "logurl" to pipe these logs to Logstash. The docker test environment includes this.)


State Log files
These are a separate class of logfiles that write data to a cluster of files.

They're called either with state.LogPrintf(filename, format string, parameters...) for general log messages, or with state.LogMessage(filename, comment, IMsg) for logging related to specific messages. In the latter case, the filename usually corresponds to the queue the message came from.


Paired with the state logger comes the command line parameter debuglog. By default it's empty and thus doesn't match to anything. If you set it to . it will create all files. Please note that this is a case-insensitive regular expression, not a simple text search. Factomd won't boot up with an invalid expression.

The files will be created in the current working directory, and the possible filenames are:
Code:
fnode0_ackchange.txt
fnode0_apilog.txt
fnode0_balancehash.txt
fnode0_dbsig-eom.txt
fnode0_dbsig.txt
fnode0_dbstateprocess.txt
fnode0_duplicatesend.txt
fnode0_election.txt
fnode0_entrycredits_trans.txt
fnode0_entrysync.txt
fnode0_executemsg.txt
fnode0_factoids_trans.txt
fnode0_factoids.txt
fnode0_holding.txt
fnode0_inmsgqueue.txt
fnode0_missing_messages.txt
fnode0_msgqueue.txt
fnode0_networkoutputs.txt
fnode0_processlist.txt
fnode0_processstatus.txt
fnode0_process.txt
graphdata.txt
marshalsizes.txt

Note: If you change the config setting ControlPanelSetting to readwrite or run factomd with --controlpanelsetting=readwrite, you can change this setting on the fly via: control panel (port 8090) -> "More Detailed Node Information" -> "Log Settings". This is very much not recommended for production since entering an invalid regular expression will crash the node.

(Thanks to Veena Gondkar and Brian Deery for elaborating on these)


pprof
Factomd also opens up the standard port for pprof (6060), which allows you insight into memory and thread debugging. This is general golang stuff with plenty of resources out there on how to use it. I personally found https://jvns.ca/blog/2017/09/24/profiling-go-with-pprof/ to explain it fairly well.
 
Last edited: