log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'publicFile' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'privateFile' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'product' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'metrics' log4j:WARN Failed to set property [rollingPolicy] to value "org.apache.log4j.rolling.TimeBasedRollingPolicy". log4j:WARN Please set a rolling policy for the DatabricksRollingFileAppender named 'usage

2

2 Answers

As of Datatbricks runtime 11.0 with Spark 3.3.0, log4j has been upgraded to log4j2. I haven't seen official documentation for custom log support but this is what's been working for me. Similar with log4j 1 support, you need a custom init script to run on the cluster and create a log4j2.properties file. This script might look something like:

#! /bin/bash set -euxo pipefail echo "Running on the driver? ${DB_IS_DRIVER}" echo "Driver ip: ${DB_DRIVER_IP}" cat >>/databricks/spark/dbconf/log4j/driver/log4j2.properties <<EOL appender.customFile.type = RollingFile appender.customFile.name = customFile appender.customFile.layout.type = PatternLayout appender.customFile.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex appender.customFile.filePattern = logs/log4j.custom.%d{yyyy-MM-dd-HH}.log.gz appender.customFile.policies.type = Policies appender.customFile.policies.time.type = TimeBasedTriggeringPolicy appender.customFile.policies.time.interval = 1 appender.customFile.fileName = logs/stdout.custom-active.log logger.custom=DEBUG, customFile logger.custom.name = com.custom logger.custom.additivity = true EOL

In databricks , you can check the complete details on databricks with log4j2 in here

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password

Post as a guest

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.