Hadoop log4j日志說明
log4j.propertites # Define some default values that can be overridden by system properties
hadoop.root.logger=INFO,console
hadoop.log.dir=.
hadoop.log.file=hadoop.log
#
# Job Summary Appender
#
# Use following logger to send summary to separate file defined by
# hadoop.mapreduce.jobsummary.log.file rolled daily:
# hadoop.mapreduce.jobsummary.logger=INFO,JSA
#
hadoop.mapreduce.jobsummary.logger=${hadoop.root.logger}
hadoop.mapreduce.jobsummary.log.file=hadoop-mapreduce.jobsummary.log
# Define the root logger to the system property "hadoop.root.logger".
log4j.rootLogger=${hadoop.root.logger}, EventCounter
# Logging Threshold
log4j.threshhold=ALL
#
# Daily Rolling File Appender
#
log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFA.File=${hadoop.log.dir}/${hadoop.log.file}
# Rollver at midnight
log4j.appender.DRFA.DatePattern=.yyyy-MM-dd
# 30-day backup
#log4j.appender.DRFA.MaxBackupIndex=30
log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
# Pattern format: Date LogLevel LoggerName LogMessage
#log4j.appender.DRFA.layout.ConversionPattern=%l %m%n
# Debugging Pattern format 日志文件格式
log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
#
# console
# Add "console" to rootlogger above if you want to use this
#
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%l: %m%n
#
# TaskLog Appender
#
#Default values
hadoop.tasklog.taskid=null
hadoop.tasklog.iscleanup=false
hadoop.tasklog.noKeepSplits=4
hadoop.tasklog.totalLogFileSize=100
hadoop.tasklog.purgeLogSplits=true
hadoop.tasklog.logsRetainHours=12
log4j.appender.TLA=org.apache.hadoop.mapred.TaskLogAppender
log4j.appender.TLA.taskId=${hadoop.tasklog.taskid}
log4j.appender.TLA.isCleanup=${hadoop.tasklog.iscleanup}
log4j.appender.TLA.totalLogFileSize=${hadoop.tasklog.totalLogFileSize}
log4j.appender.TLA.layout=org.apache.log4j.PatternLayout
log4j.appender.TLA.layout.ConversionPattern=%l %p %c: %m%n
#
#Security audit appender
#
hadoop.security.log.file=SecurityAuth.audit
log4j.appender.DRFAS=org.apache.log4j.DailyRollingFileAppender
log4j.appender.DRFAS.File=${hadoop.log.dir}/${hadoop.security.log.file}
log4j.appender.DRFAS.layout=org.apache.log4j.PatternLayout
log4j.appender.DRFAS.layout.ConversionPattern=%l %p %c: %m%n
#new logger
log4j.logger.SecurityLogger=OFF,console
log4j.logger.SecurityLogger.additivity=false
#
# Rolling File Appender
#
#log4j.appender.RFA=org.apache.log4j.RollingFileAppender
#log4j.appender.RFA.File=${hadoop.log.dir}/${hadoop.log.file}
# Logfile size and and 30-day backups
#log4j.appender.RFA.MaxFileSize=1MB
#log4j.appender.RFA.MaxBackupIndex=30
#log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} - %m%n
#log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p %c{2} (%F:%M(%L)) - %m%n
#
# FSNamesystem Audit logging
# All audit events are logged at INFO level
#
log4j.logger.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit=DEBUG
# Custom Logging levels
hadoop.metrics.log.level=DEBUG
#log4j.logger.org.apache.hadoop.mapred.JobTracker=DEBUG
#log4j.logger.org.apache.hadoop.mapred.TaskTracker=DEBUG
#log4j.logger.org.apache.hadoop.fs.FSNamesystem=DEBUG
#應該是設置包下類的日志級別
log4j.logger.org.apache.hadoop.metrics2=${hadoop.metrics.log.level}
# Jets3t library
log4j.logger.org.jets3t.service.impl.rest.httpclient.RestS3Service=ERROR
#
# Null Appender
# Trap security logger on the hadoop client side
#
log4j.appender.NullAppender=org.apache.log4j.varia.NullAppender
#
# Event Counter Appender
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
#
log4j.appender.EventCounter=org.apache.hadoop.log.metrics.EventCounter
#
# Job Summary Appender
#
log4j.appender.JSA=org.apache.log4j.DailyRollingFileAppender
log4j.appender.JSA.File=${hadoop.log.dir}/${hadoop.mapreduce.jobsummary.log.file}
log4j.appender.JSA.layout=org.apache.log4j.PatternLayout
log4j.appender.JSA.layout.ConversionPattern=%l %p %c{2}: %m%n
log4j.appender.JSA.DatePattern=.yyyy-MM-dd
log4j.logger.org.apache.hadoop.mapred.JobInProgress$JobSummary=${hadoop.mapreduce.jobsummary.logger}
log4j.additivity.org.apache.hadoop.mapred.JobInProgress$JobSummary=false
hadoop日志級別設置
在hadoop/bin/ hadoop-daemon.sh文件下
export HADOOP_ROOT_LOGGER="DEBUG,DRFA"
自定義日志
目標：將需要的信息寫入自己指定的獨立的日志中。
需求：這次只是一個嘗試，在DFSClient中，將部分內容寫入指定的日志文件中。在客戶端讀取HDFS數據時，將讀的blockID寫入文件。
步驟：
1、修改hadoop/conf/log4j.properties文件。在文件末尾添加如下內容：
#為寫日志的操作取個名字，MyDFSClient。用來在DFSClient中獲取該日志的實例。并指定輸出方式為自定義的OUT
log4j.logger.MyDFSClient=DEBUG,OUT
#設置OUT的輸出方式為輸出到文件
log4j.appender.OUT=org.apache.log4j.FileAppender
#設置文件路徑
log4j.appender.OUT.File=${hadoop.log.dir}/DFSClient.log
#設置文件的布局
log4j.appender.OUT.layout=org.apache.log4j.PatternLayout
#設置文件的格式
log4j.appender.OUT.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
#設置該日志操作不與父類日志操作重疊
log4j.additivity.MyDFSClient=false
2、保存該文件，復制到集群各個節點的hadoop/conf目錄下，替換原有的文件。
3、修改DFSClient類
這里只是簡單的為了驗證這個過程的正確性，以后還回加入更有意義的日志內容。
首先在DFSClient類中聲明一個LOG實例：
public static final Log myLOG = LogFactory.getLog("MyDFSClient");
在read(byte buf[], int off, int len)函數中，添加如下代碼：
myLOG.info("Read Block!!!!");
if(currentBlock!=null)
myLOG.info("Read block: "+currentBlock.getBlockId());
4、重新啟動hadoop。
5、這里使用dfs命令進行測試。
$bin/hadoop dfs -cat /user/XXX/out/part-r-00000
可以看到文件part-r-00000的內容輸出到屏幕。這時在/hadoop/logs/DFSClient.log文件中，可以看到剛才在類中記錄的日志。驗證成功。