网页资讯视频图片知道文库贴吧地图采购
进入贴吧全吧搜索

 
 
 
日一二三四五六
       
       
       
       
       
       

签到排名:今日本吧第个签到,

本吧因你更精彩,明天继续来努力!

本吧签到人数:0

一键签到
成为超级会员,使用一键签到
一键签到
本月漏签0次!
0
成为超级会员,赠送8张补签卡
如何使用?
点击日历上漏签日期,即可进行补签。
连续签到:天  累计签到:天
0
超级会员单次开通12个月以上,赠送连续签到卡3张
使用连续签到卡
03月19日漏签0天
hadoop吧 关注:15,531贴子:56,442
  • 看贴

  • 图片

  • 吧主推荐

  • 游戏

  • 1回复贴,共1页
<<返回hadoop吧
>0< 加载中...

在使用MultipleOutputs输出时的问题

  • 取消只看楼主
  • 收藏

  • 回复
  • 辣鸡网易05
  • 托儿所
    1
该楼层疑似违规已被系统折叠 隐藏此楼查看此楼
在使用MultipleOutputs输出时报could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation
即只能复制到0个节点,而不是minReplication(=1)。有3个数据节点正在运行,此操作中不排除任何节点。
1. 我的防火墙也关了
2. 在不使用MultipleOutputs的代码时没有任何问题
3. 在本地跑没有问题,在集群上一跑到就出了问题,疯狂报这个错
could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
4. 向HDFS上传文件等都没这个问题
查了好多资料有说空间不足的
但我的群集节点上的hdfs dfsadmin -report显示的空间应该是没问题的:
Configured Capacity: 143516123136 (133.66 GB)
Present Capacity: 109557473280 (102.03 GB)
DFS Remaining: 103351971840 (96.25 GB)
DFS Used: 6205501440 (5.78 GB)
DFS Used%: 5.66%
Under replicated blocks: 2
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
-------------------------------------------------
Live datanodes (3):
Name: 192.168.100.100:50010 (node01)
Hostname: node01
Decommission Status : Normal
Configured Capacity: 47838707712 (44.55 GB)
DFS Used: 2064539648 (1.92 GB)
Non DFS Used: 9394999296 (8.75 GB)
DFS Remaining: 33928613888 (31.60 GB)
DFS Used%: 4.32%
DFS Remaining%: 70.92%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Jan 29 02:18:12 CST 2020
Name: 192.168.100.102:50010 (node02)
Hostname: node02
Decommission Status : Normal
Configured Capacity: 47838707712 (44.55 GB)
DFS Used: 2069651456 (1.93 GB)
Non DFS Used: 8647630848 (8.05 GB)
DFS Remaining: 34670870528 (32.29 GB)
DFS Used%: 4.33%
DFS Remaining%: 72.47%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Jan 29 02:18:13 CST 2020
Name: 192.168.100.103:50010 (node03)
Hostname: node03
Decommission Status : Normal
Configured Capacity: 47838707712 (44.55 GB)
DFS Used: 2071310336 (1.93 GB)
Non DFS Used: 8564355072 (7.98 GB)
DFS Remaining: 34752487424 (32.37 GB)
DFS Used%: 4.33%
DFS Remaining%: 72.65%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Wed Jan 29 02:18:13 CST 2020
这是我的Driver代码
package com.czxy;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.NullWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
/**
* @Author 嗨皮骚
* @贴吧用户_00JG4KJ615 v 1.0
* @date 2020/2/14
*/
public class Driver extends Configured implements Tool {
@Override
public int run(String[] args) throws Exception {
Job job = Job.getInstance(super.getConf(), "xm");
job.setNumReduceTasks(3);
job.setJarByClass(Driver.class);
// job.setInputFormatClass(TextInputFormat.class);
FileInputFormat.setInputPaths(job,new Path("hdfs://node01:8020/cs/sj.txt"));
job.setMapperClass(Map.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(Text.class);
job.setReducerClass(Reduce.class);
job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(Text.class);
MultipleOutputs.addNamedOutput(job, "output", TextOutputFormat.class, NullWritable.class, Text.class);
// job.setOutputFormatClass(TextOutputFormat.class);
FileOutputFormat.setOutputPath(job,new Path("hdfs://node01:8020/cs/wc"));
return job.waitForCompletion(true)?0:-1;
}
public static void main(String[] args) throws Exception {
Configuration configuration = new Configuration();
int run = ToolRunner.run(configuration, new Driver(), args);
System.out.println(run==0?"成功":"失败");
System.exit(run);
}
}
求大佬帮忙分析分析这是什么问题


  • 辣鸡网易05
  • 托儿所
    1
该楼层疑似违规已被系统折叠 隐藏此楼查看此楼
报错信息
20/01/29 03:22:05 INFO mapreduce.Job: Task Id : attempt_1580235429588_0001_r_000001_2, Status : FAILED
Error: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /cs/wc/_temporary/1/_temporary/attempt_1580235429588_0001_r_000001_2/2014/03/12-r-00001 could only be replicated to 0 nodes instead of minReplication (=1). There are 3 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1724)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3448)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:690)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:217)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:506)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2281)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2277)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2275)
at org.apache.hadoop.ipc.Client.call(Client.java:1504)
at org.apache.hadoop.ipc.Client.call(Client.java:1441)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy14.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:423)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy15.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1858)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1654)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:788)


登录百度账号

扫二维码下载贴吧客户端

下载贴吧APP
看高清直播、视频!
  • 贴吧页面意见反馈
  • 违规贴吧举报反馈通道
  • 贴吧违规信息处理公示
  • 1回复贴,共1页
<<返回hadoop吧
分享到:
©2026 Baidu贴吧协议|隐私政策|吧主制度|意见反馈|网络谣言警示