Grafana dashboard is very useful to check the hadoop services like hbase,hive,hdfs etc services load,utilization and other metrics. Recently we upgraded the Hadoop cluster from HDP 3.1.X to HDP 3.1.X, after that we observed Grafana dashboard not showing correct data.
In our cluster we have 4 region servers, but when we saw in grafana hbase dashboard it is showing 95 region servers. when i checked the hiveserver2 heap size utilization grafana hiveserver2 dashboard showing heap utilization is only in bytes, but heap utilization is in gbs.
So i decided to troubleshoot the grafana dashboard issue, I checked the ambari metrics, metrics collector and grafana services are running fine and all metrics moniters are running fine on all the nodes. I restarted the ambari metrics service and checked the dashboard but no use.
I thoroughly checked the metrics collector,grafana and metrics monitor logs, then also no use. Still grafana not showing correct data on dashboards.
I sign in to grafana web ui as admin user and checked the data sources, in our grafana URL value was empty, so thats the reason, ambari collector and other services are running fine but the data is showing the correct values on grafana dashboard.
Finally I followed below steps and fixed this issue.
- Open Grafana WebUI
- Sign in as admin and give password which you provided while installing the ambari metrics service
- Left side click on data sources
- In data sources page Name and Url of grafana data source are available
- If you not see any url details, left side click on edit option or click on add new option.
- Name : you can give any name which you like
- Type : under type we have multiple options like Ambari Metrics,Cloud watch,Elastic Search,Graphite,InfluxDB,KairosDB,OpenTsDB,Prometheus. For hadoop cluster we need to select Ambari Metrics option.
- Next Tick on default option
- Under Http settings you can see Url,Access and Http Auth options
- Url : you have to Ambari collector running host and port details, like http://ambari-metrics-host:6188
- Access : under access we have two options proxy and direct, select proxy.
- Http Auth : under Http Auth you can see options like Basic auth and with Credentials, please leave it both empty.
- Save the settings and click on Test connection to check the connection details.
- If connection is working, you entered all correct details and now grafana dashboard will show correct data.