By a Habanero alumni

Distributed Cache needs ping

Summary

Issue

The Distributed Cache service does not install correctly on additional farm servers.

Symptoms

  1. When you join a server to the farm the Distributed Cache service on the server does not start. When you try to manually start or provision the service, you receive an error or the exception: 
    cacheHostInfo is null
  2. When you try create a new Distributed Cache instance on a server that is not part of the Distributed Cache cluster using the Add-SPDistributedCacheServiceInstance cmdlet you receive the exception: 
    ErrorCode<ERRCAdmin040>:SubStatus<ES0001>:Failed to connect to hosts in the cluster

In both cases:

  • The Distributed Cache service has been created and is running on one or more other servers in the farm
  • The AppFabric ports (TCP 22233-22236) are permitted between all servers in the farm
  • SharePoint has created a new Distributed Cache SPServiceInstance on the server, but it is Disabled
  • The AppFabric Windows service (AppFabric Caching Service) is not running on the server and has a Disabled startup type

Cause

Internet Control Message Protocol v4 (ICMPv4, or “ping”) traffic between the server and the first cache host in the farm is not permitted. The source of the blocked ICMP traffic could be due to:

  • One or more firewalls between SharePoint servers are not allowing ICMP traffic. e.g. a hardware firewall, Windows Firewall, or other software-based firewall
  • For servers in different networks, ICMP packets are not routed between the networks
  • Some other network policy that blocks ICMP traffic

Resolution

Allow ICMPv4 traffic between all servers running distributed cache and attempt recreating Distributed Cache instances on the additional servers or disconnecting and re-joining the servers to the farm.

Details

You've been selected to set up a new SharePoint Server 2013 farm to support a new company-wide portal. The stakeholders have a vision that the SharePoint farm will “never get hacked.” In an effort to achieve this goal, you've spent a considerable amount of time figuring out what you’ll need to do to harden SharePoint. Thankfully, there's the Plan security hardening for SharePoint 2013 TechNet article that details the networking and service requirements. In fact, you’ve spent so much time dissecting this guide that it's a mainstay of your most visited sites thumbnails when you open a new browser tab.

The guide details the requirements for Distributed Cache: Open the ports for AppFabric on the servers hosting the service and allow inbound connections. These are TCP ports 22233, 22234, 22235, and 22236 (i.e. TCP ports 22233-22236).

The day has come and you're setting up the farm. You start the process on one of your servers and by creating the configuration database and Central Administration site. Next you join some other servers to the farm without issue. You carry on setting up web applications and services.

You reach a point where you need to configure the Distributed Cache service. The first thing you want to do is change which servers are running the service. For some reason, you notice the only server running the service is the server you used to originally create the farm. This is unusual because normally Distributed Cache is created and started on a server when you join it to the farm unless you explicitly provide the -SkipRegisterAsDistributedCacheHost switch to the Connect-SPConfigurationDatabase cmdlet. Of course, in this case you did not use the switch. You expect to see Distributed Cache running on other servers.

The servers in the farm, Distributed Cache is not running on the second server (when it should)

So you click on the server and confirm the Distributed Cache service instance is stopped.

Distributed Cache is not running on this server

You click Start and after a few seconds it says there was an error.

Trying to start Distributed Cache results in an error!

If you try this in PowerShell (as you should have in the first place) you see the service instance exists, but it’s disabled.

Checking with PowerShell, Distributed Cache exists, but is disabled.

When you go to provision it, you get the excellent “cache host info is null” error which is the technical way to say the Distributed Cache configuration is messed up.

When you try to provision the service, you get the familiar yet useless exception 'cacheHostInfo is null'

At this point the only thing you think to do is to delete the service instance and manually create it again.

Delete the service instance:

Deleting the disabled Distributed Cache service instance

Add the instance by running the Add-SPDistributedCacheServiceInstance directly on the server:

When you try to add a new Distributed Cache service instance, you get the exception 'Failed to connect to hosts in the cluster'

And there we g...?

Failed to connect to hosts in the cluster? How can that be? In this case the servers are on the same network, they're even on the same VM host. We can use PortQry to validate the server can connect to the AppFabric ports:

Using Microsoft's Portqry tool we see the ports for Distributed Cache on the first server are open and listening

That checks out, the cache (22233), cluster (22234), and replication (22236) ports are listening so what’s the deal?

The Deal

The deal is there is a minimally documented requirement for the Distributed Cache service. Unfortunately this requirement is not mentioned in either the hardening guide or the Manage the Distributed Cache service in SharePoint Server 2013 articles. But it does appear in the final note at the very bottom of the Plan for feeds and the Distributed Cache service in SharePoint Server 2013 page:

If you are using more than one cache host in your server farm, you must configure the first cache host running the Distributed Cache service to allow Inbound ICMP (ICMPv4) traffic through the firewall ... If an administrator removes the first cache host from the cluster which was configured to allow Inbound ICMP (ICMPv4) traffic through the firewall, you must configure the first server of the new cluster to allow Inbound ICMP (ICMPv4) traffic through the firewall.

To set up Distributed Cache, the cache hosts must be able to ping the initial cache host. Normally this is the first server you set up in the farm provided you haven’t removed the service instance.

Sure enough, when we ping the server, it fails:

Can't ping the first cache host from our new server. There's something blocking ICMP.

The new server can’t ping the server that is already running Distributed Cache. In this case, Windows Firewall blocked incoming ICMPv4 ping requests. By creating a rule to allow ping to the server, it becomes possible to add a new Distributed Cache instance:

Now with allowing ICMP we can ping and add the Distributed Cache service instance

But it gets better. If you follow the documentation exactly and enable ICMP to only the first cache host and none of the others servers respond to pings, attempting to administer the AppFabric cluster won't work and says the other hosts are unavailable. If you then allow ping on the other hosts the instances appear online.

When ping is blocked to the other hosts, AppFabric doesn't know where the host is. Enable ICMP and now i sees it!

This means the actual networking requirements for Distributed Cache are allowing inbound TCP ports 22233-22236 and inbound ICMPv4 on all cache hosts in the farm.

Adding the service to a server that didn't have it to begin with

Let’s pretend you originally joined a server to the farm using the -SkipRegisterAsDistributedCacheHost switch and later decided you want to run Distributed Cache. If ICMP isn’t enabled on the first cache host you will encounter the issue as well. When you run Add-SPDistributedCacheServiceInstance you'll receive the “Failed to connect to hosts in the cluster” exception. The resolution is the same. Allow ICMP and retry.

In both scenarios you may need to delete and recreate the new service instance a number of times before it works. I find after enabling ICMP the first attempt doesn’t always succeed so I need to delete the instance and add it again.

Of course, if your SharePoint servers can ping each other before you join them, you’ll never run into this issue.

References

Stories say it best.

More stories

Are you ready to make your workplace awesome? We're keen to hear what you have in mind.

Let's talk!

Interested in learning more about the work we do?

Explore our culture and transformation services.

Our commitment to reconciliation

Learn how Habanero is responding to the Truth and Reconciliation Calls to Action as a settler-owned company operating on Indigenous territories across what is now called Canada.

Read about our commitment