ADC
ADC / Citrix / NetScaler

Apply security update to Citrix ADC (CTX276688)

- by Vikash Jhagroe - Leave a Comment
Reading Time: 4 minutes

At the time of writing this blog post Citrix released information about 11 new vulnerabilities discovered in their NetScaler line of products including Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP. As sysadmins we want to keep our infrastructure secure and in this blog post I will show you how to apply the security update to the Citrix ADC (CTX276688) and mitigate for these vulnerabilities.

What to do now? Simple, upgrade to the latest version of Citrix ADC firmware. At the time of writing that is NS13.0 Build 58.32. Let me show you how.

If you followed my recent post Upgrade Citrix ADC firmware using CLI and you upgraded your Citrix ADC to version 13.0-58.30 according to my blog post then you should already be fine! Yay! The newly discovered vulnerabilities are documented in the Citrix Support Knowledge Center and is know by the number CTX276688. According to Citrix these vulnerabilities can only be exploited in very unique situations and circumstances and as far as they know, they are not yet used in the wild. I will show you how to upgrade Citrix ADC to the patched firmware and keep hackers at bay.

Preparation

We will use the same steps from my blog post Upgrade Citrix ADC firmware using CLI. The difference now is that I will do the upgrade from version 13.0 Build 58.30 to the latest release version 13.0 Build 58.32.

Download the latest firmware

According to Citrix we need to upgrade our ADC with version 13.0 Build 58.32. You can find that firmware here. If you open up that page you will see the important message regarding CTX276688:

Download the firmware and safe it to your computer. We will upload this file to the ADC and then start the upgrade.

Backup your current configuration

Before we start the upgrade process we need to backup the current configuration. You can do that using the steps I described here.

Start the upgrade

I am a big fan of CLI when it comes to upgrading these NetScaler appliances. But first we need to upload the new firmware to ADC. You can use the steps described here.

Start the upgrade script

Login to the ADC using SSH. I use Putty for this. After login in you need to go to the directory where you uploaded the new firmware file. If you followed my post that should be somewhere in /var/nsinstall/<directory_name>. I have uploaded my firmware to the directory /var/nsinstall/vikash.nl:

We can see the new firmware there. Now extract it using the following command:

tar –xvzf  build-13.0-58.32_nc_64.tgz

The tar -xvzf is the command you need to extract the file with name build-13.0-58.32_nc_64.tgz. Remember to replace the filename with the correct one. After the firmware is extracted you will have a lot of files there. The one we need is names installns. In your SSH / Putty session enter the following command to start the upgrade process:

./installns

The command start with a . yes. After typing in the command you will see the a similar screen like the one below indicating that the upgrade has started:

The update process will take a while. When the upgrade is finished you will be prompted to reboot the ADC. Enter Y and hit enter:

After the reboot let’s check the version and make sure everything went well. Open the management page (webgui) and check the firmware version:

You can check it using CLI. Login using SSH / Putty and enter the following command:

show ns version

You should get the following output displaying the firmware version of ADC:

If the version displays NS13.0 Build 58.32 then your ADC is protected from the vulnerabilities as described in CTX276688.

Steps for updating a Citrix ADC High Availability pair (HA)

The steps described in this blogpost apply in general for the ADC nodes which are running in a High Availability pair configuration. Every node can be upgraded using the same method I described in this post. Make sure you to upgrade the individual nodes in the following order:

  • Upgrade the secondary node first.
  • Reboot the secondary node.
  • Disable HA-sync on the secondary node using cli: set ha node -hasync disabled.
  • Upgrade the primary node.
  • Reboot the primary node.
  • Check that all the config is still there after the reboot of the primary node.
  • Enable HA-sync on the secondary node using cli: set ha node -hasync enabled.
Apply security update to Citrix ADC (CTX276688) Read More
ADC / Citrix / NetScaler

Upgrade Citrix ADC firmware using CLI

- by Vikash Jhagroe - 2 Comments.
Reading Time: 5 minutes

When running Citrix ADC it is vital to keep the ADC up-to-date. Usually Citrix ADC is very secure but every now and then they will discover bugs. That is when you need to update the firmware of the system. In this blog post I will show you how to upgrade Citrix ADC firmware using Command Line Interface (CLI).

Why do it using the CLI of there is a nice option in the webgui? In my experience doing it using the CLI is the most reliable way of getting the job done. The webgui is just not stable enough because on numerous occasions I have seen an upgrade fail when doing it using the webgui. And when such a system is running a crucial part of your infrastructure you don’t want to end up with a broken ADC. The CLI way has been rocksolid and delivers every time. It is not hard to do it if you follow the steps in this blog post.

Preparation

I will perform the upgrade on my ADC running in my lab environment. The version I am running here is 13.0 52.24. I will upgrade to the latest version. At the time of writing this post the latest version is 13.0 58.30.

Download the latest firmware

Download the latest firmware from the Citrix website. When you visit the website choose for the Firmware option:

Then on the next page scroll down to the Build section and download the latest firmware:

Backup the Citrix ADC configuration

I’m sure you already know this but often this step is still overlooked. Backing up the components in your network infrastructure is a vital part of running an IT-infrastructure. Your backup strategy for Citrix ADC depends in the platform you are running it on. I have mine running on Windows Hyper-V 2019 so making a snapshot before starting the upgrade is pretty handy. I will also show you how to make a backup of the ADC configuration from the webgui. Making a backup using the webgui has always worked in me experience so no need for CLI here.

Login to the webgui and in the left menu expand System and click on Backup:

Then click on Backup/Import button. You will be presented with several options. Enter a file name for the backup and something in the description that makes is easy to see why this backup was made. The most important part here is to select the Full backup level. Then click on Backup:

Now that the backup is made we need to download it from the ADC and keep is somewhere safe. Do this in case the upgrade does fail and you are not able to access the ADC using webgui of ssh. You will see an overview of all backups available on the appliance once you clicked on the Backup button as seen in the screenshot above. Select the backup you just made and from the action menu select Download to save the backup file to your local computer:

Start the upgrade

We have done our preperations and now we need to get the firmware we downloaded on the ADC and start the upgrade process.

Upload the new firmware

I use WinSCP to upload the new firmware to my ADC. Start WinSCP and login to your ADC using the option SFTP option:

After loggin in go to the /nsinstall directory and create a new directory there:

Upload the firmware using WinSCP:

Start the upgrade script

Login to the ADC using SSH. I use Putty for this. After login in you need to go to the directory where you uploaded the new firmware file. First enter the command shell to enter a shell:

Go to the directory where you uploaded the firmware file using WinSCP. On my ADC that is /var/nsinstall/vikash.nl:

We can see the new firmware there. Now extract it using the following command:

tar –xvzf build-13.0-58.30_nc_64.tgz

The tar -xvzf is the command you need to extract the file with name build-13.0-58.30_nc_64.tgz. Remember to replace the filename with the correct one. After the firmware is extracted you will have a lot of files there. The one we need is names installns. In your SSH / Putty session enter the following command to start the upgrade process:

./installns

The command start with a . yes. After typing in the command you will see the a similar screen like the one below indicating that the upgrade has started:

The update process will take a while. When the upgrade is finished you will be prompted to reboot the ADC. Enter Y and hit enter:

After the ADC has rebooted login using the webgui and check the firmware version to make sure the upgrade was successful:

How do I do it for ADC’s in a High Availability pair?

The steps described in this blogpost apply in general for the ADC nodes which are running in a High Availability pair configuration. Every node can be upgraded using the same method I described in this post. Make sure you to upgrade the individual nodes in the following order:

  • Upgrade the secondary node.
  • Reboot the secondary node.
  • Disable HA-sync on the secondary node using cli: set ha node -hasync disabled.
  • Upgrade the primary node.
  • Reboot the primary node.
  • Check that all the config is still there after the reboot of the primary node.
  • Enable HA-sync on the secondary node using cli: set ha node -hasync enabled.

Upgrade Citrix ADC firmware using CLI is not that hard if you prepare beforehand and make sure that you have backups. Even upgrading nodes in a High Availability configuration is easy once you follow the steps in the same order as I described above. Good luck and stay safe!

Upgrade Citrix ADC firmware using CLI Read More
ADC / Citrix / NetScaler

Enable secure LDAP for Citrix ADC with LDAP signature signing

- by Vikash Jhagroe - Leave a Comment
Reading Time: 6 minutes

The case

In this blog post I will show you how to enable secure LDAP for Citrix ADC with LDAP signature signing policy in order to tighten security in your network. As most of you know Microsoft will be retiring insecure LDAP communication on domain controllers. Check the Microsoft article here for an in-depth explanation. You can also check the following articles about LDAP signature signing:

  • CVE-2017-8563 – Windows Elevation of Privilege Vulnerability
  • ADV190023 – Microsoft Guidance for Enabling LDAP Channel Binding and LDAP Signing

So insecure communications to Active Directory is going away and we all need to switch the components in our network (which are talking to Active Directory for authentication) to use Secure LDAP (LDAPS). LDAPS uses SSL or TLS to encrypt traffic. Because the LDAPS traffic is encrypted we don’t need to worry about someone intercepting the traffic. If you intercept it, it’s encrypted so you won’t be able to read it. This change then is a good thing!

It so happens that a lot of ADC’s out there use insecure insecure LDAP to talk to domain controllers. As you can see in the screenshot below my own ADC has been setup to talk to my domain controller using insecure LDAP on port 389:

The screenshot above shows the basic LDAP server configuration pointing to my Active Directory domain controller. When I click the Test Network Connectivity button you see that everything is fine. This is the way a lot of these ADC’s are setup and when insecure LDAP to domain controllers is not working anymore somewhere during this year, this will break and that means that users will not be able to authenticate on you ADC and login to for instance Access Gateway VPN (if they are using that). In the end it will affect everything the ADC is providing your users with that require authentication from your Active Directory.

That’s why I will show you in this blog post how to get ahead of this change from Microsoft and prepare you Citrix ADC to Enable secure LDAP for Citrix ADC with LDAP signature signing.

What needs to be done

We need to reconfigure the ADC to use Secure LDAP (LDAPS). This can be done using that same insecure LDAP port (port 389) but tell the Citrix ADC to use TLS communication. Or you can choose to communicate to the Active Directory using port 636 and use the SSL option. The Active Directory only listens to LDAP with SSL encrypted traffic on port 636.

What group policy are we talking about

Ok all those articles are fine but let’s get pragmatic and let’s find the policy on the domain controller which will (somewhere this year) block insecure LDAP. In the end you want to know what will cause the problem and how to resolve it. Let me show you. To find the policy start mmc.exe on your Active Directory domain controller. Then click on File -> Add or Remove Snap-ins and find the Group Policy Management Editor. Click on Add:

Click on Browse:

Make sure you doubleclick the Domain Controllers folder here! You will see that the folder has the name of your domain in it. In my case it is Domain Controllers.vikash.nl:

Once you are in the folder you will find the policy we are looking for. The name should be Default Domain Controllers Policy. Select that and click on OK and then Finish. Finally click on OK and you should see the mmc showing you something like this:

The policy setting Microsoft is going to change in order to enforce Secure LDAP is named Domain controller: LDAP server signing requirements. Find it under Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> Security Options:

You can see that the setting here is None. This is the default setting and Microsoft eventually will change this to Require signing and in the basic this is all there is :). So for testing purpose let’s change the setting to Required signing and re-test our LDAP Server setting on the Citrix ADC. Note that when you change the setting to Require signing you will get a popup with a warning message. Click on Yes:

Now the setting is changed and Secure LDAP is being enforced because now it says Require signing. The policy setting will look like this:

Head over to the management webgui on the ADC and go to the Authentication LDAP Server page and click on Test Network connectivity. It will show an error which basically tells us that the Active Directory domain controller is not accepting insecure LDAP anymore:

Ok now we have the policy on the Active Directory to require LDAP signature signing and the expected error on the ADC when we make a PLAINTEXT LDAP request. Now I will show you how to fix this.

Domain Controller Certificate

Before we can do secure LDAP requests to our Active Directory Domain Controllers we have to make sure that the domain controller is using a Certificate. This is a requirement so make sure that this is working and in place. On my domain controller I am running AD Certificate Authority Role (CA):

The CA role allows me to easily issue certificates in my domain. In the screenshot below you can see that I have a certificate issued and activated on my domain controller:

Note that if you don’t have a certificate on your domain controller installed and active you will not be able to let the Citrix ADC do secure LDAP requests.

Enable Secure LDAP on Citrix ADC

Ok now we have our certificate setup on our domain controller and let’s continue to setup secure LDAP on ADC. You can do secure LDAP on port 389 with TLS or switch to port 636 with SSL. Please keep in mind that, depending on which of the below solutions you choose, you might have to adjust firewall rules.

Secure LDAP using port 389 with TLS

If your ADC is setup to use insecure LDAP it is doing this using port 389. You will see in the management webgui of the ADC that Security Type is set to PLAINTEXT:

So LDAP on port 389 can either be insecure or secure. Now we want it to be secure so the only thing you have to change here is set the Security Type to TLS. You won’t even have to change your firewall rules and everything will be running fine. So set the Security Type to TLS and click on Test Network Connectivity:

You should get all greens now and no errors. Make sure to press OK on the bottom of this screen to save your changes.

Secure LDAP using port 636 with SSL

The other option is to switch over to port 636 for LDAP requests. All LDAP communication on port 636 require encryption unlike LDAP on port 389. So on the ADC management webgui navigate to the LDAP configuration server and not set the Security Type to SSL. You will see that the ADC automatically changes the port to 636. Click on the Test Network Connectivity button and everything should be fine and green:

Note that in the green field it will say that LDAP communication is making use of port 636 now. Make sure to press OK on the bottom of this screen to save your changes.

To sum up

Using Secure LDAP on the Citrix ADC not only gives you a better security but also gives you other advantages like allowing password changes for users. If your users are able to do password changes already then chances are that you already have everything in place for that moment when Microsoft decides to enforce LDAP signature signing. And if not then I hope this blog post has helped you enable secure LDAP for Citrix ADC with LDAP signature signing. Be secure!

Enable secure LDAP for Citrix ADC with LDAP signature signing Read More