Apply security update to Citrix ADC (CTX276688)

Reading Time: 4 minutes

At the time of writing this blog post Citrix released information about 11 new vulnerabilities discovered in their NetScaler line of products including Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP. As sysadmins we want to keep our infrastructure secure and in this blog post I will show you how to apply the security update to the Citrix ADC (CTX276688) and mitigate for these vulnerabilities.

What to do now? Simple, upgrade to the latest version of Citrix ADC firmware. At the time of writing that is NS13.0 Build 58.32. Let me show you how.

If you followed my recent post Upgrade Citrix ADC firmware using CLI and you upgraded your Citrix ADC to version 13.0-58.30 according to my blog post then you should already be fine! Yay! The newly discovered vulnerabilities are documented in the Citrix Support Knowledge Center and is know by the number CTX276688. According to Citrix these vulnerabilities can only be exploited in very unique situations and circumstances and as far as they know, they are not yet used in the wild. I will show you how to upgrade Citrix ADC to the patched firmware and keep hackers at bay.

Preparation

We will use the same steps from my blog post Upgrade Citrix ADC firmware using CLI. The difference now is that I will do the upgrade from version 13.0 Build 58.30 to the latest release version 13.0 Build 58.32.

Download the latest firmware

According to Citrix we need to upgrade our ADC with version 13.0 Build 58.32. You can find that firmware here. If you open up that page you will see the important message regarding CTX276688:

Download the firmware and safe it to your computer. We will upload this file to the ADC and then start the upgrade.

Backup your current configuration

Before we start the upgrade process we need to backup the current configuration. You can do that using the steps I described here.

Start the upgrade

I am a big fan of CLI when it comes to upgrading these NetScaler appliances. But first we need to upload the new firmware to ADC. You can use the steps described here.

Start the upgrade script

Login to the ADC using SSH. I use Putty for this. After login in you need to go to the directory where you uploaded the new firmware file. If you followed my post that should be somewhere in /var/nsinstall/<directory_name>. I have uploaded my firmware to the directory /var/nsinstall/vikash.nl:

We can see the new firmware there. Now extract it using the following command:

tar –xvzf  build-13.0-58.32_nc_64.tgz

The tar -xvzf is the command you need to extract the file with name build-13.0-58.32_nc_64.tgz. Remember to replace the filename with the correct one. After the firmware is extracted you will have a lot of files there. The one we need is names installns. In your SSH / Putty session enter the following command to start the upgrade process:

./installns

The command start with a . yes. After typing in the command you will see the a similar screen like the one below indicating that the upgrade has started:

The update process will take a while. When the upgrade is finished you will be prompted to reboot the ADC. Enter Y and hit enter:

After the reboot let’s check the version and make sure everything went well. Open the management page (webgui) and check the firmware version:

You can check it using CLI. Login using SSH / Putty and enter the following command:

show ns version

You should get the following output displaying the firmware version of ADC:

If the version displays NS13.0 Build 58.32 then your ADC is protected from the vulnerabilities as described in CTX276688.

Steps for updating a Citrix ADC High Availability pair (HA)

The steps described in this blogpost apply in general for the ADC nodes which are running in a High Availability pair configuration. Every node can be upgraded using the same method I described in this post. Make sure you to upgrade the individual nodes in the following order:

  • Upgrade the secondary node first.
  • Reboot the secondary node.
  • Disable HA-sync on the secondary node using cli: set ha node -hasync disabled.
  • Upgrade the primary node.
  • Reboot the primary node.
  • Check that all the config is still there after the reboot of the primary node.
  • Enable HA-sync on the secondary node using cli: set ha node -hasync enabled.
Apply security update to Citrix ADC (CTX276688) Read More

Upgrade Citrix ADC firmware using CLI

Reading Time: 5 minutes

When running Citrix ADC it is vital to keep the ADC up-to-date. Usually Citrix ADC is very secure but every now and then they will discover bugs. That is when you need to update the firmware of the system. In this blog post I will show you how to upgrade Citrix ADC firmware using Command Line Interface (CLI).

Why do it using the CLI of there is a nice option in the webgui? In my experience doing it using the CLI is the most reliable way of getting the job done. The webgui is just not stable enough because on numerous occasions I have seen an upgrade fail when doing it using the webgui. And when such a system is running a crucial part of your infrastructure you don’t want to end up with a broken ADC. The CLI way has been rocksolid and delivers every time. It is not hard to do it if you follow the steps in this blog post.

Preparation

I will perform the upgrade on my ADC running in my lab environment. The version I am running here is 13.0 52.24. I will upgrade to the latest version. At the time of writing this post the latest version is 13.0 58.30.

Download the latest firmware

Download the latest firmware from the Citrix website. When you visit the website choose for the Firmware option:

Then on the next page scroll down to the Build section and download the latest firmware:

Backup the Citrix ADC configuration

I’m sure you already know this but often this step is still overlooked. Backing up the components in your network infrastructure is a vital part of running an IT-infrastructure. Your backup strategy for Citrix ADC depends in the platform you are running it on. I have mine running on Windows Hyper-V 2019 so making a snapshot before starting the upgrade is pretty handy. I will also show you how to make a backup of the ADC configuration from the webgui. Making a backup using the webgui has always worked in me experience so no need for CLI here.

Login to the webgui and in the left menu expand System and click on Backup:

Then click on Backup/Import button. You will be presented with several options. Enter a file name for the backup and something in the description that makes is easy to see why this backup was made. The most important part here is to select the Full backup level. Then click on Backup:

Now that the backup is made we need to download it from the ADC and keep is somewhere safe. Do this in case the upgrade does fail and you are not able to access the ADC using webgui of ssh. You will see an overview of all backups available on the appliance once you clicked on the Backup button as seen in the screenshot above. Select the backup you just made and from the action menu select Download to save the backup file to your local computer:

Start the upgrade

We have done our preperations and now we need to get the firmware we downloaded on the ADC and start the upgrade process.

Upload the new firmware

I use WinSCP to upload the new firmware to my ADC. Start WinSCP and login to your ADC using the option SFTP option:

After loggin in go to the /nsinstall directory and create a new directory there:

Upload the firmware using WinSCP:

Start the upgrade script

Login to the ADC using SSH. I use Putty for this. After login in you need to go to the directory where you uploaded the new firmware file. First enter the command shell to enter a shell:

Go to the directory where you uploaded the firmware file using WinSCP. On my ADC that is /var/nsinstall/vikash.nl:

We can see the new firmware there. Now extract it using the following command:

tar –xvzf build-13.0-58.30_nc_64.tgz

The tar -xvzf is the command you need to extract the file with name build-13.0-58.30_nc_64.tgz. Remember to replace the filename with the correct one. After the firmware is extracted you will have a lot of files there. The one we need is names installns. In your SSH / Putty session enter the following command to start the upgrade process:

./installns

The command start with a . yes. After typing in the command you will see the a similar screen like the one below indicating that the upgrade has started:

The update process will take a while. When the upgrade is finished you will be prompted to reboot the ADC. Enter Y and hit enter:

After the ADC has rebooted login using the webgui and check the firmware version to make sure the upgrade was successful:

How do I do it for ADC’s in a High Availability pair?

The steps described in this blogpost apply in general for the ADC nodes which are running in a High Availability pair configuration. Every node can be upgraded using the same method I described in this post. Make sure you to upgrade the individual nodes in the following order:

  • Upgrade the secondary node.
  • Reboot the secondary node.
  • Disable HA-sync on the secondary node using cli: set ha node -hasync disabled.
  • Upgrade the primary node.
  • Reboot the primary node.
  • Check that all the config is still there after the reboot of the primary node.
  • Enable HA-sync on the secondary node using cli: set ha node -hasync enabled.

Upgrade Citrix ADC firmware using CLI is not that hard if you prepare beforehand and make sure that you have backups. Even upgrading nodes in a High Availability configuration is easy once you follow the steps in the same order as I described above. Good luck and stay safe!

Upgrade Citrix ADC firmware using CLI Read More

Use Pi-hole with Microsoft Active Directory

Reading Time: 6 minutes

I’m a big fan of Pi-hole and have been using it to get rid of advertisement and tracking. Check my blogpost here if you want to know how to set Pi-hole up. It’s an amazing piece of software to protect your online privacy and provide network wide ad-blocking. In my day job I’m an IT-consultant for enterprise IT-solutions and in this post I will show you how to use Pi-hole with Microsoft Active Directory and protect all your domain joined clients from advertisement, tracking and also keep your clients secure from those malware websites.

Of course, you need to test this extensively before rolling it out in your infrastructure. I cannot stress this enough. The solution described in this blogpost did not show any kind of strange unexpected behaviour in my testlab but every infrastructure is different. Especially with endusers and applications there may be some challenges. So test before you implement!

Requirements

Microsoft Active Directory depends on Active Directory-Integrated DNS Service and Active Directory-Integrated DHCP Service. In this scenario all your domain joined clients are getting their IP-addresses and DNS settings from the Microsoft DHCP server. The DNS settings is used by the domain joined clients to talk to the Active Directory for DNS lookups and Active Directory related tasks. My testlab is running on Windows Server 2019 Active Directory and DNS Service, but this should also work if you are running a Windows Server 2016 environment. The requirement list is:

  • Microsoft Windows Server 2019
  • Microsoft Active Directory 2019
  • Microsoft Active Directory-Integrated DNS 2019
  • Microsoft Active Directory-Integrated DHCP Server 2019
  • Pi-hole Server
  • Domain joined client(s)

Let’s get started

They key Pi-hole feature we will be using in order to get this working is called Conditional Forwarding. I will explain in this post later on how we will use this feature.

DHCP Server settings

My DHCP Server is running on my Active Directory Domain controller. I’m sure a lot of you have the same setup which is fine. In the DHCP Server we have to specify certain options like DNS Servers and DNS Domain Name. My DHCP server is running on IP-address 192.168.130.10. My DNS Domain Name is vikash.nl. For DNS Servers fill in the IP-address of your Pi-hole Server. My Pi-hole server is running on IP-address 192.168.100.21.

On your DHCP server open the management console for DHCP Server and expand the scope options. Make sure the values match your network infrastructure:

Pi-hole Server settings

Now I will show you how to use Pi-hole with Microsoft Active Directory. The idea here is provide the Pi-hole Server as the DNS server to your domain joined clients. Then in the Pi-hole Server settings we will enable the option called Conditional Forwarding. Here we have to enter the IP-address of our Active Directory-Integrated DHCP server and also a Local Domain Name. This local domain name has to be your Active Directory name. In my case that is vikash.nl. What will happen now is that if the Pi-hole gets DNS requests from clients that need to resolve something.vikash.nl it will forward that request to our DHCP server which is also our Active Directory Domain controller. This makes sure that all the Active Directory related communications between my domain joined clients and Active Directory are completed successfully.

On the Pi-hole server go to Settings and select the DNS tab:

As you can see in the screenshot above I am using Cloudflare DNS Servers as my Upstream DNS. You can use any DNS Server as your upstream DNS. This basically means that for all DNS requests not related to vikash.nl the Pi-hole server will resolve those using Cloudflare. That is exactly what we want because it will make sure that internet is still working for all our domain joined clients. At the same time we will be able to see all the DNS requests in the Pi-hole Server Query Log for every client. This gives us control to protect our domain joined clients from ads, tracking or even malware.

In the DNS tab scroll to the bottom of the page and enter the DHCP server IP-address and the Local Domain Name. My DHCP server is 192.168.130.10 and my Local Domain Name is vikash.nl. Check your network infrastructure for your specific settings and click Save:

Testing

Now let’s make sure that everything works. First we will check that the correct DHCP settings are distributed to a client we want to join to the domain vikash.nl. I will use a Windows Server 2019 as client with the name vdi01.

Check IP-address

Open up a command prompt on the machine and make sure that the client is getting the correct settings from the DHCP server:

As you can see in the screenshot above the client is getting the DNS Domain Name and the DNS Server settings according to our scope options in the DHCP server. Check that the client is not already domain joined:

Join the client to the domain

Next step is to join the client (my vdi01) to my domain vikash.nl. Click in the Server Manager on WORKGROUP and then click on Change in the window that pops up:

Select the Domain option here and enter your domain name. Remember that this must be the same as DNS Domain Name entered in the DHCP Scope options and in the Conditional Forwarding on the Pi-hole. In my case this is vikash.nl. Then click on OK.

Windows will prompt you to enter Domain credentials which are allowed to do a domain join. In my testlab I use the domain administrator account for that. Enter the credentials and click on OK:

You will get a prompt from Windows telling you that the domain join was completed successfully. It looks like everything is working :). Click on OK and reboot you client.

After the client reboots login using a domain account:

Check that everything is ok and the client is a member of the domain:

Check Pi-hole Query Log

We can see the magic happening when we check the Query Log on our Pi-hole Server. Open the admin page of Pi-hole server and select the Query Log in the left menu:

As you can see in the screenshot above my client with IP-address 192.168.130.211 (vdi01.vikash.nl) is able to resolve internet queries as wel as queries related to my domain vikash.nl. Filesharing is working fine as well:

How amazing is this?! We are using Pi-hole with Microsoft Active Directory infrastructure and that means that we can now benefit from the protection of Pi-hole on enterprise level :). Of course this test is limited but imagine the possibilities. You can now provide all your endusers with a ad-free and tracking free internet experience but still be in control if some specific website needs to be unblocked.

Use Pi-hole with Microsoft Active Directory Read More

Exclude client devices with Pi-hole 5

Reading Time: 4 minutes

I am a big fan of Pi-hole and I recommend it to everyone. It is an amazing piece of software to get rid of advertisement and tracking on a network level and recently Pi-hole version 5 was released. Check my blogpost here if you want to know how to set it up. That blogpost is based on version 4 of Pi-hole but the same applies for version 5. Just follow the steps there to secure your network and take back your online privacy. Pi-hole 5 has a lot of new features but the one I want to talk about is how to exclude client devices with Pi-hole 5.

Use case

Being able to exclude individual client devices can be extreme useful during troubleshooting. There may be times that you want to bypass the ad-blocking capabilities of Pi-hole like for IoT devices. Many IoT devices are connected to some cloud solution, especially if they are using Apple HomeKit. I’ve had many IoT devices go offline because Pi-hole was blocking them and I did not want to have to whitelist all those domains. My IoT devices are on a separate VLAN and I want them to use my Pi-hole as DNS server but I don’t want anything blocked for them. Pi-hole 5 makes that possible without jumping trough any hoops.

Let’s get started

Excluding client devices with Pi-hole 5 is done using Group Management. After installing Pi-hole a default group is created. Blocklists are now called Adlists and all the adlists you add are added to the default group called Default. Check the screenshot below:

As you can see we can now also add a comment to an adlist :). Very nice for documentation purposes. Mine says Migrated from /etc/pihole/adlists.list because my Pi-hole was upgraded from version 4 to version 5. That comment is automatically added during the upgrade proces.

Create a new Group

The first this we need to do is create a new group. Go to Group Management and click on Group. Enter a name and description and click on Add.

Make sure the List of configured groups show the new group you added:

Check group assignment

Now we have to make sure that the new Exclude_Group group we created does not have adlists assigned to it.

Go to Group Management -> Adlists and check the Group Assigment column. In the above screenshot you can see that I have all my adlists assigned to the Default group. Next we can add client devices to the Exclude_Group group. Every client device added to this group will have no adlists because all our adlists are assigned to the Default group.

Adding client devices

Go to Group Management -> Clients. Find the IP address of the client device on the dropdown menu. You can also enter a custom IP address. My client device has IP address 192.168.100.185. Enter a Comment and then click on Add.

Not that after adding the client device it will automatically be added to the Default group:

Change the group to exclude client device

All we have to do now is change the Group assignment for the client device to the group we created earlier on. It is important to deselect the Default group! We only want the client device with IP address 192.168.100.185 be member of the group Exclude_Group. Rember that the Exclude_Group does not have adlists assigned so any member of that group will still use Pi-hole as DNS server without the blocking functionality.

After you have made sure that the client device is only member of the Exclude_Group click on Apply. Your screen should look something like this:

Do some testing

Now that my client device with IP address 192.168.100.185 is excluded we can do some testing. Opening a browser of my client device and visiting https://www.google.com shows the following in the query log of Pi-hole:

Note that the following DNS request is now allowed: adservice.google.com

I know that my exclusion is working because adservice.google.com is on several adlists I use:

If I change the group of this client device back to Default we will observe the following behaviour:

Well, and that is all there is if you want to exclude client devices with Pi-hole 5 blocking especially if you find that after implementing Pi-hole (or adlists) something broke in your network. Really helpful I’d say.

Exclude client devices with Pi-hole 5 Read More

Enable secure LDAP for Citrix ADC with LDAP signature signing

Reading Time: 6 minutes

The case

In this blog post I will show you how to enable secure LDAP for Citrix ADC with LDAP signature signing policy in order to tighten security in your network. As most of you know Microsoft will be retiring insecure LDAP communication on domain controllers. Check the Microsoft article here for an in-depth explanation. You can also check the following articles about LDAP signature signing:

  • CVE-2017-8563 – Windows Elevation of Privilege Vulnerability
  • ADV190023 – Microsoft Guidance for Enabling LDAP Channel Binding and LDAP Signing

So insecure communications to Active Directory is going away and we all need to switch the components in our network (which are talking to Active Directory for authentication) to use Secure LDAP (LDAPS). LDAPS uses SSL or TLS to encrypt traffic. Because the LDAPS traffic is encrypted we don’t need to worry about someone intercepting the traffic. If you intercept it, it’s encrypted so you won’t be able to read it. This change then is a good thing!

It so happens that a lot of ADC’s out there use insecure insecure LDAP to talk to domain controllers. As you can see in the screenshot below my own ADC has been setup to talk to my domain controller using insecure LDAP on port 389:

The screenshot above shows the basic LDAP server configuration pointing to my Active Directory domain controller. When I click the Test Network Connectivity button you see that everything is fine. This is the way a lot of these ADC’s are setup and when insecure LDAP to domain controllers is not working anymore somewhere during this year, this will break and that means that users will not be able to authenticate on you ADC and login to for instance Access Gateway VPN (if they are using that). In the end it will affect everything the ADC is providing your users with that require authentication from your Active Directory.

That’s why I will show you in this blog post how to get ahead of this change from Microsoft and prepare you Citrix ADC to Enable secure LDAP for Citrix ADC with LDAP signature signing.

What needs to be done

We need to reconfigure the ADC to use Secure LDAP (LDAPS). This can be done using that same insecure LDAP port (port 389) but tell the Citrix ADC to use TLS communication. Or you can choose to communicate to the Active Directory using port 636 and use the SSL option. The Active Directory only listens to LDAP with SSL encrypted traffic on port 636.

What group policy are we talking about

Ok all those articles are fine but let’s get pragmatic and let’s find the policy on the domain controller which will (somewhere this year) block insecure LDAP. In the end you want to know what will cause the problem and how to resolve it. Let me show you. To find the policy start mmc.exe on your Active Directory domain controller. Then click on File -> Add or Remove Snap-ins and find the Group Policy Management Editor. Click on Add:

Click on Browse:

Make sure you doubleclick the Domain Controllers folder here! You will see that the folder has the name of your domain in it. In my case it is Domain Controllers.vikash.nl:

Once you are in the folder you will find the policy we are looking for. The name should be Default Domain Controllers Policy. Select that and click on OK and then Finish. Finally click on OK and you should see the mmc showing you something like this:

The policy setting Microsoft is going to change in order to enforce Secure LDAP is named Domain controller: LDAP server signing requirements. Find it under Computer Configuration -> Policies -> Windows Settings -> Security Settings -> Local Policies -> Security Options:

You can see that the setting here is None. This is the default setting and Microsoft eventually will change this to Require signing and in the basic this is all there is :). So for testing purpose let’s change the setting to Required signing and re-test our LDAP Server setting on the Citrix ADC. Note that when you change the setting to Require signing you will get a popup with a warning message. Click on Yes:

Now the setting is changed and Secure LDAP is being enforced because now it says Require signing. The policy setting will look like this:

Head over to the management webgui on the ADC and go to the Authentication LDAP Server page and click on Test Network connectivity. It will show an error which basically tells us that the Active Directory domain controller is not accepting insecure LDAP anymore:

Ok now we have the policy on the Active Directory to require LDAP signature signing and the expected error on the ADC when we make a PLAINTEXT LDAP request. Now I will show you how to fix this.

Domain Controller Certificate

Before we can do secure LDAP requests to our Active Directory Domain Controllers we have to make sure that the domain controller is using a Certificate. This is a requirement so make sure that this is working and in place. On my domain controller I am running AD Certificate Authority Role (CA):

The CA role allows me to easily issue certificates in my domain. In the screenshot below you can see that I have a certificate issued and activated on my domain controller:

Note that if you don’t have a certificate on your domain controller installed and active you will not be able to let the Citrix ADC do secure LDAP requests.

Enable Secure LDAP on Citrix ADC

Ok now we have our certificate setup on our domain controller and let’s continue to setup secure LDAP on ADC. You can do secure LDAP on port 389 with TLS or switch to port 636 with SSL. Please keep in mind that, depending on which of the below solutions you choose, you might have to adjust firewall rules.

Secure LDAP using port 389 with TLS

If your ADC is setup to use insecure LDAP it is doing this using port 389. You will see in the management webgui of the ADC that Security Type is set to PLAINTEXT:

So LDAP on port 389 can either be insecure or secure. Now we want it to be secure so the only thing you have to change here is set the Security Type to TLS. You won’t even have to change your firewall rules and everything will be running fine. So set the Security Type to TLS and click on Test Network Connectivity:

You should get all greens now and no errors. Make sure to press OK on the bottom of this screen to save your changes.

Secure LDAP using port 636 with SSL

The other option is to switch over to port 636 for LDAP requests. All LDAP communication on port 636 require encryption unlike LDAP on port 389. So on the ADC management webgui navigate to the LDAP configuration server and not set the Security Type to SSL. You will see that the ADC automatically changes the port to 636. Click on the Test Network Connectivity button and everything should be fine and green:

Note that in the green field it will say that LDAP communication is making use of port 636 now. Make sure to press OK on the bottom of this screen to save your changes.

To sum up

Using Secure LDAP on the Citrix ADC not only gives you a better security but also gives you other advantages like allowing password changes for users. If your users are able to do password changes already then chances are that you already have everything in place for that moment when Microsoft decides to enforce LDAP signature signing. And if not then I hope this blog post has helped you enable secure LDAP for Citrix ADC with LDAP signature signing. Be secure!

Enable secure LDAP for Citrix ADC with LDAP signature signing Read More

Setup WireGuard client on iPhone

Reading Time: 3 minutes

In my blog post here I showed you how to setup and configure WireGuard VPN Server side. I that blog post I also tell you what WireGuard is and what the benefits are. Everything in this blog post continues based on the configuration in my previous blog and in this blog post I will show you how to setup WireGuard client on iPhone.

Requirement for this blog: Setup WireGuard VPN server by me.

First we need to download WireGuard for iOS from the Apple App Store. You can do that on your iPhone or click on the link here which will take you directly to the WireGuard app in the App Store.

For your iPhone visit the App Store and search and download WireGuard:

Setup WireGuard client on iPhone - vikash.nl

Adding the configuration to the iPhone app is a lot less steps compared to the Windows client. We will generate a QR code on the WireGuard server (the DietPi) and simply scan that QR code on the iPhone WireGuard app.

To do that connect to your DietPi server using SSH and go to the /etc/wireguard directory. There you should have the configuration files for you server and your clients if you followed my blog post about setting up your own WireGuard VPN Server. Let check if everything is there with the ls -l command:

Setup WireGuard client on iPhone - vikash.nl

As you can see there I named my client configuration file iphone.conf. To generate the QR code typ the following command:

grep -v '^#' /etc/wireguard/iphone.conf | qrencode -t ansiutf8

A QR code will be generated in the terminal:

Before you do this you have to make sure that your iPhone is connected to the same network as the WireGuard VPN Server. Open the WireGuard app on your iPhone and tap Add a Tunnel and then select the Create from QR Code option:

wireguard_ios_vikash.nl_03

Scan the QR code with the camera and give the tunnel a name:

Setup WireGuard client on iPhone - vikash.nl

Enter your iPhone pincode to add the tunnel:

The iPhone will now automatically add the VPN tunnel in the settings app and then switch to the WireGuard VPN app. There you will see the VPN tunnel:

That’s all for adding a WireGuard tunnel to your iPhone.

Setup WireGuard client on iPhone Read More