All About Citrixnetscaler



Citrix Cloud Services hosts a suite of services provided by Virtual Apps and Desktop service, Citrix Gateway service, ShareFile, and so forth. All these services are delivered in a single pane using Workspace experience. Salient features of Citrix Gateway service. Some of the notable features of the Citrix Gateway service are as follows. The following operations can be performed on 'shell': shell¶. Exits to the FreeBSD command prompt. Press Control + D or type exit to return to the NetScaler command prompt.Note: The shell can be accessed only by users who have write access to the NetScaler appliance.

  1. NetScaler licenses are assigned to physical (MPX) and virtual (VPX) appliances. Logical (SDX) appliances require licenses for each physical appliance and each virtual instance. Refer to NetScaler Platform for more information. Types of NetScaler Licenses.
  2. Downloads Citrix Application Delivery Controller firmware, components, SDKs, service delivery appliances, service delivery platforms, evaluations and trial software.
  3. NetScaler Gateway communicates with StoreFront to protect apps and data delivered by XenApp and XenDesktop. The user devices run Citrix Receiver to create a secure connection and access their apps, desktops, and files. Users log on and authenticate using NetScaler Gateway. NetScaler Gateway is deployed and secured in the DMZ.

Note: This blog relates to on-going incidents. Facts, recommendations and impact may change as more information becomes available.

UPDATE 24.1.2020:

  • Citrix has released fixed builds for the ADC server, please refer here for your version: https://support.citrix.com/article/CTX267027
  • Taneli Kaivola from the Nixu DFIR team has released a script to quickly triage vulnerable servers: https://github.com/nixu-corp/citrix-check

Vulnerabilities found in Citrix Gateway, Application Delivery Controller and Citrix SD-WAN WANOP appliances are being exploited in the wild and Nixu has now seen multiple cases in the Nordics where attackers are exploiting the flaw. These devices were previously known under the NetScaler brand. The exploit allows unauthenticated attackers to carry out code execution on the device – essentially giving anyone access to the device and potentially the network behind it. The vulnerability has the CVE number CVE-2019-19781. (https://nvd.nist.gov/vuln/detail/CVE-2019-19781)

The vulnerability is based on path-traversal attack on servers that are exposed to the internet. Attackers have been proven to be able to run commands and download malicious software on customer servers. A public proof-of-concept exploit is available, greatly lowering the threshold for criminals to exploit this vulnerability.

The vulnerability is critical and is currently being exploited with automated scripts. The mitigations provided by Citrix may have also been compromised, and can not be relied on to completely protect a vulnerable server. Citrix has promised a patch for the server software, but it has not yet been released. You can follow the release schedule for your version of the server from Citrix's support page: https://support.citrix.com/article/CTX267027

Citrix has released a write-up of the incident with instructions and a list of affected devices: https://www.citrix.com/blogs/2020/01/17/citrix-updates-on-citrix-adc-citrix-gateway-vulnerability/

Any organization with these appliances should perform the recommended mitigation actions immediately to prevent the devices from being compromised. As these devices are often deployed to the borders of networks, they can act as a way to attack other services and devices behind them.

These types of incidents happen regularly in today’s connected world. Any organization should be prepared to react quickly and decisively. To ensure adequate reaction times and capabilities, the following activities are recommended:

Keep up to date asset inventory of all devices and software being used

  • When vulnerabilities are released, knowing whether your organization is affected and which assets need patches or mitigations is critical

Follow vulnerability notices from vendors or utilize a service that does it for you

  • For larger enterprises, manually following the notices is often infeasible as the number of assets and vendors grows
  • Ensure that a process exists for quickly applying patches or mitigations outside of normal patching windows when the situation calls for it
  • Test the processes regularly to ensure that they can be utilized during a real crisis

Regularly patch all assets, whether software or hardware appliances

  • Ensuring that all assets have up to date patches is generally the best defense against attacks. In most cases, patches are already available or become available within hours after attack becomes known

Ensure that you have in-house capabilities or a partner to support with incident response

  • If mitigations aren’t done in time or the organization is hit very early on, simply patching assets without addressing the on-going incident that may have escalated beyond the affected assets is no longer effective

Attack mitigations

To further complicate things, several of the NetScaler ADC servers are running business-critical functions and can not be turned off to wait for a patch to be published. If an instance of NetScaler ADC is *not* business critical, Nixu suggests that volatile data should be collected from the machine following the instructions on Evidence Preservation. After collecting evidence and storing it in a secure location, the appliance should be paused until a patch is released and applied. It is also important to take a snapshot of the running machine before pausing it to avoid losing evidence in case of an error. It is not advisable to power down the appliance as parts of the file system are held in memory during runtime.

The following mitigations can be applied if turning off the server is not a possibility:

  • Apply recommended mitigations for different device types as listed on Citrix KB article: https://support.citrix.com/article/CTX267679
  • Limiting access to the server using all available methods, for example whitelisting IP addresses, VPN or GeoIP blocking.
  • Limiting access from the NetScaler ADC server to other servers.
  • Revoking all certificates stored on the server. The attacker user level ‘nobody’ has read rights to private encryption keys.
  • Revoking all passwords stored on the server. The passwords are encrypted with a hard-coded key that is retrievable by the attacker.
  • Restoring the NetScaler ADC Server from a backup to a state earlier than 10.1.2020 when the vulnerability was published. Restoring the server should be done only after other mitigations are in place, otherwise the server can be reinfected.
  • Reinstalling the NetScaler ADC Server.

Before restoring or reinstalling the server, it is advisable to store a snapshot of the current state for further forensic analysis. As a part of the operating system is running on a ramdisk, it is important to store a snapshot that preserves memory contents.

Indicators of compromise

Appearance of the following addresses may be an indicator of a compromised system:

138[.]68.14.63
95[.]179.163.186
185[.]178.45.221
159[.]69.37.196
hxxps://pastebin.com/raw/d3SY1erQ
hxxps://pastebin.com/raw/2zds3h2T
hxxps://pastebin.com/raw/8xNac8At
hxxps://pastebin.com/raw/UrJnnijX

Evidence Preservation

Before running any commands, preserve evidence by saving the following files:
/var/cron/tabs
/netscaler
/tmp

Save the process list to a file and save that too:
ps -axu | grep -i nobody > /var/tmp/processlist.txt

You can easily combine the files to an archive with the following command:
tar -czf /var/evidence_from_ramdisk.tar.gz /var/cron/tabs /netscaler /tmp /var/tmp/processlist.txt

All About Citrix Netscaler Vpn Client

Look for processes running as nobody.
ps -axu | grep -i nobody

Check /netscaler/portal/templates directory and look for suspicious XML files.
ls -lah /netscaler/portal/templates
cat suspicious.xml

Netscaler

Remember to download the preserved evidence files before analyzing the system further.

Incident Triage

Check bash and sh logs to see what commands have been executed on the system. Note that some of the logs have been rotated and archived.
grep -i nobody /var/log/bash*
grep -i nobody /var/log/sh*

Check cronjobs
crontab -l -u nobody
cat /var/cron/tabs/nobody

Need help?

Nixu provides services around cyber defense, incident response and vulnerability management. To understand more how we can help with preventing, mitigating and responding to incidents such as this one, see our service listing for cyber defense. https://www.nixu.com/services/cyber-defense

Netscaler

If you suspect that your organization has been affected and you have an active cybersecurity incident on-going, support is available to organizations that are not yet clients. To get started quickly, visit: https://www.nixu.com/service/security-incident

Further information about the vulnerability:

downloadWhy can't I download this file?Download Citrix Netscaler

Enhancements

Citrix has added a feature enhancement for DTLS which, when enabled, addresses the susceptibility to this attack pattern. The enhancement builds are available on the Citrix downloads page for the following versions:

  • Citrix ADC and Citrix Gateway 13.0-71.44 and later releases
  • NetScaler ADC and NetScaler Gateway 12.1-60.19 and later releases
  • Citrix ADC 12.1-FIPS 12.1-55.210 and later releases
  • NetScaler ADC and NetScaler Gateway 11.1-65.16 and later releases


Customers who do not use DTLS do not need to upgrade to the enhancement build. Instead, customers are recommended to disable DTLS by using the following ADC CLI command:


Customers using DTLS are recommended to upgrade to the enhancement build and enable “HelloVerifyRequest” in each DTLS profile by using the following ADC CLI instructions:

  • List all DTLS profiles by running the command:
  • For each DTLS profile, enable the “HelloVerifyRequest” setting by running the command:

  • Save the updated configuration by running the command:

Citrix Netscaler Download Windows 10

  • To verify “Hello Verify Request” is enabled, run the command:
  • If DTLS was disabled based on a previous version of this advisory, re-enable the DTLS profile by running the following command:

Obtaining Support on This Issue

If you require technical assistance with this issue, please contact Citrix Technical Support. Contact details for Citrix Technical Support are available at http://www.citrix.com/site/ss/supportContacts.asp.

Citrix Netscaler Download

Disclaimer

What Is Citrix Netscaler Gateway

This document is provided on an 'as is' basis and does not imply any kind of guarantee or warranty, including the warranties of merchantability or fitness for a particular use. Your use of the information on the document is at your own risk. Citrix reserves the right to change or update this document at any time.

All About Citrix Netscaler Anyconnect

Changelog

Citrix Netscaler Gateway Client Download

Citrix netscaler training
DateChange
2020-12-23Initial Publication
2021-01-04Enhancements Released
2021-01-11Enhancements Released in 12.1-FIPS