Jan Tytgat

What happens during the installation of a new build?

8 min read

Table of contents

  1. Anatomy of a build directory
  2. installns: one entry point, four install modes
  3. The BSD path: installns_legacy
  4. The Linux path: NSLINUX (l2l)
  5. Cross-OS migrations: f2l and l2f
  6. Reboot: making the new build active
  7. Coming up next

↑ Back to top

Introduction

Every so often a new NetScaler firmware build lands on the Citrix download portal. You download a file such as build-14.1-73.49_nc.tgz, unpack it on your appliance or inside the build folder of a VPX, run ./installns, answer a few questions and reboot. The appliance comes back with new firmware.

But what actually happens during those minutes while the installer is running? For the “NetScaler Research” series I unpacked two builds of the same release, 14.1-73.49:

Everything quoted below was found by reading the scripts in those two archives, mainly _pkg/installns and its siblings.

↑ Back to top

Anatomy of a build directory

After unpacking, both variants show a similar structure:

14.1_73.49_nc/              14.1_73.49_lx/
├── _pkg/                   ├── _pkg/
│   ├── installns           │   ├── installns
│   ├── installns_legacy    │   ├── installns_precheck_report.pl
│   ├── installns_post_boot │   ├── bzImage-14.1-73.49
│   ├── kernel.elf          │   ├── initrd-14.1-73.49.cpio.gz
│   ├── ns-14.1-gui.tar     │   ├── rootfs-14.1-73.49.tar.gz
│   └── ~50 *.tgz packages  │   ├── grub.cfg, nsconfig, ...
├── payloads/               ├── payloads/ (grub, precheck)
├── rootfs/                 └── rootfs/
└── manifest.tsv

An early observation: the _lx build carries no per-package tarballs at all. Everything is baked into the rootfs image. That detail matters later.

↑ Back to top

installns: one entry point, four install modes

installns is a dispatcher: a bash script of roughly 3.100 lines whose job is to figure out where it runs and what is being installed, and to pick the right strategy for that combination. It distinguishes four modes:

ModeRunning OSNew buildMeaning
f2fFreeBSDnCoreclassic FreeBSD → FreeBSD upgrade
f2lFreeBSDNSLINUXmigration FreeBSD → Linux
l2fNSLINUXnCoremigration Linux → FreeBSD
l2lNSLINUXNSLINUXin-place Linux upgrade

Before any of that, the script sets up some safety machinery:

The high-level flow, straight from the bottom of installns:

flowchart TD
    A[./installns invoked] --> B[init_installer_info + singleton guard]
    B --> C{Running on FreeBSD<br>and target is nCore?}
    C -- yes --> D[f2f: hand over to installns_legacy]
    C -- no --> E[_detect_install_mode]
    E -- FreeBSD + NSLINUX pkg --> F[f2l migration]
    E -- NSLINUX + nCore pkg --> G[l2f migration]
    E -- NSLINUX + NSLINUX pkg --> H[l2l in-place upgrade]
    D --> I[prompt reboot]
    F --> J[phased multi-reboot sequence]
    G --> J
    H --> K[reboot into new build]

There is a surprise in the f2f case: on a classic FreeBSD-based appliance, the shiny new unified installns does not do the work itself. It detects the combination and hands over to installns_legacy, a ~4.200-line Perl script that is the continuation of the installer that has shipped with NetScaler for two decades.

↑ Back to top

The BSD path: installns_legacy

The legacy installer is invoked as /usr/bin/perl installns_legacy with the version metadata passed via environment variables (NSVERSION, NSVARIANT, CLVERSION). Its main sequence reads like a checklist:

  1. check_system_before_install: validates the system ID, VPX specifics, that the running configuration is saved and synced between nodes, and that enough space is available.
  2. copy_kernel: backs up the currently running kernel (timestamped, under /var/nsinstall/backup/) and copies the new kernel.elf (a FreeBSD x86-64 ELF executable) to /flash.
  3. install_kernel_signature and install_rsakey: install the signature files and public key used to verify the kernel at the next boot.
  4. mod_loaderconf / mod_loaderconflocal: update the FreeBSD bootloader configuration so the new kernel is what gets loaded on reboot.
  5. The package army: a long list of install_* functions, each extracting one payload tarball from _pkg/ into the system:
install_doc, install_help, install_help_cisco,
install_LogonPoint_workspace, install_LogonPoint, install_LoginSchemaFiles,
install_OTPToolFiles, install_app_catalog_files, install_gui,
install_epa_package, install_nitro, install_openapi_specification,
install_mastools, install_nsconnector, install_lom, install_bios,
install_nic_firmware, install_mstflint_package, install_config_wipe,
install_linux_package, install_root_certs, install_callhome_cert,
install_geoipdb, install_nfast, install_safenet, install_inCompass_db_files,
install_python, remove_golang, install_perl, install_pgsql, extract_nextgen,
install_nskrb, remove_gnostic, ...

Most of them unpack into /var/netscaler or /netscaler. The GUI payload for example is extracted to /var/netscaler/gui_new and only renamed to its final location after the reboot. This is also where the scripting runtimes are laid down: Python, Perl and PostgreSQL get replaced as part of the build installation.

  1. Post-install fixups: stopping nsclfsyncd during the operation, upgrade_before_pe_start, after_upgrade, nextgen_upgrade, permission fixes, API specs, and disabling the nsfsyncd cron entry so file sync does not interfere mid-install.
  2. prompt_reboot: unless run with -y/-Y/-g, the installer waits for a human to confirm.
flowchart TD
    A[installns_legacy Perl] --> B[check_system_before_install]
    B --> C[backup running kernel]
    C --> D[copy_kernel: kernel.elf to /flash]
    D --> E[kernel signature + RSA key]
    E --> F[update FreeBSD loader config]
    F --> G[install ~50 package payloads<br>into /var/netscaler and /netscaler]
    G --> H[post-install fixups]
    H --> I[prompt_reboot]

So on the FreeBSD platform the flow is: kernel first, everything else layered on top as separate packages afterwards, which is exactly the mental model many of us had.

↑ Back to top

The Linux path: NSLINUX (l2l)

The NSLINUX build works differently. There is no installns_legacy, and there are no package tarballs: the rootfs tarball is the product, /netscaler included. The l2l mode of installns therefore boils down to placing three artifacts on the right partitions and letting the boot chain do the rest.

NetScaler-linux uses three labelled disk partitions:

The _install_linux_artifacts function does exactly this, sets /flash/.init.nsinstall (a flag the init system reads on the next boot), and after a _fix_downgrade_issues pass the system reboots. Older kernel/initrd/rootfs sets are deleted beforehand by _delete_older_linux_images to make room.

flowchart TD
    A[installns: l2l mode] --> B[install_upgrade_prerequisites]
    B --> C[_delete_older_linux_images]
    C --> D[_install_linux_artifacts]
    D --> E[bzImage + grub.cfg<br>to bootloader partition]
    D --> F[initrd + rootfs tarball<br>to flash partition]
    D --> G[sha512.signed signatures<br>+ RSA public key]
    E --> H[set /flash/.init.nsinstall flag]
    F --> H
    G --> H
    H --> I[_fix_downgrade_issues]
    I --> J[reboot into new build]

On NSLINUX the “installation” is much closer to a transactional image update than to a package installation. The installer never unpacks individual product components: it stages kernel, initrd and rootfs, and the next boot decides whether to run them. The packages you find in the _nc build are already inside the rootfs tarball of the _lx build.

↑ Back to top

Cross-OS migrations: f2l and l2f

The same installns can also migrate a machine between operating systems: f2l (FreeBSD → NSLINUX) and l2f (NSLINUX → FreeBSD). This is a phased, multi-reboot process: the running OS backs up /var and /flash data, shrinks and reshapes partitions, installs GRUB or the FreeBSD loader respectively, and then hands over to a minimal installer environment (an initrd) that continues the next phase before the machine boots into its new OS for the final phase.

The run_upgrade function implements this as an explicit state machine: running_phase counters 1 through 6, persisted in the resume file between reboots. It is a fascinating read, but too much detail for this first post; it deserves its own article later in this series.

↑ Back to top

Reboot: making the new build active

This is where the installer hands control back to the administrator. Nothing more happens until the machine reboots:

After the reboot, assuming all went well, you are running the target firmware.

↑ Back to top

Coming up next

Installation, however, is only half the story: everything above only stages software. What happens when that reboot button is pressed, the kernel starts and the NetScaler processes come up? Which init system runs, what is pitboss and why does it watch your packet engines, and why does the AAA/Gateway GUI get regenerated on every single boot?

That is the subject of the next post in this series: “How does a NetScaler instance start?”

↑ Back to top

Tags: