In the previous post, we looked at how we can analyze and filter egress traffic via the dedicated central appliance (Palo Alto Firewall). A set of Firewall VMs was deployed behind the AWS Gateway Load Balancer. The Palo Alto Firewall was intended to solve many different problems for the organization. In the process of the firewall configuration, we faced some limitations and had to find solutions for them, which will be described in this post.
Limitation 1. Configuration synchronization
We have two VMs, deployed in different AZ for high availability, traffic will be forwarded to a random VM, so configurations should be identical.
Palo Alto firewalls support high availability with session and configuration synchronization, but unfortunately today (October 2022, PAN-OS 10.2) it has several limitations:
– AWS supports active/passive HA only.
– On AWS, when you deploy the firewall with the Amazon Elastic Load Balancing service, it does not support HA
Solution. Palo Alto Panorama
We need HA + configuration sync due to requirements and common sense. Nobody wants to repeat the same configuration several times in different VMs. Palo Alto provides a solution for this (that has to be bought separately), called Panorama. It can be used for centralized configuration and visibility. The diagram below shows how it is deployed in the “Egress VPC” (other parts will be discussed later).
Here you can find an instruction on how to add a firewall VM to the Panorama-managed devices. The only thing that needs to be kept in mind, is that versions of Panorama and Firewalls OS should be the same. Unfortunately, the Panorama version in the AWS Marketplace is not up to date, so it should be updated separately (October 2022).
Once you finished with managed devices, you can see that VMs are “In Sync”:
You can also see some basic health metrics:
Then you can create policies, objects, and other configurations and push them to the relevant device groups:
Limitation 2. Palo Alto GlobalProtect VPN
Another pair of Palo Alto virtual machines are depicted in the above diagram (VPN VPC) because they can not be configured on the same VMs, that is used for Egress traffic filtering. GlobalProtect is used as a VPN solution to get access to private resources. Instances are deployed in the public subnets because they should be accessible for end-users from the internet, a single endpoint should be exposed for users’ convenience, and the deployment should be highly available.
Active/Active HA is not available in AWS, so our choice is Active/Passive.
Here you can find an instruction on how to configure Active/Passive HA for Palo Alto VMs in AWS. When the active peer goes down, the passive peer detects the failure and becomes active. Additionally, it triggers API calls to the AWS infrastructure to move all the data-plane interfaces (ENIs) from the failed peer to itself.
We can see that the Config is Synchronized between VMs:
As a result, we need at least 5 Palo Alto VMs in order to have Egress traffic filtering with config synchronization and highly-available GlobalProtect VPN.
Limitation 3. Palo Alto GlobalProtect SSO and group mapping
In the first diagram, you can see that AWS Directory Service is present in the current architecture. First of all, it is used as an identity source for the AWS IAM Identity Center:
A custom application that supports identity federation with SAML 2.0 was created for the GlobalProtect SSO.
So we have everything on the single login page:
One more requirement is that different groups of users must have access only to appropriate private networks, for example, Team1 can access only App1 VPC, while Team2 can access only App2 VPC. The above configuration is not enough for this task.
Palo Alto states that to enable group mapping functionality, you must create an LDAP server profile that instructs the firewall how to connect and authenticate to the directory server and how to search the directory for the user and group information. Then you can define GlobalProtect configurations and/or security policies based ongroup_membership.
Limitation 4. Palo Alto VM monitoring.
Using Palo Alto VMs in production requires detailed monitoring for alerting and scaling purposes and providing appropriate SLA. We have a set of standard EC2 metrics in CloudWatch, but it is not enough. We can install the CloudWatch agent in Linux or Windows instances, but Palo Alto VM has its OS called PAN-OS, where we can not install the CloudWatch agent. Fortunately, the firewall VM has the capability to send some custom metrics to CloudWatch. As usual, we need to provide relevant permissions via IAM Role (Instance profile), for example:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": [
"*"
]
}
]
}
or managed IAM policy CloudWatchAgentAdminPolicy that also allows writing logs into CloudWatch.
Then you need to reboot the VM, otherwise, it will not start using the IAM role. Enable one checkbox in the Palo Alto WebUI:
Here is a list of available Palo Alto VM metrics.
| METRIC | DESCRIPTION |
|---|---|
| Dataplane CPU Utilization (%) | Monitors dataplane CPU usage and measures the traffic load on the firewall. |
| Dataplane Packet Buffer Utilization (%) | Monitors dataplane buffer usage and measures buffer utilization. If you have a sudden burst in traffic, monitoring your buffer utilization allows you to ensure that the firewall does not deplete the dataplane buffer, which results in dropped packets. |
| GlobalProtect™ Gateway Active Tunnels | Monitors the number of active GlobalProtect sessions on a firewall deployed as a GlobalProtect gateway. Use this metric if you use this VM-Series firewall as a VPN gateway to secure remote users. Check the datasheet for the maximum number of active tunnels supported for your firewall model. |
| GlobalProtect Gateway Tunnel Utilization (%) | Monitors the active GlobalProtect tunnels on a gateway and measures tunnel utilization. Use this metric if you use this VM-Series firewall as a VPN gateway to secure remote users. |
| panSessionConnectionsPerSecond | Monitors the new connection establish rate per second. |
| panSessionThroughputKbps | Monitors the throughput in Kbps. |
| panSessionThroughputPps | Monitors the number of packets per second. |
| Sessions Active | Monitors the total number of sessions that are active on the firewall. An active session is a session that is in the flow lookup table for which packets will be inspected and forwarded, as required by policy. |
| Session Utilization (%) | Monitors the TCP, UDP, ICMP and SSL sessions that are currently active and the packet rate, new connection establish rate, and firewall throughput to determine session utilization. |
| SSLProxyUtilization (%) | Monitors the percentage of SSL forward proxy sessions with clients for SSL/TLS decryption. |
How it looks like in CloudWatch:
There are some basic logs, like the system shutting down, but without details.
Unfortunately, there are no other metrics like MemoryUsage, FreeDiskSpace, etc. SNMP protocol is an extra option for wider monitoring. Palo Alto supports a variety of MIBS, but in this case, you will need an extra instance that will collect metrics from Palo Alto VMs and put them into CloudWatch.
Palo Alto Panorama does not even support sending basic metrics into CloudWatch. The only option for Panorama monitoring is SNMP.
Limitation 5. Panorama Logs Collector
If you have a set of Palo Alto Firewall VMs that works in parallel, you may need to check logs of going traffic. Checking every VM is not convenient, so you can configure Panorama as a central log collector, where all firewalls will be sending logs to. The first thing that we need to do is switch Panorama mode from “management only” to ‘panorama’, which in turn requires a separate disk (in our case this is EBS volume).
According to the documentation the Panorama virtual appliance on AWS supports only 2TB logging disks and, in total, supports up to 24TB of log storage. You cannot add a logging disk smaller than 2TB or a logging disk of a size that is not evenly divisible by 2TB because the Panorama virtual appliance partitions logging disks into 2TB partitions.
In reality, switching Panorama mode failed with 2TB and 2.5TB disks. Panorama will display no errors during Mode change, but ‘management only’ mode will stay in place without switching to ‘panorama’. It succeeded only with a 3TB disk.
admin@Panorama> request system system-mode panorama Broadcast message from root (Fri Sep 30 10:08:20 2022): The system is going down for reboot NOW! Changing system mode from management-only to the panorama.
admin@Panorama> show system info | match system-mode system-mode: panorama
Note 1.7T of the filesystem on the newly added disk – it stays of the same size with a 3TB overall disk size.
admin@Panorama> show system disk-partition Disk /dev/nvme1n1: 3 TiB, 3298534883328 bytes, 6442450944 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: gpt Disk identifier: 16CEDCAC-1A2C-4965-8E73-35F6D50A903B Device Start End Sectors Size Type /dev/nvme1n1p1 20480 3706898431 3706877952 1.7T Linux filesystem
Fortunately, Panorama deals with partitioning and file system creation itself. You just need to attach an EBS volume.
There is a built-in mechanism of logs rotation, that is very handy while we don’t have a simple way to monitor DiskSpace in Panorama. You can configure a quota(%) of disk space, that must remain free or set “Max Days” for keeping logs.
Capabilities of traffic inspection and filtering
URL filtering
Palo Alto Networks URL filtering solution, Advanced URL Filtering, is a subscription service that defends your network from web-based threats by giving your users safe access to the web while delivering granular policy controls to precisely define how they interact and access online content.
You can use Palo Alto managed categories of traffic or create your own and use them for black/white lists.
For example, block gambling websites:
WildFire
The WildFire Analysis Environment identifies previously unknown malware and generates signatures that Palo Alto Networks firewalls can use to then detect and block the malware.
Once it is enabled and configured, you can test it by downloading a sample malware file from http://wildfire.paloaltonetworks.com/publicapi/test/pe
Once Firewall will encounter a file that is unknown to it (is absent in the signatures database) it will forward such sample(file or link) to WildFire Public Cloud and will log such an event. Wait for 5-10 minutes and check the Firewall Logs:
DNS Security
DNS Security works in a way when Firewall catches DNS Queries going from Host to DNS and forges the reply to have a different IP Address (Sinkhole Address) – this way Host will not be able to reach the malicious domain. The firewall then will log such Sinkholed traffic together with the IP of the Host which made the DNS Query – so the Administrators could take care of the infected internal Host.
DNS Sinkhole only works if DNS Server is North of the firewall.
If the Host and DNS server are both behind the firewall, DNS queries will not pass through the Firewall and no Sinkholing would happen.
For DNS Sinkhole to work Host which queries the DNS must be located behind the Firewall whereas DNS Server queried – should be before the firewall, so the DNS query from the Host to DNS Server will pass through the Firewall where it will inject the Sinkhole address into the DNS Response if the Domain is in Palo Alto malware database.
Ensure good Domain is resolved by both Internal DNS (AWS DNS) and by External DNS (Google DNS in this case):
Try to resolve malicious Domain (i.e. zesien.com)
Notice that the Domain is not resolved by using External DNS – it was captured by a Firewall
If you use default AWS DNS servers, Palo Alto DNS security will not work, because DNS queries will not go through the firewall.
Conclusion
Palo Alto Networks provides many useful security features and products that can help you keep your environment safe. You can read more about it Prisma Cloud here and Egress traffic filtering here, but you need to know about capabilities and limitations in case of deployment in the AWS.
