A flexible file backup Strategy – Part 2

In this second article, we are going to cover the File to Tape strategy.

Why tape devices are still widely present in the IT department?

  • It’s a good way (but not the only one) to manage the offline backup data (read it as improving the Security Strategy of your data).
  •  Media can be easily carried or moved (read it as Portability).
  • Deployment is often very quickly (read it as speeding up the adoption).
  • It has a potentially infinite capacity (Just adding media).
  • The LTO is a neverending technology in a continued evolution.
  • The tape is a well-known device, IT operators have the skills to manage it.
  • The costs for GB is lower than disk technologies.
  • The costs are quite predictable, managers can budget it easily.

VBR needs a Windows Physical Server named Tape Server to control the Drives and Robotic, LTO3 or later Drives, and MS-Windows drivers (supply by the hardware vendor).

The official user guide available on the Veeam site gives all detailed info.

Just a note before starting:

VBR uses Tape Technology in two different ways.

The most used one is back up to tape (Picture 1).

In this case,  the source backup data are the backups already present and created with a backup job or backup copy job.

They are saved to Repository (Repository is a Disk technology).

It means that the scope of backup to tape is to pour out data to tape.

Picture 1

Please have a look at the following video (https://www.youtube.com/watch?v=Il8mH2KB_Uo) to get more details.

The second way is File to Tape and it is the topic of this article (picture 2).

https://lnx.gable.it/wp-content/uploads/2021/01/nas-7.jpgPicture 2

Which type of source files can be saved to tape?

  • Windows & Linux servers (virtual or physical doesn’t matter)
  • NAS file share (SMB (CIFS) and NFS ).
  • NDMP filers (it will be covered in the next article).
  • How does it work?

Picture 3

Picture 3 shows the data streams when a tape process is performed:

  1. The main components are Data Movers. These Services run on the source and on the Tape Server.
  2. VBR triggers the source Data Mover to perform a copy of the files to the target. At the destination, the target Data Mover check if the files have arrived correctly.
  3. The tape Server manages the write operation to the tape also.
  4. VBR stores all info about files saved (media used, retention, etc.)  in a catalog.
  5. In the restore scenario, the step order is four to one.
  • *Note: To perform a backup of Windows and Linux servers, it is requested to add those servers to the managed server as shown in picture 3. Through this process, the Data mover service is properly installed.
  • Network Share: Adding SMB/NFS Share as shown in the previous article (A Flexible file backup strategy – Part 1).
Picture 3
  • Common scenarios

File to Tape backup can be used by any customer. You need just a Tape Server, Tape Devices, Drivers, and VBR.

There are at least two main cases:

  • Customers who want a copy of their data to tape.
  • Customers with a small budget who doesn’t need rapid restore

The next video will show how to set it up.

Main Pro

  • There is not a room limit. It means the license doesn’t count how many GB, TB, PB will be written to Tape.
  • The VBR architecture is as usual flexible. It’s possible to add more tape servers and more than 1 tape library.

Version 11 will add more great features:

  • Tape cloning (https://community.veeam.com/blogs-and-podcasts-57/tape-improvements-in-vbr-v11-277)
  • Tape verification (https://community.veeam.com/blogs-and-podcasts-57/part-ii-tape-improvements-in-vbr-v11-289)

Cons

*This behavior is quite common to all backup software that writes data directly to Tape.

  • For saving a file, VBR needs to trigger a process of discovering the file to the source, gathering and writing it to a media.

If you consider that the common NAS scenario is composed of millions of small files and thousands of folders and that the tape technology has to choose for every file the location in the media (where the file will be copied)  it’s clear that this process, common to all backup servers, stresses the hardware architecture and in particular the drive header.

The backup process has a small speed advantage compared to restoring because writings to media are often sequential and not random.

Image to restore 10k files located in 10k different positions in a single tape.

The drive has to perform a great job. It is going to suffer from an effect called shoe-shining (also known as tape back-hitching)  which occurs when a tape drive cannot transfer data at an acceptable speed.

Shoe shining can contribute to data loss over time, as the repeated back-and-forth motion will wear the tape drive’s read/write heads and negatively affect the readable portion of the tape

  • Loss of Tape Cartridge Capacity
  • Increased Risk of Read/Write Issues
  • Excessively Worn Tape Drive Heads
  • Low Data Transfer Rates
  • Data Loss
  • The Veeam DB needs to be sized correctly and the best practice is to switch from SQL Express to SQL Standard
  • Media management is quite challenging when the amount of tapes is big. Remember to store them in a fireproof and non-magnetic safe.

Do you also prefer the NAS backup feature introduced in v.10? Let me know!

That’s all for now.  

See you next week for talking about NDMP

Ransomware defense part 2: Hardening

There are many documents on the internet that describe how to address this common request.

In this article, I’ll give you a track to move easier around this topic pointing out the most interesting articles.

Before starting let me thank Edwin Weijdema who created an  exhaustive guide to answer the common question (please click here to get it)

Are you ready? Let’s start

1- The first magic point for starting is Wikipedia where I got a good definition:

In computinghardening is usually the process of securing a system by reducing its surface of vulnerability, which is larger when a system performs more functions; in principle, a single-function system is more secure than a multipurpose one. Reducing available ways of attack typically includes changing default passwords, the removal of unnecessary software, unnecessary usernames or logins, and the disabling or removal of unnecessary services.

2- The second point is to understand the concept of Perimeter security:

It is natural barriers or artificially built fortifications that have the goal of keeping intruders out of the area . The strategies can be listed as:

  • Use rack-mount servers
  • Keep intruders from opening the case
  • Disable the drives
  • Lock up the server room
  • Set up surveillance

A complete article is available by clicking here

3- The third point is  Network segmentation:

It is the division of an organization network into smaller and, consequently, a more manageable grouping of interfaces called zones. These zones consist of IP ranges, subnets, or security groups designed typically to boost performance and security.

In the event of a cyberattack, effective network segmentation will confine the attack to a specific network zone and contain its impact by blocking lateral movement across the network via logical isolation through access controls.

Designating zones allows organizations to consistently track the location of sensitive data and assess the relevance of an access request based on the nature of that data.  Designating where sensitive data reside permits network and security operations to assign resources for more aggressive patch management and proactive system hardening.

A complete article is available by clicking here

4- Hardening your Backup Repositories

The next good rules involve your backup architecture and in specific the Backup Repositories:

Windows:

a. Use the built-in local administrator account

b. Set permissions on the repository directory

c. Modify the Firewall

d. Disable remote RDP services

Linux:

e. Create a Dedicated Repository Account

f. Set Permissions on the Repository Directory

g. Configure the Linux Repository in VeeamModify the Firewall

h. Use Veeam Encryption

Do you want to know more about security? If so the Veeam Best Practices are for sure the answer.

The next article will cover monitoring and automatic actions using Veeam-ONE.

5- Prevent injection of shady boot code​

Code injection, also called Remote Code Execution (RCE), occurs when an attacker exploits an input validation flaw in software to introduce and execute malicious code.

To prevent the attack please follow the following rules:

a. Run with UEFI Native Mode​
b. Use UEFI with Secure Boot Standard Mode​
c. Combine Secure Boot with TPM
d. Equip critical servers with a TPM 2.0

Stay tuned and see you soon

Ransomware defense – part 1: Advanced product features are an mandatory requirement

A lot of new challenges came to people who work in IT-Departments these last months.

The number of ransomware attacks has been growing day by day and their attack strategies are becoming more and more evil and dangerous.

The common questions the Managers ask the IT guys are:

a) Are the company protected against these risks?

A good answer is that a successful approach is when the percentage of certainty is more than the percentage of risk.

b) Which are the best practices to be safer?

The key is defining the right process of protection.

The scope of these articles is showing the correct behavior to keep your architecture as safer as possible or, in case of attack, gain as much time as possible to fend off the assault.

The articles will cover the storage point of view and do not deal with perimetral defenses, antimalware, antiviruses, networking strategies, and so on.

Which are the main strategies to adopt?

  1. Having more copies of your data
  2. Hardening the infrastructure
  3. Monitoring behaviors

Are you ready? Let’s start with the first topic !!!

    1. Having more copies of  your data:

Backup software is the right tool to score the goals of this first part.

It has to be able to:

a) Create application consistency backup.

b) Copy backup data to different locations.

Almost all backup software can do that but some additional features can address better the biggest challenges:

Flexible: Backup software should write backup data to different types of repositories and be able to restore it without any required dependency. To be clearer, the backup data have to be self-consistent. The advantage is being able to fit different architecture scenarios (Let’s call it “Data mobility”).

Data-Offline:  back up data should be put into a “quarantine” area where they cannot be either re-written or read. The classic deployment is a Tape Devices architecture or any scripts that automatically detach the repository devices.

Immutability: The backup data cannot be changed until the immutability period is over. This has a double advantage in comparison to data-offline strategy: It changes the repository status as written & online just for the new backup file. It is offline (as Tape technologies) for re-writing to already present backup data. The speed restore option has to remain unchanged.

Immutability can be reached in two ways:

By WORM  (Write Once, Read Many) devices, where the backup files can be used just to restore once they have been added to repositories. For example, technology can be the optical disk, a technology I have been working on in the past.

At Veeam Software this common customer and partner request has been addressed using the immutability propriety of the Object Storage. The good news is that VBR v. 11 implements this great feature directly in Linux Repositories.

Is this enough? I’m still thinking that the backup solution should at least be able to:

  • Check the backup file and the backup content. The only way to check if a backup file is really reusable is restoring it in a separate area where communication with the production environment is forbidden. At Veeam it is called Sure-Backup.
  • Check with your anti-virus/anti-malware that the backup files have not been already attacked somewhere and sometime. At Veeam the technology used is the Data integration API.
  • Before restoring files or VMs in production, check with your anti-virus/anti-malware if your data has been already attacked. At Veeam it is called Secure Restore
  • Perform Replica Jobs. It helps to create a Disaster Recovery Site useful in performing a quick restart of the service.  At Veeam this feature is included from the beginning and the Sure-Backup can be applied with replica too (it is called Sure-Replica). V.11 has a very powerful feature: CDP.
  • Restore backup data to the public cloud when the primary and replication site is totally out of order. I call it Cold Disaster Recovery and it needs at least one restore point available.

The next article topic is how to hardening your backup architecture

See you soon and take care!

VBR – Proxy linux server UUID

When a Linux VM is added to Veeam console as a Proxy Server,  you can fall out in the error shown in picture 1

Picture 1

The reason for this behavior is that the default VM config does not allow another software to see the UUID of the VM.

What is UUID?

It’s the unique identifier used to uniquely identify partitions in Linux operating systems.

Why is it important to use it?

A backup where the proxy is a Linux VM only works with virtual appliance transport mode. It uses the VMware hot add capability.

Easier: when a job starts, the proxy Linux mounts the disks of the VM that have to be processed and then send a copy of data to the Veeam Repository.

If the backup server knows which are the proxy disks it can process the others easily and without errors.

The result is that it’s mandatory to set it up correctly as shown in the user guide and in Veeam forum

Note 1: the Linux command to show UUID is blkid

To address the issue just switch off the VM and, from vCENTER Console, follow the procedure showed in the next 4 pictures highlighted in yellow.

Picture 2

Picture 3

Picture 4

Picture 5

That’s all folks

Veeam Backup & Replication – Agent Licensing

Today I’m covering how to enable VBR to use VUL licensing to backup Physical Machines (they could be server or workstation both)

If you need more details to understand better the implementation scenarios please refer to the three articles I already wrote

Veeam Agent  Part 1     Veeam Agent Part 2     Veeam Agent Part 3

As many of you already know it is possible to work with the free Veeam Agent version and a paid VBR version.

The only thing that this architecture allows is using the VBR repositories as a global backup container.

What customers forget is that you can’t have a mixed infrastructure composed of free and paid agents.

Let’s see an example:

An end-user with already 10 VBR sockets license has a new project to protect 30 workstations and he wants to use Veeam agent Free.

The end-user just has to install and set up on every single workstation the free agent to write backup data to VBR repository.

From VBR version 9.5.u4, Veeam is gifting 1 VUL license for every socket customer bought (up to 6 sockets). In our example it means the end-user can protect up to 6 Physical Server or 18 Workstation (1 Vul x 3 Workstation) for free or a mixed architecture.

What happens if you enable VBR to assign the gifted license from your VBR server?

First thing how to enable it? The next three pictures (Picture 1, 2 and 3) explain how to perform it

Picture 1

Picture 2

        Picture 3

From now on VBR will use the agent license up to consuming them.

In our example, the end-user will protect 18 Workstations but the last 12 will be out from the backup procedure.

Which is the solution?

Easy one, just Buying a new license pack.

Why you should have to buy new licenses?

There are at least three good reasons:

a) It is possible to manage your workstation architecture directly from VBR console.
b) There are more restore options.
c) Veeam support.

Gems:

1) One of the good news about the licensing of VBR v.10 is that from now on the license bought will be added to the gifted one. It means that you buy just a VUL packet (10 Vul) and have a total of 16 licenses.

2) If you have two license files (the first for VBR, the second for Veeam Agent) you have to merge them in just one license file.

The License rule and the procedure are available reading the following links https://www.veeam.com/kb3085  (rule)                         https://www.veeam.com/kb3116    (procedure)

3) Is there a turnback procedure?

Yes, please refer to the following KB https://www.veeam.com/kb2235 and contact Veeam’s support.

To Remember:

4) It’s not possible to protect VM with sockets and VUL license. It means that sockets license has the priority to protect VM with respect to VUL.

https://www.veeam.com/it/availability-suite-faq.html

5) Product comparison edition

https://www.veeam.com/it/products-edition-comparison.html

Take care

Veeam VBR DB Moving with SQL – Management Studio

In these last days, I had enough time to analyze my personal lab performances.

For testing purposes, I launched the backup of the whole architecture at the same time; the VM that suffered more was the backup server (VBR) and in particular the SQL Service.

This article will explain the steps I followed to move the VBR Database SQL Express from Backup & Replication to a SQL Server standard using SQL-Management Studio as a migration tool.

Before continuing reading the article, please watch at the following Veeam KBs and contact the Veeam Support

To make the description easier I’ll use the following acronyms :

  1. VBR = Backup Server
  2. SQLServer = Target Server where SQL Standard is installed
  3. SQLExpress = Source DB
  4. DB = VeeamBackup
  5. DBFile = VeeamBackup.mdf & VeeamBackup.ldf

The main steps to get the goal are:

  1. Stopping the Veeam service on VBR server
  2. Detaching DB from SQLExpress
  3. Copying DBFile from VBR to SQLServer
  4. Attaching DB to SQL Server
  5. Using the Veeam Migration tool
  6. Changing the service account name on VBR Service (optional)
  7. Checking up the register key
  8. Launching Backup and Restore tests

Let’s go!

  1. The first step is quite easy. Just connect to VBR, click on service and stop the SQL instance (Picture 1).

Picture 1

2. The second step is detaching the DB from SQL Express using SQL Management Studio (Picture 2).

Picture 2

If you need a good and short video guide to install SQL Management Studio please refer to the following link:

Another interesting video guide to understand how to enable the remote connection with SQL server is available here

Remember:  for enabling SQL Server to talk via Network (1433 is default port) you also have to set-up the firewall correctly.

3. Now it’s time to copy DBFiles from VBR to SQLServer

Pay attention to the default path where the files have to be copied and pasted.

Generally, it is in C:\Program Files\Microsoft SQLServer\ MSSQL.xx.INSTANCENAME\MSSQL\DATA (Picture 3).

Picture 3

4. Next step is attaching the DB to the new SQL server following the easy SQL Management studio menu ((Picture 4).

Picture 4

5. Now from the programs menu of VBR server, just select the voice Veeam and then “Configuration DataBase Connection Settings“.

Now choose which DBs you want to move to the new architecture. It can be Backup & Replication or the Enterprise Manager or both (Picture 5)

 

(Picture 5)

Now fill in the Database Name and Server/Instance and proceeding with the final step migration (Picture 6).

Picture 6

If everything is correctly configurated you have finally migrated your DBs.

Troubleshooting:

TS-1

If you see that the process runs out of time (600 seconds), it means that the VBR service account can’t access the database

How to solve it?

Please contact your DB experts before doing any tasks!!!

6. The first thing is creating a user able to manage the SQL services.

The procedure is quite easy using a Domain Controller (Picture 7-9)

Picture 7

Picture 8

Picture 9

Now you have to add the new user to Domain Users and Domain Admin groups (Picture 10-12).

Picture 10

Picture 11

Picture 12

From the Veeam Services window, select the Logon Service tab and set up the right user (and for all services that need it) (Picture 13 & 14)

Picture 13

Picture 14

Re-apply the procedure shown at point 5.

In my case, I’ve had another issue.

TS-2

The issue I unlucky met during my setup was the following:

When I tried to connect to remote DB with the “Configuration DataBase Connection Settings” command appeared the following error (Picture 15).

Picture 15

This issue happens when the SQL Server driver on a client computer that uses integrated security and the Windows security token, can’t connect to the SQL Server

If you want to have all details please refer to the following Microsoft article:

Cannot generate SSPI context

Please contact your DB experts before doing any tasks!!!

After some google research and test, I found a solution that addressed my issue always working with Domain Controller.

The AD console needs to be switched to advanced (Picture 16).

Picture 16

Now left-click on the SQL server and  select “attribute editor”

From this menu, you have to delete all the entries with the writing MSSQL.svc (Picture 17)

It also needs a server reboot.

Please contact your DB experts before doing any tasks!!!

Picture 17

For the last two points (7 and 8),  check-up, if the procedure followed, has solved the request.

TS-3

If you are not able to discover the SQL server, please check on the target Server if the SQL Server browser is up and running

Picture 18

 

From VBR Server open the register key (HKEY_LOCAL_MACHINE\ Software\Veeam\Veeam Backup and Replication) and check up if the items SqlDatabaseName, SQLinstanceName e SqlServerName are correctly filled in (Picture 19).

Do the same check-up for HKEY_LOCAL_MACHINE\Software\Veeam\Veeam Backup Catalog)  (Picture 20).

Picture 19

Picture 20

Now start backup Jobs and do some restore tasks to be sure that your Backup architecture is up and running.

In my case, the Backup Server can manage more tasks without any issue.

One more recommendation before ending the article:

Before doing any activities please read the official documentation and ask Veeam support