How does the Lift CLI migrate my data to Db2 Warehouse on Cloud?

If you're migrating data from your IBM PureData System for Analytics (Netezza) database, you will first need to locally extract a database table to a CSV file using `lift extract`. Then, you will transfer your CSV data file to the IBM Db2 Warehouse on Cloud landing zone using `lift put`. The IBM Db2 Warehouse on Cloud landing zone is a pre-allocated volume used for data loading & scratch. Finally, you will load the uploaded CSV data file into the engine using `lift load`. Once the load is complete, you can delete the data file using `lift rm`.

If you're migrating a set of CSV files, you'll follow a similar set of steps to above. You'll start by transferring your CSV data files to the Db2 Warehouse on Cloud landing zone using `lift put`. The Db2 Warehouse on Cloud landing zone is a pre-allocated volume used for data loading & scratch. Finally, you will load the uploaded CSV data file into the engine using `lift load`. Once the load is complete, you can delete the data file using `lift rm`.

Is there a maximum database size that I can migrate?

No, you can migrate any size database. But, keep in mind that the duration of your database migration depends on your network connection speed, the volume of uncompressed data that you need to move, and the hardware profiles of your source and target computers. In other words, mileage may vary.

What database artifacts can I migrate?

The Lift CLI will migrate your tables or CSV files to an IBM Cloud data target. If you need to migrate other database artifacts, such as tables, views, stored procedures, please use the IBM Database Conversion Workbench (see section above).

Where should I install the CLI package for best performance?

We recommend that you install and run the CLI from a machine that is network-close (minimal latency) to your database source. This will ensure that your data is extracted and staged faster in your on-premises environment, thus improves your overall end-to-end data migration time.

What are the on-premises storage requirements to extract my data?

For Linux and macOS, the minimum storage should be greater than or equal to the on-disk representation of your largest table (uncompressed).
For Windows, the minimum storage should be 2X greater than or equal to the on-disk representation of your largest table (uncompressed).

What if there isn't enough disk space on the Db2 Warehouse on Cloud for Analytics landing zone to fit my largest table?

You can run `lift df` to check for available disk space on the Db2 Warehouse on Cloud for Analytics landing zone. You can free up space by running `lift rm`. If you still don't have enough space, you can split your table into multiple file chunks and upload those individually using the `lift extract --size` option.

I love that Lift is so fast, but can I slow down my upload if I'm on a bandwidth-limited network?

Sure. You can use the `lift put --max-throughput` option to limit the throughput utilized by the data transfer.

I don't want to keep typing in my database credentials and other common options. Can I set these as environment variables or store someplace to reference?

Yes. You can set connection credentials as environment variables. You can also create a properties file, and place your database credentials and common options there. Take a look at `lift help <command ` to see a list of options that the Lift CLI supports.

Can I use IBM Lift CLI for data regulated under HIPAA?

IBM Lift CLI may be used to process Protected Health Information regulated under HIPAA if Client, as the data controller, determines that the technical and organizational security measures are appropriate to the risks presented by the processing and the nature of the data to be protected. IBM Lift CLI is not designed to process data to which additional regulatory requirements apply.

How do I configure Lift CLI to work with an HTTP proxy?

An environment configuration file called lift.environment located in the Lift CLI installation bin directory ( <Lift CLI install dir>/bin ) may be created with the following contents to add an HTTP proxy configuration:
For the proxy host use proxy.host=<hostname> . For the proxy port use proxy.port=<port number> . Both must be specified for the settings to take effect. If the proxy requires authentication, the Lift CLI uses basic authentication in the CONNECT request by providing proxy.user=<user> and proxy.password=<password> . Both must be specified for the authentication settings to take effect.
Example of <Lift CLI install dir>/bin/lift.environment contents (each property is on a new line):
proxy.host=myproxy.host
proxy.port=3029
proxy.user=myproxyuser
proxy.password=proxypassword

How can I add a new X509 certificate to the list of trusted certificates for the Lift CLI?

An environment properties file called lift.environment located in the Lift CLI installation bin directory ( <Lift CLI install dir>/bin ) may be created with the following contents to add a X509 certificate to be imported to the trust store.
For the proxy host use proxy.certificate.path=<fully qualified path to the X509 certificate file> .
The certificate is added with alias ibm-lift-imported-proxy-cert . If the alias already exists the file is not imported and must be manually removed prior to running the CLI again. You can remove the certificate using the Java keytool on the Lift CLI Java trust store in <Lift CLI install dir>/jre/lib/security/cacerts (i.e. keytool -delete -alias ibm-lift-imported-proxy-cert -keystore <Lift CLI install dir>/jre/lib/security/cacerts -storepass changeit )

Does Lift support General Data Protection Regulation (GDPR) readiness requirements?

Notice:
This document is intended to help you in your preparations for GDPR readiness. It provides information about features of IBM Lift that you can configure, and aspects of the product's use, that you should consider to help your organization with GDPR readiness. This information is not an exhaustive list, due to the many ways that clients can choose and configure features, and the large variety of ways that the product can be used in itself and with third-party applications and systems.
Clients are responsible for ensuring their own compliance with various laws and regulations, including the European Union General Data Protection Regulation. Clients are solely responsible for obtaining advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulations that may affect the clients' business and any actions the clients may need to take to comply with such laws and regulations.
The products, services, and other capabilities described herein are not suitable for all client situations and may have restricted availability. IBM does not provide legal, accounting, or auditing advice or represent or warrant that its services or products will ensure that clients are in compliance with any law or regulation.

Table of contents
1. General Data Protection Regulation (GDPR)
2. Product configuration for GDPR readiness
3. Data life cycle
4. Data collection
5. Data storage
6. Data access
7. Data processing
8. Data deletion
9. Data monitoring
10.Capability for restricting use of personal data

General Data Protection Regulation (GDPR)
General Data Protection Regulation (GDPR) has been adopted by the European Union (EU) and applies from May 25, 2018.

Why is GDPR important?
GDPR establishes a stronger data protection regulatory framework for processing of personal data of individuals. GDPR brings:
•New and enhanced rights for individuals
•Widened definition of personal data
•New obligations for processors
•Potential for significant financial penalties for non-compliance
•Compulsory data breach notification

Read more about GDPR
• (EU GDPR Information Portal) [ https://www.eugdpr.org/]
• (ibm.com/GDPR website) [ http://ibm.com/GDPR]

Product configuration for GDPR readiness
The following sections provide considerations for configuring IBM Lift to help your organization with GDPR readiness.

Configuration to support data handling requirements
The GDPR legislation requires that personal data is strictly controlled and that the integrity of the data is maintained. This requires the data to be secured against loss through system failure and also through unauthorized access or via theft of computer equipment or storage media.

Configuration to support data privacy
IBM Lift is a tool that migrates data from a supported on-premises data center (source) to an IBM Cloud data property (target). Most of the configuration to support data privacy is enabled on the source and target data repositories. Follow the guidelines provided by those offerings to ensure the best configurations to meet data privacy requirements. Although the user specifies which data sets are moved by Lift and defines the source and targets, Lift itself is not configurable in a way that affects how the tool moves data. Lift can be used through either the IBM Integrated Analytics System (IIAS) console or Lift CLI.
When using Lift CLI, ensure that access to it is controlled. Lift CLI does not save credentials to the source or target systems. However, you can opt to save access credentials to the source or target repositories in a configuration file so that the credentials do not need to be entered explicitly for each Lift command that is issued. If the credentials are saved in a configuration file, access to the file should be properly controlled. Any instance of Lift CLI on a local system might have been used to generate files that were extracted from source systems that contain personal data.
When using the IIAS console, the user automatically has access to the source repository and might opt to save access credentials for the target repository. The credentials for the target repository that are saved by the IIAS console are available to anyone who uses the same IIAS user ID and password. This allows someone to send new data or overwrite existing information in the target system. The user ID and password of the target system can also be used to access the target system directly. Access to specific areas of the IIAS source repository can be restricted to different users. Certain functions performed in the IIAS console cause data files from the IIAS source repository to be saved to the local file system. Proper measures to control the access to the directories where these data files are stored is key to ensuring data privacy.

Configuration to support data security
Lift is preconfigured to support data security; thus there is no additional configuration required. When using IIAS console, the user can opt to save access credentials to the target system using the IIAS console with Lift. The credentials to the target system are saved securely in a repository within IIAS. The IIAS console credentials are transmitted in encrypted form (token) between components within the on-premises IIAS. The encrypted token is a one-time token with expiration. The token is decrypted with a key that is stored within the IIAS system, and used as credentials to invoke the IIAS REST API. The credentials for the source (IIAS) are securely stored and managed by IIAS backend. The user id and password can be retrieved via the IIAS REST API. No credentials are stored on the client side as browser cookies. The credentials are isolated to the current session, in which the user would be carrying out Lift functions.

Data life cycle
GDPR requires that personal data is:
• Processed lawfully, fairly, and in a transparent manner in relation to individuals.
• Collected for specified, explicit, and legitimate purposes.
• Adequate, relevant, and limited to what is necessary.
• Accurate and, where necessary, kept up to date. Every reasonable step must be taken to ensure that inaccurate personal data is erased or rectified without delay.
• Kept in a form that permits identification of the data subject for no longer than necessary.

What is the end-to-end process that personal data go through when using our offering?
Lift does not collect, process, or store customer personal data.
It merely provides a mechanism to move data from supported database sources to selected targets using IBM Aspera for high-speed transport over a wide area network. Safeguards for data privacy can be managed at the source or target repository.
Personal data used for online contact with IBM
IBM Lift clients can submit online comments, feedback, or requests to contact IBM about IBM Lift subjects in a variety of ways, primarily:
• IBM ServiceNow support portal: https://watson.service-now.com/wcp
• Feedback on the IBM Lift product page: https://www.lift-cli.cloud.ibm.com/ using the blue "Feedback" tab
• Public comments area on pages in the IBM Lift community on IBM developerWorks
• Public comments area on pages of IBM Lift documentation in IBM Cloud Docs
• Public comments in the IBM Lift space of dWAnswers
Typically, only the client name and email address are used to enable personal replies for the subject of the contact. The use of personal data conforms to the [IBM Online Privacy Statement] ( https://www.ibm.com/privacy/us/en/). Customers are advised not to directly add or upload, to any of these systems, files that contain personal data.

Data collection
When assessing your use IBM Lift and the requirements of GDPR, you should consider the types of personal data which in your circumstances are passing through IBM Lift.
Lift does not collect, process or store customer data. Other sections in this document provide more information about how Lift orchestrates data migration, and help inform customers about how to use Lift. Product usage metrics are collected to help improve the user experience and interactions with users. This is done in accordance with the IBM Online Privacy Policy, available at http://www.ibm.com/privacy/

Data storage
Lift itself does not store any customer personal data. The following data storage mechanisms are used by IBM Lift, which users may wish to consider when assessing their GDPR readiness.
Depending on the usage, Lift might extract data from the original source repository and store that data in a local temporary staging area. Lift might also migrate data in CSV files which are stored locally. While Lift doesn’t provide any mechanism to encrypt the data at rest, customers can take measures to control access and encrypt data for their local systems. After moving the data to the target using Lift, a temporary landing zone can be used before loading data into the final target destination. During data movement, temporary files might be created in these staging areas both near the source and target. Care should be taken to manage, restrict, or clean up these areas when that data is no longer needed. No other data storage occurs while migrating data using Lift.

Data access
Although Lift does not store any data, it does provide a means to connect to data sources.
After Lift is installed, each user who requires access to it must be authenticated on the local environment. There is no support for finer-grained access control to Lift.
A Lift user still requires access credentials for the source data repository. If those credentials are stored in a configuration file, steps must be taken to prevent unauthorized access to that file. Steps must also be taken to prevent unauthorized access to other files on the local system, including those containing data extracted from previous data transfers. The onus is on the customer to ensure that the local environment on which Lift is installed, including temporary storage directories, is not accessible to unauthorized users.

Data processing
What happens to the data?
Lift uses a 3-step process to migrate data: extract, put, and load.
Data extraction is performed from the source data repository (such as IBM PureData System for Analytics (PDA), formerly called Netezza) to a temporary landing zone where Lift is installed. Lift is installed in a local environment that is usually near the source data. The user determines where Lift is installed and what controls are in place to secure this local environment. When data is being moved (put) by Lift and in transit, it is always encrypted using the AES-256 encryption algorithm. (See https://asperasoft.com/technology/transport/fasp/fasp-security-model/ for more information.) Data landing in the IBM Cloud target is staged in a temporary area that is encrypted when the data is at rest. The final step in data movement is the loading of data from the staging area that is associated with the IBM Cloud target into the actual provisioned data repository.

Data deletion
IBM Lift does not support deletion of customer data in the source repository.
However, Lift can append or replace information in the target repository by overwriting what is there.
Lift can also invoke commands to delete temporary files in the landing zones at the target. Otherwise, data stored in temporary staging areas, both at the source and target, should be access controlled, encrypted, and cleaned up if no longer needed. Since these systems are outside the scope of Lift, follow the guidelines provided by those offerings or your own internal controls to ensure the best configurations to meet your data privacy requirements.IBM Cloud target into the actual provisioned data repository.

Data monitoring
Customers should regularly test, assess, and evaluate the effectiveness of their technical and organizational measures to comply with GDPR. These measures should include ongoing privacy assessments, threat modeling, centralized security logging, and monitoring.
Logging capabilities embedded within Lift provide a means to help troubleshoot product issues and record certain usage. The logs generated are stored on the local environment indefinitely and can be sent to IBM to help with problem determination should the customer need assistance. The logs generated are system logs that capture events, commands, error codes, and messages that do not contain any customer data. Lift does not maintain and monitor logs of security events, or support threat detection and investigation activities.

Capability for restricting use of personal data
Using the facilities summarized in this document, IBM Lift enables a user to restrict usage of any technical data within the platform that is considered personal data.

Can the Lift CLI be installed directly on PureData System for Analytics?

Yes, the Lift CLI can be installed on the PureData System for Analytics, but additional storage must be attached to provide the extracted data sufficient staging disk space.

What are the best practices for attaching and configuring additional storage for PureData System for Analytics?

The following tech notes provide steps to mount SAN/NFS systems on PureData Systems for Analytics. They also include best practices for attaching and configuring additional storage for PureData System for Analytics.

1. Adding SAN Storage to PureData Systems for Analytics:
http://www-01.ibm.com/support/docview.wss?uid=swg21700900
2. IBM PureData System for Analytics Mounting NFS on the appliance:
http://www-01.ibm.com/support/docview.wss?qid=63d511603b6e6fb3a0216ea69fc0d500&uid=swg21568933
3. Mounting NFS filesystem on PureData for Analytics systems:
http://www-01.ibm.com/support/docview.wss?qid=a9cf4df18f89517d8338b99a8743f522&uid=swg21971589

On what part of my on-premises system should I install the Lift CLI if I'm using IBM PureData System for Analytics sources?

When you are installing the Lift CLI for PureData System for Analytics sources, install the Lift CLI on your "injection" system (the system that you use to stage data to load into the PureData System for Analytics database). That system will have good connectivity to the PureData System for Analytics and will have lots of disk space for staging data. However, if your injection system is already fully loaded, then install the Lift CLI on a similar system that is similarly connected and with significant staging disk space.

Where should I install the Lift CLI for the best throughput if I'm using IBM PureData System for Analytics as my source?

We strongly recommend that you install the Lift CLI on a Linux machine. When your Lift CLI is installed on a Linux machine, data extraction from PureData System for Analytics sources is done with high speed unload facilities. Install your Lift CLI on Linux for significantly better overall throughput when your source is PureData System for Analytics.

Where should I install the Lift CLI for the best throughput if I'm using IBM Db2 for Linux UNIX and Windows as my source?

We strongly recommend that you install the Lift CLI on a Linux machine with Db2 client installation. Installing Lift CLI on your Linux machine with Db2 client significantly improves overall throughput. When Lift CLI fails to detect Db2 client, different extract strategy is used and you may notice reduced extract throughput. A message on console "Lift is extracting data at sub-light speeds. You can improve extraction time by installing and configuring the Db2 client. For more information, visit http://ibm.biz/BdZcFp, is printed when Db2 client is not available on your machine where Lift CLI is installed.

What are the Db2 client requirements to extract my data from IBM Db2 for Linux UNIX and Windows with better performance?

Prerequisites for Lift CLI to use Db2 client:
1. db2 (for Linux) or db2cmd (for Windows) command have to be available in PATH.
      • Linux : <INSTANCE_OWNER_HOME>/sqllib/db2profile has to be applied to environment before executing
    Lift CLI
      • Windows : The Db2 client has to be set as default instance
2. The OS user has to be included in SYSADM group of the Db2 client instance if Lift CLI is run remotely from IBM Db2 for Linux UNIX and Windows server.
3. Version of the Db2 client has to be same or higher than version of IBM Db2 for Linux UNIX and Windows server.

Can I migrate table that has hidden columns from Db2 source database?

Yes. Extracting hidden columns is supported by Lift CLI. By default, hidden columns are not included in the extracted CSV file. If you want to include hidden columns in the extracted data, explicitly specify all column names along with hidden column names by using column selection option. Refer `lift extract --help` for more information on column selection option

What are the Oracle client requirements to extract my data from an Oracle Server with better performance?

The following are prerequisites for Lift CLI to use the Oracle client:
1. Install the basic and tools modules of the Oracle client
2. Ensure the exp program path is added to the PATH environment variable
3. Depending on your environment, you might need to add the Oracle client library path to the operating system library path (for example, LD_LIBRARY_PATH)
4. The Oracle client version must be the same or later than the Oracle server version
    Note : You do not need to pre-configure the source database connection from the Oracle client.

Can I migrate a table that has international characters that include a multi-byte character set?

Yes. Lift CLI uses a UTF-8 code page and supports data conversion from commonly used Oracle character sets.

The following is a list of supported Oracle Database Character Sets (NLS_CHARACTERSET):
AL32UTF8, AR8ISO8859P6, AR8MSWIN1256, BLT8ISO8859P13, BLT8MSWIN1257, CL8ISO8859P5, CL8MSWIN1251, EE8ISO8859P2, EE8MSWIN1250, EL8ISO8859P7, EL8MSWIN1253, IW8ISO8859P8, IW8MSWIN1255, JA16EUC, JA16EUCTILDE, JA16SJIS, JA16SJISTILDE, KO16MSWIN949, NEE8ISO8859P4, TH8TISASCII, TR8MSWIN1254, VN8MSWIN1258, WE8ISO8859P15, WE8ISO8859P9, WE8MSWIN1252, ZHS16GBK, ZHT16HKSCS, ZHT16MSWIN950, ZHT32EUC, ALE16UTF16

You can determine the character set of the source Oracle database by using the following SQL query:
SELECT PROPERTY_VALUE FROM DATABASE_PROPERTIES WHERE PROPERTY_NAME = 'NLS_CHARACTERSET';

The Oracle client is configured on my Lift CLI machine, but Lift CLI displays a message like "Lift is extracting data at sub-light speeds. The source table contains data types that reduce the speed at which Lift can extract data." Why can't I extract data at faster speeds?

Lift CLI uses a degraded mode if the source table has any of the following data types:
BINARY_FLOAT, BINARY_DOUBLE, LOB, BLOB, CLOB, BFILE, XML, ROWID, UROWID
Each table is evaluated separately. This degraded mode impacts only the current table extraction operation due to the presence of the affected data types.

How do I use SSL to connect Lift CLI to an Oracle server?

Use Oracle's "TCP/IP with SSL" protocol, also called TCPS, to establish a connection between Lift CLI and an Oracle server. The procedure for doing this depends on whether you use an Oracle client. If so, Lift CLI must be installed on the same machine as that client.
• If you use an Oracle client, you must create an Oracle wallet for it and import a server certificate to that wallet before establishing the connection:
    1. On the Oracle server, issue the following command to export a server certificate from the server wallet to a certificate file:
    orapki wallet export -wallet <server_wallet_name> -pwd <password> -dn <domain> -cert <certificate_file_name>

    2. If the orapki command is available on the Oracle client machine:
      a. Copy the certificate file from Step 1 to a directory on the Oracle client machine.
      b. On the Oracle client machine, issue the following command to create a client wallet:
      orapki wallet create -wallet <client_wallet_name> -auto_login -pwd <password>
      c. On the Oracle client machine, issue the following command to add the server certificate to the client wallet:
      orapki wallet add -wallet <client_wallet_name> -pwd <password> -trusted_cert -cert <certificate_file_name>

    If the orapki command is not available on the Oracle client machine:
      d. On the Oracle server machine, issue the following command to create a new wallet:
      orapki wallet create -wallet <wallet_name> -auto_login -pwd <password>
      e. On the Oracle server machine, issue the following command to add the server certificate to the new wallet:
      orapki wallet add -wallet <wallet_name> -pwd <password> -trusted_cert -cert <certificate_file_name>
      f. Copy the wallet from the Oracle server machine to the Oracle client machine. It is now the client wallet.

    3. To establish the connection, issue the following lift extract command, which includes both -ssl (or --ssl-connection) and -owl (or --oracle-wallet-location) parameters:

• If you do not use an Oracle client, or if Lift CLI runs on the same machine as the Oracle server, a wallet and server certificate are not needed. To establish the connection, issue the following lift extract command: 
      lift extract -ss <schema> -st <table> -ssl

Do I need to set up Microsoft SQL Server Tools or client to use Lift CLI?

No. Lift CLI doesn't need any other additional setup of tools for data migration.

When I use the DDL or db2look command, how can I migrate objects that have dependencies?

Lift CLI creates, in the source database, target objects that are ordered by their creation time. It detects object dependencies and generates DDL statements for any dependent objects that were created before the objects that depend on them. However, you must handle the following exceptional cases manually:

a. A required object that is not included in the DDL operation
b. A required object that is in another schema.>
c. A dependent object that was changed by an ALTER statement after objects that depend on it were created.

When I use the DDL or db2look command, can I migrate a table that contains an identity column?

Yes. However, because the GENERATE ALWAYS definition of the identity column is migrated to the corresponding column of the target table, you will need to alter the target table column to disable GENERATE ALWAYS before loading the data into the target table.

When I use the DDL or db2look command to migrate a table, is information related to the tablespace migrated to the target?

No, the migrated table at the target uses the tablespace of the target.

When I use the DDL or db2look command, how is the table organized (row or column) at the target?

The DDL for the target table will have the same ORGANIZE BY clause (row or column) as the source table.

When I use the DDL or db2look command, I need to modify some DDL statements. How can I do this?

You can modify the DDL statements in one of the following ways:

a. Using generate and execute commands
1. Use "lift ddl --generate" to generate the DDL statements.
2. Modify the generated DDL file as per your requirements.
3. Use "lift ddl --execute" to apply the generated DDLs on the target.

b. Using your own DDL file and execute
1. Save your modified DDL statements to a filename ending with .ddl extension.
2. Use "lift ddl --execute" to apply your modified DDL statements on the target.

I run the DDL or db2look command but it fails with the error message "The user does not have the necessary privileges". How can I fix this problem?

The privileges and authorities that are needed to run the DDL or db2look command is described in http://ibm.biz/BdZ5wE. Ask an administrator to grant the necessary privileges and authorities to you and run the command again. After the command has run successfully once, you can relinquish these privileges and authorities without affecting further DDL generation.