1. RAPID DEPLOYMENT AND PREDICTABLE PERFORMANCE : The Greenplum Data Integration Accelerator is a purpose-built, open systems data accelerator

that architecturally integrates Greenplum data loading software (gpfdist), server, storage and networking into a single, easy-to-implement system. The packaging and pre-tuning ensures predictable performance, while dramatically simplifying your data loading activates, resulting in reduced administration overhead.

2. TIGHTLY INTEGRATED WITH THE DATA COMPUTING APPLIANCE FAMILY: The DIA was designed for tight integration with the DCA family of data warehousing and analytic appliances. Removing the need for custom solutions and non-supported hardware, the DIA enables an end-to-end solution with a single support and management infrastructure. By leveraging a common 10 GB/s Ethernet network, the DIA enables the fastest data loading directly into the DCA segment servers.

3. ENGINEERED FOR PARALLEL EXECUTION OF DATA LOADING: The DIA, combined with the Greenplum DCA manages the flow of data into all nodes of the

appliance using the EMC Greenplum’s MPP Scatter/Gather Streaming™ (SG Streaming) technology. The system uses a “parallel-everywhere” approach to loading, in which data flows from all the nodes on the DIA to every segment server of the database without any sequential choke points. The combined solution achieves loading speeds of more than 10 terabytes per hour, two- to five-times faster than other appliance solutions.

4. ENTERPRISE HIGH AVAILABILITY: The Greenplum DIA is a system that meets the reliability requirements of the most mission- critical enterprises with data availability consisting of RAID protection at the disk level. This provides no data loss when losing a disk within any server.

5. GREENPLUM PERFORMANCE MONITOR: The DIA is managed via the Greenplum Performance Monitor application that provides a single view of the Data Computing Appliance and the Data Integration Accelerator from a single management console. The system includes Secure Remote Support (call home) and provides email and SNMP notification in the case of any event needing attention.

6. PROACTIVE EMC ONE SUPPORT STRUCTURE: EMC Customer Support Services provides resources and services to quickly and proactively resolve solution-related issues and questions to ensure business continuity and a highly- available data environment. EMC’s global maintenance and support is available

around-the-clock via comprehensive online support tools including Live Chat and online service request management, Secure Remote Support (call home), live telephone support, and onsite support through the industry’s leading global field service organization.

Available in three configurations:

 

You can configure the DIA servers as nodes in an Informatica PowerCenter Grid, to make use of the many beneficial features of the Informatica Enterprise Grid Option, such as:

1.  adaptive load balancing

2.  high availability

3.  dynamic partitioning

The Informatica Enterprise Grid Option is an extra-cost option which you can separately purchase from Informatica.

When you configure the DIA servers into a grid, you can configure workflows and sessions to run on the grid. You can run a workflow, or a session on the grid.

When you run a workflow on a grid, the PowerCenter Integration Service runs a service process on each available DIA server in the grid, to load balance, and increase the performance and scalability of the computing processes.

When you run a session on a grid, the Integration Service distributes session threads to multiple Data Transformation Manager (DTM) processes on DIA

servers in the grid.

To set up the DIA servers as a PowerCenter Grid, you use the Administrator’s Console, and create a new Grid option for the domain you are using (see figure below.)

Note: To run workflows on a PowerCenter grid, you must have the Server grid option. To run sessions on a grid, you must have the Session on Grid option.

The Enterprise Grid option is useful when handling mission-critical,enterprise-wide data integration from a single platform. An example of where a grid may be useful is when a workflow has many  sessions that must be completed at the same time. Running this workflow on a single node will force it to run serially, sacrificing performance and usability. When this workflow is run on a grid, the sessions can be distributed to all the available nodes in the grid, according to the computing and memory resources required, ensuring maximum performance and maximum return of investment.

Once the Integration servers are set up, you can read and write Greenplum database data using an ODBC (Open DataBase Connectivity) driver. For example, you may use the Progress Software Data Direct ODBC drivers to fetch meta data or write data to a Greenplum database, or you may use an open source psqlODBC PostgreSQL ODBC driver. The Greenplum connectivity package also provides the psqlODBC drivers for use with the PWX adapter. There are some commercial PostgreSQL ODBC drivers that may be suitable as well. All will perform adequately for read and write operations to Greenplum databases.

For the best data loading performance, EMC suggest you consider using the Greenplum PWX Adapter. The PWX Adapter will call the gpload utility, which is another EMC Greenplum utility. gpload will read the format of the source file, and create a control file using YAML syntax, and start a gpfdist session. It will then create an external table, and insert the data into the Greenplum target using gpfdist.

In order to use the PWX Adapter, you will need to install at the DIA server:

The DIA servers are well-suited to host a number of industry data loading tools. These tools perform the tasks of Extract, Transform, and Load (ETL) of source data into Greenplum databases and data warehouses.

Informatica™ is a popular ETL tool. A large percentage of EMC’s customers are Informatica users. The DIA servers serves as the Integration service servers (RedHat Linux hosts.)

gpfdist is Greenplum’s parallel file distribution server utility software. It is used with read-only external tables for fast, parallel data loading of text files into a Greenplum data warehouse.

gpfdist can be considered as a networking protocol, much like the http protocol. Running gpfdist is similar to running a HTTP server. It exposes the target file via TCP/IP to a local file directory containing the files. The target files are usually delimited files or CSV files, although it can also read tar and gziped files. In the case of tar and gzip files, the PATH must contain the location of the tar and gzip utilities.

For data uploading into a Greenplum database or data warehouse, you generate the flat files from an operational database or transactional database, using export, COPY, dump, or user-written software, depending on your business requirements. This process can be automated to run periodically.

gpfdist is installed on each DIA server by default. If it is not already installed, there are two ways you can install it:

1. copy gpfdist from another source to the DIA server, and put the location of the file into the PATH environment variable.

2. download and install the Greenplum loaders package. Select the loaders package that is appropriate to your DCA database version.

For example, if your DCA Greenplum database is of version 4.0.5.0, you will want to download and install the Greenplum loaders package: Greenplum-loaders-4.0.50.0-build-8-RHEL5-x86_64.bin.

For EMC customers, the Greenplum loaders packages are available for download at the EMC Powerlink web site. (go to https://emc.subscribnet.com and click the Greenplum link.) 

gpfdist runs in a client-server model. The DIA server acts as the server, while the master server of the DCA is the client. You start the gpfdist process on a DIA server, by indicating the directory where you drop your source files. Optionally, you may also designate the TCP port number to be used.

A simple startup of the gpfdist server is the following command syntax:

gpfdist –d <file_files_directory> –p <port_number> –l <log_file> &

For example,

In the above example, we create an external table. This table has all the attributes of a table called ‘performance_table’ (using the like performance_table clause), and makes uses of flat files stored in the host ‘etl3’, using port 8887. The flat file name is expected to be ‘performance_test.dat’.

Since the DIA servers are RedHat Linux hosts, they can also be configured as hosts for data integration software, such as Informatica, Talend, and Pentaho.

Now let's discuss about common questions asked by the customers such as,

EMC Greenplum Data Integration Accelerator (DIA), is used for fast data loading, using the included gpfdist utility, as well as using a popular data integration (DI) tool called Informatica. 

For customers interested in using Informatica as a DI tool, we will explain the reference architecture and how the tool is integrated into the DIA. There is a brief section on using the Greenplum PWX Adapter with the DIA and Informatica DI tool, and finally, how the DIA servers can serve as nodes in the Informatica Enterprise Grid.