C&C Communicating-Processes View

C C Communicating Processes View

7.1 C&C Communicating-Processes View Packet 1: Ingest Subsystem

The Ingest subsystem deals with the initial reception of all data received at an EOSDIS facility and triggers subsequent archiving and processing of the data. It makes data ingest schedules available for access by users and programs and reports deviations from the schedules as exceptions so they can be handled through intervention by operations or science users.

7.1.1 Primary Presentation

7.1.2 Element Catalog

7.1.2.1 Elements and Their Properties

Properties of Ingest subsystem communicating-processes components and connectors, given in the following table, are

• Name
• Type
• Description

Element Name	Type	Description
EcInAuto	Process	Auto Ingest: Provides fundamental capabilities for data ingest into the SDPS, on receipt of a data availability notice (DAN). This process can be tailored for a specific interface. RPCs are used to request ingest services to schedule data transfer from the source. EcInAuto also Manages single requests at a time Invokes an RPC to the EcInReqMgr to begin request processing Checks DAN information Sends and receives data status messages
EcInPolling	Process	Polling Ingest Creates the appropriate polling request Detects new files of interest at tunable periods of time in either external or local disk locations by checking an agreed on network location for available data Creates a unique identifier for the request Submits requests Reports the status of the ongoing requests
EcInGUI	Process	Ingest GUI: provides Maintenance and Operations (M&O) personnel with the capability, via GUI Interface, to Perform physical media ingest: to ingest data from hard media Monitor the ingest history log, to monitor the status of ongoing ingest requests, to cancel ingest requests and granules, and to resume suspended ingest requests and granules Modify ingest configuration parameters
EcInReqMgr	Process	Request Manager Manages the ingest request traffic and the processing of the ingest requests Provides the capability to process multiple ingest requests concurrently by placing the request in a queue In the event of a failure, the EcInReqMgr process restores ongoing requests from the Ingest database.
EcIn Email GWServer	Process	Ingest E-mail Gateway Server Receives e-mail distribution notification messages Stores e-mail messages into files Detects new files of interest at a regular time interval, which can be configured, on a local disk Creates a polling request and puts it on a local disk location
EcInGran	Process	Granule Server provides services to perform the required data preprocessing and the subsequent data insertion into the appropriate Data Server. The preprocessing of data consists of Converting the data, if needed Extracting the metadata into the standard SDPS metadata format, if needed Performing required metadata existence and parameter range checks Updating the metadata with ingest-specific metadata, such as start and stop date/time for ingest EcInGran coordinates the ingest granule processing, including Performing data preprocessing Sending an insertion request to the appropriate Data Server Updating the granule state Transferring data files into Ingest Building file lists Grouping files with a valid ESDT
Sybase Server	Process	Stores and provides access to the INS internal data. In particular, the database stores the Ingest operations databases: Ingest History Logs and the Ingest request checkpoint state and template information.
RPC	Synch call	Exchanges data status, detects and reads data, sends updates, deletions, and other instructions.
Notification	Event	Notifies of activity completion and data distribution. The two approaches to initiating event communication are called the push model and the pull model. The push model allows a supplier of events to initiate the transfer of the event data to consumers. The pull model allows a consumer of events to request the event data from a supplier. In the push model, the supplier is taking the initiative.
SQL	DB Query/Resp	Database query/response mechanism.
[etc.]	[etc.]	[etc.]

7.1.2.2 Relations and Their Properties

Attachment, showing how the components and connectors are attached to each other. The primary presentation shows all attachments.

7.1.2.3 Interfaces

Interfaces for the elements shown in this view are specified under the corresponding element in the module decomposition view (Volume II, Chapter 1). To identify those elements, consult Volume I, Chapter 4.

7.1.2.4 Element Behavior

[omitted]

7.1.3 Context Diagram

7.1.4 Variability Guide

None.

7.1.5 Architecture Background

7.1.5.1 Design Rationale

• The subsystem supports the ingest of data into ECS repositories on a routine and ad hoc basis. Given the variety of possible data formats and structures, each external interface and each ad hoc ingest task may have unique aspects. Therefore, the Ingest subsystem is organized into a collection of software componentsingest management software, translation tools, media-handling softwarefrom which those required in a specific situation can be readily configured. The resultant configuration is called an ingest client, which can either operate on a continuous basis to serve a routine external interface or exist only for the duration of a specific ad hoc ingest task.
• Ingest subsystem software services are designed to support fault tolerance, graceful degradation, and effective recovery, such as automatic switchover to backup resources when primary resources fail; flexibility to handle various types of quality assurance for products, on-site as well as off-site; and retaining production history to allow reprocessing in case of data loss.

[etc.]

7.1.5.2 Results of Analysis

None.

7.1.5.3 Assumptions

• A DAAC will often preprocess ingested ancillary data, correlative data, documents, and so on into another format for the purposes of processing. This will result in data represented in both the original native format and the preprocessed format.
• Only preprocessed data will be archived unless there is no means by which the native data could be reacquired.

[etc.]

7.1.6 Other Information

[omitted]

7.1.7 Related View Packets

• Parent: None in this view. View packets in other views showing the Science Data Processing Segment (SDPS) are beyond views parents.
• Children: None
• Siblings

  - C&C Communicating-Processes View Packet 2: Data Server Subsystem (Volume II, Section 7.2, page 456)

  - C&C Communicating-Processes View Packet 3: Data Management Subsystem (Volume II, Section 7.3, page 456)

  - C&C Communicating-Processes View Packet 4: Client Subsystem (Volume II, Section 7.4, page 456)

  - C&C Communicating-Processes View Packet 5: Interoperability Subsystem (Volume II, Section 7.5, page 456)

  - C&C Communicating-Processes View Packet 6: Planning Subsystem (Volume II, Section 7.6, page 456)

  - C&C Communicating-Processes View Packet 7: Data Processing Subsystem (Volume II, Section 7.7, page 456)

  - C&C Communicating-Processes View Packet 8: Communications Subsystem (Volume II, Section 7.8, page 456)

  - C&C Communicating-Processes View Packet 9: System Management Subsystem (Volume II, Section 7.9, page 456)

  - C&C Communicating-Processes View Packet 10: Internetworking Subsystem (Volume II, Section 7.10, page 456)

7.2 C&C Communicating-Processes View Packet 2: Data Server Subsystem

[omitted]

7.3 C&C Communicating-Processes View Packet 3: Data Management Subsystem

[omitted]

7.4 C&C Communicating-Processes View Packet 4: Client Subsystem

[omitted]

7.5 C&C Communicating-Processes View Packet 5: Interoperability Subsystem

[omitted]

7.6 C&C Communicating-Processes View Packet 6: Planning Subsystem

[omitted]

7.7 C&C Communicating-Processes View Packet 7: Data Processing Subsystem

[omitted]

7.8 C&C Communicating-Processes View Packet 8: Communications Subsystem

[omitted]

7.9 C&C Communicating-Processes View Packet 9: System Management Subsystem

[omitted]

7.10 C&C Communicating-Processes View Packet 10: Internetworking Subsystem

[omitted]