SFTP Folders: Standard Configurations

For the functions in the AIR platform which require file transfer, we use an SFTP system. This platform has a series of folders for different functions in the platform.

General Principals

Below are general principles applied to our ingestion jobs and SFTP folder structures.

All files sent to AIR are subject to our Data Retention Policies.
Even though not all jobs require all the folders, they are created to be consistent.
No files should be placed directly in the root.
There should be no cross-environment data transfer. E.g. production environments sending data to the sandbox or vice-versa.

Standard Sub-Directories

Inside a root-level folder, all directories will contain the below sub-folders. These are used for job management and file storage.

inbound

Files are placed here by a client sending data into the air platform.

outbound

Files here are files being sent out from AIR for a client to download/pick up.

processed

When a job starts processing a file, the file from the inbound directory is moved to this folder. This serves two purposes:

Stop another job from processing the same file
Prevent duplicate file processing. If a file in the inbound directory already exists with the same file name in the processed directory, the file is moved to the error folder and not ingested.

error

Files are moved to the error directory if there is a major issue with the entire file. E.g. badly structured, unreadable, wrong file encoding etc. Please see the individual ingestion jobs for the required formatting of files.

logs

Where an ingestion job generates a processing log file, this will be placed here.

Folders

ℹ️
In this table, <bannerName> refers to the incomingIdentifier for the banner unit

Functionality Name	Folder Structures	Notes
Store Ingestion	/stores/`<bannerName>`/inbound /stores/`<bannerName>`/outbound /stores/`<bannerName>`/processed /stores/`<bannerName>`/error /stores/`<bannerName>`/logs
Products Ingestion	/products/`<bannerName>`/products/inbound /products/`<bannerName>`/products/outbound /products/`<bannerName>`/products/processed /products/`<bannerName>`/products/error /products/`<bannerName>`/products/logs	`<bannerName>` should always be provided This ingestion must always sit alongside the Product Groups ingestion as noted below
Groups Ingestion	/products/`<bannerName>`/groups/inbound /products/`<bannerName>`/groups/outbound /products/`<bannerName>`/groups/processed /products/`<bannerName>`/groups/error /products/`<bannerName>`/groups/logs
Offer Allocation	/offer-allocation/inbound /offer-allocation/outbound /offer-allocation/processed /offer-allocation/error /offer-allocation/logs
Recommendation Ingestion	/recommendations/inbound /recommendations/outbound /recommendations/processed /recommendations/error /recommendations/logs
Plan Members Report	/plan-members/inbound /plan-members/processed /plan-members/error /plan-members/outbound	Please note this ingestion is often the feed into the Plan Refresh job so the outbound folder for this job, where applicable, is set up to be the inbound folder for the plan refresh job.
Plan Refresh	/plan-refresh/inbound /plan-refresh/outbound /plan-refresh/processed /plan-refresh/error /plan-refresh/logs
Points Expiry	/points-expiry/inbound /points-expiry/outbound /points-expiry/processed /points-expiry/error /points-expiry/logs
Joiners/Leavers	/joinersleavers/inbound /joinersleavers/outbound /joinersleavers/processed /joinersleavers/error /joinersleavers/logs
Reporting Jobs	/reporting/`<jobName>`/outbound	As these jobs to not take an inbound file there is only an outbound directory required. Each reporting job should be set up into its own folders.
Backend Points	/backend-points/inbound /backend-points/outbound /backend-points/processed /backend-points/error /backend-points/logs
Manual File Transfer	/manual/`<jobNameWithDateTime>`/	For ad-hoc manual file transfers, new folders will be created inside this for each file transfer. The folder name will include any detail to itdentify the job E.g. ticket number, short description and the date/time.

Updated about 4 hours ago